SitePoint Sponsor

User Tag List

Results 1 to 3 of 3
  1. #1
    SitePoint Zealot w1nk5's Avatar
    Join Date
    Aug 2009
    Location
    Ontario, Canada
    Posts
    115
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Importing Posts to WP using API

    Below is the code I am using to pull data from an API to import it into my WordPress database. Getting the data from the API works fine and everything is gravy on api calls that return fewer results. However, with this particular function it is returning just over 9000 and breaking during the foreach loop that goes through each returned element; sometimes after 2500, sometimes after 5000. Also, as you can see I set the max execution time for PHP to 5 minutes but it stops just after a minute and doesn't return an error.

    This ones driving me crazy! Any help would be greatly appreciated.

    Cheers

    PHP Code:
    /**
     * Import vehicles
     */
    function n4s_import_vehicles() {

        
    ini_set('max_execution_time'300);

        
    //Initialize curl
        
    $ch curl_init(); 

        
    curl_setopt($chCURLOPT_URL'http://api.curtmfg.com/v2/GetVehicle?year=&make=&model=&style=&dataType=JSON');
        
    curl_setopt($chCURLOPT_CONNECTTIMEOUT5);
        
    curl_setopt($chCURLOPT_RETURNTRANSFERtrue);
        
    $ch_data curl_exec($ch);
        
    curl_close($ch);  

        if(!empty(
    $ch_data)) {

            
    $i 0;

            
    $json_data json_decode($ch_datatrue); //print_r(count($json_data)); exit();
            
    foreach($json_data as $data) {

                
    $post_title $data['year'] . ' ' $data['make'] . ' ' $data['model'] . ' ' $data['style'];

                
    $post = array(
                    
    'post_type' => 'vehicle',
                    
    'post_status' => 'publish',
                    
    'post_title' => $post_title
                
    );

                
    $post_id wp_insert_post($post);

                
    $vehicle_year str_replace('.''-'$data['year']);
                
    $vehicle_make str_replace(' ''-'strtolower($data['make']));

                if(!empty(
    $post_id)) {

                    
    // Update Year
                    
    wp_set_object_terms($post_id$vehicle_year'vehicle_year');

                    
    // Update Make
                    
    wp_set_object_terms($post_id$vehicle_make'make');

                    
    // Update Model
                    
    update_post_meta($post_id'n4s-vehicle-model'$data['model']);

                    
    // Update Style
                    
    update_post_meta($post_id'n4s-vehicle-style'$data['style']);
                }

                
    $i++;

                if(
    $i == 1000) {
                    
    $wpdb->queries = array();
                    
    $wp_actions = array();
                    
    wp_cache_flush();
                }
            }
        }


  2. #2
    SitePoint Wizard bronze trophy
    Join Date
    Jul 2006
    Location
    Augusta, Georgia, United States
    Posts
    4,139
    Mentioned
    16 Post(s)
    Tagged
    3 Thread(s)
    Looking at the code I wonder if there is a way to save the data in chunks. From the looks of it at the very least 36,000 individual queries are going to be issued and that is a significant problem regardless of whether it works or not. I'm not familiar with WP but one would hope there is a way to update multiple rows/entities using a single query rather than issuing separate queries for every post. If not that is something your going to need to write because the way your doing it right now is going to be prone to failure not only short term but also long term as the data set scales.

  3. #3
    SitePoint Zealot w1nk5's Avatar
    Join Date
    Aug 2009
    Location
    Ontario, Canada
    Posts
    115
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Yeah, you're definitely right. I guess I'll head back to the WordPress codex and see if I missed something. If not, I'll have to write some custom functions.

    Thanks


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •