SitePoint Sponsor

User Tag List

Results 1 to 11 of 11
  1. #1
    SitePoint Enthusiast
    Join Date
    Aug 2007
    Posts
    28
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)

    Memory usage from recursive function

    I wrote a function that queries a database and limits the results to 100. Then it does some processing and checks to see if there are any more database entries to process. If there are more to process then it calls itself again. The problem that I am having it that it keeps using memory until it runs out and crashes. I have followed it in a debugger and with each iteration it overwrites existing variables and none of the grow in size. What should I start looking for to correct the memory usage problem? What are some suggested solutions?

  2. #2
    Follow Me On Twitter: @djg gold trophysilver trophybronze trophy Dan Grossman's Avatar
    Join Date
    Aug 2000
    Location
    Philadephia, PA
    Posts
    20,580
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    Any recursive function can be rewritten as an iterative function with loops. You might try that to avoid growing the call stack.

  3. #3
    SitePoint Enthusiast
    Join Date
    Aug 2007
    Posts
    28
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I have tow different version both of which appear to use the same amount of memory.

    The first is a do while that call the function outlined above. When the function is done, if there is more to do then it returns FALSE and the do while kicks off again.

    The second is iterative, they both appear to use the same amount of memory.

    If this is not what you mean above then can you provide an example?

  4. #4
    Follow Me On Twitter: @djg gold trophysilver trophybronze trophy Dan Grossman's Avatar
    Join Date
    Aug 2000
    Location
    Philadephia, PA
    Posts
    20,580
    Mentioned
    1 Post(s)
    Tagged
    0 Thread(s)
    No, that's it.

    Perhaps this might help?
    http://us2.php.net/mysql_free_result

  5. #5
    SitePoint Enthusiast
    Join Date
    Aug 2007
    Posts
    28
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I gave that a try but it still seems to be eating up memory.

    The code below is an import process that is kicked off by a do while. Do you see anything that could be eating up memory in the code?

    Code:
    <?php
    
    function import_notes($ss)
    {
      
      $query = 'SELECT ndb.X_CDBJOIN,ndb.`Create Timestamp`,ndb.Regarding,cdb.account_id,cdb.contact_id 
      FROM ndb join cdb on cdb.x_cdbjoin = ndb.x_cdbjoin where ndb.processed = 0 limit 100';
      
      $results = mysql_query($query);
      
      //echo mysql_error();
      
      $num_results = mysql_num_rows($results);
                
      if($num_results > 0)
      {
                //set purosoap variables.
               // $ss = new SugarSoap('http://localhost/' . 'soap.php?wsdl');
                    //$ss->setDebug(1);
                    //login to sugar via soap
                 //   $ss->login('admin','password','password');
          
          
          //check memory usage. If it gets to high then kill the script and start again          
          $allotted_mem = ini_get('memory_limit'); 
          
    
          //debugbreak();
          
          while($row = mysql_fetch_assoc($results))
          {
    
             /// debugbreak();
              //add the record to sugar via soap                 
                     
              if(isset($row['Regarding']) && $row['Regarding'] != "")
              {
                   //debugbreak();
                  $date = date("Y-m-d 00:00:00", strtotime($row['Create Timestamp']));
                  
                  $values = array
                  (
                    'name' => substr($row['Regarding'],0,50),
                    'description' => $row['Regarding'],
                    'parent_type' => 'Accounts',
                    'parent_id' => $row['account_id'],
                    'contact_id' => $row['contact_id'],
                    'date_entered' => $date,
                    'date_modified' => $date,
                    
                  );
                  $result = $ss->setEntry('Notes',$values); 
              }
              //get the id
              if(isset($result['id']))
              {
                $note_id = $result['id'];
                
                //set the id in the from db
                $update_query ='Update ndb set processed = \'1\' where X_CDBJOIN =\''. $row['X_CDBJOIN'] . '\' and `Create Timestamp` = \''. $row['Create Timestamp'] . '\'';
                mysql_query($update_query);
                $err = mysql_error();
                echo "Note Imported\n";
                echo "Regarding: ". substr($row['Regarding'],0,50); 
                
              }
              else
              {
                  echo "ERROR\n";
                //set the id in the from db
                $update_query ='Update ndb set processed = \'2\' where X_CDBJOIN =\''. $row['X_CDBJOIN'] . '\' and `Create Timestamp` = \''. $row['Create Timestamp'] . '\'';
                mysql_query($update_query);
                $err = mysql_error();
                echo "Note Imported\n";
                echo "Regarding: ". substr($row['Regarding'],0,50); 
              }
              
    
                
              $mem_used = memory_get_usage(TRUE) / 1024 / 1024;
              echo round(($mem_used / $allotted_mem)*100,2) . " &#37; Memory Utilized\n";
              
              if(round(($mem_used / $allotted_mem)*100,2) > '70')
              {
                  //false means that the script has not completed its work
                  return FALSE;
              } 
              
            
          }
          
          mysql_free_result($results);
          
                $query = 'SELECT ndb.X_CDBJOIN,ndb.`Create Timestamp`,ndb.Regarding,cdb.account_id,cdb.contact_id 
      FROM ndb join cdb on cdb.x_cdbjoin = ndb.x_cdbjoin where processed = 0 limit 100';
              
              $results = mysql_query($query);
              
              $num_results = mysql_num_rows($results);
              
              mysql_free_result($results);
              
              if($num_results > 0)
              {
                  return FALSE;
              }
      }
      else
      {
          return TRUE;
      }
    }  
      
    ?>

  6. #6
    SitePoint Wizard bronze trophy
    Join Date
    Jul 2008
    Posts
    5,757
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    How many additional bytes of memory are we talking per iteration? And about how many bytes are contained in a row from that db result(approx the sum of the strlen of each column)

    Does your ss object keep data internally? A name like setEntry() suggests its doing that.

  7. #7
    SitePoint Enthusiast
    Join Date
    Aug 2007
    Posts
    28
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I don't know how many bytes exactly because I have been computing and watching the total percentage used. It goes up about .80 percent for every 5 iterations of the while statement.

    $ss is the object used to make the SOAP query. It holds login credentials and session id but login is only called once outside of any loops. Inside of the while I call $ss->setEntry('Notes',$values); which is a SOAP call using NuSOAP.

  8. #8
    SitePoint Wizard bronze trophy
    Join Date
    Jul 2008
    Posts
    5,757
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    My point being, that object might be storing data, which uses memory. You need to read the source code to see, or kill/recreate the object before it gets too fat.

  9. #9
    SitePoint Enthusiast
    Join Date
    Aug 2007
    Posts
    28
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I added a destruct mechanism that unsets everything when called which is at the end of each loop. This seems to help but the memory is still growing.

  10. #10
    SitePoint Wizard bronze trophy
    Join Date
    Jul 2008
    Posts
    5,757
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    Finding out how much it grows by might give you a clue. But surely you arent unsetting the soap object?

  11. #11
    SitePoint Enthusiast
    Join Date
    Aug 2007
    Posts
    28
    Mentioned
    0 Post(s)
    Tagged
    0 Thread(s)
    I can make the output show me how much it is growing.

    I changed the code around a little to instantiate the soap object with every call of the function. That way I can unset it at the end of the function.


Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •