I am working on a custom WP plugin which in short simply loads products from a manufacturer’s website API and insert those products (with the needed changes) into WP and WooCommerce.
The issue I am facing is that on the staging server (with identical specifications to the production server), the plugin runs out of memory while receiving the details from the API.
Unfortunately this API can’t give paged results, and simply gives a dump (containing about 9.3MB of json data; which equates to over 8500 product variations; the 9.3mb is a dump file created of the raw returned data). During the receiving of this data, the PHP memory limit (of 128mb) is reached - I might not be able to increase this limit (busy trying but as an alternative).
The API is a SOAP type, and trying to use less memory I tried not declaring a variable first before sending it to the further processing function (though further reading concluded that this does not use more or less memory).
The failing part looks like this:
$this->translate_data( $this->SOAPClient->GetFeedByEntityID( $params )->GetFeedByEntityIDResult->ResponseData );
The “->GetFeedByEntityIDResult->ResponseData” is to get to the response data as soon as possible without declaring more variables.
The function in which this is located terminates after dumping the translated (further processed data) to a file (where a 3rd part of the script handles it as the actual import to WC - this 3rd part is modular to also handle data from other manufacturer APIs; the translate_data-method simply gets it into a usable format).
Should I perhaps try to handle the translate_data method inside the current method (to unset the data asap)? What would you suggest?
I have very limited experience with this size datasets, especially ones that can’t be paged.