Ohhh… Same guy eh? I’m giong to fall back to my original thought, from the older thread, atleast I thought I brought it up…
You need to set up a central “application server” and then a data warehouse of sorts.
We know that your going to need some multi threading in order to get those curl / soap calls completed timely (or in this case child processes since php doesnt have multi-threading ) We want to 1) ultimately increase all performance which will mean the rest: 2) lower network traffic 3) lower processing being done on each array.
I’m about to mention something that’s frowned upon by some people here, but sometimes some jobs call for something such as this, I’m currently setting up a datawarehouse using it due to the sheer volume of data I’m dealing with:
I recommend setting up a central data server with MongoDB. http://www.mongodb.org/
You can store multidimensional data (arrays) in this database. There is no strict table structure, each row can have its own structure, which means you need to program those restrictions in the the code, if they need to exist.
Say the return you get from those curl calls is xml, you’d only need the following:
//connect to mongo
$m = new Mongo();
$coll = $m->dbName->xml_responses;
$coll->insert(json_decode(json_encode(simplexml_load_string($curlResponse)), 1));
This will have loaded your array into mongodb. While yes, we are running a json_encode and decode on this, we can drop a lot of network traffic. When all of your 900 requests have been fulfilled, you can actually QUERY your return results, rather than manipulate it in memory.
MongoDB is a whole new beast that isn’t covered often here at the forums, before running away from the idea, have a look at it. I’ve grown to love it (in the right situations) and one the places it excels is those randomly structured XML / json structures. You can even run in some caching based off these results. You can index anything with the subdocument as if it were a normal column (an array element is 5 levels deep? you can index it and query for it)