Hello everyone,
We work in quite a few languages and therefore have undertaken a lot of projects using very different methods and techniques.
We currently have a project that requires a lot of CLI/shell type work to be carried out on a database population crawler type application and we made the first rough test version using PHP CLI and storing the data to an InnoDB MySQL database as we would require full text indexing and quick searching capabilities. The script makes use of a customized Curl Class to fetch most of it’s remote data. Also we work on the Linux environment.
Now after much debate we have decided that it needs to do this all a different way as we need to make use of multithreading and it would be lovely if the whole thing could run a lot quicker. This all leads to potentially moving away from PHP on the CLI and perhaps doing it over in Python or Perl? C/C++ would be a great idea, but we don’t really want to have to recompile in order to view latest revisions.
If I’m not being very clear let me know and I can elaborate where neccessary, otherwise I’d love to hear some of your opinions on the matter.
Looking forward to some interesting responses 
Andrew