Hello,
I have a very large homegrown application written in PHP using the PEAR Expect module. Basically the application retrieves data from remote network devices and stores it locally. There are about 3000 remote devices to collect data from.
Another feature of the app is a web interface where a user can enter a number of commands, and those commands would be run on a number of remote devices sequentially.
My problem is, the application will stop mid-way through executing. The time it runs is random. Sometimes it will get through a few hundred devices before stopping, sometimes it will get through a few dozen.
When it stops, the process is till running, I can see that it is still SSH’d into the remote device. There is nothing in the PHP error logs or the Apache error logs. I’ve used every set_time_limit()-like function I can find (max_execution_time, expect.timeout, etc) and it doesn’t seem to be a timeout issue. For some reason the process just stops. It doesn’t die, or crash, it just stops.
Where do I even begin looking? What info do you need to troubleshoot this?
CentOS release 4.6 (Final)
php-5.1.6-3.el4s1.8
php-expect 0.2.6 beta
httpd-2.0.59-1.el4s1.8.el4.centos
Tyler
I think I solved it!
The problem was, PHP wasn’t respecting the EXP_TIMEOUT value, and in situations where Expect would normally timeout, it wouldn’t, it would just sit there.
The fix was to ini_set(‘expect.timeout’,<integer>) somewhere at the beginning of the program. I had set it in the class and PHP didn’t respect it.
Another tip, I also tried setting it in a file that was require_once() loaded. That didn’t work either. It actually needed to be set in a file that was include() loaded.
Hopefully that helps someone!
How do you retrieve data from the remote network devices? (using what function/extension)
I was thinking about forking the process, but I’m still concerned that PHP just stops without giving any errors or messages. I’d like to solve this before adding more complexity to the code.
Is there some way I can create a “debug” log for what PHP is doing? I’ve set E_ALL in php.ini, but it doesn’t log any errors once the process stops. What more can I do to debug?
Tyler
If the processor is not a slow processor, you may run several scripts in separate processes and update the database in different time than online.
The script opens up a file handle with fopen() to an Expect:// stream. The expect stream then SSH’s to a trusted unix host. From there it SSH/Telnets to each remote device to run commands on them.
I watch the Expect stream for data and send data on this Expect stream that is opened with fopen(). I did use expect_popen() at one point but found it didn’t close the stream properly and Apache would have zombie processes. Fopen() closes the streams properly but it still hangs at random intervals.