Running an Executable with Proc_Open - Reading until end of stream?

[Jake Arkinstall]

Hey everyone.

I'm running an executable (written in Fortran) that requires multiple stages of input and outputs a few rows explaining the next input etc.

(I don't have the source code to this executable)

So, I open the process with Proc_Open. The IN (relative to the executable) buffer is in $Pipes[0], the OUT buffer is $Pipes[1] and the error buffer is $Pipes[2].

My problem is reading the whole of the OUT buffer before writing to the IN buffer. IIRC the OUT buffer needs to be fully read before the IN buffer can be written to.

Now, to read the block out of a stream, one would assume that the typical:

while(!feof($Pipes[1])){....} 

Would work, however this freezes the PHP application.

So, until I can get a solution which allows me to read up until the end, I'm having to read it line by line using fgets(). If I use fgets just one too many times, the PHP application freezes (and I have to exterminate the 'httpd.exe' processes and restart WAMP).

So, obviously I made a function to loop over reading a single line based on the parameter passed. This means I have to know how many lines need to be read - a tedious task because of the different paths the application can follow.

So, is there an automated way to read the whole of a pipe stream output from an application resource created by Proc_Open()?

It seems the status of the process is constant whether at the end of the stream or not, and feof is returning false even at the end of the buffer.

I'm running PHP5.2.6.

[forkaya]

What 'descriptorspec' are you passing to proc_open?

[forkaya]

If you post php code (forget about Fortran), we can better see what you are trying to do... Maybe the deadlock is not on feof.

[crmalibu]

see
stream_set_blocking()
or
stream_set_timeout()

fread() doesn't want to let you down and return with less bytes than you asked.

Make sure you detect EOF or your own end of data. I think once fread() fails then feof() can return true.

You may not get anything reading a non blocking stream, if you try to read before something is there.

[Jake Arkinstall]

Unfortunately that still doesn't have an affect.

This is the Process class I wrote for the task:

<?php
class Process{
    protected $Descriptors = array(array('pipe', 'r'), array('pipe', 'w'), array('file', 'error-output.txt', 'a'));
    public $Pipes;
    protected $Process;
    protected $Open = false;
    function __Construct($FileName){
        $this->Process = proc_open('"'.$FileName.'"', $this->Descriptors, $this->Pipes, GetCWD());
        stream_set_timeout($this->Pipes[1], 5);
        stream_set_blocking($this->Pipes[1], 0);
        $this->Open = true;
    }
    function IsOpen(){
        return $this->Open;
    }
    function getError(){
        return fgets($this->Pipes[2]);
    }
    function ReadLine(){
        if($this->IsOpen())
            return fgets($this->Pipes[1]);
        else
            return false;
    }
    function ReadLines($LineCount){
                       if(!$this->IsOpen()) return false;
        $Lines = array();
        for($i = 0; $i < $LineCount; $i++){
            $Lines[] = $this->ReadLine();
        }
        return $Lines;
    }
    function CanRead(){
        return ($this->IsOpen() && !feof($this->Pipes[1]));
    }
    function Read(){
        if(!$this->IsOpen()){
            return false;
        }else{
            return fgetc($this->Pipes[1]);
        }
    }
    function ReadBlock(){
        if(!$this->IsOpen()) return false;
        $String = '';
        while(!feof($this->Pipes[1])){
            $String .= fgets($this->Pipes[1]);
        }
        return $String;
    }
    function WriteLine($String){
        if($this->IsOpen())
            return $this->Write($String . "\n");
    }
    function Write($String){
        if($this->IsOpen())
            return fwrite($this->Pipes[0], $String);
    }
    function GetStatus(){
        return proc_get_status($this->Process);
    }
    function __Destruct(){
        if($this->IsOpen())
            proc_close($this->Process);
        $this->Open = false;
    }
}

and upon using it:

<?php
$Process = new Process('TheFile.exe');
$Process->ReadBlock(); //Freezes PHP
//or:
$Process->ReadLines(30); //Fine if the number of lines is 30. If it's more, not all data is returned - if it's less, the application freezes.

The actual deadlock here is definitely in the reading of the stream.

Say there are 120 lines to be read. If I read 120, the application works like a charm. If I attempt to read 121, it doesn't return false and set eof like one would expect; Instead it simply freezes. Even if it's not in a feof loop, it still freezes simply by trying to get one more character.

[forkaya]

feof is probably not working because the out stream does not contain eof file marker. Can you force Fortran app to stick one at the end of the stream? Same is probably true with the end of the line marker for a last line...

[Jake Arkinstall]

Unfortunately I cannot change the source-code of the Fortran application.

[crmalibu]

I've been playing around with this as I couldn't get it working either. One thing I noticed is that I can read the childs STDOUT just fine until the child trys to read something on it's STDIN. If the child never trys to read it's STDIN and just writes stuff to its STDOUT, the parent script never encounters blocking issues.

I've tried all kinds of doohicky with stream_select() and just can't seem to avoid the deadlock. stream_set_blocking() doesn't seem to get applied according to stream_get_meta_data().

A hacky thing that works, but is failure prone is to use fread() and detect EOF by measuring bytes read.

$msg = '';
$chunk_size = 8192;
do {
    $buf = fread($pipes[1], $chunk_size);
    $msg .= $buf;
} while (strlen($buf) == $chunk_size); 

This works except if there isn't at least 1 byte to be read, fread() will block. It will also block eventually on some iteration of the loop if the stream happens to have exactly $chunk_size, or a multiple of it, because it will have read every byte and then try to read once more on the last iteration. And the most fread() will read at a time is 8192, so you can't just read some huge number of bytes in a single call as a way to mimic stream_get_contents()(which always blocks here). This just isn't a solution.

[forkaya]

@ arkinstall
Sorry for not replying earlier... If you still struggling with this one, here is another suggestion (if you have it working, please let us know how).

Have you tried using 'stream_get_contents'? It looks like it does not care about lines or eof...

So try replacing:

    function ReadBlock(){

        if(!$this->IsOpen()) return false;

        $String = '';

        while(!feof($this->Pipes[1])){

            $String .= fgets($this->Pipes[1]);

        }

        return $String;

    } 

with

    function ReadBlock(){

        if(!$this->IsOpen()) return false;

        $String = '';

        $String = stream_get_contents($this->Pipes[1]);

        return $String;

    } 

[Jake Arkinstall]

Unfortunately, as shown above by crmalibu, that just blocks.

It's not much of a stumbling block as an annoyance at this stage - of having to know the number of rows for every outcome.