parsing XML, but skipping entities

[Chris82]

Hello.

Selkirk, your parser works really great. I have used it for the form validation again and have something working now. Altough, I don't know how well the implementation is ...

--------------------------------------------------------

This is an example of a form which uses custom tags and filters:

<form method="post" action="http://localhost/sitepoint/xmlparsertest.php">
<table>
  <tr>
    <td>name:</td>
    <td>
      <webform:textbox name="name" length="20">
        <webform:filter type="LengthFilter" max="20" />
      </webform:textbox>
    </td>
    <td style="color: #cc0000;"><webform:error /></td> 
  </tr>
  <tr>
    <td>email:</td>
    <td>
      <webform:textbox name="email" length="30">
        <webform:filter type="RegExpFilter" regexp="#.+?@.+?\.(de|com|co\.uk)#" />
        <webform:filter type="LengthFilter" min="8" max="30" />
      </webform:textbox>
    </td>
    <td style="color: #cc0000;"><webform:error /></td>
  </tr>
  <tr>
    <td>telephone</td>
    <td>
      <webform:textbox name="phone" length="20">
        <webform:filter type="RegExpFilter" regexp="#0\d{2,4}(-|/)\d{3,}#" />
      </webform:textbox>
    </td>
    <td style="color: #cc0000;"><webform:error /></td> 
  </tr>
  <tr>
    <td colspan="3" align="center">
      <input type="submit" name="submit" value="Validate" />
    </td>
  </tr>
</table>
</form> 

This doesn't look bad in Dreamweaver either

The textbox tag is firstly used to display the <input type=text> part and secondly used to create the field for the validator. The <webform:error /> tag is filled out if the form has been submitted and the validation failed.

I have attached the source of a working example. If someones wants to test this, it is best to extract it to a folder called "sitepoint" and keep the file structure of the zip file.

Edit:

Because I included Selkirk's parser without any reference, I changed the zip file. Thus if someone wants to test the examples, you also need to download Selkirk's parser from here:
http://www.sitepointforums.com/showp...6&postcount=15

Because of some require_once calls in the source files it would be easiest to save the above file as XMLParser.php in the folder XML where the other classes are located.

[lastcraft]

Hi.

Indeed. I am no expert with xml in PHP. I might have bungled the expat implementation. Here is the code I used:

$xml_parser = xml_parser_create();
xml_set_object($xml_parser, $handler);
xml_set_element_handler($xml_parser, "openHandler", "closeHandler");
xml_set_character_data_handler($xml_parser, "dataHandler");
xml_parse($xml_parser, $doc); 

I used the same handler for all three versions. One difference seems to be that expat called the dataHandler method ALOT more, possibly because of line breaks. What do you make of this?

Haven't a clue! There shouldn't be anything to bungle. Are there a lot of entities in the XML file? They will certainly split the character data, but I don't know about line endings. My guess is the speed drop (from a pure C program using expat) is the marshalling of parameters going in and out of PHP.

Not every state advances the current position. An infinite loop detecter would have to make sure that the state did not change AND the current position was not advanced.

Ok, next time I'll read the code first . If you could force the pointer advance that would solve the infinite loop problem. Hmm....

Please do.

He's trying to get me to do it. Now that's panache...

yours, Marcus

[HarryF]

He's trying to get me to do it. Now that's panache...

It's all about abstraction

Been having a more detailed look; there's a few issues to clean up such as when you have attributes like

<option name="color" selected/>

When I get a moment will see how it runs under my original unit tests for HTMLSax.

Guess we need a CVS server to put this under pretty soon, given three people working on it.

[lastcraft]

Hi.

We could all get PEAR accounts!

yours, Marcus

[HarryF]

We could all get PEAR accounts!

[Sadly?] that doesn't give you access to the PHP cvs, just to the PEAR release system.

[Dr Livingston]

... Sorry Just had to have a laugh...

[HarryF]

OK - somewhat modified version attached with SimpleTests. Fixed a couple of minor bugs with the EscapeState and the one with tag attributes that have no value, plus added a CaseFolding decorator.

On the parser options, think XML_HTMLSax v2 is going to slightly break backwards compatibility and drop too options which I think are superfluous and add unneeded complexity, plus I've renamed the remain two to be more like the native XML extension. Otherwise had to put a XML_HTML_ before everything globally declared, to prepare for PEAR.

Can see the problem now with scanUntilString() and scanUntilCharacters() (in class XML_HTMLSax_StateParser) but so far, nothing seems to be able to cause an infinite loop (even artificially modifying the "XML_HTMLSax_StateParser->length" property).

One possible solution might be passing those two methods a list of characters to halt on - e.g. if the AttributeState::parse() is hunting for a quote with scanUntilString(), the method also gets a list of characters to "brake" on, such as < > and apostrophe. Couldn't see an obvious way to do it staight away though without some very clunky code.

Anyway - need to find this thing a CVS server.

[Selkirk]

On the parser options, think XML_HTMLSax v2 is going to slightly break backwards compatibility and drop too options which I think are superfluous and add unneeded complexity, plus I've renamed the remain two to be more like the native XML extension.

Good idea. Here are a few nitpicky suggestions.

First, The XML_HTMLSax_TagCleanupState should be eliminated and its functionality rolled up into the two states that transition to it: XML_HTMLSax_OpeningTagState and XML_HTMLSax_ClosingTagState. Both of these states should transition back to XML_HTMLSax_StartingState. This fixes two bugs and increases the speed a bit.

The first is that </tag/> is currently recognized as a closing tag. Most of the hard choices have to do with invalid html. Anyway, checking for '/' when you know you are in the closing tag state seems unncessary.

The second is a bigger issue. right now, there is no way to distinguish between <TAG> and <TAG/>. This could be important. The solution is to check in the XML_HTMLSax_OpeningTagState for the '/>' versus the '>' and emit a closing event when the '/>' is detected. Then it will be impossible to tell the difference between <tag/> and <tag></tag>, but that seems Ok. This will also be more compatible with the native XML extension, which seems to do the same thing.

In the XML_HTMLSax_CaseFolding decorator, it is unnecessary to check the tags against '' before re-emiting the event as it is in XML_HTMLSax_Trim. trim() can transform a string into '' while strtoupper() cannot. The tag argument is already known not to be '' before the decorator is called.

Also, in the interest of compatability with the built in XML functions, you might want to consider a decorator for the character data event which transforms its contents to match what the built in functions are doing. (breaking on line feeds or entities, or what ever. I don't know what they are doing).

One possible solution might be passing those two methods a list of characters to halt on - e.g. if the AttributeState:****() is hunting for a quote with scanUntilString(), the method also gets a list of characters to "brake" on, such as < > and apostrophe. Couldn't see an obvious way to do it staight away though without some very clunky code.

I looked at the $breakers = array('>','='); in XML_HTMLSax_AttributeStartState. As far as I can see, this involves making a choice between two very sad alternatives:

<tag attribute="x > y">stuff</tag> and <tag attribute="abc>stuff</tag>

Both are invalid html. The parser can only really handle one of these two perversions well in XML_HTMLSax_AttributeStartState. My vote would be for handling an accidental entity in an attribute value versus handling unbalanced quotes.


Been thinking a bit more on the issue of end of string handling. My current tentative tangental thinking is that adding validation capability (ability to report if the input is valid html or xml syntax) might cause the answer to the end of string issue to appear. All the evil cases are triggered by invalid html. I'm going to have to do some reading on xml validation. I am very xml ignorant.

[HarryF]

First, The XML_HTMLSax_TagCleanupState should be eliminated and its functionality rolled up into the two states that transition to it: XML_HTMLSax_OpeningTagState and XML_HTMLSax_ClosingTagState. Both of these states should transition back to XML_HTMLSax_StartingState. This fixes two bugs and increases the speed a bit.

(Hopefully) will have a place on Sf.net shortly to place this lot under so will wait until then before further changes.

I looked at the $breakers = array('>','='); in XML_HTMLSax_AttributeStartState. As far as I can see, this involves making a choice between two very sad alternatives:

Whoops - forgot to strip that out - was a by product of experiments.

Reckon emitting a closing tag event on <TAG/> is important as is being able to (optionally) break on entities (to match the native XML extension). Linefeed "breaking" is not something the native extension does though - in fact a by product of the original code (http://sourceforge.net/projects/phpshelve) that I started out with. Linefeeds could just as well be dealt with by a handler using the parser.

As to validation, one easy solution is to have a handler perform this, perhaps counting the number of opening / closing tags, checking the contents of attributes and so on. Don't think this parser should attempt to be able to deal with well formed XML (for which there's the native extension that code can be easily adapted to) but rather guarantee a complete parse through, no matter how bad the document is.

That approach could be easily implemented with a simple Sax filter - the validator is the first to handle events, keeping a tally of what it sees, then passing on the events to object really using the parser. Might mean you have to "rollback" if something goes wrong though.

Alternatively just pass through a document twice, once with validator then again with the real handler object - of course that's slower but...

[PS: Having trouble with email today after switching DNS around - got everything but only have command line access to read it which is too much pain]

[lastcraft]

Hi.

Another twist...

Given that XML_HTML_SAX is already a PEAR module this may be an alternative approach...
http://pear.php.net/package-info.php?package=tidy
This is the method the Java HTTPUnit uses for it's HTML parsing and should be extremely robust. You will still need the existing parser for those people who do not have permission to load extensions on their servers of course.

yours, Marcus.

[anode]

One difference seems to be that expat called the dataHandler method ALOT more, possibly because of line breaks. What do you make of this?

Seems to be the case. Ken Egervari writing in the comments section of the manual:

the function handler is called several times when it parses the character data. *It doesn't return the entire string as it suggests. *There are special exceptions that will always force the parser to stop scanning and call the character data handler. *This is when:

- The parser runs into an Entity Declaration, such as &amp; (&) or &apos; (‘)
- The parser finishes parsing an entity
- The parser runs into the new-line character (\n)
- The parser runs into a series of tab characters (\t)

And perhaps others.

For instance, if we have this xml content:

<mytag name=”Ken Egervari” title=”Chief Technology Officer”>
Ken has been Positive Edge&apos;s Chief Technology Officer for 2 years.
</mytag>

The parser will call the character data handler 6 times. *This is what will happen:

1 \n
2 \t
3 Ken has been Positive Edge
4 ‘
5 s Chief Technology Officer for 2 years.
6 \n

I hope that helps people out.

Selkirk, what's the license on this?

[lastcraft]

Hi...

http://pear.php.net/package-info.php?package=tidy

Scrub this for now. I have failed to get it working on three different Linux boxes.

yours, Marcus.

[HarryF]

Another twist: http://pear.php.net/html_parse - still version 0.1 but look at the C library it's based on, seems it's intended to be SAX like.

Anode - that's for those tip offs. For some reason have never noticed that behaviour with \n or \t but perhaps that's because I was handling entities and missed it?

what's the license on this?

Will be PHP 3.x license for PEAR package - now got the SF site: http://sourceforge.net/projects/htmlsax. Reckon there should also be a non - PEAR release under LGPL.

[anode]

Offhand, do you know if that's compatible with a BSD/MIT license? There's something I'd like to bundle it with ASAP.

[HarryF]

Should be OK. Zend always talk about the PHP license having a "BSD flavor". Also Rasmus has recently been trying to get it OSI approved: http://www.mail-archive.com/license-.../msg06315.html

[Ren]

Just haven taken a peek at php source code, and noticed that strspn() and strcspn do indeed take a starting position. Wether this is a documentation bug, or whatever still remains a mystery.


But this could mean that 2 PHP loops could be removed, for faster C equivalents.

    function scanUntilCharacters($string) {
        $startpos = $this->position;
        $length = strcspn($this->rawtext, $string, $startpos);
        $this->position += $length;
        return substr($this->rawtext, $startpos, $length);
    } 

and

    function ignoreWhitespace() {
        $this->position += strspn($this->rawtext, " \n\r\t", $this->position);
    } 

[HarryF]

Just haven taken a peek at php source code, and noticed that strspn() and strcspn do indeed take a starting position. Wether this is a documentation bug, or whatever still remains a mystery.

Nice one! That's another nice performance boost.

I've added that plus all of Selkirks last remarks to the latest under CVS:

cvs -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/htmlsax login

cvs -z3 -d:pserver:anonymous@cvs.sourceforge.net:/cvsroot/htmlsax co htmlsax

Parsing the following document: http://news.php.net/group.php?group=php.general&format=rdf

Examples;
With Expat getting takes 0.100s
With HTMLSax (mark 2) takes: 0.418s (without any decorators)
With HTMLSax (mark 1) takes: 1.628s

There are also now decorators for breaking on newlines, tabs and entities. For entities I used html_entity_decode() which converts the entities to their HTML equivalents (a larger set of entities than the native extension) but it's PHP 4.3.0. Perhaps it would be better to skip entity parsing and simply pass them on as is?

[Ren]

Noticed the actual PHP unit tests for strspn() and strcspn() include the extra undocumented parameters.

http://cvs.php.net/co.php/php-src/ex...spn.phpt?r=1.1
and http://cvs.php.net/co.php/php-src/ex...spn.phpt?r=1.1

[Chris82]

in class XML_HTMLSax_Entities (file: XML_HTMLSax_Decorators.php) the regexp to split the various entities is:

$data = preg_split('/(&.+;)/',$data,-1,PREG_SPLIT_DELIM_CAPTURE); 

If you apply this to &nbsp;&nbsp; the string will remain &nbsp;&nbsp; and there are some empty results.

I think it is best to change it to:

$data = preg_split('/(&.+?;)/', $data, -1, PREG_SPLIT_DELIM_CAPTURE | PREG_SPLIT_NO_EMPTY); 

Using the above regexp on:

$data = '   ';
$data = preg_split('/(&.+?;)/', $data, -1, PREG_SPLIT_DELIM_CAPTURE | PREG_SPLIT_NO_EMPTY);
print_r($data); 

results in:

Array
(
    [0] =>  

    [1] =>  
    [2] =>  
)

[Selkirk]

Examples;
With Expat getting takes 0.100s
With HTMLSax (mark 2) takes: 0.418s (without any decorators)
With HTMLSax (mark 1) takes: 1.628s

This is probably a much more realistic benchmark. The XML document I used was a sherlock preference file on my mac. This is much more like what would actually use this for and the Expat numbers aren't so bad.

(Although, it probably wouldn't hurt to add an option to the xml_ functions in PHP to handle character data in a way that didn't result in so many call backs.)

html_entity_decode() ... but it's PHP 4.3.0.

Do you have a specific version goal for this to be able to run under? 4.1, 4.2, or 4.3?

Just haven taken a peek at php source code, and noticed that strspn() and strcspn do indeed take a starting position. Wether this is a documentation bug, or whatever still remains a mystery.

Nice find. I looked longingly at those functions, but it never occurred to me that they would have undocumented parameters. Do you know which version of PHP they were introduced? (See previous issue.)

[HarryF]

In class XML_HTMLSax_Entities (file: XML_HTMLSax_Decorators.php) the regexp to split the various entities is:

Chris; thanks for that tip off. We should call this the SPF PHP community process or something - there's been so much great input.

Do you have a specific version goal for this to be able to run under? 4.1, 4.2, or 4.3?

Generally as low is possible as I see this as a fairly fundamental architectural tool. I'm pondering whether to have two decorators; Entities_Parsed and Entities_Unparsed, the latter for older versions. That introduces a question of whether it's necessary to prevent someone from using both options. Will have to see.

Otherwise, reckon this should be OK for an alpha release in the nearish future. I've used it to parse a variety of dodgy looking websites already with much success. The only issue seems to be the unclosed attributes.

[Chris82]

concerning html_entity_decode:

This is from the php documentation page for the function:

// For users prior to PHP 4.3.0 you may do this:
function unhtmlentities ($string)
{
    $trans_tbl = get_html_translation_table (HTML_ENTITIES);
    $trans_tbl = array_flip ($trans_tbl);
    return strtr ($string, $trans_tbl);
} 

Should do the same and will work with PHP4.x.

[Ren]

Nice find. I looked longingly at those functions, but it never occurred to me that they would have undocumented parameters. Do you know which version of PHP they were introduced? (See previous issue.)

Seems the extra parameters appeared in the 1.299 revision of string.c and the first version of PHP to include that code is 4.3 (using revision 1.333 of string.c)

[Ren]

Something along the lines of

    if (!function_exists('html_entity_decode'))
    {
        function html_entity_decode($string)
        {
            static $trans_tbl;
            if (!isset($trans_tbl))
                $trans_tbl = array_flip(get_html_translation_table(HTML_ENTITIES));
            return strtr($string, $trans_tbl);
        }
    } 

would solve one of the backward incompatibilities, without losing any preformance on 4.3+.

Using the extra parameters on strspn()/strcspn() however... I'm not sure the best way to go about it atm.

[Selkirk]

Generally as low is possible as I see this as a fairly fundamental architectural tool. I'm pondering whether to have two decorators; Entities_Parsed and Entities_Unparsed, the latter for older versions. That introduces a question of whether it's necessary to prevent someone from using both options. Will have to see.

Well, one way to handle this versioning issue is for the main class to use the state parser instead of inherit from it. That way if could select which implementation of the state parser it was going to use depending upon which version of PHP was running. (really just picking which set of scan* methods to use). It could select the fastest version allowed by the current version of PHP.

To make this happen, the variables that store the callback objects and methods would have to be moved from the main class onto the state objects. (Like the attribute state is done now.) This is probably a better design anyway.