Comments on: Tokenization using regular expression sub patterns http://www.sitepoint.com/blogs/2008/01/19/tokenization-using-regular-expression-sub-patterns/ Sun, 20 Jul 2008 01:51:01 +0000 http://wordpress.org/?v=2.5 By: HarryF http://www.sitepoint.com/blogs/2008/01/19/tokenization-using-regular-expression-sub-patterns/#comment-607562 HarryF Sun, 20 Jan 2008 09:35:48 +0000 http://www.sitepoint.com/blogs/2008/01/19/tokenization-using-regular-expression-sub-patterns/#comment-607562 <blockquote> a single, long regular expression can often be more inefficient than small regular expressions due to backtracking. </blockquote> Interesting point although that might well be outweighed having to make additional calls in PHP or Python, depending on what you're doing.

a single, long regular expression can often be more inefficient than small regular expressions due to backtracking.

Interesting point although that might well be outweighed having to make additional calls in PHP or Python, depending on what you’re doing.

]]> By: Anonymous http://www.sitepoint.com/blogs/2008/01/19/tokenization-using-regular-expression-sub-patterns/#comment-607299 Anonymous Sun, 20 Jan 2008 02:55:14 +0000 http://www.sitepoint.com/blogs/2008/01/19/tokenization-using-regular-expression-sub-patterns/#comment-607299 a single, long regular expression can often be more inefficient than small regular expressions due to backtracking. a single, long regular expression can often be more inefficient than small regular expressions due to backtracking.

]]> By: Skunk http://www.sitepoint.com/blogs/2008/01/19/tokenization-using-regular-expression-sub-patterns/#comment-606383 Skunk Fri, 18 Jan 2008 23:21:57 +0000 http://www.sitepoint.com/blogs/2008/01/19/tokenization-using-regular-expression-sub-patterns/#comment-606383 Interesting fact (well, I think so at least): the Django template engine uses a single regular expression to tokenize the entire template. That's the main reason it's so fast compared to many of the other Python template languages out there. Take a look at this: http://code.djangoproject.com/browser/django/trunk/django/template/__init__.py#L70 (in particular line 90, where tag_re is declared). Interesting fact (well, I think so at least): the Django template engine uses a single regular expression to tokenize the entire template. That’s the main reason it’s so fast compared to many of the other Python template languages out there. Take a look at this: http://code.djangoproject.com/browser/django/trunk/django/template/__init__.py#L70 (in particular line 90, where tag_re is declared).

]]> By: Greg Beaver http://www.sitepoint.com/blogs/2008/01/19/tokenization-using-regular-expression-sub-patterns/#comment-606296 Greg Beaver Fri, 18 Jan 2008 20:35:36 +0000 http://www.sitepoint.com/blogs/2008/01/19/tokenization-using-regular-expression-sub-patterns/#comment-606296 PHP_LexerGenerator takes this principle and combines it with the idea of states to implement a re2c-like lexer generator grammar for PHP. http://pear.php.net/PHP_LexerGenerator PHP_LexerGenerator takes this principle and combines it with the idea of states to implement a re2c-like lexer generator grammar for PHP. http://pear.php.net/PHP_LexerGenerator

]]> By: phpimpact http://www.sitepoint.com/blogs/2008/01/19/tokenization-using-regular-expression-sub-patterns/#comment-606224 phpimpact Fri, 18 Jan 2008 19:07:52 +0000 http://www.sitepoint.com/blogs/2008/01/19/tokenization-using-regular-expression-sub-patterns/#comment-606224 Thanks to guys like you Andrei PHP is on the top. Thanks man! Thanks to guys like you Andrei PHP is on the top. Thanks man!

]]> By: Andrei Z http://www.sitepoint.com/blogs/2008/01/19/tokenization-using-regular-expression-sub-patterns/#comment-606137 Andrei Z Fri, 18 Jan 2008 17:23:30 +0000 http://www.sitepoint.com/blogs/2008/01/19/tokenization-using-regular-expression-sub-patterns/#comment-606137 Another good way to tokenize is with preg_split() and PREG_SPLIT_DELIM_CAPTURE option. See my Regex Clinic slides on http://gravitonic.com/talks Another good way to tokenize is with preg_split() and PREG_SPLIT_DELIM_CAPTURE option. See my Regex Clinic slides on http://gravitonic.com/talks

]]>