How to build a good regex
Regular expressions are often used in the developer’s daily routine – log analysis, form submission validation, find and replace, and so on. That’s why every good developer should know how to use them, but what is the best practice to build a good regex?1. Define a scenario
Using natural language to define the problem will give you a better idea of the approach to use. The words could and must, used in a definition, are useful to describe mandatory constraints or assertions. Below is an example:- The string must start with ‘h’ and finish with ‘o’ (e.g. hello, halo).
- The string could be wrapped in parentheses.
2. Develop a plan
After having a good definition of the problem, we can understand the kind of elements that are involved in our regular expression:- What are the types of characters allowed (word, digit, new line, range, …)?
- How many times must a character appear (one or more, once, …)?
- Are there some constraints to follow (optionals, lookahead/behind, if-then-else, …)?
3. Implement/Test/Refactor
It’s very important to have a real-time test environment to test and improve your regular expression. There are websites like regex101.com, regexr.com and debuggex.com that provide some of the best environments. To improve the efficiency of the regex, you could try to answer some of these additional questions:- Are the character classes correctly defined for the specific domain?
- Should I write more test strings to cover more use cases?
- Is it possible to find and isolate some problems and test them separately?
- Should I refactor my expression with subpatterns, groups, conditions, etc., to make it smaller, clearer and more flexible?
Practical examples
The goal of the following examples is not to write an expression that will only solve the problem, but to write the most effective expression for the specific use cases, using important elements like character ranges, assertions, conditions, groups and so on.Matching a password
![password match password match](https://uploads.sitepoint.com/wp-content/uploads/2015/09/1442490514password.jpg)
- 6 to 12 characters in length
- Must have at least one uppercase letter
- Must have at least one lower case letter
- Must have at least one digit
- Should contain other characters
^(?=.*[a-z])(?=.*[A-Z])(?=.*\d).{6,12}$
This expression is based on multiple positive lookahead (?=(regex))
. The lookahead matches something followed by the declared (regex)
. The order of the conditions doesn’t affect the result. Lookaround expressions are very useful when there are several conditions.
We could also use the negative lookahead (?!(regex))
to exclude some character ranges. For example, I could exclude the %
with (?!.*#)
.
Let’s explain each pattern of the above expression:
^
asserts position at start of the string(?=.*[a-z])
positive lookahead, asserts that the regex.*[a-z]
can be matched:.*
matches any character (except newline) between zero and unlimited times[a-z]
matches a single character in the range between a and z (case sensitive)
(?=.*[A-Z])
positive lookahead, asserts that the regex.*[A-Z]
can be matched:.*
matches any character (except newline) between zero and unlimited times[A-Z]
matches a single character between A and Z (case sensitive)
- (?=.*\d) positive lookahead, asserts that the regex
*\d
can be matched:.*
matches any character (except newline) between zero and unlimited times\d
matches a digit [0-9]
.{6,12}
matches any character (except newline) between 6 and 12 times$
asserts position at end of the string
Matching URL
![URL match URL match](https://uploads.sitepoint.com/wp-content/uploads/2015/09/1442490495url.jpg)
- Must start with
http
orhttps
orftp
followed by://
- Must match a valid domain name
- Could contain a port specification (
http://www.sitepoint.com:80
) - Could contain digit, letter, dots, hyphens, forward slashes, multiple times
^(http|https|ftp):[\/]{2}([a-zA-Z0-9\-\.]+\.[a-zA-Z]{2,4})(:[0-9]+)?\/?([a-zA-Z0-9\-\._\?\,\'\/\\\+&%\$#\=~]*)
The first scenario is pretty easy to solve with ^(http|https|ftp):[\/]{2}
.
To match the domain name we need to bear in mind that to be valid it can only contain letters, digits, hyphen and dots. In my example, I limited the number of characters after the punctuation from 2 to 4, but could be extended for new domains like .rocks
or .codes
. The domain name is matched by ([a-zA-Z0-9\-\.]+\.[a-zA-Z]{2,4})
.
The optional port specification is matched by the simple (:[0-9]+)?
.
A URL can contain multiple slashes and multiple characters repeated many times (see RFC3986), this is matched by using a range of characters in a group ([a-zA-Z0-9\-\._\?\,\'\/\\\+&%\$#\=~]*)
.
It’s really useful to match every important element with a group capture ()
, because it will return only the matches we need. Remember that certain characters need to be escaped with \
.
Below, every single subpattern explained:
^
asserts position at start of the string- capturing group
(http|https|ftp)
, captureshttp
orhttps
orftp
:
escaped character, matches the character:
literally[\/]{2}
matches exactly 2 times the escaped character/
- capturing group
([a-zA-Z0-9\-\.]+\.[a-zA-Z]{2,4})
:[a-zA-Z0-9\-\.]+
matches one and unlimited times character in the range between a and z, A and Z, 0 and 9, the character-
literally and the character.
literally\.
matches the character.
literally[a-zA-Z]{2,4}
matches a single character between 2 and 4 times between a and z or A and Z (case sensitive)
- capturing group
(:[0-9]+)?
:- quantifier
?
matches the group between zero or more times :
matches the character:
literally[0-9]+
matches a single character between 0 and 9 one or more times
- quantifier
\/?
matches the character/
literally zero or one time- capturing group
([a-zA-Z0-9\-\._\?\,\'\/\\\+&%\$#\=~]*)
:[a-zA-Z0-9\-\._\?\,\'\/\\\+&%\$#\=~]*
matches between zero and unlimited times a single character in the range a-z, A-Z, 0-9, the characters:-._?,'/\+&%$#=~
.
Matching HTML TAG
![HTML TAG match HTML TAG match](https://uploads.sitepoint.com/wp-content/uploads/2015/09/1442490503tag.jpg)
- The start tag must begin with
<
followed by one or more characters and end with>
- The end tag must start with
</
followed by one or more characters and end with>
- We must match the content inside a TAG element
<([\w]+).*>(.*?)<\/\1>
Matching the start tag and the content inside it’s pretty easy with <([\w]+).*>
and (.*?)
, but in the pattern above I have added a useful thing: the reference to a capturing group.
Every capturing group defined by parentheses ()
could be referred to using its position number, (first)(second)(third)
, which will allow for further operations.
The expression above could be explained as:
- Start with
<
- Capture the tag name
- Followed by one or more chars
- Capture the content inside the tag
- The closing tag must be
</tag name captured before>
<
matches the character<
literally- capturing group
([\w]+)
matches any word charactera-zA-Z0-9_
one or more times .*
matches any character (except newline) between zero or more times>
matches the character>
literally- capturing group
(.*?)
, matches any character (except newline), zero and more times <
matches the characters<
literally\/
matches the character/
literally\1
matches the same text matched by the first capturing group:([\w]+)
>
matches the characters>
literally
Matching duplicated words
![duplications match HTML TAG match](https://uploads.sitepoint.com/wp-content/uploads/2015/09/1442490495url.jpg)
- The words are space separated
- We must match every duplication – non-consecutive ones as well
\b(\w+)\b(?=.*\1)
This regular expression seems challenging but uses some of the concept previously shown.
The pattern introduces the concept of word boundaries.
A word boundary \b
mainly checks positions. It matches when a word character (i.e.: abcDE
) is followed by a non-word character (Ie: -~,!
).
Below you can find some example uses of word boundary to make it clearer:
– Given the phrase Regular expressions are awesome
– The pattern \bare\b
matches are
– The pattern \w{3}\b
could match the last three letters of the words: lar, ion, are, ome
The expression above could be explained as:
- Match every word character followed by a non-word character (in our case space)
- Check if the matched word is already present or not
\b
word boundary- capturing group
([\w]+)
matches any word charactera-zA-Z0-9_
\b
word boundary(?=.*\1)
positive lookahead assert that the following can be matched:.*
matches any character (except newline)\1
matches same text as first capturing group
preg_match_all
for more information.
Final thoughts
Regular expressions are double-edged swords. The more complexity is added, the more difficult it is to solve the problem. That’s why, sometimes, it’s hard to find a regular expression that will match all the cases, and it’s better to use several smaller regex instead. Having a good scenario of the problem could be very helpful, and will allow you to start thinking of the character range, constraints, assertions, repetitions, optional values, etc. Paying more attention to group captures will make the matches useful for further processing. Feel free to improve the expressions in the examples, and let us know how you do!Useful resources
Below you can find further information and resources to help your regex skills grow. Feel free to add a comment to the article if you find something useful that isn’t listed.Lea Verou – /Reg(exp){2}lained/: Demystifying Regular Expressions
https://www.youtube.com/watch?v=EkluES9RvakPHP libraries
Name | Description |
---|---|
RegExpBuilder | Creates regex using human-readable chains of methods |
NooNooFluentRegex | Builds Regex expressions using fluent setters and English language terms like above |
Hoa\Regex | Provides tools to analyze regex and generate strings |
Regex reverse | Given a regular expression will generate a string |
Websites
URL | Description |
---|---|
regex101.com | PCRE online regex tester |
regextester.com | PCRE online regex tester |
rexv.org | PCRE online regex tester |
debuggex.com | Supports PCRE and provides a very useful visual regex debugger |
regexper.com | Javascript style regex, but useful for debug |
phpliveregex.com | Online tester for preg functions |
regxlib.com | Database of regular expressions ready to use |
regular-expressions.info | Regex tutorials, books review, examples |
Books
Title | Description | Author | Editor |
---|---|---|---|
Mastering Regular Expressions | The must have regex book | Jeffrey Friedl | O’Reilly |
Regular Expression Pocket Reference | Regular Expressions for Perl, Ruby, PHP, Python, C, Java and .NET | Tony Stubblebine | O’Reilly |
Frequently Asked Questions (FAQs) about Regular Expressions (Regex)
What are some practical applications of Regular Expressions (Regex)?
Regular expressions (Regex) are incredibly versatile and can be used in a variety of practical applications. They are commonly used in data validation to ensure that user input matches a specific format, such as an email address or phone number. They can also be used in web scraping to extract specific pieces of information from a webpage. In addition, Regex can be used in text processing for tasks such as finding and replacing specific strings of text, splitting a string into an array of substrings, and more.
How can I create complex Regular Expressions (Regex)?
Creating complex regular expressions involves understanding and combining various Regex components. These include literals, character classes, quantifiers, and metacharacters. By combining these components in different ways, you can create regular expressions that match a wide variety of patterns. For example, you could create a regular expression that matches email addresses, phone numbers, or URLs.
What are some common mistakes to avoid when using Regular Expressions (Regex)?
Some common mistakes to avoid when using regular expressions include overusing or misusing certain components, such as the dot (.) or asterisk (*), which can lead to unexpected results. Another common mistake is not properly escaping special characters when they are meant to be interpreted literally. Additionally, it’s important to remember that regular expressions are case-sensitive by default, so you need to use the appropriate flags if you want to ignore case.
How can I test my Regular Expressions (Regex)?
There are several online tools available that allow you to test your regular expressions. These tools typically allow you to enter a regular expression and a test string, and then they highlight the parts of the test string that match the regular expression. This can be a great way to debug your regular expressions and ensure they are working as expected.
Can Regular Expressions (Regex) be used in all programming languages?
Most modern programming languages support regular expressions in some form. However, the specific syntax and features supported can vary between languages. For example, JavaScript, Python, and Ruby all support regular expressions, but they each have their own unique syntax and features.
What are the performance implications of using Regular Expressions (Regex)?
While regular expressions can be incredibly powerful, they can also be resource-intensive if not used properly. Complex regular expressions can take a long time to execute, especially on large strings of text. Therefore, it’s important to use regular expressions judiciously and to optimize them as much as possible.
How can I optimize my Regular Expressions (Regex)?
There are several strategies for optimizing regular expressions. These include avoiding unnecessary quantifiers, using non-capturing groups when you don’t need the matched text, and using character classes instead of alternation where possible. Additionally, some regular expression engines offer optimization features, such as lazy quantifiers, that can improve performance.
What are some resources for learning more about Regular Expressions (Regex)?
There are many resources available for learning more about regular expressions. These include online tutorials, books, and interactive learning platforms. Additionally, many programming languages have extensive documentation on their regular expression syntax and features.
Can Regular Expressions (Regex) be used to parse HTML or XML?
While it’s technically possible to use regular expressions to parse HTML or XML, it’s generally not recommended. This is because HTML and XML have a nested structure that can be difficult to accurately capture with regular expressions. Instead, it’s usually better to use a dedicated HTML or XML parser.
What are some alternatives to Regular Expressions (Regex)?
While regular expressions are incredibly powerful, they are not always the best tool for the job. Depending on the task at hand, you might be better off using a different approach. For example, for simple string manipulation tasks, you might be able to use built-in string methods instead of regular expressions. For parsing HTML or XML, you would typically use a dedicated parser. And for complex text processing tasks, you might want to consider using a natural language processing library.
Nicola Pietroluongo is a software engineer with many years of experience, open source enthusiast, now creating and contributing to awesome PHP web projects.