Python Regex Substitute: re.sub, Backreferences, and the Escaping Traps

Q: What is the difference between re.sub and str.replace?

str.replace matches a literal string; re.sub matches a regular expression. For a fixed string, str.replace is faster and cannot be surprised by metacharacters in the input. Use re.sub when you need variable-length matching, capture groups, or alternatives.

Q: How do I use a variable safely inside a regex pattern?

Wrap it in re.escape(). Without that, characters like ., , + and ( are interpreted as regex syntax rather than matched literally, which silently changes what matches and can raise a syntax error on input containing brackets.

Q: How do I put a backslash in the replacement string?

Pass a function returning the replacement instead — re.sub(pattern, lambda m: path, text). Function return values are used literally with no escape processing, which avoids doubling backslashes and is the cleanest way to substitute Windows paths.

re.sub(pattern, replacement, string) replaces every non-overlapping match and returns a new string — the original is unchanged, because Python strings are immutable. The two things that bite are backslashes in the replacement string and forgetting that the pattern is regex, so characters in user input are interpreted rather than matched literally.

re.sub is one of the most useful functions in the standard library and one of the easiest to use slightly wrong in a way that only shows up on unusual input.

The basics take two minutes. The rest of this is about the cases where it does something other than what you meant.

The basics
Backreferences and named groups
A function as the replacement
Escaping user input, which is a correctness bug
When not to use regex at all
How this fits the rest of the stack
FAQ

The basics

import re

re.sub(r'\s+', ' ', 'too   many    spaces')
# 'too many spaces'

re.sub(r'\d+', 'N', 'order 12345 shipped 67')
# 'order N shipped N'

re.sub(r'cat', 'dog', 'cat catalog', count=1)
# 'dog catalog'   -- count limits replacements

Always use a raw string for the pattern. Without the r prefix, Python processes the backslashes first and the regex engine receives something different from what you wrote:

re.sub('\d+', 'N', text)    # works by luck -- \d is not a Python escape
re.sub('\\b', 'X', text)    # needs doubling without r''
re.sub(r'\b', 'X', text)    # what you meant

\b is the clearest example: '\b' in a normal Python string is a backspace character, not a word boundary.

The related functions:

re.subn(r'\d', 'N', 'a1b2')    # ('aNbN', 2) -- also returns the count

pattern = re.compile(r'\d+')    # compile once for repeated use
pattern.sub('N', text)

Compiling matters in a loop over many strings. For a handful of calls, re caches recent patterns internally and the difference is negligible.

Backreferences and named groups

Groups captured in the pattern are available in the replacement as \1, \2, and so on:

re.sub(r'(\w+)@(\w+)\.com', r'\2 user: \1', 'contact [email protected]')
# 'contact example user: bob'

re.sub(r'(\d{4})-(\d{2})-(\d{2})', r'\3/\2/\1', '2026-08-26')
# '26/08/2026'

The replacement string also needs to be raw, for the same reason as the pattern.

Named groups are worth using the moment there are more than two:

re.sub(
    r'(?P<user>\w+)@(?P<domain>[\w.]+)',
    r'\g<domain>/\g<user>',
    '[email protected]'
)
# 'example.com/bob'

\g<name> is also the way to disambiguate a numbered group followed by a digit — \g<1>0 means group 1 then a literal zero, whereas \10 means group 10:

re.sub(r'(\d)', r'\g<1>0', '5')    # '50'
re.sub(r'(\d)', r'\10', '5')       # error: invalid group reference

A function as the replacement

When the replacement depends on what was matched, pass a callable. It receives the match object and returns the replacement string. This is where re.sub becomes genuinely powerful.

def double(match):
    return str(int(match.group()) * 2)

re.sub(r'\d+', double, 'a 5 b 10')
# 'a 10 b 20'

A practical example — redacting values while keeping the structure readable:

def redact(match):
    key, value = match.group('key'), match.group('value')
    if key.lower() in {'password', 'token', 'secret', 'api_key'}:
        return f'{key}=<redacted>'
    return match.group()

re.sub(r'(?P<key>\w+)=(?P<value>\S+)', redact, log_line)

One important detail: the string returned by the function is used literally. Backreference syntax is not processed, so a returned \1 stays as those two characters. That is usually what you want, and it removes a whole class of escaping problem.

The function form is also the clean way to do conditional replacement, which regex alternation handles badly.

Escaping user input, which is a correctness bug

If any part of your pattern comes from user input or a variable, escape it. Otherwise a . matches any character, a ( is a syntax error, and a * changes the meaning entirely.

term = user_input          # e.g. 'a.b' or 'C++'

re.sub(term, 'X', text)              # wrong: regex metacharacters active
re.sub(re.escape(term), 'X', text)   # right: matched literally

The same applies to the replacement side, where the concern is backslashes rather than metacharacters:

path = r'C:\Users\new'
re.sub(r'PLACEHOLDER', path, template)
# error or corruption -- \U and \n are interpreted

re.sub(r'PLACEHOLDER', path.replace('\\', '\\\\'), template)  # awkward
re.sub(r'PLACEHOLDER', lambda m: path, template)              # better

The lambda form sidesteps the problem entirely, because function return values are not processed for escapes. It is the idiom worth reaching for whenever the replacement contains a path, a Windows filename, or anything else with backslashes.

There is also a denial-of-service angle. Certain patterns backtrack catastrophically on crafted input — (a+)+b against a long run of a characters can hang for minutes. Never build a pattern from untrusted input, and be cautious with nested quantifiers.

When not to use regex at all

re.sub is frequently reached for when a simpler tool is correct, faster, and more readable.

text.replace('old', 'new')          # fixed string -- no regex needed
text.strip()                        # trimming whitespace
text.split(',')                     # simple splitting
' '.join(text.split())              # collapse all whitespace

str.replace is several times faster than re.sub for a literal string, and it cannot be surprised by metacharacters. If the thing you are matching has no pattern in it, do not use a pattern engine.

For multiple fixed replacements, str.translate handles single characters efficiently, and a loop of str.replace is clearer than an elaborate alternation.

The cases where regex is genuinely the right tool are narrower than habit suggests:

Variable-length or structural matching — digits, whitespace runs, word boundaries.
Capturing and rearranging parts of a match.
One expression standing in for several alternatives.
Validation where the shape matters more than the exact content.

And the perennial one: do not parse HTML, XML, or JSON with regex. Use a parser. Nested structure is exactly what regular expressions cannot express, and the version that works on your sample will fail on real input.

How this fits the rest of the stack

Most re.sub bugs come from the same root — a string being treated as a pattern when it was meant literally, or a replacement being processed for escapes when it was meant verbatim. re.escape and a lambda replacement handle both, and they cost nothing.

The log-redaction example above hints at where this usually lives in a real application: cleaning output before it goes somewhere it will be read. That is worth doing carefully, because a token that reaches a log has been written down somewhere you did not intend. RunxBuild runs Python services from a GitHub repository with per-deploy build and runtime logs and environment variables held per service, so secrets stay out of the code that produces those logs in the first place. The RunxBuild hosting calculator shows the service and database costs separately.

Useful related references:

FAQ

What is the difference between re.sub and str.replace?

str.replace matches a literal string; re.sub matches a regular expression. For a fixed string, str.replace is faster and cannot be surprised by metacharacters in the input. Use re.sub when you need variable-length matching, capture groups, or alternatives.

Why do I get an invalid group reference error?

Usually because a numbered backreference is followed by a digit — \10 is read as group 10, not group 1 followed by a zero. Use \g<1>0 to disambiguate. The same syntax works for named groups as \g<name>.

How do I use a variable safely inside a regex pattern?

Wrap it in re.escape(). Without that, characters like ., *, + and ( are interpreted as regex syntax rather than matched literally, which silently changes what matches and can raise a syntax error on input containing brackets.

How do I put a backslash in the replacement string?

Pass a function returning the replacement instead — re.sub(pattern, lambda m: path, text). Function return values are used literally with no escape processing, which avoids doubling backslashes and is the cleanest way to substitute Windows paths.

Can I use re.sub to parse HTML?

No. Nested structure is precisely what regular expressions cannot express, so a pattern that works on your test input will fail on real markup. Use an HTML parser such as lxml or BeautifulSoup.