Python defaultdict: Useful, Until It Silently Creates Keys

defaultdict is a dict subclass that calls a factory function to create a value whenever you access a key that does not exist, instead of raising KeyError. defaultdict(list) gives you an empty list, defaultdict(int) gives you a zero, and grouping or counting code stops being three lines of existence-checking and becomes one. It is genuinely one of the nicest things in the standard library. It also has a sharp edge that the tutorials skip: merely reading a missing key creates it. Not writing - reading. That behaviour is the point of the tool and the source of its only real bug class.

The reference pages show you the grouping example and stop. The interesting part is the failure mode, because it does not look like a failure - it looks like a dictionary that grew keys nobody added.

Table of contents

• What it replaces
• The part that bites: reading creates keys
• When to use it and when not to
• Nested defaultdicts, and the infinite one
• Counter is often the better answer
• How this fits the rest of the stack
• FAQ

What it replaces

Without it, grouping looks like this:

groups = {}
for user in users:
    if user.team not in groups:
        groups[user.team] = []
    groups[user.team].append(user)

With it:

from collections import defaultdict

groups = defaultdict(list)
for user in users:
    groups[user.team].append(user)

The argument to defaultdict is a callable - not a value. defaultdict(list) calls list() to make a fresh empty list per missing key. defaultdict(int) calls int(), which is 0. Passing defaultdict([]) is a TypeError, and it is the first mistake everyone makes.

The common factories:

• defaultdict(list) - grouping items under a key.
• defaultdict(int) - counting. Though collections.Counter is usually better for pure counting.
• defaultdict(set) - collecting unique values.
• defaultdict(dict) - nested structures.
• defaultdict(lambda: 'unknown') - any constant default.

The part that bites: reading creates keys

This is the whole warning, and it is worth being blunt about.

counts = defaultdict(int)
counts['alice'] += 1

# Just looking. Surely harmless?
if counts['bob'] > 0:
    print('bob has entries')

print(dict(counts))
# {'alice': 1, 'bob': 0}   <- bob now exists

bob was never added. It was read. The read created it, because that is precisely what defaultdict promises to do. There is no bug in the implementation - the bug is in the expectation that reading a dict is a non-mutating operation, which for every other dict is true.

Where this actually hurts: you build a defaultdict, some code inspects a few keys, and then you serialise it. Now your JSON has entries that exist only because something looked at them. If you are counting distinct users, your count is wrong. If you are writing this to a database, you have inserted phantom rows. The data looks plausible, which is what makes it expensive.

The fix is to read with .get() when you mean to read:

# Does not create the key.
if counts.get('bob', 0) > 0:
    ...

# Also safe.
if 'bob' in counts:
    ...

in and .get() never trigger the factory. Only subscript access does.

When to use it and when not to

Reach for defaultdict when:

• You are accumulating into a container - grouping, appending, adding to a set.
• Every key you touch is a key you intend to create.
• The dict is short-lived and local to a function.

Use a plain dict when:

• The dict escapes the function - returned, serialised, cached, or handed to code you do not control.
• Missing keys are meaningful. If a KeyError means a genuine bug, you want that KeyError.
• You are reading more than writing. That is the exact shape of the phantom-key problem.
• You are counting - Counter is purpose-built and does not surprise you.

A useful habit: build with defaultdict, then convert before returning. return dict(groups) hands the caller a normal dict with normal behaviour, and nothing downstream inherits the sharp edge.

Nested defaultdicts, and the infinite one

defaultdict(dict) gives one level of nesting.

tree = defaultdict(dict)
tree['a']['b'] = 1   # works
tree['a']['b']['c'] = 1   # TypeError: inner dict is a plain dict

For arbitrary depth, the recursive trick:

def infinite():
    return defaultdict(infinite)

tree = infinite()
tree['a']['b']['c']['d'] = 1   # works, any depth

This is clever and I would think hard before shipping it. Every typo in a key path silently creates a branch instead of raising, which means a misspelled path produces no error and no data - just a quietly growing tree of empty dicts. Debugging that is not fun, and the alternative - a real class, or just being explicit - is usually clearer to whoever reads it next.

Counter is often the better answer

If you are counting, defaultdict(int) works, but Counter is better.

from collections import Counter

counts = Counter(user.team for user in users)
print(counts.most_common(3))

Counter gives you most_common(), arithmetic between counters, and - crucially - reading a missing key returns 0 without creating it. It is a defaultdict-like that does not have the phantom-key problem.

So: defaultdict for accumulating containers, Counter for counting. Using defaultdict(int) to count is a mild code smell that suggests the author did not know Counter existed.

How this fits the rest of the stack

A phantom key is a small bug with a real cost: wrong counts in a report, extra rows in a database, a cache entry per key anyone ever looked at. The last one is memory, and memory is a line item - a process whose dict grows on read is a process that gets restarted by the OOM killer eventually. When you are sizing the service that runs this code, the RunxBuild hosting calculator shows compute, memory, the database, and bandwidth together, so the size you pick is a decision rather than a guess. The RunxBuild dashboard is where the team watches what it actually uses.

Useful related references:

• Python Iterate a List: for, enumerate, list comprehension, and map
• Private Methods in Python: There Are None, and That Is Fine
• Build a Python VPN Client: When It Makes Sense and When It Doesn’t
• Python services on RunxBuild

FAQ

What is the difference between defaultdict and dict.get?

dict.get(key, default) returns a default without touching the dictionary - it is a pure read. defaultdict inserts the generated value into the dictionary as a side effect of the subscript access. If you only need a fallback value on read, .get() is the correct tool and it does not mutate anything. Use defaultdict when you genuinely intend every accessed key to exist afterwards.

Why did a key appear in my defaultdict without me adding it?

Because you read it with subscript access. d['missing'] on a defaultdict calls the factory and stores the result, so a read creates the key. This is the documented behaviour, not a bug, but it surprises people because reading a normal dict never mutates it. Use d.get('missing') or 'missing' in d when you want to inspect without creating.

Can you nest defaultdicts in Python?

Yes. defaultdict(dict) gives one level of nesting. For arbitrary depth, use a recursive factory: def infinite(): return defaultdict(infinite). Be careful with the recursive version in production code - every mistyped key path silently creates an empty branch instead of raising an error, which turns typos into invisible data loss.

Is defaultdict faster than a regular dict?

Marginally, for accumulation patterns, because it avoids a separate existence check per iteration. The difference is small and is not a good reason to choose it. Choose it for readability when you are accumulating into containers, and avoid it when the phantom-key behaviour would be a hazard - correctness beats a micro-optimisation here.

Should I use defaultdict or Counter for counting?

Counter. It is purpose-built for counting, provides most_common() and arithmetic between counters, and returns 0 for missing keys without inserting them - so it avoids the phantom-key problem entirely. defaultdict(int) works but offers no advantage, and it carries the sharp edge that Counter does not.