Python Dictionaries: the Key That Isn't There

I wrote a word counter in eleven lines. It ran on my test sentence, printed what I expected, and I closed the editor.

Then came a paragraph where the word "the" appears twice, and it stopped at line 7 with KeyError: 'the'. Not a wrong answer — no answer at all.

If you have written the same loop, you already know roughly where the fix goes. What takes longer than the fix is knowing which of a dict's three lookups belongs in which situation, and that is the part this article is about.

Two files, one command

Every article in this series works the same way: you write the program, the article gives you a checker, and you run the checker until it goes from red to green. No repository to clone, no Docker, no account: two files and python3.

Create wordcount.py with this, in a folder with nothing else in it:

# wordcount.py
import sys

text = sys.stdin.read().lower()
counts = {}

for word in text.split():
    counts[word] += 1

for word in sorted(counts):
    print(f"{word}: {counts[word]}")

Then create check_wordcount.py next to it. It runs your file the way you would, with a sentence piped in, and reads what comes out:

# check_wordcount.py
import subprocess
import sys

SENTENCE = "The cat and the dog and the bird"
EXPECTED = {"and": 2, "bird": 1, "cat": 1, "dog": 1, "the": 3}


def run_wordcount():
    return subprocess.run(
        [sys.executable, "wordcount.py"],
        input=SENTENCE,
        text=True,
        capture_output=True,
    )


def parse(stdout):
    counted = {}
    for line in stdout.splitlines():
        word, _, count = line.partition(":")
        if word.strip() and count.strip():
            counted[word.strip()] = count.strip()
    return counted


def report(results):
    for label, passed, detail in results:
        mark = "PASS" if passed else "FAIL"
        note = f"  <- {detail}" if detail and not passed else ""
        print(f"{mark}  {label}{note}")
    passed = sum(1 for _, ok, _ in results if ok)
    print(f"\n{passed} of {len(results)} checks passed")


def main():
    run = run_wordcount()
    crashed = run.returncode != 0
    last_error = run.stderr.strip().splitlines()[-1] if run.stderr.strip() else ""

    results = [("the script runs without crashing", not crashed, last_error)]
    if crashed:
        report(results)
        return 1

    counted = parse(run.stdout)
    results.append(
        ("every word in the sentence is counted once", sorted(counted) == sorted(EXPECTED), str(sorted(counted)))
    )
    for word, expected in sorted(EXPECTED.items()):
        results.append((f"'{word}' is counted {expected} time(s)", counted.get(word) == str(expected), f"got {counted.get(word)}"))

    report(results)
    return 0 if all(ok for _, ok, _ in results) else 1


if __name__ == "__main__":
    sys.exit(main())

You do not have to understand the checker yet: it reads your program's output, not your program. Run it:

$ python3 check_wordcount.py
FAIL  the script runs without crashing  <- KeyError: 'the'

0 of 1 checks passed

Red on the first check, and it stops there: while the script crashes, the counts are not worth measuring.

Why the first line fails

counts[word] += 1 looks like one operation. It is three:

1. read counts[word],
2. add 1,
3. write the result back to counts[word].

The read happens first, on a dict that has no shelves yet. {} holds no keys, and Python's dict does not invent a default for the subscript syntax. A missing key is an error, not a zero. That strictness is the feature: counts["teh"] returning 0 would hide every typo in your own key names as well.

Three ways to ask a dictionary for a key

They differ only in what happens when the key is not there.

• counts[word] raises KeyError. Right when the key must exist: a config field, a record you just inserted.
• counts.get(word) returns None; counts.get(word, 0) returns your default. Right when absence is a normal state, which for a counter it is: a word is absent exactly until you count it.
• counts.setdefault(word, 0) returns the value and inserts the default if it was missing. Right when you are about to append to or extend that value, such as a list of orders per customer or a set of tags per post.

And word in counts answers "is it there" without touching the value, which is what you want when the question is about membership rather than a count.

The fix, in one line

# wordcount.py
import sys

text = sys.stdin.read().lower()
counts = {}

for word in text.split():
    counts[word] = counts.get(word, 0) + 1

for word in sorted(counts):
    print(f"{word}: {counts[word]}")

$ python3 check_wordcount.py
PASS  the script runs without crashing
PASS  every word in the sentence is counted once
PASS  'and' is counted 2 time(s)
PASS  'bird' is counted 1 time(s)
PASS  'cat' is counted 1 time(s)
PASS  'dog' is counted 1 time(s)
PASS  'the' is counted 3 time(s)

7 of 7 checks passed

get(word, 0) gives the read a value when the key is absent, so the increment always has something to add to. The dict still never invents a default for counts[word]. You asked for one explicitly, on the one line where absence is expected.

Why a dict is fast

A dict does not walk through its contents looking for your key. It hashes the key and jumps straight to the slot, so a lookup costs about the same whether the dict holds ten entries or a million.

A list is the opposite: to answer "is this word in there", it checks one element at a time until it finds a match. On a list of 1,000,000 words, that took 18 milliseconds. The same question to a dict took less than a thousandth of a millisecond, about 130,000 times faster.

That gap is the whole reason counts is a dict and not a list of pairs, and why "I'll just search the list" stops being a plan once the data no longer fits on one screen.

Two properties of dicts worth knowing now:

• Insertion order is preserved (since Python 3.7). sorted(counts) in the script is there for readability, not correctness — iterating counts directly would have kept the order the words first appeared.
• Keys must be hashable. Strings, numbers, and tuples can be keys; lists cannot. You will see why in the next article.

The builtin that exists for exactly this

Counting occurrences is common enough that the standard library ships it:

from collections import Counter

counts = Counter(text.split())
print(counts.most_common(3))

Counter is a dict subclass; most_common(3) is the part you would otherwise write by hand. The rule of thumb from here on: when the job is counting, reach for Counter; when the key is a record you look up by name, write the dict yourself. Use get(key, default) when absence is expected, and setdefault when you are building a nested structure. Use [] only when a missing key really is a bug, because that is exactly what it will raise.

Before the next article

Change SENTENCE in the checker to a sentence of your own and update EXPECTED to match. Then remove .lower() from wordcount.py and run the check again — one of the seven checks catches it, and the reason it catches it is the next thing worth understanding about keys.

That is the deal this series makes: the checker is not a certificate, it is a second reader. It only knows what you told it to expect.