Python Sets: the Duplicate That Keeps Coming Back

A tag list came out of a spreadsheet. Five rows, four tags, two of them the same word typed differently.

I deduplicated it with the loop everyone writes, and got four tags back. The loop was not wrong in any way I could see: it checked every tag against the ones already kept, and it never kept the same string twice.

Two files, one command

Same contract as the rest of this series: you write the program, the article hands you a checker, you run it until it goes from red to green. No repository, no Docker, just python3.

Create unique_tags.py:

# unique_tags.py
import sys

tags = [line.strip() for line in sys.stdin if line.strip()]

seen = []
for tag in tags:
    if tag not in seen:
        seen.append(tag)

print("\n".join(seen))

Then check_unique_tags.py next to it. It feeds in the spreadsheet rows and reads what comes out:

# check_unique_tags.py
import subprocess
import sys

POST = """python
Python
web
PYTHON
web
"""
EXPECTED = ["python", "web"]


def run_script():
    return subprocess.run(
        [sys.executable, "unique_tags.py"],
        input=POST,
        text=True,
        capture_output=True,
    )


def report(results):
    for label, passed, detail in results:
        mark = "PASS" if passed else "FAIL"
        note = f"  <- {detail}" if detail and not passed else ""
        print(f"{mark}  {label}{note}")
    passed = [ok for _, ok, _ in results].count(True)
    print(f"\n{passed} of {len(results)} checks passed")


def main():
    run = run_script()
    crashed = run.returncode != 0
    last_error = run.stderr.strip().splitlines()[-1] if run.stderr.strip() else ""

    results = [("the script runs without crashing", not crashed, last_error)]
    if crashed:
        report(results)
        return 1

    lines = [line.strip() for line in run.stdout.splitlines() if line.strip()]
    lowered = [line.lower() for line in lines]

    results.append(("prints one tag per line", len(lines) == len(run.stdout.splitlines()), str(lines)))
    results.append(("leaves two tags", len(lines) == len(EXPECTED), f"got {len(lines)}: {lines}"))
    results.append(("no tag appears twice", len(set(lowered)) == len(lowered), str(lines)))
    results.append(("the tags are lowercase", lines == lowered, str(lines)))
    results.append(("the first appearance order is kept", lowered == EXPECTED, str(lines)))

    report(results)
    return 0 if all(ok for _, ok, _ in results) else 1


if __name__ == "__main__":
    sys.exit(main())

$ python3 check_unique_tags.py
PASS  the script runs without crashing
PASS  prints one tag per line
FAIL  leaves two tags  <- got 4: ['python', 'Python', 'web', 'PYTHON']
FAIL  no tag appears twice  <- ['python', 'Python', 'web', 'PYTHON']
FAIL  the tags are lowercase  <- ['python', 'Python', 'web', 'PYTHON']
FAIL  the first appearance order is kept  <- ['python', 'Python', 'web', 'PYTHON']

2 of 6 checks passed

Four failures, one cause.

not in is not "looks the same"

tag not in seen asks one question: is this exact string already in the list. Python and python are two different strings to Python, the same way 2 and "2" are different values — the characters differ, so the comparison fails.

Which means the loop is only correct if the data is already normalised. Spreadsheet exports never are.

The fix has to happen while you read, not while you compare. Lower-case each tag before it enters the list, and the comparison problem disappears:

tags = [line.strip().lower() for line in sys.stdin if line.strip()]

A set answers "is it there" faster

Once the data is normalised, there is a simpler tool for the job. A set is a collection that cannot hold duplicates, so "have I seen this tag" becomes a single membership check instead of a walk through a list.

# unique_tags.py
import sys

tags = [line.strip().lower() for line in sys.stdin if line.strip()]

seen = set()
unique = []
for tag in tags:
    if tag not in seen:
        seen.add(tag)
        unique.append(tag)

print("\n".join(unique))

$ python3 check_unique_tags.py
PASS  the script runs without crashing
PASS  prints one tag per line
PASS  leaves two tags
PASS  no tag appears twice
PASS  the tags are lowercase
PASS  the first appearance order is kept

6 of 6 checks passed

Two containers, two jobs: the set answers "seen or not", the list keeps the order for the output. That is not a workaround for a language limitation, it is the normal division of labour.

The one-liner that loses the order

You will meet this soon, and it is worth knowing why it is a trap:

print("\n".join(set(tags)))

It does remove the duplicates. It also throws away the order, because a set has none to keep. Run it four times on the same six tags and you get four different orders:

['sql', 'web', 'go', 'python', 'rust', 'ai']
['go', 'rust', 'python', 'ai', 'web', 'sql']
['web', 'rust', 'go', 'ai', 'python', 'sql']
['rust', 'python', 'ai', 'sql', 'go', 'web']

If the order matters to the reader, the set cannot be the thing you print. Use it to decide, then print from the list.

What a set is good for

The obvious use is membership: if word in vocabulary where vocabulary is a set, instead of walking a list every time you ask. The less obvious use is comparison, and it is the one people reinvent badly.

Finding the tags two posts have in common is a nested loop by hand. With sets it is a & b. Tags that are new in the second post are a - b. Everything mentioned anywhere is a | b. Asking whether two posts overlap at all is bool(a & b), and it is true or false without a loop. None of it needs your own bookkeeping, and none of it can be subtly wrong the way an inner loop can.

That last part deserves a sentence. A hand-written overlap loop has three ways to be wrong: the wrong comparison, a duplicate appended twice, and a case that looks fine because the test data happened to match. Set operations have none of them, because you did not write the loop.

Two things a set cannot do. You cannot index it, so tags[0] is an error rather than a surprise. And its members must be hashable, which is why a set of lists is not a thing — lists cannot go in a set for the same reason they cannot be dict keys.

The rule of thumb: a list when order matters, a set when only membership does, a dict when each key carries a value. When you need both order and uniqueness, that is two containers doing two jobs, not one clever line.

Before the next article

Remove .strip() from the reading line, add one row with a trailing space (" web ") to POST, and run the check again. Five of six pass; the one that fails is about the shape of the output, not about duplicates. A line that looks fine in the terminal is not the same string as the tag without the space.

Put .strip() back and it passes. That is the argument for normalising while you read, in one run.