Duplicate Reference Images: Remove Redundant Files Without Losing the Useful Version or Source Trail

A duplicate detector answers one question: do these two files hold the same data? It does not answer the question you actually have, which is which of them should survive. That decision belongs to a criterion you name in advance — the framing the current page uses, the copy an active document links to, the copy whose source record you can still vouch for. Name it and the answer is usually obvious. Leave it unnamed and a similarity percentage will make the decision for you, using a rule you never chose.

So the sequence is fixed. Establish what kind of sameness was actually detected. Look at the context the comparison cannot see. Declare the keeper criterion. Preserve the source trail. Then, only then, stage removals reversibly, with the retained paths and the quarantined paths both written down. When the set is small, the honest outcome of that sequence may be to keep every file and write the map anyway.

Three tests wear the same word

"Duplicate" covers at least three different measurements, and they are not degrees of the same thing.

Byte identity is a hash over the file. Two files with the same digest have the same bytes, including every embedded field, every thumbnail blob and any trailing data the encoder left behind. This is the only one of the three tests that is a statement about the files rather than about the pictures. It is cheap and decisive. It also answers less than people expect: a digest can tell you two files are the same, but it can never tell you which arrived first. Copying produces identical bytes in both directions, and a copy inherits whatever date it was carrying.

Decoded-pixel identity is what you get when a re-encode, a metadata strip or a colour-profile conversion changes the bytes while the picture stays the same. Proving it means decoding both files and comparing pixels, which requires deciding on dimensions, decoder and colour handling. "Identical pixels" is therefore a claim relative to a method, not a property of the world. Two files can be pixel-identical under one comparison and not under another.

Perceptual similarity is a score. The Similarity View page of the digiKam manual (https://docs.digikam.org/en/left_sidebar/similarity_view.html, documentation text inspected 18 September 2026) describes visual fingerprints, settings for the duplicate search, and a configurable policy for selecting which image is presented as the reference. The percentage that comes out of that interface is a comparison result. It is not a statement about authorship, it is not a statement about provenance completeness, and it is not a statement about whether removal is safe.

The practical consequence is that a similarity scan hands you a list containing three different kinds of thing: your genuine copies, your re-encodes, and your crops, graded versions and resized exports. The last group are not duplicates. They are versions with jobs. A crop fixes attention on the part of the frame that carries the argument. A larger encode may be the only one that survives being shown on a big screen. A differently graded version may be the one the room responded to.

In July 2020 someone asked on SuperUser how to remove all duplicate photos in digiKam but keep the reference images (https://superuser.com/questions/1566897/). The question is well put, because it names the real problem: "the reference image" is a role, not a property of a file. The replies underneath are people's recipes, not a standard, and the vote counts are not evidence that any recipe is correct.

What no image comparison can see

Whether a copy needs to survive is frequently recorded outside the image entirely.

Reference context tends to live in three places at once: inside the file, in a sidecar beside it, and in a catalog or database. The metadata settings page of the digiKam manual (https://docs.digikam.org/en/setup_application/metadata_settings.html, inspected 18 September 2026) documents sidecar write modes, a preference governing which source is read, pending or lazy writes, and configuration of which fields are handled. The manual's own framing is the useful part: settings and format support determine what actually gets persisted, and the catalog, the embedded metadata and an XMP sidecar must not be assumed to be the same thing or interchangeable.

That has two consequences that matter before you delete anything. First, which record you are looking at depends on a setting — the same file can present one credit in one configuration and a different credit in another. Second, "this file has no metadata" is ambiguous in three ways: nothing was ever recorded, something was recorded somewhere else, or something was recorded and the write has not happened yet.

Then there is the case that no metadata model covers: identical bytes with two different external trails. A text note beside one copy, a line in a project's sources file beside the other. Nothing in the pixels distinguishes them, and nothing in a hash does either. This is the situation the whole procedure exists for.

Declare the criterion before you look at the list

The reference image the similarity interface highlights is chosen by criteria you can change. If adjusting those criteria changes which file is highlighted, then the highlight was a policy, not a fact about the file. Nothing about being the largest or the oldest makes a file the original; it makes it the largest or the oldest. Embedded capture timestamps travel with copies and can be edited, and filesystem times change on copy, so neither establishes the order of events on your disk.

There is often no original in a reference set at all — only a nearest ancestor, and not always that. What you can usually establish is which copy is currently doing work. So pick a criterion that can actually be supported:

• the encode you would use if the pitch goes large;
• the framing that the page it appears on depends on;
• the copy whose source is still traceable;
• the copy an active document already links to;
• the copy the catalog currently displays to whoever opens it.

These point at different files, which is the point. Write the criterion down next to the decision, in the same note as the file list. A criterion recorded after the fact tends to be a description of what you did, not a reason for doing it.

A set to work on

The following is invented so the reasoning has something concrete to hold. The filenames, encodings, adjacent records and page references are stipulated; nothing here was produced by running a duplicate search, and no application was opened.

Now apply two criteria to the same five files.

Criterion A — keep the copy with the most complete embedded provenance. The only file with an embedded creator and source URL is courtyard_a.png, so it survives and courtyard_b.png plus its sidecar are proposed for removal. But the atrium pair carries no embedded provenance at all, so this criterion does not distinguish them. The decision falls to an undeclared tie-break — folder neatness, say — which keeps refs/atrium.png and proposes removing client_acme/atrium.png. That is the copy treatment page 2 points at. The criterion never mentioned links; the tie-break quietly broke one.

Criterion B — keep whatever the catalog currently displays, plus whatever an active document points at. Under a sidecar-preferring read configuration, the credit shown for the courtyard pair comes from courtyard_b.png.xmp, so courtyard_b.png survives and courtyard_a.png is proposed for removal. On the atrium side the linked copy is client_acme/atrium.png, so that one stays and refs/atrium.png goes.

Same five files, same evidence, opposite proposal for the courtyard pair. Neither criterion touches courtyard_crop.png, which both correctly recognise as a version with a job — yet a perceptual scan will almost certainly list it as a candidate, so a batch accept on the scan's output would remove a file that no declared criterion wanted removed.

Note also that the byte-identical atrium pair is only interchangeable at the level of bytes. Their paths are load-bearing. Keeping client_acme/atrium.png costs nothing and preserves the link. Keeping refs/atrium.png instead would require editing treatment page 2 — a real change, recorded somewhere, by someone.

Carry the trail forward before you move anything

Whichever criterion you adopt, the context attached to the losing file has to survive the loss. Map each candidate to the record that will outlive it, and transcribe what exists in only one place.

Under Criterion B, the creator name and collection URL live only inside the bytes of courtyard_a.png. Those need to be written out — into the retained record, if not into the retained file — and then read back to confirm they landed where you expect. This is where a blind metadata merge does damage. courtyard_b.png already has a sidecar with a different credit. A merge that produces one tidy credit has not preserved the trail; it has manufactured agreement between two sources that never agreed. Record both, note which carrier each came from, and note which one the catalog is currently displaying.

The same applies to the atrium pair, and here the two external records collide rather than simply fall silent. atrium.source.txt beside refs/atrium.png names a photographer's portfolio page and carries the "internal use, pitch only" restriction, which appears nowhere else. The line in sources.md beside the client copy attributes the picture to an image-board post and says nothing about licence. Removing refs/atrium.png would take the first record with it, leaving only the second. Its silence is not permission, and it is not agreement either: the two records disagree about where the picture came from, and only one of them states a use restriction. Write the conflict as it stands. Use restrictions belong to the picture, not to the folder.

A retained-record entry that is actually useful carries: path, digest, dimensions, what is embedded, what sits in a sidecar, what the catalog shows, every external source with its date, the use restriction, the documents that reference it, and an explicit note of what is still unknown or disputed.

Make the route reversible

Two mechanisms, chosen by consequence rather than by file count.

Flag in place when moving a file would break a live link. courtyard_crop.png is referenced from page 4 and client_acme/atrium.png from page 2. Marking them in place — tag, note, spreadsheet row — leaves every reference intact while still recording that the file is under review.

Quarantine only when the consequences are understood and the return path is written down. A quarantine folder is reversible exactly to the extent that its manifest records each file's original path, digest, the reason it was moved, and the adjacent records that had to travel with it. A quarantined file with no recorded original path has been deleted, just more slowly.

Before any removal that someone has actually authorised, verify the things that would break: the treatment pages, any exports already generated, and the metadata read-back. This article does not grant permission to delete anything because a comparison scored two files alike.

Two complete alternatives, same conditions

Held constant: Criterion B, the five-file set above, and one inspection pass.

Alternative A — retain all five and write the map. Cost: roughly the size of two PNG files, which for this set is single-digit megabytes, plus the time to write the retained-record entries. Gain: no transcription to get wrong, no link to repair, no dependence on the read configuration staying where it is.

Alternative B — quarantine courtyard_a.png and refs/atrium.png with their context carried forward. Cost: two transcriptions, one of them against a conflicting sidecar; a quarantine folder that has to remain findable; and a decision that now depends on a setting nobody may revisit. Gain: two files fewer.

For five files, A is the better trade, and it would be strange to pretend otherwise. B starts to pay when the count is large enough that the storage or the confusion is real, or when the duplicates cause active harm — a stale credit rendering in a client-facing export, a wrong version pulled into a live page, two people picking different copies of the same picture for the same slide. Those are the conditions that justify the transcription work. The size of the folder is not, by itself.

What is not optional in either alternative is the map. Without it, retaining everything still leaves you unable to say which copy to send, and removing anything leaves you unable to say what was lost.

What to end up holding

If the work is done under Criterion B, the note at the top of the map reads as follows. Retained: courtyard_b.png with its sidecar, because the displayed credit comes from there and the file is doing current work; courtyard_crop.png, flagged in place, because page 4 depends on that framing; client_acme/atrium.png, flagged in place, because page 2 points at that path. Held for review, not removed: courtyard_a.png, pending confirmation that its embedded creator and collection URL have been transcribed and read back; refs/atrium.png with atrium.source.txt, because the restriction note travels with it and no page has been checked for a reference to that path. Open questions: which of the two credits for the courtyard picture is correct, and which atrium source record is correct — and, given that only one of them states a restriction while the other is silent on licence, what the licence status of that picture actually is.

If the set is small, the honest version of that note is shorter: retained, all five; context, mapped; held for review, none. Less storage is worth having. Less knowledge is not the price worth paying for it, and the deletion count is not the measure of a job done well.

Frequently asked questions

Why can't a high similarity score tell me which duplicate reference image to keep?

A similarity percentage is a comparison result, not a statement about authorship, provenance completeness, or whether removal is safe. A perceptual scan can mix genuine copies, re-encodes, and crops, graded versions, or resized exports that are versions with jobs. The keeper decision needs a criterion named in advance.

What are byte identity, decoded-pixel identity, and perceptual similarity?

Byte identity is a hash over the file; the same digest means the same bytes, including embedded fields, thumbnail blobs, and trailing data. Decoded-pixel identity is what can remain when a re-encode, metadata strip, or colour-profile conversion changes bytes while the picture stays the same; proving it depends on chosen dimensions, decoder, and colour handling. Perceptual similarity is a score from visual fingerprints, settings, and a configurable policy for which image is presented as the reference.

How can metadata outside the image complicate removing a duplicate?

Reference context tends to live inside the file, in a sidecar beside it, and in a catalog or database. Settings and format support determine what actually gets persisted and which source is read, so the same file can present one credit in one configuration and a different credit in another. “This file has no metadata” is ambiguous: nothing was ever recorded, something was recorded somewhere else, or something was recorded and the write has not happened yet.

What is the difference between flagging in place and quarantine?

Flag in place when moving a file would break a live link; a tag, note, or spreadsheet row leaves references intact while recording that the file is under review. Quarantine only when the consequences are understood and the return path is written down. A quarantine manifest must record each file’s original path, digest, reason it was moved, and adjacent records that had to travel with it; a quarantined file with no recorded original path has been deleted, just more slowly.

When is keeping all duplicates better than removing some?

For a small set, retaining all files and writing the map can be the better trade. The cost is roughly the storage of the files plus the time to write retained-record entries; the gain is no transcription to get wrong, no link to repair, and no dependence on the read configuration staying where it is. Removal starts to pay when the count is large enough that storage or confusion is real, or when duplicates cause active harm such as a stale credit rendering or the wrong version being pulled into a live page. The map is not optional either way.

Further reading

• Trace a Treatment Reference Back to Its Source Before You Describe It — When the surviving copy's only record is an unfamiliar page or caption, read this to log attribution and permission separately before retiring the other copy.
• Archive Treatment Work So Reusable Methods Stay Separate From Client Material — Read this when writing the retained-record entry and the conflicts it holds, so permitted purpose and unresolved questions are recorded rather than tidied away.