Verify a Copied Documentary Research Folder With a File Manifest

The transfer finishes, the drive unmounts, and the copy looks like the original: the same folder names, the same number of items, a total size that reads plausibly. None of that tells you what arrived. For documentary research — interview recordings, scans of photographs, permission notes, transcripts — the useful question is narrower and answerable: do the files you selected before the copy still have the same bytes after it?

That question has a method. Write down the source set before you copy it. Copy without moving anything. Inspect the destination independently. Compare path membership first, then compare the recorded checksums. Sort what you find into missing paths, changed bytes, and extra files, and deal with each group on its own terms.

The method has a boundary that matters as much as the method. A matching checksum says one file's bytes today match the bytes recorded earlier. It says nothing about whether the recording is what someone claims it is, whether the permission you were given covers what you are doing with it, or whether the story it documents is true. Integrity is the narrow, checkable part. Keep it narrow, and it stays useful.

Define the source set before copying it

Decide what is in scope and write it down. Name the authorized root — the folder you were cleared to copy — and name what you are leaving out. Exclusions should be explicit and recorded, not implied by whatever the copy tool happened to skip. A field drive might contain a scratch cache, a folder of odds and ends you have no permission to take, or a _rejected directory the researcher set aside. If those are out, say so in the manifest, so that a later comparison does not report them as missing.

Three things go into a manifest entry: the path relative to the root, the size in bytes, and a checksum produced by a named algorithm. Paths should be relative — interviews/2026-03-04-ranger-a.wav, not /Volumes/field-source/vernon-bridge-2026/interviews/2026-03-04-ranger-a.wav — because the whole point is to compare two folders that live in different places. The root itself is recorded once, separately, so you know which drive you inventoried.

SHA-256 is a reasonable default. MD5 and SHA-1 still appear in transfer tools because they are fast, but both have practical collision attacks and neither is worth defending in a new workflow. Name the algorithm in the manifest rather than assuming the reader — including your future self — knows which one produced the numbers.

An inventory has to walk the tree and read every selected file in full. That means a Python script that opens each file in binary read mode and feeds it through hashlib.sha256 in blocks, rather than reading a 40 GB ProRes file into memory at once. hashlib provides the primitive; the walking, the path handling, the error reporting and the record format are the parts you write and test.

A sketch of the shape, so the decisions are visible:

import hashlib, json, os, sys

def sha256_of(path, chunk=1024 * 1024):
    h = hashlib.sha256()
    with open(path, "rb") as fh:
        for block in iter(lambda: fh.read(chunk), b""):
            h.update(block)
    return h.hexdigest()

def inventory(root):
    records, errors = [], []
    for dirpath, dirnames, filenames in os.walk(root):
        for name in sorted(dirnames):
            full = os.path.join(dirpath, name)
            if os.path.islink(full):
                rel = os.path.relpath(full, root).replace(os.sep, "/")
                errors.append({"path": rel, "problem": "directory symlink not descended"})
        dirnames[:] = sorted(
            d for d in dirnames if not os.path.islink(os.path.join(dirpath, d))
        )
        for name in sorted(filenames):
            full = os.path.join(dirpath, name)
            rel = os.path.relpath(full, root).replace(os.sep, "/")
            if os.path.islink(full) or not os.path.isfile(full):
                errors.append({"path": rel, "problem": "not a regular file"})
                continue
            try:
                records.append({
                    "path": rel,
                    "bytes": os.path.getsize(full),
                    "sha256": sha256_of(full),
                })
            except OSError as exc:
                errors.append({"path": rel, "problem": str(exc)})
    return records, errors

Nothing here has been run for this article; it is a sketch to argue about, not a tested tool. The interesting parts are the decisions it makes. A symbolic link is never followed, and it is never silently skipped either: a linked file goes into errors as not a regular file, and a linked directory — which os.walk would otherwise step straight past, leaving everything beneath it neither hashed nor reported — is pruned from the descent and recorded by name, so a skipped subtree is always visible in the manifest. A read failure lands in errors too. The wrapper should write a JSON record containing the algorithm name, the absolute root, the tool version, a UTC start time, the entries, the error list, and exit with a non-zero status if errors is non-empty.

That exit status is the whole safeguard. If a directory holds seven files and the script cannot read one of them, a manifest with six entries and no complaint looks exactly like a folder that contained six files. You then compare six good records against six good records, get a clean report, and conclude that everything arrived. Run the script against a file you know is unreadable, on each platform you actually use, and confirm that the run fails loudly rather than producing a tidy, incomplete baseline.

Save the manifest outside the tree you are inventorying. Otherwise the manifest file becomes part of the folder, and the next run inventories it too, and the run after that has to explain why the file counts keep growing.

One caution about the phrase read-only. Hashing only reads file contents, so the source survives it. But on many filesystems a read updates the access time, so the source is not untouched in every metadata sense. That is not a problem for byte comparison. It is a reason not to use access times, or modification times, as evidence of anything.

Build the destination inventory under the same rules

Copy rather than move. A move removes the reference copy, and without it there is nothing to compare against except your memory of what the folder used to look like. Keep the source drive mounted and untouched through the whole check.

At the destination, run the same script with the same chunk size, the same algorithm, the same exclusion rules, and the same treatment of unreadable files. Point it at the new root — a different absolute path, which is why the manifest stores the root separately and compares relative paths only. Normalize path separators before comparing sets, since a Windows drive and a macOS or Linux drive will not agree about / and \, and be aware that filenames that differ only in case, or only in Unicode normalization, may be distinct files on one filesystem and the same file on another. Those collisions show up as a phantom pair: one path missing, one path unexpected.

Do not hash the destination and call it done. A checksum computed only after the copy has no earlier reference, so it establishes nothing about continuity — it tells you what the destination files are now, which you could have learned by looking. The comparison needs two records made at two recorded moments.

Not every entry can be inventoried as a file. Symbolic links, sockets, device nodes, and paths whose contents live on a remote volume that is currently unreachable are not ordinary readable files. A workflow can follow links, refuse them, or record them as unsupported, but it should not quietly pretend a link's target was checked or that a remote file's contents were hashed. Whichever rule you choose, write it into both runs, and treat every unsupported entry as something still unresolved.

Separate missing paths, changed bytes, and unexpected files

Compare membership before you compare bytes. Take the sorted set of relative paths in the source manifest and the sorted set of relative paths in the destination manifest. Paths on both sides go into the candidate group. Paths only in the source are missing. Paths only in the destination are unexpected. Only then do you look at checksums in the candidate group.

This ordering keeps the three kinds of trouble from blurring together, which matters because they call for different responses and because a single number can hide all three. Here is a constructed fixture — invented to show the shape of the report, not a captured session — with a source root of six selected files and a destination holding six files.

The source manifest, made before the copy, recorded:

The destination manifest, made afterward, came back like this:

• Four paths matched with identical checksums: both recordings, the scan, and transcripts/ranger-a.md.
• rights/permission-2026-02-19.txt was absent from the destination.
• transcripts/intake-notes.md was present at exactly 2,046 bytes, with a different checksum.
• transcripts/ranger-b-summary.md was present at 3,301 bytes, and no such path existed in the source.

Look at what a casual check would have said. The destination contains six files. The source contains six files. The file counts match. The destination's total is 81,766,393 bytes against the source's 81,764,296 — a difference of 2,097 bytes, which is just 3,301 arrived minus 1,204 departed, and which reads as "roughly the same." Change one number in the fixture and the totals match exactly: if the file that failed to copy had been 3,301 bytes, the aggregate byte total would have agreed perfectly while a path was still missing. Aggregates tell you something moved. They do not tell you what.

The changed file is worth dwelling on, because it is the case that size checks are built to miss. Its size was identical on both sides, because the only edit was a single digit in a date in the header — 2026-03-04 became 2026-03-05, the same number of characters. A byte-for-byte change of one character produces a completely different SHA-256 value, so the checksum group catches it immediately. Size never would have. That is the argument for checksums in one line: equal size is not equal content, and no amount of comparing sizes gets you past that.

Now the same constructed method applied to a folder with an unreadable file. Say a root holds seven files and one of them cannot be read. A correct manifest run records six checksum entries, one error entry naming the unreadable path, and exits non-zero. An incorrect run records six entries and exits zero. Compare the incorrect manifest against an equally incomplete destination manifest and you get a perfectly clean report of zero discrepancies — clean because almost nothing was examined. A quiet run is not a passing run. Check that the number of entries in the manifest matches the number of paths you expected to select, and check the exit status.

Two smaller habits belong here. First, expect the unexpected-file group to contain operating-system metadata or the copy tool's own log, and resist the urge to build those into an automatic ignore list. The same rule that hides a .DS_Store will hide a genuinely new transcript someone dropped into the folder before you arrived. Second, keep every report. The source manifest, the destination manifest, and the discrepancy report are three different documents, and you will want all three when someone asks what happened — including if the answer turns out to be that the original baseline was wrong.

Resolve discrepancies without rewriting the evidence

Missing and changed are usually repairable. Re-copy the affected paths from the known source, then run the destination inventory and the comparison again, writing to a new report. Do not edit the source manifest to make the second comparison come out cleaner. The baseline is a record of what you selected before the copy; overwriting it destroys the only evidence that a discrepancy ever existed, which is exactly the evidence you would want if the same drive misbehaves on the next job.

Changed bytes need a conversation before they need a re-copy. Ask the person responsible for the material whether the change was deliberate. If a collaborator opened the transcript and corrected a date, that is an intentional edit, and the destination file is now a different document with a relationship to the source — not the same file that arrived intact. Re-copying over it would erase a deliberate correction; leaving it in place and calling it "identical" would be false. The honest move is to give the corrected file its own declared baseline and record what it derives from, which is the same discipline preservation guidance applies to deliberate migrations: when content is intentionally transformed, the new version needs a new documented baseline rather than a claim that nothing changed.

The fixture resolves that way. The intake note's edit was deliberate, so it stays as a derivative with its own baseline; the missing permission note is re-copied from the source; the summary file, which exists only at the destination, is moved into a labeled holding folder outside the inventoried tree rather than deleted, since nothing in the comparison establishes whether it is the only copy of anything. The recheck then reports five paths byte-identical to the source baseline, one path matching its declared derivative baseline, no missing paths inside the root, and no unexpected files inside the root. The extra file is not gone. It is accounted for.

Notice, too, that the baseline describes the source at a moment, not the source forever. If the source folder changed between the inventory and the copy — new files added, a file replaced — then the destination may faithfully match the copy-time source while failing the baseline. That is not a failed transfer; it is a stale baseline. Re-inventory the source, record why, and start again.

Two things not to do. Do not delete the originals because the check passed. A verified copy is not a reason to give up a reference copy, and the check only speaks about the paths you selected, not about anything you left out of scope. And do not let a good result travel further than it can. What you have at the end is this: a particular set of recorded files matched the bytes recorded at the baseline, after this transfer, on these two drives, under these rules. The recordings may still be mislabeled. The permission note may still not cover the use you have in mind. The scan may still be of something other than what the caption says. Those are separate questions with separate evidence, and mixing them into a checksum report is how a filmmaker ends up telling a broadcaster that the research is "verified" when the only thing verified is that a copy survived a Thursday afternoon.

End with a retained baseline and a comparison that accounts for every selected path. If some discrepancies are still open, leave them named as open rather than rounding them off, and keep the same-size alteration somewhere visible in the record, because it is the case that best explains why any of this was worth doing. Then say exactly what happened: the file set matched after transfer.

Frequently asked questions

What belongs in each file manifest entry?

A relative path from the authorized root, the size in bytes, and a checksum from a named algorithm. Record the algorithm, the absolute root separately, tool version, UTC start time, entries and error list. Relative paths matter because source and destination live at different absolute paths. SHA-256 is a reasonable default; MD5 and SHA-1 have practical collision attacks and are not worth defending in a new workflow.

Why compare path membership before checksums?

It separates missing paths, unexpected files, and candidate paths before byte comparison. Those groups call for different responses, and a single count or aggregate can hide all three. Counts can match while a path is missing, and aggregate totals can look roughly the same or even match exactly while something changed. Only paths present on both sides should go into the checksum comparison.

Why is a size check not enough to verify a copy?

Equal size is not equal content. In the constructed example, a transcript changed one digit in a date and kept the same byte count, so size stayed identical while SHA-256 changed completely. That is why checksums catch same-size alterations that size checks miss. A matching checksum still speaks only about the selected file bytes, not about labeling, permission, or whether the material is true.

What should happen if a destination file has a different checksum?

Ask the person responsible whether the change was deliberate before re-copying. A deliberate edit, such as a collaborator correcting a date, is a different document with a relationship to the source; give it its own declared baseline and record what it derives from. Re-copying over it would erase a correction, and calling it identical would be false. If the change is not wanted, re-copy from the known source and rerun the destination inventory and comparison into a new report.

What does a clean comparison actually establish?

It establishes that a particular set of recorded files matched the bytes recorded at the baseline, after this transfer, on these two drives, under these rules. It does not establish that the recordings are what someone claims, that permission covers the intended use, or that the story documented is true. It is also not a reason to delete originals. Keep the source manifest, destination manifest, and discrepancy report.

Further reading

• Build a Film Research Dossier Another Writer Can Work From — Once the copy is verified, this explains how to organize the folder by the questions the film must answer so another writer can use your reasoning.
• Preserve an Email Exchange as Source Material for a Documentary Pitch — If the research folder contains .eml exports, this covers freezing and hashing the originals, identifying parts, and treating quoted text carefully.