Hash Large Files in Python with Streaming SHA-256

A cryptographic file hash is a compact fingerprint of a sequence of bytes. It is useful when checking whether a downloaded file matches a trusted published digest or whether a stored training image has changed. You do not need to read a multi-gigabyte file into memory to compute one.

Read bytes in bounded chunks

Hashing operates on bytes, so open the input in binary mode. Text mode can decode characters or translate line endings, changing what you measure. Each call to update adds another chunk to the same digest state. The chunk size affects memory use and throughput, not the resulting digest.

import hashlib
from pathlib import Path

def sha256_file(path):
    digest = hashlib.sha256()
    with Path(path).open("rb") as stream:
        for chunk in iter(lambda: stream.read(1024 * 1024), b""):
            digest.update(chunk)
    return digest.hexdigest()

# Known test vector; no file creation needed
assert hashlib.sha256(b"abc").hexdigest() == (
    "ba7816bf8f01cfea414140de5dae2223"
    "b00361a396177a9cb410ff61f20015ad"
)

Compare against a trusted reference

Call sha256_file with an existing file path, then compare its hexadecimal string with the expected SHA-256 value. Record the algorithm alongside the digest; a bare hash does not explain how it was produced. Normalize surrounding whitespace in a copied reference, but do not silently accept truncated or malformed values.

What a matching hash does not prove

A match supports the claim that the measured bytes agree with the reference. It does not establish who created the file or whether the reference itself is trustworthy. Downloading a file and a checksum from the same compromised source can give you two matching but untrustworthy artifacts. File hashes also do not describe ownership, permissions or other filesystem metadata.

Avoid measuring a moving target

If another process writes to the file while it is being read, the digest might not represent a stable version. Use a stable copy or a workflow that prevents concurrent modification. Let read failures remain visible; returning an empty string on an exception can make a failed check look like ordinary data.

Practice and check your understanding

Hash a disposable text file, copy it and compare both digests. Change one character in the copy and repeat. Finally change only the filename: explain why that does not change a digest calculated from file contents.

Continue learning

Explore the related Code Hacker course syllabus for structured learning. Then read Build a Digital Evidence Manifest for a Practice Case to connect this exercise to the next skill.

Reference

Python hashlib documentation provides the underlying API or technical guidance. Check the documentation for the version you use; examples here are small learning exercises rather than complete production implementations.