Computing an MD5 or SHA hash without getting it wrong

You have two hexadecimal strings in front of you: the one published by a software vendor and the one you have just computed. They should be identical, character for character. When they are not, the cause is almost always more mundane than an attack — one line break too many, a different encoding, the wrong algorithm. Here is how to produce a digest you can trust, and how to diagnose a mismatch.

Five algorithms, five lengths

A hash function turns an input of any size into a fixed-size digest. The length of the result therefore identifies the algorithm used, and that alone is a useful diagnostic reflex: if you are handed 40 characters and you produce 64, you are not using the same function.

A digest can be written in upper or lower case with no change in meaning: D41D8CD9… and d41d8cd9… are the same value. PowerShell prints hexadecimal in upper case, sha256sum in lower case — that difference in case alone causes an unreasonable number of false alarms.

Hashing a piece of text

For a string, a token you need to compare or a snippet of configuration, the MD5 calculator returns the expected 32 characters and shows two counters: the number of characters and the number of UTF-8 bytes. The second figure is the more interesting of the two, and we come back to it below. The SHA calculator produces SHA-1, SHA-256, SHA-384 and SHA-512 at the same time: rather than guessing which algorithm the other side used, you compare all four at once.

Both run entirely in the browser; the text you type never reaches a server. That has one concrete technical consequence: the SHA tool relies on the WebCrypto API, which is only available in a secure context. On a page opened over plain HTTP, or in an old browser, it shows an explicit error rather than a wrong result.

Checking that the tool is telling the truth

A handful of reference values, taken from the algorithms' official test vectors, let you check any implementation in ten seconds:

• Empty string, MD5: d41d8cd98f00b204e9800998ecf8427e
• Empty string, SHA-1: da39a3ee5e6b4b0d3255bfef95601890afd80709
• Empty string, SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
• The three letters abc, MD5: 900150983cd24fb0d6963f7d28e17f72
• The three letters abc, SHA-256: ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad

If abc does not give you 900150983cd24fb0d6963f7d28e17f72, the tool is not the one at fault: you have typed something other than three letters. A space, a capital, a line break.

Why two digests of the same content differ

A hash is computed over bytes, never over what you see on screen. A single changed bit alters about half the bits of the result: there is no such thing as a small discrepancy; two digests either match or have nothing in common. The usual culprits, in decreasing order of frequency:

• The trailing line break. Most editors end a text file with a newline. The word secret on its own and the same word followed by a line break produce two unrelated digests. This is the leading cause of mismatch on short strings copied out of a terminal.
• CRLF versus LF. A file that has been through Windows contains the bytes 0D 0A where the original had only 0A. Across a 500-line file that is a 500-byte difference. Git performs the conversion on the fly depending on the core.autocrlf setting, which explains digests that diverge between two clones of the same repository.
• The UTF-8 BOM. Three invisible bytes, EF BB BF, added at the start of a file by Notepad or by a spreadsheet export. The content looks exactly the same; the digest bears no resemblance.
• The encoding. The character é is one byte in ISO-8859-1 (E9) and two in UTF-8 (C3 A9). Across a 2,000-word French text the gap runs into hundreds of bytes, for content that is visually identical.
• Unicode normalisation. That same é can be a precomposed character (U+00E9) or the letter e followed by a combining accent (U+0301). Same rendering, two byte sequences. macOS has traditionally stored filenames in decomposed form, which catches out comparisons made across systems.
• Stray characters. A non-breaking space pasted out of a word processor, a tab turned into spaces, a curly apostrophe substituted for a straight one, a space left at the end of a line.

Diagnosis in three steps

1. Paste each of the two versions of the text into the MD5 calculator in turn and compare the UTF-8 byte counter. Two different values for the same number of visible characters: the problem is an encoding or an invisible character.
2. Same byte count but different digests: a character really has been replaced. Look at smart quotes, em dashes and curly apostrophes inserted automatically.
3. Delete the last character if it is a line break, then recompute. In the majority of cases the discrepancy disappears there.

For a file, use your own system

The online calculators work on text you type or paste. For the digest of a multi-gigabyte disk image or a large installer, the local command is faster, avoids any transfer and reads the file exactly as it sits on disk:

• Windows, PowerShell: Get-FileHash file.iso -Algorithm SHA256
• Windows, Command Prompt: certutil -hashfile file.iso SHA256
• macOS: shasum -a 256 file.iso or md5 file.iso
• Linux: sha256sum file.iso or md5sum file.iso
• Checking a whole batch against a supplied checksum file: sha256sum -c checksums.sha256

Do not compare 64 characters by eye. Checking the first six and the last six is a widespread habit and an inadequate one: a deliberately manufactured collision has no reason to be caught that way. Line the two values up in an editor, or let the verification command settle it.

What a digest does not prove

It establishes that a piece of content matches a reference. It says nothing about where that reference came from. If the file and its digest are published in the same place, anyone who replaces one replaces the other: verification becomes a ritual with no effect. The reference digest has to arrive through a separate channel, ideally accompanied by a signature.

As for how robust these algorithms are, the record is settled. MD5 collisions can be manufactured in seconds on an ordinary machine, and have been since the work published in 2004. SHA-1 fell in 2017, with the release of two different PDF files sharing the same digest, then again in 2020 with a chosen-prefix attack that is far more usable in practice. The conclusion: MD5 and SHA-1 remain useful for spotting accidental corruption, an interrupted transfer or a duplicate. They are worth nothing against someone who is trying to deceive you. In that second case, SHA-256.

A digest is not a secret either. The MD5 of a common word turns up in a single search, and whole tables of them have been published for years. That is why hashing a password with MD5 or SHA-256 does not protect it: these functions are designed to be fast, and therefore to be tried billions of times a second. Storing passwords calls for bcrypt, scrypt or Argon2, with a unique salt per account. And before any of that, the length and randomness of the password — what the password generator produces — matter more than the choice of function.

Finally, hashing is not encrypting. The operation is one-way; there is no such thing as un-hashing. Sites that claim to decrypt an MD5 are in fact querying a table of values already computed. The most common confusion involves Base64, which looks like gibberish but reads in both directions, as the Base64 encoder and decoder demonstrates in one click. A password Base64-encoded in a configuration file is a password in plain text.

Key points

• Unexpected length: it is the wrong algorithm.
• Different digests for text that looks identical: trailing line break, BOM, encoding.
• Security at stake: SHA-256, with the reference digest obtained through a channel other than the one hosting the file.
• Passwords: neither MD5 nor SHA, ever.

Both of Convertu's calculators run inside the tab: neither the text nor its digest leaves your machine, which is the least you can ask when you are hashing an API key just to compare it.