Decrypting a download while it is sent

Every file on tmp0 is encrypted before it is written to disk. Until recently, each one was sealed as a single AES-256-GCM message: one key, one nonce, one authentication tag at the very end. That is the textbook way to do it, and it had a cost we didn't like.

The problem with one big message

GCM gives you two things, secrecy and a check that nothing was changed. The check is the tag, and the tag covers the whole message. Java takes that seriously: it will not hand out a single decrypted byte until it has seen the tag and verified it. That is correct behaviour. Decrypted data that hasn't been checked is data an attacker may have shaped.

For us it meant that sending a file started with decrypting all of it. Small files went into memory. Everything from 4 MB up went into a temporary file, in plaintext, and the response was served from there.

So every download of a large file wrote that file to disk a second time, unencrypted, before the first byte left the server. A 2 GB download made you wait while 2 GB were decrypted, and a handful of them at once could fill the disk. For a service whose whole point is that your file exists once and briefly, a plaintext copy per download was the wrong shape, even if the copy only lived for the length of the response.

Segments

A stored file now looks like this:

"tmp0enc" 0x02 | salt (32 bytes) | segment 0 | segment 1 | ... | last segment

Each segment is up to 64 KiB of the file, sealed with AES-256-GCM by itself and followed by its own 16-byte tag. A download reads one segment, checks its tag, sends it, and moves on to the next. The server holds two small buffers per download, the first byte goes out at once, and nothing decrypted touches the disk.

Three details make this safe rather than just fast.

A key per file. The key is not the server's master key. It is derived from it and from the random salt in the header, with HKDF-SHA256. Every file gets its own key, so the nonces can be simple counters starting at zero without one ever being used twice under the same key. It also means a segment copied in from another file fails its check: it was sealed under a different key.

The nonce is the position. The nonce of a segment is its number. Swap two segments, or drop one from the middle, and every segment after that point is checked against the wrong number and fails.

The last segment says so. One byte of the nonce is 1 for the final segment and 0 for all others. Without it, a file cut off exactly between two segments would look complete, because every segment that remains is genuine. With it, the segment that is now at the end was sealed as "not the last", and the check fails. Bytes added after the real end fail the same way.

This is not our invention. It is the STREAM construction from a 2015 paper by Hoang, Reyhanitabar, Rogaway and Vizár, and it is what age and Google's Tink use for the same job. We took 64 KiB from them as well.

One consequence shows up in the code: a segment can only be sealed once it is known whether another one follows. So the writer always reads one segment ahead, and the reader looks one byte past every full segment.

What it costs

The header is 40 bytes and every segment adds 16. For a 2 GB file that is 32,768 segments and 512 KiB of tags, about 0.02 percent.

When a file is damaged

The first segment is verified when the file is opened, before the download is counted. A file that was sealed under another key, or isn't ours at all, fails right there. That matters for burn-after-reading: the one permitted download isn't spent on a file that can't be read.

Damage further in is found when the download reaches it. The response stops at that point, short of the length it announced, so the browser reports a failed download instead of saving a file that is quietly wrong. Everything sent before the break had passed its check.

The tests do this on purpose: flip one bit, cut the file at seven different places, append bytes, swap segments, remove one from the middle, splice in a segment from another file. None of them reads as intact.

Deleting a file that is being read

Streaming from the stored file created a problem the temporary copy had hidden. A large download to a slow connection takes minutes, and in that time the file may be due for deletion: it expired, or this very download was its last permitted one. Deleting it then would cut the download off. Overwriting it first would send garbage.

So every reader holds a lease on the file. A deletion requested in the meantime is carried out when the last reader is done. For burn-after-reading the database record is gone the moment the download is claimed, and the bytes on disk follow when the response ends.

What this doesn't do

• It doesn't protect your file from us. The master key is in the server's memory, and the server decrypts the file to send it. This layer is about what a stolen or discarded disk gives away. If we shouldn't be able to read a file, use the end-to-end mode, where the key never leaves your browser.
• The end-to-end mode has not caught up. In the browser, a file is still sealed as one message and has to fit in memory to be encrypted or decrypted. The server wraps the result in segments like everything else, but your browser still does it in one piece. That is the same limitation, on your side of the connection.
• No resuming. A download is a stream that is read once and counted once. Range requests aren't offered; a client that asks for a part gets the whole file, and an interrupted download starts again.
• A file's record isn't bound to its bytes. The segments prove that a stored file is complete and unchanged. They don't prove it is the file the database says it is. Someone who can write to the disk could exchange two whole stored files and both would pass. Someone with that access has larger options, but it is a gap, and it belongs on this list.
• There was no migration. The old format has no reader any more. tmp0 wipes its storage on every restart, so there were no old files to convert. Most services can't do that.