Binary Files: Writing and Reading Raw Bytes

Start with the problem

Text files are easy to read and edit, but they are expensive. Every number is converted to digits when writing and parsed back when reading, the length of a record varies with its values, and finding record number 1,000,000 means reading every line before it.

A binary file stores the bytes of the values themselves. A million 32-bit integers take exactly 4 MB, reading them back needs no parsing, and every record can have the same size. The price is that the stream no longer makes decisions for you. Text formatting hid questions such as how big a number is, which byte comes first, and where the padding goes. In a binary file you have to answer them yourself.

How the idea works

Opening. Open both the writing and the reading stream with the flag std::ios::binary. Without it, a text-mode stream on Windows translates newline bytes, which corrupts data, and may treat a particular byte value as the end of the file.

Writing and reading. The functions write and read are unformatted: they move a given number of bytes and interpret nothing.

std::uint32_t id = 258;
out.write(reinterpret_cast<const char*>(&id), sizeof id);
 
std::uint32_t back = 0;
in.read(reinterpret_cast<char*>(&back), sizeof back);

The expression reinterpret_cast<const char*>(&id) does not convert the number. It only lets the function look at the bytes of the object as characters, which the language allows for any object. The second argument is the number of bytes. The trace follows the round trip: the bytes in memory are copied to the file unchanged (step 2) and back again (step 4).

Checking. A read can come up short, for example from a truncated file. After a read, test the stream with if (!in), and use in.gcount() to see how many bytes arrived. Never use the data of a failed read.

Fixed-width types. The size of int and long varies between platforms, so a file format uses the types of the header <cstdint>, such as std::uint32_t and std::int64_t, which have the same size everywhere, as in lesson 4.

Byte order. Step 5 is the central trap. The same four bytes mean 258 on a little-endian machine and 33,619,968 on a big-endian one. A portable format defines the byte order, and the writer produces it explicitly with shifts and masks, instead of relying on the host:

void writeU32(std::ofstream& out, std::uint32_t v) {
    char bytes[4] = {
        static_cast<char>(v & 0xFF),
        static_cast<char>((v >> 8) & 0xFF),
        static_cast<char>((v >> 16) & 0xFF),
        static_cast<char>((v >> 24) & 0xFF)
    };
    out.write(bytes, 4);
}

This writes the least significant byte first, whatever machine runs it, and a matching reader reassembles the value in the same order. The shifts and masks are the bitwise operators of lesson 2 in module 2.

Padding and structs. Do not write a whole struct with one write. Its padding bytes, from lesson 2, have unspecified contents, its layout differs between compilers, and a member such as std::string contains pointers that mean nothing in another run. Write the fields one at a time. For text, write the length first and then the characters.

A small format. A real format starts with a magic number and a version, so that a reader can recognize the file and handle changes:

const char magic[4] = {'C', 'B', 'K', '1'};   // the format name and its version
out.write(magic, 4);

Random access. If every record has the same size, record i starts at the header size plus i times the record size. A call such as in.seekg(offset) jumps straight there. This is the array formula of lesson 1 stretched over a file.

Why it is correct

The functions write and read do no formatting, so what you read is exactly what you wrote, provided that both sides agree on the layout. The language allows access to the bytes of any object through a char pointer, so the casts above are defined behavior. A format that fixes the sizes of its fields, their order and their byte order means the same thing on every platform, and the offsets of fixed-size records can be computed, which is why random access is possible.

What it costs

Size and speed. A million integers take 4 MB in binary and up to about 11 MB as text. Reading them as text means parsing ten digits each, around 50 to 100 nanoseconds per number, so the whole file takes tens of milliseconds. Reading them as binary is mostly copying memory, at the speed of the disk or the cache.

Debugging. You cannot read a binary file in an editor. A hex dump tool, such as xxd, shows the bytes:

00000000: 0201 0000                                ....

Evolution. Changing a binary format means changing a version number and writing code that still reads the old one. Text formats such as CSV tolerate small changes more easily.

Byte order. Converting costs almost nothing: a byte swap is a single instruction.

Pitfalls and variations

Forgetting the binary flag. On Windows this silently corrupts files. Set it every time.

Dumping structs. The padding, the layout and any pointer members make the bytes meaningless elsewhere. This is the lesson 2 warning in practice.

Assuming sizes. An int, a long, a bool and an enumeration can differ in size between platforms. Use fixed-width types.

Trusting length fields. A length read from a file you did not write may be 4 billion. Check it against a sensible limit, and against the size of the file, before allocating memory.

Reading through the wrong cast. Casting a char buffer to an int* and reading through it is undefined behavior. Read into the int object through a char*, as above, or use std::memcpy.

Short reads. Always check the stream and gcount() after read.

Host byte order. Never write multi-byte values by dumping them if the file will leave the machine.

Variations. C++20's std::endian, from <bit>, tells you the host's byte order, and C++23's std::byteswap reverses it. std::bit_cast converts between types of equal size without undefined behavior. Serialization libraries such as Protocol Buffers and FlatBuffers define formats and generate the code for them, and a checksum such as a CRC at the end of a file detects corruption.

Where it is used

Image, audio and video formats, database files, save games, network protocols and executable files are all binary formats with headers, fixed-width fields and defined byte order. The object files of lesson 1 and the executable that the linker produced are binary formats too, which is why tools like nm exist to read them.

The next lesson, "Parsing Lines: Splitting Text and Converting Fields", returns to text, and turns the lines you read in lesson 6 into the fields and numbers of a record.