Compression & encoding
The last lesson introduced encoding and compression. These are two distinct operations. Encoding represents a column's values as bytes and can exploit repetition or other patterns. Compression is an optional, general-purpose step applied to the encoded page data. Parquet encodes values first, then a writer can compress that data or leave it uncompressed. The size savings depend on the data and the choices made; neither operation guarantees a smaller result for every input.
Two layers of shrinking
Picture one page of ShopFlow's order status column (our running example — see Meet ShopFlow): paid, paid, paid, paid, shipped, shipped, paid, paid.
For these repeated statuses, a dictionary can replace the strings with integer IDs. Parquet encodes those IDs using RLE/bit-packing: a combination of run-length encoding (representing repeated values by counts) and packing small integers into fewer bits. An optional codec, such as Snappy, ZSTD or gzip, then works on the encoded bytes. The diagram shows the logical order; page headers, the separate dictionary page and other metadata are omitted. See Apache Parquet's encoding definitions and compression definitions for the format details.
Encodings: exploit the column's structure
Dictionary encoding
When a column has few distinct values repeated many times (low cardinality — cardinality just means the count of distinct values), storing the full value every time is wasteful. Dictionary encoding builds a small lookup table of the distinct values and replaces each value with a tiny integer index.
Our status column → dictionary {0: paid, 1: shipped}, data becomes 0,0,0,0,1,1,0,0. The strings "paid"/"shipped" are stored once; the column body is now small integers. This is Parquet's default for most columns and is hugely effective on the low-cardinality string columns analytics is full of — ShopFlow's status (placed/paid/shipped/cancelled), product category, customer region, currency.
The dictionary also supercharges skipping: the engine can check a filter value against the dictionary and discard the whole page if the value isn't present.
Run-length encoding (RLE)
When the same value repeats consecutively, run-length encoding stores the value once plus a count, instead of repeating it. The dictionary-encoded run 0,0,0,0,1,1,0,0 becomes (0×4)(1×2)(0×2) — three little pairs instead of eight values. RLE shines on sorted or naturally clustered columns, which is another reason sorting your data pays off (last lesson). Parquet combines dictionary + RLE so tightly the pair is often called "RLE/dictionary."
Delta encoding
For columns whose values change by small amounts — timestamps, monotonically increasing IDs, sequential counters — delta encoding stores the first value, then only the differences between consecutive values. ShopFlow's order_ts column, with orders arriving a few seconds apart, stores one full timestamp plus a stream of small deltas (+4000, +1200, +9000… ms), which take far fewer bytes than full 8-byte timestamps and also compress well afterward. The same trick collapses the steadily-increasing order_id.
Parquet writers pick encodings per column automatically based on the data. You rarely set them by hand. The value of knowing them is diagnostic: when you understand why a low-cardinality, sorted column compresses 50× and a high-cardinality random UUID column barely compresses at all, you can design schemas and sort orders that play to encoding's strengths.
Compression codecs: the general-purpose squeeze
After encoding, a Parquet writer can apply a general-purpose compression codec to page data. Parquet also supports UNCOMPRESSED, which leaves that data uncompressed. Compression codecs work on bytes rather than column meanings; their effectiveness depends on the input. When choosing compression, consider these tradeoffs:
- Compression ratio — how much smaller the data gets (less storage, less I/O).
- CPU cost — how much processor time compressing/decompressing burns.
- Splittability — whether a compressed file can be divided and processed in parallel without decompressing the whole thing first.
The common codecs, roughly fastest-to-smallest:
| Codec | Ratio | Speed (de/compress) | Notes |
|---|---|---|---|
| Snappy | modest | very fast | Parquet's default. Optimized for speed over ratio; the safe everyday choice. |
| LZO | modest | very fast | Snappy-like; historically used in Hadoop. Less common now. |
| ZSTD | high | fast (tunable) | Modern sweet spot — near-gzip ratios at near-Snappy speed, with a level dial. Increasingly the recommended default. |
| gzip | high | slow | Great ratio, heavy CPU. Fine for cold/archival data read rarely. |
For most lake data, Snappy (the default) is fine, and ZSTD is the better modern choice when you want noticeably smaller files without much CPU penalty — pick ZSTD when storage or scan cost matters and CPU headroom exists. Reserve gzip for cold data you write once and rarely read, where maximum shrink beats decompression speed. (Which codec is "best" shifts over time as libraries improve — the tradeoff axes ratio/CPU/splittability are the durable part; the rankings are dated.)
The splittability gotcha
Splittability is why this matters beyond storage cost. A 1 GB file that a parallel engine can split into eight chunks gets processed by eight workers at once. But some compression — notably a gzipped CSV or JSON file (one giant gzip stream over the whole file) — is not splittable: to read byte 500 MB you must decompress from the start, so the whole file is stuck on one worker. That single non-splittable file becomes a serial bottleneck in an otherwise parallel job.
Parquet sidesteps this entirely: because it compresses per page within per row group, the file stays splittable (workers grab different row groups) even though each page is compressed. This is a quiet but major advantage of Parquet over compressed text formats, and it's invisible unless you understand the page/row-group structure from the last lesson.
A traced example
ShopFlow's orders history, 100 GB raw as CSV. We rewrite it as Parquet:
- Columnar layout groups like values → already more compressible than row-major CSV.
- Encoding:
status(4 distinct values) → dictionary + RLE, shrinks dramatically.customer_id(high cardinality, millions of customers) → barely helped by dictionary, stays larger.order_ts(ascending) → delta, shrinks a lot. - Compression: ZSTD over the encoded pages squeezes the remaining byte redundancy.
Result: ~100 GB CSV → roughly 8–15 GB Parquet — and it's still splittable for parallel reads, where the gzipped CSV alternative would be one-tenth the speed on read because a single worker must decompress it serially. Smaller and faster.
Why it matters
Columnar data shrinks in two stacked steps. Encodings exploit a column's structure: dictionary (few distinct values → integer indexes), RLE (consecutive repeats → value + count, loves sorted data), and delta (small changes → store differences, ideal for timestamps/IDs). General-purpose codecs then squeeze byte redundancy, trading ratio vs CPU vs splittability — Snappy (fast default), ZSTD (modern sweet spot), gzip (best ratio, slow, for cold data). The decisive practical points: choose ZSTD when scan/storage cost matters, reserve gzip for cold data, and remember that gzipped CSV/JSON is not splittable (a serial bottleneck) while Parquet stays splittable because it compresses per page. Encodings reward low cardinality and sorted columns — design for that. Next: with the bytes understood, we step up to choosing which format to use — Parquet, Avro, or ORC — and how each survives schema change.