Training Zstandard Dictionaries for Sub-2KB Telemetry Events
Compression algorithms like LZ4 or plain Zstandard build a sliding match table as they scan a buffer. When individual telemetry events or Kafka messages are only 350 to 900 bytes long, the encoder barely warms up before reaching end-of-buffer—spending most of the output bytes storing repeated JSON field names.
Why Small Messages Compress Poorly
Consider a structured audit event containing "timestamp", "service_region", "trace_id", and enum strings. Across millions of rows, 70% of the payload is identical boilerplate, yet compressing each message independently yields only a 15–25% size reduction.
Building a Representative Corpus
With Zstandard's zstd --train (COVER algorithm), we feed 5,000–10,000 representative sample payloads to generate a shared 32 KiB dictionary loaded once per worker process.
# Collect 8,000 individual sample payloads into ./samples/
zstd --train ./samples/*.json --maxdict=32768 -o telemetry.zdict
# Inspect dictionary header & verify compression on held-out test sample
zstd -D telemetry.zdict -3 --rm -k ./heldout/event_0042.json
Ratio & CPU Cost Comparison
| Codec (avg 620B JSON) | Avg Compressed Size | Ratio | Encode Speed |
|---|---|---|---|
| Uncompressed | 620 B | 1.00x | — |
| LZ4 (no dict) | 498 B | 1.24x | 1,420 MB/s |
| Zstd level 3 (no dict) | 438 B | 1.42x | 490 MB/s |
| Zstd level 3 + 32KB Dict | 151 B | 4.11x | 395 MB/s |
Version-tagging the dictionary ID (embedded automatically in the Zstd frame header) allows rolling dictionary updates every quarter while keeping old decoders backward-compatible.