· 6 min read

Training Zstandard Dictionaries for Sub-2KB Telemetry Events

Storage Performance

Compression algorithms like LZ4 or plain Zstandard build a sliding match table as they scan a buffer. When individual telemetry events or Kafka messages are only 350 to 900 bytes long, the encoder barely warms up before reaching end-of-buffer—spending most of the output bytes storing repeated JSON field names.

Why Small Messages Compress Poorly

Consider a structured audit event containing "timestamp", "service_region", "trace_id", and enum strings. Across millions of rows, 70% of the payload is identical boilerplate, yet compressing each message independently yields only a 15–25% size reduction.

Building a Representative Corpus

With Zstandard's zstd --train (COVER algorithm), we feed 5,000–10,000 representative sample payloads to generate a shared 32 KiB dictionary loaded once per worker process.

bash — train_dict.sh
# Collect 8,000 individual sample payloads into ./samples/
zstd --train ./samples/*.json --maxdict=32768 -o telemetry.zdict

# Inspect dictionary header & verify compression on held-out test sample
zstd -D telemetry.zdict -3 --rm -k ./heldout/event_0042.json

Ratio & CPU Cost Comparison

Codec (avg 620B JSON)Avg Compressed SizeRatioEncode Speed
Uncompressed620 B1.00x—
LZ4 (no dict)498 B1.24x1,420 MB/s
Zstd level 3 (no dict)438 B1.42x490 MB/s
Zstd level 3 + 32KB Dict151 B4.11x395 MB/s

Version-tagging the dictionary ID (embedded automatically in the Zstd frame header) allows rolling dictionary updates every quarter while keeping old decoders backward-compatible.