🌀 Using Parquet? This new columnar format is up to 100x faster for random access.
Vortex is an open source columnar file format designed for modern hardware. Now a Linux Foundation sub-project. Its benchmarks are impressive:
| Metric | Improvement vs Parquet |
|---|---|
| Random access | 100x faster |
| Scans | 10-20x faster |
| Writes | 5x faster |
| Size | Comparable |
What makes it so fast?
- ⚡ Cascaded encodings: “white box” compression that allows compute directly on compressed data
- 🧮 SIMD & GPU: designed for modern vectorized instructions
- 🔗 Zero-copy with Apache Arrow: interoperable without conversion
- 📦 Optimized metadata: efficient reads from object storage
Compatible with Polars, Pandas, DuckDB, DataFusion, and Spark.
💡 Explanation in a nutshell#
Parquet is the most popular file format for tabular data in data lakes. Vortex aims to replace it by being much faster, especially when you need to look up specific rows (random access). It achieves this with smart compression techniques that allow operations without decompressing data first, leveraging modern CPU and GPU capabilities.
More information at the link 👇
More in the following external reference.
Also published on LinkedIn.
