Show HN: Minarrow – Fast Rust columnar engine with Apache Arrow compatibility

1 point

10 months ago

Minarrow is a fast, from-scratch columnar data engine written in Rust, designed for systems and workloads where Apache Arrow is a fit—but Arrow-rs is too heavy.

I built it for use cases like:

- Structured tick/quote/event ingestion in HFT - Native async streaming (Tokio, etc.) - Edge and embedded systems that need zero-copy structured memory - SIMD-native kernels (all buffers are 64-byte aligned) - polyglot apps

Minarrow gives you:

- Arrow-compatible layout and schema support - <1.5s build times - Always-typed data access (no downcasting, no Any) - Async IPC and memory-mapped file support (via a sibling crate) - .to_arrow() and .to_polars() for interop

It doesn’t try to replace the Arrow ecosystem—but for Rust-first, performance-sensitive systems, it offers Arrow compatibility without the abstraction tax.

Repo: https://github.com/pbower/minarrow Docs: https://docs.rs/minarrow/latest/minarrow/

Benchmarks (full results in repo): Minarrow sums 1k integers in 88 ns (direct), within a few ns of raw Vec (64 ns) and ~2× faster than Arrow-rs struct (162 ns). On 1B integers with SIMD + Rayon: ~114 ms.