Parquet is the columnar format that ate the data lake, and for good reason: it compresses well, it reads fast, every analytics engine from DuckDB to Spark to DataFusion speaks it natively, and the files are just sitting there on disk waiting to be queried by anything that can open them, which is the same property that made SQLite appealing for a different class of problem. The catch, and it is a significant catch, is that Parquet is immutable by design, the format spec has no concept of updating a row in place, deleting a tuple, or merging two versions of a record, and the moment you need any of those operations you are in the business of rewriting files or building a layer on top that pretends mutability exists; IcefallDB is that layer, and the interesting work was making the pretension convincing enough that you stop noticing it.
Mutations on an immutable format
The approach is copy-on-write at the row-group level, not the file level, which matters because Parquet files in practice contain multiple row groups and rewriting an entire file, often several hundred megabytes across multiple row groups, to delete three rows is a wasteful operation that nobody wants to pay for. When a DELETE or UPDATE lands, IcefallDB reads the affected row group, filters or modifies the rows in memory, and writes a replacement row group into a new file alongside the original, then updates a plain JSON manifest that records which row groups are live and which are superseded; a subsequent read through DataFusion sees only the manifest’s current view, and the old row groups are retained as history rather than discarded, which is where the “time travel” part of the design becomes real rather than marketing.
The manifest is a flat JSON file per table, intentionally human-readable and intentionally small, containing an array of row-group descriptors with file paths, column statistics, row counts, and a monotonic version integer; the format is boring on purpose, because the manifest is the one thing in the system that must survive corruption, and a format that a human can read and fix in a text editor is a format that survives operational mistakes that would brick a binary catalog; writes go through write-temp-then-atomic-rename with an fsync before the rename, because the property that matters under a crash is not readability after the fact but atomicity during the replacement, and a human reading a JSON file that a torn write never reached is still reading the previous version, intact and correct.
{
"table": "events",
"version": 47,
"row_groups": [
{
"file": "events/v047-rg000.parquet",
"rows": 8192,
"min_ts": "2026-06-01T00:00:00Z",
"max_ts": "2026-06-15T12:30:00Z"
},
{
"file": "events/v047-rg001.parquet",
"rows": 6550,
"min_ts": "2026-06-15T12:30:01Z",
"max_ts": "2026-06-30T23:59:59Z"
}
],
"superseded": ["events/v046-rg000.parquet", "events/v046-rg001.parquet"]
}
A MERGE operation follows the same pattern: the target row groups are read, the merge keys are matched against the source, the matched rows are rewritten with the merged values, and the result lands in a new row group file while the old one moves to superseded; the important property is that no Parquet file is ever modified after it is written, which means every prior version of the table is recoverable by pointing the manifest at an earlier version integer, and that is the full mechanism for time travel; prior manifests are retained alongside superseded row groups, so reconstructing version 46’s view is a matter of loading the version-46 manifest snapshot and reading the row groups it references, with no separate log store and no WAL replay required.
The aggregate sidecar
Aggregate queries are the other place where a plain-Parquet lake burns cycles unnecessarily: SELECT COUNT(*), SUM(amount) FROM orders WHERE status = 'shipped' forces a full column scan across every row group in the table even though Parquet’s own column statistics already give you min and max per row group, and the reason that is not enough is that statistics do not include arbitrary filtered aggregates like “sum of amount where status equals shipped,” so the scan happens anyway, repeatedly, for queries that are asking the same question of the same data.
IcefallDB’s sidecar .agg cache sits next to the Parquet files and stores precomputed aggregate values keyed by (column, predicate_hash, row_group_id), and the cache is populated lazily on the first scan that touches a given row group with a given predicate; the next query that asks the same question with the same predicate reads the precomputed value instead of scanning the row group, and the speedup is the difference between a 4-second scan and a 3-millisecond lookup, which is the kind of improvement that changes how you write downstream code because the cost of re-querying disappears as a concern; the cache is exact-match only, not a predicate decomposition engine, so a different filter on the same column gets its own entry rather than reusing a partial result.
The cache invalidation strategy is conservative: any mutation that touches a row group marks all aggregate entries for that row group as stale, and a background compaction pass optionally recomputes them; the tradeoff is that the first aggregate query after a mutation pays the scan cost again, but subsequent queries are warm, and since most analytical workloads are read-heavy with occasional batch writes, the cold path is infrequent enough that the simplicity of full invalidation wins over a more sophisticated incremental approach that would inevitably have edge cases around partial predicates.
Why not just use Delta or Iceberg
This is the fair question, and the honest answer is that Delta Lake and Apache Iceberg solve a harder problem than IcefallDB sets out to solve: they target multi-writer distributed table formats with ACID guarantees across a cluster, with snapshot isolation and schema evolution and partition pruning at petabyte scale, and they do it well, but they also carry the weight of that complexity in their metadata layer, their compaction strategies, and their dependency footprint, which is a lot of machinery to pull into a project that wants columnar storage with mutations and warm aggregates on a single node or a small cluster.
IcefallDB’s scope is narrower, and that is the point; if you need a distributed table format with optimistic concurrency control across a hundred nodes, Delta and Iceberg are the right tools, and we use them when the problem calls for it, but when the problem is “I have Parquet files on one machine or one NFS mount and I want to run SQL with mutations and cached aggregates without standing up a JVM stack,” the overhead of those formats is the obstacle rather than the feature, and a JSON manifest with copy-on-write row groups is enough machinery to get there without the weight.
The tradeoff, stated plainly
What you lose by choosing IcefallDB over a proper table format is multi-writer concurrency: the manifest is a single-writer model, one process mutates at a time, and concurrent readers see the last-committed version; you also lose the ecosystem of Delta-compatible tools, the Delta-RS connectors, the Iceberg readers, the Snowflake integrations, because IcefallDB speaks DataFusion’s table provider interface and nothing else, and the Parquet files it writes are standard Parquet but the manifest and aggregate cache are IcefallDB-specific, which means you can always pull the Parquet files out and point DuckDB at them directly, but you lose the mutation history and the warm aggregates when you do.
What you gain is a storage engine that fits in a Rust crate, runs on a laptop or a VPS, stores its data in files that are independently readable by every columnar-aware tool in the ecosystem, and adds the two features that plain Parquet refuses to give you, row-level mutations and cached aggregates, without asking you to adopt a JVM runtime or a distributed coordinator, and the whole thing is a few thousand lines of Rust on top of DataFusion and the parquet crate, which is about the right size for a component that does one thing and stops.