From 1173ec90eb50f1947cbbc7c0108a0b86856e5fd8 Mon Sep 17 00:00:00 2001 From: Dewey Dunnington Date: Thu, 30 Jul 2026 13:39:55 -0400 Subject: [PATCH] post --- _posts/2026-08-01-nanoarrow-0.9.0-release.md | 156 +++++++++++++++++++ 1 file changed, 156 insertions(+) create mode 100644 _posts/2026-08-01-nanoarrow-0.9.0-release.md diff --git a/_posts/2026-08-01-nanoarrow-0.9.0-release.md b/_posts/2026-08-01-nanoarrow-0.9.0-release.md new file mode 100644 index 00000000000..2eff097c48b --- /dev/null +++ b/_posts/2026-08-01-nanoarrow-0.9.0-release.md @@ -0,0 +1,156 @@ +--- +layout: post +title: "Apache Arrow nanoarrow 0.9.0 Release" +date: "2026-08-01 00:00:00" +author: pmc +categories: [release] +--- + + +The Apache Arrow team is pleased to announce the 0.9.0 release of +Apache Arrow nanoarrow. This release consists of 38 resolved GitHub issues from +5 contributors. + +## Release Highlights + +In addition to a number of bugfixes and minor build system improvements, we +added several new features in nanoarrow 0.9.0. + +- Dictionary decoding support in IPC reader +- Reference-counted array/buffer support +- LZ4 decompression support in R and Python bindings + +See the +[Changelog](https://github.com/apache/arrow-nanoarrow/blob/apache-arrow-nanoarrow-0.9.0/CHANGELOG.md) +for a detailed list of contributions to this release. + +## Features + +### Dictionary decode support + +Whereas the nanoarrow IPC reader suppports most Arrow IPC features, dictionary support +was a long requested gap in the reader functionality (mostly requested by users of +the [DuckDB nanoarrow extension](https://github.com/paleolimbot/duckdb-nanoarrow), which uses nanoarrow's reader). Dictionary encoding is used to reduce the size of frequently +repeated values and is the serialized equivalent of the "dictionary" data type that +is exposed in most Arrow implementations. + +In nanoarrow 0.9.0 built with the IPC feature enabled, streams that include the most +common forms of dictionary encoding (i.e., dictionary replacement) should now work +out of the box. This includes nested/complex dictionary types and dictionary +replacement but does not include "delta" dictionaries (i.e., dictionaires that +grow larger as more values are encountered in the encoded values). + +In R this is accessible via `read_nanoarrow()`; in Python this is accessible via +`nanoarrow.ArrayStream.from_readable()`; in C this is available via the +higher level `ArrowIpcArrayStreamReader` API. Lower level users of the +`ArrowIpcDecoder` will have to update existing usage to use the +`ArrowIpcDecoder...WithDictionaries()` variants of some functions +to support input with dictionary schemas or batches. + +### Reference-counted array/buffer support + +In previous versions (since the introduction of the IPC reader), the +`ArrowIpcSharedBuffer` has supported reading IPC streams and sharing an +underlying set of data buffers for a group of arrays; however, this wasn't +quite sufficient for the more complex case of decoding a dictionary and +attaching cheaply-cloned shared "values" arrays for potentially many batches. +Version 0.9.0 moves this functionality to the `ArrowSharedBuffer` and expands it +to support moving all of an array's buffers into a shared state that can be +more cheaply cloned. + +### LZ4 decompression support in R and Python + +While LZ4 decompression support was has long been available via the pluggable decoder +framework (and available since 0.8.0 as a built-in compile time option), reading +IPC streams with LZ4 buffer compression was not possible in the R or Python bindings. +In 0.9.0, the requisite configuration options were added such that the packages are +built with LZ4 when it is available on the system. + +```python +import io +import nanoarrow as na +import pyarrow as pa + +buf = io.BytesIO() +batch = pa.record_batch({"x": range(1000)}) +with pa.ipc.new_stream(buf, batch.schema, options=pa.ipc.IpcWriteOptions(compression="lz4")) as w: + w.write_batch(batch) + +buf.seek(0) +na.ArrayStream.from_readable(buf).read_all() +# nanoarrow.Array>[1000] +# {'x': 0} +# {'x': 1} +# {'x': 2} +# {'x': 3} +# {'x': 4} +# {'x': 5} +# {'x': 6} +# {'x': 7} +# {'x': 8} +# {'x': 9} +# ...and 990 more items +``` + +```r +library(nanoarrow) +library(reticulate) + +# IPC Write with compression not available in arrow/R +pa <- reticulate::import("pyarrow") +io <- reticulate::import("io") + +buf <- io$BytesIO() +batch <- arrow::record_batch(x = 1:1000) +writer <- pa$ipc$new_stream(buf, batch$schema, options = pa$ipc$IpcWriteOptions(compression = "lz4")) +writer$write_batch(batch) +writer$close() + +nanoarrow::read_nanoarrow(as.raw(buf$getvalue())) |> + tibble::as_tibble() +#> # A tibble: 1,000 × 1 +#> x +#> +#> 1 1 +#> 2 2 +#> 3 3 +#> 4 4 +#> 5 5 +#> 6 6 +#> 7 7 +#> 8 8 +#> 9 9 +#> 10 10 +#> # ℹ 990 more rows +``` + +## Contributors + +This release consists of contributions from 5 contributors in addition +to the invaluable advice and support of the Apache Arrow community. + +```console +$ git shortlog -sn apache-arrow-nanoarrow-0.9.0.dev..apache-arrow-nanoarrow-0.9.0 + 36 Dewey Dunnington + 2 Andrew Kane + 2 Bryce Mecum + 1 Michael Osipov + 1 Oliver Borchert +```