vera-rubin opened a new issue, #961: URL: https://github.com/apache/arrow-nanoarrow/issues/961
`ArrowArrayViewGetDoubleUnsafe()` reads binary16 `0x0001` as `0x1p-136` instead of `0x1p-24`. `ArrowArrayAppendDouble()` accepts the exactly representable value `0x1p-24` but stores `0x0000` instead of `0x0001`. ## Version and build Apache Arrow nanoarrow **0.10.0-SNAPSHOT**, commit `bb5d8d700b57fe99b8fdadf2babe680a3de60673`. This was reproduced on x86-64 Linux with GCC 13.3.0, CMake 3.28.3 and Python 3.12.3. CMake Debug and Release builds used `NANOARROW_IPC=ON`. The reproducer was compiled with `-O0` and `-O2`, without fast-math. Current `main` was source-checked on 2026-10-12 and still points to the tested commit. The defect is in the core half-float conversion helpers; IPC support was enabled for the recorded checks. ## Reproduce Run these commands in a fresh directory on Linux with Git, CMake and a C compiler. The inline reproducer is offered under CC0-1.0. ```sh git clone https://github.com/apache/arrow-nanoarrow.git nanoarrow cd nanoarrow git checkout --detach bb5d8d700b57fe99b8fdadf2babe680a3de60673 cmake -S . -B build -DNANOARROW_IPC=ON -DCMAKE_BUILD_TYPE=Debug cmake --build build --target nanoarrow_static -j3 cat > half-subnormal.c <<'EOF' #include <stdint.h> #include <stdio.h> #include <string.h> #include "nanoarrow/nanoarrow.h" #define OK(call) do { int rc = (call); if (rc) { \ fprintf(stderr, "%s: %d\n", #call, rc); return 2; } } while (0) static void release_stack(struct ArrowArray* a) { a->release = NULL; } static int sample(uint16_t bits, double value, int verbose) { const void* buffers[] = {NULL, &bits}; struct ArrowArray input = {.length=1, .null_count=0, .n_buffers=2, .buffers=buffers, .release=release_stack}; struct ArrowArrayView view; ArrowArrayViewInitFromType(&view, NANOARROW_TYPE_HALF_FLOAT); OK(ArrowArrayViewSetArray(&view, &input, NULL)); OK(ArrowArrayViewValidate(&view, NANOARROW_VALIDATION_LEVEL_FULL, NULL)); double got = ArrowArrayViewGetDoubleUnsafe(&view, 0); struct ArrowArray output; OK(ArrowArrayInitFromType(&output, NANOARROW_TYPE_HALF_FLOAT)); OK(ArrowArrayStartAppending(&output)); OK(ArrowArrayAppendDouble(&output, value)); OK(ArrowArrayFinishBuildingDefault(&output, NULL)); uint16_t written; memcpy(&written, output.buffers[1], sizeof(written)); if (verbose) printf("input=0x%04x expected=%a read=%a appended=0x%04x\n", bits, value, got, written); else printf("%u %a %u\n", bits, got, written); ArrowArrayRelease(&output); ArrowArrayViewReset(&view); return (got != value || written != bits) ? 1 : 0; } int main(int argc, char** argv) { (void)argv; /* IEEE binary16: exponent zero, fraction one => 1 * 2^-24. */ if (argc == 1) return sample(0x0001, 0x1p-24, 1); unsigned bits; double exact_value; while (scanf("%u %la", &bits, &exact_value) == 2) { if (sample((uint16_t)bits, exact_value, 0) == 2) return 2; } return 0; } EOF cc -std=c11 -Wall -Wextra -O0 -Isrc -Ibuild/src \ half-subnormal.c build/libnanoarrow_static.a -o build/half-subnormal ./build/half-subnormal ``` The program prints: ```text input=0x0001 expected=0x1p-24 read=0x1p-136 appended=0x0000 ``` It exits 1 for the value mismatch. Exit 2 indicates an API error. ## Expected and observed Arrow's [C Data Interface format table](https://arrow.apache.org/docs/21.0/format/CDataInterface.html#data-type-description-format-strings) identifies `e` as float16. For exponent field zero and fraction `F` in 1..1023, [IEEE 754-2019, sections 3.4 and 3.6](https://doi.org/10.1109/IEEESTD.2019.8766229) defines the binary16 value as `(-1)^sign * F * 2^-24`. The [getter contract](https://github.com/apache/arrow-nanoarrow/blob/bb5d8d700b57fe99b8fdadf2babe680a3de60673/src/nanoarrow/nanoarrow.h#L1344-L1349) returns an element as a double. The [appender contract](https://github.com/apache/arrow-nanoarrow/blob/bb5d8d700b57fe99b8fdadf2babe680a3de60673/src/nanoarrow/nanoarrow.h#L1020-L1026) appends by value and rejects values that cannot be represented exactly. The input here is non-null, in bounds and exactly representable in binary16, float and double. Reading `0x0001` should return `0x1p-24`. Appending `0x1p-24` should store `0x0001`. ## Cause The [conversion helpers](https://github.com/apache/arrow-nanoarrow/blob/bb5d8d700b57fe99b8fdadf2babe680a3de60673/src/nanoarrow/common/inline_buffer.h#L106-L152) explain both results. The decoder sets the binary32 exponent to zero and shifts the fraction left by 13 without rescaling it. This yields `F * 2^-136`. The encoder sets the fraction to zero when the rebased exponent is below 1. Both helpers need an explicit binary16 subnormal conversion. ## Scope An exhaustive sweep of all 63,488 finite binary16 encodings found 2,046 getter mismatches and 2,046 appender mismatches. Each set is exactly every nonzero subnormal of either sign. All 61,442 normal and signed-zero controls pass. Expected values came from exact Python rational arithmetic and were cross-checked with Python `struct`. An independently written harness using an owned array, an explicit schema and offset 1 produced the same sets. With arrays initialized from raw binary16 encodings, IPC wire bytes and raw read-back values match all 63,488 finite encodings. PyArrow 21.0.0 validates the stream and preserves its raw data buffer. The nanoarrow double getter after IPC reading still has 2,046 mismatches. Infinities, NaNs, rounding of inexact inputs, other architectures and language-binding conversions were not tested for this report. Found by ChipLab's AI verification agents and independently reproduced. The reproducer, verification checks, and report text were prepared with assistance. ChipLab ref: CLAB-2026-867 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
