PV_GENERATOR_NaN fails on Ubuntu

[markccchiang]

Summary

src/test/PV_GENERATOR_NaN.test.ts fails on the Ubuntu ICD runners at steps 6 & 7. The backend answers ADD_REQUIRED_TILES for the PV image with two tiles; the 5×25 edge tile, the one whose pixels are NaN, arrives with a correct descriptor (layer, width, height, x all as expected) but with no bytes in image_data, so the very first byte the test reads back is undefined.

Observed failure

Stage pv_generator, Ubuntu 22.04 runner, port 9002:

FAIL src/test/PV_GENERATOR_NaN.test.ts (7.61 s)
  PV_GENERATOR_NaN:Testing PV generator with a region covers NaN and none pixel.
    Register a session
      ✓ check connection (1 ms)
      ✓ Get basepath and modify the directory path (191 ms)
      Go to "set_QA" folder
        ✓ (step 1): Open image (96 ms)
        ✓ (step 2): set cursor and add required tiles (8 ms)
        ✓ (step 3): set SET_SPATIAL_REQUIREMENTS (2 ms)
        ✓ (Step 4): set SET_REGION (1 ms)
        ✓ (Step 5): PV Request (2103 ms)
        ✕ (Step 6 & 7): request 2 tiles after PV response (3 ms)
        ✓ (step 8): set cursor and check the return value (1 ms)

    expect(received).toEqual(expected) // deep equality

    Expected: 0
    Received: undefined

      223 |                     expect(Tile2.tiles[0].x).toEqual(1);
      224 |                     for (let i = 0; i < assertItem.imageDataSequence1.length; i++) {
    > 225 |                         expect(Tile2.tiles[0].imageData[i]).toEqual(assertItem.imageDataSequence1[i]);
          |                                                             ^
      226 |                     }

Test Suites: 1 failed, 1 total
Tests:       1 failed, 8 passed, 9 total

PV_GENERATOR_WIDE.test.ts, run next in the same stage against the same backend: 9 passed.

What the failure tells us

The test requests two tiles of the generated PV image (file_id = 1, ZFP, quality 11):

tiles: [16777216, 16777217]   // layer 1, (x=0,y=0) and (x=1,y=0)

The PV image is 261×25, so tile x=0 is 256 wide and tile x=1 is the 5-wide right-hand edge. That edge tile is the interesting one: step 8 confirms the cursor at (260, 11) reads NaN, i.e., this tile is where the NaN column lives.

In the failing run, the 5-wide tile arrived as Tile2, and everything about it was right except its payload:

assertion	line	result
Tile2.tiles[0].layer === 1	220	pass
Tile2.tiles[0].width === 5	221	pass
Tile2.tiles[0].height === 25	222	pass
Tile2.tiles[0].x === 1	223	pass
Tile2.tiles[0].imageData[0] === 0	225	undefined

imageData is a protobuf bytes field, which protobufjs materialises as a Uint8Array. Reading index 0 and getting undefined rather than a TypeError, which means the array exists and is empty. So the backend sent a TileData with width, height, layer and x populated and image_data of length zero.

Where an empty image_data can come from

Frame::FillRasterTileData (src/Frame/Frame.cc:506-584, carta-backend dev) sets the descriptor first, then compresses:

tile_ptr->set_width(tile_width);
tile_ptr->set_height(tile_height);
...
} else if (compression_type == CARTA::CompressionType::ZFP) {
    auto nan_encodings = GetNanEncodingsBlock(*tile_data_ptr, 0, tile_width, tile_height);
    ...
    auto find_precision = [&](const auto& self, int current, float previous_ratio) -> int {
        Compress(*tile_data_ptr, 0, compression_buffer, compressed_size, tile_width, tile_height, current);
        float compression_ratio = (float)tile_image_data_size / compressed_size;
        ...
    };
    int precision(find_precision(find_precision, requested_precision, -1));
    ...
    tile_ptr->set_image_data(compression_buffer.data(), compressed_size);

The descriptor being correct while the payload is empty pins the problem to compressed_size == 0 at line 572. Two things in this path make that possible and make it pass silently:

1. Compress()'s return value is discarded. carta::Compress (src/DataStream/Compression.cc:22-59) sets compressed_size = zfp_compress(...) and returns status 1 when that is zero — but find_precision ignores the status. A failed compression is therefore indistinguishable from a successful one, and an empty buffer goes straight onto the wire.

2. An all-NaN tile is compressed with its NaNs intact. GetNanEncodingsBlock (src/DataStream/Compression.cc) only substitutes block averages for blocks that contain both valid values and NaNs; its comment says "All-NaN blocks won't affect ZFP compression". So the 5×25 edge tile is handed to zfp_compress as literal NaNs, and what zfp does with that is version- and platform-dependent. That is also exactly why this differs between the macOS runners and the apptainer images.

Note that the knock-on effect: if compressed_size is 0, find_precision computes (float)tile_image_data_size / 0 = inf, decides the ratio is unacceptable, recurses to precision 22, gets inf again, sees compression_ratio == previous_ratio and returns, so the reported compression_quality is 22 for a tile that carries no data at all.

The first item is a defect regardless of the outcome of this issue: a compression failure should be reported, not shipped as a zero-length tile.

Why this is not showing up in dev branch

The Linux ICD action on carta-backend dev runs the whole stage in a bare loop:

while IFS= read -r test_file || [[ -n "$test_file" ]]; do
  CI=true npm test -- "$test_file"
done < $TEST_STAGE

A while loop exits with the status of the last command its body ran, so only the last file in the stage decides whether the job passes. PV_GENERATOR_NaN is the 6th of 7 in ICD_test_stages/pv_generator.tests, followed by PV_GENERATOR_WIDE, which passes. The failure has been swallowed on every Linux run.

On the other hand, macOS has always checked per-file status, but its action also retried a failed file silently, so a file that failed once and passed once looked clean there too.

Reproducing

# with a backend running on the Ubuntu image, config.json pointed at it
npm test src/test/PV_GENERATOR_NaN.test.ts

Useful backend-side evidence, since the backend runs with --verbosity=5: the performance log line

Compress 5x25 tile data in ... ms at ... MPix/s

is emitted for the failing tile, which confirms the compression path ran; and Upgraded precision to 22 (originally requested precision: 11) at debug level would confirm the inf-ratio recursion described above.

The zfp version in each environment is worth recording as part of the report: ldd build/carta_backend | grep zfp inside the Apptainer image versus otool -L on macOS.

Suggested fixes

Backend (carta-backend), the actual defect:

• Check Compress()'s return status in find_precision / FillRasterTileData and fail the tile loudly (log + do not send a zero-length image_data) instead of sending an empty payload with a valid-looking descriptor.

• Decide what an all-NaN tile should transmit. nan_encodings already describes the tile completely in that case, so the payload is redundant; either substitute a constant (e.g. zeros) before compressing so that zfp gets defined input, or short-circuit the compression and send a documented empty payload that clients can rely on. Right now the behaviour is neither defined nor consistent across platforms.

Test (ICD-RxJS), independent of the above:

• imageDataSequence1: [0, 0, 0, 0, 0, 0, 0, 0] asserts the first eight bytes of a raw ZFP stream. That is an assertion about the compressor's byte output, not about the image, and it will keep breaking whenever the zfp version, the precision selection, or the NaN handling changes. Asserting imageData.length > 0 plus the nanEncodings content would test the thing the test actually cares about. 16 test files currently assert raw imageData[...] bytes, so this exposure is not limited to this one file.
• The step 6 & 7 body is if (width === 5) {...} else if (width === 256) {...} with no else. If the first tile were ever neither, no assertion would run, and the test would pass green. Add an else fail(...), or sort the two tiles by width instead of branching.