Intermittent silently-incorrect results from Series/DataFrame boolean comparison on large arrays (100% reproducible, fixed by compute.use_numexpr=False)

[ylsleeping]

Description

df[df['col'] == value] on a large (~7M row) DataFrame intermittently
and silently returns zero matching rows for values that are confirmed
to exist in the column. The set of affected values differs between
runs of the exact same script, even against the exact same in-memory
DataFrame within a single process, and across sessions with an
MD5-verified identical source file.

Setting pd.set_option('compute.use_numexpr', False) before running
the comparisons makes the problem disappear completely and reliably
(10/10 trials pass when disabled, vs. 10/10 trials fail when enabled,
on both test machines below).

Environment

• numpy: 2.3.5
• pandas: 2.3.3
• numexpr: 2.14.1
• BLAS: Intel MKL 2025 (mkl-sdl)
• OS: Windows 11
• Reproduced on two separate machines with different CPU architectures:
  • Intel Core i7-9700, 32GB RAM (traditional architecture)
  • Intel Core Ultra 7 256V, 16GB RAM (Lunar Lake hybrid architecture)

Minimal reproducible example

"""
Minimal reproducible example: numexpr / pandas large-array boolean comparison
produces intermittent, silently incorrect results under default (multithreaded)
settings, but is 100% stable when compute.use_numexpr is disabled.

Environment where this was observed:
numpy : 2.3.5
pandas : 2.3.3
numexpr : 2.14.1
BLAS : Intel MKL 2025 (mkl-sdl)
OS : Windows 11, reproduced on TWO different machines:
- Intel Core i7-9700, 32GB RAM
- Intel Core Ultra 7 256V, 16GB RAM

Summary of findings:

1. Pure numpy array comparisons (no pandas) on a same-sized random int64
   array are 100% stable across repeated trials, with or without MKL
   multithreading enabled.
2. A DataFrame with ~7,000,000 rows, when repeatedly filtered by
   df[df['col'] == value] for every unique value in col, will
   sometimes (not always, not deterministically) report zero matching
   rows for a handful of values that are known to exist in the column
   (confirmed via .isin(), value_counts(), and independent numpy-only
   checks against the same underlying array).
3. The set of "missing" values differs between runs, even when reading
   the exact same on-disk file (verified via MD5 checksum) into a fresh
   process each time.
4. Setting pd.set_option('compute.use_numexpr', False) before running
   the filtering loop makes the problem disappear completely (tested
   5+ consecutive runs, 100% consistent, zero false negatives).
5. This strongly points to numexpr's multithreaded evaluation path
   (which pandas invokes automatically for Series/DataFrame comparisons
   once the array exceeds ~100,000 elements) as the source of the
   incorrect results, rather than numpy or the underlying BLAS library.

This script generates a synthetic dataset that mimics the shape of the
data where the bug was originally observed (~7M rows, ~1000 distinct
group IDs) and runs the same filter-loop pattern many times to check
for inconsistent results.

NOTE: On smaller arrays / fewer trials this may not reproduce every time.
If you don't see any "MISMATCH" lines after running this, try increasing
N_ROWS, N_GROUPS, or N_TRIALS.
"""

import sys
import platform
import numpy as np
import pandas as pd

---- Config ----

N_ROWS = 7_000_000
N_GROUPS = 1000
N_TRIALS = 10
SEED = 12345

def build_dataframe():
"""Build a synthetic DataFrame with the same shape/dtype characteristics
as the dataset where the bug was originally observed: a large number of
rows, an int64 'group id' column with every id from 0..N_GROUPS-1
guaranteed present, plus a couple of extra float/int columns to make
it a realistic multi-column DataFrame (the bug was NOT reproducible with
a bare 1-D numpy array; it required a pandas DataFrame with Series-level
comparison)."""
rng = np.random.default_rng(SEED)
group_id = rng.integers(0, N_GROUPS, size=N_ROWS).astype(np.int64)
# guarantee every group id appears at least once
group_id[:N_GROUPS] = np.arange(N_GROUPS, dtype=np.int64)
rng.shuffle(group_id)

df = pd.DataFrame({
    "group_id": group_id,
    "value_a": rng.random(N_ROWS),
    "value_b": rng.random(N_ROWS),
    "label": rng.integers(0, 5, size=N_ROWS).astype(np.int64),
})
return df

def run_filter_loop(df, unique_ids):
"""Reproduces the real-world usage pattern: loop over every known
group id and filter the DataFrame with a boolean comparison."""
missing = []
for gid in unique_ids:
subset = df[df["group_id"] == gid]
if subset.empty:
missing.append(int(gid))
return missing

def main():
print("=" * 70)
print("Environment")
print("=" * 70)
print("Python :", sys.version.split()[0])
print("Platform:", platform.platform())
print("numpy :", np.version)
print("pandas :", pd.version)
try:
import numexpr
print("numexpr :", numexpr.version)
print("numexpr detected cores:", numexpr.detect_number_of_cores())
except ImportError:
print("numexpr : not installed")

df = build_dataframe()
unique_ids = df["group_id"].unique()
assert len(unique_ids) == N_GROUPS, "sanity check failed: not all group ids present"

print(f"\nDataFrame shape: {df.shape}, unique group_id count: {len(unique_ids)}")

# ---- Phase 1: default settings (numexpr enabled, as pandas ships by default) ----
print("\n" + "=" * 70)
print(f"Phase 1: default settings (compute.use_numexpr = "
      f"{pd.get_option('compute.use_numexpr')}), {N_TRIALS} trials")
print("=" * 70)
any_mismatch_default = False
for trial in range(N_TRIALS):
    missing = run_filter_loop(df, unique_ids)
    if missing:
        any_mismatch_default = True
        print(f"  Trial {trial}: MISMATCH - {len(missing)} group_id(s) "
              f"incorrectly reported as missing: {missing}")
    else:
        print(f"  Trial {trial}: OK")

# ---- Phase 2: numexpr disabled ----
pd.set_option("compute.use_numexpr", False)
print("\n" + "=" * 70)
print(f"Phase 2: compute.use_numexpr forced to "
      f"{pd.get_option('compute.use_numexpr')}, {N_TRIALS} trials")
print("=" * 70)
any_mismatch_disabled = False
for trial in range(N_TRIALS):
    missing = run_filter_loop(df, unique_ids)
    if missing:
        any_mismatch_disabled = True
        print(f"  Trial {trial}: MISMATCH - {len(missing)} group_id(s) "
              f"incorrectly reported as missing: {missing}")
    else:
        print(f"  Trial {trial}: OK")

# ---- Summary ----
print("\n" + "=" * 70)
print("Summary")
print("=" * 70)
print(f"Mismatches with numexpr ENABLED : {'YES' if any_mismatch_default else 'no'}")
print(f"Mismatches with numexpr DISABLED: {'YES' if any_mismatch_disabled else 'no'}")

if name == "main":
main()

Actual output — Machine 1 (Intel Core i7-9700, 32GB RAM)

======================================================================
Environment

Python : 3.13.9
Platform: Windows-11-10.0.26200-SP0
numpy : 2.3.5
pandas : 2.3.3
numexpr : 2.14.1
numexpr detected cores: 8
DataFrame shape: (7000000, 4), unique group_id count: 1000

Phase 1: default settings (compute.use_numexpr = True), 10 trials

Trial 0: MISMATCH - 8 group_id(s) incorrectly reported as missing: [140, 81, 389, 100, 596, 473, 693, 50]
Trial 1: MISMATCH - 9 group_id(s) incorrectly reported as missing: [493, 698, 761, 327, 93, 393, 477, 619, 960]
Trial 2: MISMATCH - 8 group_id(s) incorrectly reported as missing: [871, 642, 997, 772, 122, 571, 239, 685]
Trial 3: MISMATCH - 9 group_id(s) incorrectly reported as missing: [229, 708, 724, 874, 956, 909, 577, 896, 396]
Trial 4: MISMATCH - 7 group_id(s) incorrectly reported as missing: [460, 850, 665, 671, 758, 914, 340]
Trial 5: MISMATCH - 4 group_id(s) incorrectly reported as missing: [635, 180, 188, 103]
Trial 6: MISMATCH - 9 group_id(s) incorrectly reported as missing: [651, 181, 462, 97, 237, 503, 714, 497, 202]
Trial 7: MISMATCH - 5 group_id(s) incorrectly reported as missing: [879, 657, 68, 912, 197]
Trial 8: MISMATCH - 9 group_id(s) incorrectly reported as missing: [779, 460, 179, 636, 866, 739, 528, 469, 625]
Trial 9: MISMATCH - 3 group_id(s) incorrectly reported as missing: [203, 854, 950]

Phase 2: compute.use_numexpr forced to False, 10 trials

Trial 0: OK
Trial 1: OK
Trial 2: OK
Trial 3: OK
Trial 4: OK
Trial 5: OK
Trial 6: OK
Trial 7: OK
Trial 8: OK
Trial 9: OK

Summary

Mismatches with numexpr ENABLED : YES
Mismatches with numexpr DISABLED: no

Actual output — Machine 2 (Intel Core Ultra 7 256V, 16GB RAM)

======================================================================
Environment

Python : 3.13.9
Platform: Windows-11-10.0.26200-SP0
numpy : 2.3.5
pandas : 2.3.3
numexpr : 2.14.1
numexpr detected cores: 8
DataFrame shape: (7000000, 4), unique group_id count: 1000

Phase 1: default settings (compute.use_numexpr = True), 10 trials

Trial 0: MISMATCH - 7 group_id(s) incorrectly reported as missing: [152, 750, 578, 492, 898, 412, 341]
Trial 1: MISMATCH - 2 group_id(s) incorrectly reported as missing: [136, 580]
Trial 2: MISMATCH - 4 group_id(s) incorrectly reported as missing: [253, 630, 262, 360]
Trial 3: MISMATCH - 1 group_id(s) incorrectly reported as missing: [376]
Trial 4: MISMATCH - 2 group_id(s) incorrectly reported as missing: [469, 202]
Trial 5: MISMATCH - 3 group_id(s) incorrectly reported as missing: [574, 472, 877]
Trial 6: MISMATCH - 3 group_id(s) incorrectly reported as missing: [991, 519, 443]
Trial 7: MISMATCH - 3 group_id(s) incorrectly reported as missing: [969, 306, 775]
Trial 8: MISMATCH - 1 group_id(s) incorrectly reported as missing: [463]
Trial 9: MISMATCH - 2 group_id(s) incorrectly reported as missing: [493, 957]

Phase 2: compute.use_numexpr forced to False, 10 trials

Trial 0: OK
Trial 1: OK
Trial 2: OK
Trial 3: OK
Trial 4: OK
Trial 5: OK
Trial 6: OK
Trial 7: OK
Trial 8: OK
Trial 9: OK

Summary

Mismatches with numexpr ENABLED : YES
Mismatches with numexpr DISABLED: no

Additional notes

• A pure numpy 1-D array comparison (same size, same dtype, no pandas
  involved) does NOT reproduce the issue across 10 trials on either
  machine, with or without MKL multithreading forced to 1 thread.
• This suggests the issue is specific to the code path pandas uses
  for large Series/DataFrame comparisons (i.e. numexpr's evaluate()),
  rather than numpy or the BLAS backend itself.
• Forcing single-threaded execution via
  MKL_NUM_THREADS=1 / OMP_NUM_THREADS=1 / NUMEXPR_NUM_THREADS=1
  (set before importing numpy) also incidentally resolves the issue,
  consistent with numexpr honoring the same thread-count environment
  variables.
• The number and identity of affected group_ids differs between runs
  and between machines, consistent with a race condition rather than
  a deterministic logic error.

[NamoKoala]

I independently reproduced what appears to be the same issue.

Environment on my reproducing machine: Python 3.13.9, pandas 2.3.3, NumPy 2.3.5, NumExpr 2.14.1, Windows 11.

Using completely synthetic data with 3,232,193 float rows:

ne.set_num_threads(5): intermittent incorrect result; one run failed at iteration 845 with expected=2271418, actual=0
ne.set_num_threads(1): 2000/2000 iterations completed successfully
pd.set_option("compute.use_numexpr", False): 2000/2000 iterations completed successfully

With my original data-processing pipeline, I also observed the same class of intermittent silent mismatch, where the same boolean comparison could sometimes produce zero matching rows despite the expected matches being present.

These results independently support the hypothesis that the multithreaded NumExpr execution path is involved.

[NamoKoala]

Update: I also tested NumExpr 2.14.2 using the same synthetic reproducer and 5 threads. I ran the 2,000-iteration test four times (8,000 iterations total), and all runs completed successfully without reproducing the incorrect result.

In contrast, with NumExpr 2.14.1 and 5 threads, I was able to reproduce the intermittent failure.

I noticed that the 2.14.2 release notes include a fix for run_interpreter() incorrectly returning success when the VM engine failed (#557). I don't know whether that change is related to this issue, but I thought the version-dependent behavior might be useful information.

[ylsleeping]

Confirmed — upgraded to numexpr 2.14.2 (with numpy 2.5.2) and ran
the same reproducer script: 10/10 trials pass with default settings
(previously 10/10 trials failed on 2.14.1 on the same two machines).
This appears to be fixed by the run_interpreter() fix mentioned above.

[27rabbitlt]

I noticed that the 2.14.2 release notes include a fix for run_interpreter() incorrectly returning success when the VM engine failed (#557). I don't know whether that change is related to this issue, but I thought the version-dependent behavior might be useful information.

That fix should not fix the problem in this issue. I think the problem is something wrong happens in that specific Windows build. Because when i test with 2.14.1 from PyPI on Windows, it reproduces; but if i compile from source (still with 2.14.1, still on Windows), I cannot reproduce. Somehow the new 2.14.2 build on Windows, this problem disappears. Still not sure what is the real cause.