pybind miss NONE type checking cause core dump

[ericyuanhui]

fix issue : #49

Python 3.13 pybind Parameter Conversion Crash Analysis

Conclusion

This issue is located in Ladybug's pybind Python API C++ parameter conversion layer. It is not in the C API, the query execution engine, or the SQL UNWIND / MERGE implementation.

The lbug_tcdml03_gdb.py script explicitly creates a database with backend="pybind". The core dump also confirms that the loaded extension is:

/home/eric/project/ladybug/tools/python_api/build/ladybug/
_lbug.cpython-313-x86_64-linux-gnu.so

The root cause is that the pybind binding code reinterprets a Python None object as a py::list without validation, then reads it according to CPython's internal list layout. This is undefined behavior. Python 3.12 does not crash only because the adjacent memory happens to contain zeroes, and it incorrectly converts None to an empty list. The uv-managed Python 3.13 binary has mimalloc global state at the same adjacent memory location, which reliably triggers SIGSEGV.

Reproduction Script and Input

Reproduction script:

/home/eric/install/lbug_debug/lbug_tcdml03_gdb.py

Relevant input:

rows = [
    {"id": "r0", "c_list": ["a", "b"]},
    {"id": "r1", "c_list": ["single"]},
    {"id": "r2", "c_list": []},
    {"id": "r3", "c_list": None},
    {"id": "r4", "c_list": ["x", "y"]},
]

The query is:

UNWIND $rows AS row
WITH row WHERE row.`id` IS NOT NULL AND row.`id` <> ""
MERGE (e:`lbuginvest` {`id`: row.`id`})
SET e.`c_list` = row.`c_list`;

The list values in the preceding rows cause the pybind parameter binder to infer a type approximately equivalent to:

LIST<STRUCT<id: STRING, c_list: LIST<STRING>>>

When processing the fourth row, r3, the target type of c_list has already been inferred as LIST<STRING>, while its actual Python value is None.

Core Dump Evidence

Core dump:

/home/eric/core_dump/core-python-26849-1785470797

It was generated by the uv-managed Python 3.13 interpreter. The relevant crash-thread stack trace is:

pybind11::str::check_
pybind11::isinstance<pybind11::str>
PyConnection::transformPythonValueAs
PyConnection::transformPythonValueFromParameterAs
PyConnection::transformPythonValueFromParameterAs
PyConnection::transformPythonValueFromParameterAs
transformPythonParameters
PyConnection::prepare

Relevant local variables from the stack frames:

val   = _Py_NoneStruct
type  = LIST<STRING>
child = 0xf1c300 <_mi_page_empty>

The crash occurs in pybind11::str::check_. It attempts to read the Python object type of child, but child is not a valid Python object. It is mimalloc's empty-page sentinel.

Defective Code Path

File: tools/python_api/src_cpp/py_connection.cpp

PyConnection::transformPythonValueAs() already handles null values at its entry point:

if (val.is_none()) {
    return Value::createNullValue(type);
}

However, prepared-statement parameters use a different code path:

Value PyConnection::transformPythonValueFromParameterAs(const py::handle& val,
    const LogicalType& type) {
    switch (type.getLogicalTypeID()) {
    case LogicalTypeID::LIST: {
        // ...
        py::list lst = py::reinterpret_borrow<py::list>(val);
        for (auto child : lst) {
            children.push_back(std::make_unique<Value>(
                transformPythonValueFromParameterAs(child, ListType::getChildType(type))));
        }

This function does not check val.is_none() before dispatching to the LIST, MAP, or STRUCT branches.

py::reinterpret_borrow<py::list>(val) is an unchecked type wrapper. It does not perform PyList_Check, and it does not convert None into a list. It only wraps the Py_None pointer in a C++ object declared as py::list.

The pybind11 list iterator then uses PySequence_Fast_ITEMS and PyList_GET_SIZE to access internal object fields directly. These macros are only valid for actual list objects or objects returned by PySequence_Fast(). Passing None violates the CPython C API preconditions.

Why Python 3.12 Does Not Crash

The system interpreter is:

Python 3.12.3 [GCC 13.3.0]

A read-only memory inspection around id(None) shows the following consecutive 64-bit words:

offset +0 : 0x00000000ffffffff  # reference count
offset +8 : 0x0000000000a408c0  # None type object
offset +16: 0x0000000000000000
offset +24: 0x0000000000000000

Py_None is actually only a PyObject; its valid memory range only covers the first 16 bytes. The following words are not fields of None, so reading them is already out-of-bounds access.

However, RelWithDebInfo defines NDEBUG, so the PyList_Check() assertion inside CPython's PyList_GET_SIZE() does not execute. The invalid path interprets None + 16 as PyVarObject.ob_size; that word happens to be zero. Therefore:

1. pybind11 treats the forged py::list as having length zero.
2. begin() == end().
3. for (auto child : lst) never executes.
4. The code returns a LIST<STRING> with no child values.

Therefore, Python 3.12 does not correctly preserve the value as SQL NULL. It accidentally converts None into an empty list, []. This explains why the code can execute many times without generating a core dump.

Why uv Python 3.13 Reliably Crashes

The uv-managed interpreter is:

Python 3.13.14 [Clang 22.1.3]

This interpreter binary includes mimalloc, as confirmed by strings, GDB symbols such as _mi_heap_main and _mi_page_empty, and the core dump. Its memory immediately after _Py_NoneStruct is:

offset +0 : 0x00000000ffffffff  # reference count
offset +8 : None type object
offset +16: _mi_heap_main
offset +24: _mi_page_empty
offset +32: _mi_page_empty

The same invalid operation now behaves as follows:

1. None + 16 is incorrectly interpreted as the list's ob_size; it is now a non-zero address value.
2. begin() != end(), so the loop necessarily enters.
3. None + 24 is incorrectly treated as the first list item, yielding _mi_page_empty.
4. transformPythonValueAs(..., STRING) calls py::isinstance<py::str>.
5. pybind11 reads the object type of _mi_page_empty and triggers SIGSEGV.

The reliability of the crash comes from the actual global object layout and embedded mimalloc in the current uv Python 3.13 binary, not from random query data.

Python 3.12 vs. 3.13

The public PyListObject layout and the relevant parts of the PySequence_Fast_ITEMS macro are the same in the local Python 3.12 and Python 3.13 headers. The root cause must not be described as “Python 3.13 changed the list ABI.”

The real difference is the actual binary layout of the two interpreter distributions:

Environment | Interpreter build | None + 16 | Result -- | -- | -- | -- Ubuntu Python 3.12.3 | GCC 13.3.0 | 0 | The forged list is incorrectly treated as empty and does not crash. uv Python 3.13.14 | Clang 22.1.3 with embedded mimalloc | _mi_heap_main | The forged list is treated as non-empty, then dereferences _mi_page_empty and crashes.

Python 3.13 free-threading support, JIT support, and related language changes are not involved. The active interpreter is not python3.13t. Python 3.13 merely exposes the pre-existing bug consistently through the layout of this particular uv-managed distribution.

Missing Test Coverage

Existing tests cover null values inside a list:

[1, 2, 3, None]

In this case, None eventually reaches transformPythonValueAs(), which already has a null check, so it works.

The missing case is a field whose inferred nested type is LIST, MAP, or STRUCT, while the entire field value is None, for example:

{
    "rows": [
        {"id": "r0", "c_list": ["a"]},
        {"id": "r1", "c_list": None},
    ]
}

This test should assert that c_list remains SQL NULL rather than becoming an empty list, and that execution does not crash. Equivalent tests should cover complete None values for inferred MAP and STRUCT fields as well.

Recommended Fix

Handle null values at the entry point of PyConnection::transformPythonValueFromParameterAs() before the switch statement:

if (val.is_none()) {
    return Value::createNullValue(type);
}

This preserves a null value with its target logical type before entering any py::reinterpret_borrow<py::list> or py::reinterpret_borrow<py::dict> branch. It fixes nullable list, map, and struct parameters together.

Upgrading pybind11 alone cannot fix this issue, because the root problem is that Ladybug's caller code disguises None as py::list, violating the preconditions of both pybind11 and the CPython C API.