feat(pyarrow): describe conversions in PyO3 introspection data

[jonasdedden]

Which issue does this PR close?

• Closes arrow-pyarrow offers no way to generate stubs through PyO3's experimental-inspect #10491.

Rationale for this change

PyO3's experimental-inspect feature records the Python type of every conversion into the built binary, so that tools like maturin generate-stubs can emit a real .pyi. The type comes from the INPUT_TYPE / OUTPUT_TYPE associated constants on FromPyObject / IntoPyObject.

PyArrowType implements both traits but leaves those constants at their default, _typeshed.Incomplete. Because PyArrowType is how bindings exchange every Arrow value with Python, that one default erases the entire public API of any such binding. A function like

#[pyfunction]
fn cast_record_batch(
    record_batch: PyArrowType<RecordBatch>,
    schema: PyArrowType<Schema>,
) -> PyResult<PyArrowType<RecordBatch>> { ... }

generates

def cast_record_batch(record_batch: Incomplete, schema: Incomplete) -> Incomplete: ...

What changes are included in this PR?

• A new experimental-inspect feature on arrow-pyarrow, which just turns on PyO3's. arrow gets a matching feature that implies pyarrow.
• FromPyArrow::INPUT_TYPE, ToPyArrow::OUTPUT_TYPE and IntoPyArrow::OUTPUT_TYPE, defaulting to _typeshed.Incomplete so that out-of-tree implementors are unaffected.
• Implementations for every type the crate converts: DataType, Field, Schema, ArrayData (→ pyarrow.Array), RecordBatch, Vec<T>, ArrowArrayStreamReader, Box<dyn RecordBatchReader + Send> and Table.
• PyArrowType's FromPyObject / IntoPyObject impls forward them.
• Unit tests pinning the rendered hint for each type, and a note in the crate docs.

Input and output are separate constants because they genuinely differ: Vec<T> is built from anything iterable but handed back as a list, so it is collections.abc.Iterable[pyarrow.RecordBatch] in and list[pyarrow.RecordBatch] out.

Are these changes tested?

Yes. arrow-pyarrow gains unit tests asserting the rendered hint for every conversion, run with cargo test -p arrow-pyarrow --features experimental-inspect.

Beyond that, the change was verified end to end against a real binding: with arrow patched to this branch and the binding's own code left on plain PyArrowType, maturin generate-stubs produces the signatures and the from pyarrow import ... line shown above.

Are there any user-facing changes?

No behaviour changes, and nothing is compiled unless experimental-inspect is enabled. The traits gain associated constants, but they are defaulted and feature-gated, so existing implementations keep compiling untouched.

[Jefffrey]

do these tests get run in our CI

[jonasdedden]

They are now (sorry), by putting an additional cargo test call here in .github/workflows/integration.yml. Because the test is feature gated, I chose that place; if you have a better suggestion, please let me know.

[Jefffrey]

took the liberty of moving the tests to arrow.yml workflow which i think is a better location

edit: i missed the part about needing a python to link to, so reverted my change 👍

[Jefffrey]

thanks @jonasdedden