✨ Preprocessor-aware C/C++ extraction via libclang#85
Open
ubmarco wants to merge 25 commits into
Open
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #85 +/- ##
==========================================
+ Coverage 91.40% 91.93% +0.53%
==========================================
Files 34 43 +9
Lines 3012 3521 +509
Branches 322 364 +42
==========================================
+ Hits 2753 3237 +484
- Misses 160 174 +14
- Partials 99 110 +11 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
ubmarco
added a commit
that referenced
this pull request
Jun 30, 2026
…ls.STYLE_TYPES) (#86) ## Problem `sphinxcontrib-typer 0.9.1` (released after #84) imports `typer.rich_utils.STYLE_TYPES`, which does not exist in the `typer<0.26.8` range pinned by #84. Any fresh docs build now resolves `sphinxcontrib-typer 0.9.1` and fails during `sphinx-build`: ``` AttributeError: module 'typer.rich_utils' has no attribute 'STYLE_TYPES' ``` This affects `main` and every open PR (e.g. #85) on a fresh dependency resolution. Note #84 capped *typer* for a **different** removed attribute (`STYLE_METAVAR`); this is a new break originating from the sphinxcontrib-typer side. ## Fix Cap `sphinxcontrib-typer>=0.5.1,<0.9.1`. `0.9.0` builds cleanly with the capped typer `0.26.7`; verified locally that `sphinx-build -nW` no longer raises the `AttributeError`. Dependency-only change — no source, tests, or typing impact.
Pass file_field (raw entry["file"]) instead of abs_file to filter_args so that when compile_commands.json encodes the input as a relative path (e.g. "src/a.cpp") the argument is correctly stripped rather than leaking to libclang as a spurious positional argument. Also fix the existing command-form test whose entry had an inconsistent absolute path in the command string but a relative "file" field; both now use the same relative form. Regression test added: directory=tmp, file="src/a.cpp", arguments=[..., "src/a.cpp"] -> flags == ["-DA=1"].
…harden test _resolve_preproc_args now returns None when a compile DB is found but the source file is not listed in it, causing create_src_objects_libclang to skip the file with a debug log rather than falling back to defines_to_args. No-DB path (defines fallback) is unchanged. Also: add IMPL_ALWAYS vacuous-pass guard to test_libclang_engine_other_variant and a new test_libclang_skip_file_absent_from_compile_commands that verifies the skip behaviour end-to-end.
Add test_libclang_active_matches_golden to assert Python libclang engine produces the same 4 active needs as the shared Rust golden (IMPL_ALWAYS, IMPL_VAR_A, IMPL_PROTO_3, IMPL_LINUX_A) with defines VARIANT_A=1, PLATFORM_LINUX=1, PROTOCOL_VERSION=3, excluding IMPL_VAR_B as expected.
SourceAnalyseConfig.preprocessor carried a flat metadata schema
{type: [object, null]}, so check_schema validated the constructed
PreprocessorConfig *instance* against JSON type object and raised at
config-inited: 'PreprocessorConfig(...) is not of type object, null'.
Nested dataclass config fields (need_id_refs_config, marked_rst_config,
oneline_comment_style) carry no flat schema; the preprocessor field must
match. Its structure is enforced by convert_analyse_config. Adds a
regression test exercising the validation path.
The opt-in libclang engine adds preprocessor-aware extraction (skipping comments inside inactive #if/#ifdef regions) for C, C++, and Objective-C. Tree-sitter already covers C/C++ extraction, so the extra was misnamed `cpp`: it implied "C++ support" rather than the libclang backend it actually pulls in. Rename it to `libclang`, matching the dependency and the install hint. Add tests/test_libclang_optional.py, a regression guard that blocks `clang` in a fresh interpreter (a tree-sitter-only install, by construction) and asserts: the default tree-sitter path imports and runs without libclang; and selecting the libclang engine without the extra raises the documented "pip install 'sphinx-codelinks[libclang]'" hint. It proves the block is effective first, so it cannot pass vacuously under CI, which installs the extra.
codelinks is the OSS reference implementation; cross-impl feature parity is ubCode's concern and lives there. Remove the shared Rust golden + its test; codelinks keeps its own behavioral tests (inactive-branch exclusion, resilience, compile_commands, header extraction).
Restore the libclang golden snapshot (its earlier removal was a mistake) and add a tree-sitter golden beside it. Both engines are now pinned to declarative JSON snapshots through a shared projection helper: - variants_branching.expected.json (libclang active set, 4 markers) - variants_branching.treesitter.expected.json (tree-sitter, all 5 markers) The snapshots are language-agnostic declarative data so any conforming implementation can assert against the same fixtures.
Add an 'engine' dimension to the declarative extraction harness: a fixture case
may set 'engine: libclang' + 'defines' to exercise the preprocessor-aware path
(inactive #if/#ifdef branches excluded); libclang cases skip when the clang
bindings are absent. variants_branching now lives as declarative fixtures under
both engines (tests/data/extraction/preproc_variants.yaml): tree-sitter -> all 5
markers, libclang -> the 4-marker active set.
Drop the bespoke goldens (variants_branching{,.treesitter}.expected.json and
test_libclang_active_matches_golden / test_treesitter_matches_golden); the
libclang behavioral tests (inactive-branch exclusion, resilience, compile_commands,
§3.3 skip, header-standalone) remain.
- mypy: ignore missing clang.cindex stubs; int() the libclang parse flags; import CommentType from its defining module; fix preproc TOML ignore codes - ruff-format the preproc engine + its tests Fixes the MyPy and Pre-commit CI checks on the libclang PR.
ubmarco
force-pushed
the
feat/libclang-preprocessor
branch
from
af5c95c to
b97c66a
Compare
…ers, don't crash) A header carrying oneline markers never appears in compile_commands.json, so the engine parses it standalone with the global defines. libclang infers the C language from a .h/.c/.inc extension, and a C++ -std flag then makes clang reject the combination and return a NULL TU (TranslationUnitLoadError) — which aborted the whole Sphinx build (notably 'sphinx-build -b ubtrace -W' on the presets). Pin -x to match -std in defines_to_args so such files parse as C++ and their markers are EXTRACTED rather than dropped. Keep a last-resort TranslationUnitLoadError guard (logged at info, not warning, so -W stays green) for files that still cannot load as a TU at all. Adds a regression test + .h fixture proving a C-extension header's markers extract.
ubmarco
force-pushed
the
feat/libclang-preprocessor
branch
from
893e9dd to
3618ba4
Compare
Adds a .h header case to the shared declarative extraction suite: a header is parsed standalone under libclang (pinned to C++ via defines_to_args) and its active-branch markers extract while inactive-branch markers are excluded. Adds a 'cpp_header' lang -> .h mapping. This is the engine-agnostic golden that surfaces the same requirement for the Rust engine (ubc_codelinks).
Declarative libclang cases covering nested #ifdef, #elif chains, #if 0 dead blocks, defined()/boolean logic, arithmetic comparisons, and an in-source #define driving a branch. Each keeps markers in both the taken and untaken branches; only the active-region markers survive, proving the preprocessor is evaluated (tree-sitter would keep them all).
The declarative harness can now materialize a compile_commands.json (inline `compile_commands` entries) or point at an explicit `compile_commands_path`. New preproc_compile_db cases cover DB-provided flags, a header absent from a present DB (parsed standalone), a translation-unit source absent from the DB (skipped per spec §3.3), and an unreadable explicit DB path (defines fallback).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an optional libclang preprocessor engine for C/C++ marker extraction.
The default tree-sitter path sees every comment; the libclang engine evaluates
the preprocessor (
#if/#ifdef/defines) and emits only the markers inactive branches — so a need behind an inactive
#elseis correctly dropped.analyse/preproc/— libclang loader,compile_commands.jsonwalk-updiscovery + per-file flag resolution, and an active-branch parser.
libclangextra (pip install sphinx-codelinks[libclang]);plain tree-sitter extraction is unaffected when it is absent.
PreprocessorConfig) +analyse/configurationdocs.
Testing — declarative fixtures
Tests use the declarative fixture + snapshot harness from #83. A new
tests/data/extraction/preproc_variants.yamlfixture runs the same sourcethrough both engines with per-engine golden snapshots:
…[preproc_variants-variants_branching_treesitter]keeps every branch;…[preproc_variants-variants_branching_libclang]drops the inactive#elseand keeps only the active set under
-DVARIANT_A -DPLATFORM_LINUX -DPROTOCOL_VERSION=3.Plus focused unit tests for the loader,
compile_commandshandling, and theparser. Full extraction-fixture + preproc suites pass locally.
Rebase note
Rebased onto current
main(post-#83). The branch carried its own copy of thedeclarative harness (developed before #83 merged); those commits were dropped
in the rebase, so this PR is purely the libclang additions layered on top of
the merged harness.