Skip to content

docs: add guide for adding a CodeQL data extension - #205

Merged
felickz merged 4 commits into
mainfrom
docs/data-extension-contributing-guide
Aug 4, 2026
Merged

docs: add guide for adding a CodeQL data extension#205
felickz merged 4 commits into
mainfrom
docs/data-extension-contributing-guide

Conversation

@felickz

@felickz felickz commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Adds a new "Adding a data extension" section to CONTRIBUTING.md, filling the gap noted while working on #204: there was no documentation on how to build a CodeQL data-extension (MaD) pack, and in particular no clear guidance on when to use ext vs. ext-library-sources.

What's covered

  • Explicit rule of thumb for ext vs ext-library-sources, grounded in what's actually in the repo today:
    • ext = sink models, summary models, and hand-curated framework source models (almost every contribution, including Java: model Spring R2DBC DatabaseClient and io.r2dbc.spi as SQL injection sinks #204's Spring R2DBC DatabaseClient sink models).
    • ext-library-sources (csharp/java only) = a narrower, more conservatively-reviewed pack for bulk-mined sourceModel/kind: "remote" rows (see java/src/library_sources/ExternalAPIsUsedWithUntrustedData.ql), since RemoteFlowSource feeds every standard security query.
  • MaD row shape, with a pointer to a real example (java/ext/manual/org.springframework.r2dbc.model.yml) instead of re-deriving the column reference from scratch.
  • The manual/ vs generated/ convention.
  • A troubleshooting callout for a real gotcha hit during Java: model Spring R2DBC DatabaseClient and io.r2dbc.spi as SQL injection sinks #204: if the local CodeQL CLI's bundled codeql/<language>-all doesn't satisfy the pack's pinned extensionTargets range, new models silently attach zero results instead of failing loudly.
  • How <language>/test/qlpack.yml's dependency on the extensions pack makes codeql test run auto-load new data extensions, and how to confirm in CI that a test is actually exercising the new model (not just compiling).

Also tightens one pre-existing line in "Shipping a change to a query/library pack" that bundled ext/ext-library-sources under the same "csharp/go/java/python" language list — ext-library-sources is actually csharp/java only — and links it forward to the new section.

Why a separate PR from #204

This is general documentation, not scoped to the Spring R2DBC work, so it's branched fresh from main rather than added to #204's branch.

Add a new "Adding a data extension" section to CONTRIBUTING.md covering
how to build a data-extension (MaD) pack: an explicit rule of thumb for
choosing between `ext` and `ext-library-sources` (sink/summary models
vs. remote-source models, and which languages have each), the MaD row
shape, the `manual/` vs `generated/` convention, the `extensionTargets`
version-pinning gotcha that causes silent zero-result failures on a
mismatched local CLI, and how data extensions get picked up automatically
by `codeql test run`.

Also tightens the pack-directory bullet in "Shipping a change to a
query/library pack" to correctly scope `ext-library-sources` to
csharp/java only (previously bundled with `ext`'s wider csharp/go/java/
python list), and links it to the new section.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f02ef23b-e35d-4536-bd58-f2fbc2976ba9
Copilot AI review requested due to automatic review settings August 3, 2026 17:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The new guide references a non-existent in-repo example file (java/ext/manual/org.springframework.r2dbc.model.yml), which makes the documentation incorrect as written.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Pull request overview

This PR updates CONTRIBUTING.md to document how to add CodeQL Model-as-Data (MaD) data extensions, with guidance on choosing the right extensions pack, structuring model rows, and validating changes via tests/CI.

Changes:

  • Adds a new “Adding a data extension” section covering ext vs ext-library-sources, row shape, manual/ vs generated/, and an extensionTargets troubleshooting note.
  • Clarifies the “Shipping a change to a query/library pack” checklist to reflect that ext-library-sources is only for csharp/java, and links to the new section.
  • Adds a new reference link for PR #204.
File summaries
File Description
CONTRIBUTING.md Adds end-to-end contributor documentation for creating and testing data extensions, and tightens the publishing checklist language for extension pack types.
Review details

Suppressed comments (1)

CONTRIBUTING.md:112

  • This section references java/ext/manual/org.springframework.r2dbc.model.yml as a row-shape example, but that file does not exist in the repo right now. Also, the text claims the file has explanatory comments, which may not hold for the replacement; consider wording that only asserts what the referenced file actually contains.
    for the full column reference (`namespace, type, subtypes, name, signature, ext, input/output,
    kind, provenance`). `java/ext/manual/org.springframework.r2dbc.model.yml` shows a real
    `sinkModel`/`summaryModel` pair side by side, with comments explaining *why* each row is shaped
    the way it is — comment your own rows the same way whenever the mapping from source code to a
    MaD row isn't obvious at a glance.
  • Files reviewed: 1/1 changed files
  • Comments generated: 1
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Comment thread CONTRIBUTING.md
…ts pack consumers

Two follow-ups on the "Adding a data extension" section from the initial
commit:

- Drop the "this bit us during development of the Spring R2DBC models"
  framing and the "our own file shows a real pair... comment your own
  rows the same way" phrasing in the row-shape bullet. Both read as
  insider narrative rather than generic guidance. State the
  comment-your-rows recommendation directly instead.
- Clarify that the extensionTargets version gotcha is not a local-dev-only
  quirk: per CodeQL's model pack docs, a model pack's extensions are only
  applied if the resolved codeql/<language>-all version satisfies the
  pinned extensionTargets range, silently and with no warning otherwise.
  This affects anyone consuming the published pack with a CLI outside
  that range, not only contributors testing locally.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f02ef23b-e35d-4536-bd58-f2fbc2976ba9
Copilot AI review requested due to automatic review settings August 3, 2026 17:53

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The new testing guidance currently makes an incorrect repo-wide claim about <language>/test/qlpack.yml depending on the extensions pack (not true for Python).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (1)

CONTRIBUTING.md:151

  • The docs claim that <language>/test/qlpack.yml already depends on githubsecuritylab/codeql-<language>-extensions, but this is not true for all languages in the repo (notably python/test/qlpack.yml does not list githubsecuritylab/codeql-python-extensions). This makes the statement misleading for Python contributors.
    `<language>/test/qlpack.yml` already depends on `githubsecuritylab/codeql-<language>-extensions`
    (and `-library-sources` where applicable) — that dependency is what makes `codeql test run`'s
    normal dependency resolution load your new data extensions automatically, unlike ad-hoc `codeql
    database analyze`/`query run`, which need an explicit `--model-packs` flag. Confirm the test
  • Files reviewed: 1/1 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

…ontributing-guide

# Conflicts:
#	CONTRIBUTING.md
Copilot AI review requested due to automatic review settings August 4, 2026 02:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Not ready to approve

The new testing guidance contains an inaccurate claim about test-pack dependencies that would mislead contributors (notably for Python).

Once you've addressed the issues Copilot identified, you can request another Copilot review.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (1)

CONTRIBUTING.md:141

  • The testing guidance claims <language>/test/qlpack.yml already depends on githubsecuritylab/codeql-<language>-extensions, but that isn’t true for all languages (e.g. python/test/qlpack.yml has no dependency on githubsecuritylab/codeql-python-extensions). As written, this will mislead Python contributors into thinking codeql test run will auto-load new models without any additional setup.
    `<language>/test/qlpack.yml` already depends on `githubsecuritylab/codeql-<language>-extensions`
    (and `-library-sources` where applicable) — that dependency is what makes `codeql test run`'s
    normal dependency resolution load your new data extensions automatically, unlike ad-hoc `codeql
    database analyze`/`query run`, which need an explicit `--model-packs` flag. Confirm the test
  • Files reviewed: 1/1 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

Don't assert <language>/test/qlpack.yml uniformly depends on the
extensions pack - it's not true for python/test/qlpack.yml today.
Reword to tell contributors to check/add the dependency instead of
claiming it's already there.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot-Session: f02ef23b-e35d-4536-bd58-f2fbc2976ba9
Copilot AI review requested due to automatic review settings August 4, 2026 02:30
@felickz

felickz commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

Also addressing the second Copilot review (python/test/qlpack.yml gap, review 4847148821): confirmed that's real - python/test/qlpack.yml doesn't depend on githubsecuritylab/codeql-python-extensions today, unlike java/csharp/go's test packs. Fixed in 3c383c8 by rewording the testing-section claim from "already depends on" to "check that it depends on ... add the dependency if it's missing", so the doc no longer overclaims for Python.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Ready to approve

Changes are documentation-only and the only noted issue is a small clarity tweak to the listed MaD column names.

This review doesn't count toward merge requirements. Sign up for the private preview to control whether Copilot approvals count.

Review details

Suppressed comments (1)

CONTRIBUTING.md:109

  • The MaD column list says input/output, but the actual row shape uses separate input and output columns (e.g. Argument[0].ReturnValue, ReturnValue in java/ext/manual/org.springframework.r2dbc.model.yml). Listing them as one column is misleading for contributors writing summary models.
    for the full column reference (`namespace, type, subtypes, name, signature, ext, input/output,
    kind, provenance`), and any existing file under `manual/` in this repo for a real example of the
  • Files reviewed: 1/1 changed files
  • Comments generated: 0 new
  • Review effort level: Lite

We're testing this review assessment. Please use 👍 or 👎 to tell us if it's correct.

@felickz
felickz merged commit c643627 into main Aug 4, 2026
24 checks passed
@felickz
felickz deleted the docs/data-extension-contributing-guide branch August 4, 2026 02:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants