feat(transformations): add external data source data classes by Khyat-Cognite · Pull Request #2729 · cognitedata/cognite-sdk-python

Khyat-Cognite · 2026-07-30T06:31:47Z

Summary

Data-class layer for the transformations external data sources API — Fabric OneLake connections for ext_onelake() in transform SQL. Data models only, no client API or docs guide (see follow-up plan below).

Supersedes #2714. That PR targeted /transformations/external_data; the contract has since moved to /transformations/externaldata with a materially different payload:

OneLakeLocationDescription is now workspaceId/containerId (UUIDs), not workspaceName/containerName
settings, credentials, locationDescription, format are all required on both read and write
clientSecret is required on every write (full replace); never returned on read
The usability response wraps the id: {"externalId": {"externalId": "..."}, "usableVersion": "..."}
The endpoint is public beta

Endpoints / ACL

Endpoint	Method	ACL
`/transformations/externaldata`	GET list, POST upsert (201)	READ / WRITE
`/transformations/externaldata/delete`	POST	WRITE
`/transformations/externaldata/usability`	POST	USE

Production code

cognite/client/data_classes/transformations/externaldata.py — read/write/list models, OneLake credentials/settings, usability
cognite/client/data_classes/capabilities.py — TransformationsExternalDataSourcesAcl
cognite/client/data_classes/transformations/__init__.py — re-exports (kept out of the top-level data_classes/__init__.py per review)
docs/source/transformations.rst — one automodule line for the new module

Tests

tests/tests_unit/test_base.py — two exclusion lines (mirroring the existing Source/SourceList exclusions) so the generic as_write tests correctly skip this hierarchy; round-trip dump/load is otherwise covered automatically by that same generic suite
tests/tests_unit/test_data_classes/test_capabilities.py — new ACL entries in all_acls()
tests/tests_unit/test_data_classes/test_transformations/test_externaldata.py — format dispatch, unknown-format fallback (including the order-dependent behavior inherited from CogniteResourceList when an unknown-format item is first in a list — a known trade-off already present for SourceList, not something new here), as_write raising, secret redaction in repr/str, usability status

What comes next

PR	Scope
Client API	`client.transformations.external_data_sources` (list, upsert, delete, verify_usability), sync codegen, mocks, API tests
Docs	`transformations_external_data.rst` quickstart + toctree

Test plan

pytest tests/tests_unit/test_base.py -k "OneLake or ExternalDataSource" — 48 passed
pytest tests/tests_unit/test_base.py — 3170 passed / 7 pre-existing failures (missing optional pandas dep, identical on baseline)
pytest tests/tests_unit/test_data_classes/test_transformations/test_externaldata.py tests/tests_unit/test_data_classes/test_capabilities.py — all passing
ruff check / ruff format --check / mypy on all touched files — clean
pytest docs — passing

Adds the data-class layer for the /transformations/externaldata beta endpoint (Fabric OneLake external data sources for transformations). Mirrors the polymorphic read/write pattern already used by hosted_extractors.Source/SourceWrite, since both cases share the same shape: a write-only secret that the API never returns on read.

READ/WRITE/USE actions for the new transformations external data sources endpoint, scoped to all-projects or a data set.

… redaction Round-trip dump/load is covered automatically by the generic CogniteResource suite in test_base.py (two new exclusion lines for the as_write generic test, since the client secret is never returned on read). The focused tests here cover only what that suite can't reach: format dispatch, the unknown-format fallback (including the order-dependent behavior inherited from CogniteResourceList when an unknown-format item is first in a list), as_write raising, secret redaction in repr/str, and usability status.

Khyat-Cognite · 2026-07-30T08:17:25Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for external data sources in CDF Transformations, specifically adding models and capabilities for Microsoft Fabric OneLake external data sources. It includes the implementation of data classes, credentials handling with redaction, usability checks, documentation updates, and comprehensive unit tests. I have no feedback to provide as there are no review comments.

gemini-code-assist

Code Review

This pull request introduces support for external data sources (specifically Microsoft Fabric OneLake) in CDF Transformations by adding new data classes, ACL capabilities, and corresponding unit tests. The review feedback suggests improving the readability and safety of the ExternalDataSourceUsability.dump method by avoiding mutating a dictionary with pop inside a dictionary literal construction.

…ity.dump Addresses gemini-code-assist review on #2729: replace pop-inside-literal with a direct assignment guarded by a membership check.

Khyat-Cognite · 2026-07-31T03:59:18Z

The CI failure is not my PR it is pre-existing (someone messed up i guess 🤔 ):
ERROR tests/tests_integration/test_api/test_workflows.py::TestWorkflowVersions::test_list_workflow_versions - cognite.client.exceptions.CogniteAPIError: ['integration_test-workflow_bhxnp:2:functions']: Failed validating referenced function status: [400] Request to 'https://api.cognitedata.com/api/v1/projects/python-sdk-contributor/functions/byids', with requestId: 'fd143294-79cd-9018-a3f0-37fef7feea77' failed: Function id not found | code: 400 | X-Request-ID: 9b66e2e3-09d8-98b6-8b76-99df04dccd4c | cluster: api | project: python-sdk-contributor

haakonvt

Very clean PR 👌 A handful of comments, otherwise LGTM

… models - Dispatching _load methods now declare their concrete return type instead of Self, since the returned instance may be a different subclass than the one the classmethod was called on. - ExternalDataSourceWrite._load's error message hints that an unrecognized format may require an SDK upgrade. - OneLakeExternalDataSourceWrite's docstring example reads the client secret from an environment variable instead of a placeholder string. - Collapse the two issubclass checks in test_base.py's exclusion filter into one using a union type. - Drop the two tests that only exercised CogniteResourceList's base-class type-check behavior, not anything specific to ExternalDataSourceList.

haakonvt

LGTM

Khyat-Cognite · 2026-07-31T08:28:15Z

🦄 -- will wait for CI to be fixed ^^

codecov · 2026-07-31T12:12:08Z

Codecov Report

❌ Patch coverage is 98.23009% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 93.69%. Comparing base (b60a3a8) to head (7009b66).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
...lient/data_classes/transformations/externaldata.py	97.48%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2729      +/-   ##
==========================================
- Coverage   93.72%   93.69%   -0.04%     
==========================================
  Files         505      507       +2     
  Lines       51512    51738     +226     
==========================================
+ Hits        48281    48477     +196     
- Misses       3231     3261      +30

Files with missing lines	Coverage Δ
cognite/client/data_classes/capabilities.py	`97.04% <100.00%> (+0.03%)`	⬆️
...te/client/data_classes/transformations/__init__.py	`85.76% <100.00%> (+0.05%)`	⬆️
tests/tests_unit/test_base.py	`98.94% <100.00%> (+<0.01%)`	⬆️
.../tests_unit/test_data_classes/test_capabilities.py	`100.00% <ø> (ø)`
..._classes/test_transformations/test_externaldata.py	`100.00% <100.00%> (ø)`
...lient/data_classes/transformations/externaldata.py	`97.48% <97.48%> (ø)`

... and 9 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Khyat-Cognite added 3 commits July 30, 2026 12:00

feat(iam): add transformationsExternalDataSourcesAcl capability

db85859

READ/WRITE/USE actions for the new transformations external data sources endpoint, scoped to all-projects or a data set.

Khyat-Cognite self-assigned this Jul 30, 2026

gemini-code-assist Bot reviewed Jul 30, 2026

View reviewed changes

Khyat-Cognite marked this pull request as ready for review July 30, 2026 08:22

Khyat-Cognite requested review from a team as code owners July 30, 2026 08:22

gemini-code-assist Bot reviewed Jul 30, 2026

View reviewed changes

Comment thread cognite/client/data_classes/transformations/externaldata.py

fix(transformations): avoid dict mutation in ExternalDataSourceUsabil…

9b2dada

…ity.dump Addresses gemini-code-assist review on #2729: replace pop-inside-literal with a direct assignment guarded by a membership check.

haakonvt reviewed Jul 31, 2026

View reviewed changes

Khyat-Cognite requested a review from haakonvt July 31, 2026 08:08

haakonvt approved these changes Jul 31, 2026

View reviewed changes

haakonvt added the waiting-for-risk-review Waiting for a member of the risk review team to take an action label Jul 31, 2026

Khyat-Cognite added risk-review-ongoing Risk review is in progress waiting-for-team Waiting for the submitter or reviewer of the PR to take an action and removed waiting-for-risk-review Waiting for a member of the risk review team to take an action labels Jul 31, 2026

Khyat-Cognite enabled auto-merge July 31, 2026 08:28

This was referenced Jul 31, 2026

feat(transformations): add external data sources client API #2732

Merged

docs(transformations): add external data sources guide #2734

Merged

Merge branch 'master' into feat/transformations-externaldata-models

7009b66

Khyat-Cognite added this pull request to the merge queue Jul 31, 2026

Merged via the queue into master with commit 4f449dd Jul 31, 2026
22 checks passed

Khyat-Cognite deleted the feat/transformations-externaldata-models branch July 31, 2026 12:20

erlendvollset mentioned this pull request Jul 31, 2026

chore(master): release cognite-sdk-python 8.11.0 #2706

Merged

Navigation Menu

Uh oh!

feat(transformations): add external data source data classes - #2729

feat(transformations): add external data source data classes#2729
Khyat-Cognite merged 6 commits into
masterfrom
feat/transformations-externaldata-models

Khyat-Cognite commented Jul 30, 2026 •

edited

Loading

Uh oh!

Khyat-Cognite commented Jul 30, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Khyat-Cognite commented Jul 31, 2026

Uh oh!

haakonvt left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

haakonvt left a comment

Uh oh!

Khyat-Cognite commented Jul 31, 2026

Uh oh!

codecov Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Khyat-Cognite commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Endpoints / ACL

Production code

Tests

What comes next

Test plan

Uh oh!

Khyat-Cognite commented Jul 30, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Khyat-Cognite commented Jul 31, 2026

Uh oh!

haakonvt left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

haakonvt left a comment

Choose a reason for hiding this comment

Uh oh!

Khyat-Cognite commented Jul 31, 2026

Uh oh!

codecov Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Khyat-Cognite commented Jul 30, 2026 •

edited

Loading

codecov Bot commented Jul 31, 2026 •

edited

Loading