feat(transformations): add external data sources client API - #2732
Merged
Conversation
Adds the data-class layer for the /transformations/externaldata beta endpoint (Fabric OneLake external data sources for transformations). Mirrors the polymorphic read/write pattern already used by hosted_extractors.Source/SourceWrite, since both cases share the same shape: a write-only secret that the API never returns on read.
READ/WRITE/USE actions for the new transformations external data sources endpoint, scoped to all-projects or a data set.
… redaction Round-trip dump/load is covered automatically by the generic CogniteResource suite in test_base.py (two new exclusion lines for the as_write generic test, since the client secret is never returned on read). The focused tests here cover only what that suite can't reach: format dispatch, the unknown-format fallback (including the order-dependent behavior inherited from CogniteResourceList when an unknown-format item is first in a list), as_write raising, secret redaction in repr/str, and usability status.
…ity.dump Addresses gemini-code-assist review on #2729: replace pop-inside-literal with a direct assignment guarded by a membership check.
… models - Dispatching _load methods now declare their concrete return type instead of Self, since the returned instance may be a different subclass than the one the classmethod was called on. - ExternalDataSourceWrite._load's error message hints that an unrecognized format may require an SDK upgrade. - OneLakeExternalDataSourceWrite's docstring example reads the client secret from an environment variable instead of a placeholder string. - Collapse the two issubclass checks in test_base.py's exclusion filter into one using a union type. - Drop the two tests that only exercised CogniteResourceList's base-class type-check behavior, not anything specific to ExternalDataSourceList.
Adds client.transformations.external_data_sources with list, upsert, delete and verify_usability against /transformations/externaldata, plus the generated sync API, the docs accessor and the testing mocks. The API is in public beta, so the methods emit a FeaturePreviewWarning. Delete takes no ignore_unknown_ids as the endpoint accepts no fields other than the items to delete.
Unit tests assert the wire format of each request and the parsing of each response, including that delete sends nothing but the items and that the usability request takes a flat external id. The integration test runs a full upsert/list/verify/delete lifecycle against a live Fabric lakehouse, and skips unless credentials for one are present in the environment.
Contributor
Author
|
/gemini review |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces the TransformationExternalDataSourcesAPI (and its synchronous counterpart SyncTransformationExternalDataSourcesAPI) to manage external data sources available to transformations, such as Microsoft Fabric OneLake. It includes methods for listing, upserting, deleting, and verifying the usability of external data sources, along with comprehensive unit and integration tests, documentation updates, and testing mocks. I have no feedback to provide as there are no review comments.
Base automatically changed from
feat/transformations-externaldata-models
to
master
July 31, 2026 12:20
Khyat-Cognite
marked this pull request as ready for review
August 3, 2026 02:46
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces the TransformationExternalDataSourcesAPI (and its synchronous counterpart SyncTransformationExternalDataSourcesAPI) to manage external data sources available to transformations, such as Microsoft Fabric OneLake. It implements methods for listing, iterating, upserting, deleting, and verifying the usability of these data sources, along with corresponding unit and integration tests, mock support, and documentation updates. There are no review comments, and I have no additional feedback to provide.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2732 +/- ##
==========================================
- Coverage 93.77% 93.74% -0.03%
==========================================
Files 509 513 +4
Lines 52252 52434 +182
==========================================
+ Hits 48999 49156 +157
- Misses 3253 3278 +25
🚀 New features to boost your workflow:
|
haakonvt
reviewed
Aug 7, 2026
haakonvt
left a comment
haakonvt
left a comment
Contributor
There was a problem hiding this comment.
Looks solid! Just some minor stuff left:
…s API Remove the class docstring to match sibling API classes, add missing API docs links to list/upsert/delete/verify_usability, add a doc-only upsert_mode kwarg to upsert to showcase replace semantics, clarify that delete does not silently ignore unknown external IDs, switch to os.getenv in the integration test, and add a TODO for wiring all of Transformations into the docstring-example doctests.
Contributor
Author
|
🦄 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CDF-28251
Stacked on #2729 (
feat/transformations-externaldata-models) — review #2729 first; this PR's diff is client-API only, no data class file is touched.Adds
client.transformations.external_data_sourcesfor the/transformations/externaldataendpoint.Endpoint surface
list(limit=25)/__call__(chunk_size, limit)GET /transformations/externaldatatransformationsExternalDataSourcesAcl:READupsert(data_source)POST /transformations/externaldatatransformationsExternalDataSourcesAcl:WRITEdelete(external_id)POST /transformations/externaldata/deletetransformationsExternalDataSourcesAcl:WRITEverify_usability(external_id)POST /transformations/externaldata/usabilitytransformationsExternalDataSourcesAcl:USEThere is no
/byidsendpoint, so there is noretrieve().Tests
Unit (
tests/tests_unit/test_api/test_transformations_external_data.py, 8 tests) — asserts the wire, not dump/load round trips:list()parses intoOneLakeExternalDataSourcewith typed nested settings__call__()yields one-by-one, and__call__(chunk_size=25)yields 25 + 5 for a 30-item responseupsert()single and sequence — request items carry"format": "one_lake"and complete nested settings includingclientSecretdelete()sends onlyitemsverify_usability()sends a flatexternalIdand unwraps the nested response; the missing-usableVersioncase givesis_usable is FalseIntegration (
tests/tests_integration/test_api/test_transformations/test_externaldata.py) — upsert → list → verify_usability → delete against a live Fabric lakehouse, skipped unlessONELAKE_CLIENT_ID,ONELAKE_TENANT_ID,ONELAKE_CLIENT_SECRET,ONELAKE_WORKSPACE_IDandONELAKE_CONTAINER_IDare set.Also verified locally: sync codegen
verifyclean,pre-commitclean (the only mypy errors are pre-existingtests/utils.pynumpy-typing ones, present on the base commit too), full unit suite failure set identical to the base commit, andmake linkcheck/make htmlwith-Wboth succeed with the new autosummary pages generated.