feat(tangle-cli): add python_pipeline authoring DSL#20
Open
Silin144 wants to merge 3 commits into
Open
Conversation
Vendor the Python-first pipeline authoring surface into OSS as `tangle_cli.python_pipeline`, the first phase of moving `compile from-python` out of the internal `tangle-deploy` package. This is a clean additive port: the 17 DSL modules (pipeline/task/subpipeline/ registered/ref/raw + trace/graph/emit IR + cfg/types/errors/ids/ placeholders/task_env/compiler_context) carry no runtime Shopify seam — the only zone-root marker resolution lives in the compiler, not the DSL, and is deferred to a later phase. The package is fully OSS-native: all `tangle_deploy` docstring/reference mentions are genericized, and its public surface is `pipeline, task, registered, ref, raw, subpipeline, TaskEnv, In, Out, Outputs`. External deps are stdlib + PyYAML (already required). Generalize the component_from_func authoring-import strip to recognize BOTH authoring surfaces. `_AUTHORING_IMPORT_MODULE` (a single string) becomes `_AUTHORING_IMPORT_MODULES`, a tuple of `tangle_deploy.python_pipeline` (legacy, re-exports the OSS objects) and `tangle_cli.python_pipeline` (canonical), matched via a new `_is_authoring_module` helper that also covers submodules. Non-authoring `tangle_cli.*` / `tangle_deploy.*` runtime helpers and relative imports are still preserved. The authoring shim (`_ensure_tangle_deploy_ authoring_shim`) is intentionally NOT generalized: faking `tangle_cli.python_pipeline` in sys.modules would shadow the real DSL process-wide and break tracing — the real module is importable in-package so no shim is needed for that path. Tests: new tests/test_python_pipeline_dsl.py locks the public surface, the no-`tangle_deploy`-references guard, snake_to_title_case, the CompileError hierarchy, the In/Out/Outputs markers and the raw() contract; tests/test_component_from_func.py gains parallel `tangle_cli.python_pipeline` strip cases plus unit tests for the `_is_authoring_module` / `_is_authoring_import` predicates. Full suite: 671 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Volv-G
reviewed
Jul 9, 2026
Volv-G
reviewed
Jul 9, 2026
Volv-G
reviewed
Jul 9, 2026
Volv-G
reviewed
Jul 9, 2026
…e, genericize tangle-deploy refs
- graph.py: remove unused TaskNode.edge_kw field + its docstring paragraph;
wait_for/depends_on travel through `arguments` and the emitter never read it
- trace.py: remove stale `_ = typing` guard — typing.get_type_hints is called
above, so the import is used and the "we don't call typing.*" comment was wrong
- types.py: fix invalid In[] docstring example (a non-default param followed a
defaulted one -> SyntaxError); reorder to `def my_pipeline(cfg, threshold=5)`
- python_pipeline/*: genericize internal `tangle-deploy` references to the
distribution-agnostic OSS surface ("the hydrator", `tangle sdk pipelines
compile`); tighten the no-internal-references guard to catch both the
`tangle_deploy` (import) and hyphenated `tangle-deploy` (product) spellings
Addresses Volv's review comments on PR TangleML#20.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Volv-G
approved these changes
Jul 9, 2026
Volv-G
left a comment
Volv-G
left a comment
Collaborator
There was a problem hiding this comment.
(AI-assisted) Re-reviewed after commit aff19eb — all four earlier review comments are fully addressed:
- Genericization guard now catches both
tangle_deploy(module) andtangle-deploy(product/command) spellings, and every hyphenated reference in the vendored DSL was genericized totangle sdk pipelines compile/ "the hydrator's own resolver" (verified: the DSL package is now clean of both spellings, andtangle sdk pipelinesis a real CLI namespace). TaskNode.edge_kwdead field and its misleading docstring removed._ = typingstale shim removed.Indocstring example fixed to valid Python (cfgbefore the defaulted param).
Clean additive port; the follow-up commit is doc/dead-code cleanup + a strengthened guard test with no new logic. LGTM.
Volv-G
reviewed
Jul 9, 2026
Volv's review: OSS codegen must not hardcode the downstream package name. ``_AUTHORING_IMPORT_MODULES`` was a frozen tuple that baked in ``tangle_deploy.python_pipeline`` alongside the canonical ``tangle_cli.python_pipeline`` path, so OSS knew about its own consumer — the dependency pointed outward. Replace it with an inward-pointing registry seam mirroring the resolver/reader registries in ``pipeline_hydrator``: - ``_AUTHORING_IMPORT_MODULES`` is now a mutable list seeded with ONLY the canonical ``tangle_cli.python_pipeline`` surface. - ``register_authoring_import_module(module)`` (idempotent) lets a downstream package contribute its own re-export path at import time. - ``authoring_import_modules()`` returns the current tuple snapshot. Delete the OSS-side ``_ensure_tangle_deploy_authoring_shim`` (and its now orphaned ``_identity_decorator`` / ``_AuthoringGeneric`` helpers): OSS no longer fabricates a fake downstream authoring module. A downstream package makes its module importable itself and registers the path. All remaining ``tangle_deploy`` / ``tangle-deploy`` references in ``component_from_func.py`` are genericized, and ``TestNoInternalReferences`` now scans that module too. Tests: the downstream authoring surface (registration + a stand-in ``sys.modules`` module) moves to a suite-wide autouse fixture in ``tests/conftest.py``, standing in for the downstream package. The old constant-contents assertion becomes proper API tests (OSS seed, register, idempotency, immutable snapshot). 674 passed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Phase 1 of migrating
compile from-pythonout of the internaltangle-deploypackage: vendor the Python-first pipeline authoring DSL into OSS astangle_cli.python_pipeline.This is a clean additive port — no existing behavior changes, no new runtime dependency (stdlib + PyYAML, already required).
The DSL (17 modules)
pipeline,task,subpipeline,registered,ref,rawauthoring entry points; the trace-time IR (trace,graph) lowered to the dehydrated dict shape byemit; pluscfg,types,errors,ids,placeholders,task_env,compiler_context.Public surface:
The DSL carries no runtime Shopify seam — the zone-root marker resolution lives in the compiler (a later phase), not the DSL. All
tangle_deploydocstring/reference mentions are genericized (a guard test enforces zero references remain).Authoring-import strip generalization
component_from_funcstrips authoring imports from baked runtime component source so a component running on a thin image (without the DSL) imports cleanly. Generalized to recognize both authoring surfaces:_AUTHORING_IMPORT_MODULE(single string) →_AUTHORING_IMPORT_MODULEStuple oftangle_deploy.python_pipeline(legacy; will re-export the OSS objects) andtangle_cli.python_pipeline(canonical)._is_authoring_modulehelper also covers submodules (...python_pipeline.types).tangle_cli.*/tangle_deploy.*runtime helpers and relative imports are still preserved.The authoring shim (
_ensure_tangle_deploy_authoring_shim) is intentionally not generalized: fakingtangle_cli.python_pipelineinsys.moduleswould shadow the real DSL process-wide and break tracing. The real module is importable in-package, so no shim is needed for that path.Tests
tests/test_python_pipeline_dsl.py— public import surface +__all__, identity offrom-imported objects, the no-tangle_deploy-references guard,snake_to_title_case, theCompileErrorhierarchy,In/Out/Outputsmarkers, and theraw()contract (str-only, rejects Jinja{%%}/{##}, accepts{{...}}runtime sentinels).tests/test_component_from_func.py— paralleltangle_cli.python_pipelinestrip cases (from-import, dotted@tp.task, aliased, plain import, submodule, multi-name, non-authoring preserved) + unit tests for_is_authoring_module/_is_authoring_import.Full suite: 671 passed.
uv lock --checkandgit diff --checkclean.Follow-ups (not this PR)
PipelineCompiler+ the OSScompilecommand; implement the zone-root marker behind a mutable-global/registry seam.tangle-deployto an identity-preserving re-export of the OSS DSL so every existingfrom tangle_deploy.python_pipeline import ...pipeline keeps working unchanged.🤖 Generated with Claude Code