Report non-secret external payload store URI on worker handshake#765
Draft
YunchuWang wants to merge 1 commit into
Draft
Report non-secret external payload store URI on worker handshake#765YunchuWang wants to merge 1 commit into
YunchuWang wants to merge 1 commit into
Conversation
DTS externalizes large orchestration payloads to Azure Blob Storage, persisting only an opaque "blob:v1:<container>:<blobName>" token (no storage account) in the backend DB. To let the DTS dashboard render those payloads by reading the blob directly with the signed-in user's own AAD token, the backend needs the worker's non-secret storage account base URI. This adds a new external_payload_store_uri field (number 12) to GetWorkItemsRequest and reports it on the work-item handshake: - PayloadStore gains a virtual ExternalPayloadStoreBaseUri (null by default). - BlobPayloadStore derives the account base URI from the container client URI via BlobUriBuilder, stripping the container, blob, SAS and query so it never contains a secret (verified for connection-string and AccountUri + credential configurations). - GrpcDurableTaskWorkerOptions carries the value, the AzureBlobPayloads worker registration sets it alongside the LargePayloads capability, and the processor sends it on GetWorkItemsRequest. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 44c27836-c49c-45fd-ae2b-3309f9c3f0f0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
DTS externalizes large orchestration payloads to Azure Blob Storage, persisting only an opaque token
blob:v1:<container>:<blobName>(no storage account) in the backend DB. To let the DTS dashboard render those payloads by reading the blob directly with the signed-in user's own AAD token, the backend needs the worker's non-secret storage account base URI.The worker already configures the storage account (
LargePayloadStorageOptions) and advertisesWorkerCapability.LargePayloadson itsGetWorkItemshandshake. This PR makes the worker also report its non-secret account base URI on that handshake, via a new proto fieldexternal_payload_store_uri(field number 12) onGetWorkItemsRequest.Changes
src/Grpc/orchestrator_service.proto): addstring external_payload_store_uri = 12toGetWorkItemsRequest. C# stubs are generated at build time byGrpc.Tools(no checked-in stubs), yielding aGetWorkItemsRequest.ExternalPayloadStoreUriproperty.PayloadStore(abstract base): addvirtual Uri? ExternalPayloadStoreBaseUri => null.BlobPayloadStore: override it to return the storage account base URI derived fromthis.containerClient.UriusingBlobUriBuilder, stripping the container, blob, SAS and query. This never contains a secret/SAS/key — even when configured from a connection string (e.g.https://acct.blob.core.windows.net/, orhttp://127.0.0.1:10000/devstoreaccount1for Azurite).GrpcDurableTaskWorkerOptions: addstring? ExternalPayloadStoreUrinext toCapabilities.PostConfigurethat addsWorkerCapability.LargePayloads, setopt.ExternalPayloadStoreUri = store.ExternalPayloadStoreBaseUri?.ToString().GrpcDurableTaskWorker.Processor: sendExternalPayloadStoreUrion theGetWorkItemsRequesthandshake.Tests
Added
test/Worker/Grpc.Tests/ExternalPayloadStoreUriTests.cs(that project already referencesAzureBlobPayloads, keeping the change minimal — no new test project/solution wiring):BlobPayloadStore.ExternalPayloadStoreBaseUrireturns the account base URI with no query/SAS for a connection-string store (UseDevelopmentStorage=true), a connection-string store with an account key (asserts the key is never leaked), and anAccountUri+ credential store.GrpcDurableTaskWorkerOptions.ExternalPayloadStoreUrifrom the store.Validation
dotnet build Microsoft.DurableTask.sln→ 0 errors.Worker.Grpc.Tests→ 141 passed, 0 failed (137 existing + 4 new).Compatibility
Adding a new optional proto field is backward-compatible; all existing field numbers are preserved.