diff --git a/README.md b/README.md index b86e183cd..3aba62b03 100644 --- a/README.md +++ b/README.md @@ -195,6 +195,10 @@ Please report issues or suggest improvements in the [GitHub Issues](https://gith ## ❔ Frequently asked questions +### Can inference be cancelled? + +Inference cancellation is cooperative. Low-level requests can be cancelled with `Request.Cancel()`, while streaming APIs provide language-specific cancellation mechanisms. Cancellation may not take effect until the current generation step completes, and inference requests do not currently have a built-in timeout. + ### Is Foundry Local a web server and CLI tool? No. Foundry Local is an **end-to-end local AI solution** that your application ships with. It handles model acquisition, hardware acceleration, and inference inside your app process through the SDK. The optional web server and CLI are available for development workflows, but the core product is the local AI runtime and SDK that you integrate directly into your application. diff --git a/sdk_v2/cpp/src/download/blob_downloader.cc b/sdk_v2/cpp/src/download/blob_downloader.cc index 9254a8b7f..4bb964afc 100644 --- a/sdk_v2/cpp/src/download/blob_downloader.cc +++ b/sdk_v2/cpp/src/download/blob_downloader.cc @@ -181,7 +181,8 @@ void AzureBlobDownloader::DownloadBlob(const std::string& sas_uri, // Single shared Azure context for the whole blob; calling Cancel() on it // propagates into every in-flight chunk read. - Azure::Core::Context azure_ctx; + auto azure_ctx = Azure::Core::Context{}.WithDeadline( + Azure::DateTime(std::chrono::system_clock::now() + std::chrono::hours{3})); // Internal cancel flag flipped by the orchestrator on first chunk failure // or by external cancellation; checked by workers between iterations. std::atomic internal_cancel{false}; @@ -409,7 +410,11 @@ void AzureBlobDownloader::DownloadBlob(const std::string& sas_uri, // All chunks done — sidecar is no longer needed. BlobDownloadState::DeleteState(local_path, logger_); } catch (const Azure::Core::OperationCancelledException&) { - FL_THROW(FOUNDRY_LOCAL_ERROR_OPERATION_CANCELLED, "download cancelled"); + if (cancelled && cancelled->load(std::memory_order_relaxed)) { + FL_THROW(FOUNDRY_LOCAL_ERROR_OPERATION_CANCELLED, "download cancelled"); + } + + FL_THROW(FOUNDRY_LOCAL_ERROR_NETWORK, "model download timed out after 3 hours"); } catch (const Azure::Core::RequestFailedException& e) { FL_THROW(FOUNDRY_LOCAL_ERROR_NETWORK, std::string("failed to download blob '") + blob_name + "': " + e.what()); diff --git a/sdk_v2/cs/README.md b/sdk_v2/cs/README.md index 253c6a8a0..b555e3e20 100644 --- a/sdk_v2/cs/README.md +++ b/sdk_v2/cs/README.md @@ -12,7 +12,7 @@ The Foundry Local C# SDK provides a .NET interface for running AI models locally - **Model variants** — select specific hardware/quantization variants per model alias - **Optional web service** — start an OpenAI-compatible REST endpoint (`/v1/chat_completions`, `/v1/models`) - **WinML acceleration** — built-in Windows hardware acceleration with automatic EP download -- **Full async/await** — every operation supports `CancellationToken` and async patterns +- **Async APIs** — idiomatic async/await support across the SDK - **IDisposable** — deterministic cleanup of native resources ## Installation