Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,10 @@ Please report issues or suggest improvements in the [GitHub Issues](https://gith

## ❔ Frequently asked questions

### Can inference be cancelled?

Inference cancellation is cooperative. Low-level requests can be cancelled with `Request.Cancel()`, while streaming APIs provide language-specific cancellation mechanisms. Cancellation may not take effect until the current generation step completes, and inference requests do not currently have a built-in timeout.

### Is Foundry Local a web server and CLI tool?

No. Foundry Local is an **end-to-end local AI solution** that your application ships with. It handles model acquisition, hardware acceleration, and inference inside your app process through the SDK. The optional web server and CLI are available for development workflows, but the core product is the local AI runtime and SDK that you integrate directly into your application.
Expand Down
9 changes: 7 additions & 2 deletions sdk_v2/cpp/src/download/blob_downloader.cc
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,8 @@ void AzureBlobDownloader::DownloadBlob(const std::string& sas_uri,

// Single shared Azure context for the whole blob; calling Cancel() on it
// propagates into every in-flight chunk read.
Azure::Core::Context azure_ctx;
auto azure_ctx = Azure::Core::Context{}.WithDeadline(
Azure::DateTime(std::chrono::system_clock::now() + std::chrono::hours{3}));
Comment thread
bmehta001 marked this conversation as resolved.
// Internal cancel flag flipped by the orchestrator on first chunk failure
// or by external cancellation; checked by workers between iterations.
std::atomic<bool> internal_cancel{false};
Expand Down Expand Up @@ -409,7 +410,11 @@ void AzureBlobDownloader::DownloadBlob(const std::string& sas_uri,
// All chunks done — sidecar is no longer needed.
BlobDownloadState::DeleteState(local_path, logger_);
} catch (const Azure::Core::OperationCancelledException&) {
FL_THROW(FOUNDRY_LOCAL_ERROR_OPERATION_CANCELLED, "download cancelled");
if (cancelled && cancelled->load(std::memory_order_relaxed)) {
FL_THROW(FOUNDRY_LOCAL_ERROR_OPERATION_CANCELLED, "download cancelled");
}

FL_THROW(FOUNDRY_LOCAL_ERROR_NETWORK, "model download timed out after 3 hours");
} catch (const Azure::Core::RequestFailedException& e) {
FL_THROW(FOUNDRY_LOCAL_ERROR_NETWORK,
std::string("failed to download blob '") + blob_name + "': " + e.what());
Expand Down
2 changes: 1 addition & 1 deletion sdk_v2/cs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ The Foundry Local C# SDK provides a .NET interface for running AI models locally
- **Model variants** — select specific hardware/quantization variants per model alias
- **Optional web service** — start an OpenAI-compatible REST endpoint (`/v1/chat_completions`, `/v1/models`)
- **WinML acceleration** — built-in Windows hardware acceleration with automatic EP download
- **Full async/await** — every operation supports `CancellationToken` and async patterns
- **Async APIs** — idiomatic async/await support across the SDK
- **IDisposable** — deterministic cleanup of native resources

## Installation
Expand Down
Loading