diff --git a/README.md b/README.md index c24a238..67b452d 100644 --- a/README.md +++ b/README.md @@ -6,18 +6,20 @@ Each library ships as its own DLL and NuGet package. There is no mandatory aggre ## Packages -| Package | Purpose | -|---|---| -| `FunctionFoundry.Security` | Envelope encryption, deterministic pseudonymization, threshold secret sharing, key-rotation planning | -| `FunctionFoundry.Storage` | Transactional file-set writes, content-addressed storage, content-defined chunking, Merkle file-tree diffs | -| `FunctionFoundry.Integrity` | Canonical JSON, hash manifests, Merkle proofs, tamper-evident hash chains | -| `FunctionFoundry.Observability` | Sensitive-data redaction, event fingerprinting, adaptive sampling, burst coalescing | -| `FunctionFoundry.Data` | External merge sort, structural tree diff, three-way merge, temporal interval join | -| `FunctionFoundry.Text` | Unicode spoof detection, secret scanning, near-duplicate indexing, delimited-text dialect inference | -| `FunctionFoundry.Resilience` | Adaptive concurrency, hedged execution, execution budgets, checkpointed batch execution | -| `FunctionFoundry.Networking` | Resumable parallel downloads, mirror selection, transfer planning, streaming integrity verification | -| `FunctionFoundry.Distributed` | Weighted rendezvous hashing, version clocks, phi-accrual failure detection, quorum aggregation | -| `FunctionFoundry.Scheduling` | Interval-set algebra, recurring availability, business calendars, critical-path scheduling | +Each product has a dedicated README under `src//README.md` describing exactly what it does. + +| Package | Purpose | README | +|---|---|---| +| `FunctionFoundry.Security` | Envelope encryption, deterministic pseudonymization, threshold secret sharing, key-rotation planning | [README](src/FunctionFoundry.Security/README.md) | +| `FunctionFoundry.Storage` | Transactional file-set writes, content-addressed storage, content-defined chunking, Merkle file-tree diffs | [README](src/FunctionFoundry.Storage/README.md) | +| `FunctionFoundry.Integrity` | Canonical JSON, hash manifests, Merkle proofs, tamper-evident hash chains | [README](src/FunctionFoundry.Integrity/README.md) | +| `FunctionFoundry.Observability` | Sensitive-data redaction, event fingerprinting, adaptive sampling, burst coalescing | [README](src/FunctionFoundry.Observability/README.md) | +| `FunctionFoundry.Data` | External merge sort, structural tree diff, three-way merge, temporal interval join | [README](src/FunctionFoundry.Data/README.md) | +| `FunctionFoundry.Text` | Unicode spoof detection, secret scanning, near-duplicate indexing, delimited-text dialect inference | [README](src/FunctionFoundry.Text/README.md) | +| `FunctionFoundry.Resilience` | Adaptive concurrency, hedged execution, execution budgets, checkpointed batch execution | [README](src/FunctionFoundry.Resilience/README.md) | +| `FunctionFoundry.Networking` | Resumable parallel downloads, mirror selection, transfer planning, streaming integrity verification | [README](src/FunctionFoundry.Networking/README.md) | +| `FunctionFoundry.Distributed` | Weighted rendezvous hashing, version clocks, phi-accrual failure detection, quorum aggregation | [README](src/FunctionFoundry.Distributed/README.md) | +| `FunctionFoundry.Scheduling` | Interval-set algebra, recurring availability, business calendars, critical-path scheduling | [README](src/FunctionFoundry.Scheduling/README.md) | ## Requirements diff --git a/src/FunctionFoundry.Data/README.md b/src/FunctionFoundry.Data/README.md new file mode 100644 index 0000000..b5af408 --- /dev/null +++ b/src/FunctionFoundry.Data/README.md @@ -0,0 +1,66 @@ +# FunctionFoundry.Data + +Bounded-memory and conflict-aware data processing for hard cases the BCL does not solve as a reusable package. + +## What this product does + +It handles **large sorts, structured diffs, merges with conflicts, and temporal overlaps** with explicit limits and deterministic outputs. + +### 1. Bounded-memory external merge sort (`BoundedMemoryExternalMergeSort`) + +Sorts generic records that do not fit in memory: + +* configurable memory budget +* temporary run files and k-way merge +* optional stable ordering +* cancellation and cleanup after success, cancel, or failure +* diagnostics for abandoned temporary data + +### 2. Structural tree diff (`StructuralTreeDiff`) + +Diffs nested objects, arrays, and scalars into ordered operations: + +* add / remove / replace / move +* configurable identity keys for array elements +* complexity limits against hostile inputs +* human-readable and machine-readable results + +### 3. Three-way merge (`ThreeWayMerge`) + +Merges **base + local + remote** values with: + +* explicit conflict objects (no silent conflict loss) +* configurable merge policies +* path-level conflict reporting +* deterministic output ordering + +### 4. Temporal interval join (`TemporalIntervalJoin`) + +Joins records whose validity intervals overlap: + +* open / closed boundary policies +* sorted-input friendly processing +* bounded buffering where possible +* clear handling of invalid intervals + +## When to use it + +* Offline ETL or analytics sorts larger than RAM +* Document / config sync that must show structural changes +* CRDT-adjacent or collaborative edits needing honest conflicts +* Validity-window joins (pricing, entitlements, employment periods) + +## Non-goals + +* A general query engine or ORM +* Silently “winning” merges by last-write-wins defaults + +## Install + +```bash +dotnet add package FunctionFoundry.Data +``` + +## Runtime dependencies + +None beyond the .NET shared framework. diff --git a/src/FunctionFoundry.Distributed/README.md b/src/FunctionFoundry.Distributed/README.md new file mode 100644 index 0000000..49d0180 --- /dev/null +++ b/src/FunctionFoundry.Distributed/README.md @@ -0,0 +1,68 @@ +# FunctionFoundry.Distributed + +Reusable distributed-systems algorithms without requiring a network framework. + +## What this product does + +It provides **membership hashing, causality tracking, failure suspicion, and quorum aggregation** as pure in-process building blocks. + +### 1. Weighted rendezvous hashing (`WeightedRendezvousHasher`) + +Picks stable nodes for keys with: + +* node weights +* optional bounded-load behavior +* deterministic hashing +* minimal remapping when membership changes + +### 2. Vector and dotted version clocks (`VectorClock` / `DottedVersionClock`) + +Tracks causal history across replicas: + +* increment, merge, dominance comparison +* concurrent-version detection +* deterministic serialization +* configurable growth limits and compaction trade-offs + +### 3. Phi-accrual failure detector (`PhiAccrualFailureDetector`) + +Computes a continuous **suspicion score** from heartbeat intervals: + +* sample window and warm-up behavior +* injectable clock abstraction +* outlier handling +* deterministic simulations for tests + +Suspicion is **not** proof of failure. + +### 4. Quorum result aggregator (`QuorumResultAggregator`) + +Collects concurrent replica results under a quorum policy: + +* agreement and conflict detection +* early completion +* failure evidence +* cancellation of unnecessary work +* deterministic winner selection only when the policy allows it + +## When to use it + +* Client-side shard / replica selection +* Conflict detection for multi-writer records +* Heartbeat-based soft failure detection +* Reading from N replicas and waiting for quorum + +## Non-goals + +* Shipping a cluster membership protocol or RPC stack +* Treating phi scores as hard failovers without host policy + +## Install + +```bash +dotnet add package FunctionFoundry.Distributed +``` + +## Runtime dependencies + +None beyond the .NET shared framework. diff --git a/src/FunctionFoundry.Integrity/README.md b/src/FunctionFoundry.Integrity/README.md new file mode 100644 index 0000000..57ab25c --- /dev/null +++ b/src/FunctionFoundry.Integrity/README.md @@ -0,0 +1,60 @@ +# FunctionFoundry.Integrity + +Deterministic canonicalization and verifiable data structures. + +## What this product does + +It makes data **comparable and verifiable** across machines by fixing encoding rules and building cryptographic evidence structures. + +### 1. Canonical JSON (`CanonicalJson`) + +Converts JSON into a **documented deterministic profile**: + +* sorted object properties +* exact number formatting rules +* Unicode / escape handling +* duplicate-property rejection +* streaming canonicalize where feasible + +Identical logical documents produce identical bytes, which is required for hashing and signing. + +### 2. Streaming hash manifest (`StreamingHashManifest`) + +Hashes many files or streams into a deterministic manifest: + +* SHA-256 (SHA-384 optional) +* path traversal protection +* explicit metadata inclusion policy +* verification reports that list exact mismatches + +### 3. Merkle tree and inclusion proofs (`MerkleTree`) + +Builds a Merkle tree with domain-separated leaf/internal nodes, odd-node handling, and inclusion proof generate/verify — so you can prove one item belongs to a set without shipping the whole set. + +### 4. Tamper-evident hash chain (`HashChain`) + +Append-only record chain with canonical encoding and verification that identifies the **first invalid record**. Sequence and timestamp policies are explicit. + +**Warning:** a hash chain alone is not trusted timestamping or external anchoring. + +## When to use it + +* Signing / hashing APIs that need stable JSON bytes +* Release or dataset integrity manifests +* Light-weight audit logs that detect truncation or mutation +* Inclusion claims for large collections + +## Non-goals + +* Replacing a blockchain or TSA +* Generic JSON helpers unrelated to determinism + +## Install + +```bash +dotnet add package FunctionFoundry.Integrity +``` + +## Runtime dependencies + +None beyond the .NET shared framework. diff --git a/src/FunctionFoundry.Networking/README.md b/src/FunctionFoundry.Networking/README.md new file mode 100644 index 0000000..9e6efd4 --- /dev/null +++ b/src/FunctionFoundry.Networking/README.md @@ -0,0 +1,68 @@ +# FunctionFoundry.Networking + +Robust large-data transfer orchestration over HTTP using a **caller-provided** `HttpClient`. + +## What this product does + +It plans and executes **resumable, integrity-aware downloads** with mirror selection and streaming verification. + +### 1. Resumable parallel downloader (`ResumableParallelDownloader`) + +Downloads large objects using HTTP range requests when available: + +* ETag / Last-Modified validation +* persistent checkpoint format +* configurable parallelism +* partial-file integrity and final cryptographic verification +* safe restart after interruption +* capability detection and correct fallback when ranges are unsupported + +Never creates a new `HttpClient` per request. + +### 2. Mirror selector (`MirrorSelector`) + +Scores candidate mirrors using latency, throughput, availability, and integrity failures: + +* failure decay over time +* hysteresis to avoid rapid oscillation +* deterministic tie-breaking +* exportable selection evidence + +### 3. Transfer plan (`TransferPlan`) + +Generates chunk assignments independently from execution: + +* resumable state +* incompatible checkpoint detection +* no overlapping writes +* full-coverage verification +* repair assignments after corrupt chunks + +### 4. Streaming integrity verifier (`StreamingIntegrityVerifier`) + +Verifies data while it streams: + +* per-chunk and complete-object checks +* detailed mismatch location +* bounded memory (no full-file buffering) + +## When to use it + +* IDE/plugin/runtime update downloaders +* Multi-CDN artifact fetchers +* Resumable media or dataset transfers with integrity requirements + +## Non-goals + +* Owning `HttpClient` lifetime / DNS / proxy policy for the host +* A general REST client library + +## Install + +```bash +dotnet add package FunctionFoundry.Networking +``` + +## Runtime dependencies + +None beyond the .NET shared framework (`HttpClient` supplied by the caller). diff --git a/src/FunctionFoundry.Observability/README.md b/src/FunctionFoundry.Observability/README.md new file mode 100644 index 0000000..e05aa38 --- /dev/null +++ b/src/FunctionFoundry.Observability/README.md @@ -0,0 +1,60 @@ +# FunctionFoundry.Observability + +Advanced processing for structured logs and diagnostics — without forcing a logging framework. + +## What this product does + +It transforms diagnostic events for **privacy, grouping, and volume control** before they hit expensive storage or sinks. + +### 1. Sensitive-data redactor (`SensitiveDataRedactor`) + +Walks nested objects, dictionaries, and collections (with cycle protection) and redacts: + +* property-name policies (`password`, `token`, …) +* pattern-based values +* high-entropy secret candidates + +Supports replacement strategies and hard limits on depth / item counts so inspection stays bounded. Designed to avoid arbitrary user-code invocation where practical. + +### 2. Deterministic event fingerprinting (`EventFingerprinter`) + +Normalizes exception stacks and strips unstable values (timestamps, GUIDs, correlation noise) while keeping causal structure. Equivalent failures get a **stable fingerprint** plus an explanation of which normalized parts contributed. + +### 3. Adaptive deterministic sampling (`AdaptiveSampler`) + +Decides whether to keep an event based on fingerprint and observed volume: + +* always preserves high-severity events +* identical fingerprints get stable decisions +* adapts rates under high cardinality / volume +* bounded memory, thread-safe, exportable stats + +### 4. Burst coalescing (`BurstCoalescer`) + +Merges repeated events inside a time window: + +* keeps first and last occurrence +* keeps representative samples +* emits dropped / coalesced counts +* handles concurrent producers without unbounded cardinality growth + +## When to use it + +* Log pipelines that must strip secrets before export +* Alert grouping / error fingerprinting across instances +* Cost control when a single bug floods telemetry + +## Non-goals + +* Being a full logging framework (Serilog/MEL replacement) +* Guaranteeing zero secret leakage for all custom objects + +## Install + +```bash +dotnet add package FunctionFoundry.Observability +``` + +## Runtime dependencies + +None beyond the .NET shared framework. Logging-framework-neutral by design. diff --git a/src/FunctionFoundry.Resilience/README.md b/src/FunctionFoundry.Resilience/README.md new file mode 100644 index 0000000..db0f3b4 --- /dev/null +++ b/src/FunctionFoundry.Resilience/README.md @@ -0,0 +1,70 @@ +# FunctionFoundry.Resilience + +Adaptive execution mechanisms that go beyond ordinary retries, timeouts, and circuit breakers. + +## What this product does + +It controls **how much work runs, when secondary copies start, how shared deadlines are split, and how large batches resume safely**. + +### 1. Adaptive concurrency controller (`AdaptiveConcurrencyController`) + +AIMD-style concurrency regulation driven by latency and error feedback: + +* configurable min/max concurrency +* warm-up behavior +* thread-safe acquisition and queue limits +* exportable state snapshots +* deterministic simulation hooks for tests + +### 2. Hedged execution (`HedgedExecution`) + +Starts secondary attempts based on rolling latency percentiles: + +* shared cancellation; first valid result wins +* failure aggregation +* maximum amplification limits +* idempotency warnings and metrics for extra work + +Use only when the operation is safe to duplicate. + +### 3. Execution budget (`ExecutionBudget`) + +Allocates one overall deadline across multiple steps: + +* reserve cleanup time +* rebalance unused budget +* reject impossible / negative allocations +* nested budgets +* monotonic time sources + +### 4. Checkpointed batch executor (`CheckpointedBatchExecutor`) + +Processes large batches with bounded concurrency: + +* persists successful item checkpoints through an abstraction +* resumes safely after interruption +* records partial failures +* requires deterministic item ids and explicit idempotency +* no hidden infinite retries + +## When to use it + +* Client libraries calling overloaded backends +* Fan-out calls where hedging reduces tail latency +* Multi-step workflows with a hard end-to-end deadline +* Overnight batches that must restart mid-way + +## Non-goals + +* Replacing Polly-style basic retry / circuit-breaker APIs +* Making non-idempotent APIs safe to hedge automatically + +## Install + +```bash +dotnet add package FunctionFoundry.Resilience +``` + +## Runtime dependencies + +None beyond the .NET shared framework. diff --git a/src/FunctionFoundry.Scheduling/README.md b/src/FunctionFoundry.Scheduling/README.md new file mode 100644 index 0000000..801ec5b --- /dev/null +++ b/src/FunctionFoundry.Scheduling/README.md @@ -0,0 +1,67 @@ +# FunctionFoundry.Scheduling + +Advanced interval, calendar, and dependency scheduling. + +## What this product does + +It computes **when things can happen** using interval algebra, calendars, recurring availability, and critical-path analysis. + +### 1. Interval-set algebra (`IntervalSetAlgebra`) + +Exact set operations on intervals: + +* union, intersection, difference +* complement within a boundary +* open and closed endpoints +* normalization and deterministic ordering +* efficient handling of large sorted interval sets + +### 2. Recurring availability resolver (`RecurringAvailabilityResolver`) + +Finds meeting / slot candidates across participants: + +* time zones and DST transitions +* recurring local-time windows and exceptions +* minimum duration constraints +* ranked candidate windows +* explicit policies for ambiguous / invalid local times + +### 3. Business calendar (`BusinessCalendar`) + +Working-time math without a hard-coded global holiday database: + +* working days and split working hours +* holidays / closures supplied as region-independent inputs +* add or subtract working duration +* explainable calculation path + +### 4. Critical-path scheduler (`CriticalPathScheduler`) + +Schedules dependency graphs: + +* cycle validation with an actual reported cycle +* earliest / latest start, slack, and critical path +* optional resource-capacity heuristic +* clearly marks when a heuristic result is not globally optimal + +## When to use it + +* Calendar math for SLAs and maintenance windows +* Multi-timezone meeting finders +* Project / workflow planning with dependencies +* Working-hours deadline calculation + +## Non-goals + +* Embedding a complete worldwide holiday catalog +* Guaranteeing globally optimal resource leveling when heuristics are used + +## Install + +```bash +dotnet add package FunctionFoundry.Scheduling +``` + +## Runtime dependencies + +None beyond the .NET shared framework. diff --git a/src/FunctionFoundry.Security/README.md b/src/FunctionFoundry.Security/README.md new file mode 100644 index 0000000..64b1b0f --- /dev/null +++ b/src/FunctionFoundry.Security/README.md @@ -0,0 +1,54 @@ +# FunctionFoundry.Security + +Cryptographic orchestration built on BCL primitives — not a cipher invention kit, and not basic hashing wrappers. + +## What this product does + +It helps you **protect, pseudonymize, split, and re-key sensitive data** with explicit formats and safety rules. + +### 1. Versioned envelope encryption (`EnvelopeEncryptor`) + +Encrypts payloads with **AES-256-GCM** into a versioned binary envelope that carries: + +* a key identifier +* associated authenticated data (AAD) +* nonce, auth tag, and ciphertext + +Decrypt resolves the key by id through `IDataKeyResolver`. Authentication failures **never return partial plaintext**. Input size limits reject oversized or truncated envelopes. + +### 2. Deterministic pseudonymization (`Pseudonymizer`) + +Produces stable HMAC-based tokens with **tenant + context domain separation** and key versioning. Useful for irreversible-looking identifiers when the clear value is known at generation time. + +**Important:** this is **not encryption**. Anyone with the HMAC key can regenerate or verify tokens for known inputs. + +### 3. Threshold secret sharing (`SecretSharer`) + +Shamir-style sharing over **GF(256)**. Split a secret into *n* shares with threshold *t*; any *t* shares reconstruct it. Includes share validation, duplicate detection, and binary serialization. + +### 4. Key rotation planning (`KeyRotationPlanner`) + +Inspects envelope metadata and builds a **deterministic plan** of which payloads still need re-encryption under a target key. It never silently deletes old ciphertext. + +## When to use it + +* Application-level envelope crypto with key rotation +* Privacy-preserving stable identifiers (pseudonyms) +* Split custody of high-value secrets +* Auditable migration when keys change + +## Non-goals + +* Inventing new ciphers or modes +* HSM / key-distribution services +* Claiming formal security proofs + +## Install + +```bash +dotnet add package FunctionFoundry.Security +``` + +## Runtime dependencies + +None beyond the .NET shared framework. diff --git a/src/FunctionFoundry.Storage/README.md b/src/FunctionFoundry.Storage/README.md new file mode 100644 index 0000000..2dae708 --- /dev/null +++ b/src/FunctionFoundry.Storage/README.md @@ -0,0 +1,63 @@ +# FunctionFoundry.Storage + +Reliable file and object workflows beyond ordinary `File` / `Directory` APIs. + +## What this product does + +It provides **transactional multi-file writes, content-addressed storage, content-defined chunking, and deterministic directory comparison**. + +### 1. Transactional file-set writer (`TransactionalFileSetWriter`) + +Creates or replaces **multiple files as one logical operation**: + +* staging directory +* write-ahead journal +* per-file checksums +* atomic replace where the OS supports it (documented fallback otherwise) +* rollback and **crash recovery** with deterministic recovery reports +* cancellation without undocumented half-state + +### 2. Content-addressed object store (`ContentAddressedStore`) + +Streams blobs into a hash-addressed layout (SHA-256): + +* deduplication by content hash +* integrity verification +* safe concurrent writers (temp + rename) +* orphan detection +* garbage-collection **planning** without automatic destructive deletion + +### 3. Content-defined chunking (`ContentDefinedChunker`) + +Rabin-style rolling-hash chunking with configurable min / target / max sizes. Streams input with bounded memory so similar content tends to share chunk boundaries. + +### 4. Merkle file-tree snapshot and diff (`MerkleFileTree`) + +Builds deterministic directory snapshots and diffs: + +* added / removed / modified / moved detection +* content hashing and metadata policies +* symlink policy and cycle protection +* cross-platform path normalization + +## When to use it + +* Installers, exporters, or config deployers that must not leave partial file sets +* Local caches or artifact stores with dedup +* Backup/dedup pipelines needing stable chunk boundaries +* Sync tools that need a deterministic tree diff + +## Non-goals + +* Thin wrappers around `File.WriteAllText` or `FileStream` +* A full distributed object store or cloud SDK + +## Install + +```bash +dotnet add package FunctionFoundry.Storage +``` + +## Runtime dependencies + +None beyond the .NET shared framework. diff --git a/src/FunctionFoundry.Text/README.md b/src/FunctionFoundry.Text/README.md new file mode 100644 index 0000000..64eac37 --- /dev/null +++ b/src/FunctionFoundry.Text/README.md @@ -0,0 +1,65 @@ +# FunctionFoundry.Text + +Complex text analysis for spoof detection, secret hunting, near-duplicates, and CSV/TSV dialect discovery. + +## What this product does + +It analyzes text for **risk signals and similarity**, with explainable results and hard processing limits. + +### 1. Unicode spoof / confusable detection (`UnicodeSpoofDetector`) + +Finds lookalike and mixed-script abuse patterns: + +* Unicode normalization checks +* confusable skeleton generation (versioned compact dataset) +* mixed-script detection +* invisible / control-character findings +* explainable risk findings + +Does **not** claim that a heuristic result proves malicious intent. + +### 2. Secret-candidate scanner (`SecretCandidateScanner`) + +Scans large text for likely secrets: + +* high-entropy tokens +* configurable patterns +* context-aware confidence scores and false-positive controls +* streaming support with line/column reporting +* **redacted findings by default** + +### 3. Near-duplicate text index (`NearDuplicateTextIndex`) + +Indexes documents with MinHash / LSH-style techniques: + +* configurable similarity thresholds +* incremental indexing +* deterministic seeds +* candidate retrieval plus exact similarity verification +* documented memory / accuracy trade-offs + +### 4. Delimited-text dialect inference (`DelimitedTextDialectInferrer`) + +Infers delimiter, quote, escape, newline, and header likelihood from samples. Returns **ranked candidates** with confidence and evidence — never silently claims certainty. Sample size and complexity are limited. + +## When to use it + +* Homograph / phishing username checks +* Pre-commit or CI secret scanning of text blobs +* Deduplicating support tickets, docs, or logs +* Auto-detecting messy CSV dialects before parsing + +## Non-goals + +* Full OCR or NLP pipelines +* Guaranteeing zero secret false positives + +## Install + +```bash +dotnet add package FunctionFoundry.Text +``` + +## Runtime dependencies + +None beyond the .NET shared framework.