diff --git a/specs/003-pool-warmup/spec.md b/specs/003-pool-warmup/spec.md new file mode 100644 index 0000000000..8c41b88470 --- /dev/null +++ b/specs/003-pool-warmup/spec.md @@ -0,0 +1,95 @@ +# Feature Specification: Background Pool Warmup + +**Feature Branch**: `dev/mdaigle/pool-warmup` (branched from `dev/mdaigle/pool-channel-rate-limiting`) +**Created**: 2026-07-16 +**Status**: Draft + +## Description + +Implement background pool warmup for `ChannelDbConnectionPool`. When the pool starts, it asynchronously pre-creates connections up to Min Pool Size in the background, serially (one at a time), through the shared rate-limiting mechanism used by user-initiated requests. + +Whenever the pool count drops below Min Pool Size for any reason (a connection destroyed on return, idle-timeout eviction, or any other removal that crosses the floor), the pool automatically triggers replenishment using the same serial, rate-limited creation path. Pruning itself respects the Min Pool Size floor and never reduces the pool below the minimum; it is only mentioned here because it shares the same removal choke point that triggers replenishment. + +## User Scenarios & Testing + +### User Story 1 — Background Warmup on Pool Creation (P1) + +When a pool is created with Min Pool Size > 0, it pre-creates connections in the background so early application requests find ready-to-use connections. + +**Acceptance Scenarios**: + +1. **Given** a pool is created with minimum pool size of N, **When** startup completes, **Then** the pool begins creating connections in the background up to N. +2. **Given** a pool is warming up, **When** a user opens a connection before warmup finishes, **Then** the user request is served immediately without waiting for warmup to complete. +3. **Given** a pool is warming up, **When** warmup creates each connection, **Then** each connection is created one at a time (serially), not in parallel. + +--- + +### User Story 2 — Warmup Through Shared Rate Limiter (P1) + +Warmup creates connections through the same rate-limiting mechanism as user-initiated requests, preventing warmup from overwhelming the server. + +**Acceptance Scenarios**: + +1. **Given** warmup is creating connections, **When** a user request also needs a new connection, **Then** both warmup and user requests compete fairly through the shared rate limiter. +2. **Given** the rate limiter is at capacity, **When** warmup attempts to create the next connection, **Then** warmup does not bypass the limiter: it ends the current pass without creating (rather than spinning or waiting on a permit). Saturation only occurs while user requests are actively creating connections, which fill the pool themselves; any later drop below the minimum re-triggers warmup once capacity has freed up. + +--- + +### User Story 3 — Warmup Failure Resilience (P1) + +If a connection fails to open during warmup, the failure is silently absorbed on the warmup loop — it is never propagated to the caller as an unhandled exception. Warmup participates in the pool's blocking-period error state exactly like an on-demand creation (mirroring the legacy WaitHandle pool): a genuine open failure enters the error state, and a successful open clears it. The pool remains operational — once the blocking period expires, user requests create connections on demand as needed. + +**Acceptance Scenarios**: + +1. **Given** warmup is in progress, **When** a connection fails to open, **Then** the failure is traced/logged and absorbed by the warmup loop rather than surfacing as an unhandled exception. +2. **Given** warmup fails and has entered the blocking-period error state, **When** a user opens a connection during the blocking window, **Then** the user request fast-fails with the cached exception (matching the WaitHandle pool); once the blocking period expires it creates a connection on demand and succeeds normally. +3. **Given** warmup's open fails, **When** the pool's error state is checked, **Then** the pool IS in the blocking-period error state — warmup uses the same shared creation path as user requests and enters/clears that state identically. +4. **Given** the pool is already in the blocking-period error state (driven there by a failing request), **When** warmup would otherwise replenish, **Then** warmup stands down while the error state is active rather than piling more doomed opens onto a struggling server (the warmup loop skips replenishment while `ErrorOccurred`, mirroring the WaitHandle pool). + +--- + +### User Story 4 — Warmup Cancellation on Shutdown (P2) + +When a pool is shut down while warmup is still in progress, warmup stops promptly. No new connections are created after shutdown begins. + +**Acceptance Scenarios**: + +1. **Given** warmup is in progress, **When** the pool is shut down, **Then** warmup stops and no further connections are created. +2. **Given** warmup is in progress, **When** the pool is shut down, **Then** any connections already created by warmup are cleaned up as part of normal shutdown. + +--- + +### User Story 5 — Replenishment on Any Below-Minimum Event (P2) + +Whenever the pool count drops below Min Pool Size, the pool automatically triggers warmup-style replenishment to restore to the minimum. Every connection removal funnels through a single choke point (`RemoveConnection`), which triggers replenishment when the removal leaves the pool below the minimum. + +**Acceptance Scenarios**: + +1. **Given** a pruning cycle runs, **When** it removes idle connections, **Then** it stops at the Min Pool Size floor and does not reduce the pool below the minimum (pruning shares the removal choke point but does not itself cause a below-minimum replenishment). +2. **Given** a connection is destroyed on return to the pool (broken, lifetime expired), **When** the pool count drops below the minimum, **Then** the pool queues a replenishment request. +3. **Given** idle-timeout eviction removes an expired connection on retrieval, **When** the pool count drops below minimum, **Then** the pool queues a replenishment request. +4. **Given** replenishment is triggered, **When** connections are created, **Then** they are created serially through the shared rate limiter, identical to initial warmup. +5. **Given** the pool is at or above the minimum pool size, **When** a connection is destroyed, **Then** no replenishment is triggered. + +## Implementation Notes + +- Warmup starts automatically when the pool's `Startup()` is called. +- Warmup runs on a background task, so it never blocks the caller (`Startup`/connection return) and uses no sync-over-async in the loop. The physical connection open itself is currently synchronous (executed on the background task); the connection factory does not yet expose an async open. +- Creates connections serially (one at a time) through the shared rate limiter. If the limiter is saturated (no permit available), warmup ends the pass rather than bypassing or spinning on the limiter; convergence to the minimum is handled by the concurrent user creations that saturated it and by re-triggering on the next below-minimum event. +- Warmup failures are traced/logged and absorbed by the warmup loop (never surfaced as an unhandled exception). Warmup goes through the same shared creation path as user requests, so a genuine open failure enters the blocking-period error state and a successful open clears it (mirroring the WaitHandle pool). While the error state is active, the warmup loop stands down rather than replenishing. +- If a `Clear` races with an in-flight warmup creation, the freshly created (now stale-generation) connection is not special-cased by warmup; it is harmlessly discarded by the liveness/generation check on its next retrieval. After a `Clear`, replenishment refills the pool to the minimum with fresh-generation connections. +- Idle-timeout eviction is lazy: an idle connection past its timeout is removed when it is next retrieved (via the liveness check), not by a background sweep. Pruning is separate and always respects the Min Pool Size floor. +- Concurrent warmup/replenishment requests are coalesced — a single `_warmupLoopRunning` guard ensures only one warmup loop executes at a time, and requests that arrive while it is running are dropped (the running loop re-reads the pool count each iteration and drives to the minimum on its own). +- Warmup is a no-op when Min Pool Size = 0. + +## Acceptance Criteria + +- When a pool starts with Min Pool Size > 0, background warmup begins immediately. +- After warmup completes, the pool contains Min Pool Size idle connections. +- Warmup runs on a background task without blocking callers and without sync-over-async in the loop (the physical open is currently synchronous, executed on the background task, pending async-open support in the factory). +- Warmup connections are created serially through the shared rate limiter. +- Warmup errors are traced/logged and never crash the application. Because warmup shares the on-demand creation path, a genuine open failure enters the pool's blocking-period error state (as it would for a user request) and a successful open clears it. +- If the pool is shut down or cleared during warmup, warmup stops gracefully. +- After any event that reduces the pool below Min Pool Size, the pool restores to the minimum through replenishment: a single warmup pass suffices when it is not racing a concurrent drain (e.g. `Clear`) or a saturated rate limiter; otherwise convergence completes across subsequent re-triggered passes. +- Concurrent warmup/replenishment requests are coalesced so only one warmup loop executes at a time. +- Unit tests validate warmup behavior for various Min Pool Size values (0, 1, N) and all replenishment triggers. diff --git a/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/ConnectionPool/ChannelDbConnectionPool.cs b/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/ConnectionPool/ChannelDbConnectionPool.cs index 298c38b9db..a948d04844 100644 --- a/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/ConnectionPool/ChannelDbConnectionPool.cs +++ b/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/ConnectionPool/ChannelDbConnectionPool.cs @@ -127,6 +127,23 @@ internal sealed class ChannelDbConnectionPool : IDbConnectionPool, IDisposable /// this pool group. See . /// private readonly BlockingPeriodErrorState? _errorState; + + /// + /// Cancels in-flight background warmup/replenishment when the pool shuts down so no new + /// connections are created after shutdown begins (Story 4). The token is observed at the + /// warmup loop's await points and before each creation attempt, so it stops the loop and + /// prevents further attempts promptly. It cannot abort an already in-progress physical open: + /// that path is synchronous and the connection factory does not yet accept a cancellation + /// token (see the TODO in ). + /// + private readonly CancellationTokenSource _warmupCts = new(); + + /// + /// Coalescing guard ensuring only one warmup loop executes at a time. Transitioned + /// 0 -> 1 via by the first + /// requester to start the loop, and reset to 0 by the loop when it drains. + /// + private int _warmupLoopRunning; #endregion /// @@ -221,6 +238,16 @@ public ConcurrentDictionary< private uint MaxPoolSize { get; } private int MinPoolSize => PoolGroupOptions.MinPoolSize; + + /// + /// The most recently launched warmup/replenishment loop task, exposed so tests can await a + /// warmup pass to a deterministic completion instead of polling pool counters. May be null + /// (warmup never requested) or reference an already-completed pass (requests are coalesced); + /// a caller that only needs "some warmup pass has finished" can await it regardless. Only + /// consumed by unit tests, which read it after synchronously triggering warmup on their own + /// thread, so a plain auto-property is sufficient. + /// + internal Task? WarmupLoopTask { get; private set; } #endregion #region Methods @@ -349,6 +376,26 @@ public void Shutdown() // routes returning connections to RemoveConnection. State = ShuttingDown; + // Cancel any in-flight background warmup/replenishment so no new connections are + // created after shutdown begins (Story 4). The warmup loop also observes the + // State transition above; cancelling here stops the loop promptly by tripping its + // await points and pre-create checks. It does not abort an already in-progress + // synchronous physical open (see _warmupCts field docs). Cancel is idempotent. + try + { + _warmupCts.Cancel(); + } + catch (ObjectDisposedException) + { + // Expected no-op: the cancellation source is already disposed, so there is nothing + // left to cancel. This is not a failure, so it is not traced. + } + catch (Exception ex) + { + SqlClientEventSource.Log.TryPoolerTraceEvent( + " {0}, _warmupCts.Cancel threw, continuing shutdown: {1}", Id, ex); + } + // Each cleanup step is independent and best-effort. A failure in one step must not // prevent later steps from running, otherwise the pool can be left half-shut-down // (e.g. timer disposed but channel never completed -> waiters stuck forever). @@ -422,6 +469,23 @@ public void Shutdown() " {0}, RemoveConnection threw during drain, continuing: {1}", Id, ex); } } + + // Release the warmup cancellation source now that the loop has been signalled to stop + // and no further connections will be created. Dispose is best-effort and idempotent. + try + { + _warmupCts.Dispose(); + } + catch (ObjectDisposedException) + { + // Expected no-op: the cancellation source is already disposed. Dispose is documented + // as safe to call repeatedly, so this is not a failure and is not traced. + } + catch (Exception ex) + { + SqlClientEventSource.Log.TryPoolerTraceEvent( + " {0}, _warmupCts.Dispose threw, continuing shutdown: {1}", Id, ex); + } } /// @@ -432,14 +496,16 @@ public void Shutdown() /// public void Startup() { - // Startup is currently a no-op for this pool: State is set to Running in the - // constructor, and PoolPruner (when present, i.e. MinPoolSize < MaxPoolSize) is - // also constructed eagerly there; its timer arms/disarms via UpdateTimer() calls - // from OpenNewInternalConnection and RemoveConnection as the pool grows/shrinks. - // This method exists as the symmetrical counterpart of Shutdown and as a hook - // for future warmup behavior. + // State is set to Running in the constructor, and PoolPruner (when present, i.e. + // MinPoolSize < MaxPoolSize) is also constructed eagerly there; its timer arms/disarms + // via UpdateTimer() calls from OpenNewInternalConnection and RemoveConnection as the + // pool grows/shrinks. SqlClientEventSource.Log.TryPoolerTraceEvent( " {0}", Id); + + // Kick off background warmup so the pool pre-creates connections up to MinPoolSize + // without blocking the caller (Story 1). No-op when MinPoolSize == 0. + RequestWarmup(); } /// @@ -571,7 +637,11 @@ public bool TryGetConnection( { cancellationToken.ThrowIfCancellationRequested(); - // Fast-fail in the error state. FR-006. + // Fast-fail if the pool is in the blocking-period error state. FR-006. Warmup goes + // through this same path (it has no isWarmup exemption): it mirrors the legacy WaitHandle + // pool, whose replenishment enters/clears the same error state as user requests. In + // practice the warmup loop already stands down before reaching here (its loop condition + // checks ErrorOccurred); this covers the narrow race where the state flips in between. _errorState?.ThrowIfActive(); try @@ -688,8 +758,9 @@ _connectionCreationRateLimiter is not null && // start the pruning timer so idle connections can be reclaimed. Pruner?.UpdateTimer(); - // A successful creation clears error/backoff state - // FR-009. + // A successful creation clears error/backoff state (FR-009). Warmup goes through + // this same path and clears the state on success too, mirroring the legacy + // WaitHandle pool: a connection that opens proves the server is reachable. _errorState?.Clear(); } @@ -697,13 +768,27 @@ _connectionCreationRateLimiter is not null && } catch (Exception ex) when (ADP.IsCatchableExceptionType(ex) && ex is not OperationCanceledException) { - // Enter the blocking period error state on creation failure if configured. + // Enter the blocking period error state on creation failure if configured. Warmup + // goes through this same path (the warmup loop absorbs the rethrow in its own catch), + // mirroring the legacy WaitHandle pool, whose replenishment failures also enter the + // error state. + // // We deliberately exclude OperationCanceledException: that is thrown when the // caller's own timeout/cancellation budget expires while waiting, which is // client-side contention rather than a physical connection creation failure and // must not poison the pool into fast-fail/backoff for other callers. // FR-006, FR-007. - _errorState?.Enter(ex); + // + // Only enter the error state while the pool is still Running. A synchronous physical + // open cannot be cancelled once it is in progress (see _warmupCts field docs), so a + // warmup/user open that began before Shutdown can complete with a failure after + // Shutdown has already disposed _errorState. Entering here would re-arm the exit + // timer and keep callbacks/logging alive past teardown, so we skip it once shutdown + // has begun; the throw below still propagates the failure to the caller/warmup loop. + if (State == Running) + { + _errorState?.Enter(ex); + } throw; } @@ -770,6 +855,14 @@ private void RemoveConnection(DbConnectionInternal connection) // If this removal brought us back to MinPoolSize, disable the pruning timer. Pruner?.UpdateTimer(); + + // Any removal that drops the pool below MinPoolSize triggers replenishment through the + // shared serial, rate-limited warmup path (Story 5). This is the single choke point for + // every below-minimum event: connections destroyed on return (broken/lifetime), idle + // timeout eviction, and pruning all funnel through RemoveConnection. RequestWarmup is a + // no-op when MinPoolSize == 0, the pool is not Running, or a warmup loop is already + // running/queued, so calling it unconditionally here is cheap. + RequestWarmup(); } /// @@ -962,6 +1055,189 @@ private void ValidateOwnershipAndSetPoolingState(DbConnectionInternal connection } #endregion + #region Warmup + /// + /// Requests background warmup/replenishment: the pool asynchronously pre-creates connections + /// up to , serially and through the shared rate limiter. Safe to + /// call from any thread and from hot paths; it is cheap and non-blocking. + /// + /// This is the single entry point for every warmup trigger: pool startup and any event that + /// drops the pool below (connection destruction on return, idle + /// timeout eviction, pruning). It is also invoked directly by tests to exercise the loop + /// deterministically. Concurrent requests are coalesced by the + /// guard so that only one warmup loop ever runs at a time: a + /// request that arrives while a loop is already running is simply dropped, because that loop + /// re-reads on every iteration and will drive the pool to + /// regardless. + /// + internal void RequestWarmup() + { + // No-op when there is nothing to pre-create (MinPoolSize == 0), the pool is not running, + // or shutdown has cancelled background activity. + if (MinPoolSize == 0 || State != Running || _warmupCts.IsCancellationRequested) + { + return; + } + + // Fast path: the pool is already at or above the minimum, so there is nothing to warm + // up. Return before scheduling a thread-pool work item. This keeps hot-path callers + // (e.g. RemoveConnection on every return) cheap. The check is best-effort under + // concurrency; a below-minimum condition missed here is still observed by the running + // loop, which re-reads Count on every iteration. + if (Count >= MinPoolSize) + { + return; + } + + // Coalesce: only start a loop if one is not already running. + if (Interlocked.CompareExchange(ref _warmupLoopRunning, 1, 0) != 0) + { + return; + } + + try + { + // Fire-and-forget on the thread pool so warmup never blocks the caller. The loop + // absorbs its own exceptions and always releases the single-loop guard on exit. The + // task is published so tests can await a warmup pass to a deterministic completion. + WarmupLoopTask = Task.Run(RunWarmupLoopAsync); + } + catch (Exception ex) + { + // Scheduling the loop failed (e.g. the thread pool refused the work item). Release the + // guard so warmup isn't permanently pinned off for the life of the pool; the next + // below-minimum trigger will try again. + Interlocked.Exchange(ref _warmupLoopRunning, 0); + SqlClientEventSource.Log.TryPoolerTraceEvent( + " {0}, Failed to schedule warmup loop, absorbing: {1}", Id, ex); + } + } + + /// + /// The coalesced warmup loop: creates connections one at a time (serially) up to + /// , then releases the single-loop guard. Each creation goes through + /// the same slot-reservation and rate-limited path as user requests + /// (), and freshly created connections are published + /// to the idle channel for waiting or future user requests. All failures are absorbed so + /// warmup can never surface an unhandled exception (Story 3); a genuine open failure still + /// enters the pool's blocking-period error state via the shared creation path, mirroring the + /// legacy WaitHandle pool. + /// + private async Task RunWarmupLoopAsync() + { + try + { + CancellationToken token; + try + { + token = _warmupCts.Token; + } + catch (ObjectDisposedException) + { + // The pool shut down and disposed the cancellation source before this loop + // started. There is nothing to warm up; the finally releases the guard. + return; + } + + // Single pass: re-reads Count each iteration, so any below-minimum drop that happens + // while the loop is running is picked up here without a separate request flag. + // + // The loop keeps creating while the pool is running, shutdown has not cancelled + // background work, and the pool is not in the blocking-period error state. While user + // requests are failing and the pool is blocking (ErrorOccurred), warmup stands down + // rather than piling more doomed opens onto a struggling server - mirroring the legacy + // WaitHandle pool, which skips replenishment while blocking. (Warmup still + // participates in the error state on its own creations - entering it on failure and + // clearing it on success - via OpenNewInternalConnection.) + while (State == Running + && !token.IsCancellationRequested + && !ErrorOccurred + && Count < MinPoolSize) + { + // Fresh per-attempt timeout budget based on the pool's CreationTimeout, since + // warmup has no owning Open() call to inherit a budget from. Matches the + // replenishment behavior of the legacy WaitHandle pool. + TimeoutTimer timeout = TimeoutTimer.StartNew( + TimeSpan.FromMilliseconds(PoolGroupOptions.CreationTimeout)); + + DbConnectionInternal? connection; + try + { + // owningConnection is null: warmup connections are created unattached and + // enter the pool as idle. A null return means the shared rate limiter was + // saturated; a thrown exception means the physical open genuinely failed. + connection = OpenNewInternalConnection( + owningConnection: null, + cancellationToken: token, + timeout: timeout); + } + catch (OperationCanceledException) + { + // Shutdown/cancellation unwound the in-flight create. Stop warming up. + break; + } + catch (Exception ex) when (ADP.IsCatchableExceptionType(ex)) + { + // A genuine connection-open failure (not rate limiting). It has already + // entered the pool's blocking-period error state (see OpenNewInternalConnection); + // trace and absorb the rethrow here, then stop this pass rather than retrying + // on a tight cadence. The pool stays operational: user requests fast-fail + // during the blocking window and resume creating on demand once it expires, + // and the next below-minimum trigger re-requests warmup (Story 3). + SqlClientEventSource.Log.TryPoolerTraceEvent( + " {0}, Warmup connection creation failed, stopping pass: {1}", Id, ex); + break; + } + + if (connection is null) + { + // A slot is guaranteed available here (Count < MinPoolSize <= MaxPoolSize), + // and creation failures throw rather than return null, so a null return means + // the shared rate limiter is currently saturated. Rather than bypassing the + // limiter or spinning on it (Story 2), end this warmup pass. Saturation only + // happens while user requests are actively creating connections - those very + // creations fill the pool toward MinPoolSize, and any later drop below the + // minimum re-triggers warmup through RemoveConnection - so warmup does not need + // to compete for a permit here. + break; + } + + // Publish the freshly created connection as idle. It was PrePush'd at creation + // (CreatePooledConnection) and never activated, so it is in the correct state + // to enter the idle channel directly. If a Clear raced and bumped the + // generation, the stale connection is harmlessly removed by IsLiveConnection + // on its next retrieval, so we don't check the generation here. + if (!_idleChannel.TryWrite(connection)) + { + // Channel completed (pool shutting down). Destroy instead of pooling. + RemoveConnection(connection); + break; + } + + // OpenNewInternalConnection is synchronous and blocks the loop's thread for + // the duration of the physical open. Yield between creations so a multi- + // connection warmup returns its thread-pool worker to the scheduler between + // opens (rather than monopolizing one worker for the whole sequence) and stays + // responsive to cancellation. There is no sync-over-async anywhere in this path. + await Task.Yield(); + } + } + catch (Exception ex) + { + // Defense in depth: the loop must never throw onto the thread pool. + SqlClientEventSource.Log.TryPoolerTraceEvent( + " {0}, Warmup loop failed, absorbing: {1}", Id, ex); + } + finally + { + // Always release the single-loop guard, whatever exit path we took, so a future + // below-minimum trigger can start a new loop. Interlocked.Exchange mirrors the + // Interlocked.CompareExchange acquire in RequestWarmup. + Interlocked.Exchange(ref _warmupLoopRunning, 0); + } + } + #endregion + #region Pruning /// /// Manages idle connection pruning. Null when the pool is fixed-size (MinPoolSize >= MaxPoolSize) diff --git a/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlConnectionFactory.cs b/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlConnectionFactory.cs index 3aa89f5332..8840864958 100644 --- a/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlConnectionFactory.cs +++ b/src/Microsoft.Data.SqlClient/src/Microsoft/Data/SqlClient/SqlConnectionFactory.cs @@ -650,8 +650,14 @@ protected virtual DbConnectionInternal CreateConnection( // used below to connect to the SQL Express User Instance. instanceName = sseConnection.InstanceName; - // Set future transient fault handling based on connection options - sqlOwningConnection._applyTransientFaultHandling = opt != null && opt.ConnectRetryCount > 0; + // Set future transient fault handling based on connection options. + // sqlOwningConnection is null for background warmup/replenishment creations, + // which have no owning connection to carry this state forward; only propagate + // it when an owning connection is present (matches the null-safe reads above). + if (sqlOwningConnection != null) + { + sqlOwningConnection._applyTransientFaultHandling = opt != null && opt.ConnectRetryCount > 0; + } if (!instanceName.StartsWith(@"\\.\", StringComparison.Ordinal)) { diff --git a/src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/ChannelDbConnectionPoolWarmupTest.cs b/src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/ChannelDbConnectionPoolWarmupTest.cs new file mode 100644 index 0000000000..266aa73ea8 --- /dev/null +++ b/src/Microsoft.Data.SqlClient/tests/UnitTests/ConnectionPool/ChannelDbConnectionPoolWarmupTest.cs @@ -0,0 +1,688 @@ +// Licensed to the .NET Foundation under one or more agreements. +// The .NET Foundation licenses this file to you under the MIT license. +// See the LICENSE file in the project root for more information. + +using System; +using System.Data.Common; +using System.Threading; +using System.Threading.RateLimiting; +using System.Threading.Tasks; +using System.Transactions; +using Microsoft.Data.Common; +using Microsoft.Data.Common.ConnectionString; +using Microsoft.Data.ProviderBase; +using Microsoft.Data.SqlClient.ConnectionPool; +using Microsoft.Data.SqlClient.Tests.Common; +using Xunit; + +namespace Microsoft.Data.SqlClient.UnitTests.ConnectionPool +{ + /// + /// Unit tests for background pool warmup and replenishment in + /// . See specs/003-pool-warmup/spec.md. + /// + public class ChannelDbConnectionPoolWarmupTest + { + #region Helpers + + private static ChannelDbConnectionPool ConstructPool( + SqlConnectionFactory connectionFactory, + int minPoolSize = 0, + int maxPoolSize = 50, + int idleTimeout = 0, + int loadBalanceTimeout = 0, + ConcurrencyLimiter? connectionCreationRateLimiter = null) + { + var poolGroupOptions = new DbConnectionPoolGroupOptions( + poolByIdentity: false, + minPoolSize: minPoolSize, + maxPoolSize: maxPoolSize, + creationTimeout: 15, + loadBalanceTimeout: loadBalanceTimeout, + hasTransactionAffinity: true, + idleTimeout: idleTimeout); + + var dbConnectionPoolGroup = new DbConnectionPoolGroup( + new SqlConnectionOptions("Data Source=localhost;"), + new ConnectionPoolKey("TestDataSource", credential: null, accessToken: null, accessTokenCallback: null, sspiContextProvider: null), + poolGroupOptions); + + return new ChannelDbConnectionPool( + connectionFactory, + dbConnectionPoolGroup, + DbConnectionPoolIdentity.NoIdentity, + new DbConnectionPoolProviderInfo(), + connectionCreationRateLimiter); + } + + /// + /// Generous upper bound for waits that guard a background handshake which, in correct code, + /// completes in milliseconds. It is only ever hit on a genuine failure/hang, so it does not + /// make passing runs slow or introduce timing races. + /// + private static readonly TimeSpan HandshakeTimeout = TimeSpan.FromSeconds(30); + + /// + /// Awaits background warmup passes until the pool settles at . + /// A single coalesced pass can race with concurrent removals - for example a + /// draining the pool while replenishment refills + /// it - so this re-requests warmup until the pool is full. Only safe with a factory whose + /// warmup creations succeed and where the pool is not in the error state, so every pass makes + /// forward progress and the loop terminates. Bounded by so a + /// regression that prevents the pool from ever reaching the minimum surfaces as a + /// deterministic timeout instead of hanging the whole test run. + /// + private static async Task AwaitWarmupToMinimum(ChannelDbConnectionPool pool, int minPoolSize) + { + DateTime deadline = DateTime.UtcNow + HandshakeTimeout; + while (pool.Count < minPoolSize) + { + Assert.True( + DateTime.UtcNow < deadline, + $"Pool did not reach MinPoolSize {minPoolSize} within {HandshakeTimeout}; Count={pool.Count}."); + + pool.RequestWarmup(); + Task? warmup = pool.WarmupLoopTask; + if (warmup is not null) + { + await warmup; + } + } + } + + #endregion + + #region Story 1 - Background warmup on pool creation + + /// + /// Warmup is a no-op when MinPoolSize == 0: no connections are ever created in the background. + /// + [Fact] + public void Startup_MinPoolSizeZero_DoesNotWarmUp() + { + var factory = new ChannelDbConnectionPoolTest.CountingSuccessfulConnectionFactory(); + using var pool = ConstructPool(factory, minPoolSize: 0, maxPoolSize: 10); + + pool.Startup(); + + // MinPoolSize == 0, so warmup is a no-op: no loop is ever launched (WarmupLoopTask stays + // null) and nothing is created in the background. + Assert.Null(pool.WarmupLoopTask); + Assert.Equal(0, factory.CreateCount); + Assert.Equal(0, pool.Count); + } + + /// + /// Warmup pre-creates connections up to MinPoolSize for various sizes (1 and N). + /// + [Theory] + [InlineData(1)] + [InlineData(3)] + [InlineData(10)] + public async Task Startup_WithMinPoolSize_WarmsUpToMinimum(int minPoolSize) + { + var factory = new ChannelDbConnectionPoolTest.CountingSuccessfulConnectionFactory(); + using var pool = ConstructPool(factory, minPoolSize: minPoolSize, maxPoolSize: minPoolSize + 10); + + pool.Startup(); + + // Awaiting the warmup loop to completion is deterministic: the single serial pass runs + // with no competing removals, so when it finishes the pool is exactly at MinPoolSize. + await pool.WarmupLoopTask!; + + // Warmup reaches exactly MinPoolSize (no overshoot) and pools each created connection. + Assert.Equal(minPoolSize, pool.Count); + Assert.Equal(minPoolSize, pool.IdleCount); + Assert.Equal(minPoolSize, factory.CreateCount); + } + + /// + /// A user request during warmup is served immediately and does not wait for warmup to finish. + /// The gated factory blocks warmup's first (serial) creation, yet a user open still completes + /// by creating its own connection, proving warmup does not block user requests and does not + /// create connections in parallel. + /// + [Fact] + public async Task Startup_UserRequestDuringWarmup_ServedImmediately() + { + using var createGate = new ManualResetEventSlim(initialState: false); + var factory = new ChannelDbConnectionPoolTest.GatedSuccessfulConnectionFactory(createGate); + using var pool = ConstructPool(factory, minPoolSize: 3, maxPoolSize: 10); + + // Warmup begins and its single, serial creation blocks on the gate. Waiting on this + // explicit signal (not a sleep) establishes a deterministic happens-before: warmup is now + // parked inside its first create. + pool.Startup(); + Assert.True( + factory.FirstCreateStarted.Wait(HandshakeTimeout), + "Timed out waiting for warmup to begin its first creation."); + + // Exactly one creation - warmup's - is in flight and parked on the closed gate. The + // warmup loop is serial, so it cannot advance to a second create until the gate opens. + Assert.Equal(1, factory.CreateCount); + + // Issue a user request while warmup is parked. Run it on a dedicated (LongRunning) thread + // so it never competes with warmup for the same pool worker. Because warmup does not block + // user requests, it must complete promptly by creating its own connection. A generous + // upper bound converts a hypothetical "user blocked behind warmup" regression into a clean + // failure instead of an indefinite hang; correct code completes in milliseconds. + Task userOpen = Task.Factory.StartNew( + () => + { + bool completed = pool.TryGetConnection( + new SqlConnection(), + taskCompletionSource: null, + TimeoutTimer.StartNew(TimeSpan.FromSeconds(15)), + out DbConnectionInternal? connection); + Assert.True(completed); + Assert.NotNull(connection); + return connection!; + }, + CancellationToken.None, + TaskCreationOptions.LongRunning, + TaskScheduler.Default); + + Task completed = await Task.WhenAny(userOpen, Task.Delay(HandshakeTimeout)); + Assert.True(completed == userOpen, "User request was blocked behind warmup."); + Assert.NotNull(await userOpen); + + // Seriality: warmup is still parked on its single in-flight create (the gate is closed), + // so the only additional creation is the user's. CreateCount == 2 proves warmup did not + // create connections in parallel. This is deterministic: nothing can advance warmup's + // loop until the gate is set below. + Assert.Equal(2, factory.CreateCount); + + createGate.Set(); + } + + #endregion + + #region Story 2 - Warmup through the shared rate limiter + + /// + /// Warmup creates connections through the shared rate limiter and still reaches MinPoolSize. + /// + [Fact] + public async Task Warmup_ThroughSharedRateLimiter_ReachesMinimum() + { + using var rateLimiter = new ConcurrencyLimiter( + new ConcurrencyLimiterOptions { PermitLimit = 1, QueueLimit = 0 }); + var factory = new ChannelDbConnectionPoolTest.CountingSuccessfulConnectionFactory(); + using var pool = ConstructPool( + factory, + minPoolSize: 3, + maxPoolSize: 10, + connectionCreationRateLimiter: rateLimiter); + + pool.Startup(); + + // Warmup is the only actor here, so each serial create acquires and releases the single + // permit in turn; the pass never sees a saturated limiter and reaches MinPoolSize. + await pool.WarmupLoopTask!; + + Assert.Equal(3, pool.Count); + // Every warmup creation acquired (and released) a permit from the shared limiter. + Assert.True(rateLimiter.GetStatistics()!.TotalSuccessfulLeases >= 3); + } + + /// + /// When the shared rate limiter is saturated by a user request, warmup waits its turn rather + /// than bypassing the limiter: the user request that arrives while warmup holds the only + /// permit is denied by the same limiter. + /// + [Fact] + public async Task Warmup_RateLimiterSaturated_UserSharesSameLimiter() + { + using var createGate = new ManualResetEventSlim(initialState: false); + var factory = new ChannelDbConnectionPoolTest.GatedSuccessfulConnectionFactory(createGate); + using var rateLimiter = new ConcurrencyLimiter( + new ConcurrencyLimiterOptions { PermitLimit = 1, QueueLimit = 0 }); + using var pool = ConstructPool( + factory, + minPoolSize: 2, + maxPoolSize: 5, + connectionCreationRateLimiter: rateLimiter); + + // Warmup begins, acquires the only permit, and blocks in creation while holding it. + pool.Startup(); + Assert.True( + factory.FirstCreateStarted.Wait(HandshakeTimeout), + "Timed out waiting for warmup to begin its first creation."); + + // A concurrent user request competes for the same limiter. There is no idle connection + // and warmup holds the only permit, so the request cannot be satisfied inline: it is + // deferred to the TaskCompletionSource, proving warmup did not bypass the limiter. + var tcs = new TaskCompletionSource(); + bool completedSynchronously = pool.TryGetConnection( + new SqlConnection(), tcs, TimeoutTimer.StartNew(TimeSpan.FromSeconds(15)), out DbConnectionInternal? immediateConnection); + Assert.False(completedSynchronously, "Async request unexpectedly completed synchronously."); + Assert.Null(immediateConnection); + + // Deterministically prove the shared limiter is saturated by warmup: while warmup holds + // its only permit (guaranteed by FirstCreateStarted above and the still-closed gate), + // attempting to acquire that same limiter from the test thread is denied. This is the + // shared-limiter proof without any spin - the denial is observed synchronously here + // rather than by polling for the deferred user request's transient failed-lease, which + // races the permit being released below. The denied attempt is a no-op lease. + long failedBefore = rateLimiter.GetStatistics()!.TotalFailedLeases; + using (RateLimitLease probe = rateLimiter.AttemptAcquire(1)) + { + Assert.False( + probe.IsAcquired, + "Shared rate limiter was not saturated while warmup held its permit."); + } + Assert.True( + rateLimiter.GetStatistics()!.TotalFailedLeases > failedBefore, + "Denied acquire was not recorded by the shared rate limiter."); + + // Release warmup. With rate-limit retry removed, warmup's pass may end early if the + // user request grabs the freed permit first - but the pool must still converge to + // MinPoolSize: whichever creations win the permit (warmup's or the user's) fill it. + createGate.Set(); + + // Await the deferred user request directly rather than spin-polling limiter statistics. + // Bound the wait so a regression that never completes the async request fails fast + // instead of hanging the whole test run. + Task userRequest = tcs.Task; + Assert.True( + userRequest == await Task.WhenAny(userRequest, Task.Delay(HandshakeTimeout)), + "Timed out waiting for the deferred user request to complete."); + DbConnectionInternal? userConnection = await userRequest; + Assert.NotNull(userConnection); + + // Once both the warmup pass and the deferred user request have settled, the pool holds + // exactly MinPoolSize connections (a warmup idle connection plus the user's, or two + // warmup connections one of which the user took). + await AwaitWarmupToMinimum(pool, 2); + Assert.Equal(2, pool.Count); + } + + #endregion + + #region Story 3 - Warmup failure resilience + + /// + /// Warmup creation failures are absorbed - no exception surfaces onto the thread pool and no + /// connections are pooled - but, mirroring the legacy WaitHandle pool, a genuine open failure + /// does enter the pool's blocking-period error state. Warmup then stops the pass rather than + /// spinning on the persistent failure. + /// + [Fact] + public async Task Warmup_AllCreationsFail_AbsorbedAndEntersErrorState() + { + var factory = new WarmupFailingConnectionFactory(); + using var pool = ConstructPool(factory, minPoolSize: 3, maxPoolSize: 10); + + pool.Startup(); + + // Await the warmup pass. The first open fails: OpenNewInternalConnection enters the + // blocking-period error state and rethrows, the warmup loop absorbs the rethrow and ends + // the pass. Awaiting is deterministic - no exception surfaces onto the thread pool. + await pool.WarmupLoopTask!; + + // Exactly one attempt was made: a genuine failure ends the pass rather than spinning on + // the persistent failure, and nothing is pooled. + Assert.Equal(1, factory.WarmupAttemptCount); + Assert.Equal(0, pool.IdleCount); + + // Mirroring the legacy WaitHandle pool, the failed open entered the blocking-period error + // state, exactly as an on-demand creation failure would. + Assert.True( + pool.ErrorOccurred, + "Warmup failure did not enter the pool error state as the WaitHandle pool does."); + } + + /// + /// After warmup fails, the pool enters its blocking-period error state (mirroring the + /// WaitHandle pool), so a user request during the blocking window fast-fails with the cached + /// exception rather than attempting a fresh on-demand open. The pool remains operational and + /// resumes creating on demand once the blocking period expires. + /// + [Fact] + public async Task Warmup_Fails_UserRequestFastFailsDuringBlockingPeriod() + { + var factory = new WarmupFailingConnectionFactory(); + using var pool = ConstructPool(factory, minPoolSize: 2, maxPoolSize: 10); + + pool.Startup(); + + // Warmup's failure drives the pool into the blocking-period error state. + await pool.WarmupLoopTask!; + Assert.True(pool.ErrorOccurred, "Warmup failure did not enter the pool error state."); + + // A user request while the pool is blocking fast-fails with the cached exception. + Assert.ThrowsAny(() => pool.TryGetConnection( + new SqlConnection(), taskCompletionSource: null, TimeoutTimer.StartNew(TimeSpan.FromSeconds(15)), out _)); + } + + /// + /// Warmup respects the pool's blocking-period error state: once a failing user request has + /// driven the pool into the error state, invoking warmup must make it stand down (its loop + /// condition sees ErrorOccurred) and create nothing, mirroring the legacy WaitHandle pool, + /// which skips replenishment while the pool is blocking. Driving the error state with a user + /// request (rather than warmup itself) keeps the two behaviors independent, and awaiting the + /// warmup task makes the stand-down deterministic. + /// + [Fact] + public async Task Warmup_RespectsErrorState_StandsDownWhileBlocking() + { + var factory = new UserFailingWarmupCountingConnectionFactory(); + using var pool = ConstructPool(factory, minPoolSize: 3, maxPoolSize: 10); + + // Drive the pool into the blocking-period error state with a failing user request. We do + // NOT call Startup here: that would kick off a *successful* warmup and fill the pool, + // which is not the behavior under test. + Assert.ThrowsAny(() => pool.TryGetConnection( + new SqlConnection(), taskCompletionSource: null, TimeoutTimer.StartNew(TimeSpan.FromSeconds(15)), out _)); + Assert.True(pool.ErrorOccurred, "Failing user request did not enter the pool error state."); + + // Now request warmup directly. Because the pool is blocking, the loop's error-state guard + // makes it stand down immediately: it creates nothing rather than piling doomed opens + // onto a struggling server. + pool.RequestWarmup(); + await pool.WarmupLoopTask!; + + Assert.Equal(0, factory.WarmupCreateCount); + Assert.Equal(0, pool.IdleCount); + Assert.True(pool.ErrorOccurred, "Warmup must not clear the error state it stood down for."); + } + + #endregion + + #region Story 4 - Warmup cancellation on shutdown + + /// + /// When the pool is shut down mid-warmup, warmup stops promptly: no connections created after + /// shutdown begins, and any in-flight connection is cleaned up. + /// + [Fact] + public async Task Shutdown_DuringWarmup_StopsAndCleansUp() + { + using var createGate = new ManualResetEventSlim(initialState: false); + var factory = new ChannelDbConnectionPoolTest.GatedSuccessfulConnectionFactory(createGate); + using var pool = ConstructPool(factory, minPoolSize: 3, maxPoolSize: 10); + + // Warmup begins and blocks on its first serial creation. + pool.Startup(); + Assert.True( + factory.FirstCreateStarted.Wait(HandshakeTimeout), + "Timed out waiting for warmup to begin its first creation."); + + // Shut down while warmup is parked. This cancels warmup and completes the idle channel. + pool.Shutdown(); + + // Release the in-flight creation and await the loop to completion. The freshly created + // connection cannot be published to the completed idle channel, so it is destroyed, and + // the loop stops without creating anything further. + createGate.Set(); + await pool.WarmupLoopTask!; + + Assert.Equal(0, pool.Count); + Assert.Equal(1, factory.CreateCount); + } + + #endregion + + #region Story 5 - Replenishment on any below-minimum event + + /// + /// Destroying a connection on return (non-poolable/broken) drops the pool below MinPoolSize and + /// triggers replenishment back to the minimum. + /// + [Fact] + public async Task Replenish_AfterDoomedReturn_RefillsToMinimum() + { + var factory = new DoomableConnectionFactory(); + using var pool = ConstructPool(factory, minPoolSize: 2, maxPoolSize: 10); + + pool.Startup(); + await pool.WarmupLoopTask!; + Assert.Equal(2, pool.Count); + + // Check out a connection, mark it non-poolable, and return it: it is destroyed on return, + // dropping the pool to 1 and triggering replenishment through RemoveConnection. + var owner = new SqlConnection(); + pool.TryGetConnection(owner, null, TimeoutTimer.StartNew(TimeSpan.FromSeconds(15)), out DbConnectionInternal? connection); + Assert.NotNull(connection); + ((DoomableStubConnection)connection!).MarkDoNotPool(); + pool.ReturnInternalConnection(connection, owner); + + // The doomed return re-triggered warmup synchronously; awaiting the replenishment pass + // (a single serial create with no competing removal) restores the pool to MinPoolSize. + await pool.WarmupLoopTask!; + Assert.Equal(2, pool.Count); + } + + /// + /// Clearing the pool drops it to zero (below MinPoolSize) and triggers replenishment back to + /// the minimum with fresh-generation connections. + /// + [Fact] + public async Task Replenish_AfterClear_RefillsToMinimum() + { + var factory = new ChannelDbConnectionPoolTest.CountingSuccessfulConnectionFactory(); + using var pool = ConstructPool(factory, minPoolSize: 3, maxPoolSize: 10); + + pool.Startup(); + await pool.WarmupLoopTask!; + Assert.Equal(3, pool.Count); + + pool.Clear(); + + // Clear drains the pool concurrently with replenishment, so a single warmup pass can race + // with the drain; await passes until the pool settles at MinPoolSize. + await AwaitWarmupToMinimum(pool, 3); + Assert.Equal(3, pool.Count); + } + + /// + /// Destroying a connection while still at/above the minimum does NOT trigger replenishment + /// beyond the minimum: the pool stays at MinPoolSize. + /// + [Fact] + public async Task Replenish_DestroyWhileAtMinimum_NoOvershoot() + { + var factory = new DoomableConnectionFactory(); + using var pool = ConstructPool(factory, minPoolSize: 2, maxPoolSize: 10); + + pool.Startup(); + await pool.WarmupLoopTask!; + Assert.Equal(2, pool.Count); + + // Grow above the minimum. Two checkouts reuse the two idle warmup connections; a third + // finds no idle connection and creates a new one synchronously, bringing the total to 3. + var owner1 = new SqlConnection(); + var owner2 = new SqlConnection(); + var owner3 = new SqlConnection(); + pool.TryGetConnection(owner1, null, TimeoutTimer.StartNew(TimeSpan.FromSeconds(15)), out DbConnectionInternal? c1); + pool.TryGetConnection(owner2, null, TimeoutTimer.StartNew(TimeSpan.FromSeconds(15)), out DbConnectionInternal? c2); + pool.TryGetConnection(owner3, null, TimeoutTimer.StartNew(TimeSpan.FromSeconds(15)), out DbConnectionInternal? c3); + Assert.NotNull(c1); + Assert.NotNull(c2); + Assert.NotNull(c3); + Assert.Equal(3, pool.Count); + + // Destroy the third on return. Dropping from 3 to 2 stays at the minimum, so RequestWarmup + // hits its fast path and launches no new pass. + Task? warmupBeforeReturn = pool.WarmupLoopTask; + ((DoomableStubConnection)c3!).MarkDoNotPool(); + pool.ReturnInternalConnection(c3, owner3); + + // No replenishment was triggered: the exposed warmup task is unchanged (no new pass) and + // the pool settled at MinPoolSize without overshooting. + Assert.Same(warmupBeforeReturn, pool.WarmupLoopTask); + Assert.Equal(2, pool.Count); + } + + /// + /// An idle connection evicted by the idle-timeout check (IsLiveConnection) drops the pool + /// below MinPoolSize and triggers replenishment via RemoveConnection → RequestWarmup, + /// restoring the pool to the minimum. + /// + [Fact] + public async Task Replenish_AfterIdleTimeoutEviction_RefillsToMinimum() + { + using LocalAppContextSwitchesHelper switchesHelper = new(); + switchesHelper.UseLegacyIdleTimeoutBehavior = false; + + // Pool with a 1-second idle timeout and MinPoolSize=2 so an evicted connection + // triggers replenishment. + var factory = new DoomableConnectionFactory(); + using var pool = ConstructPool(factory, minPoolSize: 2, maxPoolSize: 10, idleTimeout: 1); + + pool.Startup(); + await pool.WarmupLoopTask!; + Assert.Equal(2, pool.Count); + int createCountAfterWarmup = factory.CreateCount; + + // Check out both warm-up connections. + var owner1 = new SqlConnection(); + var owner2 = new SqlConnection(); + pool.TryGetConnection(owner1, null, TimeoutTimer.StartNew(TimeSpan.FromSeconds(15)), out DbConnectionInternal? conn1); + pool.TryGetConnection(owner2, null, TimeoutTimer.StartNew(TimeSpan.FromSeconds(15)), out DbConnectionInternal? conn2); + Assert.NotNull(conn1); + Assert.NotNull(conn2); + + // Return conn1 first, then backdate its ReturnedTime to simulate idle expiry beyond the 1s timeout. + pool.ReturnInternalConnection(conn1!, owner1); + conn1!.ReturnedTime = DateTime.UtcNow - TimeSpan.FromSeconds(2); + + // Return conn2 normally — not expired. + pool.ReturnInternalConnection(conn2!, owner2); + + // TryGetConnection pops conn1 (head of channel), finds it idle-expired, calls + // RemoveConnection (which triggers RequestWarmup), then pops conn2 and returns it. + var owner3 = new SqlConnection(); + pool.TryGetConnection(owner3, null, TimeoutTimer.StartNew(TimeSpan.FromSeconds(15)), out DbConnectionInternal? conn3); + Assert.NotNull(conn3); + Assert.Same(conn2, conn3); + + // The eviction triggered replenishment; await the pass and verify the pool refilled to MinPoolSize. + await pool.WarmupLoopTask!; + Assert.Equal(2, pool.Count); + Assert.Equal(createCountAfterWarmup + 1, factory.CreateCount); + } + + #endregion + + #region Test doubles + + /// + /// A stub internal connection that can be marked non-poolable so a return destroys it. + /// + private sealed class DoomableStubConnection : DbConnectionInternal + { + internal void MarkDoNotPool() => DoNotPoolThisConnection(); + + public override string ServerVersion => throw new NotImplementedException(); + + public override ConnectionCapabilities Capabilities => throw new NotImplementedException(); + + public override DbTransaction BeginTransaction(System.Data.IsolationLevel il) + => throw new NotImplementedException(); + + public override void EnlistTransaction(Transaction transaction) + { + } + + protected override void Activate(Transaction transaction) + { + } + + protected override void Deactivate() + { + } + + internal override void ResetConnection() + { + } + } + + /// + /// Produces instances so replenishment tests can force a + /// connection to be destroyed on return. + /// + private sealed class DoomableConnectionFactory : SqlConnectionFactory + { + private int _createCount; + + internal int CreateCount => Volatile.Read(ref _createCount); + + protected override DbConnectionInternal CreateConnection( + SqlConnectionOptions options, + ConnectionPoolKey poolKey, + DbConnectionPoolGroupProviderInfo poolGroupProviderInfo, + IDbConnectionPool pool, + DbConnection owningConnection, + TimeoutTimer timeout) + { + Interlocked.Increment(ref _createCount); + return new DoomableStubConnection(); + } + } + + /// + /// Fails only warmup creations (identified by a null owning connection) and succeeds for real + /// user requests. Lets tests assert that warmup failures are absorbed while on-demand user + /// creation still works. + /// + private sealed class WarmupFailingConnectionFactory : SqlConnectionFactory + { + private int _warmupAttemptCount; + + internal int WarmupAttemptCount => Volatile.Read(ref _warmupAttemptCount); + + protected override DbConnectionInternal CreateConnection( + SqlConnectionOptions options, + ConnectionPoolKey poolKey, + DbConnectionPoolGroupProviderInfo poolGroupProviderInfo, + IDbConnectionPool pool, + DbConnection owningConnection, + TimeoutTimer timeout) + { + if (owningConnection is null) + { + Interlocked.Increment(ref _warmupAttemptCount); + throw ADP.PooledOpenTimeout(); + } + + return new DoomableStubConnection(); + } + } + + /// + /// Fails user requests (non-null owning connection) so the pool enters its blocking-period + /// error state, while succeeding for and counting warmup creations (null owning connection). + /// Lets a test drive the pool into the error state and then verify warmup stands down + /// (creates nothing) rather than continuing to replenish. + /// + private sealed class UserFailingWarmupCountingConnectionFactory : SqlConnectionFactory + { + private int _warmupCreateCount; + + internal int WarmupCreateCount => Volatile.Read(ref _warmupCreateCount); + + protected override DbConnectionInternal CreateConnection( + SqlConnectionOptions options, + ConnectionPoolKey poolKey, + DbConnectionPoolGroupProviderInfo poolGroupProviderInfo, + IDbConnectionPool pool, + DbConnection owningConnection, + TimeoutTimer timeout) + { + if (owningConnection is not null) + { + // User request: fail so the pool enters the blocking-period error state. + throw ADP.PooledOpenTimeout(); + } + + // Warmup creation: count it. In the stand-down test this must never be reached while + // the pool is blocking. + Interlocked.Increment(ref _warmupCreateCount); + return new DoomableStubConnection(); + } + } + + #endregion + } +}