Merge upstream

.agents/skills/minimizing-ty-ecosystem-changes/SKILL.md

@@ -7,19 +7,24 @@ description: Use when a user says "minimize this ty ecosystem change", "reproduc
 
 ## Invariants
 
-1. Use the exact Ruff revisions, PR config, dependency cutoff, mypy-primer revision, and project Python version from the Actions run.
+1. Use the exact Ruff revisions, user-level PR config, dependency cutoff, mypy-primer revision, project Python version, and strictness settings from the Actions run.
 2. Reproduce the reported project difference before explaining it or writing a smaller example.
 3. Treat copied binaries and config as read-only, and verify every reduction against both binaries.
+4. Derive every candidate from the preceding verified candidate; NEVER substitute an independently constructed example.
+5. Preserve the underlying trigger, not merely the diagnostic rule, message, or displayed type.
 
 Start each investigation from fresh artifacts. Do not trust retained memories, previous minimizations, current upstream project state, or the helper script's default lockfile.
 
 ## Collect Exact-Run Metadata
 
-Run the bundled helper with the Actions run ID or URL and every affected mypy-primer project name:
+If a primary agent supplied an existing run-metadata manifest, verify that its run ID and attempt match the frozen report and that it contains each assigned project. Reuse the manifest without modifying it.
+
+Otherwise, run the bundled helper with the Actions run ID or URL, matching attempt, and every affected mypy-primer project name:
 
 ```bash
 scripts/collect_ty_ecosystem_run_metadata.py \
   <actions-run> <project-name>... \
+  --attempt <actions-attempt> \
   --output target/ty-ecosystem-run.json
 ```
 
@@ -29,7 +34,7 @@ The current workflow splits compilation into `Build ty (base)` and `Build ty (pr
 
 ## Prepare ty
 
-If a primary agent supplied freshly copied base and PR binaries plus the PR ecosystem config, verify the paths exist and reuse them. Do not rebuild, switch Ruff refs, or overwrite the shared artifacts.
+If a primary agent supplied freshly copied base and PR profiling binaries plus the PR ecosystem config, preserve their absolute paths as `TY_ECOSYSTEM_BASE_BINARY` and `TY_ECOSYSTEM_PR_BINARY`, verify they exist, and reuse them. Do not rebuild those binaries, switch shared Ruff refs, or overwrite the shared artifacts. An agent may build an exact-revision debug binary on demand to identify an ambiguous internal type, using an isolated worktree if necessary; the profiling binaries remain the behavioral oracle.
 
 Otherwise, require a clean working tree, copy `.github/ty-ecosystem.toml` from the PR revision, and build ty on the manifest's merge base and PR revision:
 
@@ -55,40 +60,71 @@ cp target/profiling/ty target/ty-ecosystem-bins/ty-pr
 
 ## Reproduce
 
-Create a unique temporary directory for each project. Read its Python version and the pinned mypy-primer revision from the manifest, then bypass the adjacent script lockfile:
+Create a unique temporary directory for each project and use its absolute path. Read its Python version and the pinned mypy-primer revision from the manifest. Obtain the project revision from the `/blob/<commit>/` component of the original diagnostic's source permalink, and check that links for the same project agree. If no diagnostic permalink exists, inspect the matching diagnostics shard or Actions logs; if the exact revision cannot be recovered, explicitly report that limitation. Then bypass the adjacent script lockfile:
 
 ```bash
 uv run \
   --python <project-python> \
   --with "mypy-primer @ git+https://github.com/hauntsaninja/mypy_primer@<mypy-primer-revision>" \
   --no-project \
   python scripts/setup_primer_project.py \
-  <project-name> <temporary-directory> \
+  <project-name> <absolute-temporary-directory> \
   --revision <report-project-revision> \
   --exclude-newer <EXCLUDE_NEWER>
 ```
 
-Use absolute paths and re-export `TY_CONFIG_FILE` in every new shell before running either binary:
+Use the ecosystem config as user-level configuration, matching CI without replacing project-level config discovery, and re-export `XDG_CONFIG_HOME` in each new shell. If a primary agent supplied `TY_ECOSYSTEM_CONFIG_HOME`, reuse its installed config without modifying it; otherwise, install the copied config locally. Read the project's `strict` or `non-strict` label from the frozen detailed report, or its `strict_settings` value from the matching diagnostics shard. Preserve that mode when running either binary:
 
 ```bash
-export TY_CONFIG_FILE="$PWD/target/ty-ecosystem-bins/ty-ecosystem.toml"
-test -f "$TY_CONFIG_FILE"
-project_dir="$PWD/<temporary-directory>"
-ty_base="$PWD/target/ty-ecosystem-bins/ty-base"
-ty_pr="$PWD/target/ty-ecosystem-bins/ty-pr"
+if [[ -n "${TY_ECOSYSTEM_CONFIG_HOME:-}" ]]; then
+  export XDG_CONFIG_HOME="$TY_ECOSYSTEM_CONFIG_HOME"
+  test -f "$XDG_CONFIG_HOME/ty/ty.toml" || exit 1
+else
+  export XDG_CONFIG_HOME="$PWD/target/ty-ecosystem-config"
+  mkdir -p "$XDG_CONFIG_HOME/ty"
+  cp "$PWD/target/ty-ecosystem-bins/ty-ecosystem.toml" "$XDG_CONFIG_HOME/ty/ty.toml"
+fi
+unset TY_CONFIG_FILE
+
+project_dir="<absolute-temporary-directory>"
+ty_base="${TY_ECOSYSTEM_BASE_BINARY:-$PWD/target/ty-ecosystem-bins/ty-base}"
+ty_pr="${TY_ECOSYSTEM_PR_BINARY:-$PWD/target/ty-ecosystem-bins/ty-pr}"
+test -x "$ty_base" && test -x "$ty_pr" || exit 1
+ecosystem_analysis_mode="<strict-or-non-strict-from-detailed-report>"
+
+if [[ "$ecosystem_analysis_mode" != strict && "$ecosystem_analysis_mode" != non-strict ]]; then
+  echo "Unknown ecosystem analysis mode: $ecosystem_analysis_mode" >&2
+  exit 1
+fi
+
+run_ecosystem_ty() {
+  if [[ "$ecosystem_analysis_mode" == strict ]]; then
+    <project-specific command printed by setup_primer_project.py> \
+      --config analysis.strict-equality-semantics=true \
+      --config analysis.strict-generic-narrowing=true
+  else
+    <project-specific command printed by setup_primer_project.py>
+  fi
+}
 
 cd "$project_dir"
 ty_binary="$ty_base"
-<project-specific command printed by setup_primer_project.py>
+base_exit_status=0
+run_ecosystem_ty || base_exit_status=$?
 ty_binary="$ty_pr"
-<project-specific command printed by setup_primer_project.py>
+pr_exit_status=0
+run_ecosystem_ty || pr_exit_status=$?
 ```
 
-Confirm the detailed report's difference exactly, including duplicate diagnostics when present.
+Confirm the detailed report's difference exactly, including duplicate diagnostics and both exit statuses. Ordinary diagnostics can produce exit status 1; do not mistake that for a failed reproduction.
 
 ## Minimize
 
-Reduce the reproduced project iteratively, using the base-versus-PR output as the oracle after every change. Prefer a single file, no third-party imports, and the least complex code that preserves the difference. For nontrivial reductions, follow [references/advanced-minimization.md](references/advanced-minimization.md).
+Reduce the reproduced project toward a self-contained single-file reproducer with minimal code and dependencies. A reduction is trivial only when the difference already occurs in one self-contained file and can be preserved solely by deleting obviously unrelated code. Multiple files, imports or dependencies, inlining, replacing language constructs, ambiguous types such as `@Todo`, or an uncertain cause make a reduction nontrivial. Before attempting any nontrivial reduction, read and follow [references/advanced-minimization.md](references/advanced-minimization.md). If in doubt, treat the reduction as nontrivial.
+
+Matching diagnostics or displayed types do not establish a shared cause. When the output is ambiguous, identify the original and minimized triggers using exact-revision debug output, a targeted `reveal_type`, or the producing Rust call site.
+
+Record the original source permalink, accepted reductions, both binaries' results, and any causal fingerprint. If source provenance or a matching cause cannot be established, return the original project excerpt explicitly marked as unminimized.
 
 ## Return
 

.agents/skills/minimizing-ty-ecosystem-changes/references/advanced-minimization.md

@@ -8,7 +8,7 @@ Prefer a single-file reproducer with no third-party imports, few definitions, an
 
 ## Reduction Loop
 
-Work systematically from the reproduced project. Do not skip ahead to an explanation, hand-written reproducer, or a guessed subset of relevant code. Follow the stages below in order and exhaust each stage before advancing. Try one controlled reduction at a time, run both copied ty binaries after every change, and keep the reduction only if the original difference remains. After every successful reduction, restart at step 1 because it may make earlier reductions possible.
+Work systematically from the reproduced project. NEVER skip ahead to an explanation, hand-written reproducer, or a guessed subset of relevant code. Follow the stages below in order and exhaust each stage before advancing. Try one controlled reduction at a time, run both copied ty binaries after every change, and keep the reduction only if the original difference and underlying trigger remain. After every successful reduction, restart at step 1 because it may make earlier reductions possible.
 
 1. Delete unrelated files.
 2. Remove imports, definitions, decorators, annotations, statements, and branches.
@@ -23,4 +23,6 @@ Repeat the full loop until an exhaustive pass through every stage finds no furth
 
 Attempt to remove every remaining import and inline every remaining third-party definition. Record why any surviving import is essential. Keep these notes as working evidence; the caller decides whether they belong in its final artifact.
 
+Verify that the recorded reduction chain connects the final reproducer to the original ecosystem entry and, when diagnostic output is ambiguous, preserves the original causal fingerprint. If either check fails, return the original project excerpt as unminimized instead of substituting an unrelated example.
+
 Delete transient project and dependency copies after the investigation.

.agents/skills/summarise-ecosystem-results/SKILL.md

@@ -8,7 +8,7 @@ description: Use when a user says "summarise ecosystem results", "summarize this
 ## Priorities
 
 1. Reproduce every retained behavior with the exact environment used by the Actions run.
-2. Lead the report with analysis of diagnostic changes and clear minimized examples.
+2. Lead the report with new or changed project failures, then cover meaningful flaky behavior, diagnostic changes, and clear minimized examples.
 3. Keep execution, audit, and traceability bookkeeping out of the report.
 
 ## Deliverable
@@ -21,9 +21,11 @@ If summarising an ecosystem report is the only thing you're asked to do in a Cod
 
 ## Workflow
 
-1. **Locate the evidence.** Normalize the input to a PR number, find the ty ecosystem-results comment, open the linked detailed HTML report, and identify the exact Actions run that produced it. Use the comment as the change list and the detailed report as evidence.
-2. **Reproduce from scratch.** Ignore retained memories and previous local artifacts. Load the `minimizing-ty-ecosystem-changes` skill, use its metadata helper and exact-run workflow, and reproduce each report entry before explaining or minimizing it.
-3. **Minimize and curate.** Retain the smallest clear reproducer for each distinct behavior change. Group entries only when the same base-to-PR behavior, explanation, and reproducer account for every entry in the group.
-4. **Write and verify.** Fill the report template, check every link and diagnostic, then run `uvx prek run --files PR_<number>_ECOSYSTEM_SUMMARY.md`. Present the Markdown file as the finished product.
+1. **Freeze the evidence.** Preserve any report URL or ecosystem-results comment explicitly supplied by the user before identifying the PR. For PR-only input, find its ecosystem-results comment and linked detailed report. Capture the matching Actions run and attempt as described in [references/evidence-acquisition.md](references/evidence-acquisition.md); never replace a supplied report with the PR's current report. Ignore later comment edits, PR updates, and workflow runs. Use the frozen detailed report as the authoritative change list and the comment for orientation when available.
+2. **Identify changed outcomes.** Check the detailed report for new, fixed, or changed project failures, panics, timeouts, abnormal exits, and meaningful flaky diagnostic or exit-status changes. Omit unchanged persistent failures. If neither project outcomes nor diagnostics changed, say explicitly that the run had no ecosystem impact and omit project-specific sections and reproduction details.
+3. **Reproduce from scratch.** Ignore retained memories and previous local artifacts. Load the `minimizing-ty-ecosystem-changes` skill, use its metadata helper and exact-run workflow, and reproduce each report entry before explaining or minimizing it. Reproduce flaky behavior with the reported run counts.
+4. **Minimize with provenance.** Include a standalone reproducer only when a verified reduction chain connects it to a cited ecosystem entry and preserves the same underlying trigger. If either cannot be verified, retain the original source excerpt and identify it as unminimized.
+5. **Group by cause.** Group entries only when the same base-to-PR behavior, underlying trigger, explanation, and reproducer account for every entry. Identical diagnostic text or displayed `@Todo` types do not establish equivalence.
+6. **Write and verify.** Fill the report template, record each affected project's strict or non-strict analysis mode, and include both strict-analysis flags in the comparison method when applicable. Check every link, diagnostic, reproducer's source provenance, and causal fingerprint when required, then run `uv run --only-group dev --locked prek run --files PR_<number>_ECOSYSTEM_SUMMARY.md`. Present the Markdown file as the finished product.
 
-When parallelizing step 2, read [references/subagent-handoff.md](references/subagent-handoff.md). Otherwise, keep batches small and work through them sequentially.
+When parallelizing reproduction or minimization, read [references/subagent-handoff.md](references/subagent-handoff.md). Otherwise, keep batches small and work through them sequentially.

.agents/skills/summarise-ecosystem-results/assets/report-template.md

@@ -1,8 +1,20 @@
-<!-- Replace every placeholder and remove all HTML comments before presenting the report. Keep each prose paragraph and list item on one source line. Do not add change-count tables, bot-update timestamps, reproduction-completeness bookkeeping, import-audit details, exhaustive traceability appendices, raw URLs, or artifact hashes. -->
+<!-- Replace every placeholder and remove all HTML comments before presenting the report. Keep each prose paragraph and list item on one source line. Omit project-specific reproduction details when there are no affected projects. Do not add change-count tables, bot-update timestamps, reproduction-completeness bookkeeping, import-audit details, exhaustive traceability appendices, raw URLs, or artifact hashes. -->
 
 # [PR #<number>](https://github.com/astral-sh/ruff/pull/<number>) ecosystem summary
 
-<Summarize the distinct diagnostic behavior changes and their significance. Lead with the analysis readers need; do not describe how the report was generated.>
+<Summarize the distinct failure, flakiness, and diagnostic behavior changes and their significance. Lead with the analysis readers need; do not describe how the report was generated.>
+
+<!-- Omit this entire section if no project failures or meaningful flaky outcomes changed. -->
+
+## <New, fixed, or changed project failure or flaky outcome>
+
+**Affected projects:**
+
+- [<project>](<project-url>): merge base: `<base outcome>`; PR: `<PR outcome>`.
+
+<Explain the crash, panic, timeout, abnormal exit, or flaky change, including relevant stderr and observed base/PR run frequencies where applicable.>
+
+<!-- Omit this entire section if no diagnostic behavior changed. -->
 
 ## <Distinct behavior change>
 
@@ -14,19 +26,37 @@
 
 <Explain the exact behavior on the merge base and PR. Group additional entries here only when the same explanation and minimized reproducer account for all of them.>
 
+<!--
+A minimal reproducer should annotate every line with a new, changed, or removed diagnostic using comments immediately above that line. Include the full error messages and error codes from both revisions, including duplicates.
+
+For example:
+
+```python
+from typing import Final
+
+# Merge base: `[error-code-1] "Some error message"`
+# PR: no diagnostic
+x: Final = 42
+
+if x:
+    # Merge base: `[error-code-2] "Some error message"`
+    # PR: `[error-code-2] "Some other error message"`
+    Y = 56
+```
+-->
+
 ```python
-# Merge base: <every full diagnostic message and error code, including duplicates, or no diagnostic>
-# PR: <every full diagnostic message and error code, including duplicates, or no diagnostic>
 <minimal reproducer>
 ```
 
 ## Reproduction
 
 - Detailed report: [ecosystem-analyzer report](<report-url>)
-- Actions run: [run <id>](<run-url>)
+- Actions run: [run <id>, attempt <attempt>](<run-url>)
 - Ruff comparison: [`<merge-base>`](https://github.com/astral-sh/ruff/commit/<merge-base>) to [`<pr-revision>`](https://github.com/astral-sh/ruff/commit/<pr-revision>)
 - `ecosystem-analyzer`: [`<revision>`](https://github.com/astral-sh/ecosystem-analyzer/commit/<revision>)
 - `mypy-primer`: [`<revision>`](https://github.com/hauntsaninja/mypy_primer/commit/<revision>)
-- Project Python: `<project: version, ...>`
 - Dependency cutoff: `<EXCLUDE_NEWER>`
-- Comparison method: `<concise exact commands or method used to run both copied ty binaries>`
+- Project Python: `<project: version, ...>`
+- Project analysis mode: `<project: strict or non-strict, ...>`
+- Comparison method: `<concise exact commands or method used to run both copied ty binaries, including --config analysis.strict-equality-semantics=true and --config analysis.strict-generic-narrowing=true for strict projects>`