DOC-6973: add a skill to vet user-suggested client libraries - #3814
Merged
andy-stark-redis merged 10 commits intoAug 14, 2026
Merged
Conversation
Adds a repeatable checklist for scoring a community-suggested Redis client library (e.g. PR #3681, Sage/Scala) before it goes into the community-supported clients table on content/develop/clients/_index.md. 13 criteria across maintenance, adoption, engineering quality, Redis-specific fit, and documentation; Pass/Warn/Fail (2/1/0) out of 26, with thresholds for include / needs-discussion / decline, plus three hard stops (archived repo, no OSI license, open unpatched advisory) that override the total. Design decisions worth keeping: - Only verifiable repo/registry facts count. Reviews, blog posts, the project's own benchmarks and social popularity are excluded on purpose: they are the signals an author can manufacture, and they are what makes a decline hard to defend. - Every evidence cell must carry a URL *and* the concrete fact (date, count, version). "Looks actively maintained" is not evidence, and without the fact a reviewer cannot re-check the score six months later. - Missing evidence caps a criterion at Warn and must be stated. Absence is not a Fail, but it must not silently read as a Pass either. - Download thresholds are language-relative and the applied threshold must be stated (npm/PyPI ~100k/month vs pub.dev/hex.pm ~5k/month), otherwise B1 quietly punishes every small-ecosystem client. - The where-to-look column names the pages that actually work unauthenticated: the contributors graph is JS-rendered so bus factor comes from the commits list, and PyPI download counts come from pepy.tech rather than the PyPI page. Learned: the checklist splits cleanly into mechanical criteria (commit/release dates, license, advisories, archived flag, committer count, downloads, dependents) and judgment criteria (responsiveness, feature coverage, compatibility statement, quickstart quality, docs beyond README). That split is what a CI implementation should follow - the mechanical half needs no model at all, and the hard stops live entirely in it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Contributor
🧠 Redis MemoryFound 9 related items from repository history:
Memory updated at 2a8dbde |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
First real run of the checklist (Sage/Scala, PR #3681) found three ways the evidence-gathering step produces a confidently wrong score, and left the question of where a scorecard should be stored unanswered. Both fixed here. The traps, each verified rather than assumed: - search.maven.org's solr index returned 0 hits for com.github.ghostdogpr: sage-core_3 while all seven published versions sat in the artifact store. A naive C2 check reads that as "not on the official registry" - an outright Fail on a project with clean release hygiene. repo1.maven.org's maven-metadata.xml is the authoritative store and it agreed with the tags 1:1. - GitHub's dependents graph doesn't parse build.sbt, so it reports 0 for any sbt project. Scored B2 Fail on the absence of positive evidence instead, and said which instrument was blind - the zero itself proves nothing. - mvnrepository.com does expose a "Used By" count, and returns HTTP 403 to scripted requests. Checked before writing it in as a pointer; it would have been a source no re-run could reproduce. Generalised into one rule at the top of the gathering step: a single index returning zero is not evidence of absence, so confirm against the authoritative store before any Fail that rests on a missing reading. Also recorded that Maven Central and pkg.go.dev publish no download figures at all, so B1 is unscoreable on the JVM and in Go and must be capped at Warn with the gap named - otherwise every JVM client silently reads as poor uptake. Go keeps B2 through pkg.go.dev's "Imported by" (verified: 17,374 for go-redis); the JVM has no scriptable equivalent. New section 6 settles storage. Not a file in this repo: readers expect a checked-in file to be current and half a scorecard's facts are false within months. On an accept it goes in the squash body of the commit that adds the table row, because that commit touches the one line being justified - blame the row, get the evidence. Trailers (Vetted:/Gaps:/Recheck:) keep the set enumerable, since commit messages are otherwise unqueryable as a set. Declines have no commit at all, so they live in the PR thread; the asymmetry is deliberate. Learned: the checklist's failure mode is not misjudging a project, it is trusting one lookup - two of the three traps would have scored a criterion Fail on a project that passes it. Constraint: cap B1 at Warn on the JVM and in Go rather than scoring it down; no download figure exists to apply a threshold to. Constraint: under COMMIT_MESSAGES the squash body defaults to the contributor's WIP commits, so an accept must merge with --body-file or the scorecard is lost. Rejected: mvnrepository.com "Used By" for JVM dependents | HTTP 403 to scripted requests, so a re-run can't reproduce it Directive: keep non-criteria observations out of the scorecard and out of any public reply - they carry no score and read as a judgement on the author. Ticket: DOC-6973 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Bugbot found two real defects on a 100-line PR, both in the arithmetic of an instrument whose whole job is to emit a reproducible number. Overlapping bands: A1 claimed a last commit at exactly 6 months as both Pass (<= 6 months) and Warn (6-18 months), and B1 claimed exactly 5k downloads/month as both Pass (>= 5k) and Warn (500-5k). Two honest runs on the same project could therefore differ by a point. That matters more than the "low severity" label suggests, because the verdict thresholds are hard cutoffs and the first real submission (Sage) cleared >= 21 by exactly one point. Bands are now half-open: "> 6 and <= 18 months", ">= 500 and < 5k". Contradictory download instructions, and self-inflicted: f9c11e1 added Maven Central to the gather list, whose trailing clause promises "download counts", while the note added in the same commit says the JVM and Go publish none and B1 must be capped at Warn there. An agent reading top-to-bottom hits the promise thirty lines before the correction, finds no number, and can score B1 Fail for negligible uptake - exactly the misreading the note exists to stop. Fixed at the point of first contact rather than by rewording the note, because the failure was acting before reaching it. Also swept the remaining bands for the same class of defect, since one overlapping band implies nobody had checked the others: D1's Pass band read "4-5+ of these covered" for a six-item list, so Sage was scored 6/6 against a band whose top was written 5+. Now ">= 4 of these 6" and "<= 1". A2/A3/C*/E* are qualitative or already disjoint. Learned: a checklist that outputs a score is code, and its bands are comparisons - review them as arithmetic, not as prose. Every defect here was invisible when the row was read as an English sentence and obvious the moment it was read as an interval. Constraint: keep scoring bands half-open and mutually exclusive; a value that satisfies two bands makes the total irreproducible and the verdict thresholds are hard cutoffs. Directive: when a criterion's evidence has an ecosystem-wide gap, state it in the gather step too, not only beside the score - an agent acts in reading order. Gaps: the scorecard already squashed into main (ddb6789) recorded D1 as 6/6 against the old "4-5+" wording; the score stands, the wording it cited moved. Ticket: DOC-6973 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Contributor
Author
|
Scope note for reviewers, since the branch name promises something this PR deliberately does not deliver: the GitHub Action was considered and rejected. Suggestions arrive fewer than ten times a year, and at that volume a model-in-CI adds an API-key secret, a cost line, prompt-injection surface (the job would fetch a submitter-controlled README with a token that can write) and score variance — to save work done a handful of times a year. The skill gets run by hand instead. PR body updated, full reasoning on DOC-6973. The checklist has had one real run since: Sage (Scala) scored 21/26 and #3681 is merged, with the scorecard in its squash body ( |
Third round of Bugbot findings on this file, and the third defect in the scoring arithmetic rather than the prose. A2's Fail band disqualified a project with "no tags at all", while the same row cites the registry version list as evidence. A client publishing twelve releases a year to npm without git tags therefore scored A2 Fail (0) for *cadence* - a claim its own evidence refutes - and took a second hit under C2, which is where tagging hygiene actually belongs. One flaw, two penalties, and the A2 number was wrong about the thing the criterion names. A release now counts whether a git tag or a registry publish evidences it. Swept the neighbours again, which turned up the ambiguity I flagged during the Sage run and then left in place: E1 was titled "README quickstart" and scoped its evidence to the README, but its Pass band describes a followable quickstart. Sage's quickstart is excellent and lives on the docs site, so I scored Pass while writing down that a strict reading gives Warn - i.e. two honest runs differ by a point. E1 now accepts a quickstart in the README or one hop from it, and asks the reviewer to record where it lives, so the observation survives as a rule instead of a judgement call. Loosening E1 opened the mirror-image double-count - one getting-started page collecting 4 points across E1 and E2 - so E2 now says explicitly that the quickstart already scored in E1 does not satisfy it. Learned: the review that finds these is not a read-through, it is checking each row as a set of comparisons over a partition. Both defects I found myself came from sweeping the rows *next to* a Bugbot finding, never from re-reading the file; the sweep is the technique, the bot's finding is only the prompt. Constraint: a criterion scores only the signal it names. Tagging hygiene lives in C2, not A2; the quickstart lives in E1, not E2. A flaw that shows up in two criteria is one flaw scored twice and makes the total say more than the evidence does. Rejected: keeping "no tags at all" as an A2 Fail | duplicates C2 and contradicts A2's own registry evidence source Gaps: the Sage scorecard in ddb6789 is unaffected (7 tags and 7 Central versions; E1 already noted the docs-site location), but its cited band wording has now moved twice - diff against the checklist at that date, not today's. Ticket: DOC-6973 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fourth Bugbot round, fourth defect in the scoring arithmetic. B1 completed the band triple for the small ecosystems (pub.dev/hex.pm: >= 5k Pass, >= 500 Warn, < 500 Fail) but gave npm/PyPI a Pass threshold only, leaving Warn vs Fail as "Middling" vs "Negligible". At 20k downloads/month on npm the row supported either 1 or 0, which can move a total across the 14 or 21 cutoff. Kept the design and fixed the execution. Delegating the threshold to the reviewer is deliberate - fifteen registries can't be enumerated and volumes drift - so the requirement is still "state the number and the triple you applied". What was wrong was one worked example being complete and the other not, which reads as though npm/PyPI's Warn/Fail split is defined somewhere it isn't. B1's bands now reference a small threshold table (npm/PyPI, the mid-tier registries, pub.dev/hex.pm), with the rule for unlisted registries stated rather than implied. The npm/PyPI floor is not a new judgement: the existing pub.dev example used a 10:1 ratio between Pass threshold and Fail floor, so a 100k Pass gives a 10k floor. Swept the neighbours as before and tightened B2, the last numeric band still written as prose: "hundreds+" / "Some" / "None" does partition if you read some as >= 1 and hundreds+ as >= 100, but it made the reader infer the boundaries. Now >= 100 / 1-99 / 0. Also pre-empted a reading-order trap I introduced in the same edit: a reviewer meeting "for a registry not listed, pick a triple" would apply it to Maven Central, which is absent because it publishes nothing, not because a triple is missing. Said so at the point of first contact - the same failure mode Bugbot caught in its second finding. Every numeric band in the checklist is now an explicit partition (A1, A2, A3, B1, B2, D1); the rest (A4, C1-C3, D2, E1, E2) are qualitative and can't usefully be made intervals. That is a falsifiable claim rather than a feeling, which is the point. Learned: four rounds, four defects, all in the arithmetic and none in the prose. A rubric that emits a score is code whose rows are intervals over a partition, and it needs reviewing as such - reading it as English is what let every one of these through. Two of the four were found by sweeping the rows adjacent to a bot finding rather than by re-reading the file; the sweep is the technique and the bot's finding is only the prompt. Constraint: every numeric band in a scoring rubric must state all three boundaries explicitly. A band that names only its Pass threshold lets two honest reviewers differ by a point, and with hard verdict cutoffs that changes the outcome, not just the presentation. Directive: when a rubric delegates a judgement to the reviewer, the delegation must be explicit and the worked examples symmetric - an incomplete example reads as a definition that exists elsewhere. Ticket: DOC-6973 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fifth Bugbot round, and the first one that invalidates a score already published rather than one that might mislead a future run. Three parts of this file, written across three commits, instructed differently for the same case. The traps section says a GitHub dependents zero for an sbt project is no-signal and "not a Fail on its own". The scoring preamble says missing evidence is "Warn at best". B2's Fail band, made explicit two commits ago, says "0 dependents and none named". For a JVM client with no named adopter, those produce Warn or Fail depending on which line the reader reaches first - and B2 offered no band at all for an instrument that cannot measure. Fixed as a precedence rule rather than a patch to B2, because the collision can recur in any criterion: a Fail now requires positive evidence of the deficiency from an instrument known to work for that ecosystem, and the rule explicitly overrides any band whose wording would Fail on a missing or unmeasurable reading. B2's bands now name the unmeasurable case in Warn and require a working instrument for Fail. Swept every other Fail band for the same hazard. B2 was the only one that could fire on unmeasurability: A1/A3 rest on commit dates, A2 on releases from either tags or a registry publish, C1 on the repo tree, C3 on the license and the advisories API, D and E on documents - all obtainable. C2's "not on the official registry" is the near miss, and the Maven index trap plus this rule now cover it: a failed lookup is Warn, a confirmed absence from the authoritative store is Fail. Consequence for the record, stated rather than buried: Sage was scored B2 Fail (0) in ddb6789 on reasoning this rule now forbids - no named adopters is absence of a claim, and the only instrument available was blind to sbt. Under the corrected checklist that criterion is Warn (1) and the total is 22/26 not 21/26. The verdict is unchanged at Recommend include, so the merged table row stands, but a re-vet diffing against that scorecard will find a number the current rules would not produce. Learned: the fifth defect was not in a band, it was between three sections that were each correct alone. Every earlier round hardened one row; this one shows the checklist also needs reading as a whole, because a rule stated in the preamble, an exception stated in the traps, and a band stated in a table can each be right and still contradict. Reading order decided the outcome, and I had already been warned about that by the second finding in this same file. Constraint: in a scoring rubric, a Fail must rest on positive evidence of the deficiency from an instrument known to work; an unmeasurable or missing reading is a Warn with the blindness named. Absence of a claim is not evidence of absence. Constraint: where a cross-cutting rule and a per-row band can both apply, state which wins. Two locally correct statements in different sections are a contradiction the reader resolves by reading order. Gaps: the Sage scorecard in ddb6789 records 21/26 with B2 Fail; the corrected rules give 22/26 with B2 Warn, same verdict. Not amendable in place - the squash commit is immutable - so a correction note on PR #3681 is the only reachable fix. Ticket: DOC-6973 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dwdougherty
approved these changes
Aug 14, 2026
dwdougherty
left a comment
dwdougherty
left a comment
Collaborator
There was a problem hiding this comment.
This is great! Approved.
Sixth Bugbot round, and the first to land two findings at once - both band *gaps*, cases matching no band at all, which is the mirror image of round one's overlaps. Both were in qualitative rows, and that is the interesting part: the previous commit claimed every numeric band was now an explicit partition and that the qualitative rows "can't usefully be made intervals". The second half was wrong. Intervals are the wrong tool there, but exhaustiveness is not - qualitative bands still have to cover every plausible case, and five of mine did not. The two found: - C2 had no band for a project on the registry with semver and a changelog but no git tags. Pass demands tags, Warn demands a missing changelog, Fail demands not-being-on-the-registry. Self-inflicted in e474f71, which made A2 defer missing tags to C2 without giving C2 anywhere to put them - the third of six findings caused by the preceding fix. C2 now scores traceability of published versions to a source state, with tags as one way to evidence it. - D2 had no band for an explicit statement of *older* support. "Supports Redis 6.x" is explicit so not Warn, is not 7.x/8.x so not Pass, and is a statement so not Fail. Original wording, never touched by any earlier fix. Then swept the rest of the qualitative rows for the same defect rather than waiting to be walked through them one per round, which found three more: A4 had no band for a project with no inbound issues or PRs to judge (Sage came close - zero open of both, scored Pass off two same-day external closures); C1 had none for CI that is green while running lint only; E2 had none for a hand-written docs site that has been abandoned. All three now sit in Warn with the observation required. The structural fix is in the preamble: bands must be mutually exclusive and exhaustive, numeric ones as half-open intervals and qualitative ones as cases, and a project matching two bands or none means the row is broken rather than the reviewer being indecisive. Learned: I split the criteria into "numeric, needs intervals" and "qualitative, needs judgement" and only audited the first half. The real distinction is that both halves need a partition and only the representation differs. Asserting a property held for six rows while excusing seven from the check is what left five gaps in the excused set - and the excuse was in a commit message, stated confidently, one round before the counter-example arrived. Constraint: in a scoring rubric, every criterion's bands must be exhaustive as well as mutually exclusive, qualitative rows included. A case that fits no band is a defect in the rubric, not a judgement call for the scorer. Rejected: "qualitative criteria can't usefully be made intervals, so they need no partition check" | intervals are the wrong representation, exhaustiveness still applies; five gaps hid behind this Gaps: whether this converges is now testable - three of six findings were caused by the previous fix, so if a whole-class sweep still draws a seventh finding, the file needs a human read rather than another patch. Ticket: DOC-6973 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seventh Bugbot round, two findings, and both landed on the exact branches added in 21f4340 - A4's "too little inbound activity to judge" and C1's "CI green while exercising no real tests". Bugbot's effort setting was raised around this point, which explains the volume and the subtlety, but not the location: these sit on new clauses, not on untouched text. So the diagnosis is about method, not about the reviewer. Six rounds of fixes have mostly appended another "or" clause to a band, and each appended clause closes one case while opening an edge against every neighbouring band. A4's new Warn branch collided with Fail (issues disabled is also too-thin-to-judge), and C1's new Warn branch, by requiring CI to be green, left CI-red-with-no-tests matching nothing at all. Both rows are now restructured around the single axis they measure rather than extended: - C1 scores {is there a real suite} x {does CI run it green}: suite plus green CI is Pass, a suite that CI does not run green (absent, red, or lint-only) is Warn, and no meaningful suite is Fail whatever CI does. This deliberately reverses the previous commit, which had put lint-only CI at Warn: on a criterion named "CI + tests", missing tests is the Fail condition and a linter should not buy a point. - A4 scores whether each inbound channel is served: all served is Pass, served slowly or too little inbound to judge is Warn, any channel unserved is Fail. A4's bands can still co-occur, and that is inherent rather than sloppy - a repo has several inbound channels, each in its own state, so "issues answered slowly" and "PRs untouched" are simultaneously true and describe one real project. Not claiming otherwise this time. The preamble now resolves it deterministically: where more than one band applies, score the lowest and record that the row needs repair, because a reviewer mid-review cannot rewrite the rubric and reading order must not decide a score. It also says to prefer restructuring a row over appending a clause. Learned: the fix technique was the defect source. Appending a disjunct to a band is locally correct and globally corrosive, because a band's meaning is defined by its boundaries with the other two - editing one band silently edits two relationships. Three of the last four findings were on clauses added by the preceding fix, which is a property of how I was fixing, not of what was wrong. Also: stop forecasting convergence. I predicted a stopping point twice; one prediction held, the other was confounded when the reviewer's sensitivity changed mid-series, and neither prediction improved the work. Constraint: prefer restructuring a rubric row around its measurement axis over appending an "or" clause to one band; a disjunct closes one case and opens an edge against every neighbour. Constraint: a rubric needs a deterministic tie-break (score the lowest applicable band) as well as disjoint bands, because residual overlaps are resolved in the field by whoever is reading, and the rubric cannot be repaired mid-review. Gaps: a trend across these rounds is not readable any more - the reviewer's effort setting changed partway, so finding counts before and after are not comparable. Ticket: DOC-6973 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eighth Bugbot round, and the first High severity finding of the series. Both findings were caused by the tie-break rule added in c830a12, which is the interesting part: that rule made the rubric deterministic and thereby converted latent, harmless overlaps into mandatory downgrades. Overlaps a human reader used to resolve with common sense now silently cost a point. B2 was the High. Pass offered two routes - at least 100 dependents, or at least one named independent user of substance - while Warn covered 1-99 dependents. For a project with 5 dependents and a named substantial adopter, both bands matched, so "score the lowest" mandated Warn and the named-user route to Pass became unreachable at any non-zero dependent count. Warn now requires "no named user of substance and either 1-99 dependents or no figure obtainable", which makes it the complement of Pass rather than an overlapping neighbour. C2 was the same shape one level down: Warn listed "unverified publisher" as a downgrade while Pass never required publisher verification, so a package meeting every Pass conjunct also matched Warn and lost a point to the tie-break. Dropped the clause rather than adding verification to Pass: npm provenance, PyPI trusted publishing and signed artifacts are real signals but most registries do not expose them, so scoring it would penalise the language rather than the project - the same reasoning that caps B1 at Warn on the JVM and in Go. It is now recorded as an unscored note where visible. The general rule is in the preamble: where a band offers alternative routes, every lower band must explicitly exclude those routes, or the tie-break kills the alternative. That is the corollary of determinism I missed when adding it - the tie-break did not create these overlaps, it raised their price from zero to a point, which is also why the severity finally jumped. Learned: making a rubric deterministic is not a free improvement, it is a change of failure mode. Before the tie-break an overlap was slack that a sensible reader absorbed; after it, every overlap is a live downgrade, so the rule must arrive together with a sweep for disjunctions in higher bands rather than on its own. Second-order effects of a governance rule land on rows the rule never mentions. Constraint: in a rubric with a lowest-band-wins tie-break, any band offering alternative qualifying routes must be mirrored by explicit exclusions in every lower band, or the alternative route is dead. Rejected: adding publisher verification to C2 Pass | most registries do not expose it, so it would score the ecosystem rather than the project Ticket: DOC-6973 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ninth Bugbot round: D2 (High), E2 and C2. Two were the corollary rule added in 7c81089 - "where a band offers alternative routes, every lower band must exclude them" - which I wrote and then applied only to the two rows that had been flagged. That is the same fix-only-what-is-reported error as three rounds ago, one level up: the rule was general, my sweep was not. - D2 (High): Warn's "implied by recent feature work but not stated" also matched Fail's "no compatibility statement anywhere", so the tie-break forced Fail and the implied-but-unstated route was dead. Fail is now "no statement, and nothing implying one". The row also had a gap - an explicit Redis 9.x claim, or a versionless "works with Redis" - so Pass is now "a current major, 7.x or later" and Warn absorbs versionless statements. - E2: a CI-updated docs.rs or javadoc was both "maintained API reference" (Pass) and "auto-generated only" (Warn). Pass now requires hand-written material; Warn takes any generated reference however current. That is the honest distinction - the criterion is asking whether somebody wrote documentation, not whether a generator ran recently. - C2: Pass and Warn both required semver while Fail only caught chaos, leaving an orderly CalVer or date-versioned client matching nothing. Both bands now ask for a consistently applied scheme, and Fail catches versioning disorderly enough that release order is unclear. Also fixed "missing one of those" to "either or both", which was a gap of the same kind waiting to be reported. Then swept every remaining row properly rather than only the reported ones, which found C3: an empty advisories response was readable as either "none open" (Pass) or "hygiene unclear" (Warn). Under the precedence rule a working instrument answering zero is positive evidence, so Pass now requires the endpoint to answer with none open and Warn covers the case where status cannot be established. Learned: a general rule plus a local sweep is not a general fix. Both times a cross-cutting rule was added here (missing-evidence precedence, then alternative-route exclusion) it was applied to the rows in front of me and left the rest to be reported back one round later. The rule is cheap; the sweep is the work, and skipping it converts a governance improvement into a fresh batch of findings. Constraint: when adding a cross-cutting rule to a rubric, re-audit every row against it in the same commit. A rule applied to the rows that prompted it will be reported back as defects in the rows it was not applied to. Ticket: DOC-6973 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
andy-stark-redis
deleted the
DOC-6973-github-action-to-vet-user-suggested-client-libraries
branch
August 14, 2026 14:43
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 4 potential issues.
Reviewed by Cursor Bugbot for commit 2a8dbde. Configure here.
| | # | Criterion | Pass (2) | Warn (1) | Fail (0) | Where to look | | ||
| |---|-----------|----------|----------|----------|---------------| | ||
| | C1 | CI + tests | A real test suite, run by CI, green on the default branch | A real test suite exists but CI does not run it green — absent, red, or lint/build only | No meaningful test suite, whatever CI does | README badges, `…/actions`, `test/` or `spec/` dirs | | ||
| | C2 | Release hygiene | On the official registry, versions ordered under a consistently applied scheme (semver, CalVer or similar), changelog/release notes maintained, **and** published versions traceable to a source state (git tags or equivalent) | On the registry with consistent versioning but missing either or both of the other two — no changelog, or no tags to trace versions to | Not on the official registry, or versioning inconsistent enough that release order is unclear | Registry page + changelog tab, `…/tags` vs registry versions (on the JVM, diff the tags against `repo1.maven.org/maven2/…/maven-metadata.xml`, not the search index). Publisher verification (npm provenance, PyPI trusted publishing, signed artifacts) is worth recording as a note where the registry shows it, but is **not** scored — most ecosystems do not expose it, so scoring it would penalise the language rather than the project | |
There was a problem hiding this comment.
C2 gap for stale changelog
Medium Severity
C2 Pass requires a maintained changelog, but Warn only covers a missing changelog (or missing tags). A registry project with consistent versioning, tags, and a present but abandoned changelog matches none of the three bands, so scorers have to improvise.
Reviewed by Cursor Bugbot for commit 2a8dbde. Configure here.
| | # | Criterion | Pass (2) | Warn (1) | Fail (0) | Where to look | | ||
| |---|-----------|----------|----------|----------|---------------| | ||
| | E1 | Quickstart | Install → connect → set/get a fresh user can follow, in the README or one hop from it | Partial (examples exist but no clean end-to-end path, or non-English gaps) | No usable quickstart anywhere | Repo README and the page it links to for getting started (a quickstart on the docs site counts; note where it lives, since a README with no inline snippet is worth saying out loud) | | ||
| | E2 | Docs beyond README | Hand-written guides or a hand-written docs site beyond the README, currently maintained | A generated API reference only, however current it is (docs.rs, dartdoc, javadoc, readthedocs autodoc), **or** hand-written docs that are no longer maintained (say how you can tell) | Nothing beyond README | Docs link in README/registry sidebar (readthedocs, pub.dev dartdoc, docs.rs, etc.) — the quickstart already scored in E1 does not by itself satisfy this; E2 asks what exists *beyond* it | |
There was a problem hiding this comment.
E2 quickstart-only docs gap
Medium Severity
The E2 note says an E1 quickstart alone does not satisfy this row, but Pass still matches a hand-written docs site beyond the README, and Fail is only “nothing beyond README.” A docs site whose only page is that quickstart therefore matches Pass by the table, is excluded by the note, and matches neither Warn nor Fail.
Reviewed by Cursor Bugbot for commit 2a8dbde. Configure here.
| |---|---|---| | ||
| | npm, PyPI | ≥ 100k | < 10k | | ||
| | crates.io, packagist, nuget | ≥ 20k | < 2k | | ||
| | pub.dev, hex.pm | ≥ 5k | < 500 | |
There was a problem hiding this comment.
B1 threshold table breaks formulas
Medium Severity
B1 bands are defined as comparisons against a Pass threshold and Fail floor, but the threshold table cells already embed those operators (≥ 100k, < 10k). Applying the band text literally yields nonsensical comparisons, so agents can mis-score boundary download counts.
Reviewed by Cursor Bugbot for commit 2a8dbde. Configure here.
| |---|-----------|----------|----------|----------|---------------| | ||
| | C1 | CI + tests | A real test suite, run by CI, green on the default branch | A real test suite exists but CI does not run it green — absent, red, or lint/build only | No meaningful test suite, whatever CI does | README badges, `…/actions`, `test/` or `spec/` dirs | | ||
| | C2 | Release hygiene | On the official registry, versions ordered under a consistently applied scheme (semver, CalVer or similar), changelog/release notes maintained, **and** published versions traceable to a source state (git tags or equivalent) | On the registry with consistent versioning but missing either or both of the other two — no changelog, or no tags to trace versions to | Not on the official registry, or versioning inconsistent enough that release order is unclear | Registry page + changelog tab, `…/tags` vs registry versions (on the JVM, diff the tags against `repo1.maven.org/maven2/…/maven-metadata.xml`, not the search index). Publisher verification (npm provenance, PyPI trusted publishing, signed artifacts) is worth recording as a note where the registry shows it, but is **not** scored — most ecosystems do not expose it, so scoring it would penalise the language rather than the project | | ||
| | C3 | License & security | OSI-approved license, **and** the advisories endpoint answers with none open | OSI-approved license, but advisory status cannot be established (endpoint unavailable, advisories not enabled) | Non-OSI or no license, or an open unpatched advisory | Repo sidebar License, `…/security/advisories` | |
There was a problem hiding this comment.
C3 open versus unpatched gap
Medium Severity
C3 Pass requires the advisories endpoint to show none open, while Fail and the hard stop require an open unpatched advisory. A listed or open advisory that already has a patch matches neither Pass, Warn, nor Fail, so security scoring becomes ambiguous on ordinary post-fix GHSA history.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit 2a8dbde. Configure here.
EliShteinman
added a commit
to EliShteinman/docs
that referenced
this pull request
Aug 17, 2026
Adds a /vet-redis-client skill: 13-criterion checklist for evaluating suggested client libraries, with 3 hard stops and only verifiable repo/registry facts counted. Run by hand, not CI, to avoid prompt-injection surface on submitter-controlled READMEs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
A
/vet-redis-clientskill (.claude/skills/vet-redis-client/SKILL.md) that scores a community-suggested Redis client library against a fixed checklist, so decisions about the community-supported clients table are consistent and defensible rather than case-by-case. Motivating example: #3681 (Sage, Scala).The checklist
13 criteria, each scored Pass (2) / Warn (1) / Fail (0), out of 26:
Verdicts: >= 21 recommend include, 14-20 needs discussion, < 14 decline. Three hard stops override the total regardless of score: archived repo, no OSI-approved license, open unpatched security advisory.
Ground rules baked into the skill:
No user-facing docs change - this is tooling only.
Follow-up: the GitHub Action was considered and rejected
The ticket asked for this to run automatically on suggestion PRs, or to be triggered by a label. We looked at it and decided not to build it — suggestions arrive fewer than ten times a year, and at that volume a model-in-CI adds an API-key secret, a cost line, prompt-injection surface (the job would fetch a submitter-controlled README with a token that can write) and score variance, to save work done a handful of times a year. The skill is run by hand instead. The branch and ticket names predate that decision; see DOC-6973 for the full reasoning and the findings to revisit if it is ever reopened.
The automation worth building is the opposite direction: a cron re-running the mechanical half of the checklist (commit/release dates, license, advisories, archived flag, committer count — where all three hard stops live) over the rows already in the table, so a listed client going stale gets noticed. No model, no secret, no
pull_request_target. Not part of this PR.🤖 Generated with Claude Code
Note
Low Risk
Documentation-only internal skill; no runtime, auth, or published site behavior changes.
Overview
Adds a
/vet-redis-clientClaude skill (.claude/skills/vet-redis-client/SKILL.md) so docs reviewers can score suggested community Redis clients against a fixed 13-criterion / 26-point checklist and reach a consistent verdict (Recommend include / Needs discussion / Decline).The skill defines how to gather URL-backed evidence (repo, registry, CI, advisories), score maintenance/adoption/engineering/Redis-fit/documentation bands, handle ecosystem blind spots (e.g. Maven Central search lag, sbt dependents graph), and apply hard stops (archived repo, non-OSI license, open advisory). It also specifies output format and how to record accepts via squash commit trailers (
Vetted:/Gaps:/Recheck:per_shared/commit-trailers.md) rather than checking in stale scorecards.No redis.io user-facing docs change—internal reviewer tooling only (DOC-6973).
Reviewed by Cursor Bugbot for commit 2a8dbde. Bugbot is set up for automated code reviews on this repo. Configure here.