Skip to content

feat: profile precision enhancement v2 — Beta confidence model with 44 fixes#152

Merged
tickernelz merged 2 commits into
tickernelz:mainfrom
rookie136:main
Jul 9, 2026
Merged

feat: profile precision enhancement v2 — Beta confidence model with 44 fixes#152
tickernelz merged 2 commits into
tickernelz:mainfrom
rookie136:main

Conversation

@rookie136

@rookie136 rookie136 commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

Summary

Comprehensive upgrade from linear confidence model to Beta posterior model, plus injection quality improvements, semantic deduplication, Thompson sampling isolation, and validator extensions.

Core Changes

Beta Confidence Model

  • syncConfidence(item): confidence = alpha/(alpha+beta) × min(timeFactor, trendMultiplier)
  • Alpha/beta accumulation replaces direct confidence manipulation
  • lazyMigrateAlpha: detects legacy Thompson prior (0.5, 1.5) and migrates to evidence-based alpha
  • Fills missing fields: weakAlpha, weakBeta, lastMatchTime, firstSeen, pendingValidation

Merge Method Constraints

  • combineItemsmergeConfirmedMatch: only called by exact/strong/weak-upgrade
  • Forced merge and cross-validation use direct field manipulation (avoid F5-9/F5-10 alpha desync)
  • combineThree: alpha=a1+a2+1, beta=b1+b2, frequency=f1+f2+1

Thompson Sampling (Isolated)

  • Weak hits use isolated weakAlpha/weakBeta
  • Cumulative limit > 7 triggers forced judgment (mean ≥ 0.45 upgrades)
  • Upgrade: total alpha +1.0 (mergeConfirmedMatch +0.5 + bonus +0.5)

Decay & Staleness

  • Removed linear decay and conf < 0.3 hard threshold
  • Stale condition: alpha ≤ 2 && age > 30 days
  • Removed decayWeakHits, applyConfidenceDecay, capConfidences

Semantic Deduplication

  • deduplicateItems: LLM checks same-category pairs with cos ≥ 0.50
  • Max 5 LLM calls per cycle, dedupCheckedCache LRU (1000 limit)
  • Merged items accumulate alpha/beta/weakAlpha/weakBeta

pendingValidation

  • New items start with pendingValidation = true
  • Confirmed next cycle: alpha += 1.0 + syncConfidence

Validator (5 verdicts)

  • confirmed/contradicted/no_evidence/inaccurate/oversimplified
  • Covers preferences (top-5) + patterns (top-3)
  • oversimplified triggers evolveAndUpdate when evidence ≥ 3

Injection Quality

  • dedupByCategory + scoreByRecency (conf×0.7 + exp(-age/90)×0.3)
  • Workflow context anchor in analysis prompt

Bug Fixes

  • P3: retry path now calls decayInMemory
  • P10: evolve evidence < 3 no longer adopts

Deployment Results (2026-07-03)

Metric Before After
conf saturation (≥0.98) 7/22 2/20
conf=1.0 6 items 0
conf range 0.50~0.998 0.42~0.99
Dedup merges 0 2 pairs
Forced merge conf→1 16/cycle 0
Decay false positives 12/cycle 0
Build 0 errors
image image image

@rookie136 rookie136 force-pushed the main branch 3 times, most recently from 1022f81 to 5d86a51 Compare July 3, 2026 08:26
@tickernelz

Copy link
Copy Markdown
Owner

Thanks for this substantial PR. I attempted to merge it as part of a batch release, but I'm holding it back because it does not pass the test suite on its own.

Merging #152 cleanly onto the current main (no git conflicts) still fails 3 tests:

  • tests/config.test.ts › CONFIG defaults › should have user profile settings as numbers
  • tests/profile-write.test.ts › merges a new explicit preference into an existing profile without clobbering other prefs
  • tests/profile-write.test.ts › deduplicates when the same explicit preference is written twice, boosting confidence

Example failure:

expect(conf).toBeGreaterThan(0.9)
Expected: > 0.9
Received: 0.7142857142857143

The new Beta-confidence model changes the resulting confidence values (e.g. ~5/7 ≈ 0.714), but tests/profile-write.test.ts in this PR still asserts the previous clamped-near-1.0 behavior, so the PR contradicts itself. Please reconcile the model and the assertions (update the tests to match the intended Beta model, and fix the config.test.ts numeric-defaults expectation) so bun test is green. Once it passes, please also rebase onto the latest main and I'll merge it in the next release. I did not modify your tests myself — that's your design call to make.

@rookie136

Copy link
Copy Markdown
Contributor Author

Fixed all 3 test failures:

  • config.test.ts: updated userProfileMax*userProfileDisplay* field names
  • profile-write.test.ts: confidence assertion changed from >0.9 to >0.6 (Beta model ~0.714)
  • profile-write.test.ts: explicit writes now bypass cold buffer (previously buffered items were silently eaten)
  • Added frequency field to test preference items

All 4 checks pass. Ready for review.

1 similar comment
@rookie136

Copy link
Copy Markdown
Contributor Author

Fixed all 3 test failures:

  • config.test.ts: updated userProfileMax*userProfileDisplay* field names
  • profile-write.test.ts: confidence assertion changed from >0.9 to >0.6 (Beta model ~0.714)
  • profile-write.test.ts: explicit writes now bypass cold buffer (previously buffered items were silently eaten)
  • Added frequency field to test preference items

All 4 checks pass. Ready for review.

rookie136 added 2 commits July 8, 2026 14:12
…4 fixes

Core changes:
- Beta unified confidence model (syncConfidence, alpha/beta accumulation replaces confidence saturation)
- mergeConfirmedMatch rename (combineItems → called only for confirmed matches)
- forced merge / cross-validation use direct field manipulation (avoid alpha desync)
- Thompson weak-hit sampling uses isolated weakAlpha/weakBeta
- Semantic dedup pass: deduplicateItems + checkSemanticDuplicate + pair cache LRU
- pendingValidation cross-batch confirmation + lazy migration
- Remove capConfidences / applyConfidenceDecay / decayWeakHits dead code
- Validator extended: 5-value verdict + patterns coverage + oversimplified → evolve
- Injection quality: dedupByCategory + scoreByRecency + workflow context anchor
- AI-cleanup preserveKeys updated for Beta fields + alpha accumulation
- Bug fixes: P3 retry path decay, P10 evolve evidence threshold >= 3

Deployment results (2026-07-03):
- confidence distribution: 0.42~0.99 (distinguishable)
- 0 saturation events
- dedup: steady-state 0 LLM calls per cycle
- 0 decay false positives
- Thompson weak/familiar isolation correct
- injection: 5-dimension full coverage
- config.test: update renamed fields (userProfileMax* → userProfileDisplay*)
- config.test: relax storagePath assertion for custom paths
- profile-write.test: update confidence assertion to match Beta model (~0.714)
- profile-write.test: add frequency field to test preference items
- mergeItems: bypass cold buffer for explicit user writes (manual-write evidence)
@tickernelz tickernelz merged commit b234f81 into tickernelz:main Jul 9, 2026
4 checks passed
@tickernelz

Copy link
Copy Markdown
Owner

Merged into main as part of the v2.19.0 batch release. Thanks for aligning the test assertions with the Beta confidence model (fix: align test assertions with Beta confidence model) — the suite is now green (197 pass / 0 fail) with typecheck and build clean on the fully-merged tree. Appreciate the follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants