Skip to content

Harness Proposal#628

Draft
FScholPer wants to merge 30 commits into
mainfrom
harness
Draft

Harness Proposal#628
FScholPer wants to merge 30 commits into
mainfrom
harness

Conversation

@FScholPer

Copy link
Copy Markdown
Contributor

📌 Description

🚨 Impact Analysis

  • This change does not violate any tool requirements and is covered by existing tool requirements
  • This change does not violate any design decisions
  • Otherwise I have created a ticket for new tool qualification

✅ Checklist

  • Added/updated documentation for new or changed features
  • Added/updated tests to cover the changes
  • Followed project coding standards and guidelines

FScholPer and others added 30 commits April 17, 2026 11:07
Co-authored-by: Maximilian Sören Pollak <maximilian.pollak@qorix.com>
Signed-off-by: Frank Scholter Peres(MBTI) <145544737+FScholPer@users.noreply.github.com>
Co-authored-by: Andreas Zwinkau <95761648+a-zw@users.noreply.github.com>
Signed-off-by: Frank Scholter Peres(MBTI) <145544737+FScholPer@users.noreply.github.com>
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot <copilot@github.com>
- Add outer loop with both metrics_json and needs_json task modes
- Add lightweight candidate validation and query tooling
- Add baseline + rule-retrieval harness candidates
- Add provenance metadata and responsibility model for audit compliance
- Add tool safety restrictions and compliance documentation
- Create executable seed corpus (4 tasks)

Addresses eclipse-score/score#2850, eclipse-score/score#2856
Part of eclipse-score/score#2852
- Log validation failures to validation_failures.jsonl
- Instruct LLM proposer to read failure log first (SKILL.md Step 1)
- Document learning mechanism in SELF_HEALING.md
- Enables system to learn from linting/type/interface mistakes
- Auto-logs failures in outer_loop.py when validation fails

Self-healing allows the proposer to see past mistakes (with exact
error messages and fix commands) so it can avoid repeating them.
This is more effective than explicit instructions because the LLM
sees actual failures it made, not generic rules.

Related: eclipse-score/score#2850
Moved internal documentation outside the public repository.
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown

License Check Results

🚀 The license check job ran with the Bazel command:

bazel run --lockfile_mode=error //src:license-check

Status: ⚠️ Needs Review

Click to expand output
[License Check Output]
Extracting Bazel installation...
Starting local Bazel server (8.4.2) and connecting to it...
INFO: Invocation ID: e63db857-5c55-45b5-ae34-2d1bbc0774c1
Computing main repo mapping: 
Loading: 
Loading: 0 packages loaded
Loading: 0 packages loaded
WARNING: Target pattern parsing failed.
ERROR: Skipping '//src:license-check': no such target '//src:license-check': target 'license-check' not declared in package 'src' defined by /home/runner/work/docs-as-code/docs-as-code/src/BUILD
ERROR: no such target '//src:license-check': target 'license-check' not declared in package 'src' defined by /home/runner/work/docs-as-code/docs-as-code/src/BUILD
INFO: Elapsed time: 4.889s
INFO: 0 processes.
ERROR: Build did NOT complete successfully
ERROR: Build failed. Not running target

Comment thread score_harness/common.py
def load_task_specs(tasks_dir: Path) -> list[dict]:
"""Load all task spec JSON files from the tasks directory."""
specs = []
for path in sorted(tasks_dir.glob("*.json")):
Comment on lines +81 to +82
[sys.executable, str(gate_script), "--metrics-json", str(metrics_json)]
+ gate_args,
Comment on lines +106 to +107
[sys.executable, str(gate_script), "--metrics-json", str(metrics_json)]
+ gate_args,
# Check 1-6: Harness interface and basic functionality
try:
harness = load_harness(candidate_path)
task_spec = json.loads(task_spec_path.read_text())
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown

The created documentation from the pull request is available at: docu-html

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

2 participants