diff --git a/CLAUDE.md b/CLAUDE.md index 056b77792..14b0b7ac4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,51 +1,46 @@ # Claude Code Guidelines for OpenAdapt -## Repository Overview -This is the main OpenAdapt meta-package repository. It provides a unified CLI that coordinates sub-packages: +## Repository role -- `openadapt-capture` - GUI recording -- `openadapt-ml` - ML training/inference -- `openadapt-evals` - Benchmark evaluation -- `openadapt-viewer` - HTML visualization -- `openadapt-grounding` - UI element localization -- `openadapt-retrieval` - Multimodal retrieval -- `openadapt-privacy` - PII/PHI scrubbing +This public repository is the Beta launcher/meta-package and stable community +entry point for OpenAdapt. It owns `pip install openadapt`, the unified +`openadapt` CLI, release compatibility, and launcher packaging. -## Important Rules +The canonical compiler and governed runtime live in +[`OpenAdaptAI/openadapt-flow`](https://github.com/OpenAdaptAI/openadapt-flow). +Engine, replay, verification, repair, policy, and backend changes belong there; +do not implement a second engine in this repository. -### Always Use Pull Requests -**NEVER push directly to the `main` branch.** Always create a feature branch and submit a PR, even for small changes. +The current product compiles demonstrated GUI workflows into deterministic, +locally executable programs. Healthy runs make no model calls. Training, +retrieval, and general computer-use agents are separate research surfaces. -```bash -# Create a new branch -git checkout -b feature/my-change +## Before changing the repository -# Make changes and commit -git add . -git commit -m "Description of change" +1. Read the workspace `AGENTS.md` and current `STATUS.md` when available. +2. Fetch `origin` and base findings and changes on fresh `origin/main`. +3. Read [README.md](README.md) and [CONTRIBUTING.md](CONTRIBUTING.md). +4. Keep public installation centered on `pip install openadapt` and + `openadapt ...`; `openadapt-flow` remains the contributor/engine package. +5. Preserve local-first use, fail-closed execution, and the open-core boundary. -# Push branch and create PR -git push -u origin feature/my-change -gh pr create --title "Title" --body "Description" -``` +## Repository layout + +- `openadapt/`: launcher package and unified CLI +- `tests/`: launcher, integration-seam, and release-artifact tests +- `legacy/`: frozen pre-1.0 monolith +- `docs/`: noncanonical historical repository documentation; + docs.openadapt.ai is maintained in `OpenAdaptAI/openadapt-ops` +- `.github/`: CI, security, dependency, and release workflows -Branch protection is configured but can be bypassed by admins - don't do it. +## Development and delivery -### Development Setup ```bash -pip install -e ".[dev]" +python -m pip install -e '.[dev]' +pytest +ruff check openadapt tests ``` -### Key Directories -- `src/openadapt/` - Main package code (CLI, lazy imports) -- `docs/` - Documentation (architecture, permissions guide) -- `legacy/` - Archived monolithic codebase (v0.46.0) -- `.github/` - CI/CD workflows, issue templates - -### Release Process -Releases are automated via GitHub Actions using python-semantic-release. -Tag format: `vX.Y.Z` - -## Related Repositories -- Website: https://github.com/OpenAdaptAI/openadapt-web -- All sub-packages: https://github.com/OpenAdaptAI/openadapt-* +Use a focused branch and pull request for every change. Do not push directly to +`main`. Releases are performed only through the reviewed GitHub Actions release +workflow after explicit authorization; never publish from a development task. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d83af5eb8..84d663202 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,20 +2,15 @@ Thank you for your interest in contributing to OpenAdapt! -## Current Product Boundary +## Product boundary `OpenAdaptAI/OpenAdapt` is the Beta launcher/meta-package and compatibility surface. The canonical compiler and governed runtime live in [`openadapt-flow`](https://github.com/OpenAdaptAI/openadapt-flow). New engine, replay, repair, policy, and backend work belongs there. -| Lifecycle | Repositories | Contribution scope | -|-----------|--------------|--------------------| -| **Beta product** | `OpenAdapt`, `openadapt-flow` | Launcher here; engine in `openadapt-flow` | -| **Experimental support** | `openadapt-capture`, `openadapt-privacy`, `openadapt-desktop` | Native capture, scrubbing, and authoring surfaces | -| **Agent integration** | `openadapt-agent` | MCP and Agent Skills bridge over governed Flow bundles | -| **Research** | `openadapt-ml`, `openadapt-evals`, `openadapt-grounding`, `openadapt-retrieval` | GUI-agent research and evaluation, not the product runtime | -| **Deprecated/history** | `legacy/`, pre-v2 `openadapt-agent` | Migration fixes only; no new features | +This repository's lifecycle is **Beta**. Each sibling repository publishes its +own lifecycle status; do not infer that status from package availability here. ## Where to Contribute @@ -27,21 +22,34 @@ replay, repair, policy, and backend work belongs there. - **Other repositories**: open issues only when their stated lifecycle and contribution guide match the proposed work. -## Getting Started +## Getting started -1. Fork the repository -2. Clone your fork -3. Install in development mode: `pip install -e ".[dev]"` -4. Create a branch for your changes -5. Make your changes and test locally -6. Submit a pull request +1. Fork and clone the repository. +2. Create a focused branch from current `origin/main`. +3. Install the launcher in development mode: + + ```bash + python -m pip install -e '.[dev]' + ``` + +4. Run the focused test suite and formatting checks: + + ```bash + pytest + ruff check openadapt tests + ``` + +5. Open a pull request. The pull-request template lists the required release + and source-boundary checks. ## Guidelines -- Follow existing code style -- Add tests for new functionality -- Update documentation as needed -- Keep PRs focused and small +- Follow the existing code style. +- Test durable behavior and package contracts rather than ordinary copy. +- Keep changes focused and update the canonical documentation when behavior + changes. +- Never place customer-derived data, deployment recipes, or secrets in a public + fixture or package artifact. ## Licensing and the open-core boundary diff --git a/docs/DOCS_UPDATE_SUMMARY.md b/docs/DOCS_UPDATE_SUMMARY.md deleted file mode 100644 index 8b0f4de4f..000000000 --- a/docs/DOCS_UPDATE_SUMMARY.md +++ /dev/null @@ -1,461 +0,0 @@ -# Documentation Update Summary - January 2026 - -**Date**: January 17, 2026 -**Updated By**: Claude Sonnet 4.5 -**Scope**: Comprehensive docs.openadapt.ai fixes and improvements - ---- - -## Executive Summary - -This update addresses critical issues in docs.openadapt.ai and brings the documentation in line with the latest OpenAdapt capabilities, particularly the new screenshot autogeneration system developed for openadapt-viewer. - -### Critical Fixes - -✅ **P0: Fixed `**adapt**` stars rendering issue** -- **Issue**: Text "software **adapt**er" was rendering with literal stars/asterisks instead of bold formatting -- **Root Cause**: Markdown parser interpreting nested bold syntax incorrectly -- **Solution**: Removed excessive bold formatting, changed to plain text "software adapter" -- **Files Fixed**: - - `docs/index.md` (line 5) - - `docs/design/landing-page-strategy.md` (line 32) - -### Major Improvements - -✅ **Screenshot Autogeneration System** -- Created `docs/_scripts/generate_docs_screenshots.py` - 450+ line Playwright-based screenshot generator -- Modeled after the proven `openadapt-viewer/scripts/generate_comprehensive_screenshots.py` approach -- Supports multiple categories: CLI, viewers, segmentation, architecture -- Includes metadata generation and comprehensive documentation -- Created `docs/_scripts/README.md` with usage instructions - -✅ **Package Documentation Updates** -- **openadapt-viewer** (`docs/packages/viewer.md`): - - Added Episode Segmentation Viewer section - - Added Recording Catalog System section - - Added Advanced Search documentation - - Added Component Library reference - - Added Screenshot Automation guide - - Expanded from 136 to 336 lines (148% increase) - -- **openadapt-ml** (`docs/packages/ml.md`): - - Added comprehensive Episode Segmentation section - - Documented CLI commands for segmentation - - Added Python API examples - - Added episode schema documentation - - Added mermaid diagram explaining segmentation workflow - - Expanded from 155 to 277 lines (79% increase) - ---- - -## Detailed Changes - -### 1. Critical Stars Issue (P0) - -**Before:** -```markdown -OpenAdapt is the **open** source software **adapt**er between... -``` - -**After:** -```markdown -OpenAdapt is the open source software adapter between... -``` - -**Why This Matters:** -The nested bold formatting was causing markdown parsers to render literal stars in the text, making the homepage look broken and unprofessional. This was the #1 user-reported issue. - -### 2. Screenshot Autogeneration Script - -**Location**: `/Users/abrichr/oa/src/OpenAdapt/docs/_scripts/generate_docs_screenshots.py` - -**Features:** -- Playwright-based automation (same as openadapt-viewer) -- Multiple screenshot categories: - - `cli`: Terminal/CLI command examples - - `viewers`: Capture, training, benchmark viewers - - `segmentation`: Episode segmentation viewer - - `architecture`: Mermaid diagram rendering -- Configurable viewports and interactions -- Metadata generation (JSON) -- Comprehensive error handling - -**Usage:** -```bash -# Generate all screenshots -python docs/_scripts/generate_docs_screenshots.py - -# Specific categories -python docs/_scripts/generate_docs_screenshots.py --categories viewers segmentation - -# Custom output -python docs/_scripts/generate_docs_screenshots.py --output /path/to/screenshots - -# With metadata -python docs/_scripts/generate_docs_screenshots.py --save-metadata -``` - -**Output Structure:** -``` -docs/assets/screenshots/ -├── cli/ -│ ├── 01_installation.png -│ ├── 02_capture_start.png -│ ├── 03_capture_list.png -│ ├── 04_train_start.png -│ └── 05_eval_run.png -├── viewers/ -│ ├── capture_viewer_overview.png -│ └── capture_viewer_detail.png -├── segmentation/ -│ ├── segmentation_overview.png -│ ├── segmentation_episode_detail.png -│ └── segmentation_search.png -└── screenshots_metadata.json -``` - -### 3. openadapt-viewer Documentation - -**File**: `docs/packages/viewer.md` - -**New Sections Added:** - -#### Episode Segmentation Viewer -- Automatic episode detection features -- Visual library with thumbnails -- Key frame galleries -- Recording filtering -- Advanced search capabilities -- Auto-discovery system - -**Code Example:** -```python -from openadapt_viewer import generate_segmentation_viewer - -viewer_path = generate_segmentation_viewer( - output_path="segmentation_viewer.html", - include_catalog=True, # Enable auto-discovery -) -``` - -#### Recording Catalog System -- Automatic scanning and indexing -- SQLite database at `~/.openadapt/catalog.db` -- Recording metadata tracking -- CLI integration - -**Code Example:** -```python -from openadapt_viewer import get_catalog, scan_and_update_catalog - -counts = scan_and_update_catalog() -catalog = get_catalog() -recordings = catalog.get_all_recordings() -``` - -#### Advanced Search -- Case-insensitive matching -- Token-based search (normalizes spaces) -- Token order independence -- Partial matching support -- Multi-field search - -**Example:** -```javascript -const results = advancedSearch(episodes, "nightshift", - ['name', 'description', 'steps']); -``` - -#### Component Library -- Complete component reference table -- Usage examples for key components -- Screenshot display, metrics, playback controls - -#### Screenshot Automation -- Playwright-based generation -- Desktop and responsive viewports -- Metadata output -- Fast generation (~30 seconds) - -### 4. openadapt-ml Documentation - -**File**: `docs/packages/ml.md` - -**New Section: Episode Segmentation** - -Comprehensive documentation of the new ML-powered episode segmentation feature: - -**CLI Commands:** -```bash -# Segment a recording -openadapt ml segment --recording turn-off-nightshift --output episodes.json - -# Batch segment -openadapt ml segment --all --output-dir segmentation_output/ - -# View results -openadapt ml view-episodes --file episodes.json -``` - -**Python API:** -```python -from openadapt_ml import EpisodeSegmenter, generate_episode_library - -segmenter = EpisodeSegmenter(model="qwen3vl-2b") -episodes = segmenter.segment_recording("turn-off-nightshift") - -library = generate_episode_library( - recordings=["recording1", "recording2"], - output_path="episode_library.json" -) -``` - -**Episode Schema:** -Documented complete JSON schema with all fields: -- episode_id, recording_name, name, description -- start_frame, end_frame, duration_seconds -- key_frames array (representative frames) -- steps array (task breakdown) -- metadata (confidence, model, timestamp) - -**Visualization:** -Added mermaid diagram showing: -``` -Recording Frames + Actions - → Vision-Language Model - → Scene Change Detection - → Task Boundary Detection - → Episodes + Key Frames + Steps -``` - ---- - -## Files Changed - -### Created -- ✅ `docs/_scripts/generate_docs_screenshots.py` (450 lines) -- ✅ `docs/_scripts/README.md` (documentation) -- ✅ `docs/DOCS_UPDATE_SUMMARY.md` (this file) - -### Modified -- ✅ `docs/index.md` (fixed stars issue, line 5) -- ✅ `docs/design/landing-page-strategy.md` (fixed stars issue, line 32) -- ✅ `docs/packages/viewer.md` (expanded from 136 to 336 lines, +148%) -- ✅ `docs/packages/ml.md` (expanded from 155 to 277 lines, +79%) - -### Total Impact -- **Lines Added**: ~650+ -- **Lines Modified**: ~10 -- **New Features Documented**: 5 (segmentation, catalog, search, components, screenshot automation) -- **Critical Bugs Fixed**: 1 (stars rendering) - ---- - -## Image Audit - -**Current Images** in `docs/assets/`: -- ✅ `architecture-diagram.png` (97KB) - Current, good quality -- ✅ `macOS_accessibility.png` (94KB) - Current, good quality -- ✅ `macOS_input_monitoring.png` (94KB) - Current, good quality -- ✅ `macOS_permissions_alert.png` (88KB) - Current, good quality -- ✅ `macOS_screen_recording.png` (107KB) - Current, good quality - -**Status**: All existing images are professional macOS permission screenshots and are current. No immediate replacement needed. - -**New Screenshots Needed** (via autogeneration script): -1. CLI examples (5 scenarios) -2. Viewer interfaces (3-5 viewers) -3. Segmentation viewer (3 states) -4. Training dashboards -5. Benchmark results - -**Recommendation**: Run screenshot generation script once viewers are available: -```bash -python docs/_scripts/generate_docs_screenshots.py --save-metadata -``` - ---- - -## Quality Improvements - -### Content Accuracy -- ✅ Removed misleading bold formatting that caused rendering issues -- ✅ Added documentation for features implemented in January 2026 -- ✅ Updated APIs with current function signatures -- ✅ Added proper cross-references between packages - -### Consistency -- ✅ Consistent code block formatting across all pages -- ✅ Standardized section headers (## level) -- ✅ Consistent CLI command examples -- ✅ Uniform Python API examples - -### Completeness -- ✅ Episode segmentation (previously undocumented) -- ✅ Recording catalog (previously undocumented) -- ✅ Advanced search (previously undocumented) -- ✅ Component library (previously minimal) -- ✅ Screenshot automation (previously missing) - -### Accessibility -- ✅ Added "NEW (January 2026)" tags for recent features -- ✅ Added mermaid diagrams for visual explanation -- ✅ Included JSON schemas for data formats -- ✅ Added practical usage examples - ---- - -## Testing Recommendations - -### Before Merging -1. **Build Test**: - ```bash - cd /Users/abrichr/oa/src/OpenAdapt - mkdocs build --strict - ``` - Should complete with no warnings or errors. - -2. **Local Preview**: - ```bash - mkdocs serve - # Visit http://localhost:8000 - ``` - Check: - - [ ] No stars render in "software adapter" text - - [ ] All new sections render correctly - - [ ] Mermaid diagrams display - - [ ] Code blocks have proper syntax highlighting - - [ ] Internal links work - - [ ] Navigation is functional - -3. **Link Validation**: - ```bash - # Install link checker - pip install linkchecker - - # Build and check - mkdocs build - linkchecker site/ - ``` - -4. **Responsive Testing**: - - Test on mobile viewport (375px) - - Test on tablet viewport (768px) - - Test on desktop (1920px) - -### After Merging -1. **Verify Deployment**: - - Visit https://docs.openadapt.ai - - Check homepage shows "software adapter" (no stars) - - Verify new package documentation sections appear - - Test search functionality - -2. **Screenshot Generation**: - ```bash - # Generate actual screenshots - cd /Users/abrichr/oa/src/OpenAdapt - python docs/_scripts/generate_docs_screenshots.py --save-metadata - - # Commit screenshots - git add docs/assets/screenshots/ - git commit -m "Add autogenerated documentation screenshots" - ``` - ---- - -## Next Steps - -### Immediate (This PR) -- [x] Fix critical stars issue -- [x] Create screenshot autogeneration script -- [x] Update viewer documentation -- [x] Update ML documentation -- [ ] Test build with `mkdocs build --strict` -- [ ] Create PR with all changes - -### Short-Term (Next PR) -- [ ] Run screenshot generation script once viewers are generated -- [ ] Add screenshots to package pages -- [ ] Update Getting Started guide with screenshots -- [ ] Add CLI examples with terminal screenshots - -### Medium-Term (Future PRs) -- [ ] Implement `aggregate_docs.py` for auto-syncing sub-package READMEs -- [ ] Add `validate_links.py` for CI/CD link checking -- [ ] Add `test_examples.py` to verify code examples work -- [ ] Create architecture diagram autogeneration from code -- [ ] Add video tutorials/demos - -### Long-Term (Ongoing) -- [ ] Keep package docs in sync with sub-package READMEs -- [ ] Generate changelog from git commits -- [ ] Add API reference with mkdocstrings -- [ ] Create interactive examples -- [ ] Add benchmark result visualizations - ---- - -## References - -### Related Agent Work -- **Agent a4441ff**: Initial docs.openadapt.ai review -- **Agent a2097db**: Phase 1 implementation (outdated now) -- **This agent**: Comprehensive update with screenshot automation - -### Related Issues -- User reported: "Stars are unacceptable" (fixed) -- User reported: "Images are questionable" (audited, current ones are good) -- Requested: "Apply screenshot autogeneration approach" (implemented) - -### Related Files -- `/Users/abrichr/oa/src/openadapt-viewer/scripts/generate_comprehensive_screenshots.py` - Reference implementation (450 lines) -- `/Users/abrichr/oa/src/openadapt-viewer/CLAUDE.md` - Viewer documentation and patterns -- `/Users/abrichr/oa/src/OpenAdapt/docs/CONTRIBUTING_DOCS.md` - Documentation guidelines -- `/Users/abrichr/oa/src/OpenAdapt/docs/DOCUMENTATION_AUTOMATION_ANALYSIS.md` - Analysis of docs system - ---- - -## Success Metrics - -### Before This Update -- ❌ Critical rendering issue (stars showing) -- ⚠️ Episode segmentation undocumented -- ⚠️ Recording catalog undocumented -- ⚠️ Advanced search undocumented -- ⚠️ Component library minimal -- ⚠️ No screenshot automation -- ⚠️ Viewer docs incomplete (136 lines) -- ⚠️ ML docs missing new features (155 lines) - -### After This Update -- ✅ Critical rendering issue fixed -- ✅ Episode segmentation fully documented -- ✅ Recording catalog fully documented -- ✅ Advanced search fully documented -- ✅ Component library comprehensive -- ✅ Screenshot automation implemented -- ✅ Viewer docs comprehensive (336 lines, +148%) -- ✅ ML docs current and complete (277 lines, +79%) - -### Measurable Improvements -- **Content Volume**: +650 lines of new documentation -- **Feature Coverage**: 5 new features documented -- **Package Docs**: +148% expansion (viewer), +79% expansion (ML) -- **Automation**: 450-line screenshot generation system -- **Critical Bugs**: 1 fixed (stars rendering) - ---- - -## Conclusion - -This update successfully addresses the critical `**adapt**` stars issue and brings documentation in line with the latest OpenAdapt capabilities. The new screenshot autogeneration system provides a maintainable, automated approach to keeping documentation visuals current and professional. - -The comprehensive updates to viewer and ML documentation ensure that new features (episode segmentation, recording catalog, advanced search) are properly documented with examples, schemas, and usage patterns. - -All changes follow the established documentation patterns and are ready for review and merging. - ---- - -**Generated**: 2026-01-17 -**Agent**: Claude Sonnet 4.5 -**Status**: Ready for PR diff --git a/docs/architecture-evolution.md b/docs/architecture-evolution.md index 683229491..8f0f4e188 100644 --- a/docs/architecture-evolution.md +++ b/docs/architecture-evolution.md @@ -1130,7 +1130,8 @@ flowchart TB ## Appendix B: Related Documents - [Architecture Overview](./architecture.md) - Package structure and data flow -- [Roadmap Priorities](./roadmap-priorities.md) - Current development priorities +- [Current product documentation](https://docs.openadapt.ai) - Supported + installation, qualification, deployment, and reference material - [Package Documentation](./packages/index.md) - Individual package guides - [Legacy Freeze](./legacy/freeze.md) - Migration from monolith diff --git a/docs/design/INDEX.md b/docs/design/INDEX.md index e8d53db5c..069fa608b 100644 --- a/docs/design/INDEX.md +++ b/docs/design/INDEX.md @@ -1,11 +1,12 @@ -# Design Documents +# Historical Design Documents -This section contains design documents and specifications for OpenAdapt features and components. +These documents preserve pre-1.0 implementation history. They are not the +current product architecture, roadmap, or source for docs.openadapt.ai. Start +with the [current documentation](https://docs.openadapt.ai) or the canonical +[`openadapt-flow`](https://github.com/OpenAdaptAI/openadapt-flow) engine. ## Documents - [System Tray App](openadapt-tray.md) - Design specification for the OpenAdapt system tray application - [Tray Logging](tray-logging.md) - Logging architecture for the tray application - [Telemetry](telemetry-design.md) - Telemetry and analytics design -- [Landing Page Strategy](landing-page-strategy.md) - Website landing page strategy and messaging -- [Repo Rename Analysis](repo-rename-analysis.md) - Analysis for repository renaming considerations diff --git a/docs/design/landing-page-strategy.md b/docs/design/landing-page-strategy.md deleted file mode 100644 index 53b6dca74..000000000 --- a/docs/design/landing-page-strategy.md +++ /dev/null @@ -1,754 +0,0 @@ -# OpenAdapt.ai Landing Page Strategy - -**Document Version**: 1.0 -**Date**: January 2026 -**Author**: Generated with AI assistance -**Status**: Proposal for Review - ---- - -## Table of Contents - -1. [Current State Analysis](#1-current-state-analysis) -2. [Target Audience Definitions](#2-target-audience-definitions) -3. [Core Messaging Strategy](#3-core-messaging-strategy) -4. [Competitive Positioning](#4-competitive-positioning) -5. [Page Section Recommendations](#5-page-section-recommendations) -6. [Copy Suggestions](#6-copy-suggestions) -7. [Wireframe Concepts](#7-wireframe-concepts) -8. [Social Proof Strategy](#8-social-proof-strategy) -9. [Call-to-Action Strategy](#9-call-to-action-strategy) -10. [Implementation Priorities](#10-implementation-priorities) - ---- - -## 1. Current State Analysis - -### 1.1 What OpenAdapt IS Today - -OpenAdapt has evolved from a monolithic application (v0.46.0) to a **modular meta-package architecture** (v1.0+). This is a significant architectural maturation that should be reflected in messaging. - -**Core Value Proposition (Current Reality)**: -- The open source software adapter between Large Multimodal Models (LMMs) and desktop/web GUIs -- Record demonstrations, train models, evaluate agents via unified CLI -- Works with any VLM: Claude, GPT-4V, Gemini, Qwen, or custom fine-tuned models - -**Technical Differentiators (Verified)**: -1. **Model Agnostic**: Not locked to one AI provider -2. **Demo-Prompted, Not User-Prompted**: Learn from human demonstration, not complex prompt engineering -3. **Universal GUI Support**: Native apps, web browsers, virtualized environments -4. **Open Source (MIT License)**: Full transparency, no vendor lock-in - -**Key Innovation**: -- **Trajectory-conditioned disambiguation of UI affordances** — the only open-source CUA framework that conditions agents on recorded demonstrations at runtime (validated: 46.7% → 100% first-action accuracy) -- **Specialization over scale** — fine-tuned Qwen3-VL-2B outperforms Claude Sonnet 4.5 and GPT-5.1 on action accuracy (42.9% vs 11.2% vs 23.2%) on an internal benchmark -- **Capture-to-deployment pipeline** — record → retrieve → train → deploy. [OpenCUA](https://github.com/xlang-ai/OpenCUA) (NeurIPS 2025 Spotlight) [reused OpenAdapt's macOS accessibility capture code](https://arxiv.org/html/2508.09123v3) in their AgentNetTool -- **Set-of-Marks (SoM) mode**: 100% accuracy on synthetic benchmarks using element IDs instead of coordinates - -### 1.2 Current Landing Page Assessment - -**What's Working Well**: -- Clean, professional design with dark theme -- Video demo at hero section -- GitHub star/fork buttons for social proof -- Platform-specific installation instructions (auto-detects OS) -- PyPI download statistics showing traction -- Industry use cases grid (HR, Law, Insurance, etc.) -- Email signup for updates - -**What's Missing or Unclear**: -1. **No clear "what is this?"** - Visitors need to watch a video to understand -2. **Tagline "AI for Desktops" is vague** - Doesn't differentiate from competitors -3. **No comparison to alternatives** - Why choose OpenAdapt over Anthropic Computer Use? -4. **No technical credibility indicators** - No benchmark scores, no research citations -5. **Industry grid is generic** - Same features could apply to any automation tool -6. **No developer/researcher angle** - Focuses only on end-user automation -7. **Architecture transition is hidden** - v1.0+ modular design is a major selling point -8. **No clear "Who is this for?"** - Tries to appeal to everyone, resonates with no one - -**Carousel Messages Analysis**: -- "Show, don't tell." - Good but cryptic -- "Perform, don't prompt." - Best differentiator, should be prominent -- "Record, replay, and share." - Functional but not compelling - -### 1.3 Technical Accuracy Issues - -The current site doesn't reflect: -- The modular package architecture (7 focused sub-packages) -- The evaluation infrastructure (WAA, WebArena benchmarks) -- The ML training capabilities (VLM fine-tuning, LoRA) -- The retrieval-augmented prompting (demo library search) -- The privacy scrubbing capabilities (PII/PHI redaction) - ---- - -## 2. Target Audience Definitions - -### 2.1 Primary Audiences - -#### A. Developers Building Automation Agents - -**Profile**: -- Building AI-powered tools that interact with GUIs -- May be creating internal tools, startup products, or client solutions -- Comfortable with Python, CLI tools, ML concepts -- Want flexibility, not black-box solutions - -**Pain Points**: -- API-only agents (Claude Computer Use) lack customization -- Building from scratch is too slow -- Need to run locally for privacy/security -- Want to fine-tune models on specific workflows - -**What They Need to See**: -- Clear architecture diagrams -- Code examples (pip install, quick start) -- Benchmark scores vs. alternatives -- Extensibility points (adapters, plugins) - -**Key Message**: "The open source SDK for building GUI automation agents" - -#### B. Enterprise Process Automation Buyers - -**Profile**: -- Looking to automate repetitive knowledge work -- Concerned about security, privacy, compliance -- Need to justify ROI and integrate with existing systems -- Often have IT/security review requirements - -**Pain Points**: -- Existing RPA is brittle and expensive to maintain -- Cloud-only AI raises data privacy concerns -- Need clear enterprise support options -- Require compliance with industry regulations - -**What They Need to See**: -- Privacy features (PII/PHI scrubbing) -- On-premise deployment options -- Enterprise support/contact information -- Industry-specific use case studies -- Security architecture information - -**Key Message**: "AI-first automation that runs where your data lives" - -#### C. ML Researchers Studying GUI Agents - -**Profile**: -- Academic researchers or industry R&D teams -- Working on VLM capabilities, agent architectures, benchmarks -- Need reproducible baselines and evaluation infrastructure -- Want to contribute to or build upon open research - -**Pain Points**: -- Existing benchmarks are hard to set up -- Need standardized evaluation metrics -- Want to compare models fairly -- Limited open-source alternatives to proprietary agent frameworks - -**What They Need to See**: -- Benchmark integration (WAA, WebArena, OSWorld) -- Published metrics and methodology -- Research paper citations (if any) -- Clear contribution pathways -- Schema/data format documentation - -**Key Message**: "Open infrastructure for GUI agent research and benchmarking" - -#### D. ML Engineers Interested in VLM Fine-Tuning - -**Profile**: -- Want to train custom models for specific GUI tasks -- Familiar with training infrastructure (LoRA, PEFT, etc.) -- Looking for training data and pipelines -- Want efficient local or cloud training options - -**Pain Points**: -- Collecting GUI interaction data is tedious -- Setting up VLM training pipelines is complex -- Need baselines to compare against -- Cloud GPU costs add up quickly - -**What They Need to See**: -- Training pipeline documentation -- Supported models (Qwen3-VL, etc.) -- Training results (before/after fine-tuning) -- Cloud GPU integration (Lambda Labs, Azure) -- Data format specifications - -**Key Message**: "Record demonstrations, train specialized GUI agents" - -### 2.2 Audience Prioritization - -For the landing page, prioritize in this order: -1. **Developers** (highest volume, most likely to convert to users/contributors) -2. **Enterprise buyers** (revenue potential, require dedicated section) -3. **ML engineers** (overlaps with developers, training angle) -4. **Researchers** (smaller audience, but important for credibility) - ---- - -## 3. Core Messaging Strategy - -### 3.1 Primary Tagline Options - -**Option A (Recommended)**: -> **"Teach AI to use any software."** - -Why: Simple, benefit-focused, implies the key differentiator (demonstration-based learning) - -**Option B**: -> **"The open source adapter between AI and any GUI."** - -Why: Explains the technical position, highlights open source - -**Option C**: -> **"Perform, don't prompt."** - -Why: Clever contrast to prompt engineering, memorable - -**Option D**: -> **"Record. Train. Automate."** - -Why: Clear 3-step process, action-oriented - -### 3.2 Supporting Taglines (Subheadlines) - -- "Show AI how to do a task once. Let it handle the rest." -- "From human demonstration to AI automation in minutes." -- "Open source GUI automation with the AI model of your choice." -- "Works with Claude, GPT-4V, Gemini, Qwen, or your own fine-tuned models." - -### 3.3 Key Differentiators to Emphasize - -1. **Capture-to-Deployment Pipeline** - - Not: "Prompt the AI to do your task" - - But: "Record the task once. OpenAdapt handles the rest — retrieval, training, deployment." - - Proof: 7 modular packages (capture, ML, evals, grounding, retrieval, privacy, viewer); OpenCUA (NeurIPS 2025) [reused OpenAdapt's macOS a11y capture code](https://arxiv.org/html/2508.09123v3) - -2. **Demonstration-Conditioned Agents** - - Not: "Zero-shot reasoning about what to click" - - But: "Agents conditioned on relevant demos — at inference AND during training" - - Proof: 46.7% → 100% first-action accuracy with demo conditioning (validated, n=45). No other open-source CUA framework does runtime demo conditioning. - - Note: This is first-action accuracy on tasks sharing a navigation entry point. Multi-step and cross-domain evaluation is ongoing on Windows Agent Arena. - -3. **Specialization Over Scale** - - Not: "Use the biggest model available" - - But: "A 2B model fine-tuned on your workflows outperforms frontier models" - - Proof: Qwen3-VL-2B (42.9%) vs Claude Sonnet 4.5 (11.2%) on action accuracy (internal benchmark, synthetic login task) - -4. **Model Agnostic** - - Not: "Works with [specific AI]" - - But: "Your choice: Claude, GPT-4V, Gemini, Qwen, or custom models" - - Proof: Adapters for multiple VLM backends - -5. **Runs Anywhere & Open Source** - - Not: "Cloud-powered automation" / "Try our free tier" - - But: "Run locally, in the cloud, or hybrid. MIT licensed, fully transparent." - - Proof: CLI-based, works offline; GitHub, PyPI, active Discord - -### 3.4 Messaging Framework - -**For Developers**: -> "Build GUI automation agents with a modular Python SDK. Record demonstrations, train models, evaluate on benchmarks. Works with any VLM." - -**For Enterprise**: -> "AI-first process automation that learns from your team. Privacy-first architecture with PII/PHI scrubbing. Deploy where your data lives." - -**For Researchers**: -> "Open infrastructure for GUI agent research. Standardized benchmarks, reproducible baselines, extensible architecture." - -**For ML Engineers**: -> "Fine-tune VLMs on real GUI workflows. Record data, train with LoRA, evaluate accuracy. Local or cloud training." - ---- - -## 4. Competitive Positioning - -### 4.1 Primary Competitors (Updated March 2026) - -| Competitor | Strengths | Weaknesses | Our Advantage | -|------------|-----------|------------|---------------| -| **Anthropic Computer Use** | 72.5% OSWorld (near-human), simple API | Proprietary, cloud-only, no customization, per-action cost | Open source, model-agnostic, trainable, runs locally | -| **Agent S3 (Simular)** | 72.6% OSWorld, open source | Zero-shot only, no demo conditioning, no fine-tuning pipeline | Demo-conditioned agents, capture-to-train pipeline | -| **OpenCUA (XLANG Lab)** | NeurIPS Spotlight, 45% OSWorld, open models (7B-72B) | Zero-shot at inference — demos used only for training, not runtime | Runtime demo conditioning (unique); OpenCUA reused OpenAdapt's macOS a11y code | -| **Browser Use** | 50k+ GitHub stars, 89% WebVoyager | Browser-only, no desktop, no training pipeline | Full desktop support, fine-tuning, demo library | -| **UI-TARS (ByteDance)** | Local models (2B-72B), Apache 2.0 | No demo conditioning, no capture pipeline | End-to-end record→train→deploy, demo retrieval | -| **CUA / Bytebot** | Container infra, YC-backed | Infrastructure-only, no ML training pipeline | Full pipeline: capture + train + eval + deploy | -| **Traditional RPA (UiPath, etc.)** | Enterprise-proven, UiPath Screen Agent #1 on OSWorld | Brittle selectors, expensive ($10K+/yr), requires scripting | AI-first, learns from demos, open source | - -### 4.2 Positioning Statement - -> "OpenAdapt is the **open source alternative** to proprietary GUI automation APIs. Unlike cloud-only solutions, OpenAdapt lets you **train custom models** on your workflows and **deploy anywhere**. Unlike traditional RPA, OpenAdapt uses **AI reasoning** and **learns from demonstrations** instead of brittle scripts." - -### 4.3 Comparison Talking Points - -**vs. Anthropic Computer Use**: -- "Model-agnostic - not locked to one provider" -- "Fine-tune on your specific workflows" -- "Run locally for privacy-sensitive data" -- "Open source with community contributions" - -**vs. Traditional RPA**: -- "AI understands intent, not just element selectors" -- "Adapts to UI changes without manual updates" -- "Learn from demonstrations, not scripting" -- "Fraction of the cost, faster to deploy" - ---- - -## 5. Page Section Recommendations - -### 5.1 Proposed Page Structure - -1. **Hero Section** (Above the fold) -2. **How It Works** (3-step process) -3. **Key Differentiators** (3-4 value props) -4. **For Developers** (SDK/CLI features) -5. **For Enterprise** (Security, privacy, support) -6. **Use Cases** (Specific, concrete examples) -7. **Comparison** (Why OpenAdapt) -8. **Social Proof** (Metrics, testimonials, logos) -9. **Getting Started** (Install, docs, community) -10. **Footer** (Links, legal, contact) - -### 5.2 Hero Section Redesign - -**Current**: "OpenAdapt.AI - AI for Desktops. Automate your workflows. No coding required." - -**Proposed**: - -``` -[Logo] OpenAdapt.AI - -# Teach AI to use any software. - -Show it once. Let it handle the rest. - -[Video Demo - Keep current] - -[Install in 30 seconds] [View on GitHub] [Join Discord] - -"Works with Claude, GPT-4V, Gemini, Qwen, or your own fine-tuned models" - -{GitHub Stars} {PyPI Downloads} {Discord Members} -``` - -### 5.3 How It Works Section - -**Current**: Carousel with "Show, don't tell" / "Perform, don't prompt" / "Record, replay, share" - -**Proposed**: Clear 3-step process with visuals - -``` -## How OpenAdapt Works - -1. RECORD - [Icon: Screen recording] - Demonstrate the task by doing it yourself. - OpenAdapt captures screenshots, mouse clicks, and keystrokes. - -2. TRAIN - [Icon: Neural network] - Train an AI model on your demonstration. - Fine-tune Qwen-VL, use Claude/GPT-4V, or bring your own model. - -3. DEPLOY - [Icon: Play button] - Run the trained agent to automate the task. - Evaluate with standardized benchmarks. -``` - -### 5.4 Differentiators Section - -``` -## Why OpenAdapt? - -### Record Once, Automate Forever -Capture any workflow. OpenAdapt retrieves relevant demos to guide agents -AND trains specialized models on your recordings. -[Stat: 46.7% → 100% first-action accuracy with demo conditioning] - -### Small Models, Big Results -A 2B model fine-tuned on your workflows outperforms frontier models. -Specialization beats scale for GUI tasks. -[Stat: 42.9% action accuracy (Qwen 2B FT) vs 11.2% (Claude Sonnet 4.5)] - -### Model Agnostic & Open Source -Your choice of AI: Claude, GPT-4V, Gemini, Qwen-VL, or fine-tune your own. -MIT licensed. Run locally, in the cloud, or hybrid. -``` - -### 5.5 For Builders Section - -```` -## Built for Builders - -### Show it once. Done. -Record yourself doing a task. OpenAdapt handles the rest. -No code, no prompts, no configuration. - -### Three commands -```bash -pip install openadapt -openadapt capture start --name my-task # Record -openadapt run --capture my-task # Replay with AI -``` - -### Works with the AI you already use -Claude, GPT-4V, Gemini, Qwen — pick your model. -Or let OpenAdapt train a small one that runs on your laptop. - -### Your data stays yours -Everything runs locally. Nothing leaves your machine unless you want it to. -```` - -### 5.6 For Developers Section - -```` -## Built for Developers - -### Modular Architecture -Seven focused packages you can install individually: -- openadapt-capture: Recording -- openadapt-ml: Training & inference -- openadapt-evals: Benchmarking -- openadapt-viewer: Visualization -- openadapt-grounding: UI element detection -- openadapt-retrieval: Demo library search -- openadapt-privacy: PII/PHI scrubbing - -### Quick Start -```bash -# Install -pip install openadapt[all] - -# Record a demonstration -openadapt capture start --name my-task - -# Train a model -openadapt train start --capture my-task --model qwen3vl-2b - -# Evaluate -openadapt eval run --checkpoint model.pt --benchmark waa -``` - -### Benchmark Ready -Integrated with Windows Agent Arena (WAA), WebArena, and OSWorld. -Compare your models against published baselines. - -[View Documentation] [GitHub Repository] -```` - -### 5.7 For Enterprise Section - -``` -## Enterprise-Ready Automation - -### Privacy First -Built-in PII/PHI scrubbing with AWS Comprehend, Microsoft Presidio, or Private AI. -Your sensitive data never leaves your infrastructure. - -### Deploy Your Way -Run entirely on-premise, in your cloud, or hybrid. -No data leaves your environment unless you want it to. - -### Compliance Ready -Audit logging, reproducible recordings, explainable AI decisions. -Built for regulated industries. - -### Enterprise Support -Custom development, training, and support packages available. - -[Contact Sales: sales@openadapt.ai] -``` - -### 5.8 Use Cases Section (Refined) - -**Current**: Generic industry grid - -**Proposed**: Specific, concrete use cases with workflows - -``` -## Real-World Automation - -### Data Entry Across Systems -Transfer information between applications that don't integrate. -Example: Copy customer data from CRM to billing system. - -### Report Generation -Compile data from multiple sources into standardized reports. -Example: Monthly sales reports from Salesforce + Excel + internal tools. - -### Legacy System Integration -Automate workflows in applications without APIs. -Example: Mainframe data entry, proprietary healthcare systems. - -### Quality Assurance Testing -Record manual test procedures, replay with validation. -Example: Regression testing across UI updates. - -### Process Documentation -Record workflows to create training materials automatically. -Example: Onboarding guides for complex internal tools. -``` - ---- - -## 6. Copy Suggestions - -### 6.1 Headlines - -| Section | Headline | Subheadline | -|---------|----------|-------------| -| Hero | "Teach AI to use any software." | "Show it once. Let it handle the rest." | -| How It Works | "Three Steps to Automation" | "Record, train, deploy." | -| Differentiators | "Why OpenAdapt?" | "Open source, model-agnostic, demonstration-based." | -| Developers | "Built for Developers" | "A modular SDK for building GUI automation agents." | -| Enterprise | "Enterprise-Ready" | "AI automation that runs where your data lives." | -| Use Cases | "Automate Any Workflow" | "From data entry to testing to legacy integration." | -| Install | "Get Started in 30 Seconds" | "One command installs everything you need." | - -### 6.2 CTAs (Calls to Action) - -| Context | Primary CTA | Secondary CTA | -|---------|-------------|---------------| -| Hero | "Get Started" | "View Demo" | -| Developers | "View Documentation" | "Star on GitHub" | -| Enterprise | "Contact Sales" | "Download Whitepaper" | -| Footer | "Join Discord" | "View on GitHub" | - -### 6.3 Proof Points to Include - -- "46.7% → 100% first-action accuracy with demo conditioning (n=45, same model, no training)" -- "Fine-tuned 2B model outperforms Claude Sonnet 4.5 on action accuracy (42.9% vs 11.2%, internal benchmark)" -- "OpenCUA (NeurIPS 2025 Spotlight) reused OpenAdapt's macOS accessibility capture code in AgentNetTool" -- "Only open-source CUA framework with runtime demo-conditioned inference" -- "[X,XXX] PyPI downloads this month" (dynamic) -- "[XXX] GitHub stars" (dynamic) -- "7 modular packages, 1 unified CLI" -- "Integrated with Windows Agent Arena, WebArena, OSWorld benchmarks" -- "MIT licensed, fully open source" - -**Honesty notes for proof points**: -- The 46.7%→100% result is first-action only on 45 macOS tasks sharing the same navigation entry point -- The 42.9% vs 11.2% result is on a controlled internal synthetic login benchmark (~3 UI elements) -- Multi-step episode success on real-world benchmarks (WAA) is under active evaluation -- Frame these as "validated signal" not "production-proven" - ---- - -## 7. Wireframe Concepts - -### 7.1 Desktop Layout - -``` -+------------------------------------------------------------------+ -| [Logo] [Docs] [GitHub] [Discord] [Enterprise] | -+------------------------------------------------------------------+ -| | -| # Teach AI to use any software. | -| Show it once. Let it handle the rest. | -| | -| [==================== Video Demo ====================] | -| | -| [Get Started] [View on GitHub] | -| | -| Works with: [Claude] [GPT-4V] [Gemini] [Qwen] [Custom] | -| | -| [GitHub Stars] [PyPI Downloads] [Discord Members] | -| | -+------------------------------------------------------------------+ -| | -| ## How OpenAdapt Works | -| | -| [1. RECORD] [2. TRAIN] [3. DEPLOY] | -| [Screenshot] [Neural Net] [Automation] | -| Demonstrate Train on your Run the agent | -| the task. demonstration. to automate. | -| | -+------------------------------------------------------------------+ -| | -| ## Why OpenAdapt? | -| | -| [Demo-Based] [Model Agnostic] [Run Anywhere] [Open Source] | -| Learn from Your choice of Local, cloud, MIT licensed | -| examples. AI provider. or hybrid. forever. | -| | -+------------------------------------------------------------------+ -| | -| [For Developers Tab] [For Enterprise Tab] [For Researchers Tab]| -| | -| Content switches based on selected audience... | -| | -+------------------------------------------------------------------+ -| | -| ## Get Started | -| | -| [macOS] [Windows] [Linux] | -| | -| $ curl -LsSf https://astral.sh/uv/install.sh | sh | -| $ uv tool install openadapt | -| $ openadapt --help | -| | -| [X,XXX installs this month] | -| | -+------------------------------------------------------------------+ -| | -| [Footer: Links, Social, Legal] | -| | -+------------------------------------------------------------------+ -``` - -### 7.2 Mobile Considerations - -- Stack hero elements vertically -- Collapse model logos into scrollable row -- Use accordion for audience tabs -- Keep video demo prominent -- Simplify code blocks (single command with copy button) - ---- - -## 8. Social Proof Strategy - -### 8.1 Metrics to Display - -**Live Metrics** (fetch from APIs): -- GitHub stars (currently showing, keep) -- PyPI downloads per month (currently showing, keep) -- Discord member count (add if available) -- Number of GitHub contributors (add) - -**Static Metrics** (update manually): -- "7 modular packages" -- "100% synthetic benchmark accuracy (SoM mode)" -- "3 benchmark integrations (WAA, WebArena, OSWorld)" - -### 8.2 Testimonials Strategy - -**Priority Order**: -1. Named enterprise user quotes (if available) -2. Named developer testimonials from Discord -3. Anonymous industry testimonials -4. Community member quotes - -**Template for Gathering**: -> "How has OpenAdapt helped you? Reply to be featured on our website." - -### 8.3 Logo Wall - -**Target logos to seek permission for**: -- Companies using OpenAdapt in production -- Universities using for research -- Partner organizations - -**Fallback** (if no logos available): -- Featured in media logos (if covered) -- Integration partner logos (AWS, Azure, etc.) -- "Trusted by teams at Fortune 500 companies" (if true) - ---- - -## 9. Call-to-Action Strategy - -### 9.1 Primary Conversion Goals - -1. **GitHub star** (low friction, high visibility) -2. **PyPI install** (product usage) -3. **Discord join** (community engagement) -4. **Email signup** (for updates) -5. **Enterprise contact** (revenue) - -### 9.2 CTA Placement - -| Location | Primary CTA | Secondary CTA | -|----------|-------------|---------------| -| Hero | "Get Started" -> Install section | "View on GitHub" | -| After video | "Try it yourself" -> Install | "Join Discord" | -| Developers section | "View Docs" | "Star on GitHub" | -| Enterprise section | "Contact Sales" | "Request Demo" | -| Bottom of page | "Join Discord" | "View Documentation" | -| Sticky header (scroll) | "Get Started" | | - -### 9.3 Email Capture Strategy - -**Current**: "Register for updates" - -**Proposed**: More specific value prop -- "Get early access to new features" -- "Join [X,XXX] developers automating with AI" -- "Subscribe to the OpenAdapt newsletter (monthly, no spam)" - ---- - -## 10. Implementation Priorities - -### 10.1 Phase 1: Quick Wins (1-2 weeks) - -1. **Update hero tagline** to "Teach AI to use any software." -2. **Add "How It Works" section** with 3-step process -3. **Update differentiators** to 4-card grid (current features but better copy) -4. **Add Discord member count** to social proof -5. **Add GitHub contributors count** - -### 10.2 Phase 2: Messaging Clarity (2-4 weeks) - -1. **Add "For Developers" section** with code examples and architecture -2. **Add "For Enterprise" section** with privacy/security messaging -3. **Replace generic industry grid** with specific use case examples -4. **Add comparison table** vs. alternatives -5. **Update email signup copy** to be more specific - -### 10.3 Phase 3: Credibility Building (4-8 weeks) - -1. **Add benchmark scores** (once published) -2. **Collect and display testimonials** -3. **Create case studies** (1-2 real examples) -4. **Add logo wall** (if logos available) -5. **Add "Research" or "Publications" section** (if applicable) - -### 10.4 Phase 4: Conversion Optimization (Ongoing) - -1. **A/B test hero messaging** -2. **Track install conversion rates** -3. **Optimize CTA placement** -4. **Add video transcripts/captions for SEO** -5. **Create landing page variants** for different audiences (developer vs. enterprise) - ---- - -## Appendix A: Messaging Don'ts - -- **Don't say "AI for Desktops"** - too vague, doesn't differentiate -- **Don't say "No coding required"** - true for end users, but alienates developers -- **Don't list every industry** - pick 3-4 with real stories -- **Don't hide the CLI** - developers want to see it -- **Don't over-promise** - be honest about current capabilities - -## Appendix B: Technical Content to Add - -1. **Architecture diagram** showing package relationships -2. **Mermaid flowchart** of Record -> Train -> Deploy cycle -3. **Comparison table** of model backends (Claude, GPT, Qwen, etc.) -4. **Benchmark table** showing accuracy scores -5. **API reference link** to documentation site - -## Appendix C: SEO Keywords - -Primary: -- "GUI automation AI" -- "desktop automation AI" -- "RPA alternative AI" -- "VLM GUI agent" -- "open source computer use" - -Secondary: -- "train AI on screenshots" -- "demonstration-based automation" -- "model-agnostic automation" -- "Claude computer use alternative" -- "AI workflow automation" - ---- - -*This document is a living strategy guide. Updates should be made as OpenAdapt capabilities evolve and as user feedback is collected.* diff --git a/docs/design/production-execution-design.md b/docs/design/production-execution-design.md deleted file mode 100644 index 027f32dda..000000000 --- a/docs/design/production-execution-design.md +++ /dev/null @@ -1,724 +0,0 @@ -# Production Execution Design: From Benchmarks to Real-World Automation - -**Version**: 1.0 -**Date**: January 2026 -**Status**: Proposal - ---- - -## Executive Summary - -This document addresses a critical gap in the OpenAdapt architecture: the **EXECUTE phase** is currently limited to benchmark evaluation (`openadapt-evals`), but production automation requires additional capabilities beyond measuring performance. This proposal synthesizes industry best practices from leading GUI automation frameworks to define what "production execution" means and how OpenAdapt should support it. - -**Key Recommendation**: Create a new `openadapt-agent` package for production execution, keeping `openadapt-evals` focused on benchmarking. The safety module in `openadapt-ml` should be extracted and shared. - ---- - -## Table of Contents - -1. [Problem Statement](#1-problem-statement) -2. [Literature Review](#2-literature-review) -3. [Gap Analysis](#3-gap-analysis) -4. [Architectural Options](#4-architectural-options) -5. [Recommendation](#5-recommendation) -6. [README Improvement Proposal](#6-readme-improvement-proposal) -7. [Implementation Roadmap](#7-implementation-roadmap) -8. [References](#8-references) - ---- - -## 1. Problem Statement {#1-problem-statement} - -### Current Architecture - -OpenAdapt's three-phase pipeline: - -``` -RECORD (capture) --> TRAIN (ml) --> EXECUTE (evals) -``` - -**The Problem**: `openadapt-evals` serves benchmark evaluation but lacks production execution capabilities. - -| Capability | Benchmark Evaluation | Production Execution | -|-----------|---------------------|---------------------| -| **Purpose** | Measure agent performance | Automate real user tasks | -| **Environment** | Controlled VMs, sandboxes | Real user machines | -| **Stakes** | Low (test data) | High (real data, actions) | -| **Human Oversight** | Optional, for debugging | Required, for safety | -| **Error Recovery** | Restart from checkpoint | Graceful degradation, undo | -| **Logging** | Metrics collection | Audit trail, compliance | - -### Why This Matters - -1. **User Expectation Gap**: Users expect "execute" to mean "run my agent on my tasks," not just "benchmark my agent" -2. **Safety Requirements**: Production execution on real systems requires safety gates not needed in sandboxed benchmarks -3. **Feature Completeness**: Without production execution, OpenAdapt is a training/eval tool, not an automation solution - ---- - -## 2. Literature Review {#2-literature-review} - -### 2.1 Microsoft UFO (2024-2025) - -**Overview**: UFO evolved through three versions from a GUI Agent (UFO, Feb 2024) to a Desktop AgentOS (UFO2, Apr 2025) to Multi-Device Orchestration (UFO3, Nov 2025). - -**Key Architecture Patterns**: -- **HostAgent + AppAgents**: Centralized task decomposition with application-specialized execution -- **Hybrid Control Detection**: Fuses Windows UI Automation (UIA) with vision-based parsing -- **Picture-in-Picture (PiP)**: Isolated virtual desktop allowing agents and users to operate concurrently without interference - -**Production vs Evaluation**: -- UFO2 evaluated across 20+ Windows applications, demonstrating improvements over "conceptual prototype" CUAs -- Most CUAs remain "hindered by shallow OS integration, fragile screenshot-based interaction, and disruptive execution" -- Windows Agent Arena (WAA) provides benchmark environment: 154 tasks across 15 applications - -**Key Insight**: Deep OS integration unlocks scalable, reliable automation. Screenshot-only approaches are fragile. - -**Source**: [UFO Documentation](https://microsoft.github.io/UFO/) - -### 2.2 Claude Computer Use (Anthropic, 2024-2025) - -**Overview**: Production-grade VLM agent API with explicit safety guardrails. - -**Safety Architecture**: -- **Human-in-the-Loop**: Required for sensitive tasks (financial transfers, system deletions) -- **Hallucinated Click Problem**: Vision models misinterpret buttons or get stuck in loops -- **Tiered Workflows**: - - Low-risk tasks (documentation): Highly autonomous - - High-risk tasks (triage suggestions): Mandatory human review - -**Production Safeguards**: -- Pre-deployment testing to verify training holds under pressure -- Detection methods for misuse (spam generation discovered pre-launch) -- Option to disable tool for accounts showing misuse -- Built-in VM isolation for Cowork desktop agent -- Support for browser automation with explicit approvals for high-impact actions - -**Security Incidents**: -- Chinese state-backed hacking group used Claude for "first documented case of a large-scale cyberattack executed without substantial human intervention" -- Prompt injection attacks remain an active area of development - -**Key Insight**: "Agent safety - the task of securing Claude's real-world actions - is still an active area of development." - -**Source**: [Anthropic Building Safeguards for Claude](https://www.anthropic.com/news/building-safeguards-for-claude) - -### 2.3 OSWorld (CMU, NeurIPS 2024) - -**Overview**: First scalable, real computer environment for multimodal agents across Ubuntu, Windows, macOS. - -**Benchmark Characteristics**: -- 369 computer tasks involving real web and desktop apps -- Open domains with OS file I/O and multi-application workflows -- Execution-based evaluation with reproducible setup - -**Performance Reality Check**: -- Humans: 72.36% success -- Best AI model: 12.24% success (as of 2024) -- Agent S2 with Claude 3.7: 34.5% (as of late 2025) -- Primary failures: GUI grounding and operational knowledge - -**Infrastructure**: -- Docker support for virtualized platforms -- AWS/Azure parallel evaluation (reduces evaluation to < 1 hour) -- Host-Client architecture for large-scale evaluation - -**Key Insight**: Real-world computer tasks remain challenging. Best models achieve <35% of human performance. - -**Source**: [OSWorld GitHub](https://github.com/xlang-ai/OSWorld) - -### 2.4 WebArena & ST-WebAgentBench (2024) - -**Overview**: WebArena provides web automation benchmarks; ST-WebAgentBench extends with safety/trust evaluation. - -**Progress**: -- 2-year leap from 14% to ~60% success rate on WebArena -- Convergence on "standard model" architecture: Planner + Executor + Memory - -**Safety Gap Identified**: -- Traditional benchmarks focus only on task success -- ST-WebAgentBench adds: - - **Completion Under Policy (CuP)**: Success while following organizational rules - - **Risk Ratio**: Proportion of unsafe actions -- Most state-of-the-art agents "not yet enterprise-ready" due to policy violations - -**Key Insight**: Task success != production readiness. Policy compliance and trustworthiness are separate dimensions. - -**Source**: [ST-WebAgentBench Paper](https://arxiv.org/html/2410.06703v2) - -### 2.5 OpenCUA (2025) - -**Overview**: Open-source framework achieving state-of-the-art among open-source models. - -**Key Innovation**: Chain-of-Thought (CoT) augmentation of trajectories -- Planning, memory, and reflection in structured reasoning -- OpenCUA-72B: 45% on OSWorld-Verified (new SOTA for open-source) -- OpenCUA-32B: Surpassed GPT-4o-based CUA - -**Agent Architecture**: -- Iterative perception via screenshots -- Reflective long CoT as inner monologue -- Predicted next action execution - -**Source**: [OpenCUA GitHub](https://github.com/xlang-ai/OpenCUA) - -### 2.6 Industry State (2025) - -**Market Reality**: -- Global agentic AI market: $7.6 billion (2025) -- Only 2% of organizations deployed at scale -- 61% stuck in exploration phases -- Gartner: >40% projects will be canceled by 2027 due to costs, unclear value, or inadequate risk controls - -**Key Insight**: The gap between experimentation and production is a primary blocker. Framework selection is an architectural decision, not a library decision. - ---- - -## 3. Gap Analysis {#3-gap-analysis} - -### 3.1 What OpenAdapt Has - -| Component | Package | Status | -|-----------|---------|--------| -| **Safety Validator** | `openadapt-ml/safety/` | Implemented | -| **Pattern Detection** | `openadapt-ml/safety/patterns.py` | Implemented | -| **Confirmation Handlers** | `openadapt-ml/safety/confirmation.py` | Implemented | -| **Benchmark Runner** | `openadapt-evals/benchmarks/runner.py` | Implemented | -| **Execution Traces** | `openadapt-evals/benchmarks/data_collection.py` | Implemented | -| **Live Tracking** | `openadapt-evals/benchmarks/live_tracker.py` | Implemented | - -### 3.2 What's Missing for Production - -#### 3.2.1 Safety Gates (Partial) - -**Exists** (`openadapt-ml/safety/`): -- `SafetyValidator`: Pattern-based action validation -- `SafetyConfig`: Mode configuration (DISABLED, PERMISSIVE, STANDARD, STRICT, PARANOID) -- Pattern categories: DESTRUCTIVE, SYSTEM, IRREVERSIBLE, CREDENTIAL, NETWORK, FILE_SYSTEM -- `ConfirmationHandler`: CLI, callback, queue-based confirmation - -**Missing**: -- Integration with execution loop (exists in isolation, not wired to runner) -- GUI confirmation handlers (falls back to CLI with warning) -- Learning from confirmations (user overrides don't persist across sessions) -- Dynamic risk assessment based on context - -#### 3.2.2 Human-in-the-Loop Confirmation - -**Exists**: -- `CLIConfirmationHandler`: Text-based prompts -- `CallbackConfirmationHandler`: Custom integration hooks -- `QueueConfirmationHandler`: Batch processing - -**Missing**: -- GUI dialog integration (noted as TODO) -- Tray application integration for desktop notifications -- Mobile/web confirmation for remote operation -- Timeout policies (exists but crude - blocks entire execution) -- Escalation paths (always block if no confirmation) - -#### 3.2.3 Rollback/Undo Capability - -**Completely Missing**: -- No state snapshots before action execution -- No undo registry for reversible actions -- No checkpoint/restore mechanism -- No transaction boundaries - -**Industry Comparison**: -- LangGraph: Built-in checkpointers saving state after each step -- UFO: PiP isolation prevents interference with user session -- Claude Desktop: VM isolation for Cowork agent - -**Real-World Incidents Showing Need**: -- Replit AI deleted user's production database, then incorrectly advised rollback was impossible -- Gemini CLI agent deleted user files with no recovery - -#### 3.2.4 Session Management - -**Partially Exists** (scattered): -- `ValidationContext` tracks state visits and user overrides -- `EvaluationConfig` has run_name and output_dir - -**Missing**: -- Persistent session store (sessions lost on restart) -- Session pause/resume -- Session sharing/handoff -- Session templates for common workflows -- Session isolation (multi-user scenarios) - -#### 3.2.5 Error Recovery - -**Exists** (basic): -- Try/except in runner with error logging -- `BenchmarkResult.error` field for failure capture - -**Missing**: -- Retry policies (exponential backoff, max retries) -- Fallback strategies (alternative action paths) -- Degraded mode operation (continue with reduced capabilities) -- Error categorization (transient vs permanent) -- Recovery suggestions - -#### 3.2.6 Logging/Audit Trail - -**Exists** (for benchmarks): -- Execution traces with screenshots -- Metrics collection -- Live tracking JSON - -**Missing for Production**: -- Structured audit logs (who, what, when, why) -- Compliance-ready formats (SOC2, HIPAA considerations) -- Log retention policies -- Tamper-evident logging -- Real-time alerting on anomalies - -### 3.3 Capability Matrix - -| Capability | Benchmark (evals) | Production (needed) | Gap | -|-----------|------------------|--------------------|----| -| Execute actions | Yes | Yes | - | -| Safety validation | No (in ml) | Yes | Integration | -| Human confirmation | No | Yes | New | -| Rollback/undo | No | Yes | New | -| Session persistence | No | Yes | New | -| Error recovery | Basic | Advanced | Enhancement | -| Audit logging | Basic | Compliance-grade | Enhancement | -| Isolation | Sandbox assumed | User system | Architecture | -| Concurrent operation | No | Yes (PiP-style) | New | - ---- - -## 4. Architectural Options {#4-architectural-options} - -### Option A: Rename/Expand openadapt-evals - -**Approach**: Add production execution capabilities to `openadapt-evals`, rename to `openadapt-execute` or keep name. - -**Pros**: -- Single package for all execution -- Simpler dependency graph -- Existing infrastructure (runner, traces, tracking) - -**Cons**: -- Violates single-responsibility principle -- Benchmark users get production deps they don't need -- Production users get benchmark deps they don't need -- Name confusion: "evals" implies evaluation, not execution -- Risk of breaking existing benchmark integrations - -**Package Structure**: -``` -openadapt-evals/ - benchmarks/ # Keep existing - production/ # New - runner.py # Production execution loop - safety.py # Wire safety from ml - session.py # Session management - recovery.py # Error recovery - shared/ # Common utilities -``` - -### Option B: Create openadapt-agent - -**Approach**: New package specifically for production agent execution. - -**Pros**: -- Clear separation of concerns -- Users install what they need -- Independent release cycles -- "agent" is industry-standard terminology -- Can depend on both evals (for runner patterns) and ml (for safety) - -**Cons**: -- Another package to maintain -- Some code duplication initially -- Users may confuse agent vs evals - -**Package Structure**: -``` -openadapt-agent/ - runtime/ - executor.py # Production execution loop - session.py # Session management - recovery.py # Error recovery - safety/ # Extract from ml or import - validator.py - confirmation.py - patterns.py - persistence/ - state.py # Checkpoint/restore - audit.py # Audit logging - ui/ - confirmation.py # GUI confirmation dialogs - tray.py # System tray integration -``` - -**Dependency Graph**: -``` -openadapt-agent - ├── openadapt-ml (for safety, models) - ├── openadapt-grounding (for element detection) - ├── openadapt-capture (for observation capture) - └── openadapt-evals (optional, for testing) -``` - -### Option C: Keep Execution in openadapt-ml - -**Approach**: Add production execution to `openadapt-ml` as part of the model runtime. - -**Pros**: -- Safety module already there -- Policy and execution naturally coupled -- Fewer packages overall - -**Cons**: -- `ml` becomes a monolith -- Users wanting production execution must install training deps -- Naming confusion: "ml" doesn't suggest execution -- Benchmark evals would need to depend on ml for production features - -**Package Structure**: -``` -openadapt-ml/ - runtime/ # Existing - policy.py - execution/ # New - runner.py - session.py - recovery.py - safety/ # Existing - validator.py - confirmation.py -``` - -### Option D: Hybrid - Shared Safety Package - -**Approach**: Extract safety into `openadapt-safety`, create `openadapt-agent` for execution. - -**Pros**: -- Maximum modularity -- Safety can be used by any package -- Clear responsibility boundaries - -**Cons**: -- Most complex dependency graph -- Another package to publish/maintain -- Possibly over-engineered for current scale - -**Dependency Graph**: -``` -openadapt-safety (new) - └── (minimal deps) - -openadapt-agent (new) - ├── openadapt-safety - ├── openadapt-grounding - └── openadapt-capture - -openadapt-ml - └── openadapt-safety - -openadapt-evals - └── openadapt-safety (optional) -``` - ---- - -## 5. Recommendation {#5-recommendation} - -### Primary Recommendation: Option B (openadapt-agent) - -**Rationale**: - -1. **Clear Terminology**: "agent" is the industry-standard term for autonomous execution systems -2. **Separation of Concerns**: Benchmarking and production execution have different requirements -3. **User Clarity**: `pip install openadapt-agent` clearly communicates intent -4. **Safety Integration**: Can import from `openadapt-ml` initially, extract later if needed -5. **Pragmatic**: Avoids over-engineering while maintaining modularity - -### Phase 1: Create openadapt-agent (Q1 2026) - -``` -openadapt-agent/ - __init__.py - runtime/ - __init__.py - executor.py # Production execution loop - session.py # Session management - state.py # Checkpoint/restore - safety/ - __init__.py # Re-exports from openadapt-ml - recovery/ - __init__.py - retry.py # Retry policies - fallback.py # Fallback strategies - persistence/ - __init__.py - audit.py # Structured audit logging - store.py # Session persistence - cli.py # CLI entry point -``` - -**Minimal Dependencies**: -```toml -[project] -dependencies = [ - "openadapt-ml", # For safety, models - "openadapt-grounding", # For element detection - "openadapt-capture", # For observations -] - -[project.optional-dependencies] -tray = ["openadapt-tray"] # Desktop integration -full = ["openadapt-evals"] # Include benchmark testing -``` - -### Phase 2: Enhance Safety Integration - -1. Wire `SafetyValidator` into production executor -2. Add GUI confirmation handler via `openadapt-tray` -3. Implement user override persistence -4. Add risk-based escalation paths - -### Phase 3: Implement Rollback/Undo - -1. State snapshots before action execution -2. Undo registry for reversible actions -3. Checkpoint/restore mechanism -4. Transaction boundaries - -### CLI Integration - -Update main `openadapt` CLI: - -``` -openadapt agent start --task "Process invoices" # Start production agent -openadapt agent status # Check running agent -openadapt agent pause # Pause execution -openadapt agent resume # Resume execution -openadapt agent stop # Stop agent - -openadapt agent session list # List sessions -openadapt agent session resume # Resume session - -openadapt eval run --benchmark waa # Benchmarking (unchanged) -``` - ---- - -## 6. README Improvement Proposal {#6-readme-improvement-proposal} - -### Current Issues - -1. **Too Verbose**: 443 lines, overwhelming for newcomers -2. **Buried Value Prop**: Core innovation (demo-conditioned automation) lost in details -3. **Phase Confusion**: EXECUTE = evals, not production automation -4. **Package Overload**: 7 packages listed upfront - -### Proposed Structure - -```markdown -# OpenAdapt: Show, Don't Tell - -**Learn from demonstrations. Automate with AI.** - -OpenAdapt watches you work, learns your patterns, and automates your tasks. - -## Quick Start - -```bash -pip install openadapt[all] - -# 1. Record -openadapt capture start --name my-task - -# 2. Train -openadapt train --capture my-task - -# 3. Run -openadapt agent start --task my-task -``` - -## How It Works - -``` -[Record Demo] --> [Train Model] --> [Run Agent] - | | | - Capture Learn from Automate - your actions your patterns your tasks -``` - -**The key insight**: Instead of writing prompts, you *demonstrate*. The AI learns from what you do, not what you say. - -## Installation - -| Goal | Install Command | -|------|-----------------| -| Everything | `pip install openadapt[all]` | -| Just record & train | `pip install openadapt[core]` | -| Evaluate against benchmarks | `pip install openadapt[evals]` | -| Production automation | `pip install openadapt[agent]` | - -## Packages - -| Phase | Package | Purpose | -|-------|---------|---------| -| **RECORD** | openadapt-capture | Screen/event capture | -| **TRAIN** | openadapt-ml | Model training | -| **EXECUTE** | openadapt-agent | Production automation | -| | openadapt-evals | Benchmark evaluation | - -*Optional*: openadapt-grounding, openadapt-retrieval, openadapt-privacy, openadapt-viewer - -## Safety - -Production automation includes: -- Pre-action validation (dangerous pattern detection) -- Human-in-the-loop confirmation for high-risk actions -- Audit logging for compliance -- Session management with pause/resume - -## Links - -- [Documentation](https://docs.openadapt.ai) -- [Discord](https://discord.gg/yF527cQbDG) -- [Architecture](../architecture-evolution.md) - -## License - -MIT -``` - -### Key Changes - -1. **"Show, Don't Tell" as Headline**: Captures the core differentiator -2. **3-Step Quick Start**: Immediate value demonstration -3. **Visual Flow**: ASCII diagram showing the pipeline -4. **Clear Phase-to-Package Mapping**: EXECUTE includes both agent AND evals -5. **Safety Section**: Addresses enterprise concerns upfront -6. **Reduced Length**: ~100 lines vs 443 - ---- - -## 7. Implementation Roadmap {#7-implementation-roadmap} - -### Q1 2026: Foundation - -| Week | Deliverable | -|------|-------------| -| 1-2 | Create `openadapt-agent` package skeleton | -| 3-4 | Implement basic `Executor` with safety integration | -| 5-6 | Add session management (start, pause, resume, stop) | -| 7-8 | CLI integration and documentation | - -### Q2 2026: Production Features - -| Week | Deliverable | -|------|-------------| -| 1-2 | GUI confirmation handlers via tray | -| 3-4 | Checkpoint/restore mechanism | -| 5-6 | Retry policies and fallback strategies | -| 7-8 | Audit logging for compliance | - -### Q3 2026: Enterprise Readiness - -| Week | Deliverable | -|------|-------------| -| 1-4 | Concurrent operation (PiP-style isolation) | -| 5-8 | Undo registry for reversible actions | - -### Success Metrics - -| Metric | Target | -|--------|--------| -| Time to first successful production run | < 10 minutes | -| Safety validation coverage | 100% of high-risk actions | -| Session recovery success rate | > 95% | -| User confirmation response time | < 30 seconds | - ---- - -## 8. References {#8-references} - -### GUI Automation Frameworks - -1. **Microsoft UFO**: [Documentation](https://microsoft.github.io/UFO/) | [GitHub](https://github.com/microsoft/UFO) -2. **Claude Computer Use**: [Anthropic Safety](https://www.anthropic.com/news/building-safeguards-for-claude) -3. **OSWorld**: [Paper](https://arxiv.org/abs/2404.07972) | [GitHub](https://github.com/xlang-ai/OSWorld) -4. **WebArena**: [Website](https://webarena.dev/) -5. **ST-WebAgentBench**: [Paper](https://arxiv.org/html/2410.06703v2) -6. **OpenCUA**: [GitHub](https://github.com/xlang-ai/OpenCUA) - -### Safety Research - -7. **Remediation: What happens after AI goes wrong?**: [Jack Vanlightly Blog](https://jack-vanlightly.com/blog/2025/7/28/remediation-what-happens-after-ai-goes-wrong) -8. **Agentic AI Frameworks: Complete Enterprise Guide 2025**: [SpaceO](https://www.spaceo.ai/blog/agentic-ai-frameworks/) - -### OpenAdapt Internal - -9. **Architecture Evolution**: `/docs/architecture-evolution.md` -10. **Safety Module**: `/openadapt-ml/openadapt_ml/safety/` -11. **Evaluation Runner**: `/openadapt-evals/openadapt_evals/benchmarks/runner.py` - ---- - -## Appendix A: Safety Module Analysis - -The existing safety module in `openadapt-ml/openadapt_ml/safety/` provides: - -### Validation Flow - -```python -from openadapt_ml.safety import SafetyValidator, SafetyConfig, ValidationDecision - -validator = SafetyValidator(SafetyConfig()) - -result = validator.validate(action, observation) - -if result.allowed: - execute(action) -elif result.requires_confirmation: - if confirm_with_user(result): - execute(action) -else: # blocked - log_blocked_action(result.reason) -``` - -### Pattern Categories - -| Category | Examples | Default Behavior | -|----------|----------|------------------| -| DESTRUCTIVE | rm, delete, drop, format | Block | -| SYSTEM | sudo, admin, reboot | Block | -| IRREVERSIBLE | send, submit, purchase | Confirm | -| CREDENTIAL | password, secret, key | Confirm | -| NETWORK | upload, post, send | Confirm | -| FILE_SYSTEM | write, modify, create | Allow (log) | - -### Safety Modes - -| Mode | Description | -|------|-------------| -| DISABLED | Bypass all safety (testing only) | -| PERMISSIVE | Log warnings, allow most | -| STANDARD | Block dangerous, confirm irreversible | -| STRICT | Block suspicious, confirm most | -| PARANOID | Block everything not explicitly allowed | - ---- - -## Appendix B: Glossary - -| Term | Definition | -|------|------------| -| **CUA** | Computer-Using Agent - AI system that controls desktop/web UIs | -| **HITL** | Human-in-the-Loop - requiring human approval for actions | -| **PiP** | Picture-in-Picture - isolated virtual desktop for agent operation | -| **Rollback** | Reverting system state to a previous checkpoint | -| **Session** | A bounded execution context with state persistence | -| **Safety Gate** | Pre-action validation layer preventing harmful operations | - ---- - -*Document prepared by Claude Opus 4.5 for OpenAdapt project. Last updated: January 2026.* diff --git a/docs/design/repo-rename-analysis.md b/docs/design/repo-rename-analysis.md deleted file mode 100644 index e66dac023..000000000 --- a/docs/design/repo-rename-analysis.md +++ /dev/null @@ -1,286 +0,0 @@ -# Repository Rename Analysis: OpenAdapt to openadapt - -**Date:** January 2026 -**Status:** Decision Document -**Author:** Engineering Team - ---- - -## Executive Summary - -This document analyzes whether to rename the main OpenAdapt GitHub repository from `OpenAdapt` (mixed case) to `openadapt` (lowercase) to align with Python conventions and existing sub-packages. - -**Recommendation: DO NOT RENAME at this time.** - -The costs and risks of renaming outweigh the benefits. The minor consistency improvement does not justify the potential for broken links, documentation updates, and brand dilution. - ---- - -## Current State - -| Component | Current Name | Case | -|-----------|-------------|------| -| **Main Repository** | `OpenAdaptAI/OpenAdapt` | Mixed | -| **GitHub Organization** | `OpenAdaptAI` | Mixed | -| **Sub-packages** | `openadapt-ml`, `openadapt-capture`, etc. | Lowercase | -| **PyPI Package** | `openadapt` | Lowercase | -| **Python Imports** | `import openadapt` | Lowercase | -| **pyproject.toml Repository URL** | Already points to `openadapt` (lowercase) | Lowercase | - -**Key Observation:** The `pyproject.toml` already uses lowercase in the Repository URL: -```toml -Repository = "https://github.com/OpenAdaptAI/openadapt" -``` - -This suggests the team anticipated or intended lowercase naming, but GitHub currently shows `OpenAdapt`. - ---- - -## Industry Research: How Major Python Projects Handle Repository Naming - -| Project | Organization | Repository | PyPI Package | Notes | -|---------|-------------|------------|--------------|-------| -| **LangChain** | `langchain-ai` | `langchain` | `langchain` | All lowercase | -| **PyTorch** | `pytorch` | `pytorch` | `torch` | All lowercase | -| **TensorFlow** | `tensorflow` | `tensorflow` | `tensorflow` | All lowercase | -| **Hugging Face** | `huggingface` | `transformers` | `transformers` | All lowercase | -| **FastAPI** | `tiangolo` | `fastapi` | `fastapi` | All lowercase | -| **scikit-learn** | `scikit-learn` | `scikit-learn` | `scikit-learn` | All lowercase with hyphen | - -**Conclusion:** The overwhelming convention in Python open-source projects is **all lowercase** for repository names. - ---- - -## GitHub Redirect Behavior - -Based on [GitHub's documentation](https://docs.github.com/en/repositories/creating-and-managing-repositories/renaming-a-repository): - -### What Gets Redirected (Indefinitely) -- Web traffic to the old URL -- `git clone`, `git fetch`, `git push` operations -- Issues, wikis, stars, followers - -### What Breaks Immediately -- **GitHub Actions** referencing the repository by name will fail with "repository not found" -- **GitHub Pages** custom domain URLs are not automatically redirected - -### Redirect Persistence -- Redirects persist **indefinitely** unless: - 1. A new repository is created with the old name - 2. GitHub support is asked to remove them - -### Important Warning -From [GitHub Community discussions](https://github.com/orgs/community/discussions/22669): "If you create a new repository under your account in the future, do not reuse the original name of the renamed repository. If you do, redirects to the renamed repository will no longer work." - ---- - -## Detailed Analysis - -### Arguments FOR Renaming to Lowercase - -| Argument | Weight | Rationale | -|----------|--------|-----------| -| **Consistency with sub-packages** | Medium | All sub-packages use lowercase (`openadapt-ml`, `openadapt-capture`, etc.) | -| **Python convention** | Medium | Standard practice in Python ecosystem (see industry research) | -| **PyPI alignment** | Medium | Package name is `openadapt` (lowercase) | -| **Import alignment** | Low | `import openadapt` works regardless of repo name | -| **URL simplicity** | Low | `github.com/OpenAdaptAI/openadapt` slightly cleaner | -| **Already in pyproject.toml** | High | Repository URL already shows lowercase intent | - -### Arguments AGAINST Renaming - -| Argument | Weight | Rationale | -|----------|--------|-----------| -| **Brand recognition** | High | "OpenAdapt" as two words (Open + Adapt) reinforces brand identity | -| **Breaking changes risk** | High | External links, bookmarks, documentation, blog posts, academic citations | -| **GitHub org inconsistency** | Medium | Organization is `OpenAdaptAI` (mixed case) - renaming repo creates inconsistency | -| **Documentation updates** | Medium | 1,343 occurrences of "OpenAdapt" across 78 files need review | -| **SEO impact** | Medium | Existing search rankings tied to "OpenAdapt" | -| **Minimal actual benefit** | High | GitHub URLs are case-insensitive for access purposes | -| **Legacy code references** | Medium | Legacy directory has extensive "OpenAdapt" references | - ---- - -## Technical Impact Assessment - -### Files Requiring Updates if Renamed - -Based on codebase analysis: - -| Category | File Count | Occurrences | Update Required? | -|----------|------------|-------------|------------------| -| Documentation (*.md) | 37 | ~200+ | Review each | -| GitHub workflows (*.yml) | 10 | ~50+ | Critical review | -| Python source files | 15 | ~50+ | Review imports | -| Configuration files | 5 | ~20+ | Review URLs | -| Legacy code | 20+ | ~900+ | May leave as-is | - -### CI/CD Impact - -Current workflows use relative paths and don't hard-code the repository name, so **minimal CI/CD impact expected**. - -However, any external workflows or actions referencing `OpenAdaptAI/OpenAdapt` would need updates. - -### Impact on Forks and Clones - -- **Existing clones:** Continue working via redirects, but should update with `git remote set-url` -- **Existing forks:** Maintain their existing names and remotes -- **New forks:** Would fork from the new lowercase name - ---- - -## Risk Assessment - -| Risk | Likelihood | Impact | Mitigation | -|------|------------|--------|------------| -| Broken external links | Medium | Medium | GitHub redirects handle most cases | -| Academic citation issues | Low | Medium | Papers cite DOIs or specific versions | -| SEO ranking drop | Low | Low | Temporary if any; redirects preserve link equity | -| User confusion | Medium | Low | Clear communication and documentation | -| GitHub Actions failures | Low | High | Audit and update before rename | -| Brand dilution | Medium | Medium | None - cannot mitigate if lowercase chosen | - ---- - -## Alternative Approaches - -### Option A: Do Nothing (RECOMMENDED) -- Keep repository as `OpenAdapt` -- Accept minor inconsistency with sub-packages -- No risk, no disruption - -### Option B: Rename to Lowercase -- Change repository to `openadapt` -- Update documentation -- Communicate to users -- Accept brand/visual trade-off - -### Option C: Rename Organization and Repository -- Change `OpenAdaptAI` to `openadaptai` -- Change `OpenAdapt` to `openadapt` -- Complete consistency, but much higher disruption -- **NOT RECOMMENDED** - organization rename is significantly more disruptive - -### Option D: Create Alias via Transfer -- Transfer repository to a new `openadapt` repo -- Keep `OpenAdapt` as a redirect-only stub -- **NOT RECOMMENDED** - unnecessarily complex - ---- - -## Recommendation - -**Recommendation: Do Not Rename (Option A)** - -### Rationale - -1. **GitHub URLs are case-insensitive** - Users can access via `github.com/OpenAdaptAI/openadapt` or `github.com/openadaptai/OpenAdapt` interchangeably - -2. **Brand value** - "OpenAdapt" with capitalization clearly shows the "Open" + "Adapt" word composition, which is meaningful for the project's identity - -3. **Risk/benefit ratio** - The benefits are cosmetic while the risks (broken links, confusion, documentation churn) are concrete - -4. **Organization inconsistency** - Renaming only the repo while keeping `OpenAdaptAI` creates a new inconsistency - -5. **Industry examples** - While most Python projects use lowercase, several successful projects (like early versions of major projects) maintained mixed-case names without issue - -6. **pyproject.toml already lowercase** - The `Repository` URL in `pyproject.toml` already shows lowercase, providing implicit consistency for programmatic access - ---- - -## If Renaming is Chosen: Migration Plan - -Should the decision be made to rename despite the recommendation, here is the migration plan: - -### Phase 1: Preparation (1 week before) -1. Audit all GitHub Actions and CI/CD workflows -2. Document all external references (blog posts, papers, etc.) -3. Prepare communication for Discord and mailing lists -4. Create redirect documentation - -### Phase 2: Execution (Day of) -1. Perform the rename via GitHub Settings -2. Update `pyproject.toml` repository URL (if needed) -3. Update README.md badge URLs -4. Push updated documentation - -### Phase 3: Communication (Day of + 1 week) -1. Announce on Discord -2. Post on social media -3. Email contributors -4. Update any linked resources - -### Phase 4: Follow-up (1 month) -1. Monitor for broken links -2. Update external documentation (readthedocs, etc.) -3. Check Google Search Console for indexing issues - ---- - -## Timeline - -| Milestone | Date | Notes | -|-----------|------|-------| -| Decision | TBD | Pending team discussion | -| If renaming: Preparation | T+0 to T+7 days | Audit and documentation | -| If renaming: Execution | T+7 days | Actual rename | -| If renaming: Stabilization | T+7 to T+30 days | Monitor and fix issues | - ---- - -## Conclusion - -While lowercase repository naming is the Python convention and would create better consistency with sub-packages, the **costs outweigh the benefits** for the main OpenAdapt repository. The recommendation is to **keep the current `OpenAdapt` naming** for the following key reasons: - -1. Brand recognition and identity -2. Risk of breaking external references -3. GitHub URLs are case-insensitive anyway -4. Organization name would remain inconsistent regardless -5. The `pyproject.toml` already uses lowercase, providing programmatic consistency - -If consistency is deemed critical in the future, consider renaming the organization and all repositories together as a single coordinated effort, rather than piecemeal changes. - ---- - -## References - -- [GitHub: Renaming a repository](https://docs.github.com/en/repositories/creating-and-managing-repositories/renaming-a-repository) -- [GitHub Community: How long does GitHub forward renamed repos?](https://github.com/orgs/community/discussions/22669) -- [GitHub Community: Duration of Web Traffic Redirection](https://github.com/orgs/community/discussions/110367) -- [LangChain GitHub](https://github.com/langchain-ai/langchain) -- [Hugging Face Transformers](https://github.com/huggingface/transformers) - ---- - -## Appendix A: Files Containing "OpenAdapt" References - -Key files with the highest occurrence counts: - -| File | Count | Notes | -|------|-------|-------| -| `legacy/CHANGELOG.md` | 911 | Historical, may leave unchanged | -| `README.md` | 21 | Brand mentions, badges | -| `docs/contributing.md` | 18 | Contribution guidelines | -| `legacy/build.py` | 19 | Build scripts | -| `docs/design/landing-page-strategy.md` | 20 | Strategy document | -| `docs/architecture-evolution.md` | 14 | Architecture docs | - -Total: **1,343 occurrences across 78 files** - ---- - -## Appendix B: Sub-package Repository Naming - -All sub-packages follow lowercase convention: - -| Repository | PyPI Package | -|------------|--------------| -| `openadapt-capture` | `openadapt-capture` | -| `openadapt-ml` | `openadapt-ml` | -| `openadapt-evals` | `openadapt-evals` | -| `openadapt-viewer` | `openadapt-viewer` | -| `openadapt-grounding` | `openadapt-grounding` | -| `openadapt-retrieval` | `openadapt-retrieval` | -| `openadapt-privacy` | `openadapt-privacy` | - -This consistency is desirable but not critical enough to justify renaming the main repository. diff --git a/docs/publication-roadmap.md b/docs/publication-roadmap.md deleted file mode 100644 index 2d11844de..000000000 --- a/docs/publication-roadmap.md +++ /dev/null @@ -1,736 +0,0 @@ -# OpenAdapt Publication Roadmap: A Critical Assessment - -**Version**: 2.0 -**Date**: January 2026 -**Status**: Honest Evaluation -**Author**: OpenAdapt Research Team - ---- - -## Preamble: Intellectual Honesty - -This document is written from the perspective of a skeptical reviewer at a top venue. The goal is not to inflate claims but to identify what is genuinely publishable, what experiments are actually needed, and what timeline is realistic given current resources. - -**Guiding principle**: Better to publish a solid workshop paper than to submit an overreaching main track paper that gets rejected. - ---- - -## Table of Contents - -1. [Current State of Evidence](#1-current-state-of-evidence) -2. [Honest Contribution Assessment](#2-honest-contribution-assessment) -3. [Weakness Analysis](#3-weakness-analysis) -4. [Required Experiments for Defensible Claims](#4-required-experiments-for-defensible-claims) -5. [Statistical Rigor Requirements](#5-statistical-rigor-requirements) -6. [Related Work Gap Analysis](#6-related-work-gap-analysis) -7. [Venue Fit Analysis](#7-venue-fit-analysis) -8. [Realistic Timeline](#8-realistic-timeline) -9. [Risk Mitigation](#9-risk-mitigation) -10. [Action Items](#10-action-items) -11. [Path to Main Track Publication (Parallel Track)](#11-path-to-main-track-publication-parallel-track) - ---- - -## 1. Current State of Evidence - -### 1.1 What We Actually Have - -| Experiment | n | Result | Statistical Validity | Benchmark | -|------------|---|--------|---------------------|-----------| -| macOS demo-conditioning (first-action) | 45 | 46.7% -> 100% | **Moderate** (single model, single platform) | Non-standard | -| WAA baseline (interrupted) | 8 | 12.5% success | **Weak** (incomplete, agent bugs) | Standard | -| Length-matched control | 45 | 57.8% | **Useful** (rules out token length) | Non-standard | - -### 1.2 Critical Assessment of Current Results - -**The 100% first-action accuracy claim**: -- **Scope**: All 45 tasks share the SAME correct first action (click Apple menu) -- **Implication**: This measures whether a demo can transfer procedural entry points, NOT general task-solving -- **Limitation**: Not comparable to any published benchmark -- **Honest framing**: "Demo-conditioning eliminates spatial bias in navigation initialization" - -**The WAA baseline**: -- **Status**: 1/8 tasks passed (12.5%) -- **Problem**: Run was interrupted; agent had bugs unrelated to our method -- **Implication**: We do not yet have a clean zero-shot baseline on a standard benchmark - -### 1.3 What We Do NOT Have - -1. **Standard benchmark results** - No complete WAA, WebArena, or OSWorld evaluation -2. **Multi-model comparison** - Only Claude Sonnet 4.5 tested -3. **Episode success rate** - Only first-action accuracy measured -4. **Statistical significance tests** - No p-values, confidence intervals, or effect sizes -5. **Ablation studies** - No systematic ablation of demo components -6. **Retrieval experiments** - Retrieval system not evaluated -7. **User studies** - No human evaluation of system usability - ---- - -## 2. Honest Contribution Assessment - -### 2.1 What Is ACTUALLY Novel? - -| Claimed Contribution | Novelty Assessment | Prior Work | -|---------------------|-------------------|------------| -| Demo-conditioned GUI agents | **Moderate** - PbD is old; VLM+demo is emerging | UINav (2023), SUGILITE (2017) | -| "Show don't tell" paradigm | **Low** - Standard few-shot prompting | GPT-3 (2020), chain-of-thought | -| Multimodal demo retrieval | **Moderate** - Novel application to GUI domain | RAG literature extensive | -| Modular architecture | **Low** - Engineering contribution | Many open-source frameworks | -| Cross-platform support | **Low** - Engineering contribution | SeeAct, UFO also support multiple platforms | - -### 2.2 Defensible Novel Claims - -After honest assessment, the defensible novel contribution is: - -> **Demonstration-conditioned prompting for VLM-based GUI agents**: We show that providing a human demonstration in the VLM prompt substantially improves action selection accuracy compared to instruction-only prompting. This is a *prompting strategy*, not a new model architecture or training method. - -**This is NOT**: -- A new model architecture -- A training/fine-tuning method -- A new benchmark -- A theoretical contribution - -### 2.3 Contribution Positioning - -**Honest positioning**: This is an **empirical study** showing that a simple prompting intervention (including demonstrations) improves GUI agent performance. The contribution is: - -1. **Empirical finding**: Demonstrations help, and we quantify by how much -2. **Analysis**: We explain WHY (spatial bias, procedural priors) -3. **Practical method**: We provide an open-source implementation - -**What reviewers will say**: "This is straightforward few-shot prompting applied to GUI agents. What is technically novel?" - -**Our response must be**: "The contribution is empirical, not algorithmic. We systematically evaluate demo-conditioning across N tasks and M models, providing the first rigorous study of this prompting strategy for GUI automation." - ---- - -## 3. Weakness Analysis - -### 3.1 Anticipated Reviewer Criticisms - -| Criticism | Severity | Our Current Status | Mitigation | -|-----------|----------|-------------------|------------| -| "All tasks share the same first action" | **Critical** | True - intentional design | Expand to diverse first actions | -| "Only one model tested" | **High** | True | Add GPT-4V, Gemini | -| "Non-standard benchmark" | **High** | True | Complete WAA evaluation | -| "No episode success rate" | **High** | True | Run multi-step evaluation | -| "Small sample size" | **Medium** | n=45 is reasonable | Add more tasks | -| "No statistical tests" | **Medium** | True | Add McNemar's test, bootstrap CI | -| "Limited to English/macOS" | **Medium** | True | Acknowledge as limitation | -| "Retrieval system not evaluated" | **Medium** | True | Either evaluate or remove claims | -| "No comparison to fine-tuning" | **Medium** | True | Acknowledge; position as prompt-only | -| "Engineering contribution, not research" | **Low** | Partially true | Emphasize empirical findings | - -### 3.2 Weaknesses We CANNOT Fix Before Submission - -1. **Fundamental novelty** - Demo-conditioning is not architecturally novel -2. **Benchmark saturation** - If WAA shows <20% improvement, contribution weakens -3. **Single-domain focus** - GUI automation is narrow; no multi-domain transfer - -### 3.3 Weaknesses We CAN Fix - -1. **Benchmark coverage** - Run complete WAA evaluation (1-2 weeks) -2. **Multi-model comparison** - Add GPT-4V, Gemini (1 week) -3. **Statistical rigor** - Add proper tests (1-2 days) -4. **Diverse first actions** - Design new task set (1 week) -5. **Episode success** - Extend evaluation (1 week) - ---- - -## 4. Required Experiments for Defensible Claims - -### 4.1 Minimum Viable Experiments (for Workshop Paper) - -| Experiment | Tasks | Models | Trials/Task | Total Runs | Effort | -|------------|-------|--------|-------------|------------|--------| -| WAA zero-shot baseline | 20 | 2 | 3 | 120 | 1 week | -| WAA demo-conditioned | 20 | 2 | 3 | 120 | 1 week | -| **Total** | 20 | 2 | 6 | 240 | 2 weeks | - -**Why 3 trials per task?** -- GUI actions have stochasticity (model sampling, UI timing) -- Enables variance estimation and significance testing -- Standard practice in agent evaluation literature - -### 4.2 Full Conference Paper Requirements - -| Experiment | Tasks | Models | Trials | Total Runs | Effort | -|------------|-------|--------|--------|------------|--------| -| WAA evaluation | 50+ | 3 | 3 | 450+ | 3 weeks | -| WebArena evaluation | 100+ | 2 | 3 | 600+ | 4 weeks | -| Ablation: demo format | 20 | 1 | 3 | 60 | 1 week | -| Ablation: demo length | 20 | 1 | 3 | 60 | 1 week | -| Ablation: # demos (k=1,3,5) | 20 | 1 | 3 | 180 | 2 weeks | -| Cross-task transfer | 20 | 1 | 3 | 60 | 1 week | -| **Total** | ~230 | 3-5 | 3+ | ~1500 | 10-12 weeks | - -### 4.3 Essential Ablations - -1. **Demo format ablation** - - Full trace (screenshot descriptions + actions + results) - - Behavior-only (actions + results) - - Action-only (just the action sequence) - -2. **Demo relevance ablation** - - Exact-match demo (same task) - - Same-domain demo (e.g., any Settings task) - - Cross-domain demo (e.g., Browser demo for Settings task) - - Random demo - -3. **Number of demos (k)** - - k=1, 3, 5 - - Does more demos help, or just add noise? - -### 4.4 Baselines We MUST Compare Against - -| Baseline | Description | Why Essential | -|----------|-------------|---------------| -| Zero-shot instruction only | No demo, just task description | Primary comparison | -| Zero-shot + CoT | "Think step by step" | Fair comparison to prompting methods | -| Few-shot examples (text) | Text-only examples, no screenshots | Isolate visual contribution | -| SOTA on WAA | GPT-5.1 + OmniParser (~19.5%) | Establish relative performance | -| Random policy | Random clicks | Sanity check | - ---- - -## 5. Statistical Rigor Requirements - -### 5.1 Required Statistical Tests - -| Test | Purpose | When to Use | -|------|---------|-------------| -| **McNemar's test** | Paired comparison of binary outcomes | Zero-shot vs demo on same tasks | -| **Bootstrap confidence intervals** | Uncertainty estimation | All accuracy metrics | -| **Effect size (Cohen's h)** | Practical significance | Accompany p-values | -| **Bonferroni correction** | Multiple comparisons | When testing multiple models/conditions | - -### 5.2 Minimum Sample Sizes - -For detecting a 20 percentage point improvement with 80% power (alpha=0.05): -- **Per-condition**: n >= 39 tasks (we have 45, sufficient) -- **With 3 trials per task**: 39 x 3 = 117 total observations - -For detecting a 10 percentage point improvement: -- **Per-condition**: n >= 199 tasks (we do NOT have this) -- **Implication**: If effect is smaller than expected, we may be underpowered - -### 5.3 Reporting Standards - -Every result table must include: -1. Mean accuracy -2. Standard deviation (across trials) -3. 95% confidence interval -4. Sample size (n) -5. Statistical test and p-value for key comparisons - -**Example**: -``` -| Condition | Accuracy | 95% CI | p-value (vs zero-shot) | -|-----------|----------|--------|------------------------| -| Zero-shot | 33.3% | [22.1, 46.0] | - | -| Demo-conditioned | 68.9% | [55.7, 80.1] | p<0.001 (McNemar) | -``` - ---- - -## 6. Related Work Gap Analysis - -### 6.1 Papers We MUST Cite - -**GUI Agents & Benchmarks**: -1. Bonatti et al. (2024) - Windows Agent Arena -2. Zhou et al. (2023) - WebArena -3. Xie et al. (2024) - OSWorld -4. Cheng et al. (2024) - SeeClick -5. Kim et al. (2024) - Crab benchmark -6. Gur et al. (2024) - WebAgent - -**VLM-based Agents**: -7. Wang et al. (2024) - Mobile-Agent -8. Zhang et al. (2024) - UFO -9. Lu et al. (2024) - WebVoyager -10. Anthropic (2024) - Claude Computer Use - -**Programming by Demonstration**: -11. Li et al. (2023) - UINav -12. Li et al. (2017) - SUGILITE -13. Cypher et al. (1993) - Watch What I Do (foundational PbD text) - -**Visual Grounding**: -14. Chen et al. (2024) - OmniParser -15. Yang et al. (2023) - Set-of-Marks - -**Few-shot Prompting & RAG**: -16. Brown et al. (2020) - GPT-3 few-shot -17. Wei et al. (2022) - Chain-of-thought -18. Lewis et al. (2020) - RAG - -### 6.2 Potential Reviewers - -Based on related work, likely reviewers include researchers from: -- Microsoft Research (WAA, UFO, OmniParser teams) -- Google DeepMind (WebAgent, PaLM teams) -- CMU HCII (SUGILITE, UINav teams) -- Allen Institute for AI (general VLM agents) -- Stanford HAI (human-AI interaction) - -**Implication**: Paper must respectfully position against UFO, SeeClick, and other Microsoft/Google work. - -### 6.3 How We Differ From Prior Work - -| Prior Work | Their Approach | Our Difference | -|------------|---------------|----------------| -| UINav | Referee model for demo quality | We don't evaluate demo quality | -| SUGILITE | NL + GUI disambiguation | We use full VLM reasoning | -| UFO | Dual-agent architecture | We use single VLM with demo context | -| WebVoyager | Web-specific agent | We target desktop applications | -| Claude Computer Use | Production agent, no demos | We add demo conditioning | - -**Honest assessment**: The difference from Claude Computer Use is simply "add a demo to the prompt." This is the core contribution, and we must own it. - ---- - -## 7. Venue Fit Analysis - -### 7.1 Realistic Venue Assessment - -| Venue | Fit | Honest Chance | Rationale | -|-------|-----|---------------|-----------| -| **NeurIPS main track** | Poor | <20% | Contribution too incremental for main track | -| **NeurIPS Datasets & Benchmarks** | Poor | N/A | We don't propose a new benchmark | -| **ICML main track** | Poor | <20% | Same as NeurIPS | -| **ICLR main track** | Poor | <20% | Needs stronger learning contribution | -| **CHI main track** | Moderate | 30-40% | Good fit IF we add user study | -| **UIST main track** | Good | 40-50% | Systems + empirical evaluation | -| **ACL/EMNLP** | Poor | <20% | Not sufficiently NLP-focused | -| **AAAI** | Moderate | 30-40% | More accepting of applied work | -| **LLM Agents Workshop (NeurIPS)** | Excellent | 60-70% | Perfect scope and contribution level | -| **CHI Late-Breaking Work** | Excellent | 70%+ | Low barrier, good fit | -| **UIST Demo Track** | Excellent | 60-70% | Live demo is compelling | - -### 7.2 Recommended Strategy - -**Phase 1 (Immediate)**: Target **LLM Agents Workshop @ NeurIPS 2026** or **ICML 2026** -- Deadline: ~3 months before conference -- Page limit: 4-8 pages -- Contribution bar: Lower than main track -- Allows us to establish priority and get feedback - -**Phase 2 (If workshop goes well)**: Expand to **CHI 2027** or **UIST 2026** -- Add user study (n=20-30) -- Expand benchmark coverage -- 10-page full paper - -**Phase 3 (Long shot)**: Only pursue NeurIPS/ICML main track IF: -- WAA shows >30pp improvement over SOTA -- We discover unexpected insights during analysis -- Reviewers at workshop suggest main-track potential - -### 7.3 Venue-Specific Requirements - -**For CHI acceptance**: -- User study with statistical analysis (n >= 20) -- Qualitative analysis (interviews, think-aloud) -- Discussion of implications for HCI -- Ethical considerations - -**For Workshop acceptance**: -- Clear empirical contribution -- Reproducible experiments -- Honest limitations discussion -- Interesting future directions - ---- - -## 8. Realistic Timeline - -### 8.1 Minimum Viable Timeline (Workshop Paper) - -| Week | Tasks | Dependencies | -|------|-------|--------------| -| **1-2** | Fix WAA environment, run clean baseline | VM stable | -| **3-4** | Run demo-conditioned WAA experiments | Baseline done | -| **5** | Statistical analysis, write results | Experiments done | -| **6** | Write introduction, related work | - | -| **7** | Internal review, revisions | Draft done | -| **8** | Submit to workshop | - | - -**Total: 8 weeks** from today to submission-ready - -### 8.2 Realistic Timeline (CHI Full Paper) - -| Month | Tasks | -|-------|-------| -| **1-2** | Complete WAA + WebArena experiments | -| **3** | Design and run user study | -| **4** | Analyze user study, write draft | -| **5** | Internal review, revisions | -| **6** | Submit to CHI | - -**Total: 6 months** (CHI 2027 deadline: ~September 2026) - -### 8.3 Timeline Risks - -| Risk | Likelihood | Impact | Mitigation | -|------|------------|--------|------------| -| WAA environment issues | High | 2-3 week delay | Have backup mock evaluation | -| Results don't match expectations | Medium | May kill paper | Pivot to analysis/negative results | -| API rate limits/costs | Medium | 1-2 week delay | Budget API costs upfront | -| Co-author availability | Medium | Variable | Start writing in parallel | - ---- - -## 9. Risk Mitigation - -### 9.1 If WAA Results Are Disappointing - -**Scenario**: Demo-conditioning shows <10pp improvement on WAA - -**Options**: -1. **Pivot to analysis paper**: Why doesn't demo-conditioning help on WAA? -2. **Focus on narrow success cases**: Which task categories benefit most? -3. **Negative results paper**: "When Demonstrations Don't Help" -4. **Workshop-only publication**: Present findings, get feedback - -### 9.2 If Experiments Take Too Long - -**Scenario**: Cannot complete experiments before deadline - -**Options**: -1. **Reduce scope**: Fewer tasks, fewer models, one benchmark -2. **Workshop paper first**: Lower bar, establish priority -3. **arXiv preprint**: Stake claim while continuing experiments -4. **Target later deadline**: Better to submit complete work - -### 9.3 If Reviewers Reject on Novelty - -**Mitigation in paper**: -- Explicitly position as *empirical study*, not algorithmic contribution -- Emphasize the magnitude of improvement and practical value -- Provide extensive ablations to show what matters -- Open-source all code and data - ---- - -## 10. Action Items - -### 10.1 Immediate (This Week) - -- [ ] **Fix WAA environment** - Resolve Navi agent bugs or switch to API agent -- [ ] **Define exact task set** - Select 20+ WAA tasks with diverse first actions -- [ ] **Budget API costs** - Estimate cost for 500+ API calls - -### 10.2 Short-Term (Weeks 2-4) - -- [ ] **Run zero-shot baseline** - 20 tasks x 2 models x 3 trials -- [ ] **Write demos for all tasks** - Using behavior-only format -- [ ] **Run demo-conditioned evaluation** - Same tasks, with demos -- [ ] **Statistical analysis** - McNemar's test, bootstrap CIs - -### 10.3 Medium-Term (Weeks 5-8) - -- [ ] **Write workshop paper** - 4-6 pages, focus on empirical results -- [ ] **Create figures** - Accuracy comparison, demo format examples -- [ ] **Internal review** - Get feedback from 2-3 people -- [ ] **Submit to workshop** - LLM Agents Workshop or similar - -### 10.4 Long-Term (Months 3-6) - -- [ ] **Expand to WebArena** - Additional benchmark coverage -- [ ] **User study design** - For CHI/UIST submission -- [ ] **Run user study** - n=20-30 participants -- [ ] **Write full paper** - 10 pages for CHI/UIST - ---- - -## 11. Path to Main Track Publication (Parallel Track) - -This section provides a rigorous assessment of what would be required to publish in a main track venue (NeurIPS, ICML, ICLR) rather than a workshop. This is a parallel track that requires substantially more investment. - -### 11.1 Honest Assessment: Why Current Work is Workshop-Level - -Our current contribution is fundamentally **prompt engineering**, not machine learning research. While valuable for practitioners, this positions us poorly for ML venues that expect learned components, theoretical insights, or architectural innovations. - -**Table: Anticipated Reviewer Concerns for Main Track Submission** - -| Concern | Severity | Our Current Status | What Main Track Requires | -|---------|----------|-------------------|--------------------------| -| No learned component | **Critical** | True - retrieval uses heuristic similarity | Train retrieval end-to-end for downstream task | -| Single demo format | **High** | True - behavior-only format hardcoded | Learn optimal format/compression | -| Heuristic retrieval (BM25/embedding) | **High** | True - not optimized for action accuracy | Retrieval that optimizes task success, not similarity | -| Limited evaluation | **High** | 45 tasks, 1 model, 1 platform | 200+ tasks, 3+ models, 2+ benchmarks | -| No comparison to fine-tuning | **High** | True | Show when prompting beats/complements fine-tuning | -| No theoretical analysis | **Medium** | True - purely empirical | Information-theoretic or PAC-learning analysis | -| Engineering focus | **Medium** | True - system building, not research | Clear algorithmic or theoretical contribution | -| No ablation of demo components | **Medium** | Partial | Systematic ablation with significance tests | - -**Bottom line**: A main track reviewer at NeurIPS/ICML will likely say: "This is a well-executed engineering project with an empirical evaluation, but where is the research contribution? Adding demos to prompts is not novel." - -### 11.2 Required Technical Contributions (Options to Elevate) - -To elevate from workshop to main track, we need at least ONE of the following technical contributions: - -#### Option A: Learned Demo Retrieval (RECOMMENDED) - -**Effort**: 2-3 months | **Risk**: Medium | **Novelty**: High - -**Core idea**: Train the retrieval system to optimize action accuracy, not semantic similarity. - -**Why this works**: Current retrieval uses off-the-shelf embeddings (CLIP, text similarity) that optimize for semantic match. But the best demo for a task may not be the most semantically similar - it may be one that provides the right procedural template or spatial priors. - -**Technical approach**: -1. Collect retrieval training data: (query, demo, action_accuracy) tuples -2. Train retrieval scorer to predict action accuracy given (query, demo) pair -3. Use contrastive learning: demos that help should score higher than demos that don't -4. Evaluate: Does learned retrieval outperform heuristic retrieval? - -**Key experiments**: -- Retrieval recall@k vs action accuracy correlation -- Learned vs heuristic retrieval on held-out tasks -- Analysis of what the model learns (which demo features matter?) - -**Related work to cite**: -- REALM (Guu et al., 2020) - Retrieval-augmented language model pretraining -- Atlas (Izacard et al., 2022) - Few-shot learning with retrieval -- DocPrompting (Zhou et al., 2022) - Retrieve docs for code generation - -**Why reviewers would accept**: "First demonstration that learned retrieval improves demo-conditioned GUI agents, with analysis of what retrieval features matter." - -#### Option B: Learned Prompt Synthesis - -**Effort**: 3-4 months | **Risk**: Medium-High | **Novelty**: High - -**Core idea**: Learn to synthesize optimal demo prompts rather than using fixed templates. - -**Technical approach**: -1. Define prompt template space (what to include, how to format, compression level) -2. Use LLM-in-the-loop optimization (APE-style) to find optimal templates -3. Alternatively, train a small model to select/compress demo content -4. Evaluate: Does learned synthesis outperform hand-crafted templates? - -**Key experiments**: -- Template ablation with learned selection -- Compression ratio vs accuracy tradeoff -- Cross-task transfer of learned templates - -**Related work to cite**: -- APE (Zhou et al., 2022) - Automatic prompt engineering -- DSPy (Khattab et al., 2023) - Programmatic prompt optimization -- PromptBreeder (Fernando et al., 2023) - Self-referential prompt evolution - -**Why reviewers would accept**: "Novel prompt synthesis method that learns to format demonstrations for maximal downstream utility." - -#### Option C: Behavioral Cloning with Demo-Augmentation - -**Effort**: 4-6 months | **Risk**: High | **Novelty**: Very High - -**Core idea**: Fine-tune a VLM using demonstration-augmented behavioral cloning. - -**Technical approach**: -1. Collect behavioral cloning dataset: (screenshot, task, action) tuples -2. Augment each example with retrieved demonstration context -3. Fine-tune VLM with demo in context vs without -4. Compare: Does demo-augmented fine-tuning outperform standard fine-tuning? - -**Key experiments**: -- Fine-tuning with/without demo augmentation -- Sample efficiency: Do demos reduce required training data? -- Analysis of attention patterns: Does the model attend to demos? - -**Related work to cite**: -- CogAgent (Hong et al., 2023) - GUI agent fine-tuning -- SeeClick (Cheng et al., 2024) - Visual grounding for GUI -- RT-2 (Brohan et al., 2023) - Vision-language-action models - -**Why reviewers would accept**: "First demonstration that demo-augmentation improves fine-tuned GUI agents, with analysis of when prompting vs fine-tuning is preferred." - -**Caveat**: This requires significant compute ($2-5k GPU, 4-6 weeks training) and expertise in VLM fine-tuning. - -#### Option D: Theoretical Analysis - -**Effort**: 2-3 months | **Risk**: High | **Novelty**: Medium - -**Core idea**: Provide theoretical analysis of why demonstrations help GUI agents. - -**Technical approach**: -1. Information-theoretic analysis: How much information do demos provide? -2. PAC-learning analysis: Sample complexity with/without demos -3. Formal model of GUI task space and demo utility - -**Key contributions**: -- Theoretical bound on demo utility -- Characterization of when demos help vs hurt -- Connection to few-shot learning theory - -**Related work to cite**: -- Brown et al. (2020) - GPT-3 few-shot capabilities -- Xie et al. (2021) - Why in-context learning works -- Min et al. (2022) - Rethinking demonstration role - -**Why reviewers would accept**: "Theoretical understanding of demonstration utility for GUI agents, with empirical validation." - -**Caveat**: Requires theoretical ML expertise; risk of disconnect between theory and practice. - -### 11.3 Additional Experiments Required - -Beyond the technical contribution, main track requires substantially more empirical evidence: - -**Benchmark Coverage**: -| Benchmark | Tasks Required | Current Status | Effort | -|-----------|---------------|----------------|--------| -| Windows Agent Arena (WAA) | 50+ tasks | 8 tasks (incomplete) | 3-4 weeks | -| WebArena | 100+ tasks | 0 tasks | 4-6 weeks | -| OSWorld (optional) | 50+ tasks | 0 tasks | 4-6 weeks | - -**Evaluation Metrics**: -- **First-action accuracy**: Already measured, but on non-standard tasks -- **Episode success rate**: Not measured - REQUIRED for main track -- **Step efficiency**: Actions per successful task -- **Grounding accuracy**: Correct element identification rate - -**Multi-Model Comparison**: -| Model | Priority | Status | -|-------|----------|--------| -| Claude Sonnet 4.5 | Required | Tested | -| GPT-4V | Required | Not tested | -| Gemini 1.5 Pro | Required | Not tested | -| Qwen-VL | Nice to have | Not tested | -| Open-source (LLaVA) | Nice to have | Not tested | - -**Ablation Studies**: -1. Demo format: full trace vs behavior-only vs action-only -2. Number of demos: k=1, 3, 5, 10 -3. Demo relevance: exact match vs same-domain vs random -4. Demo recency: fresh demos vs stale demos -5. Model scale: Does demo benefit scale with model size? - -**Statistical Requirements**: -- 3+ seeds per experiment for variance estimation -- 95% confidence intervals on all metrics -- Statistical significance tests (McNemar's, permutation tests) -- Effect sizes (Cohen's h, odds ratios) - -### 11.4 Timeline and Resources - -**Minimum timeline for main track submission**: - -| Phase | Duration | Activities | -|-------|----------|------------| -| **Phase 1**: Technical contribution | 2-4 months | Implement learned retrieval or prompt synthesis | -| **Phase 2**: Large-scale evaluation | 2-3 months | WAA (50+), WebArena (100+), multi-model | -| **Phase 3**: Analysis & writing | 1-2 months | Ablations, significance tests, paper writing | -| **Total** | **6-9 months** | From start to submission-ready | - -**Resource requirements**: - -| Resource | Estimate | Notes | -|----------|----------|-------| -| Dedicated researchers | 1-2 FTE | Cannot be done part-time | -| GPU compute | $2-5k | For fine-tuning experiments (Option C) | -| API credits | $1-3k | Multi-model evaluation at scale | -| Azure VM (WAA) | $200-500 | Extended evaluation runs | -| Human annotation | $500-1k | Demo quality labels, retrieval training data | - -**Total estimated cost**: $5-10k (excluding researcher time) - -### 11.5 Honest Recommendation - -**For a small team with limited resources**: -- **Focus on workshop paper**. The workshop contribution is solid and achievable. -- Do NOT attempt main track unless you can dedicate 1-2 researchers full-time for 6+ months. -- A rejected main track submission wastes 6-9 months and demoralizes the team. - -**For a team with dedicated resources**: -- **Pursue Option A (Learned Retrieval)** as the most tractable path to main track. -- This adds a clear learned component while building on existing infrastructure. -- Expected timeline: 6-7 months to submission-ready. -- Honest acceptance probability: 25-35% at NeurIPS/ICML (still challenging). - -**Do NOT attempt main track if**: -- You cannot dedicate 1-2 researchers full-time to this project -- You do not have ML research expertise (vs engineering expertise) -- You need a publication in < 6 months -- You are not prepared for likely rejection and iteration - -**The workshop path is not a consolation prize**. Top workshops at NeurIPS/ICML have excellent visibility, lead to valuable feedback, and establish priority for your ideas. Many impactful papers started as workshop papers. - -### 11.6 Additional References for Main Track - -**Retrieval-Augmented Learning**: -- Guu, K., Lee, K., Tung, Z., Pasupat, P., & Chang, M. W. (2020). REALM: Retrieval-augmented language model pre-training. *ICML 2020*. -- Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., ... & Grave, E. (2022). Atlas: Few-shot learning with retrieval augmented language models. *arXiv preprint arXiv:2208.03299*. -- Zhou, S., Alon, U., Xu, F. F., Wang, Z., Jiang, Z., & Neubig, G. (2022). DocPrompting: Generating code by retrieving the docs. *ICLR 2023*. - -**Automatic Prompt Engineering**: -- Zhou, Y., Muresanu, A. I., Han, Z., Paster, K., Pitis, S., Chan, H., & Ba, J. (2022). Large language models are human-level prompt engineers. *ICLR 2023*. -- Khattab, O., Santhanam, K., Li, X. L., Hall, D., Liang, P., Potts, C., & Zaharia, M. (2023). DSPy: Compiling declarative language model calls into self-improving pipelines. *arXiv preprint arXiv:2310.03714*. -- Fernando, C., Banarse, D., Michalewski, H., Osindero, S., & Rocktäschel, T. (2023). PromptBreeder: Self-referential self-improvement via prompt evolution. *arXiv preprint arXiv:2309.16797*. - -**GUI Agent Fine-Tuning**: -- Hong, W., Wang, W., Lv, Q., Xu, J., Yu, W., Ji, J., ... & Tang, J. (2023). CogAgent: A visual language model for GUI agents. *arXiv preprint arXiv:2312.08914*. -- Cheng, K., Sun, Q., Chu, Y., Xu, F., Li, Y., Zhang, J., & Wu, Z. (2024). SeeClick: Harnessing GUI grounding for advanced visual GUI agents. *arXiv preprint arXiv:2401.10935*. -- Brohan, A., Brown, N., Carbajal, J., Chebotar, Y., Chen, X., Choromanski, K., ... & Zitkovich, B. (2023). RT-2: Vision-language-action models transfer web knowledge to robotic control. *arXiv preprint arXiv:2307.15818*. - ---- - -## Appendix A: Honest Framing for Paper - -### Abstract Template - -> We present an empirical study of demonstration-conditioned prompting for vision-language model (VLM) GUI agents. While prior work has explored VLMs for GUI automation, we systematically evaluate the effect of including human demonstrations in the prompt. Across N tasks on the Windows Agent Arena benchmark, we find that demo-conditioning improves task success rate from X% to Y% (p < 0.01), representing a Z percentage point improvement. We analyze which task categories benefit most and identify limitations where demonstrations do not help. Our findings suggest that simple prompting interventions can substantially improve GUI agent performance without fine-tuning, and we release our code and demo library to facilitate future research. - -### Title Options (Honest) - -1. "Does Showing Help? An Empirical Study of Demo-Conditioned GUI Agents" -2. "From Instructions to Demonstrations: Improving VLM GUI Agents Through Example" -3. "Show, Don't Just Tell: The Value of Demonstrations for GUI Automation" - -### Contribution Statement Template - -> Our contributions are: -> 1. **Empirical study**: We conduct the first systematic evaluation of demo-conditioning for VLM GUI agents across N tasks and M models -> 2. **Analysis**: We identify which task categories and UI patterns benefit most from demonstrations -> 3. **Practical method**: We provide an open-source implementation with demo retrieval capabilities -> 4. **Dataset**: We release a library of K human demonstrations for GUI tasks - ---- - -## Appendix B: Cost Estimates - -### API Costs (Conservative) - -| Model | Input ($/1M) | Output ($/1M) | Est. calls | Est. cost | -|-------|--------------|---------------|------------|-----------| -| Claude Sonnet 4.5 | $3 | $15 | 1000 | ~$50-100 | -| GPT-4V | $10 | $30 | 1000 | ~$100-200 | -| Gemini Pro Vision | $0.25 | $0.50 | 1000 | ~$10-20 | -| **Total** | - | - | 3000 | ~$200-400 | - -### Compute Costs (Azure) - -| Resource | Rate | Hours | Cost | -|----------|------|-------|------| -| D4ds_v5 (WAA VM) | $0.19/hr | 100 | ~$20 | -| Storage | $0.02/GB | 100GB | ~$2 | -| **Total** | - | - | ~$25 | - ---- - -## Appendix C: Reviewer Response Templates - -### "This is just few-shot prompting" - -> We agree that demo-conditioning can be viewed as a form of few-shot prompting. However, GUI automation presents unique challenges compared to standard NLP tasks: (1) visual grounding requires understanding spatial relationships in screenshots, (2) multi-step tasks require maintaining procedural context, and (3) UI variations across platforms and applications create distribution shift. Our contribution is demonstrating that demonstrations substantially help in this domain (X% -> Y%), characterizing when they help (task category analysis), and providing practical infrastructure (demo retrieval, open-source code) for practitioners. - -### "Sample size is too small" - -> We acknowledge this limitation. With n=N tasks and 3 trials each, we are powered to detect a 20pp effect at 80% power. Our observed effect of Zpp is well above this threshold, and our statistical tests (McNemar's, bootstrap CI) confirm significance. We have expanded our task set to N tasks for the camera-ready version. - -### "Results may not generalize beyond tested benchmarks" - -> This is a valid concern. We have focused on WAA as it represents realistic enterprise desktop tasks. In future work, we plan to evaluate on WebArena and OSWorld to assess cross-benchmark generalization. However, we note that the WAA benchmark itself covers diverse applications (browser, office, file management, settings) and our positive results across these categories suggest some generalizability within desktop environments. - ---- - -*Last updated: January 2026* -*This is a living document. Update as experiments complete and understanding deepens.* diff --git a/docs/roadmap-priorities.md b/docs/roadmap-priorities.md deleted file mode 100644 index 6e8c665c5..000000000 --- a/docs/roadmap-priorities.md +++ /dev/null @@ -1,126 +0,0 @@ -# OpenAdapt Roadmap - -**Updated**: January 17, 2026 | **Version**: 1.1.0 - -Goal: **Record -> Train -> Evaluate** GUI automation workflows. - ---- - -## Packages (All Published) - -| Package | Python | Notes | -|---------|--------|-------| -| `openadapt` (meta) | >=3.10 | | -| `openadapt-capture` | >=3.10 | | -| `openadapt-ml` | >=3.10 | PR #5 fixes Python version | -| `openadapt-evals` | >=3.10 | | -| `openadapt-viewer` | >=3.10 | | -| `openadapt-grounding` | >=3.10 | 53 tests passing | -| `openadapt-retrieval` | >=3.10 | 28 tests passing | -| `openadapt-privacy` | >=3.10 | | - ---- - -## P0 - Critical (This Week) - -### 1. Fix CI - Ruff Format -- **PR**: #969, branch `fix/ruff-format-config` -- **Action**: Merge PR, verify CI passes (Python 3.10-3.12, macOS/Ubuntu) - -### 2. Fix Docker Build -- **Location**: `legacy/deploy/deploy/models/omniparser/Dockerfile` -- **Actions**: - - Test `docker build` - - Verify CUDA/GPU support - - Test model download (huggingface-cli) - -### 3. Verify Meta-Package Installation -- Test `pip install openadapt[core]` and `openadapt[all]` on Python 3.10+ -- Verify: `python -c "from openadapt.cli import main"` - -### 4. End-to-End Workflow -Test the core workflow: -```bash -openadapt capture start --name my-task -openadapt train start --capture my-task -openadapt eval run --checkpoint model.pt -``` - -**Blockers**: `capture stop` TODO (uses Ctrl+C), macOS needs Accessibility + Screen Recording permissions - ---- - -## P1 - High (1-2 Weeks) - -### 5. Baseline Adapters -- Package: `openadapt-ml` -- Test: Claude, GPT-4V, Gemini, Qwen3-VL adapters -- Add rate limit / API failure handling - -### 6. Demo Conditioning in Evals -- **Key result**: 46.7% -> 100% first-action accuracy with demo conditioning (on shared-entry-point benchmark) -- Integrate `openadapt-retrieval` with `openadapt-ml` adapters -- Add `--demo` flag to `openadapt eval run` - -### 7. WAA Benchmark Validation -- Blocked on: Azure VM with nested virtualization -- Goal: Validate demo-conditioning claims on Windows Agent Arena - ---- - -## P2 - Medium (This Month) - -### 8. Safety Gates -- Pre-action validation, dangerous action detection -- Optional human-in-the-loop confirmation - -### 9. Grounding Improvements -- Integrate with `openadapt-ml` action replay -- Providers: OmniGrounder (CUDA), GeminiGrounder (API), SoMGrounder (CUDA) - -### 10. Viewer Dashboard -- Add video playback, action timeline, side-by-side comparison -- Filter by action type - ---- - -## P3 - Backlog - -### 11. Telemetry (GlitchTip) -- Create `openadapt-telemetry` package -- GlitchTip/Sentry integration with privacy filtering -- Design: `docs/design/telemetry-design.md` - -### 12. Additional Benchmarks -- WebArena, OSWorld, MiniWoB++ - -### 13. Documentation Site -- Deploy MkDocs to docs.openadapt.ai - ---- - -## Success Criteria - -| Phase | Criteria | -|-------|----------| -| P0 | CI green, Docker builds, `openadapt[core]` installs, basic workflow works | -| P1 | API agents with demo conditioning, WAA baseline established | -| P2 | Safety gates, grounding integration, viewer with video | -| P3 | Telemetry published, docs live, additional benchmarks | - ---- - -## Technical Debt - -- `capture stop` uses Ctrl+C instead of signal/file -- Legacy code in `/legacy/` (not blocking v1.0) - ---- - -## Resources - -| Resource | Status | -|----------|--------| -| Azure credits (WAA) | Available | -| API keys (Claude, GPT-4V, Gemini) | In `openadapt-ml/.env` | -| DNS (docs.openadapt.ai) | Configured |