Built an end-to-end computer-use system that discovers a browser workflow with an LLM, compiles the successful run into a reusable typed capability, and replays it deterministically without calling a model.
Architecture
A Playwright-backed browser layer exposes bounded semantic operations rather than raw CSS, JavaScript, or coordinates. During discovery, the system observes the accessible UI, requests one structured action from the model, validates it against policy, executes it, and records redacted evidence. The artifact builder then converts the successful run into a versioned JSON capability with typed parameters, outputs, ordered steps, expectations, and a final checkpoint.
Deterministic Replay
- Executes saved capabilities without a model or prompt dependency
- Resolves semantic targets through ARIA roles, accessible names, labels, and bounded fallbacks
- Validates runtime inputs, step expectations, extracted outputs, and final checkpoints
- Separates success, expected business outcomes, and operational failures in the result contract
- Handles slow states and locator failures with bounded waits, refreshed observations, and retries
Safety and Human Handoff
A centralized policy classifies actions as safe, human-reviewed, or blocked and constrains navigation by origin and route. The evidence writer recursively redacts credentials, tokens, selected PII, and operator-entered values. On a recoverable failure, automation can transfer exclusive control of the same browser session to a human operator, record the intervention, and resume at the next step.
Full-Stack Test Application
The repository includes a synthetic credit-union application used to exercise real browser workflows and controlled failure scenarios. Its React and TypeScript frontend communicates with a FastAPI service backed by SQLAlchemy and SQLite. The test suite covers deterministic success, multi-record extraction, expected business outcomes, timeouts, policy enforcement, failure evidence, and human control transfer.