Autonomous Web Agent: What Makes It Reliable Is Not the Prompt
Building a form-filling agent that actually works: perception pipelines, structured error recovery, and the engineering that surrounds the LLM.
Most online forms are fifteen questions in a different order. Dropdowns, repeated fields, nothing that required me to actually think. I wanted to point an agent at a form, have it fill what it could, then review and submit. The hard part turned out to be not the form-filling. It was making the agent reliable enough to actually trust with it.
The Loop That Matters
The agent runs a four-step cycle until the goal is done or a budget runs out: perceive the page, plan the next action, act on it with Playwright, and record what happened. Then repeat.
That loop is not the interesting part. What surrounds it is.
Every useful agent needs to handle what happens when a step doesn't go as planned: the model picks something that no longer exists, the page doesn't respond the way it expected, or it gets stuck repeating itself. Most "autonomous agent" demos work once and skip all of that. The engineering question is what you build for the cases where it doesn't, because on the real web those cases come up constantly.
One design decision that runs through the whole system: the planner talks to an abstract LLM client, not a provider SDK. Two backends ship, one Anthropic-compatible (used here with MiniMax M3 via a custom base URL) and one OpenAI-compatible for OpenRouter, DeepSeek, and anything else OpenAI-shaped. Switching models is a config change. This mattered in practice when comparing models without wanting to touch the agent loop.
What the Agent Actually Sees
The first decision that shapes everything else is what the page looks like to the model. The obvious approach is to dump the HTML and ask "what now?" That falls apart on real pages: a product page can be tens of thousands of tokens of nested divs, most of it irrelevant to anything you can click.
Instead, the agent extracts a reduced view from the accessibility tree, only the interactive, role-bearing elements, each tagged in the live DOM with a data-agent-id. Here is the local sandbox the agent runs against:
The local sandbox site. The agent sees only the interactive elements: the textbox and the Add button. Not the surrounding structure.
The model sees the page as a short indexed list:
It picks an action by id: click 1, type "buy milk" into 2. The executor resolves that id back to a Playwright locator. Perception runs fresh every step, so ids always reflect the current DOM rather than a stale snapshot from earlier in the run.
Two things follow from this. The prompt is an order of magnitude smaller, which is cheaper and faster. And the model's choices are constrained to things that actually exist and are actionable right now, which removes an entire category of hallucinated selectors.
Parser injection surface: Accessible names are user-controlled text. An aria-label containing literal double quotes, like Mark "buy milk" done, splits a line formatted as [7] button "name" and the model starts referencing the wrong element ids. Sanitizing the accessible name on serialization fixed it. Treat any user-controlled string going into a structured format as untrusted input.
Reliability Is a Layer
A ReliabilityController wraps every execution step. It handles three failure types differently because they require different responses.
Transient failures
A locator times out because the page hadn't settled, a click arrived a beat early. These get retried with exponential backoff. Most of the time the next attempt just works.
Structural failures
The model picked an element id that no longer exists, or tried to navigate somewhere outside the domain allowlist. Retrying is pointless; the action is wrong for reasons that won't change. Instead of crashing, the failure becomes a result the planner sees on the next step: "that element is not here, here's the current page." The model re-plans against reality.
No-progress loops
The quiet failure mode. The agent acts, the page doesn't change, and it acts again. The system hashes (page state, action) each step. If the same pair repeats, the agent gets one nudge to try something different. If it repeats again, the run aborts as stuck rather than consuming its entire budget going nowhere.
On top of all of this sits a hard step budget. No run is ever unbounded.
The action whitelist is the companion piece. The model can only emit a fixed set of Pydantic-validated actions. Anything else fails validation before it touches the browser. A confirmation gate can also require human approval before a sensitive action executes, which is the practical form of the human-in-the-loop model I wanted from the start.
The Run That Proved It
The first real run against a live model was a trivial task: add "buy milk" to a todo list and mark it done. The agent reported success. The trace said 9 steps and 8 recoveries.
That gap is the whole story.
Reading the per-step trace, the task was actually finished at step 2. The agent clicked "Done," the item was marked complete. Then it spent the next six steps trying to do it again. At one point it burned 26 seconds retrying a click on a button that was disabled and was never going to respond.
The problem was in the perception layer, not the model. The sandbox's "Done" button changed its visible text to "Completed" and went disabled when clicked, but its aria-label stayed as Mark 'buy milk' done. The accessibility tree is exactly what the model sees instead of raw HTML, so the model read button "Mark 'buy milk' done" (disabled) and drew the obvious conclusion: here is the control for finishing the task, and for some reason it is not responding.
It was not broken. It was already done. The label was lying.
Two fixes came out of one run. The sandbox bug was a genuine accessibility bug: an aria-label contradicting the element's actual state. So the label was updated to tell the truth once the action was complete. The executor also had no business retrying a disabled element, so it now checks before handing off to Playwright and fails fast rather than eating the full timeout.
Here is the same task after both fixes, with the trace open:
Run #1: "Add 'buy milk' to my todo list and mark it done." Four steps, one recovery. Step 2 shows the disabled element check firing and the model's reasoning about it. Step 3 concludes correctly.
Four steps, one recovery. The recovery still fires at step 2 because the model sees the now-correctly-labelled button ("'buy milk' is completed"), tries to click it to confirm, and gets error · disabled back immediately instead of spinning for 26 seconds. Step 3 reads that outcome and finishes correctly.
The reliability machinery did its job the entire time. Without the step budget and loop detection, the original run doesn't end in an ugly but correct success; it spins until something external kills it. And without the per-step trace, I just see "success" and ship a perception bug that would surface later as random flakiness on real sites. The thing that broke was revealing. The only reason I caught it was that I had built the boring logging first.
You Can't Fix What You Can't See
Every step writes a record to SQLite: the action, the model's reasoning, the outcome, a screenshot, the page URL, a DOM-state hash, and how long it took. The dashboard reads this and renders each run step by step, with the screenshot of exactly what the agent was looking at when it made each decision.
The run list dashboard: status, step count, and recovery count for every run. The STUCK and FAILED rows are where the debugging starts.
When a run does something baffling, I don't guess. I open the trace.
That same data feeds the benchmark. A task suite with an independent success_check: a URL, DOM, or element assertion evaluated by the harness, not the agent's own claim that it finished. The harness runs each task N times and reports task success rate, mean steps-to-completion, and recovery rate.
The distinction between harness-evaluated success and self-reported success matters. An agent that "reports" it finished is telling you what it believes happened. The harness checks whether it actually did.
Benchmark results: MiniMax M3, 3 runs per task, including two public generalization sites. The saucedemo_add_to_cart row at 33% is the honest number.
The saucedemo add-to-cart result is what the benchmark is for. Same code, same model, one run at 100% and the next at 0%. The login → add-to-cart → open-cart flow at roughly fifteen to twenty steps is where the agent is weakest on a real site. A single green demo hides that. Running each task N times surfaces which task is fragile and by how much, which is the only way to know whether a change to the prompt or the loop actually improved reliability, or just happened to work on the one run you watched.
What I'd Do Differently
Multi-Tab Support
Some real-world flows open a new tab mid-task: a login redirect, a product detail page, a confirmation popup. Tab awareness in the perception and executor layer is the natural next extension.
Per-Task Step Budgets
Right now the budget is a single global cap. A login flow and a multi-step checkout are not the same. Per-task hints in the task definition would fix this without much added complexity.
More Generalization Tasks
The local sandbox is deterministic by design. The public site tasks on saucedemo and the-internet are where the interesting generalization signal is. A broader set of varied real-world flows would make the 90% number mean more.
Richer Confirmation Gate
The gate exists and works but it is coarse: it applies to whole action types. Field-level confirmation for high-value inputs would make the human-in-the-loop model more practical for the form-filling use case.