Introduction

Core Concepts

Three ideas make Mr. Browser different: intent resolution, the accessibility tree as the source of truth, and the self-healing Memory Engine.

# Intent-Driven Resolution

An intent is a natural-language description of an element. A local NLP algorithm decomposes it into role, text, and contextual hints, then scores every node in the page against them. No cloud, no LLM required — resolution is deterministic and typically completes in under 30ms.

- click: "the red delete button in the sidebar"

# The engine decomposes this into:
#   action:    click
#   role hint: button
#   text hint: "delete"
#   modifiers: color=red, region=sidebar

# The Accessibility Tree

Instead of the raw DOM, the engine matches against the browser's Accessibility Tree — the same semantic structure screen readers use. Roles, accessible names, and states are stable across redesigns even when class names are minified into .x9f2a hashes.

# The Memory Engine

Every successful resolution stores a structural fingerprint: ancestor chain, sibling context, region, and a text hash.

{
  "intent": "Login",
  "resolved": { "role": "button", "name": "Login" },
  "fingerprint": {
    "ancestors": ["form[auth]", "main", "body"],
    "siblings":  ["input[Email]", "input[Password]"],
    "position":  { "region": "center", "order": 3 },
    "text_hash": "b2f9…"
  },
  "confidence": 0.99,
  "last_seen": "2026-07-01T09:14:22Z"
}

When a later run fails to resolve an intent directly, the engine compares the live page against historical fingerprints to find where the element moved — then updates the fingerprint and continues:

$ mrbrowser run login.yaml

[intent] resolving "Login" → no direct match (0.41 < threshold)
[memory] consulting fingerprint b2f9…
[memory] structural match found: button moved
         form[auth] → dialog[auth-modal]   (0.94)
[memory] ✔ self-healed, fingerprint updated
✔ flow login_admin passed in 3.1s