Reference

Architecture

Mr. Browser is a layered Go application. The CLI and SDKs sit at the top; Chrome DevTools Protocol (CDP) via chromedp sits at the bottom. In between, the engine resolves plain-English intent against the browser's Accessibility Tree, executes actions, and stores structural fingerprints for future self-healing.

# System diagram

architecture

bash

┌─────────────────────────────────────────────────────────┐
│                      Client Layer                       │
│     Python SDK (mrbrowser)   TypeScript SDK (@mrbrowser/sdk) │
│     YAML Workflows (mr-browser run)                     │
└──────────────────────┬──────────────────────────────────┘
                       │  HTTP REST  (localhost:7331)
┌──────────────────────▼──────────────────────────────────┐
│                   CLI / Server (Go)                     │
│   cobra commands: run / debug / screenshot / inspect    │
└──────────────────────┬──────────────────────────────────┘
                       │
┌──────────────────────▼──────────────────────────────────┐
│                   Runtime / Executor                    │
│   task.go: YAML parsing & validation                    │
│   executor.go: step dispatch, error handling            │
└──────────────────────┬──────────────────────────────────┘
                       │
          ┌────────────┴────────────┐
          │                        │
┌─────────▼──────────┐  ┌──────────▼──────────────────────┐
│  Actions Engine    │  │   Intelligence / Resolution      │
│  actions.go:       │  │   dom/extractor.go:              │
│  Click / Type /    │  │   Build Accessibility Tree       │
│  Hover / Scroll /  │  │                                 │
│  Upload / Screenshot│  │   intent/resolver.go:           │
│  verification.go:  │  │   Score elements against intent │
│  DOM-change checks │  │   (local NLP, no cloud)         │
└─────────┬──────────┘  └──────────┬──────────────────────┘
          │                        │
          └──────────┬─────────────┘
                     │
┌────────────────────▼──────────────────────────────────┐
│               Memory Engine (SQLite)                  │
│   memory.go: store & retrieve element fingerprints   │
│   self-heal: compare live DOM to historical snapshot  │
│   database: ./mrbrowser.db                           │
└────────────────────┬──────────────────────────────────┘
                     │
┌────────────────────▼──────────────────────────────────┐
│       Chrome DevTools Protocol (chromedp)             │
│   Navigate / Click / Type / Screenshot / JS execute   │
│   Accessibility tree snapshot (CDP Accessibility API) │
└────────────────────┬──────────────────────────────────┘
                     │
               [Chromium Browser]

# Key packages

Package	Responsibility
cli/cmd	Cobra subcommands: run, debug, screenshot, inspect
core/runtime	YAML task parsing (task.go) and step execution (executor.go)
core/actions	Browser action primitives: Click, Type, Hover, Scroll, Upload, Screenshot
core/browser	chromedp session wrapper: Navigate, TypeSelector, ClickSelector, ExecuteJS
intelligence/dom	CDP Accessibility Tree extraction → []PageElement
intelligence/intent	Scoring engine: matches intent strings to PageElements
memory	SQLite fingerprint store — read/write/heal operations
telemetry	Structured logging (slog-based) with step/success/error helpers

# Intent resolution pipeline

resolution algorithm

typescript

// Resolution pipeline (intelligence/dom + intent packages)
//
// 1. CDP Accessibility snapshot → []PageElement
// 2. For each element, compute score:
//      textScore     = fuzzy match(intent, element.text + aria-label)
//      roleScore     = match(role-hints in intent, element.role)
//      regionScore   = match(position-hints, element.boundingBox)
//      overallScore  = weighted average
// 3. Sort by score, take top candidate
// 4. If score < threshold → check Memory Engine fingerprints
// 5. If fingerprint match > heal_threshold → self-heal & update DB
// 6. If no match → fail step with descriptive error

# Memory Engine fingerprint

Every successful resolution is persisted to a SQLite database (./mrbrowser.db by default). The fingerprint captures the structural context of the element — not its class name or ID, but where it lives in the DOM tree and what surrounds it.

fingerprint schema

typescript

// memory/fingerprint (stored in mrbrowser.db)
{
  "intent":    "Login button",
  "resolved":  { "role": "button", "name": "Login" },
  "fingerprint": {
    "ancestors": ["form[auth]", "main", "body"],
    "siblings":  ["input[Email]", "input[Password]"],
    "position":  { "region": "center", "order": 3 },
    "text_hash": "b2f9a1…"
  },
  "confidence": 0.99,
  "last_seen":  "2026-07-01T09:14:22Z"
}

When the intent resolver fails on a future run, the Memory Engine compares the current DOM against all stored fingerprints for that intent and finds the element that most structurally resembles the historical record — even if it has moved, been re-labeled, or wrapped in additional containers.

# Technology choices

Go — chosen for performance, a single binary distribution, and excellent concurrency primitives for managing Chromium sessions.

chromedp — a pure-Go CDP client that gives direct access to the Accessibility Tree snapshot API, which other automation frameworks don't expose at the same depth.

SQLite (go-sqlite3) — zero-dependency embedded database. The fingerprint store is a single file, easily committed to version control alongside test code.

Local NLP only — no cloud calls, no API keys, no latency from LLM round-trips. Resolution is deterministic and completes in under 30 ms on a modern laptop.