Tests

Overview

A test is an abstraction on top of the agent API. Where creating an agent is a one-off operation (one prompt, one run, one result), a test lets you define a prompt once and run it as many times as you want. Each run creates a new agent that is linked back to the test, so you build up a history of results for the same test over time.

This is the core mechanism for tracking the health of your application. If you have a test called "checkout flow" and you run it after every deploy, you get a timeline of pass/fail results that tells you exactly when something broke and when it was fixed. The project health board aggregates this across all tests in a project.

When you run a test, the system resolves the role (cascading from the run parameters, to the test defaults, to the project defaults), copies any attached mailboxes onto the new agent, renders the prompt, and queues the agent for execution.

Prompt template model

A test's prompt_template is the instruction set the agent executes. When a test is run as part of a test plan, everything recorded by ancestor steps is provided to the agent automatically — refer to those values in plain language (“log in with the credentials from the register step”). The only template syntax is for plan-level variables:

• {{ plan.variables.<name> }} — replaced with a plan-level variable supplied when the run starts.

The starting URL is no longer a separate field. Inline it as literal text in the prompt body, typically as the opening “Navigate to …” line. The agent reads that and calls browser_navigate itself.

A minimal example:

Navigate to {{ plan.variables.env_url }} and log in with the credentials recorded by the register step. Open the order created by the checkout step and verify its status is “confirmed”.

When this test runs via POST /api/v1/tests/{id}/run (outside a plan), plan.variables references render as empty strings, gracefully degrading.

Success determination

Whether a test run succeeded or failed is determined by a two-step process. First, the agent itself reports its own assessment when it finishes (success or failure with a summary). Second, a server-side verification step independently analyzes the full conversation trace and produces its own verdict.

If the agent and the server-side verification disagree, the server-side verdict wins. This means an agent cannot incorrectly report success if the evidence in the conversation shows otherwise, and vice versa. The final outcome is what you see in the API response and on the health board.

Role resolution

When a test is run, the role is resolved using a cascading fallback. The system first checks if a role was specified in the run parameters. If not, it falls back to the default set on the test itself. If the test has no default, it uses the project-level default.

This lets you set sensible defaults on the test or project and override them on individual runs when needed. For example, you might have a test that normally runs with a "Functional QA" role but occasionally run it with a "Power User" role to test a different interaction style.

Create a new test

Parameters

Status Codes

Response Body

{
  "id": "770e8400-e29b-41d4-a716-446655440000",
  "tenant_id": "550e8400-e29b-41d4-a716-446655440000",
  "project_id": "660e8400-e29b-41d4-a716-446655440000",
  "name": "Checkout Flow",
  "description": "",
  "prompt_template": "Navigate to https://example.com and complete checkout",
  "status": "active",
  "file_paths": [],
  "created_at": "2025-01-15T10:30:00Z",
  "updated_at": "2025-01-15T10:30:00Z"
}

List tests

Parameters

Status Codes

Response Body

{
  "tests": [
    {
      "id": "770e8400-e29b-41d4-a716-446655440000",
      "tenant_id": "550e8400-e29b-41d4-a716-446655440000",
      "project_id": "660e8400-e29b-41d4-a716-446655440000",
      "name": "Checkout Flow",
      "description": "",
      "prompt_template": "Navigate to https://example.com and complete checkout",
      "status": "active",
      "file_paths": [],
      "created_at": "2025-01-15T10:30:00Z",
      "updated_at": "2025-01-15T10:30:00Z"
    }
  ],
  "next_cursor": "880e8400-e29b-41d4-a716-446655440000"
}

Get test by ID

Parameters

Status Codes

Response Body

{
  "id": "770e8400-e29b-41d4-a716-446655440000",
  "tenant_id": "550e8400-e29b-41d4-a716-446655440000",
  "project_id": "660e8400-e29b-41d4-a716-446655440000",
  "name": "Checkout Flow",
  "description": "",
  "prompt_template": "Navigate to https://example.com and complete checkout",
  "status": "active",
  "file_paths": [],
  "created_at": "2025-01-15T10:30:00Z",
  "updated_at": "2025-01-15T10:30:00Z"
}

Update a test

Parameters

Status Codes

Response Body

{
  "id": "770e8400-e29b-41d4-a716-446655440000",
  "tenant_id": "550e8400-e29b-41d4-a716-446655440000",
  "project_id": "660e8400-e29b-41d4-a716-446655440000",
  "name": "Checkout Flow",
  "description": "",
  "prompt_template": "Navigate to https://example.com and complete checkout",
  "status": "active",
  "file_paths": [],
  "created_at": "2025-01-15T10:30:00Z",
  "updated_at": "2025-01-15T10:30:00Z"
}

Delete a test

Parameters

Status Codes

Run a test

Creates a new test run from the test, inheriting the test's stored device, browser, and mobile app configuration. Files attached to the test are automatically copied into the test run workspace.

Parameters

Status Codes

Response Body

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "tenant_id": "660e8400-e29b-41d4-a716-446655440000",
  "project_id": "770e8400-e29b-41d4-a716-446655440000",
  "status": "pending",
  "prompt": "Navigate to https://example.com and complete checkout",
  "model": "gemini-3.5-flash",
  "browser_type": "chrome",
  "is_discovery": false,
  "messages": [],
  "iteration": 0,
  "max_iterations": 200,
  "tokens_used": 0,
  "test_id": "990e8400-e29b-41d4-a716-446655440000",
  "source": "api",
  "created_at": "2025-01-15T10:30:00Z",
  "updated_at": "2025-01-15T10:30:00Z"
}