Tests

Overview

A test stores the setup for repeated agent runs. This setup includes the prompt, device, browser or app, files, mailboxes, and role.

Running a test spawns an agent and queues it for execution. The agent receives the saved setup plus any values supplied in the run request. Each agent links to the test, so repeated runs form a result history.

The test configures the run. The agent executes it and owns the state for that execution.

Success determination

The executor performs the test first. When it finishes, it stores result and moves the agent to auditing. The result contains the executor's reported success value and a short summary.

During auditing, a separate, heavier model reviews the trace, network events, console events, and existing findings. The auditor writes the summary object, records supported findings, and can correct result.success.

The API reads result.success as the pass or fail value. The following rules control audit corrections:

• A critical or high product finding changes a reported success to false.
• A successful audit can change a reported failure to true.
• A medium or low finding does not fail the test.

The status field describes the execution lifecycle. An agent can reach completed after the auditor finds a product issue.

Instructions template model

The instructions field carries the actions every agent created from the test performs; the agent starts at the test's entry_url and must resolve every entry of validations before it may finish.

The remaining features in this section apply only to test plan runs. A plan gives each step the values recorded by its ancestor steps. Refer to those values in plain language. Use {{ plan.variables.<name> }} when the instructions need a plan variable; references resolve only inside the instructions.

Example:

Navigate to {{ plan.variables.env_url }} and log in with the credentials recorded by the register step. Open the order created by the checkout step and verify its status is "confirmed".

Standalone runs have no test plan context, so they leave plan.variables references unresolved.

List tests

Parameters

Example Request

curl "http://app.aiqaramba.com/api/v1/tests" \
  -H "Authorization: Bearer $AIQA_API_KEY"

Status Codes

Response Fields (200)

Create a test

Request Body (application/json)

Example Request

curl -X POST "http://app.aiqaramba.com/api/v1/tests" \
  -H "Authorization: Bearer $AIQA_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "project_id": "<uuid>",
    "name": "<string>",
    "instructions": "<string>",
    "validations": ["<string>"]
  }'

Status Codes

Response Fields (201)

Get a test

Parameters

Example Request

curl "http://app.aiqaramba.com/api/v1/tests/<id>" \
  -H "Authorization: Bearer $AIQA_API_KEY"

Status Codes

Response Fields (200)

Delete a test

Parameters

Example Request

curl -X DELETE "http://app.aiqaramba.com/api/v1/tests/<id>" \
  -H "Authorization: Bearer $AIQA_API_KEY"

Status Codes

Update a test

Parameters

Request Body (application/json)

Example Request

curl -X PATCH "http://app.aiqaramba.com/api/v1/tests/<id>" \
  -H "Authorization: Bearer $AIQA_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "name": "<string>",
    "description": "<string>",
    "entry_url": "<string>",
    "instructions": "<string>",
    "validations": ["<string>"],
    "source": "<string>",
    "status": "<string>",
    "importance": "<string>",
    "file_paths": ["<string>"],
    "mailbox_names": ["<string>"],
    "role_id": "<uuid>",
    "device": "<string>",
    "browser_type": "<string>",
    "app_artifact_id": "<uuid>"
  }'

Status Codes

Response Fields (200)

Run a test

Creates and queues an agent using the saved test configuration and optional overrides.

Parameters

Request Body (application/json)

Example Request

curl -X POST "http://app.aiqaramba.com/api/v1/tests/<id>/run" \
  -H "Authorization: Bearer $AIQA_API_KEY" \
  -H 'Content-Type: application/json' \
  -d '{
    "role_id": "<uuid>",
    "mailbox_names": ["<string>"],
    "model": "<string>",
    "thinking_level": "<string>",
    "auditor_thinking_level": "<string>",
    "auditor_model": "<string>",
    "max_iterations": 1,
    "files": {},
    "tunnel": "<string>"
  }'

Status Codes

Response Fields (201)