Writing Effective Tests

An agent is only as good as the instructions it receives. The whole point of testing with Aiqaramba is that those instructions no longer have to be exact. A Playwright script needs precise code that clicks this selector, then that one, in this order. Our agents need none of that. They read the page, understand your intent, and figure out the clicks themselves.

In fact, a little vagueness helps. "Submit the form and check it worked" ages far better than "click the button with id #submit-btn," because the agent adapts when your UI changes while the rigid instruction breaks. This page collects the practices we have found produce reliable, repeatable tests.

The three parts of a test

Agents cope well with vagueness, but they cannot read your mind. That is why a test is not one free-form prompt but three parts:

The entry URL. Where the agent starts. It opens a fresh browser session on this page with no prior knowledge of your application.
The instructions. What to do, in plain language: a loose sketch of the flow plus anything the agent cannot invent on its own — credentials, which account to use, how to reach the feature.
The validations. The specific things you consider "working," one per entry. The agent is forced to explicitly verify every one before it can finish, and a test with a failed validation fails.

The validations are the part people most often underspecify. The days of naming exact buttons are over, but you still have to say what you want validated. A vague validation list means the agent invents its own best-effort opinion of success, which makes your results unpredictable and lets obvious bugs slip through. Every validation is checked off individually — the agent cannot report success while one is unverified or failing — so each entry you write is a guarantee about what was actually checked.

Compare these two tests for the same login page:

Underspecified, so the agent decides what "working" means:

Entry URL:    https://app.example.com

Instructions: Log in with these credentials:
              Username: bob@example.com
              Password: Test123!

Validations:  1. logging in works

Better, because every check is its own validation the agent must resolve:

Entry URL:    https://app.example.com

Instructions: Try to log in with these credentials, then retry the form
              with incorrect credentials, unicode input, and empty values:
              Username: bob@example.com
              Password: Test123!

Validations:  1. correct credentials land on the dashboard
              2. incorrect credentials return nicely rendered, useful error messages
              3. weird unicode characters do not break validation
              4. empty values produce a clear, non-breaking error

The second test constrains the agent to the checks you care about and only reports success when they all pass. Notice that a validation can still be vague in wording, like "weird unicode characters" or "empty values." You just have to list it. The first test runs fine but rarely tries odd inputs on its own, so it quietly misses the very things you wanted to catch.

Give the agent what it needs

Agents are go-getters that proactively push to complete the task. That is usually a strength, but it turns into odd behaviour when you have left out something critical. The usual culprits are missing or wrong credentials and missing URLs.

Instructions like these will not get you far:

Test my login

Which part of the login? With what credentials? Not even the best human tester could act on that. You can be vague, but not empty. Always hand over the information the task genuinely requires. Think of it exactly like briefing a colleague: the agent knows what "log in" means, but it cannot supply the test account for you. And remember the agent starts from scratch at the entry URL every run — if the feature under test is three pages deep, the navigation to it belongs in the instructions.

Testing email flows

Email has always been a friction point for automation. Magic links, one-time codes, and signup confirmations all live in an inbox the test cannot normally reach. Aiqaramba solves this with mailboxes. A mailbox is a real email address your agents can read: create one in the dashboard or via the API, pick a name, and every message sent to that address (for example qa-signups@agents.aiqaramba.com) is stored against it. Attach the mailbox to a test and its agents can read what lands there.

An agent knows which mailboxes are attached to it and will reach for one whenever a flow needs an email. That convenience has a sharp edge: attach a mailbox, ask the agent to "sign up," and it will happily use the mailbox address as the account email even if that was not your intent. So be explicit. Tell it to "use the attached mailbox to do X" and you are never surprised.

Entry URL:    https://app.example.com/signup

Instructions: Sign up for a new account. Use the attached mailbox
              (qa-signups@agents.aiqaramba.com) as the email address.
              After submitting, read the inbox of that mailbox, open the
              confirmation email, and follow the verification link.

Validations:  1. the confirmation email arrives in the mailbox
              2. the verification link confirms the account and lands on the dashboard

Handling credentials securely

For test credentials, we strongly recommend using accounts tied to mailboxes you created in Aiqaramba. It makes the whole workflow smoother, and it keeps you clear of real user data.

Notice what we do not offer: a vault to "securely" store and hand secrets to agents. That is deliberate, and it is not a gap we plan to fill. There is no way to give an LLM a credential in a form it cannot leak. At some point the agent needs the plain-text value to type it into a login form, and the moment it holds that value, in a non-deterministic system, it can surface anywhere. Encrypted storage in front of that reality is security theatre. We would rather tell you the truth than sell you the illusion of safety.

The real fix is simpler: do not hand over anything sensitive in the first place. Use throwaway test accounts, point at staging rather than production, use fake credit-card numbers. You cannot leak what you never provided, and that is good practice regardless of Aiqaramba.