Consistency Tests
Your documentation and getting started guides tell users how to use your product. But do the steps actually still work? Are the screenshots still accurate? Did someone rename a button three releases ago and forget to update the guide?
Consistency testing with Aiqaramba means pointing an agent at your documentation and telling it to follow the instructions step by step. The agent reads your guide, performs each action on the real application, and reports back whether everything worked as described.
How we would set it up
Not all documentation needs consistency testing. Nobody needs an agent to verify that "click the login button" still works. Focus on the flows that are important, that change often, and that would cause real frustration if the documentation is out of sync.
Think about your onboarding guides, your setup tutorials, any multi-step flow that involves configuration or integration. These are the pages that get stale the fastest because they touch many parts of the application and rarely get updated when individual features change.
Create one journey per documentation flow. "Follow the getting started guide from scratch" or "complete the API integration tutorial." Each journey should cover one complete guide, not just individual steps. The goal is to verify the entire narrative still holds together.
- Focus on high-value docs. Pick the 3 to 5 documentation pages that matter most. Your getting started guide, your main integration tutorial, any complex setup flow that users actually follow.
- Run on change, not on schedule. Unlike functional QA or chaos testing, consistency tests are most useful when you change either the documentation or the code paths it describes. Running them daily on unchanged docs is noise.
- Test the narrative, not just steps. A guide can have all its individual steps work but still be confusing. Tell the agent to flag anything that is unclear, ambiguous, or requires knowledge the documentation does not provide.