Human QA vs AI agents, on two real products
Same products, two ways of testing them. We lined up a human QA team's own reports against QA Team's autonomous agents on two live products — a consumer AI app and a SaaS analytics platform — and compared what each approach actually caught, how fast, and how consistently.
A consumer AI companion app
A mobile-web consumer app: AI avatar chat, a token wallet and billing, sign-up/login, and a bilingual (EN/ES) interface. Human QA ran multi-day manual passes — end-to-end, functional, UI, integration, exploratory, i18n and light security probing — across two October cycles.
Billing & timersConcurrencyi18n EN/ES mixingSecurity hardeningAccessibility404 handling
The standout human findings were real and specific: parallel chat sessions that locked the input and silently dropped messages, billing/timer state that counted at the wrong moments, pervasive language-string mixing between EN and ES, and a cluster of security-hardening gaps. But on a retest, 18 of 32 tracked issues were still reproducible — regressions and un-fixed defects piling up between manual cycles — and 5 more couldn't be re-verified that run.
| Dimension | Human QA | QA Team agents |
|---|---|---|
| Testing cadence | Multi-day manual passes, a few times a month | 43 sessions in two weeks — continuous, every deploy |
| New bugs reported (two weeks) | 22 (two passes) | 40 |
| Checks run · pass rate | Not tracked | 207 checks · 65.7% pass |
| Re-verified developer "fixes" | Manual re-test | Caught two claimed fixes still broken — voice still silent, rapid messages still dropped |
| Every bug reproduced & proven | 5 left unverified | Reproduced 1–2×, with screenshots & a recording |
| Time to first report after a deploy | Days (next scheduled pass) | Hours — same-day fixes retested automatically |
| Cost (two-week run) | part of a $3,500/mo team | $57.20 (main agent) · ~$95 with the admin & avatar agents |
A SaaS market-analytics platform
A data-heavy SaaS product: dashboards, keyword and audience research, AI reports and a third-party data integration, across 13 modules. Human QA ran a two-week cycle — functional, regression, UI, API and exploratory — supported by Claude and agents.
Data accuracySearch returns 0 / 10s latencyPlan & pack limitsNavigation mis-linksRevenue mismatches
The dominant theme was data accuracy: counts, dates and revenue that didn't match the source, searches returning zero results or taking ~10 seconds, and plan/pack limits behaving inconsistently. Two signals stand out for a human-vs-agent comparison: the regression rate climbed from 40% to 50% across the two weeks, and 14 of 34 defects were never itemised in the tracker (some were only shared informally over chat) — exactly the coverage and consistency gaps continuous agents are built to close.
| Dimension | Human QA | QA Team agents |
|---|---|---|
| Bugs reported | 34 over two weeks (28 open, 6 resolved) | 18 over 20 sessions |
| Checks run · pass rate | Not tracked | 65 checks · 69.1% pass |
| Tracked with full evidence | 20 of 34 itemised; some only on chat | Every bug logged, deduplicated, with proof |
| Re-tests fixed bugs automatically | Manual re-test (45% regression rate) | Yes — re-verified on the next deploy |
| Agent time | Tester days | 77 agent-minutes |
| Cost (run shown) | part of a $3,500/mo team | $70.53 |
What changes with agents
Human QA found deep, genuine bugs on both products — the kind that need judgement. The gaps weren't in skill; they were in cadence and consistency: issues reproduced for weeks between manual passes, a regression rate creeping from 40% to 50%, and a long tail of defects that never made it into the tracker. The agents didn't replace that judgement — but they ran 63 sessions across the two products for about $128 of compute, re-checked every developer-claimed fix on the next deploy (catching two that were still broken), and filed every finding with reproducible proof. So the human team spends its time on the hard calls instead of the regression sweep.