Inputs required for useful generated tests
Statements such as 'works correctly' or 'has a good experience' cannot produce decidable assertions. Rewrite them as preconditions, inputs, actions, expected results and failure behavior.
- Testable requirements or acceptance criteria.
- Interface definitions, field constraints, error codes and state changes.
- Critical business rules and outcomes that must never occur.
- Normal, boundary and known-failure samples.
- Runtime environment, dependency versions and treatment of external services.
- The current change scope and existing tests.
AI has a different role at each test layer
- Unit tests: expand boundaries, error branches and parameter combinations; verify that assertions follow business rules rather than merely matching the implementation.
- API tests: draft request combinations, status codes, field checks and permission negatives; review contracts, idempotency, concurrency and cleanup.
- Integration tests: add cross-module and dependency-failure paths; verify real dependency versions, timeouts and compensation logic.
- End-to-end tests: cover critical user journeys while limiting suite size and controlling flakiness.
- Device and field tests: AI can organize inputs and retest steps, but power, temperature, network and hardware differences require target-environment testing.
Six steps from candidate tests to CI gates
- Rewrite requirements as testable conditions, including workload, concurrency, hardware, network and measurement definitions for performance cases.
- List data corruption, privilege, duplicate request, dependency failure, resource and compatibility risks before generating cases.
- Review whether each assertion distinguishes correct from incorrect outcomes and avoids false passes from loose matching or faulty mocks.
- Prepare resettable, cleanable and authorized data, and document differences between CI and the target environment.
- Run fast checks at commit, API and integration tests at merge, and critical end-to-end, migration, rollback and target-environment tests before release.
- Make failures traceable to tests, changes and logs. Fix, quarantine or remove flaky tests instead of rerunning until they pass.
How to determine whether a generated test is valuable
- It covers changed business rules and includes normal, boundary, error and permission-negative paths.
- It fails when a fault is deliberately injected and reports a specific condition.
- It repeats in a clean environment and its maintenance cost matches the protected risk.
- It reduces manual regression or reopened defects instead of only increasing test code.
- Coverage is supporting evidence, not proof; high coverage can still hide weak assertions and missing system interactions.
Common failure patterns
- Tests copy implementation logic, so code and tests share the same wrong assumption.
- Only happy paths are generated, omitting nulls, limits, duplicate requests, denied permissions, timeouts and recovery.
- Mocks replace too much of the real database, network, third-party service, device or deployment environment.
- Flaky tests depend on repeated runs, consuming investigation time and weakening trust in the gate.
Can AI generate a complete test suite automatically?
Do not assume so. AI can expand candidate cases from available information, while business correctness, risk priority, real data and environment differences still require engineering judgment.
Are testers still needed after tests are generated?
Yes. Testers or engineers responsible for testing still design the risk model, review assertions, prepare environments, analyze failures and decide release readiness. Generation mainly reduces repetitive preparation.
Should every test block a merge?
Not necessarily. Stable, fast and change-relevant checks fit commit or merge gates. Expensive end-to-end and device tests may run later, but agreed release validation must finish before deployment.
Does a passing CI run prove the system is ready to release?
No. Deployment configuration, migration, third-party dependencies, permissions, monitoring, rollback and target-environment retesting still need confirmation. CI is part of the evidence, not the complete acceptance decision.
Sources and scope
- NIST Secure Software Development Framework: https://csrc.nist.gov/pubs/sp/800/218/final
- DORA research on generative AI in software development: https://dora.dev/ai/gen-ai-report/report/
- Gate levels and thresholds must reflect system risk, technology stack, target environment and agreed acceptance criteria.