AI-generated tests are the most underrated and most accident-prone LLM coding scenario today. Let a model spit out 20 pytest cases from a function and you have coverage numbers looking great in minutes, but a green bar is not the same as "tested correctly." The most common AI failures are assertions so loose they are meaningless, mocks that drift out of sync with the real interface, and cases that depend on each other so deleting one turns the whole suite red. Our site's AI Code Refactoring SOP (ai-code-refactoring-sop) covers "changing code," this one covers "verifying code," and both share one premise: measure first, act second. This SOP breaks AI test generation into six steps: when to use it + baseline, generating unit tests from code, regression tests from issues, batch boundary/parameterized cases, test maintenance, and human-gate + CI integration. Each step ships copy-paste Python (pytest) and JS (jest) code, followed by pitfalls and FAQ. The one iron rule: AI only drafts, a human reviews and CI passes before merge. For general prompt engineering, see our site's Coding Prompt Pack (prompt-coding-pack).
1. When to Use AI Test Generation + Quantify the Baseline
Not every module is worth AI-generating tests for. First measure two numbers: current coverage and defect escape rate. Coverage shows which modules are low and which branches are unreached; defect escape rate (bugs found in production / total bugs) shows whether tests actually caught anything, and if you test a lot offline but production bugs keep flowing, the tests are testing the wrong things. Read the two numbers together: high coverage with high escape means you tested irrelevant paths; both low means testing has barely started. AI test generation suits "stable implementation, clear boundaries, enumerable cases" pure functions and utilities; it does not suit "behavior depends on external systems, assertions hard to define" glue code, where even humans struggle and AI will fabricate fake cases.
Quantify the baseline with coverage.py (Python) and jest's built-in coverage (JS), focusing on uncovered line numbers, not the total percentage:
# Install: pip install pytest pytest-cov
# Run: pytest --cov=yourpkg --cov-report=term-missing
# Output lists uncovered line numbers per file; AI-generated tests target those lines
import subprocess, re
def coverage_by_module(pkg: str) -> dict:
"""Run pytest-cov and parse per-module statement coverage and uncovered lines"""
out = subprocess.run(
["pytest", f"--cov={pkg}", "--cov-report=term-missing", "-q"],
capture_output=True, text=True,
)
rows = {}
for line in out.stdout.splitlines():
# e.g. "src/utils.py 42 8 81% 12-15, 30"
m = re.match(r"^(\S+\.py)\s+(\d+)\s+(\d+)\s+(\d+)%\s*(.*)$", line)
if m:
path, stmts, miss, pct, missing = m.groups()
rows[path] = {
"statements": int(stmts),
"missing_lines": int(miss),
"percent": int(pct),
"missing": missing.strip(),
}
return rows// package.json script: "test": "jest --coverage --collectCoverageFrom='src/**/*.{js,ts}'"
// Run: npm test -- --coverage
// Output table lists % Stmts / % Branch / Uncovered Lines per file
// Parse jest's coverage-summary.json for a baseline snapshot
const fs = require('fs');
function coverageBaseline(reportPath = 'coverage/coverage-summary.json') {
const summary = JSON.parse(fs.readFileSync(reportPath, 'utf8'));
const rows = {};
for (const [file, m] of Object.entries(summary)) {
rows[file] = {
lines: m.lines.pct,
branches: m.branches.pct,
uncovered: m.lines.pct < 100,
};
}
return rows;
}
// Compare snapshots: after AI generates tests, uncovered lines should truly shrinkRecord the baseline, generate tests with AI, then run again and compare whether uncovered line numbers actually shrank. If coverage rose but uncovered lines are unchanged, AI is testing already-covered paths and just inflating the number.
2. Generate Unit Tests From Existing Code
What you feed the LLM is not the three words "write tests," but the function signature + full implementation + explicit case requirements (normal/boundary/exception). Ask the model to output in given-when-then structure, explicitly forbid "not-None-is-pass" loose assertions, and forbid mocking the function under test.
import ast
from openai import OpenAI
client = OpenAI() # OpenAI-compatible endpoint; model name per official docs
GEN_UNIT_PROMPT = """You are a test engineer. Below is the source of the function under test. Generate unit tests with pytest, requirements:
1. Cover the normal path, boundary values (empty, zero, negative, max), and exception inputs;
2. Each test states its intent with a given-when-then comment;
3. Assertions must check a concrete return value or thrown exception type. Do not only write assert result is not None;
4. Do not mock the function under test. Output code only, no explanation.
Source:
{code}
"""
def gen_unit_tests(source_path: str, func_name: str) -> str:
"""Read a function from source and generate a pytest draft"""
src = open(source_path, encoding="utf-8").read()
tree = ast.parse(src)
for node in ast.walk(tree):
if isinstance(node, ast.FunctionDef) and node.name == func_name:
seg = ast.get_source_segment(src, node)
resp = client.chat.completions.create(
model="qwen-plus", # per official docs
messages=[{"role": "user",
"content": GEN_UNIT_PROMPT.format(code=seg)}],
temperature=0.2,
)
return resp.choices[0].message.content
raise ValueError(f"function {func_name} not found")const fs = require('fs');
const { OpenAI } = require('openai'); // npm i openai; model name per official docs
const client = new OpenAI();
const GEN_UNIT_PROMPT = `You are a test engineer. Below is the source of the function under test. Generate unit tests with Jest, requirements:
1. Cover the normal path, boundary values (empty, zero, negative, max), and exception inputs;
2. Each test states its intent with a given-when-then comment;
3. Assertions must check a concrete return value or thrown exception type. Do not only write expect(x).toBeDefined();
4. Do not mock the function under test. Output code only.
Source:
{code}
`;
async function genUnitTests(sourcePath, funcName) {
const src = fs.readFileSync(sourcePath, 'utf8');
// Rough regex extraction (use @babel/parser for precise AST extraction in production)
const re = new RegExp(
`(?:export\\s+)?(?:async\\s+)?function\\s+${funcName}[\\s\\S]*?^\\}`, 'm');
const match = src.match(re);
if (!match) throw new Error(`function ${funcName} not found`);
const resp = await client.chat.completions.create({
model: 'qwen-plus', // per official docs
messages: [{ role: 'user',
content: GEN_UNIT_PROMPT.replace('{code}', match[0]) }],
temperature: 0.2,
});
return resp.choices[0].message.content;
}After generation, run it locally immediately. If it fails, paste the error back and let the model self-fix for 2-3 rounds, rather than hand-patching. Hand-patching hides prompt flaws, so the next batch fails the same way. This generate -> run -> paste-error -> regenerate loop can be orchestrated with a prompt-chaining framework like LangChain, but the core is feeding failures back for the model to fix.
3. Generate Regression Tests From Issue/PR Descriptions
Reproducing a production bug is the test most worth automating. Feed the issue description (repro steps, expected/actual) to the model and ask it to first produce a failing test that reproduces the bug, then confirm it turns green after the fix. Compared to writing tests from scratch, regression tests have a natural anchor: the issue states "what actually happened," and the model only needs to translate that into an assertion. But issue descriptions often lack key context (input data, environment version), so fill those in before feeding, or the model will fabricate a self-consistent repro scenario.
REGRESSION_PROMPT = """This is a GitHub issue. Write a regression test with pytest:
1. First construct an input that triggers the bug per the issue's repro steps;
2. The test should fail before the fix and pass after;
3. Use pytest.raises or an explicit assertion to lock the expected behavior;
4. Tag the issue number in the test name and comments.
issue #{number}:
{body}
"""
def gen_regression_test(number: int, body: str) -> str:
resp = client.chat.completions.create(
model="qwen-plus", # per official docs
messages=[{"role": "user",
"content": REGRESSION_PROMPT.format(number=number, body=body)}],
temperature=0.2,
)
return resp.choices[0].message.contentconst REGRESSION_PROMPT = `This is a GitHub issue. Write a regression test with Jest:
1. First construct an input that triggers the bug per the issue's repro steps;
2. The test should fail before the fix and pass after;
3. Use expect(() => fn(...)).toThrow(...) or an explicit assertion to lock the expected behavior;
4. Tag the issue number in the test name and comments.
issue #{number}:
{body}
`;
async function genRegressionTest(number, body) {
const resp = await client.chat.completions.create({
model: 'qwen-plus', // per official docs
messages: [{ role: 'user',
content: REGRESSION_PROMPT
.replace('{number}', number).replace('{body}', body) }],
temperature: 0.2,
});
return resp.choices[0].message.content;
}Key discipline: first run this test on the unfixed branch and confirm it is red, then merge the fix and confirm it is green. If it is green even unfixed, the test never reproduced the bug and is a fake regression test, throw it out.
4. Batch-Generate Boundary and Parameterized Cases
Boundary cases are the best fit for parameterization: one table of inputs, one test function runs them all. Let AI only generate the parameter table (input + expected), and you apply a fixed parameterization template, so AI's failure surface shrinks to data rows, not test structure.
import pytest
# AI only produces this table; after human review it goes into the template
PARAMS = [
("", 0), # empty string
("abc", 3), # normal
("a" * 1000, 1000), # long string
(None, TypeError), # invalid type
("中文测试", 4), # multibyte
]
@pytest.mark.parametrize("s,expected", PARAMS,
ids=["empty", "normal", "long", "invalid", "multibyte"])
def test_len_safe(s, expected):
"""A length function with type validation"""
if s is None:
with pytest.raises(TypeError):
len_safe(s)
else:
assert len_safe(s) == expected
def len_safe(s):
if s is None:
raise TypeError("input must be a string")
return len(s)// Jest parameterization: test.each with an AI-generated table
const PARAMS = [
{ input: '', expected: 0, id: 'empty' },
{ input: 'abc', expected: 3, id: 'normal' },
{ input: 'a'.repeat(1000), expected: 1000, id: 'long' },
{ input: null, expected: 'throw', id: 'invalid' },
{ input: '中文测试', expected: 4, id: 'multibyte' },
];
test.each(PARAMS)('lenSafe($id) -> $expected', ({ input, expected }) => {
if (expected === 'throw') {
expect(() => lenSafe(input)).toThrow(TypeError);
} else {
expect(lenSafe(input)).toBe(expected);
}
});
function lenSafe(s) {
if (s === null || s === undefined) {
throw new TypeError('input must be a string');
}
return s.length;
}Have AI output a JSON parameter table (not a full test code block), review each row's expected value, then apply the template. This minimizes the risk of fabricated assertions, since AI can only fabricate data, not the test skeleton.
5. Test Maintenance: Detect Stale Assertions and Remove Redundancy
Tests rot. AI generates dozens at once; six months later some assertions are stale (the implementation they depended on changed), some are duplicates, and some are forever assert True. Maintenance matters more than generation, or the test suite becomes what the literature calls "technical debt whose cost outweighs its value."
import ast, re
def find_smell_tests(test_path: str) -> dict:
"""Scan a pytest file for three categories of test smells"""
src = open(test_path, encoding="utf-8").read()
tree = ast.parse(src)
always_pass, tautology, names = [], [], []
for node in ast.walk(tree):
if isinstance(node, ast.FunctionDef) and node.name.startswith("test_"):
body_src = ast.get_source_segment(src, node)
names.append(node.name)
# Smell 1: only assert True / assert 1
if re.search(r"assert\s+(True|1)\b", body_src):
always_pass.append(node.name)
# Smell 2: tautology assert x == x (capture group for backreference)
if re.search(r"assert\s+(\w+)\s*==\s*\1\b", body_src):
tautology.append(node.name)
# Smell 3: duplicate test names (pytest runs only one, the rest silently lost)
seen, dups = set(), []
for n in names:
if n in seen:
dups.append(n)
seen.add(n)
return {"always_pass": always_pass,
"tautology": tautology, "duplicates": dups}const fs = require('fs');
function findSmellTests(testPath) {
const src = fs.readFileSync(testPath, 'utf8');
const smells = { noAssertions: [], tautology: [], duplicateNames: [] };
const names = new Set();
// Rough regex scan (use @babel/parser or ts-morph for precise analysis)
const testBlocks = src.match(/(?:test|it)\(['"`](.+?)['"`][\s\S]*?\}\);/g) || [];
for (const block of testBlocks) {
const nameMatch = block.match(/(?:test|it)\(['"`](.+?)['"`]/);
const name = nameMatch ? nameMatch[1] : '<anon>';
if (names.has(name)) smells.duplicateNames.push(name);
names.add(name);
if (!/expect\(/.test(block)) smells.noAssertions.push(name);
// Tautology expect(x).toBe(x) (capture group)
if (/expect\(([^)]+)\)\.toBe\(\1\)/.test(block)) smells.tautology.push(name);
}
return smells;
}After detecting smells, you can have AI regenerate those tests, but a human must review: AI removing redundancy tends to delete cases that look duplicated but actually test different branches.
6. Human Review Gate + CI Integration
The only channel for AI test drafts into the main branch: a human-gate plus CI double gate. CI must pass all tests with no coverage regression; humans must review assertion quality, not quantity. Both gates are indispensable: CI alone lets green bars through every "runs but tests nothing" dummy case; human review alone means a fatigued reviewer will eventually miss a mutually exclusive branch. The gate is not process overhead, it is the only forge that turns an AI draft into a trustworthy test.
# .github/workflows/ai-tests.yml - AI test gate CI
name: ai-test-gate
on: [pull_request]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: '3.12' }
- run: pip install pytest pytest-cov
# Full test suite + coverage threshold; failure blocks merge
- run: pytest --cov=src --cov-fail-under=70 -q
- uses: actions/setup-node@v4
with: { node-version: '20' }
- run: npm ci && npm test -- --coverageHuman review checklist (paste into the PR template):
- Every assertion checks a concrete value or exception type, no
is not None/toBeDefined()padding - No implicit dependencies between cases (can run in any order, individually)
- Mocks target external dependencies, not the function under test
- Regression tests confirmed red on the unfixed branch
- Coverage gains come from shrinking uncovered lines, not re-testing covered paths
7. Pitfalls
Pitfall 1: AI assertions too loose. The model loves assert result is not None, expect(x).toBeTruthy(), which pass for almost any return value and test nothing. The first review pass is rewriting every "not-None-is-pass" assertion into a concrete value comparison.
Pitfall 2: Tests depend on each other. If AI shares global state or relies on execution order, deleting one case cascades failures. Use pytest fixtures for isolation, reset state before each jest test, and run in random order in CI (such as the pytest-randomly plugin) to expose implicit dependencies.
Pitfall 3: Fake coverage. AI tests trivial getters/setters and already-covered paths heavily, so the coverage number rises but uncovered lines are unchanged. Watch the specific line numbers in --cov-report=term-missing, not the total percentage.
Pitfall 4: Mock drift. AI writes mocks against its imagined interface, mismatching the real API's field names or return structure. Tests stay green while production breaks. Periodically run de-mocked integration tests against the real interface, or use contract tests to lock the mock shape.
Pitfall 5: Not testing the real path. AI mocks out every dependency the function under test calls internally, so you end up testing the mock, not the logic, what the community calls "mocked confidence." Rule: mocks are only for external I/O (network/disk/clock); the function under test's own branches must run real code.
Pitfall 6: Ignoring flaky tests. AI-generated tests sometimes pass or fail depending on execution order, system time, or random seeds, and teams instinctively mask them with @pytest.mark.skip or retries. But flakiness is a signal, not noise: every flaky test points to an implicit dependency or non-deterministic behavior, and you must fix the root cause (freeze time, isolate state, lock the random seed) rather than skip. A skipped flaky test is no test at all, yet still gives the illusion of "covered."
FAQ
Q1: What if AI-generated tests don't run?
Do not hand-patch them into running. A failure means the prompt or the context you fed is insufficient. Paste the error back and let the model self-fix for 2-3 rounds; if it still fails, add the function signature/type annotations/call examples and retry. Hand-patching hides prompt flaws, so the next batch fails the same way.
Q2: How much coverage is enough?
There is no universal number. 70% is a common starting threshold (--cov-fail-under=70), but core business logic should aim for 90%+, while glue/UI code at 50% may be reasonable. More important than total coverage: are critical branches tested, and are uncovered lines converging. Inflating coverage is easy; testing correctly is hard.
Q3: Which code is unsuitable for AI test generation?
Glue code whose behavior strongly depends on external systems (databases, third-party APIs, message queues) and whose expected output is hard to define; non-deterministic logic involving time/concurrency/randomness; and code under frequent change, where tests get rewritten as the implementation shifts. For these, have humans write contract tests or integration tests first as a foundation.
Q4: How do I stop AI from fabricating cases (invented expected values)?
Have AI output only the "input" column of the parameter table; fill the "expected" column yourself by running the current implementation (assuming it is deemed correct), then lock it as a regression baseline. Or have AI also provide the reasoning behind each expected value, and review each row. Never blindly trust AI-supplied expected values.
Q5: Human review is too slow, can it be fully automated?
No. The core risk of AI tests is "looks green but tests nothing," and full automation hands the quality gate to a model that fabricates assertions. What you can automate are the "runs clean" and "no coverage regression" gates; assertion quality needs a human. Compromise: AI generates + auto-runs clean + human spot-checks high-risk modules (payments, auth, data migrations).
Perspective
The real value of AI test generation is not "writing fewer tests," but "turning tests from a luxury into a daily necessity." When a pure function has 12 boundary combinations, a human lazily writes 3; AI can lay out all 12 parameterized cases in 30 seconds, provided you constrain it with a parameterization template so it only produces data, not code. The litmus test for whether AI tests are actually useful is singular: delete one line of business logic from the function under test and see whether the tests turn red. If they stay green after deleting logic, they never tested that logic and are just padding the coverage number. But the safety valve of this whole flow is always the human review gate: AI drafts must be human-reviewed and CI-green before merge, and this iron rule cannot bend a single letter. Our site's AI Code Refactoring SOP covers refactoring, this one covers verification, and the two form a closed loop, refactoring lets you dare to change, tests let you dare to refactor. Bypassing human review and pushing AI tests straight to the main branch gives a beautiful coverage curve in the short term, and a rotting test suite with a false sense of security more dangerous than no tests at all in the long term.
References