QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code

Paper (2026-10-09, arXiv:2610.13811) — Meriem Terki, Nouhaila Innan, Minghao Shao, Muhammad Kashif, Alberto Marchisio, and Muhammad Shafique. Paper

What it does

QTestBench asks whether a test generated by a large language model can distinguish a correct quantum implementation from a faulty one. The framework keeps an independent benchmark checker outside the test suite as the reference. It generates syntactic, semantic, and behavioral tests, injects candidate implementations, and compares test verdicts with the checker labels.

Across eight PennyLane QHack challenges and ten Qiskit HumanEval tasks, generated tests usually executed but discriminated poorly. The best generator reached the paper's defined test-quality metric of 0.71 on QHack and 0.34 on Qiskit HumanEval. Oracle-verified synthetic mutants provide more stable estimates when naturally faulty solutions are scarce.

Contributions

• A reproducible, cross-framework protocol that separates executability from discrimination.
• An independent oracle design that prevents the tests from defining their own ground truth.
• Evidence that LLM-generated quantum tests require quality evaluation rather than execution checks alone.

Limits

The benchmark covers a limited set of tasks and frameworks. Results depend on the independent checker and its labels, and the work evaluates generated tests rather than universal formal proofs of quantum programs.

Connections

The paper extends quantum software testing and mutation testing to LLM-generated suites. It complements symbolic verification in Verification Methods and Tools. It can be compared with Retromorphic Testing of Quantum Compiler Passes, which targets compiler transformations rather than test generation.

Reference

Terki et al., QTestBench: A Test Framework for Evaluating LLM-Generated Quantum Code, arXiv:2610.13811, 2026.