Are Solved Issues in SWE-bench Really Solved Correctly? An Empirical Study

PatchDiff: test pass is not behavioral equivalence

编译摘要

1. Executable tests are necessary but not exhaustive

The study finds patches that pass SWE-bench's validation yet fail broader developer-written tests.

Therefore:

benchmark tests pass
  ≠
behavioral correctness

2. Developer patch is useful as a differential reference, but not unique truth

PatchDiff finds 29.6% of plausible patches behave differently from the developer patch. Yet only a subset of divergent behavior is manually confirmed wrong.

So:

behavior differs from developer patch
  ≠
automatically incorrect

The developer patch is a strong comparison anchor, not a complete semantic oracle.

3. Differential testing needs adjudication

PatchDiff's stable role is:

test-pass candidate
  → differential behavior probe
  → discrepancy
  → semantic/manual adjudication

This adds a behavioral-equivalence layer between test execution and final reference truth.

4. 对标

• 对 Verifiable-Agent-Engineering：execution truth must not collapse into finite test-suite pass.
• 对 Evaluation-Integrity：a benchmark can overstate capability when tests under-specify behavior.

关联概念

• Verifiable-Agent-Engineering
• Evaluation-Integrity