When a Test Fails at 3 AM: Deterministic Counterexample Repair vs. Blind Retry Loops

Why re-running the same prompt at higher temperature fails during unattended runs, and how AgentAFK's flight recorder extracts minimal counterexample vectors to guide deterministic repair.

When a candidate solution fails a test at 3:00 AM during an unattended run, a naive agent harness simply appends "Tests failed, please try again" and samples the model again. Without a concrete counterexample, the model guesses blindly, oscillating between two broken edits until its call budget runs out. agentafk.si replaces blind retries with Flight-Recorder Counterexample Self-Repair.

How the Flight Recorder extracts a minimal repair target

Every time verifier.py rejects a candidate during an overnight run, the flight recorder captures a structured failure tuple before the next turn begins:

• Pinpoint the violated invariant: Identifies the exact rule ID (rule_04_watertight_manifold or rule_11_no_ssrf_redirect) that failed.
• Shrink the failing input: Extracts the minimal input parameters (such as the exact vertex indices (u=142, v=143) with valence == 1, or the exact velocity vector that tunneled through collider #3).
• Lock already-passing invariants: Instructs the repair turn that Rules 1 through 3 and 5 through 14 are already passing and must not be regressed by the patch.

{
  "flight_recorder_event": "counterexample_repair_dispatch",
  "attempt_number": 2,
  "passing_rules_locked": [1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
  "failed_rule": {
    "rule_id": "rule_04_watertight_manifold",
    "open_boundary_edges_count": 12,
    "sample_unwelded_seam": {"u_xyz": [0.55, -1.1, 0.0], "v_xyz": [0.55, -1.1, -1e-7]},
    "repair_hint": "Modulo wrap final radial column i == radialSegs - 1 to column 0 instead of duplicating seam vertices"
  }
}

By feeding the exact unwelded seam coordinates and locking the 13 already-passing rules, gemma4:31b repairs the mesh generator on Attempt 2 without regressing lighting or physics.