Built a prototype for the LFX Mentorship on agentic CI flake categorization, designed to sit downstream of the open PR restoring logformatter rather than compete with it. Mined three proxy signals for ground truth, weighted by strength: rerun disagreement on an identical commit SHA, the same test failing across unrelated commits, and post-merge failures on main.
Case Study / Open-source prototype
Podman Flake Agent
Podman's May 2026 migration off Cirrus CI orphaned logformatter, the 38KB Perl script that classified every subtest as pass, fail, skip, or flake. Triage fell back to a human reading a bar graph, and the project does not re-run failing tests, so the obvious ground-truth signal (failed then passed on retry) is not handed to you.
Standard-library Python with the model SDK imported lazily, treating logformatter's HTML as a stream of annotated lines rather than a DOM tree, because the failure summary and its diagnostic detail never share a subtree. Ships an evaluation harness that scores verdicts against labelled real failures.
Cut log payload 76% to 93% across Podman's own logformatter fixtures before any inference runs, which is what makes 30+ jobs per PR affordable to analyse. Abstention is a first-class verdict, and the harness explicitly counts the dangerous confusion: real bugs waved through as re-runnable flakes.
Technical highlights
What to inspect.
01
Ground truth mined from three weighted proxy signals, because the project sets GINKGO_FLAKE_ATTEMPTS to 0 and never auto-reruns.
02
Line-oriented parsing yielding per-test granularity, which is what makes deduplication against existing flake issues possible at all.
03
Non-destructive classification history via ADD, UPDATE, INVALIDATE, or NOOP, so a flake that was diagnosed, fixed, then regressed stays auditable.
04
ETag-revalidated caching where a 304 consumes no rate-limit quota, and a committed dossier corpus so the agent can be iterated with no token and no API budget.