Skip to content
Devtools

Flaky Test Detector

Identifies tests that randomly fail and why.

Intermediate24h1–3 people15/20 fit
TL;DR

Build Flaky Test Detector for engineering teams with ci in ~24h with 1–3 people. Stack: Python + JUnit XML + React + Charts.

Why it wins: Every dev team feels this pain.

Build flow

01 · Problem
Flaky tests waste hours and erode trust in CI.
02 · Solution
Parse CI test reports over many runs, score flakiness, and group failures by likely cause.
03 · Build
  1. 1Parse JUnit XML
  2. 2Pass/fail history per test
  3. 3Cluster error messages
  4. 4Dashboard
04 · Demo
Load sample CI history → top 3 flaky tests with failure patterns.

MVP checklist

  • Report ingest
  • Flakiness score
  • Failure clustering
  • Trend chart

Tech stack

TechRole
PythonBackend / scripting
JUnit XMLTest reports
ReactFrontend UI
ChartsData visualisation

Related ideas

More Devtools

  • Devtools

    Visual Regression Bot

    Screenshot key pages on each PR and highlight visual changes

    Intermediate16h1–2
    PlaywrightGitHub ActionsNode.js
    17/20

  • Devtools

    Prompt Regression Tests

    Catch quality drops when prompts or models change

    Intermediate14h1–2
    Node.jsLLMGitHub Actions
    17/20

  • Devtools

    Pull Request Risk Score

    Score each PR for risk so reviewers focus where it matters

    Intermediate16h1–2
    GitHub APINode.jsLLM
    18/20