Skip to content

How does Bifrost measure up on semantic data flow?

Immutable evaluation · DataFlowBench v0.6.0

c kernel · correct decisions

Bifrost

40/48

40 of 40 decided

CodeQL

41/48

Infer

43/48

Semgrep CE

12/48

12 of 14 decided

24 templates · unanswered outcomes (Bifrost: 8 incomplete; Semgrep CE: 34 declined) are coverage, excluded from correctness and never counted against it

cpp kernel · correct decisions

Bifrost

30/56

30 of 30 decided

CodeQL

42/56

Infer

47/56

Semgrep CE

12/56

12 of 14 decided

28 templates · unanswered outcomes (Bifrost: 26 incomplete; Semgrep CE: 42 declined) are coverage, excluded from correctness and never counted against it

csharp kernel · correct decisions

Bifrost

32/58

32 of 34 decided

CodeQL

47/58

29 templates · unanswered outcomes (Bifrost: 24 incomplete) are coverage, excluded from correctness and never counted against it

go kernel · correct decisions

Bifrost

35/58

35 of 36 decided

CodeQL

45/58

Semgrep CE

12/58

12 of 14 decided

29 templates · unanswered outcomes (Bifrost: 22 incomplete; Semgrep CE: 44 declined) are coverage, excluded from correctness and never counted against it

java kernel · correct decisions

Bifrost

37/58

37 of 38 decided

CodeQL

48/58

FlowDroid

49/58

Infer

50/58

Joern

47/58

OpenTaint

49/58

Semgrep CE

12/58

12 of 14 decided

29 templates · unanswered outcomes (Bifrost: 20 incomplete; Semgrep CE: 44 declined) are coverage, excluded from correctness and never counted against it

javascript kernel · correct decisions

Bifrost

36/58

36 of 36 decided

CodeQL

48/58

Joern

44/58

Semgrep CE

12/58

12 of 14 decided

29 templates · unanswered outcomes (Bifrost: 22 incomplete; Semgrep CE: 44 declined) are coverage, excluded from correctness and never counted against it

kotlin kernel · correct decisions

Bifrost

28/58

28 of 28 decided

CodeQL

46/58

FlowDroid

49/58

OpenTaint

50/58

Semgrep CE

12/58

12 of 14 decided

29 templates · unanswered outcomes (Bifrost: 30 incomplete; Semgrep CE: 44 declined) are coverage, excluded from correctness and never counted against it

php kernel · correct decisions

Bifrost

32/58

32 of 32 decided

Joern

48/58

Semgrep CE

12/58

12 of 14 decided

29 templates · unanswered outcomes (Bifrost: 26 incomplete; Semgrep CE: 44 declined) are coverage, excluded from correctness and never counted against it

python kernel · correct decisions

Bifrost

36/58

36 of 36 decided

CodeQL

48/58

Joern

48/58

Pysa

47/58

Semgrep CE

12/58

12 of 14 decided

29 templates · unanswered outcomes (Bifrost: 22 incomplete; Semgrep CE: 44 declined) are coverage, excluded from correctness and never counted against it

ruby kernel · correct decisions

Bifrost

21/58

21 of 22 decided

CodeQL

49/58

Joern

40/58

Semgrep CE

12/58

12 of 14 decided

29 templates · unanswered outcomes (Bifrost: 36 incomplete; Semgrep CE: 44 declined) are coverage, excluded from correctness and never counted against it

rust kernel · correct decisions

Bifrost

36/54

36 of 36 decided

CodeQL

44/54

Joern

43/54

Semgrep CE

12/54

12 of 14 decided

27 templates · unanswered outcomes (Bifrost: 18 incomplete; Semgrep CE: 40 declined) are coverage, excluded from correctness and never counted against it

scala kernel · correct decisions

Bifrost

38/58

38 of 38 decided

29 templates · unanswered outcomes (Bifrost: 20 incomplete) are coverage, excluded from correctness and never counted against it

typescript kernel · correct decisions

Bifrost

34/58

34 of 34 decided

CodeQL

48/58

Semgrep CE

12/58

12 of 14 decided

29 templates · unanswered outcomes (Bifrost: 24 incomplete; Semgrep CE: 44 declined) are coverage, excluded from correctness and never counted against it

Direct-flow breadth · languages fully correct

Bifrost

13/13

one positive and one negative direct-propagation assertion per language

Confusion matrix · DataFlowBench v0.6.0

Decisive outcomes only. inconclusive, unsupported and runner-error are capability coverage: they are excluded from the matrix and from the rates, and are never converted into clean negatives.

What this result means

DataFlowBench measures whether analyzers correctly decide semantic data-flow questions — and whether they stay quiet when they should. Cases are balanced positive/negative pairs of language-neutral semantic templates (aliasing, kills, call context, branch joins, exception paths, …). The kernels and the modeling matrix run under a benchmark-controlled model profile, so the engines are compared under a common contract; the tool-native probes run under a tool-native profile, measuring the shipped product instead. The two profiles answer different questions and are never combined.

This snapshot's bounded claim covers the synthetic direct-flow breadth baseline and the propagation kernels of the 13 kernel languages above on the taint track, plus the modeling matrix and the tool-native probe set in java, javascript, and python. Cores are sized per language (24, 27, 28 or 29 templates), so each kernel is read on its own denominator; language-only constructs are reported in separate language-extension tiers on the snapshot pages. It does not estimate real-project accuracy or other languages' kernel behavior; performance is characterized separately in the latency tier, on its own terms, and never folded into these scores. The tool-native rows describe shipped coverage on six probe templates rather than product accuracy at large. inconclusive, unsupported, and runner-error are capability coverage and are never converted into clean negatives.

3336

frozen case results, across three populations

13

languages in the breadth baseline

13

kernel languages (24, 27, 28 or 29 templates each)

v0.6.0

immutable evidence release

How the benchmark and its analyzers evolved

One series per analyzer, across every published freeze. The vertical axis counts decisive-correct assertions — true positives plus true negatives — on the benchmark-controlled kernel population, the same no-pooling population the cards at the top of this page read. The grey step is the benchmark itself: the total core population of that snapshot, which grew from 32 assertions in v0.1.0 to 738 in v0.6.0. Nothing here is pooled with the modeling matrix or the tool-native probes, and there is no combined cross-analyzer score: read each line on its own. The second view divides each analyzer's decisive-correct count by the population it actually covers, so a narrow language footprint no longer reads as a weak result. The third view drops the drawing entirely and lists every figure as a table.

Decisive-correct assertions per analyzer across DataFlowBench snapshots A stepped grey line shows each snapshot's total benchmark-controlled core population, growing from 32 assertions in v0.1.0 to 738 in v0.6.0. Beneath it, one line per analyzer plots that analyzer's decisive-correct count, with a faint stub reaching that analyzer's own covered population. Every marker also carries its exact figures in a tooltip, and the toggle above this chart has a Data table view that lists every value as text. 0 200 400 600 800 32 96 316 738 738 738 Bifrost · v0.1.0 · 17 decisive-correct of 32 covered (1 of 1 kernels) = 53.1%; secondary, correct ÷ decided = 17 of 22 Bifrost · v0.2.0 · 52 decisive-correct of 96 covered (3 of 3 kernels) = 54.2%; secondary, correct ÷ decided = 52 of 68 Bifrost · v0.3.0 · 163 decisive-correct of 316 covered (10 of 10 kernels) = 51.6%; secondary, correct ÷ decided = 163 of 166 Bifrost · v0.4.0 · 222 decisive-correct of 738 covered (13 of 13 kernels) = 30.1%; secondary, correct ÷ decided = 222 of 227 Bifrost · v0.5.0 · 435 decisive-correct of 738 covered (13 of 13 kernels) = 58.9%; secondary, correct ÷ decided = 435 of 440 Bifrost · v0.6.0 · 435 decisive-correct of 738 covered (13 of 13 kernels) = 58.9%; secondary, correct ÷ decided = 435 of 440 CodeQL · v0.1.0 · 27 decisive-correct of 32 covered (1 of 1 kernels) = 84.4%; secondary, correct ÷ decided = 27 of 32 CodeQL · v0.2.0 · 84 decisive-correct of 96 covered (3 of 3 kernels) = 87.5%; secondary, correct ÷ decided = 84 of 96 CodeQL · v0.3.0 · 276 decisive-correct of 316 covered (10 of 10 kernels) = 87.3%; secondary, correct ÷ decided = 276 of 316 CodeQL · v0.4.0 · 506 decisive-correct of 622 covered (11 of 13 kernels) = 81.4%; secondary, correct ÷ decided = 506 of 622 CodeQL · v0.5.0 · 506 decisive-correct of 622 covered (11 of 13 kernels) = 81.4%; secondary, correct ÷ decided = 506 of 622 CodeQL · v0.6.0 · 506 decisive-correct of 622 covered (11 of 13 kernels) = 81.4%; secondary, correct ÷ decided = 506 of 622 Joern · v0.4.0 · 270 decisive-correct of 344 covered (6 of 13 kernels) = 78.5%; secondary, correct ÷ decided = 270 of 344 Joern · v0.5.0 · 270 decisive-correct of 344 covered (6 of 13 kernels) = 78.5%; secondary, correct ÷ decided = 270 of 344 Joern · v0.6.0 · 270 decisive-correct of 344 covered (6 of 13 kernels) = 78.5%; secondary, correct ÷ decided = 270 of 344 Semgrep CE · v0.4.0 · 132 decisive-correct of 622 covered (11 of 13 kernels) = 21.2%; secondary, correct ÷ decided = 132 of 154 Semgrep CE · v0.5.0 · 132 decisive-correct of 622 covered (11 of 13 kernels) = 21.2%; secondary, correct ÷ decided = 132 of 154 Semgrep CE · v0.6.0 · 132 decisive-correct of 622 covered (11 of 13 kernels) = 21.2%; secondary, correct ÷ decided = 132 of 154 OpenTaint · v0.6.0 · 99 decisive-correct of 116 covered (2 of 13 kernels) = 85.3%; secondary, correct ÷ decided = 99 of 116 Infer · v0.6.0 · 140 decisive-correct of 162 covered (3 of 13 kernels) = 86.4%; secondary, correct ÷ decided = 140 of 162 FlowDroid · v0.6.0 · 98 decisive-correct of 116 covered (2 of 13 kernels) = 84.5%; secondary, correct ÷ decided = 98 of 116 Pysa · v0.6.0 · 47 decisive-correct of 58 covered (1 of 13 kernels) = 81.0%; secondary, correct ÷ decided = 47 of 58 v0.1.0 1 kernel v0.2.0 3 kernels v0.3.0 10 kernels v0.4.0 13 kernels v0.5.0 13 kernels v0.6.0 13 kernels assertions
Decisive-correct share of each analyzer's own covered population across DataFlowBench snapshots One line per analyzer on a nought to one hundred percent axis. Each point is that analyzer's decisive-correct count divided by the assertions in the kernels it covers, with inconclusive and unsupported outcomes left in the denominator. Marker size grows with the share of the snapshot's kernels the analyzer covers, and every point is annotated with that kernel count. Every marker also carries its exact figures in a tooltip, and the toggle above this chart has a Data table view that lists every value as text, including the denominator each percentage is over. 0% 20% 40% 60% 80% 100% Bifrost · v0.1.0 · 53.1% — 17 of 32 covered (1 of 1 kernel); secondary, correct ÷ decided = 17 of 22 1/1 Bifrost · v0.2.0 · 54.2% — 52 of 96 covered (3 of 3 kernels); secondary, correct ÷ decided = 52 of 68 3/3 Bifrost · v0.3.0 · 51.6% — 163 of 316 covered (10 of 10 kernels); secondary, correct ÷ decided = 163 of 166 10/10 Bifrost · v0.4.0 · 30.1% — 222 of 738 covered (13 of 13 kernels); secondary, correct ÷ decided = 222 of 227 13/13 Bifrost · v0.5.0 · 58.9% — 435 of 738 covered (13 of 13 kernels); secondary, correct ÷ decided = 435 of 440 13/13 Bifrost · v0.6.0 · 58.9% — 435 of 738 covered (13 of 13 kernels); secondary, correct ÷ decided = 435 of 440 13/13 CodeQL · v0.1.0 · 84.4% — 27 of 32 covered (1 of 1 kernel); secondary, correct ÷ decided = 27 of 32 1/1 CodeQL · v0.2.0 · 87.5% — 84 of 96 covered (3 of 3 kernels); secondary, correct ÷ decided = 84 of 96 3/3 CodeQL · v0.3.0 · 87.3% — 276 of 316 covered (10 of 10 kernels); secondary, correct ÷ decided = 276 of 316 10/10 CodeQL · v0.4.0 · 81.4% — 506 of 622 covered (11 of 13 kernels); secondary, correct ÷ decided = 506 of 622 11/13 CodeQL · v0.5.0 · 81.4% — 506 of 622 covered (11 of 13 kernels); secondary, correct ÷ decided = 506 of 622 11/13 CodeQL · v0.6.0 · 81.4% — 506 of 622 covered (11 of 13 kernels); secondary, correct ÷ decided = 506 of 622 11/13 Joern · v0.4.0 · 78.5% — 270 of 344 covered (6 of 13 kernels); secondary, correct ÷ decided = 270 of 344 6/13 Joern · v0.5.0 · 78.5% — 270 of 344 covered (6 of 13 kernels); secondary, correct ÷ decided = 270 of 344 6/13 Joern · v0.6.0 · 78.5% — 270 of 344 covered (6 of 13 kernels); secondary, correct ÷ decided = 270 of 344 6/13 Semgrep CE · v0.4.0 · 21.2% — 132 of 622 covered (11 of 13 kernels); secondary, correct ÷ decided = 132 of 154 11/13 Semgrep CE · v0.5.0 · 21.2% — 132 of 622 covered (11 of 13 kernels); secondary, correct ÷ decided = 132 of 154 11/13 Semgrep CE · v0.6.0 · 21.2% — 132 of 622 covered (11 of 13 kernels); secondary, correct ÷ decided = 132 of 154 11/13 OpenTaint · v0.6.0 · 85.3% — 99 of 116 covered (2 of 13 kernels); secondary, correct ÷ decided = 99 of 116 2/13 Infer · v0.6.0 · 86.4% — 140 of 162 covered (3 of 13 kernels); secondary, correct ÷ decided = 140 of 162 3/13 FlowDroid · v0.6.0 · 84.5% — 98 of 116 covered (2 of 13 kernels); secondary, correct ÷ decided = 98 of 116 2/13 Pysa · v0.6.0 · 81.0% — 47 of 58 covered (1 of 13 kernels); secondary, correct ÷ decided = 47 of 58 1/13 v0.1.0 1 kernel v0.2.0 3 kernels v0.3.0 10 kernels v0.4.0 13 kernels v0.5.0 13 kernels v0.6.0 13 kernels % of its covered population
  • Bifrost · decisive-correct
  • CodeQL · decisive-correct
  • Joern · decisive-correct
  • Semgrep CE · decisive-correct
  • OpenTaint · decisive-correct
  • Infer · decisive-correct
  • FlowDroid · decisive-correct
  • Pysa · decisive-correct
  • benchmark core population (the denominator)
  • remainder of that analyzer's own covered population
  • Bifrost · decisive-correct ÷ its covered population
  • CodeQL · decisive-correct ÷ its covered population
  • Joern · decisive-correct ÷ its covered population
  • Semgrep CE · decisive-correct ÷ its covered population
  • OpenTaint · decisive-correct ÷ its covered population
  • Infer · decisive-correct ÷ its covered population
  • FlowDroid · decisive-correct ÷ its covered population
  • Pysa · decisive-correct ÷ its covered population
  • marker size and the k/n label under it: kernels covered of the snapshot's kernels

Absolute counts plot progress against the whole growing benchmark: the marker is the analyzer's decisive-correct count and the grey step is the entire core population of that snapshot, so an analyzer that supports only part of the benchmark sits far below the step by construction. Accuracy on covered kernels instead divides each analyzer's decisive-correct count by its own covered population — the assertions in the kernels it reported on at all — so it answers how well a tool does on the slice it supports. Coverage is stated beside every number in that view: the marker grows with the share of the snapshot's kernels covered, each point carries its k/n kernel count, and the Data table view spells out the denominator, so 78% over 6 kernels is never mistaken for 78% over 13. Inconclusive and unsupported outcomes inside covered kernels stay in the denominator — they are non-answers on cases the analyzer took on. The narrower correct ÷ decided ratio is reported only as a secondary figure, in the tooltips and the last column of the data table. Data table is the third setting of the same toggle: it carries the exact figures behind both charts — every count, every denominator and every percentage — for anyone who would rather read the numbers than the shape.

Each marker sits at the analyzer's decisive-correct count. The faint stub above it reaches that analyzer's own covered population — the assertions it reported on — so the part of the stub above the marker is wrong answers plus coverage outcomes, and the remaining gap up to the grey step is kernels it does not cover at all. A snapshot without a marker is a snapshot the analyzer was not run in: Joern first appears in v0.4.0, Semgrep CE first appears in v0.4.0, OpenTaint first appears in v0.6.0, Infer first appears in v0.6.0, FlowDroid first appears in v0.6.0, and Pysa first appears in v0.6.0. Absence is never drawn as a zero and never interpolated across, so an analyzer that appears in exactly one freeze is one marker and no line — OpenTaint, Infer, FlowDroid, and Pysa are drawn that way here, and no earlier value is implied for them. inconclusive, unsupported and runner-error are capability coverage: excluded from correctness, never converted into wrong answers. Every snapshot label links to that freeze's per-case evidence.

Each marker sits at the analyzer's decisive-correct share of its own covered population; the faint stub above it runs to 100%, which is that same covered population, so the stub is wrong answers plus coverage outcomes inside the kernels it did report on. A snapshot without a marker is a snapshot the analyzer was not run in: Joern first appears in v0.4.0, Semgrep CE first appears in v0.4.0, OpenTaint first appears in v0.6.0, Infer first appears in v0.6.0, FlowDroid first appears in v0.6.0, and Pysa first appears in v0.6.0. Absence is never drawn as a zero and never interpolated across. Percentages from different snapshots are not the same exam, and percentages from different analyzers in the same snapshot are only the same exam when their k/n kernel counts match. Every snapshot label links to that freeze's per-case evidence.

Snapshot Analyzer Decisive-correct Decided Coverage outcomes Its covered population Correct ÷ its covered population Benchmark core population Secondary: correct ÷ decided
v0.1.0 Bifrost 17 22 10 32 (1 of 1 kernel) 53.1% (17 of 32 covered, 1 of 1 kernel) 32 77.3% (17 of 22 decided)
v0.1.0 CodeQL 27 32 0 32 (1 of 1 kernel) 84.4% (27 of 32 covered, 1 of 1 kernel) 32 84.4% (27 of 32 decided)
v0.2.0 Bifrost 52 68 28 96 (3 of 3 kernels) 54.2% (52 of 96 covered, 3 of 3 kernels) 96 76.5% (52 of 68 decided)
v0.2.0 CodeQL 84 96 0 96 (3 of 3 kernels) 87.5% (84 of 96 covered, 3 of 3 kernels) 96 87.5% (84 of 96 decided)
v0.3.0 Bifrost 163 166 150 316 (10 of 10 kernels) 51.6% (163 of 316 covered, 10 of 10 kernels) 316 98.2% (163 of 166 decided)
v0.3.0 CodeQL 276 316 0 316 (10 of 10 kernels) 87.3% (276 of 316 covered, 10 of 10 kernels) 316 87.3% (276 of 316 decided)
v0.4.0 Bifrost 222 227 511 738 (13 of 13 kernels) 30.1% (222 of 738 covered, 13 of 13 kernels) 738 97.8% (222 of 227 decided)
v0.4.0 CodeQL 506 622 0 622 (11 of 13 kernels) 81.4% (506 of 622 covered, 11 of 13 kernels) 738 81.4% (506 of 622 decided)
v0.4.0 Joern 270 344 0 344 (6 of 13 kernels) 78.5% (270 of 344 covered, 6 of 13 kernels) 738 78.5% (270 of 344 decided)
v0.4.0 Semgrep CE 132 154 468 622 (11 of 13 kernels) 21.2% (132 of 622 covered, 11 of 13 kernels) 738 85.7% (132 of 154 decided)
v0.5.0 Bifrost 435 440 298 738 (13 of 13 kernels) 58.9% (435 of 738 covered, 13 of 13 kernels) 738 98.9% (435 of 440 decided)
v0.5.0 CodeQL 506 622 0 622 (11 of 13 kernels) 81.4% (506 of 622 covered, 11 of 13 kernels) 738 81.4% (506 of 622 decided)
v0.5.0 Joern 270 344 0 344 (6 of 13 kernels) 78.5% (270 of 344 covered, 6 of 13 kernels) 738 78.5% (270 of 344 decided)
v0.5.0 Semgrep CE 132 154 468 622 (11 of 13 kernels) 21.2% (132 of 622 covered, 11 of 13 kernels) 738 85.7% (132 of 154 decided)
v0.6.0 Bifrost 435 440 298 738 (13 of 13 kernels) 58.9% (435 of 738 covered, 13 of 13 kernels) 738 98.9% (435 of 440 decided)
v0.6.0 CodeQL 506 622 0 622 (11 of 13 kernels) 81.4% (506 of 622 covered, 11 of 13 kernels) 738 81.4% (506 of 622 decided)
v0.6.0 Joern 270 344 0 344 (6 of 13 kernels) 78.5% (270 of 344 covered, 6 of 13 kernels) 738 78.5% (270 of 344 decided)
v0.6.0 Semgrep CE 132 154 468 622 (11 of 13 kernels) 21.2% (132 of 622 covered, 11 of 13 kernels) 738 85.7% (132 of 154 decided)
v0.6.0 OpenTaint 99 116 0 116 (2 of 13 kernels) 85.3% (99 of 116 covered, 2 of 13 kernels) 738 85.3% (99 of 116 decided)
v0.6.0 Infer 140 162 0 162 (3 of 13 kernels) 86.4% (140 of 162 covered, 3 of 13 kernels) 738 86.4% (140 of 162 decided)
v0.6.0 FlowDroid 98 116 0 116 (2 of 13 kernels) 84.5% (98 of 116 covered, 2 of 13 kernels) 738 84.5% (98 of 116 decided)
v0.6.0 Pysa 47 58 0 58 (1 of 13 kernels) 81.0% (47 of 58 covered, 1 of 13 kernels) 738 81.0% (47 of 58 decided)

Every value the two chart views draw, one row per analyzer per snapshot, with each denominator spelled out beside the figure it belongs to. This is the same data the charts plot — nothing here is computed differently, and nothing is rounded away except the percentages, which are given to one decimal place. Switch back to Absolute counts or Accuracy on covered kernels for the shape of it.

Read the shape, not a ranking. Two different things move a line: an analyzer getting better, and the benchmark getting harder. The largest expansion so far is v0.4.0, which took the core population from 316 to 738 assertions — every analyzer's share of the population fell there without any of them changing. Because the denominator is drawn alongside the counts, saturation and expansion stay distinguishable, and no two snapshots' fractions are ever compared as if they were the same exam.

Modeling matrix — is the model surface load-bearing?

A separate population from the kernels above, on the same benchmark-controlled profile. Twelve preregistered templates in six balanced categories — declared sources and sinks, declared propagators, declared sanitizers, opaque summaries, framework entry points, persistence boundaries — ask whether each tool's own model-declaration surface can express a category and be made to carry the flow. A category a tool cannot express is unsupported, decided before the tool is invoked. The scored partition differs per adapter, so the denominators differ by construction and are never pooled or ranked. Read the scored column first: it is the load-bearing number, and the ratio inside it is only meaningful against that tool's own partition.

Bifrost 0.10.7

java 8 scored · 16 declined of 24

8/8 decided correctly

javascript 8 scored · 16 declined of 24

5/5 decided correctly · 3 inconclusive

python 8 scored · 16 declined of 24

8/8 decided correctly

CodeQL 2.26.4

java 24 scored · 0 declined of 24

24/24 decided correctly

javascript 24 scored · 0 declined of 24

24/24 decided correctly

python 24 scored · 0 declined of 24

24/24 decided correctly

Joern 4.0.614

java 16 scored · 8 declined of 24

14/16 decided correctly

javascript 16 scored · 8 declined of 24

14/16 decided correctly

python 16 scored · 8 declined of 24

14/16 decided correctly

Semgrep CE 1.175.0

java 10 scored · 14 declined of 24

10/10 decided correctly

javascript 10 scored · 14 declined of 24

10/10 decided correctly

python 10 scored · 14 declined of 24

10/10 decided correctly

  • correct
  • wrong
  • inconclusive — coverage, not a wrong answer
  • declined — outside this tool's model surface, decided before it runs

Read the scored partition, not the ratio: across the 12 language tiers it runs from 8 to 24 of the 24 assertions in the tier, so no two of these bars are the same exam. The per-tier counts behind every bar are on the analyzers page, and the template-by-template model-* outcomes on the semantic templates page.

Tool-native probes — what ships and decides on its own

A third population, under the tool-native model profile: six templates run with nothing supplied by DataFlowBench, against whatever ruleset, semantics, or policy pack the product ships. This measures product coverage, not engine accuracy. Tool-native results are never pooled with the benchmark-controlled kernels or the modeling matrix, and never compared number-to-number with them. A row of unsupported is a declared decline — the tool ships no source or sink catalog for this tier — and is never counted as a wrong answer. Those runs still witness the identity of the binary and ruleset that produced them.

Bifrost 0.10.7

java 0 scored · 12 declined of 12

declines the tier — declared coverage

javascript 0 scored · 12 declined of 12

declines the tier — declared coverage

python 0 scored · 12 declined of 12

declines the tier — declared coverage

CodeQL 2.26.4

java 12 scored · 0 declined of 12

11/12 decided correctly · 1 FP

javascript 12 scored · 0 declined of 12

9/12 decided correctly · 2 FP · 1 FN

python 12 scored · 0 declined of 12

10/12 decided correctly · 2 FP

Joern 4.0.614

java 0 scored · 12 declined of 12

declines the tier — declared coverage

javascript 0 scored · 12 declined of 12

declines the tier — declared coverage

python 0 scored · 12 declined of 12

declines the tier — declared coverage

Semgrep CE 1.175.0

java 0 scored · 12 declined of 12

declines the tier — declared coverage

javascript 0 scored · 12 declined of 12

declines the tier — declared coverage

python 12 scored · 0 declined of 12

8/12 decided correctly · 4 FP

  • correct
  • wrong
  • inconclusive — coverage, not a wrong answer
  • declined — the product ships no catalog for this tier

8 of the 12 tiers decline the tier outright — a declared decline drawn as declined coverage, never as 0/12 — and the 4 that do decide answer 48 assertions between them (CodeQL and Semgrep CE). Per-tier coverage is on the analyzers page, the native-* template outcomes on the semantic templates page, and every case — including the runs that decide nothing — on the case evidence page.

How long an answer takes — beside the answers, never inside them

A descriptive characterization of per-case analyzer wall-clock, published under the contract preregistered in docs/latency-tier.md, which merged before a single timestamp was captured — because a latency page assembled after the numbers were known, by the vendor of one of the engines measured, would deserve exactly the skepticism it would get. It is not a score and not a ranking of engines. Nothing on this site pools it with correctness: there is no combined number, no efficiency-adjusted rate, and no leaderboard that blends the two, so "correct but slow" and "fast but wrong" stay independently visible. Read this section against the correctness sections above, not through them.

Whole-invocation wall-clock per analyzer, over every timed invocation in the freeze One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of the two adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 123 ms, IQR 116 ms to 133 ms, p10–p90 111 ms to 149 ms, over 883 timed whole invocations Bifrost 883 timed 123 ms Semgrep CE — median 1.20 s, IQR 1.11 s to 1.59 s, p10–p90 1.08 s to 2.47 s, over 196 timed whole invocations Semgrep CE 196 timed 1.20 s FlowDroid — median 1.70 s, IQR 1.55 s to 1.96 s, p10–p90 1.48 s to 2.08 s, over 116 timed whole invocations FlowDroid 116 timed 1.70 s Infer — median 2.21 s, IQR 554 ms to 6.24 s, p10–p90 504 ms to 6.74 s, over 162 timed whole invocations Infer 162 timed 2.21 s Infer phase capture — median 1.78 s, IQR 160 ms to 5.87 s, p10–p90 151 ms to 6.37 s. Comparable only against this adapter's own numbers. capture 1.78 s Infer phase analyze — median 373 ms, IQR 343 ms to 440 ms, p10–p90 324 ms to 500 ms. Comparable only against this adapter's own numbers. analyze 373 ms Pysa — median 4.08 s, IQR 3.88 s to 4.21 s, p10–p90 3.77 s to 4.58 s, over 58 timed whole invocations Pysa 58 timed 4.08 s OpenTaint — median 4.60 s, IQR 4.39 s to 4.98 s, p10–p90 4.22 s to 9.51 s, over 116 timed whole invocations OpenTaint 116 timed 4.60 s Joern — median 8.70 s, IQR 7.18 s to 10.1 s, p10–p90 5.17 s to 14.6 s, over 392 timed whole invocations Joern 392 timed 8.70 s CodeQL — median 10.8 s, IQR 7.22 s to 16.7 s, p10–p90 5.30 s to 39.4 s, over 734 timed whole invocations CodeQL 734 timed 10.8 s CodeQL phase database-create — median 4.51 s, IQR 2.18 s to 10.7 s, p10–p90 1.42 s to 20.4 s. Comparable only against this adapter's own numbers. database-create 4.51 s CodeQL phase database-analyze — median 5.58 s, IQR 4.64 s to 8.29 s, p10–p90 3.45 s to 15.4 s. Comparable only against this adapter's own numbers. database-analyze 5.58 s

Every timed analyzer invocation the freeze binds — 2657 of them, across every score tier and both model profiles. This is the widest denominator on the site and the only one here that is not a single population: an adapter's median mixes the languages it runs on, whose fixtures differ in size and whose front ends differ in cost. The per-kernel views on the latency page hold the language fixed.

  • median, printed beside every row
  • interquartile range (Q1–Q3)
  • p10–p90; the minimum and maximum are in the tables, not the whiskers
  • indented rows: an adapter's own declared phases

Every bar above is cold per-invocation wall-clock, and the warm marginal is a different quantity measured separately. Cold is what this benchmark actually runs — one process per case, start-up inside the number, because start-up is not observable from inside a single invocation. Read across runtimes, though, those bars overstate the steady-state gap: a JVM engine's row carries a JVM start a long-lived deployment pays once. So the other quantity is measured directly rather than estimated and subtracted — k cases through one tool process, for increasing k, reporting the slope of batch wall-clock against k. No caret is drawn in this view: the whole-corpus view mixes languages, and a marginal measured on one kernel is not a claim about a mixed-language row. The per-kernel views carry the marks. Only adapters whose released CLI exposes a real multi-case batch have a figure at all; the rest are not observable with the released CLI, and the warm-marginal section records every verdict, measured and declined, with the evidence behind it.

The axis is logarithmic. Each labelled tick is three times the one before it, so equal distances are equal ratios, not equal durations — the gap from 100 ms to 300 ms is drawn the same width as the gap from 10 s to 30 s. That is the only way this freeze's medians fit in one picture: they span 123 ms to 10.8 s, and on a linear axis every analyzer except the slowest would be a sliver against the origin. Because a log axis is easy to misread, every median is also printed as a number at the right of its own row.

The indented rows are phases, and they are not comparable across adapters. Only the 2 adapters whose preregistered row declares more than one subprocess have them. A phase mark sits on the same axis as the totals because it is the same kind of measurement — wall-clock of a subprocess — but reading one adapter's phase against another adapter's total is precisely the comparison the granularity rule forbids. Read a phase against the adapter it is indented under, and nothing else.

Ordering is not scoring, and this is never pooled with correctness. Rows are sorted by median because an unsorted ranking is unreadable, not because latency is a result. No correctness figure appears in this chart and no number here is blended with one: there is no combined score anywhere on this site, and a fast analyzer that answers wrongly is neither rewarded nor penalised by anything drawn above. Cases an analyzer declined before invocation are absent, not entered as zero — entering them as zero would make the analyzers that decline the most look the fastest, which is exactly backwards.

The conditions these numbers were produced under. A single developer machine (Mac16,1, macos 25.6.0, 10 logical CPUs), under light concurrent load, running the benchmark's standing sequential-run discipline — one analyzer at a time, never two at once. Latency numbers are comparable within one environment and are not comparable across machines. There are no repeated trials and no warm-up iterations: the case population is the sample and its spread is the statistics, so the whiskers are that spread and not error bars on a repeated measurement. Every number is a cold start: each case spawns a fresh process, so JVM tools (Joern, OpenTaint, FlowDroid) pay full JVM start-up in every invocation, and Pysa pays Pyre initialization. In a resident deployment those costs amortize to once per session, so cross-runtime comparisons here overstate the steady-state gap; the boot share of a single invocation is not adapter-observable and is included, stated rather than estimated. How much they overstate it is now measured, not left as a caveat — separately, as the warm marginal cost of one more case in a process that has already started, wherever the released CLI let it be measured. Read this as characterization of what the benchmark actually costs to run, at the order-of-magnitude and shape level, and not as a precise figure for any engine. The full contract, the per-adapter granularity, the cache declarations and every quartile are on the latency page.

Kernels at a glance

One bar per analyzer, one panel per kernel. Each bar is that analyzer's own core population, split into decisive-correct, decisive-wrong, incomplete, and declined — the last two are capability coverage, each drawn as its own segment so the remainder of a bar is never read as a wrong answer. Panels are separate populations with separate denominators: compare bars inside a panel, never across panels, and never against the modeling or tool-native sections above. Every kernel here is on the benchmark-controlled model profile.

c 48 assertions

Bifrost

40/48

CodeQL

41/48

Infer

43/48

Semgrep CE

12/48

cpp 56 assertions

Bifrost

30/56

CodeQL

42/56

Infer

47/56

Semgrep CE

12/56

csharp 58 assertions

Bifrost

32/58

CodeQL

47/58

go 58 assertions

Bifrost

35/58

CodeQL

45/58

Semgrep CE

12/58

java 58 assertions

Bifrost

37/58

CodeQL

48/58

FlowDroid

49/58

Infer

50/58

Joern

47/58

OpenTaint

49/58

Semgrep CE

12/58

javascript 58 assertions

Bifrost

36/58

CodeQL

48/58

Joern

44/58

Semgrep CE

12/58

kotlin 58 assertions

Bifrost

28/58

CodeQL

46/58

FlowDroid

49/58

OpenTaint

50/58

Semgrep CE

12/58

php 58 assertions

Bifrost

32/58

Joern

48/58

Semgrep CE

12/58

python 58 assertions

Bifrost

36/58

CodeQL

48/58

Joern

48/58

Pysa

47/58

Semgrep CE

12/58

ruby 58 assertions

Bifrost

21/58

CodeQL

49/58

Joern

40/58

Semgrep CE

12/58

rust 54 assertions

Bifrost

36/54

CodeQL

44/54

Joern

43/54

Semgrep CE

12/54

scala 58 assertions

Bifrost

38/58

typescript 58 assertions

Bifrost

34/58

CodeQL

48/58

Semgrep CE

12/58

  • decisive-correct
  • decisive-wrong
  • incomplete — inconclusive or runner-error, coverage rather than a wrong answer
  • declined — unsupported by declared capability, decided before the analyzer runs

None of the 49 analyzer rows published across these 13 kernels answers its whole core correctly, and 25 of them answer every assertion in their kernel definitively — nothing incomplete, nothing declined. The template-by-template outcome for every one of these rows — each balanced positive/negative pair, per analyzer — is on the semantic templates page, with per-case classifications and retained raw evidence on the case evidence page.

Direct-flow breadth across languages

The breadth baseline is one balanced direct-propagation pair per language, on Bifrost's all-language smoke run: a separate, much smaller population from the kernels above, and never pooled with them. Each chip is one language, with a mark for the positive and the negative assertion.

  • c
  • cpp
  • csharp
  • go
  • java
  • javascript
  • kotlin
  • php
  • python
  • ruby
  • rust
  • scala
  • typescript

Bifrost answers both assertions correctly in 13 of 13 languages. The exact reached / not-reached outcome for each of those assertions is on the semantic templates page, under the smoke scorecard's direct-propagation rows.

Why these numbers are trustworthy

Every count on this page is generated from the freeze manifest (3228af686d09…), which digest-binds the benchmark revision, every case and fixture, every analyzer's identity, their normalized reports, and one retained raw artifact per result. CI regenerates the numbers from the manifest and fails on any drift; hand-authored prose cannot override a generated count. See reproduction to verify locally, or dive into the full snapshotanalyzers, languages, templates, and per-case evidence.