Skip to content
DataFlowBench

Analyzer profiles — v0.6.0

An exploratory view: one small radar per analyzer, four axes, drawn over the benchmark-controlled core kernel populations. It exists to make one analyzer’s shape legible at a glance — where it is fast, where it decides, where it is right when it decides, and how much of the benchmark it covers at all — and for nothing else.

It is not a ranking and it does not produce one. There is no composite score on this page, no weighted blend of the axes, and no arithmetic anywhere that crosses from one axis to another. The radars are deliberately not overlaid: eight polygons on one chart would invite the whole-shape comparison these figures cannot support, so each analyzer gets its own.

Read the axis notes under the grid before reading the shapes. The four axes have four different denominators, one of them is on a logarithmic scale, and two of them treat a non-answer in opposite ways — a case that lowers an analyzer’s recall can be entirely absent from its precision. Every value is also printed as a number with its denominator beneath its own radar, so no reading has to come from the shape.

Bifrost
Four-axis profile for Bifrost: speed 96.9% of the log scale, recall 59.3%, precision 98.2%, kernel coverage 100.0%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
124 ms median 738 timed invocations
recall
59.3% 219 of 369 positive assertions it covers
precision
98.2% 219 of 223 decided positives
coverage
100.0% 13 of 13 kernels, 738 assertions
CodeQL
Four-axis profile for CodeQL: speed 33.2% of the log scale, recall 72.3%, precision 88.2%, kernel coverage 84.6%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
10.1 s median 622 timed invocations
recall
72.3% 225 of 311 positive assertions it covers
precision
88.2% 225 of 255 decided positives
coverage
84.6% 11 of 13 kernels, 622 assertions
Semgrep CE
Four-axis profile for Semgrep CE: speed 64.1% of the log scale, recall 24.8%, precision 77.8%, kernel coverage 84.6%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
1.20 s median 154 timed invocations
recall
24.8% 77 of 311 positive assertions it covers
precision
77.8% 77 of 99 decided positives
coverage
84.6% 11 of 13 kernels, 622 assertions
Joern
Four-axis profile for Joern: speed 35.3% of the log scale, recall 70.9%, precision 83.6%, kernel coverage 46.2%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
8.75 s median 344 timed invocations
recall
70.9% 122 of 172 positive assertions it covers
precision
83.6% 122 of 146 decided positives
coverage
46.2% 6 of 13 kernels, 344 assertions
Infer
Four-axis profile for Infer: speed 55.2% of the log scale, recall 74.1%, precision 98.4%, kernel coverage 23.1%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
2.21 s median 162 timed invocations
recall
74.1% 60 of 81 positive assertions it covers
precision
98.4% 60 of 61 decided positives
coverage
23.1% 3 of 13 kernels, 162 assertions
FlowDroid
Four-axis profile for FlowDroid: speed 59.0% of the log scale, recall 84.5%, precision 84.5%, kernel coverage 15.4%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
1.70 s median 116 timed invocations
recall
84.5% 49 of 58 positive assertions it covers
precision
84.5% 49 of 58 decided positives
coverage
15.4% 2 of 13 kernels, 116 assertions
OpenTaint
Four-axis profile for OpenTaint: speed 44.6% of the log scale, recall 89.7%, precision 82.5%, kernel coverage 15.4%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
4.60 s median 116 timed invocations
recall
89.7% 52 of 58 positive assertions it covers
precision
82.5% 52 of 63 decided positives
coverage
15.4% 2 of 13 kernels, 116 assertions
Pysa
Four-axis profile for Pysa: speed 46.3% of the log scale, recall 65.5%, precision 95.0%, kernel coverage 7.7%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
4.08 s median 58 timed invocations
recall
65.5% 19 of 29 positive assertions it covers
precision
95.0% 19 of 20 decided positives
coverage
7.7% 1 of 13 kernels, 58 assertions

These are profiles, not scores, and their areas are not comparable. A radar's enclosed area depends on the order the axes happen to be drawn in — swap two of them and the area changes without a single number changing — so nothing on this page computes one, and a shape that looks "bigger" is not a better analyzer. Nor are two shapes comparable as wholes: read one axis at a time, against the denominator printed under it. There is no combined score here or anywhere else on this site.

What each axis is over, since they are four different denominators.

  • speed — the median whole-invocation wall-clock over these same kernel populations, log-normalized and inverted so that faster reads as larger. The scale is absolute, not relative to the analyzers present: 100 ms maps to the outer ring and 100.0 s to the centre, the same three decades the latency chart's axis spans. Adding or removing an analyzer cannot move anyone else's mark. It is the only axis whose spacing is not linear, so equal distances along it are equal ratios of time. Every latency caveat on the latency page applies to it unchanged: one machine, one environment stamp, no repeated trials, per-invocation start-up costs inside the number.
  • recall — true positives over every positive-polarity assertion in the kernels the analyzer covers. Non-answers count against it: a case it answered inconclusive or declined as unsupported stays in the denominator, because it is a case the analyzer took on and did not resolve.
  • precision — true positives over decided positives, true plus false. Non-answers are absent from this denominator entirely. That asymmetry with recall is deliberate and it is the thing most easily misread on this page: the very same case can lower an analyzer's recall while being invisible to its precision, so an analyzer that decides little can hold a high precision and a low recall at once.
  • coverage — kernels covered of the snapshot's 13. It is on the picture precisely because the two rates beside it are over each tool's own covered slice: recall and precision from a one-kernel slice and from a thirteen-kernel slice are not the same exam, and this axis is how far apart those exams are. It is the same honesty note the cross-snapshot evolution chart's normalized view carries, and it binds harder here, because there the coverage was an annotation and here it is an axis of the same shape.

One population, so that the four numbers describe one exam. Every axis is computed over the benchmark-controlled core kernel populations — the same no-pooling filter the landing page's cards read — and not over the whole freeze. The latency figure here is therefore narrower than the whole-corpus median on the latency page, which also includes the modeling and tool-native runs; where the two differ, they differ because they are over different populations and both say which.

Latency and correctness are still never pooled. Placing them on one picture is not combining them: no arithmetic here crosses an axis boundary, no correctness outcome in this freeze was derived from or tie-broken by a timing value, and "correct but slow" and "fast but wrong" are as independently visible on these shapes as they are in the tables. If a reader wants a single number that ranks these analyzers, this page does not have one, and that is the point of it.

Where these numbers come from

Section titled “Where these numbers come from”

The correctness axes are derived from the same frozen results/results.json the correctness pages read, through the same benchmark-controlled core population filter, and the latency axis from the timing sidecars the freeze binds, under the contract preregistered in docs/latency-tier.md. Placing latency on the same picture as correctness is not pooling them: the contract forbids a combined score, an efficiency-adjusted rate, and a leaderboard over a blend, and this page has none of those. No correctness outcome in this freeze was derived from, conditioned on, or tie-broken by a timing value.