Skip to content
DataFlowBench

Analyzer profiles — v0.7.1

An exploratory view: one small radar per analyzer, four axes, drawn over the benchmark-controlled core kernel populations. It makes one analyzer’s shape legible at a glance — where it is fast, where it decides, where it is right when it decides, and how much of the benchmark it covers — without creating a composite score.

The correctness axes use v0.7.1’s freshly rerun evidence. The latency axis uses this release’s separately qualified characterization data. The two are shown together for orientation only: they have different denominators and no calculation on this page combines them. In particular, this page is not a ranking and does not claim that one analyzer dominates another.

The four correctness deltas are confined to Java: Bifrost’s anonymous-implementation pair became reached / not-reached from inconclusive, and OpenTaint’s callback-registration and map-iteration positive cells became reached from not-reached.

Bifrost
Four-axis profile for Bifrost: speed 91.5% of the log scale, recall 61.2%, precision 100.0%, kernel coverage 100.0%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
180 ms median 738 timed invocations
recall
61.2% 226 of 369 positive assertions it covers
precision
100.0% 226 of 226 decided positives
coverage
100.0% 13 of 13 kernels, 738 assertions
CodeQL
Four-axis profile for CodeQL: speed 36.0% of the log scale, recall 72.3%, precision 88.2%, kernel coverage 84.6%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
8.30 s median 622 timed invocations
recall
72.3% 225 of 311 positive assertions it covers
precision
88.2% 225 of 255 decided positives
coverage
84.6% 11 of 13 kernels, 622 assertions
Semgrep CE
Four-axis profile for Semgrep CE: speed 63.0% of the log scale, recall 24.8%, precision 77.8%, kernel coverage 84.6%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
1.29 s median 154 timed invocations
recall
24.8% 77 of 311 positive assertions it covers
precision
77.8% 77 of 99 decided positives
coverage
84.6% 11 of 13 kernels, 622 assertions
Joern
Four-axis profile for Joern: speed 40.8% of the log scale, recall 70.9%, precision 83.6%, kernel coverage 46.2%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
5.99 s median 344 timed invocations
recall
70.9% 122 of 172 positive assertions it covers
precision
83.6% 122 of 146 decided positives
coverage
46.2% 6 of 13 kernels, 344 assertions
Infer
Four-axis profile for Infer: speed 64.8% of the log scale, recall 74.1%, precision 98.4%, kernel coverage 23.1%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
1.14 s median 162 timed invocations
recall
74.1% 60 of 81 positive assertions it covers
precision
98.4% 60 of 61 decided positives
coverage
23.1% 3 of 13 kernels, 162 assertions
FlowDroid
Four-axis profile for FlowDroid: speed 69.3% of the log scale, recall 84.5%, precision 84.5%, kernel coverage 15.4%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
836 ms median 116 timed invocations
recall
84.5% 49 of 58 positive assertions it covers
precision
84.5% 49 of 58 decided positives
coverage
15.4% 2 of 13 kernels, 116 assertions
OpenTaint
Four-axis profile for OpenTaint: speed 45.4% of the log scale, recall 93.1%, precision 83.1%, kernel coverage 15.4%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
4.35 s median 116 timed invocations
recall
93.1% 54 of 58 positive assertions it covers
precision
83.1% 54 of 65 decided positives
coverage
15.4% 2 of 13 kernels, 116 assertions
Pysa
Four-axis profile for Pysa: speed 51.1% of the log scale, recall 65.5%, precision 95.0%, kernel coverage 7.7%. The list beneath the shape carries every value and its denominator. The shape is a profile, not a score, and its area means nothing. speed recall precision coverage
speed
2.93 s median 58 timed invocations
recall
65.5% 19 of 29 positive assertions it covers
precision
95.0% 19 of 20 decided positives
coverage
7.7% 1 of 13 kernels, 58 assertions

These are profiles, not scores, and their areas are not comparable. A radar's enclosed area depends on the order the axes happen to be drawn in — swap two of them and the area changes without a single number changing — so nothing on this page computes one, and a shape that looks "bigger" is not a better analyzer. Nor are two shapes comparable as wholes: read one axis at a time, against the denominator printed under it. There is no combined score here or anywhere else on this site.

What each axis is over, since they are four different denominators.

  • speed — the median analyzer-invocation wall-clock over these same kernel populations, log-normalized and inverted so that faster reads as larger. The scale is absolute, not relative to the analyzers present: 100 ms maps to the outer ring and 100.0 s to the centre, the same three decades the latency chart's axis spans. Adding or removing an analyzer cannot move anyone else's mark. It is the only axis whose spacing is not linear, so equal distances along it are equal ratios of time. Every latency caveat on the latency page applies to it unchanged: one machine, one environment stamp, no repeated trials, per-invocation start-up costs inside the number.
  • recall — true positives over every positive-polarity assertion in the kernels the analyzer covers. Non-answers count against it: a case it answered inconclusive or declined as unsupported stays in the denominator, because it is a case the analyzer took on and did not resolve.
  • precision — true positives over decided positives, true plus false. Non-answers are absent from this denominator entirely. That asymmetry with recall is deliberate and it is the thing most easily misread on this page: the very same case can lower an analyzer's recall while being invisible to its precision, so an analyzer that decides little can hold a high precision and a low recall at once.
  • coverage — kernels covered of the snapshot's 13. It is on the picture precisely because the two rates beside it are over each tool's own covered slice: recall and precision from a one-kernel slice and from a thirteen-kernel slice are not the same exam, and this axis is how far apart those exams are. It is the same honesty note the cross-snapshot evolution chart's normalized view carries, and it binds harder here, because there the coverage was an annotation and here it is an axis of the same shape.

One population, so that the four numbers describe one exam. Every axis is computed over the benchmark-controlled core kernel populations — the same no-pooling filter the landing page's cards read — and not over the whole freeze. The latency figure here is therefore narrower than the whole-corpus median on the latency page, which also includes the modeling and tool-native runs; where the two differ, they differ because they are over different populations and both say which.

Latency and correctness are still never pooled. Placing them on one picture is not combining them: no arithmetic here crosses an axis boundary, no correctness outcome in this freeze was derived from or tie-broken by a timing value, and "correct but slow" and "fast but wrong" are as independently visible on these shapes as they are in the tables. If a reader wants a single number that ranks these analyzers, this page does not have one, and that is the point of it.

Where these numbers come from

Section titled “Where these numbers come from”

The correctness axes are derived from the same frozen results/results.json that the correctness pages read, through the benchmark-controlled core population filter. The latency axis is derived from the v0.7.1 timing evidence under the preregistered docs/latency-tier.md contract. No correctness outcome is derived from, conditioned on, or tie-broken by a timing value.