Skip to content
DataFlowBench

Latency — v0.6.0

The first publication of DataFlowBench’s latency-characterization tier. It stands beside the correctness scorecards, on its own page, and never as a column inside one.

The contract governing every number here — docs/latency-tier.md — was written and merged before any timestamp was captured, and it is explicit about why. DataFlowBench is published by the vendor of one of the engines measured, and a latency page assembled after the numbers were known, with its phase boundaries and its aggregation chosen afterwards, would deserve exactly the skepticism it would get. So the decomposition rule, the exclusion list, the per-adapter granularity, and the aggregation were all fixed while every number was still unmeasured. The four analyzers this release adds took their granularity rows by Amendment A12, dated and landed before this page published any of their numbers, rather than being backfilled into the original table.

The page also carries a second, separately labelled figure: the warm marginal cost of one more case in a tool process that has already started, added by Amendment A15 and likewise preregistered before the first warm number was measured. It exists because the cold rows below span three runtimes, and read across runtimes they overstate the steady-state gap — a difference the vendor publishing this benchmark happens to win. The correction is a direct measurement, not an estimate subtracted from anything: no cold number on this page changes, and the warm figure is never substituted for one.

What these numbers are, and what they are not

This is a descriptive characterization of per-case analyzer wall-clock, published under the contract preregistered in docs/latency-tier.md, which was merged before a single timestamp was captured. It is not a score, not a ranking, and not a benchmark result in the sense the rest of this site uses that phrase.

  • Latency is never pooled with correctness. There is no combined number anywhere on this site, no efficiency-adjusted rate, and no leaderboard that blends the two. "Correct but slow" and "fast but wrong" stay independently visible. No correctness outcome in this freeze was derived from, conditioned on, or tie-broken by a timing value.
  • A latency number is a property of a run, not of a case. No case carries a timing assertion, threshold, or budget.
  • Characterization, not micro-benchmark. There are no repeated trials, no warm-up iterations, and no steady-state harness. The case population is the sample and its spread is the statistics. The distribution columns below are that spread — not error bars on a measurement that was repeated, because it was not.
  • These are the conditions the numbers were produced under: a single developer machine, under light concurrent load, running the benchmark's standing sequential-run discipline — one analyzer at a time, never two at once. That discipline is what makes the numbers usable at all; the light concurrent load is what stops them being a controlled measurement. Read them as characterization of what this benchmark actually costs to run, at the order-of-magnitude and shape level, and not as a precise figure for any engine.

Environment stamp

Recorded once per run beside the tool identity the run witnessed, and shown here verbatim. Latency numbers are comparable within one environment and are not comparable across machines. Every published run to date executes on one maintainer machine, and that caveat travels with every number on this page.

Hardware model OS OS release Logical CPUs CPU architecture Timed runs stamped
Mac16,1 macos 25.6.0 10 aarch64 66

One row: every timed invocation in this freeze ran in the same stamped environment, so no number on this page is a cross-environment comparison. Were there two rows, this page would say so in the table rather than averaging across them. The count is of run directories that contain at least one timed invocation — a run that invoked no analyzer at all still stamps its environment and witnesses its tool identity, but contributes no latency number and is not counted here.

The granularity is unequal, and that is the point

A phase is timed if and only if the adapter already invokes it as a separate subprocess. The benchmark never instruments analyzer internals, never patches an upstream tool, and never infers a boundary a subprocess boundary does not expose. The adapters do not expose the same boundaries, so the rows below are not equally decomposed — and the two ways of papering over that would both misdescribe the evidence: summing a decomposed adapter's phases and presenting every row as one number of the same kind, or guessing phase splits for the adapters that expose only one.

So the table states each adapter's declared granularity as data. Phases compare within an adapter; whole invocations compare across adapters. A CodeQL database-create number and a Joern total are not the same kind of quantity, and nothing on this page sets them side by side as though they were.

Adapter Version Subprocesses per case Phases timed Timed invocations
Bifrost 0.10.7 1 total 883
Semgrep CE 1.175.0 1 total 196
FlowDroid 2.15.1 1 total 116
Infer v1.3.0 2 capture analyze 162
Pysa 0.10.0 1 total 58
OpenTaint analyzer/2026.08.27.17eb0fe 1 total 116
Joern 4.0.614 1 total 392
CodeQL 2.26.4 2 database-create database-analyze 734

2 of 8 adapters expose more than one subprocess and are decomposed here (Infer and CodeQL); the other 6 expose a single invocation and take one honestly labelled number, with interpreter or JVM start-up inside it. That is a fact about invocation shapes, not about engine architecture: an adapter with one number is not an adapter that does one thing.

The phase names above are the ones the retained evidence carries, so any name here can be found in the artifact it came from. For CodeQL those are database-create and database-analyze, after the subcommands that produce them, where the preregistration's table spells the same two boundaries extract and analyze. Two spellings, one pair of subprocess boundaries: nothing moved and nothing is attributed differently. The contract is immutable, so the difference is recorded in its Amendment A12 rather than edited out of its table.

Whole-invocation wall-clock, per adapter

The only cross-adapter reading this tier supports: each adapter's whole-invocation wall-clock, over every case in this freeze it actually invoked. Per-case timing at this granularity includes per-invocation fixed costs — JVM start-up, extractor initialization, interpreter start — that a long-lived deployment of the same engine would amortize. This tier does not correct for that; it characterizes what the benchmark actually runs.

Fastest to slowest, and the same ranking per kernel

Ordered by median, on a logarithmic axis, with the spread drawn beside every median rather than left to the table. The toggle scopes the same ranking to one language kernel at a time: a whole-corpus median mixes the languages an adapter runs on, and holding the language fixed is the difference between "this engine is slow" and "this front end is expensive". Only the analyzers that actually invoked on a kernel appear in its view. The chart draws the median, the quartiles and the p10–p90 spread; the disclosure beneath it opens the same distributions as a table, with the minima and maxima the whiskers deliberately leave out.

Whole-invocation wall-clock per analyzer, over every timed invocation in the freeze One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of the two adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 123 ms, IQR 116 ms to 133 ms, p10–p90 111 ms to 149 ms, over 883 timed whole invocations Bifrost 883 timed 123 ms Semgrep CE — median 1.20 s, IQR 1.11 s to 1.59 s, p10–p90 1.08 s to 2.47 s, over 196 timed whole invocations Semgrep CE 196 timed 1.20 s FlowDroid — median 1.70 s, IQR 1.55 s to 1.96 s, p10–p90 1.48 s to 2.08 s, over 116 timed whole invocations FlowDroid 116 timed 1.70 s Infer — median 2.21 s, IQR 554 ms to 6.24 s, p10–p90 504 ms to 6.74 s, over 162 timed whole invocations Infer 162 timed 2.21 s Infer phase capture — median 1.78 s, IQR 160 ms to 5.87 s, p10–p90 151 ms to 6.37 s. Comparable only against this adapter's own numbers. capture 1.78 s Infer phase analyze — median 373 ms, IQR 343 ms to 440 ms, p10–p90 324 ms to 500 ms. Comparable only against this adapter's own numbers. analyze 373 ms Pysa — median 4.08 s, IQR 3.88 s to 4.21 s, p10–p90 3.77 s to 4.58 s, over 58 timed whole invocations Pysa 58 timed 4.08 s OpenTaint — median 4.60 s, IQR 4.39 s to 4.98 s, p10–p90 4.22 s to 9.51 s, over 116 timed whole invocations OpenTaint 116 timed 4.60 s Joern — median 8.70 s, IQR 7.18 s to 10.1 s, p10–p90 5.17 s to 14.6 s, over 392 timed whole invocations Joern 392 timed 8.70 s CodeQL — median 10.8 s, IQR 7.22 s to 16.7 s, p10–p90 5.30 s to 39.4 s, over 734 timed whole invocations CodeQL 734 timed 10.8 s CodeQL phase database-create — median 4.51 s, IQR 2.18 s to 10.7 s, p10–p90 1.42 s to 20.4 s. Comparable only against this adapter's own numbers. database-create 4.51 s CodeQL phase database-analyze — median 5.58 s, IQR 4.64 s to 8.29 s, p10–p90 3.45 s to 15.4 s. Comparable only against this adapter's own numbers. database-analyze 5.58 s

Every timed analyzer invocation the freeze binds — 2657 of them, across every score tier and both model profiles. This is the widest denominator on the site and the only one here that is not a single population: an adapter's median mixes the languages it runs on, whose fixtures differ in size and whose front ends differ in cost. The per-kernel views hold the language fixed.

Whole-invocation wall-clock per analyzer on the c kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of the two adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 131 ms, IQR 124 ms to 141 ms, p10–p90 120 ms to 155 ms, over 48 timed whole invocations Bifrost 48 of 48 timed 131 ms Infer — median 540 ms, IQR 494 ms to 612 ms, p10–p90 484 ms to 755 ms, over 48 timed whole invocations Infer 48 of 48 timed 540 ms Infer phase capture — median 159 ms, IQR 150 ms to 183 ms, p10–p90 144 ms to 210 ms. Comparable only against this adapter's own numbers. capture 159 ms Infer phase analyze — median 372 ms, IQR 344 ms to 435 ms, p10–p90 331 ms to 527 ms. Comparable only against this adapter's own numbers. analyze 372 ms Semgrep CE — median 1.28 s, IQR 1.21 s to 1.33 s, p10–p90 1.17 s to 1.37 s, over 14 timed whole invocations Semgrep CE 14 of 48 timed 1.28 s CodeQL — median 6.28 s, IQR 5.41 s to 7.11 s, p10–p90 5.31 s to 8.21 s, over 48 timed whole invocations CodeQL 48 of 48 timed 6.28 s CodeQL phase database-create — median 1.56 s, IQR 1.39 s to 1.79 s, p10–p90 1.36 s to 2.20 s. Comparable only against this adapter's own numbers. database-create 1.56 s CodeQL phase database-analyze — median 4.72 s, IQR 4.01 s to 5.37 s, p10–p90 3.91 s to 6.24 s. Comparable only against this adapter's own numbers. database-analyze 4.72 s

The c kernel's 48 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 48, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Whole-invocation wall-clock per analyzer on the cpp kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of the two adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 130 ms, IQR 127 ms to 136 ms, p10–p90 120 ms to 144 ms, over 56 timed whole invocations Bifrost 56 of 56 timed 130 ms Infer — median 728 ms, IQR 550 ms to 2.99 s, p10–p90 518 ms to 3.33 s, over 56 timed whole invocations Infer 56 of 56 timed 728 ms Infer phase capture — median 234 ms, IQR 161 ms to 2.55 s, p10–p90 153 ms to 2.91 s. Comparable only against this adapter's own numbers. capture 234 ms Infer phase analyze — median 413 ms, IQR 370 ms to 466 ms, p10–p90 358 ms to 518 ms. Comparable only against this adapter's own numbers. analyze 413 ms Semgrep CE — median 1.12 s, IQR 1.10 s to 1.13 s, p10–p90 1.08 s to 1.16 s, over 14 timed whole invocations Semgrep CE 14 of 56 timed 1.12 s CodeQL — median 7.28 s, IQR 6.19 s to 19.1 s, p10–p90 5.40 s to 21.3 s, over 56 timed whole invocations CodeQL 56 of 56 timed 7.28 s CodeQL phase database-create — median 1.89 s, IQR 1.54 s to 10.9 s, p10–p90 1.41 s to 12.6 s. Comparable only against this adapter's own numbers. database-create 1.89 s CodeQL phase database-analyze — median 5.40 s, IQR 4.62 s to 8.44 s, p10–p90 3.99 s to 8.80 s. Comparable only against this adapter's own numbers. database-analyze 5.40 s

The cpp kernel's 56 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 56, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Whole-invocation wall-clock per analyzer on the csharp kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of the two adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 141 ms, IQR 130 ms to 155 ms, p10–p90 122 ms to 166 ms, over 58 timed whole invocations Bifrost 58 of 58 timed 141 ms CodeQL — median 33.1 s, IQR 24.1 s to 39.7 s, p10–p90 20.7 s to 41.3 s, over 58 timed whole invocations CodeQL 58 of 58 timed 33.1 s CodeQL phase database-create — median 20.1 s, IQR 13.6 s to 25.7 s, p10–p90 11.5 s to 26.7 s. Comparable only against this adapter's own numbers. database-create 20.1 s CodeQL phase database-analyze — median 13.0 s, IQR 9.42 s to 14.2 s, p10–p90 9.12 s to 14.6 s. Comparable only against this adapter's own numbers. database-analyze 13.0 s

The csharp kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 2 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Whole-invocation wall-clock per analyzer on the go kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of the two adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 123 ms, IQR 116 ms to 128 ms, p10–p90 115 ms to 140 ms, over 58 timed whole invocations Bifrost 58 of 58 timed 123 ms Semgrep CE — median 1.11 s, IQR 1.09 s to 1.15 s, p10–p90 1.08 s to 1.17 s, over 14 timed whole invocations Semgrep CE 14 of 58 timed 1.11 s CodeQL — median 9.43 s, IQR 9.24 s to 9.81 s, p10–p90 9.12 s to 10.9 s, over 58 timed whole invocations CodeQL 58 of 58 timed 9.43 s CodeQL phase database-create — median 4.40 s, IQR 4.30 s to 4.52 s, p10–p90 4.26 s to 5.67 s. Comparable only against this adapter's own numbers. database-create 4.40 s CodeQL phase database-analyze — median 5.05 s, IQR 4.90 s to 5.14 s, p10–p90 4.75 s to 5.41 s. Comparable only against this adapter's own numbers. database-analyze 5.05 s

The go kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 3 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Whole-invocation wall-clock per analyzer on the java kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of the two adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 120 ms, IQR 114 ms to 125 ms, p10–p90 111 ms to 130 ms, over 58 timed whole invocations Bifrost 58 of 58 timed 120 ms Semgrep CE — median 1.16 s, IQR 1.13 s to 1.22 s, p10–p90 1.11 s to 1.28 s, over 14 timed whole invocations Semgrep CE 14 of 58 timed 1.16 s FlowDroid — median 1.55 s, IQR 1.49 s to 1.61 s, p10–p90 1.43 s to 1.67 s, over 58 timed whole invocations FlowDroid 58 of 58 timed 1.55 s OpenTaint — median 4.70 s, IQR 4.55 s to 9.25 s, p10–p90 4.41 s to 9.96 s, over 58 timed whole invocations OpenTaint 58 of 58 timed 4.70 s Infer — median 6.47 s, IQR 6.15 s to 6.77 s, p10–p90 6.03 s to 7.20 s, over 58 timed whole invocations Infer 58 of 58 timed 6.47 s Infer phase capture — median 6.09 s, IQR 5.81 s to 6.39 s, p10–p90 5.65 s to 6.85 s. Comparable only against this adapter's own numbers. capture 6.09 s Infer phase analyze — median 349 ms, IQR 331 ms to 393 ms, p10–p90 314 ms to 439 ms. Comparable only against this adapter's own numbers. analyze 349 ms CodeQL — median 12.7 s, IQR 12.5 s to 12.9 s, p10–p90 12.3 s to 13.1 s, over 58 timed whole invocations CodeQL 58 of 58 timed 12.7 s CodeQL phase database-create — median 6.70 s, IQR 6.58 s to 6.81 s, p10–p90 6.48 s to 7.10 s. Comparable only against this adapter's own numbers. database-create 6.70 s CodeQL phase database-analyze — median 6.00 s, IQR 5.87 s to 6.11 s, p10–p90 5.80 s to 6.18 s. Comparable only against this adapter's own numbers. database-analyze 6.00 s Joern — median 14.6 s, IQR 14.3 s to 15.3 s, p10–p90 14.1 s to 16.2 s, over 58 timed whole invocations. Warm marginal, measured separately: 1.55 s per case in one process Joern 58 of 58 timed 14.6 s

The java kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 7 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Whole-invocation wall-clock per analyzer on the javascript kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of the two adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 117 ms, IQR 112 ms to 121 ms, p10–p90 108 ms to 127 ms, over 58 timed whole invocations Bifrost 58 of 58 timed 117 ms Semgrep CE — median 1.31 s, IQR 1.24 s to 1.33 s, p10–p90 1.20 s to 1.44 s, over 14 timed whole invocations Semgrep CE 14 of 58 timed 1.31 s Joern — median 9.15 s, IQR 8.86 s to 9.61 s, p10–p90 8.64 s to 10.2 s, over 58 timed whole invocations Joern 58 of 58 timed 9.15 s CodeQL — median 10.7 s, IQR 9.80 s to 13.5 s, p10–p90 9.45 s to 14.0 s, over 58 timed whole invocations CodeQL 58 of 58 timed 10.7 s CodeQL phase database-create — median 4.49 s, IQR 4.15 s to 5.50 s, p10–p90 3.98 s to 5.76 s. Comparable only against this adapter's own numbers. database-create 4.49 s CodeQL phase database-analyze — median 6.12 s, IQR 5.62 s to 8.01 s, p10–p90 5.43 s to 8.21 s. Comparable only against this adapter's own numbers. database-analyze 6.12 s

The javascript kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Whole-invocation wall-clock per analyzer on the kotlin kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of the two adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 122 ms, IQR 119 ms to 128 ms, p10–p90 115 ms to 132 ms, over 58 timed whole invocations Bifrost 58 of 58 timed 122 ms Semgrep CE — median 1.09 s, IQR 1.08 s to 1.58 s, p10–p90 1.05 s to 1.67 s, over 14 timed whole invocations Semgrep CE 14 of 58 timed 1.09 s FlowDroid — median 1.97 s, IQR 1.84 s to 2.05 s, p10–p90 1.76 s to 2.15 s, over 58 timed whole invocations FlowDroid 58 of 58 timed 1.97 s OpenTaint — median 4.46 s, IQR 4.26 s to 4.74 s, p10–p90 4.16 s to 4.97 s, over 58 timed whole invocations OpenTaint 58 of 58 timed 4.46 s CodeQL — median 15.8 s, IQR 15.6 s to 17.0 s, p10–p90 15.5 s to 27.1 s, over 58 timed whole invocations CodeQL 58 of 58 timed 15.8 s CodeQL phase database-create — median 10.9 s, IQR 10.8 s to 11.6 s, p10–p90 10.7 s to 17.7 s. Comparable only against this adapter's own numbers. database-create 10.9 s CodeQL phase database-analyze — median 4.85 s, IQR 4.78 s to 5.24 s, p10–p90 4.73 s to 7.61 s. Comparable only against this adapter's own numbers. database-analyze 4.85 s

The kotlin kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 5 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Whole-invocation wall-clock per analyzer on the php kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of the two adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 122 ms, IQR 116 ms to 129 ms, p10–p90 110 ms to 137 ms, over 58 timed whole invocations Bifrost 58 of 58 timed 122 ms Semgrep CE — median 1.69 s, IQR 1.61 s to 1.82 s, p10–p90 1.56 s to 1.94 s, over 14 timed whole invocations Semgrep CE 14 of 58 timed 1.69 s Joern — median 5.45 s, IQR 5.12 s to 8.39 s, p10–p90 5.05 s to 11.9 s, over 58 timed whole invocations Joern 58 of 58 timed 5.45 s

The php kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 3 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Whole-invocation wall-clock per analyzer on the python kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of the two adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 115 ms, IQR 110 ms to 121 ms, p10–p90 106 ms to 127 ms, over 58 timed whole invocations Bifrost 58 of 58 timed 115 ms Semgrep CE — median 1.10 s, IQR 1.09 s to 1.12 s, p10–p90 1.07 s to 1.13 s, over 14 timed whole invocations Semgrep CE 14 of 58 timed 1.10 s Pysa — median 4.08 s, IQR 3.88 s to 4.21 s, p10–p90 3.77 s to 4.58 s, over 58 timed whole invocations Pysa 58 of 58 timed 4.08 s CodeQL — median 5.38 s, IQR 5.30 s to 5.83 s, p10–p90 5.26 s to 5.90 s, over 58 timed whole invocations CodeQL 58 of 58 timed 5.38 s CodeQL phase database-create — median 2.00 s, IQR 1.97 s to 2.13 s, p10–p90 1.95 s to 2.18 s. Comparable only against this adapter's own numbers. database-create 2.00 s CodeQL phase database-analyze — median 3.41 s, IQR 3.31 s to 3.67 s, p10–p90 3.27 s to 3.73 s. Comparable only against this adapter's own numbers. database-analyze 3.41 s Joern — median 8.38 s, IQR 8.00 s to 8.86 s, p10–p90 7.70 s to 9.29 s, over 58 timed whole invocations Joern 58 of 58 timed 8.38 s

The python kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 5 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Whole-invocation wall-clock per analyzer on the ruby kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of the two adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 122 ms, IQR 117 ms to 125 ms, p10–p90 112 ms to 128 ms, over 58 timed whole invocations Bifrost 58 of 58 timed 122 ms Semgrep CE — median 2.53 s, IQR 2.37 s to 2.73 s, p10–p90 2.29 s to 2.89 s, over 14 timed whole invocations Semgrep CE 14 of 58 timed 2.53 s CodeQL — median 4.70 s, IQR 4.58 s to 4.89 s, p10–p90 4.47 s to 5.13 s, over 58 timed whole invocations CodeQL 58 of 58 timed 4.70 s CodeQL phase database-create — median 1.34 s, IQR 1.29 s to 1.40 s, p10–p90 1.27 s to 1.43 s. Comparable only against this adapter's own numbers. database-create 1.34 s CodeQL phase database-analyze — median 3.33 s, IQR 3.25 s to 3.49 s, p10–p90 3.19 s to 3.68 s. Comparable only against this adapter's own numbers. database-analyze 3.33 s Joern — median 8.63 s, IQR 7.35 s to 11.0 s, p10–p90 7.13 s to 13.7 s, over 58 timed whole invocations Joern 58 of 58 timed 8.63 s

The ruby kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Whole-invocation wall-clock per analyzer on the rust kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of the two adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 122 ms, IQR 118 ms to 129 ms, p10–p90 113 ms to 133 ms, over 54 timed whole invocations Bifrost 54 of 54 timed 122 ms Semgrep CE — median 1.35 s, IQR 1.22 s to 1.52 s, p10–p90 1.17 s to 1.82 s, over 14 timed whole invocations Semgrep CE 14 of 54 timed 1.35 s Joern — median 7.41 s, IQR 7.05 s to 8.31 s, p10–p90 6.82 s to 9.18 s, over 54 timed whole invocations Joern 54 of 54 timed 7.41 s CodeQL — median 47.0 s, IQR 45.4 s to 50.1 s, p10–p90 44.8 s to 61.5 s, over 54 timed whole invocations CodeQL 54 of 54 timed 47.0 s CodeQL phase database-create — median 21.0 s, IQR 20.3 s to 21.8 s, p10–p90 20.1 s to 27.7 s. Comparable only against this adapter's own numbers. database-create 21.0 s CodeQL phase database-analyze — median 25.6 s, IQR 24.9 s to 27.3 s, p10–p90 24.6 s to 31.0 s. Comparable only against this adapter's own numbers. database-analyze 25.6 s

The rust kernel's 54 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 54, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Whole-invocation wall-clock per analyzer on the scala kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of the two adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 133 ms, IQR 125 ms to 147 ms, p10–p90 121 ms to 164 ms, over 58 timed whole invocations Bifrost 58 of 58 timed 133 ms

The scala kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 1 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Whole-invocation wall-clock per analyzer on the typescript kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of the two adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 144 ms, IQR 128 ms to 163 ms, p10–p90 119 ms to 203 ms, over 58 timed whole invocations Bifrost 58 of 58 timed 144 ms Semgrep CE — median 1.11 s, IQR 1.09 s to 1.18 s, p10–p90 1.07 s to 1.20 s, over 14 timed whole invocations Semgrep CE 14 of 58 timed 1.11 s CodeQL — median 8.92 s, IQR 8.09 s to 9.80 s, p10–p90 8.01 s to 10.2 s, over 58 timed whole invocations CodeQL 58 of 58 timed 8.92 s CodeQL phase database-create — median 3.73 s, IQR 3.53 s to 4.21 s, p10–p90 3.43 s to 4.36 s. Comparable only against this adapter's own numbers. database-create 3.73 s CodeQL phase database-analyze — median 5.04 s, IQR 4.55 s to 5.57 s, p10–p90 4.50 s to 5.80 s. Comparable only against this adapter's own numbers. database-analyze 5.04 s

The typescript kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 3 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

  • median, printed beside every row
  • interquartile range (Q1–Q3)
  • p10–p90; the minimum and maximum are in the tables, not the whiskers
  • indented rows: an adapter's own declared phases
  • warm marginal per case, where the released CLI let it be measured

Every bar above is cold per-invocation wall-clock, and the warm marginal is a different quantity measured separately. Cold is what this benchmark actually runs — one process per case, start-up inside the number, because start-up is not observable from inside a single invocation. Read across runtimes, though, those bars overstate the steady-state gap: a JVM engine's row carries a JVM start a long-lived deployment pays once. So the other quantity is measured directly rather than estimated and subtracted — k cases through one tool process, for increasing k, reporting the slope of batch wall-clock against k. The caret marks it on the rows that have one. It is never subtracted from a median, never substituted for one, and never used to order the rows. Only adapters whose released CLI exposes a real multi-case batch have a figure at all; the rest are not observable with the released CLI, and the warm-marginal section records every verdict, measured and declined, with the evidence behind it.

The axis is logarithmic. Each labelled tick is three times the one before it, so equal distances are equal ratios, not equal durations — the gap from 100 ms to 300 ms is drawn the same width as the gap from 10 s to 30 s. That is the only way this freeze's medians fit in one picture: they span 123 ms to 10.8 s, and on a linear axis every analyzer except the slowest would be a sliver against the origin. Because a log axis is easy to misread, every median is also printed as a number at the right of its own row.

The indented rows are phases, and they are not comparable across adapters. Only the 2 adapters whose preregistered row declares more than one subprocess have them. A phase mark sits on the same axis as the totals because it is the same kind of measurement — wall-clock of a subprocess — but reading one adapter's phase against another adapter's total is precisely the comparison the granularity rule forbids. Read a phase against the adapter it is indented under, and nothing else.

Ordering is not scoring, and this is never pooled with correctness. Rows are sorted by median because an unsorted ranking is unreadable, not because latency is a result. No correctness figure appears in this chart and no number here is blended with one: there is no combined score anywhere on this site, and a fast analyzer that answers wrongly is neither rewarded nor penalised by anything drawn above. Cases an analyzer declined before invocation are absent, not entered as zero — entering them as zero would make the analyzers that decline the most look the fastest, which is exactly backwards.

Show the data table — whole-invocation wall-clock per adapter, with minima and maxima
Adapter Timed invocations Median Q1 Q3 Min Max
Bifrost 883 123 ms 116 ms 133 ms 103 ms 393 ms
Semgrep CE 196 1.20 s 1.11 s 1.59 s 1.03 s 4.26 s
FlowDroid 116 1.70 s 1.55 s 1.96 s 1.35 s 3.07 s
Infer 162 2.21 s 554 ms 6.24 s 468 ms 8.25 s
Pysa 58 4.08 s 3.88 s 4.21 s 3.69 s 5.95 s
OpenTaint 116 4.60 s 4.39 s 4.98 s 3.99 s 10.7 s
Joern 392 8.70 s 7.18 s 10.1 s 4.48 s 127.9 s
CodeQL 734 10.8 s 7.22 s 16.7 s 4.39 s 161.0 s

No mean is headlined anywhere on this page. The distributions are skewed by per-invocation fixed costs and by a small number of very long tails — the slowest single invocation in this freeze is 161.0 s, against a page-wide fastest median of 123 ms — and a mean over that shape would describe neither the typical case nor the tail. The adapters span roughly 88× between the fastest and slowest medians, which is the order-of-magnitude claim this tier is entitled to make, and the strongest one it makes.

Phase decomposition, within each adapter that has one

Only the 2 adapters whose declared row has phases appear here, and each is read down its own column. These numbers exist to decompose one adapter's own cost. Racing one adapter's phase against another's total is exactly the reading the granularity rule forbids. Each adapter's numbers open from its own disclosure, one per adapter, which is also the boundary the comparison rule draws.

Infer v1.3.0

Show the data table — Infer's 2 phases against its whole invocation
Phase Median Q1 Q3 Min Max Share of the median invocation
capture 1.78 s 160 ms 5.87 s 137 ms 7.88 s 80%
analyze 373 ms 343 ms 440 ms 297 ms 1.06 s 17%
whole invocation 2.21 s 554 ms 6.24 s 468 ms 8.25 s —

CodeQL 2.26.4

Show the data table — CodeQL's 2 phases against its whole invocation
Phase Median Q1 Q3 Min Max Share of the median invocation
database-create 4.51 s 2.18 s 10.7 s 1.26 s 155.9 s 42%
database-analyze 5.58 s 4.64 s 8.29 s 3.10 s 43.5 s 52%
whole invocation 10.8 s 7.22 s 16.7 s 4.39 s 161.0 s —

The share column is each phase's median against the whole invocation's median, and the shares need not sum to 100%: a median is not additive, and the phase whose median is largest is not necessarily the phase that dominated any particular case. The whole-invocation row is the distribution of per-case sums, computed per case and then summarized — not the sum of the phase medians.

Warm marginal cost, measured separately

Everything above is cold per-invocation wall-clock, and it stays the headline of this tier. It is also what this benchmark actually runs: one analyzer process per case, start-up inside the number, because start-up is not observable from inside a single invocation. Nothing in this section adjusts, corrects, or replaces one number above it.

But the ranked chart sets a native binary, a Python CLI, and five JVM or JVM-fronted engines on one axis, and read across runtimes those bars overstate the steady-state gap: a JVM row carries a JVM start that a long-lived deployment of the same engine pays once. The honest response is not to estimate that start-up and subtract it — that would be exactly the after-the-fact adjustment this tier's contract refuses. It is to measure the other quantity directly, which is what this section reports, under Amendment A15, preregistered before the first warm number existed.

The method, in one sentence: run k cases through one tool process for increasing k, and report the slope of batch wall-clock against k — the cost of one more case in a process that has already paid its start-up. A slope, never an average: an average per case at k still carries a 1/k share of the fixed cost, which is the very quantity the figure exists to remove. The batch reuses the cold runner's own case selection, endpoint resolution, workspace materialization, and query logic, so what is timed is the same work; only the number of cases sharing one process differs.

Adapter Kernel Cold median, same cases Warm marginal (least squares) Warm marginal (endpoint) Batches Fitted fixed cost
Joern java 15.6 s 1.55 s 1.54 s k = 1, 2, 4, 8, 16 4.72 s

The cold column is the median over exactly the cases the warm batch analyzed, not over the whole kernel: two figures compared across different populations would mix a population effect into a start-up effect. The two columns are never subtracted. The "fitted fixed cost" is the same fit's intercept, published as the descriptive estimate it is — it is not measured, it is not a decomposition of any cold number, and nothing on this site subtracts it from one.

What makes these figures checkable. Each batch retains the per-case evidence it produced, beside the series above. Every document the largest Joern batch produced matches that case's retained cold evidence field for field — same analyzed state, same method count, same source and sink node counts, same flow count — differing only in the scratch path each run recorded for its own workspace. A batch that had quietly analyzed less would be visible there rather than merely fast. Publication is also gated on a stability check: every measurement is run twice, back to back, and a figure whose two runs do not agree closely is withheld rather than published. Where the check passes, the second run is the one retained — not an average, not the better of the two, so publishing it is not a choice, and no repeated trial becomes a statistic. The check has already cost this page one figure: Semgrep's batch was implemented and measured, its two runs disagreed by roughly a factor of two, and the table below records that as a decline with both runs retained as its evidence.

Show the batch series — Joern on the java kernel
Cases in the batch (k) Wall-clock of the one process Average per case (not the reported figure)
1 5.90 s 5.90 s
2 7.56 s 3.78 s
4 10.9 s 2.72 s
8 18.3 s 2.29 s
16 29.0 s 1.81 s

The third column is shown precisely so it can be discarded: it falls with k because it is still paying a shrinking share of the fixed cost, and it would keep falling with a larger k without ever being the marginal cost. The reported figure is the slope of the second column against the first. Measured on Mac16,1, macos 25.6.0, 10 logical CPUs — the same stamp rule as every other number here, and not comparable across machines.

Which adapters this could be measured on, and why not the rest

Each verdict below was reached by interrogating the pinned distribution — its help output, and where that was ambiguous its own bytecode — never from a README or an assumption about the runtime. No adapter is patched, forked, or invoked outside its released interface to make a batch possible. An adapter that ships no way to analyze several cases in one process has no warm figure here, and none is estimated for it from its runtime or from another adapter's slope.

Adapter Observable with the released CLI? Evidence
Joern Yes — measured A released multi-case batch that does the same per-case work, timed above.
Semgrep CE Yes in the CLI — not measured here The batch exists and was measured: one `semgrep scan` accepts many target paths. It also accepts one `--config`, so a batch is the same work as its k cold runs only when all k cases resolve to identical rule text — which caps k at 12 here, and every Semgrep kernel in this benchmark invokes exactly 14 cases (the rest are declared-capability `unsupported`, decided before invocation), so no other language raises the ceiling. At that k the whole batch runs two to three seconds and the slope is small against the machine's own noise: the same measurement run twice back to back produced slopes differing by roughly a factor of two, with one series not even monotone in k. Both runs are retained as `reports/raw/warm-latency/semgrep-java-stability-probe/`. The figure is withheld rather than published to two significant figures it does not have.
FlowDroid Yes in the CLI — not measured here The released CLI does have a batch: `-a/--apkfile` accepts a directory, and the shipped `soot-infoflow-cmd` main class lists its APKs, builds the taint wrapper once outside the loop, and iterates them in one JVM — its own help documents `-si/--skipapkfile` as "APK file to skip when processing a directory of input files", and it refuses a non-directory output with "The output file must be a directory when analyzing multiple APKs". It is not measured here because one invocation carries one `-s` sources-and-sinks definition, so a k-APK batch runs a union of k per-case endpoint configurations rather than each case's own. Whether that changes any case's result is an empirical question that has to be answered across the whole population before a marginal derived from it may be published. Named follow-up work, not a decline.
OpenTaint Not observable with the released CLI The pinned analyzer takes one `--project` and one `--output-dir` and exits after that project. A `project.yaml` may list several `javaProjects`, but analyzing their union is one whole-program analysis over a merged call graph and a merged entry-point set — different work, not k independent case analyses — and `--semgrep-rule-set` is one rule set for the whole invocation while the benchmark resolves a rule per case. No released mode processes separate case projects in one process.
Pysa Not observable with the released CLI `pyre analyze` is one-shot. The client does expose daemon commands (`start`, `incremental`, `query`), but they serve the type checker, not the taint analysis, and `analyze` never attaches to a running server. `--source-directory` is repeatable but merges directories into one project — again one whole-program analysis rather than k — and each case carries its own `.pyre_configuration`, `pyrefly.toml`, and resolved models. A daemon-shaped measurement would be stateful and not reproducibly preregisterable against this pin, so it is declined rather than attempted.
CodeQL Not observable with the released CLI `codeql database analyze` takes exactly one mandatory database, and `database create` produces exactly one database per invocation. Neither subcommand has a multi-database form in the pinned CLI.
Infer Not observable with the released CLI `--results-dir` names one capture database for one project and the analyzer exits after it. Worth stating explicitly: Infer's analyzer is a native binary, and the JVM cost inside its Java row is the traced `javac` in `capture` — per-project compilation work, not process start-up a batch could amortize.
Bifrost Not observable with the released CLI The policy CLI takes one `--root` per invocation; the repeatable `--workspace NAME=PATH` is documented as requiring `--mcp` and does not reach the policy path. What can be said without measuring anything is a bound: Bifrost's cold median already includes its own process start, so its warm marginal lies between zero and that cold number. Warm figures can therefore only move the other rows down toward Bifrost's, never Bifrost's row down further — the asymmetry this amendment corrects is one the publishing vendor's engine loses by.

The decline that matters most is Bifrost's, and it is stated in the direction that costs this benchmark's publisher something. Bifrost has no warm figure here, and its cold median already includes its own process start — so whatever its warm marginal is, it lies between zero and a number already published. Adding warm figures can therefore only move the other rows down toward Bifrost's, never Bifrost's row down further. The conflation this section corrects is one the vendor's own engine benefits from, which is the reason to correct it in public rather than leave the cold chart to be read as a steady-state ranking.

These artifacts are auxiliary evidence, outside the freeze. They are retained under reports/raw/warm-latency/ — the batch series, the environment stamp, and the per-case evidence each batch produced, so that a reader can check the batch did the real work rather than less of it — but freeze/v1 does not digest them, and this amendment does not extend it to. A warm number carries the release commit's immutability for its bytes and no stronger guarantee, exactly as the cold timing sidecars do. No warm run wrote a normalized report, produced an outcome, or touched a scored population; validate-reports and the freeze manifest never read this directory.

Per-slice medians

The tier's declared unit of aggregation: one row per slice — adapter × language × population — because a median over a whole adapter mixes languages whose fixtures differ in size and whose front ends differ in cost. Each row is one bound report. Rows are not ranked and columns are not compared across adapters with different granularity. One disclosure per adapter, for the same reason: the rows inside one of them are the rows that share a granularity.

Bifrost — 20 slices

Show the data table — 20 slices for Bifrost
Slice Language Profile Timed / bound Median Q1 Q3
bifrost-smoke 13 languages benchmark-controlled 117 / 118 117 ms 113 ms 123 ms
bifrost-c-kernel c benchmark-controlled 50 / 50 131 ms 124 ms 140 ms
bifrost-cpp-kernel cpp benchmark-controlled 56 / 56 130 ms 127 ms 136 ms
bifrost-csharp-kernel csharp benchmark-controlled 58 / 58 141 ms 130 ms 155 ms
bifrost-go-kernel go benchmark-controlled 58 / 58 123 ms 116 ms 128 ms
bifrost-java-kernel java benchmark-controlled 58 / 58 120 ms 114 ms 125 ms
bifrost-java-modeling java benchmark-controlled 8 / 24 144 ms 126 ms 151 ms
bifrost-java-native java tool-native 0 / 12 n/a n/a n/a
bifrost-javascript-kernel javascript benchmark-controlled 58 / 58 117 ms 112 ms 121 ms
bifrost-javascript-modeling javascript benchmark-controlled 8 / 24 196 ms 171 ms 203 ms
bifrost-javascript-native javascript tool-native 0 / 12 n/a n/a n/a
bifrost-kotlin-kernel kotlin benchmark-controlled 58 / 58 122 ms 119 ms 128 ms
bifrost-php-kernel php benchmark-controlled 58 / 58 122 ms 116 ms 129 ms
bifrost-python-kernel python benchmark-controlled 58 / 58 115 ms 110 ms 121 ms
bifrost-python-modeling python benchmark-controlled 8 / 24 181 ms 141 ms 289 ms
bifrost-python-native python tool-native 0 / 12 n/a n/a n/a
bifrost-ruby-kernel ruby benchmark-controlled 58 / 58 122 ms 117 ms 125 ms
bifrost-rust-kernel rust benchmark-controlled 56 / 56 121 ms 117 ms 128 ms
bifrost-scala-kernel scala benchmark-controlled 58 / 58 133 ms 125 ms 147 ms
bifrost-typescript-kernel typescript benchmark-controlled 58 / 58 144 ms 128 ms 163 ms

Semgrep CE — 17 slices

Show the data table — 17 slices for Semgrep CE
Slice Language Profile Timed / bound Median Q1 Q3
semgrep-c-kernel c benchmark-controlled 14 / 48 1.28 s 1.21 s 1.33 s
semgrep-cpp-kernel cpp benchmark-controlled 14 / 56 1.12 s 1.10 s 1.13 s
semgrep-go-kernel go benchmark-controlled 14 / 58 1.11 s 1.09 s 1.15 s
semgrep-java-kernel java benchmark-controlled 14 / 58 1.16 s 1.13 s 1.22 s
semgrep-java-modeling java benchmark-controlled 10 / 24 1.12 s 1.09 s 1.16 s
semgrep-java-native java tool-native 0 / 12 n/a n/a n/a
semgrep-javascript-kernel javascript benchmark-controlled 14 / 58 1.31 s 1.24 s 1.33 s
semgrep-javascript-modeling javascript benchmark-controlled 10 / 24 1.14 s 1.11 s 1.17 s
semgrep-javascript-native javascript tool-native 0 / 12 n/a n/a n/a
semgrep-kotlin-kernel kotlin benchmark-controlled 14 / 58 1.09 s 1.08 s 1.58 s
semgrep-php-kernel php benchmark-controlled 14 / 58 1.69 s 1.61 s 1.82 s
semgrep-python-kernel python benchmark-controlled 14 / 58 1.10 s 1.09 s 1.12 s
semgrep-python-modeling python benchmark-controlled 10 / 24 1.63 s 1.18 s 1.85 s
semgrep-python-native python tool-native 12 / 12 4.10 s 3.88 s 4.20 s
semgrep-ruby-kernel ruby benchmark-controlled 14 / 58 2.53 s 2.37 s 2.73 s
semgrep-rust-kernel rust benchmark-controlled 14 / 54 1.35 s 1.22 s 1.52 s
semgrep-typescript-kernel typescript benchmark-controlled 14 / 58 1.11 s 1.09 s 1.18 s

FlowDroid — 2 slices

Show the data table — 2 slices for FlowDroid
Slice Language Profile Timed / bound Median Q1 Q3
flowdroid-java-kernel java benchmark-controlled 58 / 58 1.55 s 1.49 s 1.61 s
flowdroid-kotlin-kernel kotlin benchmark-controlled 58 / 58 1.97 s 1.84 s 2.05 s

Infer — 3 slices

Show the data table — 3 slices for Infer
Slice Language Profile Timed / bound Median Q1 Q3 capture median analyze median
infer-c-kernel c benchmark-controlled 48 / 48 540 ms 494 ms 612 ms 159 ms372 ms
infer-cpp-kernel cpp benchmark-controlled 56 / 56 728 ms 550 ms 2.99 s 234 ms413 ms
infer-java-kernel java benchmark-controlled 58 / 58 6.47 s 6.15 s 6.77 s 6.09 s349 ms

Pysa — 1 slice

Show the data table — 1 slice for Pysa
Slice Language Profile Timed / bound Median Q1 Q3
pysa-python-kernel python benchmark-controlled 58 / 58 4.08 s 3.88 s 4.21 s

OpenTaint — 2 slices

Show the data table — 2 slices for OpenTaint
Slice Language Profile Timed / bound Median Q1 Q3
opentaint-java-kernel java benchmark-controlled 58 / 58 4.70 s 4.55 s 9.25 s
opentaint-kotlin-kernel kotlin benchmark-controlled 58 / 58 4.46 s 4.26 s 4.74 s

Joern — 12 slices

Show the data table — 12 slices for Joern
Slice Language Profile Timed / bound Median Q1 Q3
joern-java-kernel java benchmark-controlled 58 / 58 14.6 s 14.3 s 15.3 s
joern-java-modeling java benchmark-controlled 16 / 24 9.29 s 8.95 s 9.91 s
joern-java-native java tool-native 0 / 12 n/a n/a n/a
joern-javascript-kernel javascript benchmark-controlled 58 / 58 9.15 s 8.86 s 9.61 s
joern-javascript-modeling javascript benchmark-controlled 16 / 24 5.28 s 5.07 s 5.40 s
joern-javascript-native javascript tool-native 0 / 12 n/a n/a n/a
joern-php-kernel php benchmark-controlled 58 / 58 5.45 s 5.12 s 8.39 s
joern-python-kernel python benchmark-controlled 58 / 58 8.38 s 8.00 s 8.86 s
joern-python-modeling python benchmark-controlled 16 / 24 4.95 s 4.89 s 5.03 s
joern-python-native python tool-native 0 / 12 n/a n/a n/a
joern-ruby-kernel ruby benchmark-controlled 58 / 58 8.63 s 7.35 s 11.0 s
joern-rust-kernel rust benchmark-controlled 54 / 54 7.41 s 7.05 s 8.31 s

CodeQL — 17 slices

Show the data table — 17 slices for CodeQL
Slice Language Profile Timed / bound Median Q1 Q3 database-create median database-analyze median
codeql-c-kernel c benchmark-controlled 50 / 50 6.15 s 5.42 s 7.08 s 1.55 s4.62 s
codeql-cpp-kernel cpp benchmark-controlled 56 / 56 7.28 s 6.19 s 19.1 s 1.89 s5.40 s
codeql-csharp-kernel csharp benchmark-controlled 58 / 58 33.1 s 24.1 s 39.7 s 20.1 s13.0 s
codeql-go-kernel go benchmark-controlled 58 / 58 9.43 s 9.24 s 9.81 s 4.40 s5.05 s
codeql-java-kernel java benchmark-controlled 58 / 58 12.7 s 12.5 s 12.9 s 6.70 s6.00 s
codeql-java-modeling java benchmark-controlled 24 / 24 12.3 s 11.2 s 13.2 s 6.40 s5.58 s
codeql-java-native java tool-native 12 / 12 22.6 s 17.3 s 25.9 s 7.77 s14.8 s
codeql-javascript-kernel javascript benchmark-controlled 58 / 58 10.7 s 9.80 s 13.5 s 4.49 s6.12 s
codeql-javascript-modeling javascript benchmark-controlled 24 / 24 13.3 s 8.82 s 13.8 s 5.59 s7.76 s
codeql-javascript-native javascript tool-native 12 / 12 21.7 s 21.4 s 22.0 s 5.68 s16.1 s
codeql-kotlin-kernel kotlin benchmark-controlled 58 / 58 15.8 s 15.6 s 17.0 s 10.9 s4.85 s
codeql-python-kernel python benchmark-controlled 58 / 58 5.38 s 5.30 s 5.83 s 2.00 s3.41 s
codeql-python-modeling python benchmark-controlled 24 / 24 9.85 s 9.72 s 9.98 s 3.66 s6.15 s
codeql-python-native python tool-native 12 / 12 12.6 s 12.4 s 12.9 s 3.70 s8.86 s
codeql-ruby-kernel ruby benchmark-controlled 58 / 58 4.70 s 4.58 s 4.89 s 1.34 s3.33 s
codeql-rust-kernel rust benchmark-controlled 56 / 56 47.1 s 45.5 s 49.2 s 20.9 s25.7 s
codeql-typescript-kernel typescript benchmark-controlled 58 / 58 8.92 s 8.09 s 9.80 s 3.73 s5.04 s

Cases with no latency number

Of the bound case results in this freeze, 2657 invoked an analyzer and carry a timing sidecar. 679 do not, and every one of them is accounted for:

Outcome of the untimed case Cases
unsupported 679

An unsupported case is decided from case metadata before the analyzer is invoked, so there is no subprocess and there is nothing to time. Those cases are absent from every distribution on this page, not entered as zero — entering them as zero would make the adapters that decline the most look the fastest, which is exactly backwards. Every case that did invoke an analyzer is timed; the timed and untimed columns account for the whole bound population with no remainder.

Provenance and what the freeze does and does not guarantee

Every number above was derived at build time from v0.6.0's own retained artifacts. The publication gate is the freeze manifest (3228af686d09…): a timing sidecar is read only when the manifest binds that report and that case, so a sidecar left in the tree by a run the freeze does not bind cannot reach a number here.

One limit, stated rather than left to be assumed. The freeze/v1 manifest binds one raw-evidence digest per result; the timing sidecar and the environment stamp are additive files beside that evidence and are not themselves digested by the manifest. So a number on this page carries the freeze's guarantee that the run it belongs to is bound and byte-verified, and the release commit's own immutability for the timing bytes. It does not carry a manifest digest over those bytes. Extending the manifest to bind them is a freeze/v2 question, and it is named here rather than smuggled in under a schema version that does not describe it.