Skip to content

Latency — v0.6.1

DataFlowBench’s latency-characterization tier. It stands beside the correctness scorecards, on its own page, and never as a column inside one.

The contract governing every number here — docs/latency-tier.md — was written and merged before any timestamp was captured, and it is explicit about why. DataFlowBench is published by the vendor of one of the engines measured, and a latency page assembled after the numbers were known, with its phase boundaries and its aggregation chosen afterwards, would deserve exactly the skepticism it would get. So the decomposition rule, the exclusion list, the per-adapter granularity, and the aggregation were all fixed while every number was still unmeasured. The four analyzers added in v0.6.0 took their granularity rows by Amendment A12, dated and landed before this page published any of their numbers, rather than being backfilled into the original table.

The page also carries a second, separately labelled figure: the warm marginal cost of one more case in a tool process that has already started, added by Amendment A15 and likewise preregistered before the first warm number was measured. It exists because the cold rows below span three runtimes, and read across runtimes they overstate the steady-state gap — a difference the vendor publishing this benchmark happens to win. The correction is a direct measurement, not an estimate subtracted from anything: no cold number on this page changes, and the warm figure is never substituted for one.

What these numbers are, and what they are not

Section titled “What these numbers are, and what they are not”

This is a descriptive characterization of per-case analyzer wall-clock, published under the contract preregistered in docs/latency-tier.md, which was merged before a single timestamp was captured. It is not a score, not a ranking, and not a benchmark result in the sense the rest of this site uses that phrase.

  • Latency is never pooled with correctness. There is no combined number anywhere on this site, no efficiency-adjusted rate, and no leaderboard that blends the two. "Correct but slow" and "fast but wrong" stay independently visible. No correctness outcome in this freeze was derived from, conditioned on, or tie-broken by a timing value.
  • A latency number is a property of a run, not of a case. No case carries a timing assertion, threshold, or budget.
  • Characterization, not micro-benchmark. There are no repeated trials, no warm-up iterations, and no steady-state harness in the cold figures below. The case population is the sample and its spread is the statistics. The distribution columns below are that spread — not error bars on a measurement that was repeated, because it was not. (The warm-marginal section is the one place a series is measured more than once. It says so, retains every repeat, and publishes the range they span rather than averaging them into a statistic.)
  • These are the conditions the numbers were produced under: a single developer machine, under light concurrent load, running the benchmark's standing sequential-run discipline — one analyzer at a time, never two at once. That discipline is what makes the numbers usable at all; the light concurrent load is what stops them being a controlled measurement. Read them as characterization of what this benchmark actually costs to run, at the order-of-magnitude and shape level, and not as a precise figure for any engine.

Recorded once per run beside the tool identity the run witnessed, and shown here verbatim. Latency numbers are comparable within one environment and are not comparable across machines. Every published run to date executes on one maintainer machine, and that caveat travels with every number on this page.

Hardware modelOSOS releaseLogical CPUsCPU architectureTimed runs stamped
Mac16,1macos25.6.010aarch6466

One row: every timed invocation in this bound latency corpus ran in the same stamped environment, so no number on this page is a cross-environment comparison. Were there two rows, this page would say so in the table rather than averaging across them. The count is of run directories that contain at least one timed invocation — a run that invoked no analyzer at all still stamps its environment and witnesses its tool identity, but contributes no latency number and is not counted here.

The granularity is unequal, and that is the point

Section titled “The granularity is unequal, and that is the point”

A phase is timed if and only if the adapter already invokes it as a separate subprocess. The benchmark never instruments analyzer internals, never patches an upstream tool, and never infers a boundary a subprocess boundary does not expose. The adapters do not expose the same boundaries, so the rows below are not equally decomposed — and the two ways of papering over that would both misdescribe the evidence: summing a decomposed adapter's phases and presenting every row as one number of the same kind, or guessing phase splits for the adapters that expose only one.

So the table states each adapter's declared granularity as data. Phases compare within an adapter; analyzer-invocation totals compare across adapters. A CodeQL database-create number and a Joern total are not the same kind of quantity, and nothing on this page sets them side by side as though they were.

AdapterVersionObserved subprocesses per casePhases timedTimed invocations
Bifrost0.10.70 or 1total 883
Semgrep CE1.175.00 or 1total 196
FlowDroid2.15.11total 116
Inferv1.3.02capture analyze 162
Pysa0.10.01total 58
OpenTaintanalyzer/2026.08.27.17eb0fe1total 116
Joern4.0.6140 or 1total 392
CodeQL2.26.42database-create database-analyze 734

2 of 8 adapters expose more than one subprocess and are decomposed here (Infer and CodeQL); the other 6 expose a single invocation and take one honestly labelled number, with interpreter or JVM start-up inside it. That is a fact about invocation shapes, not about engine architecture: an adapter with one number is not an adapter that does one thing.

The phase names above are the ones the retained evidence carries, so any name here can be found in the artifact it came from. For CodeQL those are database-create and database-analyze, after the subcommands that produce them, where the preregistration's table spells the same two boundaries extract and analyze. Two spellings, one pair of subprocess boundaries: nothing moved and nothing is attributed differently. The contract is immutable, so the difference is recorded in its Amendment A12 rather than edited out of its table.

Analyzer-invocation wall-clock, per adapter

Section titled “Analyzer-invocation wall-clock, per adapter”

The only cross-adapter reading this tier supports: each adapter's cold per-case analyzer-invocation wall-clock, over every case in this snapshot's explicitly bound latency population that it actually invoked. Adapter-observable setup phases remain visible below, but only phases the contract classifies as analyzer work enter this total. Per-case timing at this granularity includes per-invocation fixed costs — JVM start-up, extractor initialization, interpreter start — that a long-lived deployment of the same engine would amortize. This tier does not correct for that; it characterizes what the benchmark actually runs.

Fastest to slowest, and the same ranking per kernel

Ordered by median, on a logarithmic axis, with the spread drawn beside every median rather than left to the table. The toggle scopes the same ranking to one language kernel at a time: a whole-corpus median mixes the languages an adapter runs on, and holding the language fixed is the difference between "this engine is slow" and "this front end is expensive". Only the analyzers that actually invoked on a kernel appear in its view. The chart draws the median, the quartiles and the p10–p90 spread; the disclosure beneath it opens the same distributions as a table, with the minima and maxima the whiskers deliberately leave out.

Cold per-case analyzer-invocation wall-clock over the bound latency population One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. Materialization phases are explicitly labelled and excluded from the analyzer total. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. A solid caret below a row is that adapter's separately measured warm marginal per case; a dashed span above a row is its estimated per-invocation overhead, an upper bound measured on a trivial no-flow fixture, drawn across the range its repeats spanned rather than at a point. Neither is a cold median, neither affects the ordering, and neither is subtracted from anything. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 123 ms, IQR 116 ms to 133 ms, p10–p90 111 ms to 149 ms, over 883 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow python fixture: 108 ms to 239 ms across repeats Bifrost 883 timed 123 ms Semgrep CE — median 1.20 s, IQR 1.11 s to 1.59 s, p10–p90 1.08 s to 2.47 s, over 196 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow kotlin fixture: 1.04 s to 1.07 s across repeats Semgrep CE 196 timed 1.20 s FlowDroid — median 1.70 s, IQR 1.55 s to 1.96 s, p10–p90 1.48 s to 2.08 s, over 116 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow java fixture: 678 ms to 771 ms across repeats FlowDroid 116 timed 1.70 s Infer — median 2.21 s, IQR 554 ms to 6.24 s, p10–p90 504 ms to 6.74 s, over 162 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow c fixture: 288 ms to 491 ms across repeats Infer 162 timed 2.21 s Infer phase capture — median 1.78 s, IQR 160 ms to 5.87 s, p10–p90 151 ms to 6.37 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. capture 1.78 s Infer phase analyze — median 373 ms, IQR 343 ms to 440 ms, p10–p90 324 ms to 500 ms. Included in this adapter's analyzer total; phase detail compares only within the adapter. analyze 373 ms Pysa — median 4.08 s, IQR 3.88 s to 4.21 s, p10–p90 3.77 s to 4.58 s, over 58 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow python fixture: 2.82 s to 2.98 s across repeats Pysa 58 timed 4.08 s OpenTaint — median 4.60 s, IQR 4.39 s to 4.98 s, p10–p90 4.22 s to 9.51 s, over 116 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow kotlin fixture: 4.08 s to 4.14 s across repeats OpenTaint 116 timed 4.60 s Joern — median 8.70 s, IQR 7.18 s to 10.1 s, p10–p90 5.17 s to 14.6 s, over 392 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow php fixture: 4.42 s to 5.34 s across repeats Joern 392 timed 8.70 s CodeQL — median 10.8 s, IQR 7.22 s to 16.7 s, p10–p90 5.30 s to 39.4 s, over 734 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow ruby fixture: 4.02 s to 4.49 s across repeats CodeQL 734 timed 10.8 s CodeQL phase database-create — median 4.51 s, IQR 2.18 s to 10.7 s, p10–p90 1.42 s to 20.4 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-create 4.51 s CodeQL phase database-analyze — median 5.58 s, IQR 4.64 s to 8.29 s, p10–p90 3.45 s to 15.4 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-analyze 5.58 s

Every timed analyzer invocation the freeze binds — 2657 of them, across every score tier and both model profiles. This is the widest denominator on the site and the only one here that is not a single population: an adapter's median mixes the languages it runs on, whose fixtures differ in size and whose front ends differ in cost. The per-kernel views hold the language fixed.

Cold per-case analyzer-invocation wall-clock on the c kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. Materialization phases are explicitly labelled and excluded from the analyzer total. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. A solid caret below a row is that adapter's separately measured warm marginal per case; a dashed span above a row is its estimated per-invocation overhead, an upper bound measured on a trivial no-flow fixture, drawn across the range its repeats spanned rather than at a point. Neither is a cold median, neither affects the ordering, and neither is subtracted from anything. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 131 ms, IQR 124 ms to 141 ms, p10–p90 120 ms to 155 ms, over 48 timed analyzer invocations Bifrost 48 of 48 timed 131 ms Infer — median 540 ms, IQR 494 ms to 612 ms, p10–p90 484 ms to 755 ms, over 48 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow c fixture: 288 ms to 491 ms across repeats Infer 48 of 48 timed 540 ms Infer phase capture — median 159 ms, IQR 150 ms to 183 ms, p10–p90 144 ms to 210 ms. Included in this adapter's analyzer total; phase detail compares only within the adapter. capture 159 ms Infer phase analyze — median 372 ms, IQR 344 ms to 435 ms, p10–p90 331 ms to 527 ms. Included in this adapter's analyzer total; phase detail compares only within the adapter. analyze 372 ms Semgrep CE — median 1.28 s, IQR 1.21 s to 1.33 s, p10–p90 1.17 s to 1.37 s, over 14 timed analyzer invocations Semgrep CE 14 of 48 timed 1.28 s CodeQL — median 6.28 s, IQR 5.41 s to 7.11 s, p10–p90 5.31 s to 8.21 s, over 48 timed analyzer invocations CodeQL 48 of 48 timed 6.28 s CodeQL phase database-create — median 1.56 s, IQR 1.39 s to 1.79 s, p10–p90 1.36 s to 2.20 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-create 1.56 s CodeQL phase database-analyze — median 4.72 s, IQR 4.01 s to 5.37 s, p10–p90 3.91 s to 6.24 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-analyze 4.72 s

The c kernel's 48 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 48, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Cold per-case analyzer-invocation wall-clock on the cpp kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. Materialization phases are explicitly labelled and excluded from the analyzer total. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. A solid caret below a row is that adapter's separately measured warm marginal per case; a dashed span above a row is its estimated per-invocation overhead, an upper bound measured on a trivial no-flow fixture, drawn across the range its repeats spanned rather than at a point. Neither is a cold median, neither affects the ordering, and neither is subtracted from anything. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 130 ms, IQR 127 ms to 136 ms, p10–p90 120 ms to 144 ms, over 56 timed analyzer invocations Bifrost 56 of 56 timed 130 ms Infer — median 728 ms, IQR 550 ms to 2.99 s, p10–p90 518 ms to 3.33 s, over 56 timed analyzer invocations Infer 56 of 56 timed 728 ms Infer phase capture — median 234 ms, IQR 161 ms to 2.55 s, p10–p90 153 ms to 2.91 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. capture 234 ms Infer phase analyze — median 413 ms, IQR 370 ms to 466 ms, p10–p90 358 ms to 518 ms. Included in this adapter's analyzer total; phase detail compares only within the adapter. analyze 413 ms Semgrep CE — median 1.12 s, IQR 1.10 s to 1.13 s, p10–p90 1.08 s to 1.16 s, over 14 timed analyzer invocations Semgrep CE 14 of 56 timed 1.12 s CodeQL — median 7.28 s, IQR 6.19 s to 19.1 s, p10–p90 5.40 s to 21.3 s, over 56 timed analyzer invocations CodeQL 56 of 56 timed 7.28 s CodeQL phase database-create — median 1.89 s, IQR 1.54 s to 10.9 s, p10–p90 1.41 s to 12.6 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-create 1.89 s CodeQL phase database-analyze — median 5.40 s, IQR 4.62 s to 8.44 s, p10–p90 3.99 s to 8.80 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-analyze 5.40 s

The cpp kernel's 56 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 56, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Cold per-case analyzer-invocation wall-clock on the csharp kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. Materialization phases are explicitly labelled and excluded from the analyzer total. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. A solid caret below a row is that adapter's separately measured warm marginal per case; a dashed span above a row is its estimated per-invocation overhead, an upper bound measured on a trivial no-flow fixture, drawn across the range its repeats spanned rather than at a point. Neither is a cold median, neither affects the ordering, and neither is subtracted from anything. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 141 ms, IQR 130 ms to 155 ms, p10–p90 122 ms to 166 ms, over 58 timed analyzer invocations Bifrost 58 of 58 timed 141 ms CodeQL — median 33.1 s, IQR 24.1 s to 39.7 s, p10–p90 20.7 s to 41.3 s, over 58 timed analyzer invocations CodeQL 58 of 58 timed 33.1 s CodeQL phase database-create — median 20.1 s, IQR 13.6 s to 25.7 s, p10–p90 11.5 s to 26.7 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-create 20.1 s CodeQL phase database-analyze — median 13.0 s, IQR 9.42 s to 14.2 s, p10–p90 9.12 s to 14.6 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-analyze 13.0 s

The csharp kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 2 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Cold per-case analyzer-invocation wall-clock on the go kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. Materialization phases are explicitly labelled and excluded from the analyzer total. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. A solid caret below a row is that adapter's separately measured warm marginal per case; a dashed span above a row is its estimated per-invocation overhead, an upper bound measured on a trivial no-flow fixture, drawn across the range its repeats spanned rather than at a point. Neither is a cold median, neither affects the ordering, and neither is subtracted from anything. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 123 ms, IQR 116 ms to 128 ms, p10–p90 115 ms to 140 ms, over 58 timed analyzer invocations Bifrost 58 of 58 timed 123 ms Semgrep CE — median 1.11 s, IQR 1.09 s to 1.15 s, p10–p90 1.08 s to 1.17 s, over 14 timed analyzer invocations Semgrep CE 14 of 58 timed 1.11 s CodeQL — median 9.43 s, IQR 9.24 s to 9.81 s, p10–p90 9.12 s to 10.9 s, over 58 timed analyzer invocations CodeQL 58 of 58 timed 9.43 s CodeQL phase database-create — median 4.40 s, IQR 4.30 s to 4.52 s, p10–p90 4.26 s to 5.67 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-create 4.40 s CodeQL phase database-analyze — median 5.05 s, IQR 4.90 s to 5.14 s, p10–p90 4.75 s to 5.41 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-analyze 5.05 s

The go kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 3 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Cold per-case analyzer-invocation wall-clock on the java kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. Materialization phases are explicitly labelled and excluded from the analyzer total. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. A solid caret below a row is that adapter's separately measured warm marginal per case; a dashed span above a row is its estimated per-invocation overhead, an upper bound measured on a trivial no-flow fixture, drawn across the range its repeats spanned rather than at a point. Neither is a cold median, neither affects the ordering, and neither is subtracted from anything. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 120 ms, IQR 114 ms to 125 ms, p10–p90 111 ms to 130 ms, over 58 timed analyzer invocations Bifrost 58 of 58 timed 120 ms Semgrep CE — median 1.16 s, IQR 1.13 s to 1.22 s, p10–p90 1.11 s to 1.28 s, over 14 timed analyzer invocations. Warm marginal, measured separately: 74 ms to 77 ms per case in one process, over 2 retained repeats Semgrep CE 14 of 58 timed 1.16 s FlowDroid — median 1.55 s, IQR 1.49 s to 1.61 s, p10–p90 1.43 s to 1.67 s, over 58 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow java fixture: 678 ms to 771 ms across repeats FlowDroid 58 of 58 timed 1.55 s OpenTaint — median 4.70 s, IQR 4.55 s to 9.25 s, p10–p90 4.41 s to 9.96 s, over 58 timed analyzer invocations OpenTaint 58 of 58 timed 4.70 s Infer — median 6.47 s, IQR 6.15 s to 6.77 s, p10–p90 6.03 s to 7.20 s, over 58 timed analyzer invocations Infer 58 of 58 timed 6.47 s Infer phase capture — median 6.09 s, IQR 5.81 s to 6.39 s, p10–p90 5.65 s to 6.85 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. capture 6.09 s Infer phase analyze — median 349 ms, IQR 331 ms to 393 ms, p10–p90 314 ms to 439 ms. Included in this adapter's analyzer total; phase detail compares only within the adapter. analyze 349 ms CodeQL — median 12.7 s, IQR 12.5 s to 12.9 s, p10–p90 12.3 s to 13.1 s, over 58 timed analyzer invocations CodeQL 58 of 58 timed 12.7 s CodeQL phase database-create — median 6.70 s, IQR 6.58 s to 6.81 s, p10–p90 6.48 s to 7.10 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-create 6.70 s CodeQL phase database-analyze — median 6.00 s, IQR 5.87 s to 6.11 s, p10–p90 5.80 s to 6.18 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-analyze 6.00 s Joern — median 14.6 s, IQR 14.3 s to 15.3 s, p10–p90 14.1 s to 16.2 s, over 58 timed analyzer invocations. Warm marginal, measured separately: 996 ms to 1.01 s per case in one process, over 2 retained repeats. Estimated per-invocation overhead, an upper bound from a trivial no-flow java fixture: 8.91 s to 13.2 s across repeats Joern 58 of 58 timed 14.6 s

The java kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 7 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Cold per-case analyzer-invocation wall-clock on the javascript kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. Materialization phases are explicitly labelled and excluded from the analyzer total. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. A solid caret below a row is that adapter's separately measured warm marginal per case; a dashed span above a row is its estimated per-invocation overhead, an upper bound measured on a trivial no-flow fixture, drawn across the range its repeats spanned rather than at a point. Neither is a cold median, neither affects the ordering, and neither is subtracted from anything. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 117 ms, IQR 112 ms to 121 ms, p10–p90 108 ms to 127 ms, over 58 timed analyzer invocations Bifrost 58 of 58 timed 117 ms Semgrep CE — median 1.31 s, IQR 1.24 s to 1.33 s, p10–p90 1.20 s to 1.44 s, over 14 timed analyzer invocations Semgrep CE 14 of 58 timed 1.31 s Joern — median 9.15 s, IQR 8.86 s to 9.61 s, p10–p90 8.64 s to 10.2 s, over 58 timed analyzer invocations Joern 58 of 58 timed 9.15 s CodeQL — median 10.7 s, IQR 9.80 s to 13.5 s, p10–p90 9.45 s to 14.0 s, over 58 timed analyzer invocations CodeQL 58 of 58 timed 10.7 s CodeQL phase database-create — median 4.49 s, IQR 4.15 s to 5.50 s, p10–p90 3.98 s to 5.76 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-create 4.49 s CodeQL phase database-analyze — median 6.12 s, IQR 5.62 s to 8.01 s, p10–p90 5.43 s to 8.21 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-analyze 6.12 s

The javascript kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Cold per-case analyzer-invocation wall-clock on the kotlin kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. Materialization phases are explicitly labelled and excluded from the analyzer total. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. A solid caret below a row is that adapter's separately measured warm marginal per case; a dashed span above a row is its estimated per-invocation overhead, an upper bound measured on a trivial no-flow fixture, drawn across the range its repeats spanned rather than at a point. Neither is a cold median, neither affects the ordering, and neither is subtracted from anything. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 122 ms, IQR 119 ms to 128 ms, p10–p90 115 ms to 132 ms, over 58 timed analyzer invocations Bifrost 58 of 58 timed 122 ms Semgrep CE — median 1.09 s, IQR 1.08 s to 1.58 s, p10–p90 1.05 s to 1.67 s, over 14 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow kotlin fixture: 1.04 s to 1.07 s across repeats Semgrep CE 14 of 58 timed 1.09 s FlowDroid — median 1.97 s, IQR 1.84 s to 2.05 s, p10–p90 1.76 s to 2.15 s, over 58 timed analyzer invocations FlowDroid 58 of 58 timed 1.97 s OpenTaint — median 4.46 s, IQR 4.26 s to 4.74 s, p10–p90 4.16 s to 4.97 s, over 58 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow kotlin fixture: 4.08 s to 4.14 s across repeats OpenTaint 58 of 58 timed 4.46 s CodeQL — median 15.8 s, IQR 15.6 s to 17.0 s, p10–p90 15.5 s to 27.1 s, over 58 timed analyzer invocations CodeQL 58 of 58 timed 15.8 s CodeQL phase database-create — median 10.9 s, IQR 10.8 s to 11.6 s, p10–p90 10.7 s to 17.7 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-create 10.9 s CodeQL phase database-analyze — median 4.85 s, IQR 4.78 s to 5.24 s, p10–p90 4.73 s to 7.61 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-analyze 4.85 s

The kotlin kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 5 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Cold per-case analyzer-invocation wall-clock on the php kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. Materialization phases are explicitly labelled and excluded from the analyzer total. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. A solid caret below a row is that adapter's separately measured warm marginal per case; a dashed span above a row is its estimated per-invocation overhead, an upper bound measured on a trivial no-flow fixture, drawn across the range its repeats spanned rather than at a point. Neither is a cold median, neither affects the ordering, and neither is subtracted from anything. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 122 ms, IQR 116 ms to 129 ms, p10–p90 110 ms to 137 ms, over 58 timed analyzer invocations Bifrost 58 of 58 timed 122 ms Semgrep CE — median 1.69 s, IQR 1.61 s to 1.82 s, p10–p90 1.56 s to 1.94 s, over 14 timed analyzer invocations Semgrep CE 14 of 58 timed 1.69 s Joern — median 5.45 s, IQR 5.12 s to 8.39 s, p10–p90 5.05 s to 11.9 s, over 58 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow php fixture: 4.42 s to 5.34 s across repeats Joern 58 of 58 timed 5.45 s

The php kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 3 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Cold per-case analyzer-invocation wall-clock on the python kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. Materialization phases are explicitly labelled and excluded from the analyzer total. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. A solid caret below a row is that adapter's separately measured warm marginal per case; a dashed span above a row is its estimated per-invocation overhead, an upper bound measured on a trivial no-flow fixture, drawn across the range its repeats spanned rather than at a point. Neither is a cold median, neither affects the ordering, and neither is subtracted from anything. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 115 ms, IQR 110 ms to 121 ms, p10–p90 106 ms to 127 ms, over 58 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow python fixture: 108 ms to 239 ms across repeats Bifrost 58 of 58 timed 115 ms Semgrep CE — median 1.10 s, IQR 1.09 s to 1.12 s, p10–p90 1.07 s to 1.13 s, over 14 timed analyzer invocations Semgrep CE 14 of 58 timed 1.10 s Pysa — median 4.08 s, IQR 3.88 s to 4.21 s, p10–p90 3.77 s to 4.58 s, over 58 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow python fixture: 2.82 s to 2.98 s across repeats Pysa 58 of 58 timed 4.08 s CodeQL — median 5.38 s, IQR 5.30 s to 5.83 s, p10–p90 5.26 s to 5.90 s, over 58 timed analyzer invocations CodeQL 58 of 58 timed 5.38 s CodeQL phase database-create — median 2.00 s, IQR 1.97 s to 2.13 s, p10–p90 1.95 s to 2.18 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-create 2.00 s CodeQL phase database-analyze — median 3.41 s, IQR 3.31 s to 3.67 s, p10–p90 3.27 s to 3.73 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-analyze 3.41 s Joern — median 8.38 s, IQR 8.00 s to 8.86 s, p10–p90 7.70 s to 9.29 s, over 58 timed analyzer invocations Joern 58 of 58 timed 8.38 s

The python kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 5 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Cold per-case analyzer-invocation wall-clock on the ruby kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. Materialization phases are explicitly labelled and excluded from the analyzer total. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. A solid caret below a row is that adapter's separately measured warm marginal per case; a dashed span above a row is its estimated per-invocation overhead, an upper bound measured on a trivial no-flow fixture, drawn across the range its repeats spanned rather than at a point. Neither is a cold median, neither affects the ordering, and neither is subtracted from anything. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 122 ms, IQR 117 ms to 125 ms, p10–p90 112 ms to 128 ms, over 58 timed analyzer invocations Bifrost 58 of 58 timed 122 ms Semgrep CE — median 2.53 s, IQR 2.37 s to 2.73 s, p10–p90 2.29 s to 2.89 s, over 14 timed analyzer invocations Semgrep CE 14 of 58 timed 2.53 s CodeQL — median 4.70 s, IQR 4.58 s to 4.89 s, p10–p90 4.47 s to 5.13 s, over 58 timed analyzer invocations. Estimated per-invocation overhead, an upper bound from a trivial no-flow ruby fixture: 4.02 s to 4.49 s across repeats CodeQL 58 of 58 timed 4.70 s CodeQL phase database-create — median 1.34 s, IQR 1.29 s to 1.40 s, p10–p90 1.27 s to 1.43 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-create 1.34 s CodeQL phase database-analyze — median 3.33 s, IQR 3.25 s to 3.49 s, p10–p90 3.19 s to 3.68 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-analyze 3.33 s Joern — median 8.63 s, IQR 7.35 s to 11.0 s, p10–p90 7.13 s to 13.7 s, over 58 timed analyzer invocations Joern 58 of 58 timed 8.63 s

The ruby kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Cold per-case analyzer-invocation wall-clock on the rust kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. Materialization phases are explicitly labelled and excluded from the analyzer total. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. A solid caret below a row is that adapter's separately measured warm marginal per case; a dashed span above a row is its estimated per-invocation overhead, an upper bound measured on a trivial no-flow fixture, drawn across the range its repeats spanned rather than at a point. Neither is a cold median, neither affects the ordering, and neither is subtracted from anything. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 122 ms, IQR 118 ms to 129 ms, p10–p90 113 ms to 133 ms, over 54 timed analyzer invocations Bifrost 54 of 54 timed 122 ms Semgrep CE — median 1.35 s, IQR 1.22 s to 1.52 s, p10–p90 1.17 s to 1.82 s, over 14 timed analyzer invocations Semgrep CE 14 of 54 timed 1.35 s Joern — median 7.41 s, IQR 7.05 s to 8.31 s, p10–p90 6.82 s to 9.18 s, over 54 timed analyzer invocations Joern 54 of 54 timed 7.41 s CodeQL — median 47.0 s, IQR 45.4 s to 50.1 s, p10–p90 44.8 s to 61.5 s, over 54 timed analyzer invocations CodeQL 54 of 54 timed 47.0 s CodeQL phase database-create — median 21.0 s, IQR 20.3 s to 21.8 s, p10–p90 20.1 s to 27.7 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-create 21.0 s CodeQL phase database-analyze — median 25.6 s, IQR 24.9 s to 27.3 s, p10–p90 24.6 s to 31.0 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-analyze 25.6 s

The rust kernel's 54 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 54, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Cold per-case analyzer-invocation wall-clock on the scala kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. Materialization phases are explicitly labelled and excluded from the analyzer total. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. A solid caret below a row is that adapter's separately measured warm marginal per case; a dashed span above a row is its estimated per-invocation overhead, an upper bound measured on a trivial no-flow fixture, drawn across the range its repeats spanned rather than at a point. Neither is a cold median, neither affects the ordering, and neither is subtracted from anything. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 133 ms, IQR 125 ms to 147 ms, p10–p90 121 ms to 164 ms, over 58 timed analyzer invocations Bifrost 58 of 58 timed 133 ms

The scala kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 1 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

Cold per-case analyzer-invocation wall-clock on the typescript kernel One row per analyzer, ordered fastest median first, on a logarithmic time axis. Each row draws the tenth to ninetieth percentile as a thin line, the interquartile range as a bar, and the median as a thick tick, with the median also printed as a number. Indented rows are the declared phases of adapters whose invocation exposes more than one subprocess, and belong to the adapter above them only. Materialization phases are explicitly labelled and excluded from the analyzer total. The latency page carries every value, including the minima and maxima this chart does not draw, in the data tables behind its "Show the data table" disclosures. A solid caret below a row is that adapter's separately measured warm marginal per case; a dashed span above a row is its estimated per-invocation overhead, an upper bound measured on a trivial no-flow fixture, drawn across the range its repeats spanned rather than at a point. Neither is a cold median, neither affects the ordering, and neither is subtracted from anything. 100 ms 300 ms 1 s 3 s 10 s 30 s 100 s wall-clock per invocation — logarithmic scale Bifrost — median 144 ms, IQR 128 ms to 163 ms, p10–p90 119 ms to 203 ms, over 58 timed analyzer invocations Bifrost 58 of 58 timed 144 ms Semgrep CE — median 1.11 s, IQR 1.09 s to 1.18 s, p10–p90 1.07 s to 1.20 s, over 14 timed analyzer invocations Semgrep CE 14 of 58 timed 1.11 s CodeQL — median 8.92 s, IQR 8.09 s to 9.80 s, p10–p90 8.01 s to 10.2 s, over 58 timed analyzer invocations CodeQL 58 of 58 timed 8.92 s CodeQL phase database-create — median 3.73 s, IQR 3.53 s to 4.21 s, p10–p90 3.43 s to 4.36 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-create 3.73 s CodeQL phase database-analyze — median 5.04 s, IQR 4.55 s to 5.57 s, p10–p90 4.50 s to 5.80 s. Included in this adapter's analyzer total; phase detail compares only within the adapter. database-analyze 5.04 s

The typescript kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 3 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

  • median, printed beside every row
  • interquartile range (Q1–Q3)
  • p10–p90; the minimum and maximum are in the tables, not the whiskers
  • indented rows: an adapter's own declared phases
  • measured warm marginal per case — the range its retained repeats span
  • estimated per-invocation overhead (trivial fixture, upper bound), drawn across the range its repeats spanned, where that range starts at or above 25% of the row's median

How to read this figure

Every bar above is cold per-invocation wall-clock, and the warm marginal is a different quantity measured separately. Cold is what this benchmark actually runs — one process per case, start-up inside the number, because start-up is not observable from inside a single invocation. Read across runtimes, though, those bars overstate the steady-state gap: a JVM engine's row carries a JVM start a long-lived deployment pays once. So the other quantity is measured directly rather than estimated and subtracted — k cases through one tool process, for increasing k, reporting the slope of batch wall-clock against k. The carets mark it on the rows that have one, and they mark a range: the whole series is measured more than once and the figure published is the span its repeats cover, so the mark's width is its own precision rather than a number stated more exactly than it was measured. It is never subtracted from a median, never substituted for one, and never used to order the rows. Only adapters whose released CLI exposes a real multi-case batch have a figure at all; the rest are not observable with the released CLI, and the warm-marginal section records every verdict, measured and declined, with the evidence behind it.

The dashed spans are estimates, not measurements of the same kind as the bars. Each is the wall-clock of one complete adapter invocation — same pipeline, same policy, rule or query, same subprocess shape — over a trivial no-flow fixture: both benchmark endpoints declared, nothing connecting them, nothing to find. That is fixed per-invocation overhead plus the trivial fixture's own near-zero analysis, so it is an upper bound on what an adapter pays before it starts work, and it is a cold single-shot execution — the same posture the bars are measured in, and not a steady-state one. It is never subtracted from a median, never substituted for one, and never used to order the rows. The width of a span is the figure's precision, not a decoration: each measurement is repeated a fixed number of times and what is published is the range those repeats spanned — never a mean, never one chosen repeat, and never withheld for repeats that disagree, because a disagreement widens the range and that is the honest consequence of it. A span is drawn only where the range starts at or above 25% of that row's own cold median — a threshold fixed in the amendment before any estimate was measured, read at the low end so that no mark can appear on the strength of one slow repeat. Each estimate is measured in one named language, stated on the row it annotates, because boot cost is not language-free. The estimates table carries every value — every repeat behind every range, every unmarked row, and every adapter for which no estimate could be taken at all.

The axis is logarithmic. Each labelled tick is three times the one before it, so equal distances are equal ratios, not equal durations — the gap from 100 ms to 300 ms is drawn the same width as the gap from 10 s to 30 s. That is the only way this bound corpus's medians fit in one picture: they span 123 ms to 10.8 s, and on a linear axis every analyzer except the slowest would be a sliver against the origin. Because a log axis is easy to misread, every median is also printed as a number at the right of its own row.

The indented rows are phases, and they are not comparable across adapters. Only the 2 adapters whose preregistered row declares more than one subprocess have them. A phase mark sits on the same axis as the totals because it is the same kind of measurement — wall-clock of a subprocess — but reading one adapter's phase against another adapter's total is precisely the comparison the granularity rule forbids. Read a phase against the adapter it is indented under, and nothing else.

Ordering is not scoring, and this is never pooled with correctness. Rows are sorted by median because an unsorted ranking is unreadable, not because latency is a result. No correctness figure appears in this chart and no number here is blended with one: there is no combined score anywhere on this site, and a fast analyzer that answers wrongly is neither rewarded nor penalised by anything drawn above. Cases an analyzer declined before invocation are absent, not entered as zero — entering them as zero would make the analyzers that decline the most look the fastest, which is exactly backwards.

Show the data table — analyzer-invocation wall-clock per adapter, with minima and maxima
AdapterTimed invocationsMedianQ1Q3MinMax
Bifrost883123 ms116 ms133 ms103 ms393 ms
Semgrep CE1961.20 s1.11 s1.59 s1.03 s4.26 s
FlowDroid1161.70 s1.55 s1.96 s1.35 s3.07 s
Infer1622.21 s554 ms6.24 s468 ms8.25 s
Pysa584.08 s3.88 s4.21 s3.69 s5.95 s
OpenTaint1164.60 s4.39 s4.98 s3.99 s10.7 s
Joern3928.70 s7.18 s10.1 s4.48 s127.9 s
CodeQL73410.8 s7.22 s16.7 s4.39 s161.0 s

No mean is headlined anywhere on this page. The distributions are skewed by per-invocation fixed costs and by a small number of very long tails — the slowest single invocation in this bound latency corpus is 161.0 s, against a page-wide fastest median of 123 ms — and a mean over that shape would describe neither the typical case nor the tail. The adapters span roughly 88× between the fastest and slowest medians, which is the order-of-magnitude claim this tier is entitled to make, and the strongest one it makes.

Phase decomposition, within each adapter that has one

Section titled “Phase decomposition, within each adapter that has one”

Only the 2 adapters whose declared row has phases appear here, and each is read down its own column. These numbers exist to decompose one adapter's own cost. Racing one adapter's phase against another's total is exactly the reading the granularity rule forbids. Each adapter's numbers open from its own disclosure, one per adapter, which is also the boundary the comparison rule draws.

Inferv1.3.0

Show the data table — Infer's 2 observable phases against its analyzer total
PhaseMedianQ1Q3MinMaxShare of the median analyzer total
capture1.78 s160 ms5.87 s137 ms7.88 s80%
analyze373 ms343 ms440 ms297 ms1.06 s17%
analyzer total2.21 s554 ms6.24 s468 ms8.25 s

CodeQL2.26.4

Show the data table — CodeQL's 2 observable phases against its analyzer total
PhaseMedianQ1Q3MinMaxShare of the median analyzer total
database-create4.51 s2.18 s10.7 s1.26 s155.9 s42%
database-analyze5.58 s4.64 s8.29 s3.10 s43.5 s52%
analyzer total10.8 s7.22 s16.7 s4.39 s161.0 s

For analyzer phases, the share column is the phase median against the analyzer-total median, and the shares need not sum to 100%: a median is not additive, and the phase whose median is largest is not necessarily the phase that dominated any particular case. Materialization phases are labelled and excluded rather than made to look comparable. The analyzer-total row is the distribution of per-case contract-selected phase sums, computed per case and then summarized — not the sum of the phase medians.

Everything above is cold per-invocation wall-clock, and it stays the headline of this tier. It is also what this benchmark actually runs: one analyzer process per case, start-up inside the number, because start-up is not observable from inside a single invocation. Nothing in this section adjusts, corrects, or replaces one number above it.

But the ranked chart sets a native binary, a Python CLI, and five JVM or JVM-fronted engines on one axis, and read across runtimes those bars overstate the steady-state gap: a JVM row carries a JVM start that a long-lived deployment of the same engine pays once. The honest response is not to estimate that start-up and subtract it — that would be exactly the after-the-fact adjustment this tier's contract refuses. It is to measure the other quantity directly, which is what this section reports, under Amendment A15, preregistered before the first warm number existed.

The method, in one sentence: run k cases through one tool process for increasing k, and report the slope of batch wall-clock against k — the cost of one more case in a process that has already paid its start-up. A slope, never an average: an average per case at k still carries a 1/k share of the fixed cost, which is the very quantity the figure exists to remove. The batch reuses the cold runner's own case selection, endpoint resolution, workspace materialization, and query logic, so what is timed is the same work; only the number of cases sharing one process differs.

AdapterKernelCold median, same casesWarm marginal (least squares)Warm marginal (endpoint)BatchesRepeatsFitted fixed cost
Joernjava15.6 s996 ms – 1.01 s996 ms – 1.00 s k = 1, 2, 4, 8, 1623.41 s – 3.59 s
Semgrep CEjava1.16 s74 ms – 77 ms78 ms – 81 ms k = 1, 2, 4, 8, 122986 ms – 1.06 s

The cold column is the median over exactly the cases the warm batch analyzed, not over the whole kernel: two figures compared across different populations would mix a population effect into a start-up effect. The two columns are never subtracted. The "fitted fixed cost" is the same fit's intercept, published as the descriptive estimate it is — it is not measured, it is not a decomposition of any cold number, and nothing on this site subtracts it from one.

What makes these figures checkable. Each batch retains the per-case evidence it produced, beside the series above. Every document the largest Joern batch produced matches that case's retained cold evidence field for field — same analyzed state, same method count, same source and sink node counts, same flow count — differing only in the scratch path each run recorded for its own workspace. A batch that had quietly analyzed less would be visible there rather than merely fast. Every figure above is a range, and that is deliberate. The whole batch series is measured more than once, every repeat is retained, and what is published is the span the repeats cover. A single slope over a handful of batches on a developer machine has a precision, and there were only two other ways to give it one — publish one run and hide the spread, which understates it, or gate publication on an agreement tolerance, which means choosing that tolerance after the numbers exist, precisely the after-the-fact decision this tier's contract refuses. The range needs neither: its width is the precision, the reader sees it directly, and nothing between the measurement and this table is discretionary. The repeats are never averaged — that would turn repeated trials into the statistic the tier's non-goals rule out — and never picked between.

What these figures replaced

The figures above are not the first this page published. Amendment A15 published a warm marginal as a point estimate, under a rule that ran each measurement twice and withheld the figure unless the two runs "agreed closely". Amendment A21 supersedes both the rule and the numbers it produced. Those numbers are restated here rather than removed, because a retired figure is only auditable if it is still legible beside the one that replaced it.

AdapterA15 publishedA21 publishesRetired artifact
Joern1.55 s996 ms – 1.01 sreports/raw/warm-latency/superseded-a15/joern-java-kernel/warm-latency.json
Semgrep CEwithheld74 ms – 77 msreports/raw/warm-latency/semgrep-java-stability-probe/

Why the rule was replaced. "Agree closely" named no tolerance, and the only moment at which one could have been chosen was after the spreads were known — the after-the-fact decision this tier's contract refuses. It was found by measuring again: two repeats on a quiet machine disagreed by 12% where two under load had agreed to 5%, and no tolerance could have been written down at that point without being fitted to spreads already seen. Publishing the interval that was measured needs no tolerance at all.

Why Joern's number moved, and why Semgrep's exists at all. Machine conditions. Both runs recorded the one-minute load average before every batch, so the comparison is read off the artifacts rather than recalled: Joern at 9.1 to 9.5 when A15 measured it, against 2.0 to 3.9 for the figure above. That is most of the difference in Joern's row. It is the whole of Semgrep's: A15 withheld a Semgrep figure because its two runs' slopes differed by a factor of two, and recorded that as a property of the measurement's subject. Re-measured on an idle machine, Semgrep's two repeats agree to about 5% — tighter than Joern's. The instability was the machine, not the engine, and attributing it to Semgrep was a mistake, recorded here rather than quietly dropped. It is also why the observed load now travels with every figure on this page.

Worth stating plainly, since this benchmark is published by the vendor of one of the engines on it: both corrections move numbers against that vendor. Joern's marginal falls by about a third, and a competitor's withheld figure is restored — and turns out to be the fastest warm marginal here. The rule was replaced because it was unsound, not because of which way it pointed.

Show the batch series — Joern on the java kernel, all 2 retained repeats
Cases in the batch (k)Run 1 — wall-clock of the one processRun 2 — wall-clock of the one processAverage per case at run 2 (not the reported figure)
14.67 s4.35 s4.35 s
25.47 s5.43 s2.72 s
47.48 s7.51 s1.88 s
812.0 s11.3 s1.41 s
1619.6 s19.4 s1.21 s
slope (least squares)1.01 s996 ms

The average-per-case column is shown precisely so it can be discarded: it falls with k because it is still paying a shrinking share of the fixed cost, and it would keep falling with a larger k without ever being the marginal cost. The reported figure is the slope of a wall-clock column against the first, and the published figure is the range those slopes span. Measured on Mac16,1, macos25.6.0, 10 logical CPUs — the same stamp rule as every other number here, and not comparable across machines. Machine conditions: the one-minute load average sampled immediately before each batch ranged 2.0 to 3.9 across every batch of every repeat, on a 10-core machine. The tier's hygiene rule is that no other analyzer competes for the machine while one is timed, and background load is reported rather than asserted away — so the reader can weigh the conditions instead of taking "quiet machine" on trust.

Show the batch series — Semgrep CE on the java kernel, all 2 retained repeats
Cases in the batch (k)Run 1 — wall-clock of the one processRun 2 — wall-clock of the one processAverage per case at run 2 (not the reported figure)
11.06 s1.06 s1.06 s
21.14 s1.31 s656 ms
41.30 s1.32 s330 ms
81.60 s1.64 s205 ms
121.92 s1.95 s163 ms
slope (least squares)77 ms74 ms

The average-per-case column is shown precisely so it can be discarded: it falls with k because it is still paying a shrinking share of the fixed cost, and it would keep falling with a larger k without ever being the marginal cost. The reported figure is the slope of a wall-clock column against the first, and the published figure is the range those slopes span. Population:one `semgrep scan` carries one --config, so a batch is restricted to cases resolving to identical rule text: 12 of the 14 invocable Java kernel assertions. Measured on Mac16,1, macos25.6.0, 10 logical CPUs — the same stamp rule as every other number here, and not comparable across machines. Machine conditions: the one-minute load average sampled immediately before each batch ranged 4.1 to 4.3 across every batch of every repeat, on a 10-core machine. The tier's hygiene rule is that no other analyzer competes for the machine while one is timed, and background load is reported rather than asserted away — so the reader can weigh the conditions instead of taking "quiet machine" on trust.

Which adapters this could be measured on, and why not the rest

Each verdict below was reached by interrogating the pinned distribution — its help output, and where that was ambiguous its own bytecode — never from a README or an assumption about the runtime. No adapter is patched, forked, or invoked outside its released interface to make a batch possible. An adapter that ships no way to analyze several cases in one process has no warm figure here, and none is estimated for it from its runtime or from another adapter's slope.

AdapterObservable with the released CLI?Evidence
JoernYes — measured A released multi-case batch that does the same per-case work, timed above.
Semgrep CEYes — measured A released multi-case batch that does the same per-case work, timed above. one `semgrep scan` carries one --config, so a batch is restricted to cases resolving to identical rule text: 12 of the 14 invocable Java kernel assertions.
FlowDroidYes in the CLI — not measured hereThe released CLI does have a batch: `-a/--apkfile` accepts a directory, and the shipped `soot-infoflow-cmd` main class lists its APKs, builds the taint wrapper once outside the loop, and iterates them in one JVM — its own help documents `-si/--skipapkfile` as "APK file to skip when processing a directory of input files", and it refuses a non-directory output with "The output file must be a directory when analyzing multiple APKs". It is not measured here because one invocation carries one `-s` sources-and-sinks definition, so a k-APK batch runs a union of k per-case endpoint configurations rather than each case's own. Whether that changes any case's result is an empirical question that has to be answered across the whole population before a marginal derived from it may be published. Named follow-up work, not a decline.
OpenTaintNot observable with the released CLIThe pinned analyzer takes one `--project` and one `--output-dir` and exits after that project. A `project.yaml` may list several `javaProjects`, but analyzing their union is one whole-program analysis over a merged call graph and a merged entry-point set — different work, not k independent case analyses — and `--semgrep-rule-set` is one rule set for the whole invocation while the benchmark resolves a rule per case. No released mode processes separate case projects in one process.
PysaNot observable with the released CLI`pyre analyze` is one-shot. The client does expose daemon commands (`start`, `incremental`, `query`), but they serve the type checker, not the taint analysis, and `analyze` never attaches to a running server. `--source-directory` is repeatable but merges directories into one project — again one whole-program analysis rather than k — and each case carries its own `.pyre_configuration`, `pyrefly.toml`, and resolved models. A daemon-shaped measurement would be stateful and not reproducibly preregisterable against this pin, so it is declined rather than attempted.
CodeQLNot observable with the released CLI`codeql database analyze` takes exactly one mandatory database, and `database create` produces exactly one database per invocation. Neither subcommand has a multi-database form in the pinned CLI.
InferNot observable with the released CLI`--results-dir` names one capture database for one project and the analyzer exits after it. Worth stating explicitly: Infer's analyzer is a native binary, and the JVM cost inside its Java row is the traced `javac` in `capture` — per-project compilation work, not process start-up a batch could amortize.
BifrostNot observable with the released CLIThe policy CLI takes one `--root` per invocation; the repeatable `--workspace NAME=PATH` is documented as requiring `--mcp` and does not reach the policy path. What can be said without measuring anything is a bound: Bifrost's cold median already includes its own process start, so its warm marginal lies between zero and that cold number. Warm figures can therefore only move the other rows down toward Bifrost's, never Bifrost's row down further — the asymmetry this amendment corrects is one the publishing vendor's engine loses by.

The decline that matters most is Bifrost's, and it is stated in the direction that costs this benchmark's publisher something. Bifrost has no warm figure here, and its cold median already includes its own process start — so whatever its warm marginal is, it lies between zero and a number already published. Adding warm figures can therefore only move the other rows down toward Bifrost's, never Bifrost's row down further. The conflation this section corrects is one the vendor's own engine benefits from, which is the reason to correct it in public rather than leave the cold chart to be read as a steady-state ranking.

These artifacts are auxiliary evidence, outside the freeze. They are retained under reports/raw/warm-latency/ — the batch series, the environment stamp, and the per-case evidence each batch produced, so that a reader can check the batch did the real work rather than less of it — but freeze/v1 does not digest them, and this amendment does not extend it to. A warm number carries the pinned amendment commit ccbcd788aabe…'s immutability for its bytes and no stronger guarantee, exactly as the cold timing sidecars do. No warm run wrote a normalized report, produced an outcome, or touched a scored population; validate-reports and the freeze manifest never read this directory.

The warm marginal above answers what one more case costs a process already running, and only one adapter's released CLI let it be measured. This section answers a different question that every adapter can be asked: what does one invocation cost before it has anything to find? It is published under Amendment A24, preregistered — estimator, bias, tolerance, fixtures and presentation — before the first estimate existed.

The method, in one sentence: run one complete adapter invocation — same pipeline, same committed policy, rule or query, same flags, same subprocess shape, both subprocesses where the adapter has two — over a trivial no-flow fixture that declares the benchmark's own source and sink endpoints and never connects them, and take the runner's wall-clock around it. The fixture is generated into a scratch workspace before the process is spawned, exactly as every other fixture is, and is retained beside the measurement. Nothing is added to cases/: no population, denominator or freeze sees these files, and they are not cases.

What the number is biased by, in both directions. A trivial file is still parsed, still extracted, still queried, so the measurement is fixed overhead plus that near-zero analysis: it is an upper bound on start-up and warm-up, and the true fixed cost is at or below it. And it is a cold, single-shot execution — no warm JIT, no primed cache — which is exactly the posture the cold rows above are measured in, and exactly not the posture of a resident deployment, where both this figure and those rows would fall. It is an estimate, labelled one everywhere it appears, and it is never subtracted from a cold number, never substituted for one, and never used to order a row.

AdapterFixture languageEstimated overhead (range over every repeat)WidthEvery repeatCold median, same kernelLow end as a share of coldLoad observedMarked on the chart?
Bifrostpython108 ms – 239 ms131 ms239 ms, 114 ms, 108 ms115 ms94%3.4yes
CodeQLruby4.02 s – 4.49 s475 ms4.49 s, 4.04 s, 4.02 s4.70 s86%3.6 to 3.8yes
FlowDroidjava678 ms – 771 ms93 ms771 ms, 768 ms, 678 ms1.55 s44%3.4 to 3.5yes
Inferc288 ms – 491 ms203 ms491 ms, 297 ms, 288 ms540 ms53%3.3yes
Joernjava8.91 s – 13.2 s4.33 s8.91 s, 9.50 s, 13.2 s14.6 s61%3.7 to 3.9yes
Joernphp4.42 s – 5.34 s911 ms5.34 s, 4.45 s, 4.42 s5.45 s81%3.8 to 4.6yes
OpenTaintkotlin4.08 s – 4.14 s60 ms4.14 s, 4.12 s, 4.08 s4.46 s91%3.7 to 4.2yes
Pysapython2.82 s – 2.98 s165 ms2.82 s, 2.88 s, 2.98 s4.08 s69%3.5 to 3.6yes
Semgrep CEkotlin1.04 s – 1.07 s28 ms1.07 s, 1.07 s, 1.04 s1.09 s95%3.3yes

The published figure is a range, and its width is the measurement's precision. Each estimate is repeated a fixed number of times — the count is a constant in the runner's source, not a per-run choice — every repeat is retained and printed above, and what is published is the range those repeats span. Never a mean, never a chosen repeat, and never withheld because the repeats disagreed: a disagreement widens the range, which is the honest consequence of it, and there is no agreement threshold anywhere in this machinery to be justified or tuned — a unit test asserts that no such constant exists. This is the same convention the warm-marginal figures publish under, shared rather than re-derived. The build re-derives each range from the retained repeats and fails if it disagrees with what the runner wrote.

The last column is what decides a chart mark, and the threshold is preregistered. A row carries a dashed span only where its range starts at or above 25% of that adapter's cold median on the same kernel. The cut is relative because the chart's axis is logarithmic and its rows span two orders of magnitude, so a share of each row's own median is the same visual claim everywhere, where a fixed millisecond cut would mark every slow adapter and no fast one whatever its overhead actually was; and it reads the range's low end so that a mark can never appear on the strength of one slow repeat. An unmarked row is not an unmeasured one — every value is in the table above.

The load column is the machine's own condition, published rather than summarized as the word "quiet". Each repeat samples the one-minute load average immediately before its subprocess is spawned, and the column is the range those samples span, so a figure taken on a busy machine can be discounted instead of taken on trust. Two disciplines stand behind those numbers: no other analyzer under measurement competes for the machine, and — because nine heavy analyzers run back to back drive the load up by themselves — each adapter's measurement waits for the load to settle before it begins, so what a row records is the machine's state and not its own position in the sequence.

Joern is the one adapter where three figures can be set side by side, and they are three different things. Its estimated per-invocation overhead on the Java kernel is 8.91 s – 13.2 s — a measured upper bound, published across the range its repeats spanned. The fitted fixed cost, the intercept of the warm batch series on the same kernel, is 3.41 s – 3.59 s — not measured at all, but inferred from a line through five batches. And its measured warm marginal, the cost of one more case in a process already running, is 996 ms – 1.01 s. The first two are two routes to the same quantity and the third is a different quantity entirely.

Those first two routes disagree, by roughly a factor of three, and the disagreement is the informative part. Neither corrects the other and their difference is not published as a measurement, but the direction is exactly what the two constructions predict. The estimate is an upper bound that contains work the intercept excludes by construction: a whole trivial invocation — a fresh JVM, the Java front end loaded, a CPG built for the fixture, and the kernel script run over it — where the fitted intercept is what is left when a line through five batch sizes is extended back to zero cases, and every per-case cost the batch pays has been taken out of it. So the true once-per-process cost is bounded above by the estimate and approached from below by the intercept, and the gap between them is the part of "fixed cost" that neither construction can see on its own: not observable from inside one invocation, and not separable from per-case work inside a batch. That is the honest reading, and it is why this page publishes both figures rather than reconciling them into one.

Which adapters an estimate could be taken for

Each adapter is estimated in the language of its cheapest kernel arm — the core kernel whose cold median is that adapter's lowest — because that is the arm where fixed cost is the largest share of the number, and the arm whose trivial-fixture invocation is least dominated by analysis. Joern is additionally estimated on Java, so that A21's Java warm figures have a same-language estimate to be compared against; that second figure is labelled by its own language and is not the one the cheapest-arm rule selects.

AdapterFixture languageVerdictEvidence
Bifrostpython
its cheapest kernel arm
Measured3 repeats — 239 ms, 114 ms, 108 ms — published as the 108 ms to 239 ms range they span.
CodeQLruby
its cheapest kernel arm
Measured3 repeats — 4.49 s, 4.04 s, 4.02 s — published as the 4.02 s to 4.49 s range they span.
FlowDroidjava
its cheapest kernel arm
Measured3 repeats — 771 ms, 768 ms, 678 ms — published as the 678 ms to 771 ms range they span.
Inferc
its cheapest kernel arm
Measured3 repeats — 491 ms, 297 ms, 288 ms — published as the 288 ms to 491 ms range they span.
Joernphp
its cheapest kernel arm
Measured3 repeats — 5.34 s, 4.45 s, 4.42 s — published as the 4.42 s to 5.34 s range they span.
OpenTaintkotlin
its cheapest kernel arm
Measured3 repeats — 4.14 s, 4.12 s, 4.08 s — published as the 4.08 s to 4.14 s range they span.
Pysapython
its only kernel arm
Measured3 repeats — 2.82 s, 2.88 s, 2.98 s — published as the 2.82 s to 2.98 s range they span.
Semgrep CEkotlin
its cheapest kernel arm
Measured3 repeats — 1.07 s, 1.07 s, 1.04 s — published as the 1.04 s to 1.07 s range they span.
Joernjava
not its cheapest arm — measured so that A21's Java warm figures have a same-language estimate to be compared against
Measured3 repeats — 8.91 s, 9.50 s, 13.2 s — published as the 8.91 s to 13.2 s range they span.

There is exactly one kind of missing figure here, and it is a fact about a machine.Environment means the pinned distribution is not installed where the estimator ran, so the invocation was never attempted — explicitly not a statement about the adapter's released CLI, unlike A15's capability declines above, and resolved by running the same committed command where the distribution is installed. There is no second kind: because the figure is a range over every repeat, noisy repeats produce a wide range rather than a withheld number, so nothing is ever missing for having been measured badly. Nothing is filled in by inference either — no estimate here is derived from an adapter's runtime, its architecture, another adapter's estimate, or the same adapter's estimate in another language.

These artifacts are auxiliary evidence, outside the freeze, retained under reports/raw/invocation-overhead/: every repeat with its phase split and the one-minute load average it was taken under, the trivial fixture itself with its digest, the resolved configuration where the adapter's is per-case, the published range, and the environment stamp. freeze/v1 does not digest them, exactly as it does not digest the warm artifacts or the cold timing sidecars, so an estimate carries the pinned amendment commit ccbcd788aabe…'s immutability for its bytes and no stronger guarantee. No estimator run wrote a normalized report, produced an outcome, or touched a scored population.

The tier's declared unit of aggregation: one row per slice — adapter × language × population — because a median over a whole adapter mixes languages whose fixtures differ in size and whose front ends differ in cost. Each row is one bound report. Rows are not ranked and columns are not compared across adapters with different granularity. One disclosure per adapter, for the same reason: the rows inside one of them are the rows that share a granularity.

Bifrost — 20 slices

Show the data table — 20 slices for Bifrost
SliceLanguageProfileTimed / boundMedianQ1Q3
bifrost-smoke13 languagesbenchmark-controlled117 / 118117 ms113 ms123 ms
bifrost-c-kernelcbenchmark-controlled50 / 50131 ms124 ms140 ms
bifrost-cpp-kernelcppbenchmark-controlled56 / 56130 ms127 ms136 ms
bifrost-csharp-kernelcsharpbenchmark-controlled58 / 58141 ms130 ms155 ms
bifrost-go-kernelgobenchmark-controlled58 / 58123 ms116 ms128 ms
bifrost-java-kerneljavabenchmark-controlled58 / 58120 ms114 ms125 ms
bifrost-java-modelingjavabenchmark-controlled8 / 24144 ms126 ms151 ms
bifrost-java-nativejavatool-native0 / 12n/an/an/a
bifrost-javascript-kerneljavascriptbenchmark-controlled58 / 58117 ms112 ms121 ms
bifrost-javascript-modelingjavascriptbenchmark-controlled8 / 24196 ms171 ms203 ms
bifrost-javascript-nativejavascripttool-native0 / 12n/an/an/a
bifrost-kotlin-kernelkotlinbenchmark-controlled58 / 58122 ms119 ms128 ms
bifrost-php-kernelphpbenchmark-controlled58 / 58122 ms116 ms129 ms
bifrost-python-kernelpythonbenchmark-controlled58 / 58115 ms110 ms121 ms
bifrost-python-modelingpythonbenchmark-controlled8 / 24181 ms141 ms289 ms
bifrost-python-nativepythontool-native0 / 12n/an/an/a
bifrost-ruby-kernelrubybenchmark-controlled58 / 58122 ms117 ms125 ms
bifrost-rust-kernelrustbenchmark-controlled56 / 56121 ms117 ms128 ms
bifrost-scala-kernelscalabenchmark-controlled58 / 58133 ms125 ms147 ms
bifrost-typescript-kerneltypescriptbenchmark-controlled58 / 58144 ms128 ms163 ms

Semgrep CE — 17 slices

Show the data table — 17 slices for Semgrep CE
SliceLanguageProfileTimed / boundMedianQ1Q3
semgrep-c-kernelcbenchmark-controlled14 / 481.28 s1.21 s1.33 s
semgrep-cpp-kernelcppbenchmark-controlled14 / 561.12 s1.10 s1.13 s
semgrep-go-kernelgobenchmark-controlled14 / 581.11 s1.09 s1.15 s
semgrep-java-kerneljavabenchmark-controlled14 / 581.16 s1.13 s1.22 s
semgrep-java-modelingjavabenchmark-controlled10 / 241.12 s1.09 s1.16 s
semgrep-java-nativejavatool-native0 / 12n/an/an/a
semgrep-javascript-kerneljavascriptbenchmark-controlled14 / 581.31 s1.24 s1.33 s
semgrep-javascript-modelingjavascriptbenchmark-controlled10 / 241.14 s1.11 s1.17 s
semgrep-javascript-nativejavascripttool-native0 / 12n/an/an/a
semgrep-kotlin-kernelkotlinbenchmark-controlled14 / 581.09 s1.08 s1.58 s
semgrep-php-kernelphpbenchmark-controlled14 / 581.69 s1.61 s1.82 s
semgrep-python-kernelpythonbenchmark-controlled14 / 581.10 s1.09 s1.12 s
semgrep-python-modelingpythonbenchmark-controlled10 / 241.63 s1.18 s1.85 s
semgrep-python-nativepythontool-native12 / 124.10 s3.88 s4.20 s
semgrep-ruby-kernelrubybenchmark-controlled14 / 582.53 s2.37 s2.73 s
semgrep-rust-kernelrustbenchmark-controlled14 / 541.35 s1.22 s1.52 s
semgrep-typescript-kerneltypescriptbenchmark-controlled14 / 581.11 s1.09 s1.18 s

FlowDroid — 2 slices

Show the data table — 2 slices for FlowDroid
SliceLanguageProfileTimed / boundMedianQ1Q3
flowdroid-java-kerneljavabenchmark-controlled58 / 581.55 s1.49 s1.61 s
flowdroid-kotlin-kernelkotlinbenchmark-controlled58 / 581.97 s1.84 s2.05 s

Infer — 3 slices

Show the data table — 3 slices for Infer
SliceLanguageProfileTimed / boundMedianQ1Q3capture median analyze median
infer-c-kernelcbenchmark-controlled48 / 48540 ms494 ms612 ms159 ms372 ms
infer-cpp-kernelcppbenchmark-controlled56 / 56728 ms550 ms2.99 s234 ms413 ms
infer-java-kerneljavabenchmark-controlled58 / 586.47 s6.15 s6.77 s6.09 s349 ms

Pysa — 1 slice

Show the data table — 1 slice for Pysa
SliceLanguageProfileTimed / boundMedianQ1Q3
pysa-python-kernelpythonbenchmark-controlled58 / 584.08 s3.88 s4.21 s

OpenTaint — 2 slices

Show the data table — 2 slices for OpenTaint
SliceLanguageProfileTimed / boundMedianQ1Q3
opentaint-java-kerneljavabenchmark-controlled58 / 584.70 s4.55 s9.25 s
opentaint-kotlin-kernelkotlinbenchmark-controlled58 / 584.46 s4.26 s4.74 s

Joern — 12 slices

Show the data table — 12 slices for Joern
SliceLanguageProfileTimed / boundMedianQ1Q3
joern-java-kerneljavabenchmark-controlled58 / 5814.6 s14.3 s15.3 s
joern-java-modelingjavabenchmark-controlled16 / 249.29 s8.95 s9.91 s
joern-java-nativejavatool-native0 / 12n/an/an/a
joern-javascript-kerneljavascriptbenchmark-controlled58 / 589.15 s8.86 s9.61 s
joern-javascript-modelingjavascriptbenchmark-controlled16 / 245.28 s5.07 s5.40 s
joern-javascript-nativejavascripttool-native0 / 12n/an/an/a
joern-php-kernelphpbenchmark-controlled58 / 585.45 s5.12 s8.39 s
joern-python-kernelpythonbenchmark-controlled58 / 588.38 s8.00 s8.86 s
joern-python-modelingpythonbenchmark-controlled16 / 244.95 s4.89 s5.03 s
joern-python-nativepythontool-native0 / 12n/an/an/a
joern-ruby-kernelrubybenchmark-controlled58 / 588.63 s7.35 s11.0 s
joern-rust-kernelrustbenchmark-controlled54 / 547.41 s7.05 s8.31 s

CodeQL — 17 slices

Show the data table — 17 slices for CodeQL
SliceLanguageProfileTimed / boundMedianQ1Q3database-create median database-analyze median
codeql-c-kernelcbenchmark-controlled50 / 506.15 s5.42 s7.08 s1.55 s4.62 s
codeql-cpp-kernelcppbenchmark-controlled56 / 567.28 s6.19 s19.1 s1.89 s5.40 s
codeql-csharp-kernelcsharpbenchmark-controlled58 / 5833.1 s24.1 s39.7 s20.1 s13.0 s
codeql-go-kernelgobenchmark-controlled58 / 589.43 s9.24 s9.81 s4.40 s5.05 s
codeql-java-kerneljavabenchmark-controlled58 / 5812.7 s12.5 s12.9 s6.70 s6.00 s
codeql-java-modelingjavabenchmark-controlled24 / 2412.3 s11.2 s13.2 s6.40 s5.58 s
codeql-java-nativejavatool-native12 / 1222.6 s17.3 s25.9 s7.77 s14.8 s
codeql-javascript-kerneljavascriptbenchmark-controlled58 / 5810.7 s9.80 s13.5 s4.49 s6.12 s
codeql-javascript-modelingjavascriptbenchmark-controlled24 / 2413.3 s8.82 s13.8 s5.59 s7.76 s
codeql-javascript-nativejavascripttool-native12 / 1221.7 s21.4 s22.0 s5.68 s16.1 s
codeql-kotlin-kernelkotlinbenchmark-controlled58 / 5815.8 s15.6 s17.0 s10.9 s4.85 s
codeql-python-kernelpythonbenchmark-controlled58 / 585.38 s5.30 s5.83 s2.00 s3.41 s
codeql-python-modelingpythonbenchmark-controlled24 / 249.85 s9.72 s9.98 s3.66 s6.15 s
codeql-python-nativepythontool-native12 / 1212.6 s12.4 s12.9 s3.70 s8.86 s
codeql-ruby-kernelrubybenchmark-controlled58 / 584.70 s4.58 s4.89 s1.34 s3.33 s
codeql-rust-kernelrustbenchmark-controlled56 / 5647.1 s45.5 s49.2 s20.9 s25.7 s
codeql-typescript-kerneltypescriptbenchmark-controlled58 / 588.92 s8.09 s9.80 s3.73 s5.04 s

Of the case results in this bound latency corpus, 2657 invoked an analyzer and carry a timing sidecar. 679 do not, and every one of them is accounted for:

Outcome of the untimed caseCases
unsupported679

An unsupported case is decided from case metadata before the analyzer is invoked, so there is no subprocess and there is nothing to time. Those cases are absent from every distribution on this page, not entered as zero — entering them as zero would make the adapters that decline the most look the fastest, which is exactly backwards. Every case that did invoke an analyzer is timed; the timed and untimed columns account for the whole bound population with no remainder.

Provenance and what the freeze does and does not guarantee

Section titled “Provenance and what the freeze does and does not guarantee”

Every cold number above was derived at build time from v0.6.0's archived latency-evidence bundle. The bundle was generated from that release commit's retained timing sidecars, and its population comes from the results model generated by freeze manifest 3228af686d09…. Snapshot pages select this bundle explicitly: advancing the current correctness freeze cannot add a case, replace a timing, or relabel an analyzer on a historical page. The warm and invocation-overhead amendments are likewise read from a checked-in auxiliary bundle pinned to commit ccbcd788aabec2abe60200573f38bc42128d00f0; they never read the mutable working-tree artifact directories.

One limit, stated rather than left to be assumed. The freeze/v1 manifest binds one raw-evidence digest per result; the timing sidecar and the environment stamp are additive files beside that evidence and are not themselves digested by the manifest. So a number on this page carries the freeze's guarantee that the run it belongs to is bound and byte-verified, and the release commit plus the checked-in archive's own Git history preserve the timing bytes selected for rendering. It does not carry a freeze-manifest digest over those timing bytes. Extending the manifest to bind them is a freeze/v2 question, and it is named here rather than smuggled in under a schema version that does not describe it.