Latency — v0.7.1

DataFlowBench’s latency-characterization tier stands beside the correctness scorecards, on its own page, and never as a column inside one. This release retains 2,725 cold timings across the unchanged population. The 755 unsupported outcomes have no invented timing.

Characterization on one desktop machine

The cold rows describe the runs this freeze actually performed. Desktop activity and cache state were observed but uncontrolled, so the figures do not support an uncontended-machine claim. The Bifrost access controls establish equal outcomes for their paired sandboxed/elevated controls, not timing parity.

The contract governing every number here — docs/latency-tier.md — keeps latency descriptive and independent from correctness. No outcome was derived from, conditioned on, or tie-broken by a timing value.

The warm series retain Joern batches 1, 2, 4, 8, 16, with two repeats, and Semgrep batches 1, 2, 4, 8, 12, with two repeats. Semgrep’s original batch size-16 attempt failed during pre-analysis because its eligible identical-rule population contained twelve cases; that attempt is retained, and no size-16 comparison is claimed. Nine overhead groups were measured three times each.

OpenTaint’s functional product-probe timings are diagnostic and excluded from cold, warm, and overhead qualification. The measurement audit and warm-series audit retain the execution boundaries and limitations.

What these numbers are, and what they are not

This is a descriptive characterization of per-case analyzer wall-clock, published under the contract preregistered in docs/latency-tier.md, which was merged before a single timestamp was captured. It is not a score, not a ranking, and not a benchmark result in the sense the rest of this site uses that phrase.

• Latency is never pooled with correctness. There is no combined number anywhere on this site, no efficiency-adjusted rate, and no leaderboard that blends the two. "Correct but slow" and "fast but wrong" stay independently visible. No correctness outcome in this freeze was derived from, conditioned on, or tie-broken by a timing value.
• A latency number is a property of a run, not of a case. No case carries a timing assertion, threshold, or budget.
• Characterization, not micro-benchmark. There are no repeated trials, no warm-up iterations, and no steady-state harness in the cold figures below. The case population is the sample and its spread is the statistics. The distribution columns below are that spread — not error bars on a measurement that was repeated, because it was not. (The warm-marginal section is the one place a series is measured more than once. It says so, retains every repeat, and publishes the range they span rather than averaging them into a statistic.)
• These are the conditions the numbers were produced under: a single developer machine, under light concurrent load, running the benchmark's standing sequential-run discipline — one analyzer at a time, never two at once. That discipline is what makes the numbers usable at all; the light concurrent load is what stops them being a controlled measurement. Read them as characterization of what this benchmark actually costs to run, at the order-of-magnitude and shape level, and not as a precise figure for any engine.

Environment stamp

Recorded once per run beside the tool identity the run witnessed, and shown here verbatim. Latency numbers are comparable within one environment and are not comparable across machines. Every published run to date executes on one maintainer machine, and that caveat travels with every number on this page.

Hardware model	OS	OS release	Logical CPUs	CPU architecture	Timed runs stamped

The granularity is unequal, and that is the point

A phase is timed if and only if the adapter already invokes it as a separate subprocess. The benchmark never instruments analyzer internals, never patches an upstream tool, and never infers a boundary a subprocess boundary does not expose. The adapters do not expose the same boundaries, so the rows below are not equally decomposed — and the two ways of papering over that would both misdescribe the evidence: summing a decomposed adapter's phases and presenting every row as one number of the same kind, or guessing phase splits for the adapters that expose only one.

So the table states each adapter's declared granularity as data. Phases compare within an adapter; analyzer-invocation totals compare across adapters. A CodeQL database-create number and a Joern total are not the same kind of quantity, and nothing on this page sets them side by side as though they were.

Adapter	Version	Observed subprocesses per case	Phases timed	Timed invocations
Bifrost	0.11.0	0 or 1	total	883
FlowDroid	2.15.1	0 or 1 or 3	total compile dex analyze	130
Infer	v1.3.0	0 or 2	capture analyze	172
Semgrep CE	1.176.0	0 or 1	total	196
Pysa	0.10.0	1	total	90
OpenTaint	v0.4.6	0 or 1	total	128
Joern	4.0.621	0 or 1	total	392
CodeQL	2.26.4	2	database-create database-analyze	734

3 of 8 adapters expose more than one subprocess and are decomposed here (FlowDroid and Infer and CodeQL); the other 5 expose a single invocation and take one honestly labelled number, with interpreter or JVM start-up inside it. That is a fact about invocation shapes, not about engine architecture: an adapter with one number is not an adapter that does one thing.

The phase names above are the ones the retained evidence carries, so any name here can be found in the artifact it came from. For CodeQL those are database-create and database-analyze, after the subcommands that produce them, where the preregistration's table spells the same two boundaries extract and analyze. Two spellings, one pair of subprocess boundaries: nothing moved and nothing is attributed differently. The contract is immutable, so the difference is recorded in its Amendment A12 rather than edited out of its table.

Analyzer-invocation wall-clock, per adapter

The only cross-adapter reading this tier supports: each adapter's cold per-case analyzer-invocation wall-clock, over every case in this snapshot's explicitly bound latency population that it actually invoked. Adapter-observable setup phases remain visible below, but only phases the contract classifies as analyzer work enter this total. Per-case timing at this granularity includes per-invocation fixed costs — JVM start-up, extractor initialization, interpreter start — that a long-lived deployment of the same engine would amortize. This tier does not correct for that; it characterizes what the benchmark actually runs.

Fastest to slowest, and the same ranking per kernel

Ordered by median, on a logarithmic axis, with the spread drawn beside every median rather than left to the table. The toggle scopes the same ranking to one language kernel at a time: a whole-corpus median mixes the languages an adapter runs on, and holding the language fixed is the difference between "this engine is slow" and "this front end is expensive". Only the analyzers that actually invoked on a kernel appear in its view. The chart draws the median, the quartiles and the p10–p90 spread; the disclosure beneath it opens the same distributions as a table, with the minima and maxima the whiskers deliberately leave out.

All timed invocationsccppcsharpgojavajavascriptkotlinphppythonrubyrustscalatypescript

Every timed analyzer invocation the freeze binds — 2725 of them, across every score tier and both model profiles. This is the widest denominator on the site and the only one here that is not a single population: an adapter's median mixes the languages it runs on, whose fixtures differ in size and whose front ends differ in cost. The per-kernel views hold the language fixed.

The c kernel's 48 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 48, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The cpp kernel's 56 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 56, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The csharp kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 2 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The go kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 3 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The java kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 7 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The javascript kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The kotlin kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 5 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The php kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 3 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The python kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 5 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The ruby kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The rust kernel's 54 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 4 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 54, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The scala kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 1 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

The typescript kernel's 58 benchmark-controlled core assertions, the same no-pooling population the correctness sections read. 3 of the 8 analyzers invoked here at all; the rest do not cover this kernel and are absent from this view rather than drawn at zero. Where a row's count is below 58, the remaining assertions were declined before any subprocess was spawned and so have nothing to time.

• median, printed beside every row
• interquartile range (Q1–Q3)
• p10–p90; the minimum and maximum are in the tables, not the whiskers
• indented rows: an adapter's own declared phases
• measured warm marginal per case — the range its retained repeats span
• estimated per-invocation overhead (trivial fixture, upper bound), drawn across the range its repeats spanned, where that range starts at or above 25% of the row's median

How to read this figure

Every bar above is cold per-invocation wall-clock, and the warm marginal is a different quantity measured separately. Cold is what this benchmark actually runs — one process per case, start-up inside the number, because start-up is not observable from inside a single invocation. Read across runtimes, though, those bars overstate the steady-state gap: a JVM engine's row carries a JVM start a long-lived deployment pays once. So the other quantity is measured directly rather than estimated and subtracted — k cases through one tool process, for increasing k, reporting the slope of batch wall-clock against k. The carets mark it on the rows that have one, and they mark a range: the whole series is measured more than once and the figure published is the span its repeats cover, so the mark's width is its own precision rather than a number stated more exactly than it was measured. It is never subtracted from a median, never substituted for one, and never used to order the rows. Only adapters whose released CLI exposes a real multi-case batch have a figure at all; the rest are not observable with the released CLI, and the warm-marginal section records every verdict, measured and declined, with the evidence behind it.

The dashed spans are estimates, not measurements of the same kind as the bars. Each is the wall-clock of one complete adapter invocation — same pipeline, same policy, rule or query, same subprocess shape — over a trivial no-flow fixture: both benchmark endpoints declared, nothing connecting them, nothing to find. That is fixed per-invocation overhead plus the trivial fixture's own near-zero analysis, so it is an upper bound on what an adapter pays before it starts work, and it is a cold single-shot execution — the same posture the bars are measured in, and not a steady-state one. It is never subtracted from a median, never substituted for one, and never used to order the rows. The width of a span is the figure's precision, not a decoration: each measurement is repeated a fixed number of times and what is published is the range those repeats spanned — never a mean, never one chosen repeat, and never withheld for repeats that disagree, because a disagreement widens the range and that is the honest consequence of it. A span is drawn only where the range starts at or above 25% of that row's own cold median — a threshold fixed in the amendment before any estimate was measured, read at the low end so that no mark can appear on the strength of one slow repeat. Each estimate is measured in one named language, stated on the row it annotates, because boot cost is not language-free. The estimates table carries every value — every repeat behind every range, every unmarked row, and every adapter for which no estimate could be taken at all.

The axis is logarithmic. Each labelled tick is three times the one before it, so equal distances are equal ratios, not equal durations — the gap from 100 ms to 300 ms is drawn the same width as the gap from 10 s to 30 s. That is the only way this bound corpus's medians fit in one picture: they span 176 ms to 8.29 s, and on a linear axis every analyzer except the slowest would be a sliver against the origin. Because a log axis is easy to misread, every median is also printed as a number at the right of its own row.

The indented rows are phases, and they are not comparable across adapters. Only the 3 adapters whose preregistered row declares more than one subprocess have them. A phase mark sits on the same axis as the totals because it is the same kind of measurement — wall-clock of a subprocess — but reading one adapter's phase against another adapter's total is precisely the comparison the granularity rule forbids. Read a phase against the adapter it is indented under, and nothing else.

Ordering is not scoring, and this is never pooled with correctness. Rows are sorted by median because an unsorted ranking is unreadable, not because latency is a result. No correctness figure appears in this chart and no number here is blended with one: there is no combined score anywhere on this site, and a fast analyzer that answers wrongly is neither rewarded nor penalised by anything drawn above. Cases an analyzer declined before invocation are absent, not entered as zero — entering them as zero would make the analyzers that decline the most look the fastest, which is exactly backwards.

Show the data table — analyzer-invocation wall-clock per adapter, with minima and maxima

Adapter	Timed invocations	Median	Q1	Q3	Min	Max
Bifrost	883	176 ms	160 ms	270 ms	141 ms	21.2 s
FlowDroid	130	802 ms	726 ms	968 ms	504 ms	1.19 s
Infer	172	1.22 s	275 ms	3.88 s	256 ms	4.37 s
Semgrep CE	196	1.30 s	1.20 s	1.46 s	1.09 s	2.96 s
Pysa	90	2.92 s	2.88 s	2.97 s	2.60 s	3.24 s
OpenTaint	128	4.33 s	4.22 s	4.46 s	3.93 s	5.31 s
Joern	392	5.56 s	4.70 s	6.71 s	4.04 s	11.4 s
CodeQL	734	8.29 s	5.76 s	16.2 s	4.35 s	84.1 s

No mean is headlined anywhere on this page. The distributions are skewed by per-invocation fixed costs and by a small number of very long tails — the slowest single invocation in this bound latency corpus is 84.1 s, against a page-wide fastest median of 176 ms — and a mean over that shape would describe neither the typical case nor the tail. The adapters span roughly 47× between the fastest and slowest medians, which is the order-of-magnitude claim this tier is entitled to make, and the strongest one it makes.

Phase decomposition, within each adapter that has one

Only the 3 adapters whose declared row has phases appear here, and each is read down its own column. These numbers exist to decompose one adapter's own cost. Racing one adapter's phase against another's total is exactly the reading the granularity rule forbids. Each adapter's numbers open from its own disclosure, one per adapter, which is also the boundary the comparison rule draws.

FlowDroid2.15.1

Show the data table — FlowDroid's 4 observable phases against its analyzer total

Phase	Median	Q1	Q3	Min	Max	Share of the median analyzer total
total	836 ms	759 ms	992 ms	645 ms	1.19 s	104%
compile — materialization	501 ms	485 ms	515 ms	446 ms	527 ms	excluded by contract
dex — materialization	330 ms	322 ms	336 ms	302 ms	358 ms	excluded by contract
analyze	576 ms	567 ms	588 ms	504 ms	615 ms	72%
analyzer total	802 ms	726 ms	968 ms	504 ms	1.19 s	—

Inferv1.3.0

Show the data table — Infer's 2 observable phases against its analyzer total

Phase	Median	Q1	Q3	Min	Max	Share of the median analyzer total
capture	1.02 s	78 ms	3.68 s	72 ms	4.13 s	84%
analyze	199 ms	196 ms	202 ms	184 ms	336 ms	16%
analyzer total	1.22 s	275 ms	3.88 s	256 ms	4.37 s	—

CodeQL2.26.4

Show the data table — CodeQL's 2 observable phases against its analyzer total

Phase	Median	Q1	Q3	Min	Max	Share of the median analyzer total
database-create	3.44 s	1.99 s	10.4 s	1.16 s	21.3 s	41%
database-analyze	4.87 s	3.73 s	5.45 s	2.69 s	80.3 s	59%
analyzer total	8.29 s	5.76 s	16.2 s	4.35 s	84.1 s	—

For analyzer phases, the share column is the phase median against the analyzer-total median, and the shares need not sum to 100%: a median is not additive, and the phase whose median is largest is not necessarily the phase that dominated any particular case. Materialization phases are labelled and excluded rather than made to look comparable. The analyzer-total row is the distribution of per-case contract-selected phase sums, computed per case and then summarized — not the sum of the phase medians.

Warm marginal cost, measured separately

Everything above is cold per-invocation wall-clock, and it stays the headline of this tier. It is also what this benchmark actually runs: one analyzer process per case, start-up inside the number, because start-up is not observable from inside a single invocation. Nothing in this section adjusts, corrects, or replaces one number above it.

But the ranked chart sets a native binary, a Python CLI, and five JVM or JVM-fronted engines on one axis, and read across runtimes those bars overstate the steady-state gap: a JVM row carries a JVM start that a long-lived deployment of the same engine pays once. The honest response is not to estimate that start-up and subtract it — that would be exactly the after-the-fact adjustment this tier's contract refuses. It is to measure the other quantity directly, which is what this section reports, under Amendment A15, preregistered before the first warm number existed.

The method, in one sentence: run k cases through one tool process for increasing k, and report the slope of batch wall-clock against k — the cost of one more case in a process that has already paid its start-up. A slope, never an average: an average per case at k still carries a 1/k share of the fixed cost, which is the very quantity the figure exists to remove. The batch reuses the cold runner's own case selection, endpoint resolution, workspace materialization, and query logic, so what is timed is the same work; only the number of cases sharing one process differs.

Adapter	Kernel	Cold median, same cases	Warm marginal (least squares)	Warm marginal (endpoint)	Batches	Repeats	Fitted fixed cost
Joern	java	8.78 s	5.06 s – 5.24 s	5.06 s – 5.24 s	k = 1, 2, 4, 8, 16	2	2.90 s – 4.29 s
Semgrep CE	java	1.28 s	74 ms – 76 ms	74 ms – 77 ms	k = 1, 2, 4, 8, 12	2	896 ms – 899 ms

The cold column is the median over exactly the cases the warm batch analyzed, not over the whole kernel: two figures compared across different populations would mix a population effect into a start-up effect. The two columns are never subtracted. The "fitted fixed cost" is the same fit's intercept, published as the descriptive estimate it is — it is not measured, it is not a decomposition of any cold number, and nothing on this site subtracts it from one.

What makes these figures checkable. Each batch retains the per-case evidence it produced, beside the series above. Every document the largest Joern batch produced matches that case's retained cold evidence field for field — same analyzed state, same method count, same source and sink node counts, same flow count — differing only in the scratch path each run recorded for its own workspace. A batch that had quietly analyzed less would be visible there rather than merely fast. Every figure above is a range, and that is deliberate. The whole batch series is measured more than once, every repeat is retained, and what is published is the span the repeats cover. A single slope over a handful of batches on a developer machine has a precision, and there were only two other ways to give it one — publish one run and hide the spread, which understates it, or gate publication on an agreement tolerance, which means choosing that tolerance after the numbers exist, precisely the after-the-fact decision this tier's contract refuses. The range needs neither: its width is the precision, the reader sees it directly, and nothing between the measurement and this table is discretionary. The repeats are never averaged — that would turn repeated trials into the statistic the tier's non-goals rule out — and never picked between.

These are fresh v0.7.1 series. Historical A15/A21 measurements remain on their original snapshot pages. Semgrep's original batch-16 attempt failed before analysis; its prospectively corrected largest batch is 12. No batch-16 comparison, idle-machine explanation, or causal cross-release speedup is inferred from these desktop observations.

Show the batch series — Joern on the java kernel, all 2 retained repeats

Cases in the batch (k)	Run 1 — wall-clock of the one process	Run 2 — wall-clock of the one process	Average per case at run 2 (not the reported figure)
1	9.29 s	8.38 s	8.38 s
2	14.7 s	13.5 s	6.76 s
4	24.2 s	23.8 s	5.95 s
8	44.8 s	44.2 s	5.52 s
16	85.2 s	86.9 s	5.43 s
slope (least squares)	5.06 s	5.24 s	—

The average-per-case column is shown precisely so it can be discarded: it falls with k because it is still paying a shrinking share of the fixed cost, and it would keep falling with a larger k without ever being the marginal cost. The reported figure is the slope of a wall-clock column against the first, and the published figure is the range those slopes span. Measured on Mac16,1, macos25.6.0, 10 logical CPUs — the same stamp rule as every other number here, and not comparable across machines. Machine conditions: the one-minute load average sampled immediately before each batch ranged 3.3 to 14.0 across every batch of every repeat, on a 10-core machine. The tier's hygiene rule is that no other analyzer competes for the machine while one is timed, and background load is reported rather than asserted away — so the reader can weigh the conditions instead of taking "quiet machine" on trust.

Show the batch series — Semgrep CE on the java kernel, all 2 retained repeats

Cases in the batch (k)	Run 1 — wall-clock of the one process	Run 2 — wall-clock of the one process	Average per case at run 2 (not the reported figure)
1	977 ms	973 ms	973 ms
2	1.05 s	1.06 s	528 ms
4	1.19 s	1.20 s	300 ms
8	1.50 s	1.49 s	187 ms
12	1.79 s	1.82 s	152 ms
slope (least squares)	74 ms	76 ms	—

The average-per-case column is shown precisely so it can be discarded: it falls with k because it is still paying a shrinking share of the fixed cost, and it would keep falling with a larger k without ever being the marginal cost. The reported figure is the slope of a wall-clock column against the first, and the published figure is the range those slopes span. Population:one `semgrep scan` carries one --config, so a batch is restricted to cases resolving to identical rule text: 12 of the 14 invocable Java kernel assertions. Measured on Mac16,1, macos25.6.0, 10 logical CPUs — the same stamp rule as every other number here, and not comparable across machines. Machine conditions: the one-minute load average sampled immediately before each batch ranged 2.3 to 2.4 across every batch of every repeat, on a 10-core machine. The tier's hygiene rule is that no other analyzer competes for the machine while one is timed, and background load is reported rather than asserted away — so the reader can weigh the conditions instead of taking "quiet machine" on trust.

Which adapters this could be measured on, and why not the rest

Each verdict below was reached by interrogating the pinned distribution — its help output, and where that was ambiguous its own bytecode — never from a README or an assumption about the runtime. No adapter is patched, forked, or invoked outside its released interface to make a batch possible. An adapter that ships no way to analyze several cases in one process has no warm figure here, and none is estimated for it from its runtime or from another adapter's slope.

Adapter	Observable with the released CLI?	Evidence
Joern	Yes — measured	A released multi-case batch that does the same per-case work, timed above.
Semgrep CE	Yes — measured	A released multi-case batch that does the same per-case work, timed above. one `semgrep scan` carries one --config, so a batch is restricted to cases resolving to identical rule text: 12 of the 14 invocable Java kernel assertions.
FlowDroid	Yes in the CLI — not measured here	The shipped CLI accepts a directory batch, but a batch shares one sources-and-sinks configuration. Whole-population equivalence to independent case configurations remains unresolved (A15). No warm marginal is qualified. Fresh CLI observations: reports/raw/warm-observability-v071/.
OpenTaint	Not observable with the released CLI	Under the retained adapter contract, one project and entrypoint set forms one analysis. Multiple project inputs do not establish repeated independent case analysis in a persistent process. The fresh v0.4.6 wrapper and jar observations are retained under reports/raw/warm-observability-v071/; help absence alone is not the basis of this decline.
Pysa	Not observable with the released CLI	The retained contract uses one-shot pyre analyze; type-checker daemon commands do not establish a reusable taint-analysis process. Fresh CLI observations are retained under reports/raw/warm-observability-v071/.
CodeQL	Not observable with the released CLI	The pinned database create/analyze interfaces operate on one database per invocation. No same-work multi-database warm series was qualified; fresh interface observations are retained under reports/raw/warm-observability-v071/.
Infer	Not observable with the released CLI	The retained invocation operates on one capture database and exits. Java capture includes javac work; it is not an independently measured JVM startup term. Fresh CLI observations are retained under reports/raw/warm-observability-v071/.
Bifrost	Not observable with the released CLI	The retained policy CLI operates on one root per invocation; the workspace/MCP surface does not establish an equivalent reusable policy process. No warm marginal was measured for this pin. Cold timings do not establish a comparative warm ranking. Fresh CLI observations are retained under reports/raw/warm-observability-v071/.

The decline that matters most is Bifrost's, and it is stated in the direction that costs this benchmark's publisher something. Bifrost has no warm figure here, and its cold median already includes its own process start — so whatever its warm marginal is, it lies between zero and a number already published. Adding warm figures can therefore only move the other rows down toward Bifrost's, never Bifrost's row down further. The conflation this section corrects is one the vendor's own engine benefits from, which is the reason to correct it in public rather than leave the cold chart to be read as a steady-state ranking.

These artifacts are auxiliary evidence, outside the freeze. They are retained under reports/raw/warm-latency/ — the batch series, the environment stamp, and the per-case evidence each batch produced, so that a reader can check the batch did the real work rather than less of it — but freeze/v1 does not digest them, and this amendment does not extend it to. A warm number carries the pinned amendment commit 2007f15d687e…'s immutability for its bytes and no stronger guarantee, exactly as the cold timing sidecars do. No warm run wrote a normalized report, produced an outcome, or touched a scored population; validate-reports and the freeze manifest never read this directory.

Estimated per-invocation overhead

The warm marginal above answers what one more case costs a process already running, and only one adapter's released CLI let it be measured. This section answers a different question that every adapter can be asked: what does one invocation cost before it has anything to find? It is published under Amendment A24, preregistered — estimator, bias, tolerance, fixtures and presentation — before the first estimate existed.

The method, in one sentence: run one complete adapter invocation — same pipeline, same committed policy, rule or query, same flags, same subprocess shape, both subprocesses where the adapter has two — over a trivial no-flow fixture that declares the benchmark's own source and sink endpoints and never connects them, and take the runner's wall-clock around it. The fixture is generated into a scratch workspace before the process is spawned, exactly as every other fixture is, and is retained beside the measurement. Nothing is added to cases/: no population, denominator or freeze sees these files, and they are not cases.

What the number is biased by, in both directions. A trivial file is still parsed, still extracted, still queried, so the measurement is fixed overhead plus that near-zero analysis: it is an upper bound on start-up and warm-up, and the true fixed cost is at or below it. And it is a cold, single-shot execution — no warm JIT, no primed cache — which is exactly the posture the cold rows above are measured in, and exactly not the posture of a resident deployment, where both this figure and those rows would fall. It is an estimate, labelled one everywhere it appears, and it is never subtracted from a cold number, never substituted for one, and never used to order a row.

Adapter	Fixture language	Estimated overhead (range over every repeat)	Width	Every repeat	Cold median, same kernel	Low end as a share of cold	Load observed	Marked on the chart?
Bifrost	python	135 ms – 236 ms	101 ms	236 ms, 135 ms, 138 ms	154 ms	88%	4.2	yes
CodeQL	ruby	3.50 s – 3.79 s	290 ms	3.79 s, 3.50 s, 3.55 s	4.51 s	78%	2.8 to 2.9	yes
FlowDroid	java	594 ms – 616 ms	22 ms	616 ms, 594 ms, 608 ms	759 ms	78%	1.9	yes
Infer	c	253 ms – 463 ms	210 ms	463 ms, 253 ms, 253 ms	275 ms	92%	2.2 to 2.2	yes
Joern	java	7.80 s – 8.50 s	707 ms	8.48 s, 8.50 s, 7.80 s	8.77 s	89%	4.8 to 5.3	yes
Joern	php	3.65 s – 4.39 s	744 ms	4.39 s, 3.76 s, 3.65 s	4.71 s	77%	2.1 to 2.2	yes
OpenTaint	kotlin	3.64 s – 3.71 s	65 ms	3.70 s, 3.64 s, 3.71 s	4.43 s	82%	2.7 to 3.9	yes
Pysa	python	2.42 s – 2.51 s	95 ms	2.42 s, 2.48 s, 2.51 s	2.93 s	83%	2.4 to 2.6	yes
Semgrep CE	kotlin	936 ms – 944 ms	8 ms	940 ms, 944 ms, 936 ms	1.22 s	77%	1.6 to 1.7	yes

The published figure is a range, and its width is the measurement's precision. Each estimate is repeated a fixed number of times — the count is a constant in the runner's source, not a per-run choice — every repeat is retained and printed above, and what is published is the range those repeats span. Never a mean, never a chosen repeat, and never withheld because the repeats disagreed: a disagreement widens the range, which is the honest consequence of it, and there is no agreement threshold anywhere in this machinery to be justified or tuned — a unit test asserts that no such constant exists. This is the same convention the warm-marginal figures publish under, shared rather than re-derived. The build re-derives each range from the retained repeats and fails if it disagrees with what the runner wrote.

The last column is what decides a chart mark, and the threshold is preregistered. A row carries a dashed span only where its range starts at or above 25% of that adapter's cold median on the same kernel. The cut is relative because the chart's axis is logarithmic and its rows span two orders of magnitude, so a share of each row's own median is the same visual claim everywhere, where a fixed millisecond cut would mark every slow adapter and no fast one whatever its overhead actually was; and it reads the range's low end so that a mark can never appear on the strength of one slow repeat. An unmarked row is not an unmeasured one — every value is in the table above.

The load column is the machine's own condition, published rather than summarized as the word "quiet". Each repeat samples the one-minute load average immediately before its subprocess is spawned, and the column is the range those samples span, so a figure taken on a busy machine can be discounted instead of taken on trust. Two disciplines stand behind those numbers: no other analyzer under measurement competes for the machine, and — because nine heavy analyzers run back to back drive the load up by themselves — each adapter's measurement waits for the load to settle before it begins, so what a row records is the machine's state and not its own position in the sequence.

Joern is the one adapter where three figures can be set side by side, and they are three different things. Its estimated per-invocation overhead on the Java kernel is 7.80 s – 8.50 s — a measured upper bound, published across the range its repeats spanned. The fitted fixed cost, the intercept of the warm batch series on the same kernel, is 2.90 s – 4.29 s — not measured at all, but inferred from a line through five batches. And its measured warm marginal, the cost of one more case in a process already running, is 5.06 s – 5.24 s. The first two are two routes to the same quantity and the third is a different quantity entirely.

Those first two routes disagree, by roughly a factor of three, and the disagreement is the informative part. Neither corrects the other and their difference is not published as a measurement, but the direction is exactly what the two constructions predict. The estimate is an upper bound that contains work the intercept excludes by construction: a whole trivial invocation — a fresh JVM, the Java front end loaded, a CPG built for the fixture, and the kernel script run over it — where the fitted intercept is what is left when a line through five batch sizes is extended back to zero cases, and every per-case cost the batch pays has been taken out of it. So the true once-per-process cost is bounded above by the estimate and approached from below by the intercept, and the gap between them is the part of "fixed cost" that neither construction can see on its own: not observable from inside one invocation, and not separable from per-case work inside a batch. That is the honest reading, and it is why this page publishes both figures rather than reconciling them into one.

Which adapters an estimate could be taken for

Each adapter is estimated in the language of its cheapest kernel arm — the core kernel whose cold median is that adapter's lowest — because that is the arm where fixed cost is the largest share of the number, and the arm whose trivial-fixture invocation is least dominated by analysis. Joern is additionally estimated on Java, so that A21's Java warm figures have a same-language estimate to be compared against; that second figure is labelled by its own language and is not the one the cheapest-arm rule selects.

Adapter	Fixture language	Verdict	Evidence
Bifrost	python its cheapest kernel arm	Measured	3 repeats — 236 ms, 135 ms, 138 ms — published as the 135 ms to 236 ms range they span.
CodeQL	ruby its cheapest kernel arm	Measured	3 repeats — 3.79 s, 3.50 s, 3.55 s — published as the 3.50 s to 3.79 s range they span.
FlowDroid	java its cheapest kernel arm	Measured	3 repeats — 616 ms, 594 ms, 608 ms — published as the 594 ms to 616 ms range they span.
Infer	c its cheapest kernel arm	Measured	3 repeats — 463 ms, 253 ms, 253 ms — published as the 253 ms to 463 ms range they span.
Joern	php its cheapest kernel arm	Measured	3 repeats — 4.39 s, 3.76 s, 3.65 s — published as the 3.65 s to 4.39 s range they span.
OpenTaint	kotlin its cheapest kernel arm	Measured	3 repeats — 3.70 s, 3.64 s, 3.71 s — published as the 3.64 s to 3.71 s range they span.
Pysa	python its only kernel arm	Measured	3 repeats — 2.42 s, 2.48 s, 2.51 s — published as the 2.42 s to 2.51 s range they span.
Semgrep CE	kotlin its cheapest kernel arm	Measured	3 repeats — 940 ms, 944 ms, 936 ms — published as the 936 ms to 944 ms range they span.
Joern	java not its cheapest arm — measured so that A21's Java warm figures have a same-language estimate to be compared against	Measured	3 repeats — 8.48 s, 8.50 s, 7.80 s — published as the 7.80 s to 8.50 s range they span.

There is exactly one kind of missing figure here, and it is a fact about a machine.Environment means the pinned distribution is not installed where the estimator ran, so the invocation was never attempted — explicitly not a statement about the adapter's released CLI, unlike A15's capability declines above, and resolved by running the same committed command where the distribution is installed. There is no second kind: because the figure is a range over every repeat, noisy repeats produce a wide range rather than a withheld number, so nothing is ever missing for having been measured badly. Nothing is filled in by inference either — no estimate here is derived from an adapter's runtime, its architecture, another adapter's estimate, or the same adapter's estimate in another language.

These artifacts are auxiliary evidence, outside the freeze, retained under reports/raw/invocation-overhead/: every repeat with its phase split and the one-minute load average it was taken under, the trivial fixture itself with its digest, the resolved configuration where the adapter's is per-case, the published range, and the environment stamp. freeze/v1 does not digest them, exactly as it does not digest the warm artifacts or the cold timing sidecars, so an estimate carries the pinned amendment commit 2007f15d687e…'s immutability for its bytes and no stronger guarantee. No estimator run wrote a normalized report, produced an outcome, or touched a scored population.

Per-slice medians

The tier's declared unit of aggregation: one row per slice — adapter × language × population — because a median over a whole adapter mixes languages whose fixtures differ in size and whose front ends differ in cost. Each row is one bound report. Rows are not ranked and columns are not compared across adapters with different granularity. One disclosure per adapter, for the same reason: the rows inside one of them are the rows that share a granularity.

Bifrost — 20 slices

Show the data table — 20 slices for Bifrost

Slice	Language	Profile	Timed / bound	Median	Q1	Q3
bifrost-smoke	13 languages	benchmark-controlled	117 / 118	164 ms	157 ms	13.1 s
bifrost-c-kernel	c	benchmark-controlled	50 / 50	204 ms	177 ms	216 ms
bifrost-cpp-kernel	cpp	benchmark-controlled	56 / 56	198 ms	177 ms	210 ms
bifrost-csharp-kernel	csharp	benchmark-controlled	58 / 58	190 ms	174 ms	199 ms
bifrost-go-kernel	go	benchmark-controlled	58 / 58	228 ms	215 ms	239 ms
bifrost-java-kernel	java	benchmark-controlled	58 / 58	19.0 s	18.8 s	19.3 s
bifrost-java-modeling	java	benchmark-controlled	8 / 24	13.4 s	13.4 s	13.4 s
bifrost-java-native	java	tool-native	0 / 12	n/a	n/a	n/a
bifrost-javascript-kernel	javascript	benchmark-controlled	58 / 58	165 ms	162 ms	173 ms
bifrost-javascript-modeling	javascript	benchmark-controlled	8 / 24	165 ms	159 ms	166 ms
bifrost-javascript-native	javascript	tool-native	0 / 12	n/a	n/a	n/a
bifrost-kotlin-kernel	kotlin	benchmark-controlled	58 / 58	13.3 s	13.2 s	13.4 s
bifrost-php-kernel	php	benchmark-controlled	58 / 58	158 ms	152 ms	165 ms
bifrost-python-kernel	python	benchmark-controlled	58 / 58	154 ms	150 ms	159 ms
bifrost-python-modeling	python	benchmark-controlled	8 / 24	151 ms	148 ms	153 ms
bifrost-python-native	python	tool-native	0 / 12	n/a	n/a	n/a
bifrost-ruby-kernel	ruby	benchmark-controlled	58 / 58	163 ms	156 ms	176 ms
bifrost-rust-kernel	rust	benchmark-controlled	56 / 56	161 ms	157 ms	169 ms
bifrost-scala-kernel	scala	benchmark-controlled	58 / 58	13.1 s	13.0 s	13.2 s
bifrost-typescript-kernel	typescript	benchmark-controlled	58 / 58	161 ms	155 ms	166 ms

FlowDroid — 4 slices

Show the data table — 4 slices for FlowDroid

Slice	Language	Profile	Timed / bound	Median	Q1	Q3	total median	compile median	dex median	analyze median
flowdroid-java-kernel	java	benchmark-controlled	58 / 58	759 ms	714 ms	786 ms	n/a	n/a	n/a	n/a
flowdroid-java-modeling	java	benchmark-controlled	14 / 24	576 ms	567 ms	588 ms	n/a	501 ms	330 ms	576 ms
flowdroid-java-native	java	tool-native	0 / 12	n/a	n/a	n/a	n/a	n/a	n/a	n/a
flowdroid-kotlin-kernel	kotlin	benchmark-controlled	58 / 58	996 ms	923 ms	1.03 s	n/a	n/a	n/a	n/a

Infer — 5 slices

Show the data table — 5 slices for Infer

Slice	Language	Profile	Timed / bound	Median	Q1	Q3	capture median	analyze median
infer-c-kernel	c	benchmark-controlled	48 / 48	275 ms	271 ms	279 ms	78 ms	198 ms
infer-cpp-kernel	cpp	benchmark-controlled	56 / 56	292 ms	274 ms	1.49 s	79 ms	200 ms
infer-java-kernel	java	benchmark-controlled	58 / 58	3.90 s	3.85 s	3.96 s	3.69 s	200 ms
infer-java-modeling	java	benchmark-controlled	10 / 24	3.98 s	3.92 s	4.04 s	3.78 s	201 ms
infer-java-native	java	tool-native	0 / 12	n/a	n/a	n/a	n/a	n/a

Semgrep CE — 17 slices

Show the data table — 17 slices for Semgrep CE

Slice	Language	Profile	Timed / bound	Median	Q1	Q3
semgrep-c-kernel	c	benchmark-controlled	14 / 48	1.14 s	1.12 s	1.15 s
semgrep-cpp-kernel	cpp	benchmark-controlled	14 / 56	1.21 s	1.16 s	1.32 s
semgrep-go-kernel	go	benchmark-controlled	14 / 58	1.44 s	1.41 s	1.49 s
semgrep-java-kernel	java	benchmark-controlled	14 / 58	1.28 s	1.22 s	1.37 s
semgrep-java-modeling	java	benchmark-controlled	10 / 24	1.26 s	1.20 s	1.32 s
semgrep-java-native	java	tool-native	0 / 12	n/a	n/a	n/a
semgrep-javascript-kernel	javascript	benchmark-controlled	14 / 58	1.32 s	1.21 s	1.40 s
semgrep-javascript-modeling	javascript	benchmark-controlled	10 / 24	1.27 s	1.23 s	1.41 s
semgrep-javascript-native	javascript	tool-native	0 / 12	n/a	n/a	n/a
semgrep-kotlin-kernel	kotlin	benchmark-controlled	14 / 58	1.22 s	1.18 s	1.28 s
semgrep-php-kernel	php	benchmark-controlled	14 / 58	1.20 s	1.17 s	1.22 s
semgrep-python-kernel	python	benchmark-controlled	14 / 58	1.29 s	1.21 s	1.38 s
semgrep-python-modeling	python	benchmark-controlled	10 / 24	1.33 s	1.24 s	1.38 s
semgrep-python-native	python	tool-native	12 / 12	2.86 s	2.84 s	2.88 s
semgrep-ruby-kernel	ruby	benchmark-controlled	14 / 58	1.43 s	1.31 s	1.54 s
semgrep-rust-kernel	rust	benchmark-controlled	14 / 54	1.43 s	1.24 s	1.55 s
semgrep-typescript-kernel	typescript	benchmark-controlled	14 / 58	1.49 s	1.48 s	1.55 s

Pysa — 3 slices

Show the data table — 3 slices for Pysa

Slice	Language	Profile	Timed / bound	Median	Q1	Q3
pysa-python-kernel	python	benchmark-controlled	58 / 58	2.93 s	2.89 s	2.98 s
pysa-python-modeling	python	benchmark-controlled	20 / 24	2.93 s	2.89 s	2.96 s
pysa-python-native	python	tool-native	12 / 12	2.69 s	2.66 s	2.72 s

OpenTaint — 4 slices

Show the data table — 4 slices for OpenTaint

Slice	Language	Profile	Timed / bound	Median	Q1	Q3
opentaint-java-kernel	java	benchmark-controlled	58 / 58	4.25 s	4.18 s	4.37 s
opentaint-java-modeling	java	benchmark-controlled	12 / 24	4.28 s	4.15 s	4.34 s
opentaint-java-native	java	tool-native	0 / 12	n/a	n/a	n/a
opentaint-kotlin-kernel	kotlin	benchmark-controlled	58 / 58	4.43 s	4.32 s	4.59 s

Joern — 12 slices

Show the data table — 12 slices for Joern

Slice	Language	Profile	Timed / bound	Median	Q1	Q3
joern-java-kernel	java	benchmark-controlled	58 / 58	8.77 s	8.71 s	8.84 s
joern-java-modeling	java	benchmark-controlled	16 / 24	8.89 s	8.81 s	9.52 s
joern-java-native	java	tool-native	0 / 12	n/a	n/a	n/a
joern-javascript-kernel	javascript	benchmark-controlled	58 / 58	5.15 s	5.03 s	5.35 s
joern-javascript-modeling	javascript	benchmark-controlled	16 / 24	4.94 s	4.85 s	4.97 s
joern-javascript-native	javascript	tool-native	0 / 12	n/a	n/a	n/a
joern-php-kernel	php	benchmark-controlled	58 / 58	4.71 s	4.66 s	4.84 s
joern-python-kernel	python	benchmark-controlled	58 / 58	4.39 s	4.31 s	4.46 s
joern-python-modeling	python	benchmark-controlled	16 / 24	4.23 s	4.14 s	4.36 s
joern-python-native	python	tool-native	0 / 12	n/a	n/a	n/a
joern-ruby-kernel	ruby	benchmark-controlled	58 / 58	6.66 s	6.52 s	6.75 s
joern-rust-kernel	rust	benchmark-controlled	54 / 54	6.29 s	6.12 s	6.39 s

CodeQL — 17 slices

Show the data table — 17 slices for CodeQL

Slice	Language	Profile	Timed / bound	Median	Q1	Q3	database-create median	database-analyze median
codeql-c-kernel	c	benchmark-controlled	50 / 50	5.82 s	5.68 s	6.14 s	1.35 s	4.46 s
codeql-cpp-kernel	cpp	benchmark-controlled	56 / 56	5.74 s	5.62 s	17.9 s	1.37 s	4.41 s
codeql-csharp-kernel	csharp	benchmark-controlled	58 / 58	20.0 s	19.7 s	20.5 s	11.2 s	8.92 s
codeql-go-kernel	go	benchmark-controlled	58 / 58	6.36 s	6.20 s	6.59 s	3.36 s	2.99 s
codeql-java-kernel	java	benchmark-controlled	58 / 58	11.2 s	11.1 s	11.4 s	6.01 s	5.17 s
codeql-java-modeling	java	benchmark-controlled	24 / 24	10.9 s	10.7 s	11.0 s	6.05 s	4.81 s
codeql-java-native	java	tool-native	12 / 12	15.0 s	14.8 s	15.9 s	5.69 s	9.32 s
codeql-javascript-kernel	javascript	benchmark-controlled	58 / 58	8.23 s	8.14 s	8.30 s	3.34 s	4.89 s
codeql-javascript-modeling	javascript	benchmark-controlled	24 / 24	8.10 s	8.01 s	8.18 s	3.39 s	4.71 s
codeql-javascript-native	javascript	tool-native	12 / 12	12.6 s	12.3 s	13.1 s	3.20 s	9.36 s
codeql-kotlin-kernel	kotlin	benchmark-controlled	58 / 58	16.3 s	16.2 s	16.5 s	11.0 s	5.27 s
codeql-python-kernel	python	benchmark-controlled	58 / 58	5.55 s	5.53 s	5.65 s	1.99 s	3.57 s
codeql-python-modeling	python	benchmark-controlled	24 / 24	5.38 s	5.34 s	5.45 s	2.00 s	3.40 s
codeql-python-native	python	tool-native	12 / 12	6.78 s	6.75 s	6.87 s	1.96 s	4.82 s
codeql-ruby-kernel	ruby	benchmark-controlled	58 / 58	4.51 s	4.45 s	4.57 s	1.22 s	3.28 s
codeql-rust-kernel	rust	benchmark-controlled	56 / 56	46.8 s	46.5 s	47.4 s	20.5 s	26.2 s
codeql-typescript-kernel	typescript	benchmark-controlled	58 / 58	8.33 s	8.28 s	8.44 s	3.48 s	4.87 s

Cases with no latency number

Of the case results in this bound latency corpus, 2725 invoked an analyzer and carry a timing sidecar. 755 do not, and every one of them is accounted for:

Outcome of the untimed case	Cases
unsupported	755

An unsupported case is decided from case metadata before the analyzer is invoked, so there is no subprocess and there is nothing to time. Those cases are absent from every distribution on this page, not entered as zero — entering them as zero would make the adapters that decline the most look the fastest, which is exactly backwards. Every case that did invoke an analyzer is timed; the timed and untimed columns account for the whole bound population with no remainder.

Provenance and what the freeze does and does not guarantee

Every cold number above was derived at build time from v0.7.1's archived latency-evidence bundle. The bundle was generated from that release commit's retained timing sidecars, and its population comes from the results model generated by freeze manifest f5416cded589…. Snapshot pages select this bundle explicitly: advancing the current correctness freeze cannot add a case, replace a timing, or relabel an analyzer on a historical page. The warm and invocation-overhead amendments are likewise read from a checked-in auxiliary bundle pinned to commit 2007f15d687e0081c948e55bba39c952d248ee0f; they never read the mutable working-tree artifact directories.

One limit, stated rather than left to be assumed. The freeze/v1 manifest binds one raw-evidence digest per result; the timing sidecar and the environment stamp are additive files beside that evidence and are not themselves digested by the manifest. So a number on this page carries the freeze's guarantee that the run it belongs to is bound and byte-verified, and the release commit plus the checked-in archive's own Git history preserve the timing bytes selected for rendering. It does not carry a freeze-manifest digest over those timing bytes. Extending the manifest to bind them is a freeze/v2 question, and it is named here rather than smuggled in under a schema version that does not describe it.