Latency — v0.7.0
DataFlowBench’s latency-characterization tier. It stands beside the correctness scorecards, on its own page, and never as a column inside one.
The contract governing every number here —
docs/latency-tier.md
— was written and merged before any timestamp was captured, and it is
explicit about why. DataFlowBench is published by the vendor of one of the
engines measured, and a latency page assembled after the numbers were known,
with its phase boundaries and its aggregation chosen afterwards, would deserve
exactly the skepticism it would get. So the decomposition rule, the exclusion
list, the per-adapter granularity, and the aggregation were all fixed while
every number was still unmeasured. The four analyzers added in v0.6.0 took
their granularity rows by
Amendment A12,
dated and landed before this page published any of their numbers, rather
than being backfilled into the original table.
The page also carries a second, separately labelled figure: the warm marginal cost of one more case in a tool process that has already started, added by Amendment A15 and likewise preregistered before the first warm number was measured. It exists because the cold rows below span three runtimes, and read across runtimes they overstate the steady-state gap — a difference the vendor publishing this benchmark happens to win. The correction is a direct measurement, not an estimate subtracted from anything: no cold number on this page changes, and the warm figure is never substituted for one.
What these numbers are, and what they are not
Section titled “What these numbers are, and what they are not”
This is a descriptive characterization of per-case analyzer
wall-clock, published under the contract preregistered in
docs/latency-tier.md,
which was merged before a single timestamp was captured. It is not a score,
not a ranking, and not a benchmark result in the sense the rest of this site
uses that phrase.
- Latency is never pooled with correctness. There is no combined number anywhere on this site, no efficiency-adjusted rate, and no leaderboard that blends the two. "Correct but slow" and "fast but wrong" stay independently visible. No correctness outcome in this freeze was derived from, conditioned on, or tie-broken by a timing value.
- A latency number is a property of a run, not of a case. No case carries a timing assertion, threshold, or budget.
- Characterization, not micro-benchmark. There are no repeated trials, no warm-up iterations, and no steady-state harness in the cold figures below. The case population is the sample and its spread is the statistics. The distribution columns below are that spread — not error bars on a measurement that was repeated, because it was not. (The warm-marginal section is the one place a series is measured more than once. It says so, retains every repeat, and publishes the range they span rather than averaging them into a statistic.)
- These are the conditions the numbers were produced under: a single developer machine, under light concurrent load, running the benchmark's standing sequential-run discipline — one analyzer at a time, never two at once. That discipline is what makes the numbers usable at all; the light concurrent load is what stops them being a controlled measurement. Read them as characterization of what this benchmark actually costs to run, at the order-of-magnitude and shape level, and not as a precise figure for any engine.
Environment stamp
Section titled “Environment stamp”Recorded once per run beside the tool identity the run witnessed, and shown here verbatim. Latency numbers are comparable within one environment and are not comparable across machines. Every published run to date executes on one maintainer machine, and that caveat travels with every number on this page.
| Hardware model | OS | OS release | Logical CPUs | CPU architecture | Timed runs stamped |
|---|---|---|---|---|---|
Mac16,1 | macos | 25.6.0 | 10 | aarch64 | 66 |
One row: every timed invocation in this bound latency corpus ran in the same stamped environment, so no number on this page is a cross-environment comparison. Were there two rows, this page would say so in the table rather than averaging across them. The count is of run directories that contain at least one timed invocation — a run that invoked no analyzer at all still stamps its environment and witnesses its tool identity, but contributes no latency number and is not counted here.
The granularity is unequal, and that is the point
Section titled “The granularity is unequal, and that is the point”A phase is timed if and only if the adapter already invokes it as a separate subprocess. The benchmark never instruments analyzer internals, never patches an upstream tool, and never infers a boundary a subprocess boundary does not expose. The adapters do not expose the same boundaries, so the rows below are not equally decomposed — and the two ways of papering over that would both misdescribe the evidence: summing a decomposed adapter's phases and presenting every row as one number of the same kind, or guessing phase splits for the adapters that expose only one.
So the table states each adapter's declared granularity as data.
Phases compare within an adapter; analyzer-invocation totals compare
across adapters. A CodeQL database-create number and a Joern
total are not the same kind of quantity, and nothing on this page sets them
side by side as though they were.
| Adapter | Version | Observed subprocesses per case | Phases timed | Timed invocations |
|---|---|---|---|---|
| Bifrost | 0.10.7 | 0 or 1 | total | 883 |
| Semgrep CE | 1.175.0 | 0 or 1 | total | 196 |
| FlowDroid | 2.15.1 | 1 | total | 116 |
| Infer | v1.3.0 | 2 | capture analyze | 162 |
| Pysa | 0.10.0 | 1 | total | 58 |
| OpenTaint | analyzer/2026.08.27.17eb0fe | 1 | total | 116 |
| Joern | 4.0.614 | 0 or 1 | total | 392 |
| CodeQL | 2.26.4 | 2 | database-create database-analyze | 734 |
2 of 8 adapters expose more than one subprocess and are decomposed here (Infer and CodeQL); the other 6 expose a single invocation and take one honestly labelled number, with interpreter or JVM start-up inside it. That is a fact about invocation shapes, not about engine architecture: an adapter with one number is not an adapter that does one thing.
The phase names above are the ones the retained evidence carries, so any
name here can be found in the artifact it came from. For CodeQL those are
database-create and database-analyze, after the
subcommands that produce them, where the preregistration's table spells the
same two boundaries extract and analyze. Two
spellings, one pair of subprocess boundaries: nothing moved and nothing is
attributed differently. The contract is immutable, so the difference is
recorded in its
Amendment A12
rather than edited out of its table.
Analyzer-invocation wall-clock, per adapter
Section titled “Analyzer-invocation wall-clock, per adapter”The only cross-adapter reading this tier supports: each adapter's cold per-case analyzer-invocation wall-clock, over every case in this snapshot's explicitly bound latency population that it actually invoked. Adapter-observable setup phases remain visible below, but only phases the contract classifies as analyzer work enter this total. Per-case timing at this granularity includes per-invocation fixed costs — JVM start-up, extractor initialization, interpreter start — that a long-lived deployment of the same engine would amortize. This tier does not correct for that; it characterizes what the benchmark actually runs.
Fastest to slowest, and the same ranking per kernel
Ordered by median, on a logarithmic axis, with the spread drawn beside every median rather than left to the table. The toggle scopes the same ranking to one language kernel at a time: a whole-corpus median mixes the languages an adapter runs on, and holding the language fixed is the difference between "this engine is slow" and "this front end is expensive". Only the analyzers that actually invoked on a kernel appear in its view. The chart draws the median, the quartiles and the p10–p90 spread; the disclosure beneath it opens the same distributions as a table, with the minima and maxima the whiskers deliberately leave out.
Every timed analyzer invocation the freeze binds — 2657 of them, across every score tier and both model profiles. This is the widest denominator on the site and the only one here that is not a single population: an adapter's median mixes the languages it runs on, whose fixtures differ in size and whose front ends differ in cost. The per-kernel views hold the language fixed.
The c kernel's 48 benchmark-controlled core
assertions, the same no-pooling population the correctness
sections read. 4 of the 8 analyzers invoked here at all; the rest
do not cover this kernel and are absent from this view rather than drawn at zero.
Where a row's count is below 48, the
remaining assertions were declined before any subprocess was
spawned and so have nothing to time.
The cpp kernel's 56 benchmark-controlled core
assertions, the same no-pooling population the correctness
sections read. 4 of the 8 analyzers invoked here at all; the rest
do not cover this kernel and are absent from this view rather than drawn at zero.
Where a row's count is below 56, the
remaining assertions were declined before any subprocess was
spawned and so have nothing to time.
The csharp kernel's 58 benchmark-controlled core
assertions, the same no-pooling population the correctness
sections read. 2 of the 8 analyzers invoked here at all; the rest
do not cover this kernel and are absent from this view rather than drawn at zero.
Where a row's count is below 58, the
remaining assertions were declined before any subprocess was
spawned and so have nothing to time.
The go kernel's 58 benchmark-controlled core
assertions, the same no-pooling population the correctness
sections read. 3 of the 8 analyzers invoked here at all; the rest
do not cover this kernel and are absent from this view rather than drawn at zero.
Where a row's count is below 58, the
remaining assertions were declined before any subprocess was
spawned and so have nothing to time.
The java kernel's 58 benchmark-controlled core
assertions, the same no-pooling population the correctness
sections read. 7 of the 8 analyzers invoked here at all; the rest
do not cover this kernel and are absent from this view rather than drawn at zero.
Where a row's count is below 58, the
remaining assertions were declined before any subprocess was
spawned and so have nothing to time.
The javascript kernel's 58 benchmark-controlled core
assertions, the same no-pooling population the correctness
sections read. 4 of the 8 analyzers invoked here at all; the rest
do not cover this kernel and are absent from this view rather than drawn at zero.
Where a row's count is below 58, the
remaining assertions were declined before any subprocess was
spawned and so have nothing to time.
The kotlin kernel's 58 benchmark-controlled core
assertions, the same no-pooling population the correctness
sections read. 5 of the 8 analyzers invoked here at all; the rest
do not cover this kernel and are absent from this view rather than drawn at zero.
Where a row's count is below 58, the
remaining assertions were declined before any subprocess was
spawned and so have nothing to time.
The php kernel's 58 benchmark-controlled core
assertions, the same no-pooling population the correctness
sections read. 3 of the 8 analyzers invoked here at all; the rest
do not cover this kernel and are absent from this view rather than drawn at zero.
Where a row's count is below 58, the
remaining assertions were declined before any subprocess was
spawned and so have nothing to time.
The python kernel's 58 benchmark-controlled core
assertions, the same no-pooling population the correctness
sections read. 5 of the 8 analyzers invoked here at all; the rest
do not cover this kernel and are absent from this view rather than drawn at zero.
Where a row's count is below 58, the
remaining assertions were declined before any subprocess was
spawned and so have nothing to time.
The ruby kernel's 58 benchmark-controlled core
assertions, the same no-pooling population the correctness
sections read. 4 of the 8 analyzers invoked here at all; the rest
do not cover this kernel and are absent from this view rather than drawn at zero.
Where a row's count is below 58, the
remaining assertions were declined before any subprocess was
spawned and so have nothing to time.
The rust kernel's 54 benchmark-controlled core
assertions, the same no-pooling population the correctness
sections read. 4 of the 8 analyzers invoked here at all; the rest
do not cover this kernel and are absent from this view rather than drawn at zero.
Where a row's count is below 54, the
remaining assertions were declined before any subprocess was
spawned and so have nothing to time.
The scala kernel's 58 benchmark-controlled core
assertions, the same no-pooling population the correctness
sections read. 1 of the 8 analyzers invoked here at all; the rest
do not cover this kernel and are absent from this view rather than drawn at zero.
Where a row's count is below 58, the
remaining assertions were declined before any subprocess was
spawned and so have nothing to time.
The typescript kernel's 58 benchmark-controlled core
assertions, the same no-pooling population the correctness
sections read. 3 of the 8 analyzers invoked here at all; the rest
do not cover this kernel and are absent from this view rather than drawn at zero.
Where a row's count is below 58, the
remaining assertions were declined before any subprocess was
spawned and so have nothing to time.
- median, printed beside every row
- interquartile range (Q1–Q3)
- p10–p90; the minimum and maximum are in the tables, not the whiskers
- indented rows: an adapter's own declared phases
- measured warm marginal per case — the range its retained repeats span
- estimated per-invocation overhead (trivial fixture, upper bound), drawn across the range its repeats spanned, where that range starts at or above 25% of the row's median
How to read this figure
Every bar above is cold per-invocation wall-clock, and the warm marginal is a different quantity measured separately. Cold is what this benchmark actually runs — one process per case, start-up inside the number, because start-up is not observable from inside a single invocation. Read across runtimes, though, those bars overstate the steady-state gap: a JVM engine's row carries a JVM start a long-lived deployment pays once. So the other quantity is measured directly rather than estimated and subtracted — k cases through one tool process, for increasing k, reporting the slope of batch wall-clock against k. The carets mark it on the rows that have one, and they mark a range: the whole series is measured more than once and the figure published is the span its repeats cover, so the mark's width is its own precision rather than a number stated more exactly than it was measured. It is never subtracted from a median, never substituted for one, and never used to order the rows. Only adapters whose released CLI exposes a real multi-case batch have a figure at all; the rest are not observable with the released CLI, and the warm-marginal section records every verdict, measured and declined, with the evidence behind it.
The dashed spans are estimates, not measurements of the same kind as the bars. Each is the wall-clock of one complete adapter invocation — same pipeline, same policy, rule or query, same subprocess shape — over a trivial no-flow fixture: both benchmark endpoints declared, nothing connecting them, nothing to find. That is fixed per-invocation overhead plus the trivial fixture's own near-zero analysis, so it is an upper bound on what an adapter pays before it starts work, and it is a cold single-shot execution — the same posture the bars are measured in, and not a steady-state one. It is never subtracted from a median, never substituted for one, and never used to order the rows. The width of a span is the figure's precision, not a decoration: each measurement is repeated a fixed number of times and what is published is the range those repeats spanned — never a mean, never one chosen repeat, and never withheld for repeats that disagree, because a disagreement widens the range and that is the honest consequence of it. A span is drawn only where the range starts at or above 25% of that row's own cold median — a threshold fixed in the amendment before any estimate was measured, read at the low end so that no mark can appear on the strength of one slow repeat. Each estimate is measured in one named language, stated on the row it annotates, because boot cost is not language-free. The estimates table carries every value — every repeat behind every range, every unmarked row, and every adapter for which no estimate could be taken at all.
The axis is logarithmic. Each labelled tick is three times the one before it, so equal distances are equal ratios, not equal durations — the gap from 100 ms to 300 ms is drawn the same width as the gap from 10 s to 30 s. That is the only way this bound corpus's medians fit in one picture: they span 123 ms to 10.8 s, and on a linear axis every analyzer except the slowest would be a sliver against the origin. Because a log axis is easy to misread, every median is also printed as a number at the right of its own row.
The indented rows are phases, and they are not comparable across adapters. Only the 2 adapters whose preregistered row declares more than one subprocess have them. A phase mark sits on the same axis as the totals because it is the same kind of measurement — wall-clock of a subprocess — but reading one adapter's phase against another adapter's total is precisely the comparison the granularity rule forbids. Read a phase against the adapter it is indented under, and nothing else.
Ordering is not scoring, and this is never pooled with correctness. Rows are sorted by median because an unsorted ranking is unreadable, not because latency is a result. No correctness figure appears in this chart and no number here is blended with one: there is no combined score anywhere on this site, and a fast analyzer that answers wrongly is neither rewarded nor penalised by anything drawn above. Cases an analyzer declined before invocation are absent, not entered as zero — entering them as zero would make the analyzers that decline the most look the fastest, which is exactly backwards.
Show the data table — analyzer-invocation wall-clock per adapter, with minima and maxima
| Adapter | Timed invocations | Median | Q1 | Q3 | Min | Max |
|---|---|---|---|---|---|---|
| Bifrost | 883 | 123 ms | 116 ms | 133 ms | 103 ms | 393 ms |
| Semgrep CE | 196 | 1.20 s | 1.11 s | 1.59 s | 1.03 s | 4.26 s |
| FlowDroid | 116 | 1.70 s | 1.55 s | 1.96 s | 1.35 s | 3.07 s |
| Infer | 162 | 2.21 s | 554 ms | 6.24 s | 468 ms | 8.25 s |
| Pysa | 58 | 4.08 s | 3.88 s | 4.21 s | 3.69 s | 5.95 s |
| OpenTaint | 116 | 4.60 s | 4.39 s | 4.98 s | 3.99 s | 10.7 s |
| Joern | 392 | 8.70 s | 7.18 s | 10.1 s | 4.48 s | 127.9 s |
| CodeQL | 734 | 10.8 s | 7.22 s | 16.7 s | 4.39 s | 161.0 s |
No mean is headlined anywhere on this page. The distributions are skewed by per-invocation fixed costs and by a small number of very long tails — the slowest single invocation in this bound latency corpus is 161.0 s, against a page-wide fastest median of 123 ms — and a mean over that shape would describe neither the typical case nor the tail. The adapters span roughly 88× between the fastest and slowest medians, which is the order-of-magnitude claim this tier is entitled to make, and the strongest one it makes.
Phase decomposition, within each adapter that has one
Section titled “Phase decomposition, within each adapter that has one”Only the 2 adapters whose declared row has phases appear here, and each is read down its own column. These numbers exist to decompose one adapter's own cost. Racing one adapter's phase against another's total is exactly the reading the granularity rule forbids. Each adapter's numbers open from its own disclosure, one per adapter, which is also the boundary the comparison rule draws.
Inferv1.3.0
Show the data table — Infer's 2 observable phases against its analyzer total
| Phase | Median | Q1 | Q3 | Min | Max | Share of the median analyzer total |
|---|---|---|---|---|---|---|
capture | 1.78 s | 160 ms | 5.87 s | 137 ms | 7.88 s | 80% |
analyze | 373 ms | 343 ms | 440 ms | 297 ms | 1.06 s | 17% |
| analyzer total | 2.21 s | 554 ms | 6.24 s | 468 ms | 8.25 s | — |
CodeQL2.26.4
Show the data table — CodeQL's 2 observable phases against its analyzer total
| Phase | Median | Q1 | Q3 | Min | Max | Share of the median analyzer total |
|---|---|---|---|---|---|---|
database-create | 4.51 s | 2.18 s | 10.7 s | 1.26 s | 155.9 s | 42% |
database-analyze | 5.58 s | 4.64 s | 8.29 s | 3.10 s | 43.5 s | 52% |
| analyzer total | 10.8 s | 7.22 s | 16.7 s | 4.39 s | 161.0 s | — |
For analyzer phases, the share column is the phase median against the analyzer-total median, and the shares need not sum to 100%: a median is not additive, and the phase whose median is largest is not necessarily the phase that dominated any particular case. Materialization phases are labelled and excluded rather than made to look comparable. The analyzer-total row is the distribution of per-case contract-selected phase sums, computed per case and then summarized — not the sum of the phase medians.
Warm marginal cost, measured separately
Section titled “Warm marginal cost, measured separately”Everything above is cold per-invocation wall-clock, and it stays the headline of this tier. It is also what this benchmark actually runs: one analyzer process per case, start-up inside the number, because start-up is not observable from inside a single invocation. Nothing in this section adjusts, corrects, or replaces one number above it.
But the ranked chart sets a native binary, a Python CLI, and five JVM or JVM-fronted engines on one axis, and read across runtimes those bars overstate the steady-state gap: a JVM row carries a JVM start that a long-lived deployment of the same engine pays once. The honest response is not to estimate that start-up and subtract it — that would be exactly the after-the-fact adjustment this tier's contract refuses. It is to measure the other quantity directly, which is what this section reports, under Amendment A15, preregistered before the first warm number existed.
The method, in one sentence: run k cases through one tool process for increasing k, and report the slope of batch wall-clock against k — the cost of one more case in a process that has already paid its start-up. A slope, never an average: an average per case at k still carries a 1/k share of the fixed cost, which is the very quantity the figure exists to remove. The batch reuses the cold runner's own case selection, endpoint resolution, workspace materialization, and query logic, so what is timed is the same work; only the number of cases sharing one process differs.
| Adapter | Kernel | Cold median, same cases | Warm marginal (least squares) | Warm marginal (endpoint) | Batches | Repeats | Fitted fixed cost |
|---|---|---|---|---|---|---|---|
| Joern | java | 15.6 s | 996 ms – 1.01 s | 996 ms – 1.00 s |
k = 1, 2, 4, 8, 16 | 2 | 3.41 s – 3.59 s |
| Semgrep CE | java | 1.16 s | 74 ms – 77 ms | 78 ms – 81 ms |
k = 1, 2, 4, 8, 12 | 2 | 986 ms – 1.06 s |
The cold column is the median over exactly the cases the warm batch analyzed, not over the whole kernel: two figures compared across different populations would mix a population effect into a start-up effect. The two columns are never subtracted. The "fitted fixed cost" is the same fit's intercept, published as the descriptive estimate it is — it is not measured, it is not a decomposition of any cold number, and nothing on this site subtracts it from one.
What makes these figures checkable. Each batch retains the per-case evidence it produced, beside the series above. Every document the largest Joern batch produced matches that case's retained cold evidence field for field — same analyzed state, same method count, same source and sink node counts, same flow count — differing only in the scratch path each run recorded for its own workspace. A batch that had quietly analyzed less would be visible there rather than merely fast. Every figure above is a range, and that is deliberate. The whole batch series is measured more than once, every repeat is retained, and what is published is the span the repeats cover. A single slope over a handful of batches on a developer machine has a precision, and there were only two other ways to give it one — publish one run and hide the spread, which understates it, or gate publication on an agreement tolerance, which means choosing that tolerance after the numbers exist, precisely the after-the-fact decision this tier's contract refuses. The range needs neither: its width is the precision, the reader sees it directly, and nothing between the measurement and this table is discretionary. The repeats are never averaged — that would turn repeated trials into the statistic the tier's non-goals rule out — and never picked between.
What these figures replaced
The figures above are not the first this page published. Amendment A15 published a warm marginal as a point estimate, under a rule that ran each measurement twice and withheld the figure unless the two runs "agreed closely". Amendment A21 supersedes both the rule and the numbers it produced. Those numbers are restated here rather than removed, because a retired figure is only auditable if it is still legible beside the one that replaced it.
| Adapter | A15 published | A21 publishes | Retired artifact |
|---|---|---|---|
| Joern | 996 ms – 1.01 s | reports/raw/warm-latency/superseded-a15/joern-java-kernel/warm-latency.json | |
| Semgrep CE | withheld | 74 ms – 77 ms | reports/raw/warm-latency/semgrep-java-stability-probe/ |
Why the rule was replaced. "Agree closely" named no tolerance, and the only moment at which one could have been chosen was after the spreads were known — the after-the-fact decision this tier's contract refuses. It was found by measuring again: two repeats on a quiet machine disagreed by 12% where two under load had agreed to 5%, and no tolerance could have been written down at that point without being fitted to spreads already seen. Publishing the interval that was measured needs no tolerance at all.
Why Joern's number moved, and why Semgrep's exists at all. Machine conditions. Both runs recorded the one-minute load average before every batch, so the comparison is read off the artifacts rather than recalled: Joern at 9.1 to 9.5 when A15 measured it, against 2.0 to 3.9 for the figure above. That is most of the difference in Joern's row. It is the whole of Semgrep's: A15 withheld a Semgrep figure because its two runs' slopes differed by a factor of two, and recorded that as a property of the measurement's subject. Re-measured on an idle machine, Semgrep's two repeats agree to about 5% — tighter than Joern's. The instability was the machine, not the engine, and attributing it to Semgrep was a mistake, recorded here rather than quietly dropped. It is also why the observed load now travels with every figure on this page.
Worth stating plainly, since this benchmark is published by the vendor of one of the engines on it: both corrections move numbers against that vendor. Joern's marginal falls by about a third, and a competitor's withheld figure is restored — and turns out to be the fastest warm marginal here. The rule was replaced because it was unsound, not because of which way it pointed.
Show the batch series — Joern on the java kernel, all 2 retained repeats
| Cases in the batch (k) | Run 1 — wall-clock of the one process | Run 2 — wall-clock of the one process | Average per case at run 2 (not the reported figure) |
|---|---|---|---|
| 1 | 4.67 s | 4.35 s | 4.35 s |
| 2 | 5.47 s | 5.43 s | 2.72 s |
| 4 | 7.48 s | 7.51 s | 1.88 s |
| 8 | 12.0 s | 11.3 s | 1.41 s |
| 16 | 19.6 s | 19.4 s | 1.21 s |
| slope (least squares) | 1.01 s | 996 ms | — |
The average-per-case column is shown precisely so it can be
discarded: it falls with k because it is still paying a
shrinking share of the fixed cost, and it would keep falling with a
larger k without ever being the marginal cost. The reported
figure is the slope of a wall-clock column against the first, and the
published figure is the range those slopes span.
Measured on Mac16,1, macos25.6.0, 10 logical CPUs — the same stamp
rule as every other number here, and not comparable across
machines.
Machine conditions: the one-minute load average
sampled immediately before each batch ranged 2.0 to 3.9 across every batch of every repeat, on
a 10-core machine. The
tier's hygiene rule is that no other analyzer competes for the
machine while one is timed, and background load is reported
rather than asserted away — so the reader can weigh the
conditions instead of taking "quiet machine" on trust.
Show the batch series — Semgrep CE on the java kernel, all 2 retained repeats
| Cases in the batch (k) | Run 1 — wall-clock of the one process | Run 2 — wall-clock of the one process | Average per case at run 2 (not the reported figure) |
|---|---|---|---|
| 1 | 1.06 s | 1.06 s | 1.06 s |
| 2 | 1.14 s | 1.31 s | 656 ms |
| 4 | 1.30 s | 1.32 s | 330 ms |
| 8 | 1.60 s | 1.64 s | 205 ms |
| 12 | 1.92 s | 1.95 s | 163 ms |
| slope (least squares) | 77 ms | 74 ms | — |
The average-per-case column is shown precisely so it can be
discarded: it falls with k because it is still paying a
shrinking share of the fixed cost, and it would keep falling with a
larger k without ever being the marginal cost. The reported
figure is the slope of a wall-clock column against the first, and the
published figure is the range those slopes span.
Population:one `semgrep scan` carries one --config, so a batch is restricted to cases resolving to identical rule text: 12 of the 14 invocable Java kernel assertions.
Measured on Mac16,1, macos25.6.0, 10 logical CPUs — the same stamp
rule as every other number here, and not comparable across
machines.
Machine conditions: the one-minute load average
sampled immediately before each batch ranged 4.1 to 4.3 across every batch of every repeat, on
a 10-core machine. The
tier's hygiene rule is that no other analyzer competes for the
machine while one is timed, and background load is reported
rather than asserted away — so the reader can weigh the
conditions instead of taking "quiet machine" on trust.
Which adapters this could be measured on, and why not the rest
Each verdict below was reached by interrogating the pinned distribution — its help output, and where that was ambiguous its own bytecode — never from a README or an assumption about the runtime. No adapter is patched, forked, or invoked outside its released interface to make a batch possible. An adapter that ships no way to analyze several cases in one process has no warm figure here, and none is estimated for it from its runtime or from another adapter's slope.
| Adapter | Observable with the released CLI? | Evidence |
|---|---|---|
| Joern | Yes — measured | A released multi-case batch that does the same per-case work, timed above. |
| Semgrep CE | Yes — measured | A released multi-case batch that does the same per-case work, timed above. one `semgrep scan` carries one --config, so a batch is restricted to cases resolving to identical rule text: 12 of the 14 invocable Java kernel assertions. |
| FlowDroid | Yes in the CLI — not measured here | The released CLI does have a batch: `-a/--apkfile` accepts a directory, and the shipped `soot-infoflow-cmd` main class lists its APKs, builds the taint wrapper once outside the loop, and iterates them in one JVM — its own help documents `-si/--skipapkfile` as "APK file to skip when processing a directory of input files", and it refuses a non-directory output with "The output file must be a directory when analyzing multiple APKs". It is not measured here because one invocation carries one `-s` sources-and-sinks definition, so a k-APK batch runs a union of k per-case endpoint configurations rather than each case's own. Whether that changes any case's result is an empirical question that has to be answered across the whole population before a marginal derived from it may be published. Named follow-up work, not a decline. |
| OpenTaint | Not observable with the released CLI | The pinned analyzer takes one `--project` and one `--output-dir` and exits after that project. A `project.yaml` may list several `javaProjects`, but analyzing their union is one whole-program analysis over a merged call graph and a merged entry-point set — different work, not k independent case analyses — and `--semgrep-rule-set` is one rule set for the whole invocation while the benchmark resolves a rule per case. No released mode processes separate case projects in one process. |
| Pysa | Not observable with the released CLI | `pyre analyze` is one-shot. The client does expose daemon commands (`start`, `incremental`, `query`), but they serve the type checker, not the taint analysis, and `analyze` never attaches to a running server. `--source-directory` is repeatable but merges directories into one project — again one whole-program analysis rather than k — and each case carries its own `.pyre_configuration`, `pyrefly.toml`, and resolved models. A daemon-shaped measurement would be stateful and not reproducibly preregisterable against this pin, so it is declined rather than attempted. |
| CodeQL | Not observable with the released CLI | `codeql database analyze` takes exactly one mandatory database, and `database create` produces exactly one database per invocation. Neither subcommand has a multi-database form in the pinned CLI. |
| Infer | Not observable with the released CLI | `--results-dir` names one capture database for one project and the analyzer exits after it. Worth stating explicitly: Infer's analyzer is a native binary, and the JVM cost inside its Java row is the traced `javac` in `capture` — per-project compilation work, not process start-up a batch could amortize. |
| Bifrost | Not observable with the released CLI | The policy CLI takes one `--root` per invocation; the repeatable `--workspace NAME=PATH` is documented as requiring `--mcp` and does not reach the policy path. What can be said without measuring anything is a bound: Bifrost's cold median already includes its own process start, so its warm marginal lies between zero and that cold number. Warm figures can therefore only move the other rows down toward Bifrost's, never Bifrost's row down further — the asymmetry this amendment corrects is one the publishing vendor's engine loses by. |
The decline that matters most is Bifrost's, and it is stated in the direction that costs this benchmark's publisher something. Bifrost has no warm figure here, and its cold median already includes its own process start — so whatever its warm marginal is, it lies between zero and a number already published. Adding warm figures can therefore only move the other rows down toward Bifrost's, never Bifrost's row down further. The conflation this section corrects is one the vendor's own engine benefits from, which is the reason to correct it in public rather than leave the cold chart to be read as a steady-state ranking.
These artifacts are auxiliary evidence, outside the freeze.
They are retained under reports/raw/warm-latency/ — the batch
series, the environment stamp, and the per-case evidence each batch produced,
so that a reader can check the batch did the real work rather than less of
it — but freeze/v1 does not digest them, and this amendment does
not extend it to. A warm number carries the pinned amendment commit
ccbcd788aabe…'s immutability for
its bytes and no stronger guarantee, exactly as the cold timing sidecars do.
No warm run wrote a normalized report, produced an outcome, or touched a
scored population; validate-reports and the freeze manifest never
read this directory.
Estimated per-invocation overhead
Section titled “Estimated per-invocation overhead”The warm marginal above answers what one more case costs a process already running, and only one adapter's released CLI let it be measured. This section answers a different question that every adapter can be asked: what does one invocation cost before it has anything to find? It is published under Amendment A24, preregistered — estimator, bias, tolerance, fixtures and presentation — before the first estimate existed.
The method, in one sentence: run one complete adapter
invocation — same pipeline, same committed policy, rule or query, same flags,
same subprocess shape, both subprocesses where the adapter has two — over a
trivial no-flow fixture that declares the benchmark's own source and
sink endpoints and never connects them, and take the runner's wall-clock
around it. The fixture is generated into a scratch workspace before the
process is spawned, exactly as every other fixture is, and is retained beside
the measurement. Nothing is added to cases/: no
population, denominator or freeze sees these files, and they are not cases.
What the number is biased by, in both directions. A trivial file is still parsed, still extracted, still queried, so the measurement is fixed overhead plus that near-zero analysis: it is an upper bound on start-up and warm-up, and the true fixed cost is at or below it. And it is a cold, single-shot execution — no warm JIT, no primed cache — which is exactly the posture the cold rows above are measured in, and exactly not the posture of a resident deployment, where both this figure and those rows would fall. It is an estimate, labelled one everywhere it appears, and it is never subtracted from a cold number, never substituted for one, and never used to order a row.
| Adapter | Fixture language | Estimated overhead (range over every repeat) | Width | Every repeat | Cold median, same kernel | Low end as a share of cold | Load observed | Marked on the chart? |
|---|---|---|---|---|---|---|---|---|
| Bifrost | python | 108 ms – 239 ms | 131 ms | 239 ms, 114 ms, 108 ms | 115 ms | 94% | 3.4 | yes |
| CodeQL | ruby | 4.02 s – 4.49 s | 475 ms | 4.49 s, 4.04 s, 4.02 s | 4.70 s | 86% | 3.6 to 3.8 | yes |
| FlowDroid | java | 678 ms – 771 ms | 93 ms | 771 ms, 768 ms, 678 ms | 1.55 s | 44% | 3.4 to 3.5 | yes |
| Infer | c | 288 ms – 491 ms | 203 ms | 491 ms, 297 ms, 288 ms | 540 ms | 53% | 3.3 | yes |
| Joern | java | 8.91 s – 13.2 s | 4.33 s | 8.91 s, 9.50 s, 13.2 s | 14.6 s | 61% | 3.7 to 3.9 | yes |
| Joern | php | 4.42 s – 5.34 s | 911 ms | 5.34 s, 4.45 s, 4.42 s | 5.45 s | 81% | 3.8 to 4.6 | yes |
| OpenTaint | kotlin | 4.08 s – 4.14 s | 60 ms | 4.14 s, 4.12 s, 4.08 s | 4.46 s | 91% | 3.7 to 4.2 | yes |
| Pysa | python | 2.82 s – 2.98 s | 165 ms | 2.82 s, 2.88 s, 2.98 s | 4.08 s | 69% | 3.5 to 3.6 | yes |
| Semgrep CE | kotlin | 1.04 s – 1.07 s | 28 ms | 1.07 s, 1.07 s, 1.04 s | 1.09 s | 95% | 3.3 | yes |
The published figure is a range, and its width is the measurement's precision. Each estimate is repeated a fixed number of times — the count is a constant in the runner's source, not a per-run choice — every repeat is retained and printed above, and what is published is the range those repeats span. Never a mean, never a chosen repeat, and never withheld because the repeats disagreed: a disagreement widens the range, which is the honest consequence of it, and there is no agreement threshold anywhere in this machinery to be justified or tuned — a unit test asserts that no such constant exists. This is the same convention the warm-marginal figures publish under, shared rather than re-derived. The build re-derives each range from the retained repeats and fails if it disagrees with what the runner wrote.
The last column is what decides a chart mark, and the threshold is preregistered. A row carries a dashed span only where its range starts at or above 25% of that adapter's cold median on the same kernel. The cut is relative because the chart's axis is logarithmic and its rows span two orders of magnitude, so a share of each row's own median is the same visual claim everywhere, where a fixed millisecond cut would mark every slow adapter and no fast one whatever its overhead actually was; and it reads the range's low end so that a mark can never appear on the strength of one slow repeat. An unmarked row is not an unmeasured one — every value is in the table above.
The load column is the machine's own condition, published rather than summarized as the word "quiet". Each repeat samples the one-minute load average immediately before its subprocess is spawned, and the column is the range those samples span, so a figure taken on a busy machine can be discounted instead of taken on trust. Two disciplines stand behind those numbers: no other analyzer under measurement competes for the machine, and — because nine heavy analyzers run back to back drive the load up by themselves — each adapter's measurement waits for the load to settle before it begins, so what a row records is the machine's state and not its own position in the sequence.
Joern is the one adapter where three figures can be set side by side, and they are three different things. Its estimated per-invocation overhead on the Java kernel is 8.91 s – 13.2 s — a measured upper bound, published across the range its repeats spanned. The fitted fixed cost, the intercept of the warm batch series on the same kernel, is 3.41 s – 3.59 s — not measured at all, but inferred from a line through five batches. And its measured warm marginal, the cost of one more case in a process already running, is 996 ms – 1.01 s. The first two are two routes to the same quantity and the third is a different quantity entirely.
Those first two routes disagree, by roughly a factor of three, and the disagreement is the informative part. Neither corrects the other and their difference is not published as a measurement, but the direction is exactly what the two constructions predict. The estimate is an upper bound that contains work the intercept excludes by construction: a whole trivial invocation — a fresh JVM, the Java front end loaded, a CPG built for the fixture, and the kernel script run over it — where the fitted intercept is what is left when a line through five batch sizes is extended back to zero cases, and every per-case cost the batch pays has been taken out of it. So the true once-per-process cost is bounded above by the estimate and approached from below by the intercept, and the gap between them is the part of "fixed cost" that neither construction can see on its own: not observable from inside one invocation, and not separable from per-case work inside a batch. That is the honest reading, and it is why this page publishes both figures rather than reconciling them into one.
Which adapters an estimate could be taken for
Each adapter is estimated in the language of its cheapest kernel arm — the core kernel whose cold median is that adapter's lowest — because that is the arm where fixed cost is the largest share of the number, and the arm whose trivial-fixture invocation is least dominated by analysis. Joern is additionally estimated on Java, so that A21's Java warm figures have a same-language estimate to be compared against; that second figure is labelled by its own language and is not the one the cheapest-arm rule selects.
| Adapter | Fixture language | Verdict | Evidence |
|---|---|---|---|
| Bifrost | pythonits cheapest kernel arm | Measured | 3 repeats — 239 ms, 114 ms, 108 ms — published as the 108 ms to 239 ms range they span. |
| CodeQL | rubyits cheapest kernel arm | Measured | 3 repeats — 4.49 s, 4.04 s, 4.02 s — published as the 4.02 s to 4.49 s range they span. |
| FlowDroid | javaits cheapest kernel arm | Measured | 3 repeats — 771 ms, 768 ms, 678 ms — published as the 678 ms to 771 ms range they span. |
| Infer | cits cheapest kernel arm | Measured | 3 repeats — 491 ms, 297 ms, 288 ms — published as the 288 ms to 491 ms range they span. |
| Joern | phpits cheapest kernel arm | Measured | 3 repeats — 5.34 s, 4.45 s, 4.42 s — published as the 4.42 s to 5.34 s range they span. |
| OpenTaint | kotlinits cheapest kernel arm | Measured | 3 repeats — 4.14 s, 4.12 s, 4.08 s — published as the 4.08 s to 4.14 s range they span. |
| Pysa | pythonits only kernel arm | Measured | 3 repeats — 2.82 s, 2.88 s, 2.98 s — published as the 2.82 s to 2.98 s range they span. |
| Semgrep CE | kotlinits cheapest kernel arm | Measured | 3 repeats — 1.07 s, 1.07 s, 1.04 s — published as the 1.04 s to 1.07 s range they span. |
| Joern | javanot its cheapest arm — measured so that A21's Java warm figures have a same-language estimate to be compared against | Measured | 3 repeats — 8.91 s, 9.50 s, 13.2 s — published as the 8.91 s to 13.2 s range they span. |
There is exactly one kind of missing figure here, and it is a fact about a machine.Environment means the pinned distribution is not installed where the estimator ran, so the invocation was never attempted — explicitly not a statement about the adapter's released CLI, unlike A15's capability declines above, and resolved by running the same committed command where the distribution is installed. There is no second kind: because the figure is a range over every repeat, noisy repeats produce a wide range rather than a withheld number, so nothing is ever missing for having been measured badly. Nothing is filled in by inference either — no estimate here is derived from an adapter's runtime, its architecture, another adapter's estimate, or the same adapter's estimate in another language.
These artifacts are auxiliary evidence, outside the freeze,
retained under reports/raw/invocation-overhead/: every repeat
with its phase split and the one-minute load average it was taken under, the
trivial fixture itself with its digest, the resolved configuration where the
adapter's is per-case, the published range, and the environment stamp.
freeze/v1 does not digest them, exactly as it does not digest the
warm artifacts or the cold timing sidecars, so an estimate carries the pinned
amendment commit ccbcd788aabe…'s
immutability for its bytes and no stronger guarantee. No estimator run wrote a
normalized report, produced an outcome, or touched a scored population.
Per-slice medians
Section titled “Per-slice medians”The tier's declared unit of aggregation: one row per slice — adapter × language × population — because a median over a whole adapter mixes languages whose fixtures differ in size and whose front ends differ in cost. Each row is one bound report. Rows are not ranked and columns are not compared across adapters with different granularity. One disclosure per adapter, for the same reason: the rows inside one of them are the rows that share a granularity.
Bifrost — 20 slices
Show the data table — 20 slices for Bifrost
| Slice | Language | Profile | Timed / bound | Median | Q1 | Q3 |
|---|---|---|---|---|---|---|
bifrost-smoke | 13 languages | benchmark-controlled | 117 / 118 | 117 ms | 113 ms | 123 ms |
bifrost-c-kernel | c | benchmark-controlled | 50 / 50 | 131 ms | 124 ms | 140 ms |
bifrost-cpp-kernel | cpp | benchmark-controlled | 56 / 56 | 130 ms | 127 ms | 136 ms |
bifrost-csharp-kernel | csharp | benchmark-controlled | 58 / 58 | 141 ms | 130 ms | 155 ms |
bifrost-go-kernel | go | benchmark-controlled | 58 / 58 | 123 ms | 116 ms | 128 ms |
bifrost-java-kernel | java | benchmark-controlled | 58 / 58 | 120 ms | 114 ms | 125 ms |
bifrost-java-modeling | java | benchmark-controlled | 8 / 24 | 144 ms | 126 ms | 151 ms |
bifrost-java-native | java | tool-native | 0 / 12 | n/a | n/a | n/a |
bifrost-javascript-kernel | javascript | benchmark-controlled | 58 / 58 | 117 ms | 112 ms | 121 ms |
bifrost-javascript-modeling | javascript | benchmark-controlled | 8 / 24 | 196 ms | 171 ms | 203 ms |
bifrost-javascript-native | javascript | tool-native | 0 / 12 | n/a | n/a | n/a |
bifrost-kotlin-kernel | kotlin | benchmark-controlled | 58 / 58 | 122 ms | 119 ms | 128 ms |
bifrost-php-kernel | php | benchmark-controlled | 58 / 58 | 122 ms | 116 ms | 129 ms |
bifrost-python-kernel | python | benchmark-controlled | 58 / 58 | 115 ms | 110 ms | 121 ms |
bifrost-python-modeling | python | benchmark-controlled | 8 / 24 | 181 ms | 141 ms | 289 ms |
bifrost-python-native | python | tool-native | 0 / 12 | n/a | n/a | n/a |
bifrost-ruby-kernel | ruby | benchmark-controlled | 58 / 58 | 122 ms | 117 ms | 125 ms |
bifrost-rust-kernel | rust | benchmark-controlled | 56 / 56 | 121 ms | 117 ms | 128 ms |
bifrost-scala-kernel | scala | benchmark-controlled | 58 / 58 | 133 ms | 125 ms | 147 ms |
bifrost-typescript-kernel | typescript | benchmark-controlled | 58 / 58 | 144 ms | 128 ms | 163 ms |
Semgrep CE — 17 slices
Show the data table — 17 slices for Semgrep CE
| Slice | Language | Profile | Timed / bound | Median | Q1 | Q3 |
|---|---|---|---|---|---|---|
semgrep-c-kernel | c | benchmark-controlled | 14 / 48 | 1.28 s | 1.21 s | 1.33 s |
semgrep-cpp-kernel | cpp | benchmark-controlled | 14 / 56 | 1.12 s | 1.10 s | 1.13 s |
semgrep-go-kernel | go | benchmark-controlled | 14 / 58 | 1.11 s | 1.09 s | 1.15 s |
semgrep-java-kernel | java | benchmark-controlled | 14 / 58 | 1.16 s | 1.13 s | 1.22 s |
semgrep-java-modeling | java | benchmark-controlled | 10 / 24 | 1.12 s | 1.09 s | 1.16 s |
semgrep-java-native | java | tool-native | 0 / 12 | n/a | n/a | n/a |
semgrep-javascript-kernel | javascript | benchmark-controlled | 14 / 58 | 1.31 s | 1.24 s | 1.33 s |
semgrep-javascript-modeling | javascript | benchmark-controlled | 10 / 24 | 1.14 s | 1.11 s | 1.17 s |
semgrep-javascript-native | javascript | tool-native | 0 / 12 | n/a | n/a | n/a |
semgrep-kotlin-kernel | kotlin | benchmark-controlled | 14 / 58 | 1.09 s | 1.08 s | 1.58 s |
semgrep-php-kernel | php | benchmark-controlled | 14 / 58 | 1.69 s | 1.61 s | 1.82 s |
semgrep-python-kernel | python | benchmark-controlled | 14 / 58 | 1.10 s | 1.09 s | 1.12 s |
semgrep-python-modeling | python | benchmark-controlled | 10 / 24 | 1.63 s | 1.18 s | 1.85 s |
semgrep-python-native | python | tool-native | 12 / 12 | 4.10 s | 3.88 s | 4.20 s |
semgrep-ruby-kernel | ruby | benchmark-controlled | 14 / 58 | 2.53 s | 2.37 s | 2.73 s |
semgrep-rust-kernel | rust | benchmark-controlled | 14 / 54 | 1.35 s | 1.22 s | 1.52 s |
semgrep-typescript-kernel | typescript | benchmark-controlled | 14 / 58 | 1.11 s | 1.09 s | 1.18 s |
FlowDroid — 2 slices
Show the data table — 2 slices for FlowDroid
| Slice | Language | Profile | Timed / bound | Median | Q1 | Q3 |
|---|---|---|---|---|---|---|
flowdroid-java-kernel | java | benchmark-controlled | 58 / 58 | 1.55 s | 1.49 s | 1.61 s |
flowdroid-kotlin-kernel | kotlin | benchmark-controlled | 58 / 58 | 1.97 s | 1.84 s | 2.05 s |
Infer — 3 slices
Show the data table — 3 slices for Infer
| Slice | Language | Profile | Timed / bound | Median | Q1 | Q3 | capture median
| analyze median
|
|---|---|---|---|---|---|---|---|---|
infer-c-kernel | c | benchmark-controlled | 48 / 48 | 540 ms | 494 ms | 612 ms | 159 ms | 372 ms |
infer-cpp-kernel | cpp | benchmark-controlled | 56 / 56 | 728 ms | 550 ms | 2.99 s | 234 ms | 413 ms |
infer-java-kernel | java | benchmark-controlled | 58 / 58 | 6.47 s | 6.15 s | 6.77 s | 6.09 s | 349 ms |
Pysa — 1 slice
Show the data table — 1 slice for Pysa
| Slice | Language | Profile | Timed / bound | Median | Q1 | Q3 |
|---|---|---|---|---|---|---|
pysa-python-kernel | python | benchmark-controlled | 58 / 58 | 4.08 s | 3.88 s | 4.21 s |
OpenTaint — 2 slices
Show the data table — 2 slices for OpenTaint
| Slice | Language | Profile | Timed / bound | Median | Q1 | Q3 |
|---|---|---|---|---|---|---|
opentaint-java-kernel | java | benchmark-controlled | 58 / 58 | 4.70 s | 4.55 s | 9.25 s |
opentaint-kotlin-kernel | kotlin | benchmark-controlled | 58 / 58 | 4.46 s | 4.26 s | 4.74 s |
Joern — 12 slices
Show the data table — 12 slices for Joern
| Slice | Language | Profile | Timed / bound | Median | Q1 | Q3 |
|---|---|---|---|---|---|---|
joern-java-kernel | java | benchmark-controlled | 58 / 58 | 14.6 s | 14.3 s | 15.3 s |
joern-java-modeling | java | benchmark-controlled | 16 / 24 | 9.29 s | 8.95 s | 9.91 s |
joern-java-native | java | tool-native | 0 / 12 | n/a | n/a | n/a |
joern-javascript-kernel | javascript | benchmark-controlled | 58 / 58 | 9.15 s | 8.86 s | 9.61 s |
joern-javascript-modeling | javascript | benchmark-controlled | 16 / 24 | 5.28 s | 5.07 s | 5.40 s |
joern-javascript-native | javascript | tool-native | 0 / 12 | n/a | n/a | n/a |
joern-php-kernel | php | benchmark-controlled | 58 / 58 | 5.45 s | 5.12 s | 8.39 s |
joern-python-kernel | python | benchmark-controlled | 58 / 58 | 8.38 s | 8.00 s | 8.86 s |
joern-python-modeling | python | benchmark-controlled | 16 / 24 | 4.95 s | 4.89 s | 5.03 s |
joern-python-native | python | tool-native | 0 / 12 | n/a | n/a | n/a |
joern-ruby-kernel | ruby | benchmark-controlled | 58 / 58 | 8.63 s | 7.35 s | 11.0 s |
joern-rust-kernel | rust | benchmark-controlled | 54 / 54 | 7.41 s | 7.05 s | 8.31 s |
CodeQL — 17 slices
Show the data table — 17 slices for CodeQL
| Slice | Language | Profile | Timed / bound | Median | Q1 | Q3 | database-create median
| database-analyze median
|
|---|---|---|---|---|---|---|---|---|
codeql-c-kernel | c | benchmark-controlled | 50 / 50 | 6.15 s | 5.42 s | 7.08 s | 1.55 s | 4.62 s |
codeql-cpp-kernel | cpp | benchmark-controlled | 56 / 56 | 7.28 s | 6.19 s | 19.1 s | 1.89 s | 5.40 s |
codeql-csharp-kernel | csharp | benchmark-controlled | 58 / 58 | 33.1 s | 24.1 s | 39.7 s | 20.1 s | 13.0 s |
codeql-go-kernel | go | benchmark-controlled | 58 / 58 | 9.43 s | 9.24 s | 9.81 s | 4.40 s | 5.05 s |
codeql-java-kernel | java | benchmark-controlled | 58 / 58 | 12.7 s | 12.5 s | 12.9 s | 6.70 s | 6.00 s |
codeql-java-modeling | java | benchmark-controlled | 24 / 24 | 12.3 s | 11.2 s | 13.2 s | 6.40 s | 5.58 s |
codeql-java-native | java | tool-native | 12 / 12 | 22.6 s | 17.3 s | 25.9 s | 7.77 s | 14.8 s |
codeql-javascript-kernel | javascript | benchmark-controlled | 58 / 58 | 10.7 s | 9.80 s | 13.5 s | 4.49 s | 6.12 s |
codeql-javascript-modeling | javascript | benchmark-controlled | 24 / 24 | 13.3 s | 8.82 s | 13.8 s | 5.59 s | 7.76 s |
codeql-javascript-native | javascript | tool-native | 12 / 12 | 21.7 s | 21.4 s | 22.0 s | 5.68 s | 16.1 s |
codeql-kotlin-kernel | kotlin | benchmark-controlled | 58 / 58 | 15.8 s | 15.6 s | 17.0 s | 10.9 s | 4.85 s |
codeql-python-kernel | python | benchmark-controlled | 58 / 58 | 5.38 s | 5.30 s | 5.83 s | 2.00 s | 3.41 s |
codeql-python-modeling | python | benchmark-controlled | 24 / 24 | 9.85 s | 9.72 s | 9.98 s | 3.66 s | 6.15 s |
codeql-python-native | python | tool-native | 12 / 12 | 12.6 s | 12.4 s | 12.9 s | 3.70 s | 8.86 s |
codeql-ruby-kernel | ruby | benchmark-controlled | 58 / 58 | 4.70 s | 4.58 s | 4.89 s | 1.34 s | 3.33 s |
codeql-rust-kernel | rust | benchmark-controlled | 56 / 56 | 47.1 s | 45.5 s | 49.2 s | 20.9 s | 25.7 s |
codeql-typescript-kernel | typescript | benchmark-controlled | 58 / 58 | 8.92 s | 8.09 s | 9.80 s | 3.73 s | 5.04 s |
Cases with no latency number
Section titled “Cases with no latency number”Of the case results in this bound latency corpus, 2657 invoked an analyzer and carry a timing sidecar. 679 do not, and every one of them is accounted for:
| Outcome of the untimed case | Cases |
|---|---|
unsupported | 679 |
An unsupported case is decided from case metadata
before the analyzer is invoked, so there is no subprocess and there
is nothing to time. Those cases are absent from every distribution
on this page, not entered as zero — entering them as zero would
make the adapters that decline the most look the fastest, which is exactly
backwards. Every case that did invoke an analyzer is timed; the timed and
untimed columns account for the whole bound population with no remainder.
Provenance and what the freeze does and does not guarantee
Section titled “Provenance and what the freeze does and does not guarantee”
Every cold number above was derived at build time from
v0.6.0's archived latency-evidence bundle. The bundle
was generated from that release commit's retained timing sidecars, and its
population comes from the results model generated by freeze manifest
3228af686d09…. Snapshot pages select this
bundle explicitly: advancing the current correctness freeze cannot add a
case, replace a timing, or relabel an analyzer on a historical page.
The warm and invocation-overhead amendments are likewise read from a
checked-in auxiliary bundle pinned to commit
ccbcd788aabec2abe60200573f38bc42128d00f0; they never read the mutable
working-tree artifact directories.
One limit, stated rather than left to be assumed. The
freeze/v1 manifest binds one raw-evidence digest per result; the
timing sidecar and the environment stamp are additive files beside
that evidence and are not themselves digested by the manifest. So a
number on this page carries the freeze's guarantee that the run it belongs
to is bound and byte-verified, and the release commit plus the checked-in
archive's own Git history preserve the timing bytes selected for rendering.
It does not carry a freeze-manifest digest over those timing bytes.
Extending the manifest to bind them is a freeze/v2 question,
and it is named here rather than smuggled in under a schema version that
does not describe it.