Agent-Driven Analysis with SystemsLab

In this tutorial we'll walk through how an AI agent drove SystemsLab end to end: it wrote a parameterized experiment, launched a concurrency sweep, pulled the resulting telemetry, and analyzed it — catching two of its own mistakes along the way. The running example is a real investigation into how a vLLM server's prefix cache behaves as client concurrency climbs.

Like the Redis tutorial, this is not meant to be a perfect analysis. It's meant to show the shape of an agentic workflow on SystemsLab: the CLI surface an agent uses to issue experiments, and the discipline — unit sanity checks, cross-validation against canonical results — that keeps an agent's conclusions honest.

The concrete question the agent set out to answer:

How do the prefix-cache hit rate and GPU metrics of a vLLM server respond to client concurrency? vLLM serving Qwen/Qwen3-4B on a single NVIDIA L4, swept across client concurrency c = 12 → 32.

Prerequisites

To reproduce this kind of workflow you will need:

• The SystemsLab CLI installed locally.
• A SystemsLab server you can submit to.
• Two agents (hosts) registered with the server:
  • A GPU host — here an NVIDIA L4 — with vllm installed. We'll assume it carries a gpu tag.
  • A load-generation host with a chat load generator that can replay a prompt dataset (OpenOrca) at a fixed concurrency. We'll assume it carries a loadgen tag.
• An AI agent wired to SystemsLab's MCP server (see below), or one that can simply shell out to the systemslab CLI. Either way, the agent is the actor driving every step in this tutorial.

Make sure the host tags match your setup, or your experiments will sit queued forever waiting for a host that never appears.

Wiring the agent to SystemsLab

An agent can drive SystemsLab two ways, and this tutorial uses both:

1. Issuing experiments — the agent runs systemslab CLI commands (submit, sweep, artifact download-all, …) exactly as a human would.

2. Talking to the server programmatically — systemslab mcp exposes SystemsLab to any MCP client (Claude, Cursor, your own agent) as a set of tools for listing experiments, reading reports, and fetching artifacts. Point your MCP client at:

   // MCP client config
   {
     "mcpServers": {
       "systemslab": { "command": "systemslab", "args": ["mcp"] }
     }
   }

   See the systemslab mcp reference for details.

Getting Started

As with any SystemsLab workflow, first tell the CLI where the server lives. Create a .config/systemslab.toml in your working directory (or any parent), replacing <url>:

systemslab_url = "<url>"

INFO

If you'd rather not create a config file, pass --systemslab-url <url> to any command. An agent shelling out to the CLI can also set the SYSTEMSLAB_URL environment variable once instead of threading the flag through every call.

A quick refresher on the vocabulary we'll use:

• An experiment is one benchmark run — here, one concurrency level.
• A job is a sequence of steps that runs on one host. This experiment has two: the vLLM server and the loadgen.
• A sweep launches many experiments from a single parameterized spec — one per concurrency level.
• A context groups related experiments so SystemsLab can build cross-experiment reports. Our sweep produces one context holding all 21 levels — the unit the agent ultimately analyzed.

Writing the Experiment Specification

Because we want to sweep concurrency, the spec is a parameterized jsonnet file: its root is a function taking the swept parameter c. (Parameterization is what makes a spec sweepable — see the experiment reference.)

We'll keep this terse; the structure mirrors the Redis tutorial (two jobs synchronized with barriers, artifacts uploaded at the end). Save it as experiment.jsonnet:

local systemslab = import 'systemslab.libsonnet';

// `c` is supplied per-experiment by the sweep file.
function(c=16) {
  name: 'vllm qwen3-4b chat @ c=%d' % c,
  // Recorded on the experiment so reports and downloads can key off it.
  metadata: { concurrency: c },
  jobs: {
    // The vLLM inference server.
    server: {
      host: { tags: ['gpu'] },
      steps: [
        // Start vLLM in the background and wait until it is serving.
        systemslab.bash('vllm serve Qwen/Qwen3-4B --port 8000', background=true),
        systemslab.bash('until curl -sf localhost:8000/health; do sleep 1; done'),

        systemslab.barrier('server-ready'),  // tell loadgen we're up
        systemslab.barrier('load-done'),     // wait for loadgen to finish

        // vLLM engine metrics, captured as parquet.
        systemslab.upload_artifact('vllm-metrics.parquet'),
      ],
    },

    // The chat load generator.
    loadgen: {
      host: { tags: ['loadgen'] },
      steps: [
        systemslab.barrier('server-ready'),

        // Replay OpenOrca prompts at concurrency `c`; write a canonical summary.
        systemslab.bash(
          'chat-loadgen --target server.systemslab.internal:8000 '
          + '--dataset openorca --concurrency %d --output results.json' % c,
        ),

        systemslab.barrier('load-done'),
        systemslab.upload_artifact('results.json'),
      ],
    },
  },
}

Two things worth calling out for the analysis later:

• results.json is the load generator's own canonical summary — the source of truth for customer-facing KPIs (throughput, TTFT, TPOT).
• GPU and host telemetry is captured for you. SystemsLab's built-in Rezolus telemetry records GPU utilization, power, clocks, and host metrics as parquet on every host — the agent doesn't add a step for it, it just downloads the artifacts afterward.

Launching the Sweep

The agent varies only one parameter, so the sweep file is a one-line matrix. Save it as sweep.json:

{ "matrix": { "c": { "start": 12, "stop": 32, "step": 1 } } }

That expands to one experiment per integer concurrency from 12 to 32 — 21 in all. Launch them, grouped into a named context, with a single command:

systemslab sweep --name 'monolithic-chat-knee-fine' experiment.jsonnet sweep.json

--name creates a fresh context holding all 21 experiments and marks it complete once the sweep finishes, so SystemsLab builds the cross-experiment report the agent will read. (To fold a sweep into an existing context instead, pass --context <id>.)

TIP

Sweeps can generate a lot of experiments — the cartesian product grows fast when you vary more than one parameter. systemslab sweep refuses to launch more than --limit-experiments (default 10000) as a guardrail.

Letting the Agent Pull the Data

Once the context is complete, the agent pulls every artifact for all 21 levels in one shot, organized into per-experiment directories:

systemslab artifact download-all --context <context-id> -o ./data

This lands a vllm-metrics.parquet, a Rezolus GPU/host parquet, and a results.json under each experiment's folder. The agent can discover the context id from systemslab context list (or the MCP list_contexts tool) and the per-level artifacts from systemslab artifact list.

With the data local, the agent wrote a single analysis script (sweep.py) that auto-discovers each c<N>/ directory and aggregates, per level: block- and token-level prefix-cache hit rates, KV-pool fill, scheduler queue depths (running/waiting), and GPU compute/memory-bandwidth utilization, power, and clock.

The Agent's Analysis

This is where an agentic workflow earns its keep — not in producing numbers, but in distrusting them. Two self-checks materially changed the results:

1. A unit-sanity block fired on the first run. GPU power came out at 42,820 (an L4 caps at 72 W) and the clock at 2.9×10⁹ (Hz, not MHz), and the script was averaging the wrong clock domain (video instead of the SM compute clock). After fixing units, filtering to the SM clock, and restricting aggregation to the steady-state middle 80% of each run, the numbers passed sanity.
2. A cross-check against results.json caught an averaging-window bug. The parquet-derived absolute rates had been averaged over the whole ~1019 s capture even though load only ran ~240 s post-warmup — deflating them ~4×. Ratios (hit %, utilization %) were unaffected, but the customer-facing KPI table was rebuilt from the canonical results.json for all 21 levels.

The final story is therefore cross-validated by two independent sources — raw telemetry and the harness's own summaries — and they agree.

The Data

Prefix cache + GPU vs. concurrency (steady-state telemetry)

c	hit %	cached tok %	kv %	run	wait	GPU %	gmem %	W	MHz
12	84.1	84.1	0.0	3.8	0.1	31.9	28.8	42.3	1856
16	81.7	81.7	0.0	5.3	0.2	33.2	29.4	42.8	1852
18	78.8	81.0	0.0	5.8	0.3	32.8	29.4	42.8	1836
20	50.4	70.4	0.0	6.3	0.5	32.9	27.8	42.6	1828
23	28.8	60.5	0.0	7.0	0.9	32.5	25.0	43.6	1806
24	14.5	50.8	0.1	7.1	1.1	33.1	25.3	42.7	1810
28	2.9	4.7	0.0	7.6	1.8	32.7	20.9	42.7	1785
32	0.0	0.0	0.0	7.8	3.1	33.0	21.7	42.6	1784

(Selected rows from the 21-level table. hit % = block-level prefix-cache hit rate; kv % = vLLM KV-pool fill; run/wait = mean requests running/waiting; gmem % = memory-bandwidth utilization.)

Canonical KPIs (from results.json)

c	out tok/s	TTFT p50 (ms)	TTFT p90	TTFT p99	TPOT p50 (ms)
12	182	296	1,099	10,201	61
15	193	302	1,149	12,885	71
17	188	323	1,191	15,099	77
18	176	344	5,536	16,106	79
21	180	359	6,208	18,656	88
23	145	440	10,872	21,206	109
24	148	4,563	17,448	22,817	129
28	110	9,731	20,133	26,441	168
32	124	16,106	30,736	35,433	159

The Insights

1. The cache collapses — and it is not a capacity problem

The hit rate shows three clean regimes: a plateau at ~82% (c=12–18), a cliff from 79% down to ~20% (c=19–27, with one lucky outlier at c=21), and a floor near 0% (c=28–32). Yet kv % is essentially zero across the entire sweep and preemptions never exceed 0.01/s — no blocks are being evicted under pressure. vLLM has ~21.5 GB committed to the KV pool and barely engages any of it.

The mechanism is concurrent racing: the OpenOrca prompts share long prefixes, but a hit only registers after another request's prefill has committed its blocks. As concurrency rises, more identical-prefix requests are in flight simultaneously, so they all miss and each pays full prefill from scratch. Sequential warmth turns into parallel coldness. Token-level cache % decays more gently than block-level, since long shared prefixes still partially hit.

2. The GPU loafs through the entire sweep

Compute utilization is flat at ~32–33% from c=12 to c=32; power holds at 42–44 W against a 72 W TDP; the SM clock drifts down a negligible ~5%. The only GPU metric that moves is memory-bandwidth utilization, which decreases from ~29% to ~21% as concurrency rises — the inverse of a healthy scaling curve, and confirmation of insight 1: fewer cache hits mean more redundant prefill per useful output token, so productive HBM traffic falls even as pending work grows.

Meanwhile server-side running saturates near 8 regardless of client concurrency — a vLLM scheduler batch cap (max_num_seqs or similar), not a hardware wall.

3. Latency mirrors the cache regimes almost exactly

Regime	c range	TTFT p50	Hit %	What's happening
Cheap	12–17	~300–330 ms	~82%	Cache hits, light queueing, GPU loafing
Wobble	18–23	344–440 ms	80→29%	Hit rate sliding; tail (p90/p95) breaks before the median
Cliff	24–32	4.6 s → 16.1 s	14→0%	Cache dead; full prefill × growing queue — classic saturation queueing

The step from c=23 to c=24 is the punchline: a small change in inputs (mean waiting 0.9→1.1, hit rate 28%→14%) produces a 10× jump in median TTFT, because uncached-prefill cost and queue depth spike together. TTFT p99 tells the same story louder — 10 s at c=12, 35 s at c=32 — and TPOT climbs 61→186 ms p50, a gentler queue/batching effect since decode is bandwidth-bound on a GPU with headroom to spare.

Conclusion

Operating point: target c ≈ 15–17. That's peak output throughput (~190 tok/s) at sub-second median TTFT. Above c=23 the system is strictly worse on every customer-visible KPI — less throughput, dramatically worse latency — while the GPU sits at a third of its capacity the entire time.

The bottleneck is software, not silicon:

• The prefix cache's collapse under concurrent identical-prefix racing is the primary driver of both the throughput slide and the TTFT cliff.
• The scheduler cap near 8 running sequences prevents the GPU from ever being pushed; raising it would let the hardware work harder before the cache race ruins things.
• Worth checking whether vLLM's in-flight prefill sharing (letting concurrent identical-prefix requests share a pending prefill rather than racing it) is available and enabled.

Every related context on the server topped out at c=32, so extending the curve beyond the observed floor would require the agent to commission a new sweep — another systemslab sweep with a wider range.

Making Agent-Driven Analysis Trustworthy

The analysis above is only as good as the checks the agent ran against itself. The transferable lessons, for whatever you point an agent at next:

• Assert on units and physical bounds. A cheap sanity block ("L4 power must be ≤ 72 W") caught a units bug and a wrong-clock-domain bug on the very first run.
• Cross-validate against the canonical summary. Absolute rates derived from raw telemetry disagreed with results.json by ~4× because of an averaging-window bug. When the harness already emits a source-of-truth KPI file, prefer it — and use the raw parquet to explain, not to restate.
• Aggregate over steady state. Windowing to the middle 80% of each run (dropping warmup and drain) is what made cross-run comparison meaningful.
• Keep the script re-runnable and self-discovering. sweep.py auto-detects new c<N>/ directories, so extending the sweep and re-analyzing is a two-command loop: systemslab sweep … then re-run the script.

INFO

Reproducibility: sweep.py re-runs in under a second and auto-discovers new c<N>/ data directories; the KPI table regenerates directly from each experiment's results.json.