SystemsLab

Appearance

SystemsLab 153.0.0 ​

Added ​

  • A systemslab view <export> command opens a context export (the .zip/.tar/directory produced by systemslab export context) and serves the familiar SystemsLab dashboards for it locally — no server, database, or network required. It serves the same web UI and GraphQL schema the full server does, resolved read-only from the export, and runs entirely offline. Pass --open to launch a browser, --listen to choose an address (default 127.0.0.1:0), and --workdir to control where the export is unpacked.
  • A durable.reserved_database_connections config option (default 8) reserves connections in the durable worker's pool for its own bookkeeping. The worker now bounds concurrent task execution so a burst of ready tasks cannot consume every connection and starve the heartbeat, leader election, cleanup, and task-claiming queries that keep the worker alive.
  • New MCP tools let an agent build a context incrementally: create_context opens a new context, attach_experiments appends experiments (with tags/params) and can be called repeatedly, and seal_context freezes it for execution.
  • A get_experiment_spec MCP tool returns the experiment specification captured on submit, for a single experiment or for every experiment in a context, so an agent can recover how a run was configured.
  • An artifact_schema MCP tool returns the column/series schema (name, type, nullability, metadata) of a parquet artifact so an agent can discover what it can query before reading it.
  • Summary tables now support multi-column sorting: plain click sorts by one column, shift-click adds secondary/tertiary sorts that keep rows grouped across axes, and the header shows a priority number when more than one sort is active.
  • Home-page context rows now show per-state experiment counts as colored badges (running / pending / success / failure / error / timeout / cancelled) instead of a single total, and these counts update live as experiments progress.
  • Experiment and context pages now have a per-entity bell toggle to opt in to a browser notification when that experiment or context reaches a terminal state while the page is open.
  • PromQL metric series are now labeled with experiment_id, job_id, and job (the job name, e.g. server/client) in addition to artifact_id, so queries can slice per job (cpu_usage{job="server"}) and span experiments. This applies retroactively to existing artifacts with no re-ingest.
  • A job_dir agent config option (and --job-dir flag) relocates per-job scratch space (working directory, recorded metrics, event log) off the system temp directory, which is typically a tmpfs ramdisk, onto real disk. The packaged agent now defaults job_dir to /var/cache/systemslab-agent; unset, it falls back to the system temp directory as before.

Changed ​

  • An agent with no cluster_id of its own now adopts the cluster id reported by the server at registration, so its telemetry (and the SYSTEMSLAB_CLUSTER_ID passed to runner processes) is tagged with the same cluster as the server it connects to.

Fixed ​

  • Cancelling a running experiment now reliably kills workloads that escaped their process group (via setsid/double-fork/nohup) or ran in shell-action sub-cgroups. Since the previous release the runner only SIGKILLed tracked process groups, so such processes survived for up to several minutes until the agent tore down the cgroup tree; the runner again SIGKILLs the whole job cgroup when one is configured.
  • Experiments could get stuck in pending forever after the scheduler's Postgres LISTEN/NOTIFY connection briefly dropped and silently lost a host_updated notification. The scheduler now resynchronizes its host view on listener reconnect, and tolerates work-queue rows deleted concurrently (cancel/timeout/completion/pause) instead of tearing down and restarting.
  • The durable service no longer stays silently dead after its worker exits on its own (e.g. after a transient DB pool timeout), which had left scheduled runs undispatched and hosts stuck in assigned; the worker's exit is now observed so the supervisor restarts it.
  • A run that completes after its host was marked lost (e.g. during host teardown) now records its real terminal outcome instead of staying mislabeled lost: the runner's authoritative lost -> terminal PATCH is accepted as an idempotent upgrade rather than rejected with a 400.
  • DELETE /api/v1/experiment/{id} no longer returns 500 for experiments whose metrics had been imported; the referencing metrics-metadata rows are now cleaned up automatically on delete.
  • GET /api/internal/host/{id}/status now returns 502 Bad Gateway when the target agent is unreachable and 504 Gateway Timeout when it times out, instead of 500.
  • GET /artifact/{id}/parquet-metadata and /artifact/{id}/schema now return 4xx instead of 500 when the backing parquet object is missing (410) or its recorded size is NULL or too small to be a valid parquet file.
  • Context KPIs no longer disappear once the KPI cache is populated: non-finite (NaN) KPI values now round-trip correctly through the JSONB cache instead of failing every cached read.
  • Running an MCP tool against an auth-proxied server with no cached credentials now returns a clear message pointing to systemslab login instead of an opaque JSON-parse error from the proxy's HTML login page.