SystemsLab 165.0.0

Added

• Jobs whose host stops making progress are now marked lost and retried, even if the host's agent still responds. Previously a host that stalled (e.g. by running out of memory) held its job until the job timed out. The runner sends a heartbeat to POST /api/v1/job/{id}/{run}/heartbeat every 30 seconds, and the controller marks a running job lost once its heartbeat is older than the new run_heartbeat_timeout server option (default 10 minutes, 0 disables it). If the runner recovers afterwards, it stops the job. Jobs run by older runners, which send no heartbeats, are never timed out this way.
• Experiments that no registered host could ever run now end in the timeout state after 6 hours instead of waiting out the 7-day queue timeout. This covers a job asking for a tag no host has, and work whose hosts went away after it was submitted. Locked, draining and drained hosts still count, so maintenance does not trigger it. The new unschedulable_timeout server option (--unschedulable-timeout) sets the limit in seconds; queue_timeout remains the overall limit.
• Workspaces: named, shareable collections that group related work. POST /api/v1/workspace creates one with a slug (e.g. allocator-sweep), a name and a description, and GET, PATCH and DELETE /api/v1/workspace/{slug or id} read, change and archive it. Archiving a workspace frees its slug.
• PUT and DELETE /api/v1/workspace/{workspace}/items/{id} add a context or notebook to a workspace and take it out again, and GET /api/v1/workspace/{workspace}/items lists what a workspace holds. An item can be in any number of workspaces, and removing it from one never deletes it.
• POST /api/v1/context and POST /api/v1/submit take workspaces: [{"slug": ...}] to file the new context into workspaces in the same transaction that creates it. An unknown workspace fails the request instead of leaving the context unfiled.
• GraphQL has workspaceBySlug, workspaceById and workspaces(search, archived). A Workspace has its contexts, contextCount, notebookCount and createdBy, and Context.workspaces lists the workspaces a context is in.
• systemslab workspace manages workspaces from the command line: create, list, show, add, remove and archive. add and remove take any mix of context and notebook ids.
• systemslab submit, sweep, optimize, context new, import context and import rezolus take --workspace <slug> (repeatable) to add the context they create to workspaces. SYSTEMSLAB_WORKSPACE or workspace = "<slug>" in the CLI config file sets a default, and --no-workspace skips it. The CLI's MCP submit tool takes a workspace argument with the same default.
• MCP tools list_workspaces, get_workspace (items plus the readme notebook as markdown), create_workspace and add_to_workspace, and a workspace argument on create_context.
• The web UI lists workspaces at /workspaces, where they can be searched and created, and shows each one at /w/<slug>: its description, its notebooks with the readme first, and its contexts with their status. Contexts and notebooks can be added by pasting their ids, and removed.
• Workspace facets: PUT and DELETE /api/v1/workspace/{workspace}/facets/{key} declare and remove the keys a workspace groups its contexts by, and GET /api/v1/workspace/{workspace}/facets lists them. A label facet's values are set by hand with PATCH /api/v1/workspace/{workspace}/items/{context}/labels; a param facet takes its values from the experiment parameter of the same name. Labels belong to the workspace, and unknown keys are rejected unless the request passes declare: true. Workspace items now include their labels.
• GraphQL Workspace.facets lists a workspace's facets with each value and how many contexts have it, Workspace.contextFacetValues gives each context's values, and Workspace.contexts(facets: [{key, values}]) filters contexts by facet (values within a facet are ORed, facets are ANDed). A param facet matches a context if any of its experiments has the value.
• The workspaces targets on POST /api/v1/context and POST /api/v1/submit take labels (and declare), so a new context can be labelled in the same transaction that creates and files it.
• systemslab workspace facet <workspace> list|add|remove manages a workspace's facets (add --from-param for parameter facets, --from-label to switch back), and systemslab workspace label <workspace> <context> KEY=VALUE KEY- sets and removes labels. Commands that create a context take --label KEY=VALUE to label it in the workspaces it is filed into, and workspace show lists each item's labels.
• MCP: get_workspace returns the workspace's facets and each context's labels, a new label_workspace_context tool sets and removes labels, and create_context and the CLI's submit tool take labels.
• The workspace page has a facet rail: pick facet values to filter its contexts, and group the contexts table by a facet. The filters and grouping live in the URL (?allocator=jemalloc&_group=phase), so a filtered view can be shared, and each context shows its labels.
• The workspace page has a Follow button. Without a signed-in user, follows are kept in the browser and the page says so.
• GET /api/v1/search finds workspaces and notebooks as well as contexts and experiments, and matches free text against context and workspace descriptions and notebook titles too. ws:<slug> (repeatable) limits it to the items of a workspace, <key>=<value> filters by that workspace's facets (quote a value with spaces: phase="release candidate"), and is: may now be repeated. Results carry a workspace's slug, and state is left out for workspaces and notebooks.
• PUT and DELETE /api/v1/workspace/{workspace}/follow follow and unfollow a workspace as the signed-in user, and GET /api/v1/workspace takes following=true and mine=true to list the workspaces the user follows or created. These need a trusted identity and return 401 without one.
• GET /api/v1/search understands in:followed, which limits it to the workspaces the signed-in user follows and their items, and by:me, which limits it to what the user created.
• systemslab workspace follow and unfollow follow and unfollow a workspace, and systemslab workspace list takes --following and --mine.
• The web UI shows which idle hosts the scheduler is holding for a queued higher-priority experiment. On the hosts page a held host's state reads held instead of idle, with the holding experiment, its priority, and how long the hold has lasted underneath, and held hosts are counted separately from idle ones. A pending experiment's page lists the hosts it is holding. The server state of a held host is still idle: the hold only keeps lower-priority work off it.
• GraphQL Host has heldByExperiment, heldByExperimentName, holdPriority and heldSince, and the host query takes a heldByExperiment filter. GET /api/v1/host returns the same fields as held_by_experiment, held_by_experiment_name, hold_priority and held_since. The MCP list_hosts tool reports the holding experiment and its priority. The scheduler publishes its holds to a new host_hold table whenever they change.
• The job logs page has a Follow button that keeps the newest line in view as output arrives. Scrolling up turns it off, and scrolling down to the end turns it back on. The page still opens at the first line.
• systemslab logs --follow streams new output until the job finishes, like tail -f. --for 60s stops following after the given time. With --tail N, the last N lines written so far are printed first.
• GET /api/v1/job/{id}/{run}/events takes follow=false to return only the events written so far instead of waiting for the run to finish.
• GET /api/v1/job/{id}/{run}/events and /events2 take ?tail=N, which starts the stream at the last N events logged so far, like tail -f. Event ids stay ordinals in the full log, so a tailed stream's first id can be non-zero. /events2 messages now carry the ordinal as id. A running job starts at its tail only on agents from this release on; older agents stream the whole log.
• The server can now identify users from the headers set by an authenticating proxy such as oauth2-proxy. Enable it with the new [identity] config section; headers are only trusted from the addresses in identity.trusted_proxies (loopback by default). Users are created the first time they are seen, and GET /api/v1/user/me returns the current user, or null when the request has no trusted identity.
• With [identity] enabled, contexts, experiments, reports and report templates record the user who created them, and the REST API returns it as created_by (report templates also return updated_by). This covers the REST API and the MCP create_context tool. Instance request approvals record the verified user alongside the caller-supplied actor. Without a trusted identity these fields are null.
• GraphQL: contexts, experiments, reports and report templates have a createdBy field that resolves to the new User type (report templates also have updatedBy), and the context and experiment queries accept a createdBy user id to list one user's entities.
• EC2 virtual tags are managed through the API instead of the server config. POST /api/v1/infra/virtual-tags creates a tag, PUT /api/v1/infra/virtual-tags/{name} edits one, and /rollback, /disable and /enable under the same path restore an earlier revision or retire a tag. Every change is kept as a revision, listed by GET /api/v1/infra/virtual-tags/{name}/revisions, with who made it. Changes apply without a restart, and by default an edit retires idle instances launched from the old revision. POST /api/v1/infra/virtual-tags/validate checks a tag's AMI and instance type against AWS without saving it, and GET /api/v1/infra/launch-profiles lists the profiles a tag can use.
• The GraphQL context query takes a status filter, and systemslab context list takes --status, to list contexts by the overall status shown on their badge (success, pending, failure, error, cancelled or empty).
• The GraphQL experiment query and GET /api/v1/experiment take a running filter for experiments that do or do not have a running job. systemslab experiment list --state accepts running.
• Notebooks: documents that mix prose with live charts, KPI tables and mentions of experiments and contexts. /notebooks lists them and /notebook/{id} shows one, with an in-place editor that autosaves and a history page that can restore any revision. Experiment and context pages list the notebooks that mention them.
• Notebooks can be read and written as markdown or as a JSON block tree through /api/v1/notebook and the MCP notebook tools (create_notebook, edit_notebook_blocks and others). Every update and block edit takes a base_revision and fails with 409 if the notebook has changed since, so an agent and a person editing the same notebook cannot overwrite each other.
• systemslab notebook list, pull and push edit notebooks as local markdown files with YAML front matter.
• The web UI has an experiment queue page at /queue that lists unfinished experiments, split into running and queued, with how long each has waited. The nav link that opened the contexts list filtered to pending contexts is now "Experiment Queue" and opens this page.
• The admin infra page has an Available Instances section listing each virtual tag's instance type, market, AMI, instance cap and current on-demand and spot price. The same catalog is served by GET /api/v1/infra/catalog, GraphQL infraCatalog and the MCP infra_catalog tool. allowed_instance_types is reported only when [infra].allowed_instance_types is set; otherwise the page says any instance type is allowed.
• Virtual tags can be created, edited, disabled and rolled back from /admin/infra, which shows each tag's revision history.
• A new systemslab-agent-autoupdate deb/rpm package upgrades the installed SystemsLab agent packages and rezolus at boot, so a cloud agent image does not need rebuilding for each release. A failed upgrade does not block boot.
• [infra] instance_identity_certificate makes the server verify each EC2 agent's signed instance identity document at registration, so a registration token read off one instance cannot register as another instance of the same launch. Set it to AWS's RSA certificate for the region; unset (the default) accepts the token alone, as before. Once it is set, the agent on the AMI must be from this release or later.
• With instance_identity_certificate set, an EC2 agent whose host the reaper declared lost (a single failed status check is enough) registers again into its old host row instead of having its instance terminated. A host lost while idle, busy or assigned comes back idle, and one lost while draining or drained comes back drained. Hosts lost while locked, instances bound to a single run, and spot instances warned of an interruption are still terminated straight away. [infra] dead_host_grace (default 900 seconds) is how long the server waits for the agent before terminating the instance.
• GET /api/v1/job/{id}/{run}/events?endEvent=true sends a final end event, and a systemslab-end-event: end response header, when the stream completes without an error. systemslab logs and the MCP get_logs tool request it, and reconnect when a stream closes before end instead of stopping early.

Changed

• The web UI's homepage is a search box over workspaces, notebooks, contexts and experiments, limited to the workspaces you follow until you remove the in: followed chip. With nothing typed it lists your followed workspaces with their newest contexts, what you created recently, and the pinned contexts. The context lists moved to /contexts, with a "Contexts" link in the top bar, and old links like /?state=open redirect there.
• Breaking: [infra.virtual_tags] and [ec2.virtual_tags] are ignored, with a warning at startup naming each tag. Recreate the tags through the API. Subnet, security groups, IAM instance profile and key name move to named [infra.launch_profiles.<name>] sections, which a tag refers to by name. A profile can set allowed_image_owners to limit which accounts' AMIs its tags may launch.
• Breaking: saving a virtual tag checks its AMI and instance type against AWS, so the server's IAM role needs ec2:DescribeInstanceTypes and ec2:DescribeInstanceTypeOfferings in addition to ec2:DescribeImages.
• The web UI's contexts page now has status tabs that filter both the pinned and recent tables by the status shown on each context's badge (Success, Pending, Failure, Error, Cancelled or Empty), instead of a dropdown that filtered by the context's lifecycle state. Its URL parameter is now status instead of state.
• systemslab experiment list --state pending now lists only experiments that are still waiting to be scheduled, matching the UI's pending badge. Use --state running for experiments that are running. Scripts that used --state pending to find running experiments need --state running.
• The SQL statement behind each GraphQL query field is cancelled after 5 seconds, and the query returns a timeout error.
• systemslab logs and the MCP get_logs tool return the output written so far for a job that is still running, instead of blocking until the job finishes. Pass --follow to keep streaming until the job finishes. systemslab submit --tail still follows the run.
• Pressing Esc in a context's description editor closes it and keeps the draft; the Cancel button still discards it.

Fixed

• The scheduler now honours priority for experiments that need more than one host. A queued experiment that cannot run yet holds the idle hosts it can already use, and only experiments with a strictly lower priority are kept off them. Previously a high-priority two-host experiment lost every freed host to whatever single-host work was queued or submitted next, and could be starved indefinitely. Equal-priority experiments still backfill onto any free host, so a cluster where nobody sets priorities behaves as before. Cancelling a holding experiment, or losing a host it depended on, releases the hold and schedules whatever lower-priority work now fits. Locked, draining and drained hosts do not count as hosts a held experiment can wait for, and a held experiment holds the least specialised hosts it can use.
• Scheduler classes are keyed by the multiset of job tag sets instead of the set. A two-job experiment and a one-job experiment with the same tags used to share a class, so an unschedulable two-job head hid a runnable one-job experiment behind it and left a free host idle.
• A host pre-bound to a virtual-tag run that registers before it is ready is no longer matched until it reports ready, and it triggers scheduling when it does.
• host(after:) on the GraphQL API no longer also applies as a before bound, which returned no hosts.
• The job logs page converts each line's colour codes once, instead of converting every loaded line again for each new line, and renders lines that arrive together once per frame. In headless Chrome, a finished run's 22,320-line log took 232 s to load before and about 1 s after.
• Fixed a context summary showing only one of an experiment's KPI-bearing recordings. An experiment that records its read and write phases separately produced two values for each KPI, the table looked them up by name, and which one you saw depended on artifact iteration order -- on one run the phases were more than an order of magnitude apart on throughput and latency. The summary now renders one table per phase, and the CSV export gains a Phase column with a row per experiment and phase it has values in. A context with one recording per experiment is unchanged. An experiment whose recordings cannot be assigned to phases has its row left blank, with a note naming it, rather than a value shown under the wrong phase.
• Fixed one capture uploaded in two formats (x.rez alongside x.parquet) being counted as two measurements, which reported the same KPI as two slightly different numbers. The .rez is used, and the .parquet is read instead when the .rez cannot be read or reports no KPIs.
• Job log streams no longer fail with unexpected EOF during chunk size line when a job is quiet for a while. The server and agent send keep-alives on idle streams, an error after the stream has started is sent as an SSE error event instead of cutting the response off, and the CLI reconnects and resumes from the last event when the connection drops.
• Fixed the contexts page's Previous link, and GraphQL context(after:), returning the newest page instead of the page just newer than the cursor.
• Fixed plots showing no data until a server restart when they were viewed before the experiment's metrics artifact was uploaded or retagged.
• Fixed plots showing an empty 1970-01-01 time range on every view after the first for experiments with metrics that have no labels.
• Zooming a plot now refetches it at a finer resolution. Previously zoom never went finer than the resolution of the whole recording.
• Fixed the cgroup section's "Sum of non-selected cgroups" chart including the selected cgroups, a single selected cgroup being drawn in a different colour from its picker swatch, and cgroups with escaped names such as \x2d never matching a selection.
• Fixed a plot still drawing the previous render's envelope after switching between Mean and Total.
• Resuming a paused context now keeps its experiments' priority. Previously they were requeued at priority 0.
• Fixed an experiment whose jobs are all external staying queued indefinitely after a server restart.
• The scheduler now detects a host whose ready notification was lost and schedules work on it within about two minutes, instead of leaving it unused until a server restart.
• Fixed scheduler and host-state races that could leave hosts or runs stuck: a drain lost when the host was assigned work at the same moment, a fast job's final state overwritten with running, jobs waiting on a barrier until experiment_timeout after another job of the run failed to launch, and a run left running when the runner's final state report failed during a network partition.
• Fixed EC2 hosts staying busy after their job finished, with their instance never terminated.
• Fixed a multi-job experiment waiting on EC2 capacity staying queued until its queue timeout when a warm instance was idle, or when a launched host was lost before the run started on it.
• A spot interruption warning now moves a busy host to draining instead of drained, so reactivating it during its run no longer lets idle teardown terminate it mid-run.
• Fixed a log request for a job that had just ended keeping the host busy, and delaying the end of the experiment, by up to 7 seconds.
• Fixed the systemslab-postgresql package's prerm script aborting on an unbound variable before it downgraded the database clusters.
• Fixed nightly package upgrades failing with unresolvable dependencies between systemslab-postgresql and systemslab-dbctl.