SystemsLab

Appearance

SystemsLab 159.0.0 ​

Added ​

  • A new [infra] config section supersedes [ec2] for EC2 host provisioning and adds options [ec2] cannot express: spot instances with automatic market selection (a request escalates to on-demand above max_spot_duration, default 3600 seconds, or max_spot_hosts, default 4), a keep-warm pool that holds an idle instance for reuse for idle_ttl (default 600 seconds) instead of terminating it as soon as its work finishes, allow-lists for instance types and regions, and per-virtual-tag market, max_instances, spot_max_price, and volume_size_gb overrides. [ec2]'s max_instances is global_max_instances here. A reclaimed spot instance drains its host, its interrupted runs are retried elsewhere, and a replacement instance is launched.

Changed ​

  • EC2 provisioning now runs in a durable infrastructure manager service instead of inline in the scheduler. A host request is recorded in the database before anything is launched, so it survives a server restart, and each instance's lifecycle — requested, launched, denied, assigned, released, terminated — is recorded in a machine event log. Deployments configured with [ec2] keep their current behaviour (one on-demand instance per request, terminated once its work finishes) and need no config changes, but the server's IAM role should gain ec2:DescribeInstances for the orphan sweep, plus ec2:DescribeSpotInstanceRequests, ec2:DescribeSpotPriceHistory, and pricing:GetProducts for spot handling and price recording; without them those passes log an error and are skipped.

Fixed ​

  • Fixed a race where a multi-job experiment's workflow could be woken before the transaction assigning hosts to its runs was visible to it. The workflow then saw NULL hosts, skipped launching those jobs, and stranded the experiment holding its hosts, blocking every experiment queued behind it. (#5578)
  • Fixed the server failing to start with Peer authentication failed for user "anonymous" on deployments that connect to PostgreSQL over a Unix socket with peer authentication and no explicit DATABASE_URL. Introduced in 158.0.0.