systemslab sweep

Description	Perform a sweep over a set of parameters and launch each configuration as an experiment.
Usage	systemslab sweep [OPTIONS] --name <SPEC> <SWEEPFILE>

Description

systemslab sweep allows for easily launching a large number of experiments which are generated from a single experiment specification. It is designed to be used for launching parameter sweeps of an experiment but the sweep file format is designed to be flexible as long as only one specification file is being used.

If you are trying to compare experiments with different specification files consider using systemslab submit with the --context flag instead.

Experiment Specification

The experiment specification file describes the steps that will be taken to perform the experiment: what jobs are run, what steps each of them runs, and where they can be scheduled. The experiment specification can be written in json, yaml, or jsonnet. Which one is loaded will depend on the extension of the provided file. If the path does not have an extension then yaml will be assumed.

See the experiment specification reference for documentation of the JSON object that the experiment specification file should expand to.

Sweep Specification

A sweep specification file allows you to determine the parameters used for each experiment launched as part of the sweep.

INFO

In order to take advantage of a sweep specification file your experiment specification will need to have parameters. That is to say, it must be written in jsonnet and it must take in parameters via jsonnet top-level arguments.

What this means is that your experiment specification will be a jsonnet file whose root element is a function, such as this one:

function(cores, jdk = "jdk11", spin = false) {
  // ... your experiment here
}

The parameters set via the sweep file will set the arguments to the top-level function in your experiment specification.

There are two different formats you can use for a sweep specification.

List Specification

The first and simplest format is a list of json objects whose fields correspond to the top-level arguments in the experiment specification. Each entry in the list corresponds to exactly one experiment in the resulting sweep.

A sweep specification for the experiment spec above might look like this:

[
  { "cores": 8, "jdk": "jdk11", "spin": true },
  { "cores": 4, "jdk": "jdk11", "spin": true }
]

TIP

It is not necessary to set all parameters in the experiment specification via the sweep file. Parameters with default values will simply have their default value if not included. Since the jdk parameter defaults to "jdk11" in the experiment specification the following sweep spec is equivalent to the one above:

[
  { "cores": 8, "spin": true },
  { "cores": 4, "spin": true }
]

Matrix Specification

The second format allows you to define a matrix of parameter values. It looks like this:

{
  "matrix": {
    "cores": [8, 4],
    "jdk": ["jdk11"],
    "spin": [true]
  }
}

Each key in the matrix object corresponds to a parameter in your experiment specification and the corresponding array is the list of values it can take. The resulting sweep will have a single experiment for every point in the cartesian product of all specified parameter values.

For example, the specification above corresponds to the following parameter list:

[
  { "cores": 8, "jdk": "jdk11", "spin": true },
  { "cores": 4, "jdk": "jdk11", "spin": true }
]

Since only one parameter was varied above it only generates a small number of experiments. However, by adding more we can quickly start creating more and more experiments. For example, this specification

{
  "matrix": {
    "cores": [4, 8],
    "jdk": ["jdk11"],
    "spin": [true, false]
  }
}

results in a sweep containing 4 different experiments

[
  { "cores": 4, "jdk": "jdk11", "spin": true },
  { "cores": 4, "jdk": "jdk11", "spin": false },
  { "cores": 8, "jdk": "jdk11", "spin": true },
  { "cores": 8, "jdk": "jdk11", "spin": false }
]

Range

As an alternative to creating a large list of numbers, the matrix specification also allows you to specify a range of numbers in place of its list. This is done by adding a json object with "start", "stop", and (optionally) "step" instead of the list of parameter values.

A matrix with a range within would look like this:

{
  "matrix": {
    "cores": {
      "start": 0,
      "stop": 8,
      "step": 2
    }
  }
}

The specification above would result in the following parameter list

[
  { "cores": 0 },
  { "cores": 2 },
  { "cores": 4 },
  { "cores": 8 }
]

Note that the range is inclusive of its stop value. Since we had "stop": 8 here, 8 was included in the range.

WARNING

A large matrix specification can generate an extremely large number of experiments. This is especially true when using ranges.

As an example, the following sweep specification generates 5120 experiments.

{
  "matrix": {
    "cores": { "start": 1, "stop": 32 },
    "target_qps": { "start": 5000, "stop": 200000, "step": 5000 },
    "batch_size": [1, 10, 100, 1000]
  }
}

By default, SystemsLab will warn if you attempt to spawn more than 10000 experiments at once. However, even below that running large numbers of experiments can be expensive.

Common Errors

One thing that you may run into when attempting to use a sweep file is that the parameters passed in to the experiment specification will contain numbers and booleans, instead of just strings. By default, systemslab submit will pass all parameters as strings unless their type is explicitly specified on the command line (e.g. by doing --param cores:number=1).

TIP

You can ensure that a nicer error message is emitted when evaluating the experiment specification by adding jsonnet asserts to validate the type is what the specification expects:

local invalidTypeMsg(name, expected, value) = std.format(
  '`%s` had invalid type (expected `%s` but got `%s` instead)',
  [name, expected, std.type(value)]
);

function(cores, jdk = "jdk11", spin = false)
  assert std.type(cores) == 'number'  : invalidTypeMsg('cores', 'number', cores);
  assert std.type(jdk)   == 'string'  : invalidTypeMsg('jdk',   'string', jdk);
  assert std.type(spin)  == 'boolean' : invalidTypeMsg('spin', 'boolean', spin);
{
  // ... your experiment here
}

Options

Option	Default	Description
-p, --param		Pass a parameter to use when evaluating the experiment specification. This will be overridden by the sweep file if the sweep file specifies the same parameter.
-j, --parallel	1	Evaluate experiment specifications using multiple threads. Passing 0 is equivalent to specifying the number of CPUs.
--limit-experiments	10000	Limit the number of experiments that can be launched before giving an error. This is to prevent accidental launches of huge numbers of experiments.
-q, --quiet		Silence all progress bars.
--jpath		Add an extra include path to use when resolving jsonnet imports.
--jsonnet-max-stack	200	Limit the recursion depth that the jsonnet vm is allowed to reach when evaluating jsonnet files.

Common Options

Option	Default	Description
--systemslab-url		The URL at which to access the systemslab server.
--color	auto	Controls when color is used in the output. Possible values are auto, always, and never.
--output-format	long	Output format to use when outputting messages. long will result output designed to be human-readable while short is short output meant to be used within scripts.

Environment Variables

Variable	Description
SYSTEMSLAB_URL	Specify the URL used to contact the SystemsLab server. Equivalent to --systemslab-url.