evaluation module

Armory Experiment Configuration Classes

class evaluation.AttackProtocol(*args, **kwargs)

Bases: Protocol

An attack against the model that can be optimized

apply(batch: Batch) → None

Applies an attack to the given batch

export(sink: Sink, epoch: int) → None

Exports the attack to the given sink at a specific epoch

name: str

Descriptive name of the attack

optimizers() → Any

Configure and return the optimizer(s) that can be used to refine the attack

class evaluation.Chain(name: str, dataset: Dataset | None = None, perturbations: Iterable[PerturbationProtocol] | None = None, model: ModelProtocol | None = None, metrics: Mapping[str, Metric] | None = None, exporters: Iterable[Exporter] | None = None)

Bases: Trackable

A single chain of the following: - dataset as input - perturbations to be applied to input samples - model to generate predictions - metrics to be calculated from model predictions - exporters to save samples

add_exporter(exporter: Exporter) → None

Add an exporter to this evaluation chain

add_metric(name: str, metric: Metric) → None

Add a metric to this evaluation chain

add_perturbation(perturbation: PerturbationProtocol) → None

Add the perturbation to this evaluation chain

get_tracked_params() → Dict[str, Any]

Return the tracked parameters for this chain and all trackable components of the chain

use_dataset(dataset: Dataset) → None

Set the dataset to use for this evaluation chain

use_exporters(exporters: Iterable[Exporter]) → None

Set the exporters to use for this evaluation chain

use_metrics(metrics: Mapping[str, Metric]) → None

Set the metrics to use for this evaluation chain

use_model(model: ModelProtocol) → None

Set the model to use for this evaluation chain

use_perturbations(perturbations: Iterable[PerturbationProtocol]) → None

Set the perturbations to use for this evaluation chain

validate() → None

Ensure all required components of this chain have been defined

class evaluation.Dataset(name: str, dataloader: DataLoader)

Bases: Trackable

Configuration for the dataset to be used for model evaluation

dataloader: DataLoader

Data loader for evaluation data

name: str

Descriptive name of the dataset

class evaluation.Evaluation(name: str, description: str, author: str)

Bases: object

A collection of datasets, perturbations, models, and outputs defining an Armory evaluation.

Each combination of dataset, optional perturbations, model, and outputs represents a distinct evaluation chain. Chains are executed sequentially.

add_chain(name: str)

Add a new evaluation chain to the evaluation

Parameters:

name (str) – Name of the evaluation chain

Yield:

The evaluation chain to be configured

Return type:

Chain

autotrack()

Creates a context for automatically tracking parameters. Any trackable objects created while the context is active will be associated with the parameters recorded during the context.

Example:

with evaluation.autotrack() as track:
    model = track(load_model, split="test")

Returns:

Utility function to invoke functions or classes while recording keyword arguments as parameters

use_dataset(dataset: Dataset) → None

Set the default dataset to use for evaluation chains that do not specify a dataset

use_exporters(exporters: Iterable[Exporter]) → None

Set the default exporters to use for evaluation chains that do not specify exporters

use_metrics(metrics: Mapping[str, Metric]) → None

Set the default metrics to use for evaluation chains that do not specify metrics

use_model(model: ModelProtocol) → None

Set the default model to use for evaluation chains that do not specify a model

use_perturbations(perturbations: Iterable[PerturbationProtocol]) → None

Set the default perturbations to use for evaluation chains that do not specify perturbations

class evaluation.ModelProtocol(*args, **kwargs)

Bases: Protocol

Model being evaluated

loss(batch: Batch) → Any

Calculates the loss for the given batch

name: str

Descriptive name of the model

predict(batch: Batch)

Executes the model to generate predictions from the given batch

class evaluation.Optimization(name: str, description: str, author: str, dataset: Dataset, attack: AttackProtocol, model: ModelProtocol, transforms: Iterable[PerturbationProtocol] | None = None)

Bases: Trackable

A collection of dataset, EOT transforms, and model defining an Armory attack optimization.

get_tracked_params() → Dict[str, Any]

Return the tracked parameters for this optimization and all trackable components of the optimization

Returns:

The tracked parameters for this optimization and all trackable

components of the optimization :rtype: Dict[str, Any]

class evaluation.PerturbationProtocol(*args, **kwargs)

Bases: Protocol

A perturbation that can be applied to dataset batches

apply(batch: Batch) → None

Applies a perturbation to the given batch

name: str

Descriptive name of the perturbation

class evaluation.SysConfig(armory_home: Path = PosixPath('/home/docs/.armory'), dataset_cache: Path = PosixPath('/home/docs/.cache/armory/dataset_cache'))

Bases: object

Host system configuration. This should only need to be instantiated to customize the default system configuration.

armory_home: Path = PosixPath('/home/docs/.armory')

Path to the user-specific folder where Armory will store evaluation results, cached data, etc.

dataset_cache: Path = PosixPath('/home/docs/.cache/armory/dataset_cache')

Path to folder used as a dataset cache