engine package

Submodules

engine.adversarial_dataset module

Armory engine to create adversarial datasets

class engine.adversarial_dataset.AdversarialDatasetEngine(chain: Chain, inputs_key: str, targets_key: str, inputs_spec: DataSpecification | None = None, targets_spec: DataSpecification | None = None, output_dir: str | None = None, adapter: Callable[[Mapping[str, Any]], Mapping[str, Any]] | None = None, features: datasets.Features | None = None, num_batches: int | None = None, profiler: Profiler | None = None)

Bases: object

Armory engine to create adversarial datasets. An adversarial dataset has an adversarial attack already applied to every sample in the dataset.

Example:

from charmory.engine import AdversarialDatasetEngine

# assuming `task` has been defined using a `charmory.tasks` class
engine = AdversarialDatasetEngine(
    task,
    output_dir="dataset_output_dir",
)
engine.generate()

# to load the generated dataset...
import datasets
ds = datasets.load_from_disk("dataset_output_dir")

generate() → datasets.Dataset

Create the adversarial dataset

Returns:: Adversarial dataset
Return type:: datasets.Dataset

engine.adversarial_dataset.SampleAdapter

An adapter for generated samples. The input argument and return types are a dictionary of column names to values for a single sample.

alias of Callable[[Mapping[str, Any]], Mapping[str, Any]]

engine.evaluation module

Armory engine to perform model robustness evaluations

class engine.evaluation.EvaluationEngine(evaluation: Evaluation, profiler: Profiler | None = None, sysconfig: SysConfig | None = None, **kwargs)

Bases: object

Armory engine to perform model robustness evaluations.

Example:

from charmory.engine import EvaluationEngine

# assuming `evaluation` has been defined using `charmory.evaluation.Evaluation`
engine = EvaluationEngine(evaluation)
results = engine.run()

run(verbose: bool = False) → EvaluationResults | None

Perform the evaluation

Parameters:: verbose (bool, optional) – Verbose output, defaults to False
Returns:: Evaluation results if running on the rank zero node
Return type:: Optional[EvaluationResults]

class engine.evaluation.EvaluationProgressBar(chain_name: str, **kwargs)

Bases: RichProgressBar

property test_description

engine.evaluation_module module

Armory lightning module to perform evaluations

class engine.evaluation_module.EvaluationModule(chain: Chain, profiler: Profiler)

Bases: LightningModule

Armory lightning module to perform an evaluation on a single evaluation chain

class MetricsDict(modules: Mapping[str, Module] | None = None)

Bases: ModuleDict

compute() → Mapping[str, Tensor]

Compute

Returns:: Mapping[str, torch.Tensor]
Return type:: Mapping[str, torch.Tensor]

reset() → None: Reset

update_metrics(batch: Batch) → None

Update Metrics

Parameters:: batch (Batch) – Batch

apply_perturbations(batch: Batch)

Applies the chain’s perturbations to the batch to produce the perturbed data to be given to the model

Parameters:: batch (Batch) – Batch

evaluate(batch: Batch)

Perform evaluation on batch

Parameters:: batch (Batch) – Batch

on_test_epoch_end() → None: Logs all metric results

on_test_epoch_start() → None: Resets all metrics

record_metrics(): Record metrics

setup(stage: str) → None

Sets up the exporters

Parameters:: stage (str) – Stage

test_step(batch, batch_idx)

Performs evaluations of the model for each configured perturbation chain

Parameters:

batch (Batch) – Batch
batch_idx (int) – Batch index

engine.optimization module

Armory engine to perform adversarial attack optimization

class engine.optimization.OptimizationEngine(optimization: Optimization, profiler: Profiler | None = None, sysconfig: SysConfig | None = None, **kwargs)

Bases: object

Armory engine to perform adversarial attack optimization

run() → None: Perform the optimization

engine.optimization_module module

Armory lightning module to perform attack optimizations

class engine.optimization_module.OptimizationModule(optimization: Optimization)

Bases: LightningModule

Armory lightning module to perform attack optimizations

configure_optimizers(): Configure optimizers

on_train_epoch_end(): On train epoch end

setup(stage: str) → None

Sets up the exporters

Parameters:: stage (str) – stage

training_step(batch: Batch, batch_idx: int)

Training step

Parameters:

batch (Batch) – Batch
batch_idx (int) – Batch index

Module contents

This package contains the Armory runtime engines.