metrics package

Submodules

metrics.compute module

Profilers to collect computational metrics

class metrics.compute.BasicProfiler

Bases: NullProfiler

Profiler using time.perf_counter

measure(name)
results()
class metrics.compute.DeterministicProfiler

Bases: NullProfiler

Profiler using cProfile for deterministic profiling

measure(name)
results()
class metrics.compute.NullProfiler

Bases: object

Profiler that does nothing (no-op)

measure(name)
reset()
results()
class metrics.compute.Profiler(*args, **kwargs)

Bases: Protocol

measure(name: str)
reset() → None
results() → Mapping[str, float]

metrics.detection module

Object detection metrics

class metrics.detection.ObjectDetectionRates(mean: bool = True, iou_threshold: float = 0.5, score_threshold: float = 0.5, class_list: Sequence[int] | None = None)

Bases: Metric

Metric for the following object detection metrics:

  • True positive rate: the percent of ground-truth boxes which are predicted with iou > iou_threshold, score > score_threshold, and the correct label

  • Misclassification rate: the percent of ground-truth boxes with are predicted with iou > iou_threshold, score > score_threshold, and the incorrect label

  • Disappearance rate: 1 - true positive rate - misclassification rate

  • Hallucinations: the number of predicted boxes per image that have score > score_threshold and iou < iou_threshold for each ground-truth box

compute()

Override this method to compute the final metric value.

This method will automatically synchronize state variables when running in distributed backend.

classmethod create(*args, record_as_artifact: bool = True, record_as_metrics: Iterable[str] | None = None, **kwargs)

Creates an instance of the object detection rates metric pre-wrapped in a armory.metric.PredictionMetric

Parameters:

  • *args –

    Positional arguments are forwarded to the ObjectDetectionRates constructor

  • record_as_artifact (bool, optional) – If True, the metric result will be recorded as an artifact to the evaluation run, defaults to True

  • record_as_metrics (Iterable[str], optional) – Optional, a set of JSON paths in the metric result pointing to scalar values to record as metrics to the evaluation run. If None, no metrics will be recorded, defaults to None

  • **kwargs –

    Keyword arguments are forwarded to the ObjectDetectionRates constructor

Returns:

Armory prediction metric for object detection rates

disappearance_rate_per_img: List[Tensor]
hallucinations_per_img: List[Tensor]
misclassification_rate_per_img: List[Tensor]
true_positive_rate_per_img: List[Tensor]
update(preds: Sequence[BoxesTorch], targets: Sequence[BoxesTorch])

Override this method to update the state variables of your metric class.

metrics.perturbation module

Perturbation metrics

class metrics.perturbation.PerturbationNormMetric(ord: float | int = inf)

Bases: Metric

Metric for L-norm distance between ground truth and perturbed samples.

The calculation of the norm depends on the value used for the order of the norm, as follows. In the following table, d is the distance between ground truth and perturbed samples as calculated by x - x_adv.

order | norm formula ————-|————- torch.inf | max(abs(d)) -torch.inf | min(abs(d)) 0 | sum(d != 0) 1, 2 | sum(abs(d)**ord)**(1./ord)

compute()

Override this method to compute the final metric value.

This method will automatically synchronize state variables when running in distributed backend.

update(ground: Tensor, perturbed: Tensor)

Override this method to update the state variables of your metric class.

metrics.tide module

Module contents

This package contains custom torchmetrics.Metric implementations.