Raptor/docs/evaluation/methodology.md

# Evaluation methodology

The [documentation index](../README.md) links execution guides, model provenance,
reproduction details, and correctness experiments.

## Canonical result datasets and output roots

The authoritative default aggregate CSVs are:

- [Comparison results](../../validation/tools/pim/pimcomp/compare/results_comparison.csv)
- [Ablation results](../../validation/tools/pim/ablation/results_ablation.csv)

They are intentionally checked in beside their scripts. Default benchmark runs
update the corresponding canonical CSV; default plotting reads its current
contents. Plot generation does not run benchmarks or establish publication
provenance. Preserve the model/configuration/source/build revisions, invocation,
reference inputs, and original per-case reports when publishing measurements.

An explicit custom `--out-dir PATH` keeps aggregate CSVs under that experiment
root. Model artifacts still default to `validation/networks/pimcomp_models`;
changing aggregate placement does not change artifact or shared-reference paths.
Per-variant ablation summaries remain `<artifact-root>/ablation/<variant>/results_comparison.csv`.
Use `--input PATH` to plot a custom aggregate. Plot destinations can be selected
with `--output-dir PATH`. Cleanup of default model artifacts preserves the two
canonical datasets.

## Functional validity and performance availability

Functional PASS/FAIL and performance availability are independent. A numerically
correct program can fail performance simulation or complete no inference before
the deadline. Missing, failed, skipped, unsupported, nonfinite, or nonpositive
per-inference measurements become `NA` in aggregates, not zero. With zero
completed outputs, interval, throughput, and energy per inference are unavailable;
measured average chip power can still be reported. Functional status is retained.

Floating-point correctness requires both elementwise absolute/relative tolerance
and a relative L2 limit for each nonzero reference output. The L2 limit prevents
small reference outputs from accepting all-zero results under the absolute floor.
It cannot substitute for the elementwise check. Zero references use the absolute
check; integer outputs require exact agreement. Nonfinite outputs are rejected.
Validation reports record both maximum absolute and relative L2 errors.
Report reuse includes the validation-policy identity.

Removing normalization operations can amplify floating-point sensitivity and
change the computation. Such variants must use their own native reference.
Agreement in a global norm alone does not establish elementwise correctness.
Independent reference implementations and higher-precision calculations can
help diagnose rounding sensitivity without relaxing the recorded pass criteria.

Ablation percentages require positive finite values for both the variant and
its matching baseline. Tiny positive measurements retain significant digits
instead of being rounded to zero in the comparison CSV. An extreme valid
decrease can round to `-100.00%`; that value alone is not evidence of failure.
Reciprocal output interval and throughput cannot both have that decrease.
The ablation report uses each disabled feature as its comparison baseline.
Planning gain uses `no-plan`; throughput synchronization gain uses `no-sync`.
Throughput gain measures the relative increase. Latency, power, and energy gains
measure relative reductions. Both programs' functional statuses remain separate.

## Report reuse and implementation revisions

The [comparison guide](../../validation/tools/pim/pimcomp/compare/README.md)
describes fingerprint validation and failed-report reuse. Fingerprints cover
model/configuration contents and experiment settings, but do not identify
compiler/tool implementations, built binaries, or manually changed reference
artifacts. Reuse is not proof of unchanged tool behavior: rerun the relevant
compiler after those change, and record build/source revisions for publication.
No revision cache is inferred from Git HEAD, which would miss dirty sources,
external tools, and binaries built from another checkout.

## Metric interpretation

Metrics come from Pimsim's
[`Chip::getSimulationReport`](../../backend-simulators/pim/pimsim-nn/src/chip/Chip.cpp)
and [`EnergyCounter`](../../backend-simulators/pim/pimsim-nn/src/utils/EnergyCounter.cpp).

Pimsim-nn omits `vsoftmax` before instruction fetch. Reported performance and
energy exclude its work. The PIM queue estimator uses the same omission.
Functional validation executes Softmax and compares the same full model with
its native reference. This timing approximation does not establish hardware
support for Softmax or include its execution energy.

- Latency mode measures time to complete one inference.
- Throughput-mode latency is measurement duration / completed iterations,
  equal to `1000 / throughput` before rounding. It is not end-to-end pipelined
  inference latency; multiplying it by depth does not establish that quantity.
- Average power is total modeled static + dynamic chip energy / elapsed time.
- Energy per inference is total energy / completed iterations (one in latency
  mode). Plots convert pJ to mJ with `1e-9`; no pipeline-depth or batch-size
  division is applied.

Existing `average_energy_pj`, `raptor_energy_pj`, and validation CSV energy
columns retain their names for compatibility. They contain energy per inference.

Fast mode measures throughput over the last two convergence windows but power
and energy over the entire run up to early stop. Energy includes startup and
work in flight, so it need not equal average power / reported throughput.
If convergence fails, comparison and ablation reports retain the simulator's
full-duration fallback throughput. This measurement includes transient execution.
It does not establish steady-state convergence. The per-case JSON records its
measurement type and convergence status. Missing or invalid measurements remain `NA`.
Use `--no-fast` for full-duration measurements. Consult the
[correctness notes](../../validation/tools/pim/pimcomp/correctness/README.md) before interpreting failed rows
as performance comparisons.