NiccoloN
034c197260
Validate Operations / validate-operations (push) Failing after 3h10m55s
add plot generation scripts adjust validation tools add up-to-date results
76 lines
4.1 KiB
Markdown
76 lines
4.1 KiB
Markdown
# Evaluation methodology
|
|
|
|
The [documentation index](../README.md) links execution guides, model provenance,
|
|
reproduction details, and correctness experiments.
|
|
|
|
## Canonical result datasets and output roots
|
|
|
|
The authoritative default aggregate CSVs are:
|
|
|
|
- [Comparison results](../../validation/tools/pim/pimcomp/compare/results_comparison.csv)
|
|
- [Ablation results](../../validation/tools/pim/ablation/results_ablation.csv)
|
|
|
|
They are intentionally checked in beside their scripts. Default benchmark runs
|
|
update the corresponding canonical CSV; default plotting reads its current
|
|
contents. Plot generation does not run benchmarks or establish publication
|
|
provenance. Preserve the model/configuration/source/build revisions, invocation,
|
|
reference inputs, and original per-case reports when publishing measurements.
|
|
|
|
An explicit custom `--out-dir PATH` keeps aggregate CSVs under that experiment
|
|
root. Model artifacts still default to `validation/networks/pimcomp_models`;
|
|
changing aggregate placement does not change artifact or shared-reference paths.
|
|
Per-variant ablation summaries remain `<artifact-root>/ablation/<variant>/results_comparison.csv`.
|
|
Use `--input PATH` to plot a custom aggregate. Plot destinations can be selected
|
|
with `--output-dir PATH`. Cleanup of default model artifacts preserves the two
|
|
canonical datasets.
|
|
|
|
## Functional validity and performance availability
|
|
|
|
Functional PASS/FAIL and performance availability are independent. A numerically
|
|
correct program can fail performance simulation or complete no inference before
|
|
the deadline. Missing, failed, skipped, unsupported, nonfinite, or nonpositive
|
|
per-inference measurements become `NA` in aggregates, not zero. With zero
|
|
completed outputs, interval, throughput, and energy per inference are unavailable;
|
|
measured average chip power can still be reported. Functional status is retained.
|
|
|
|
Ablation percentages require positive finite values for both the variant and
|
|
its matching baseline. Tiny positive measurements retain significant digits
|
|
instead of being rounded to zero in the comparison CSV. An extreme valid
|
|
decrease can round to `-100.00%`; that value alone is not evidence of failure.
|
|
Reciprocal output interval and throughput cannot both have that decrease.
|
|
The ablation plotter warns and displays `N/A` for such invalid paired percentages,
|
|
while leaving the CSV and functional status unchanged.
|
|
|
|
## Report reuse and implementation revisions
|
|
|
|
The [comparison guide](../../validation/tools/pim/pimcomp/compare/README.md)
|
|
describes fingerprint validation and failed-report reuse. Fingerprints cover
|
|
model/configuration contents and experiment settings, but do not identify
|
|
compiler/tool implementations, built binaries, or manually changed reference
|
|
artifacts. Reuse is not proof of unchanged tool behavior: rerun the relevant
|
|
compiler after those change, and record build/source revisions for publication.
|
|
No revision cache is inferred from Git HEAD, which would miss dirty sources,
|
|
external tools, and binaries built from another checkout.
|
|
|
|
## Metric interpretation
|
|
|
|
Metrics come from Pimsim's
|
|
[`Chip::getSimulationReport`](../../backend-simulators/pim/pimsim-nn/src/chip/Chip.cpp)
|
|
and [`EnergyCounter`](../../backend-simulators/pim/pimsim-nn/src/utils/EnergyCounter.cpp).
|
|
|
|
- Latency mode measures time to complete one inference.
|
|
- Throughput-mode latency is measurement duration / completed iterations,
|
|
equal to `1000 / throughput` before rounding. It is not end-to-end pipelined
|
|
inference latency; multiplying it by depth does not establish that quantity.
|
|
- Average power is total modeled static + dynamic chip energy / elapsed time.
|
|
- Energy per inference is total energy / completed iterations (one in latency
|
|
mode). Plots convert pJ to mJ with `1e-9`; no pipeline-depth or batch-size
|
|
division is applied.
|
|
|
|
Fast mode measures throughput over the last two convergence windows but power
|
|
and energy over the entire run up to early stop. Energy includes startup and
|
|
work in flight, so it need not equal average power / reported throughput.
|
|
Use `--no-fast` for full-duration measurements. Consult the
|
|
[correctness notes](../../validation/tools/pim/pimcomp/correctness/README.md) before interpreting failed rows
|
|
as performance comparisons.
|