NiccoloN
034c197260
add plot generation scripts adjust validation tools add up-to-date results
4.1 KiB
Evaluation methodology
The documentation index links execution guides, model provenance, reproduction details, and correctness experiments.
Canonical result datasets and output roots
The authoritative default aggregate CSVs are:
They are intentionally checked in beside their scripts. Default benchmark runs update the corresponding canonical CSV; default plotting reads its current contents. Plot generation does not run benchmarks or establish publication provenance. Preserve the model/configuration/source/build revisions, invocation, reference inputs, and original per-case reports when publishing measurements.
An explicit custom --out-dir PATH keeps aggregate CSVs under that experiment
root. Model artifacts still default to validation/networks/pimcomp_models;
changing aggregate placement does not change artifact or shared-reference paths.
Per-variant ablation summaries remain <artifact-root>/ablation/<variant>/results_comparison.csv.
Use --input PATH to plot a custom aggregate. Plot destinations can be selected
with --output-dir PATH. Cleanup of default model artifacts preserves the two
canonical datasets.
Functional validity and performance availability
Functional PASS/FAIL and performance availability are independent. A numerically
correct program can fail performance simulation or complete no inference before
the deadline. Missing, failed, skipped, unsupported, nonfinite, or nonpositive
per-inference measurements become NA in aggregates, not zero. With zero
completed outputs, interval, throughput, and energy per inference are unavailable;
measured average chip power can still be reported. Functional status is retained.
Ablation percentages require positive finite values for both the variant and
its matching baseline. Tiny positive measurements retain significant digits
instead of being rounded to zero in the comparison CSV. An extreme valid
decrease can round to -100.00%; that value alone is not evidence of failure.
Reciprocal output interval and throughput cannot both have that decrease.
The ablation plotter warns and displays N/A for such invalid paired percentages,
while leaving the CSV and functional status unchanged.
Report reuse and implementation revisions
The comparison guide describes fingerprint validation and failed-report reuse. Fingerprints cover model/configuration contents and experiment settings, but do not identify compiler/tool implementations, built binaries, or manually changed reference artifacts. Reuse is not proof of unchanged tool behavior: rerun the relevant compiler after those change, and record build/source revisions for publication. No revision cache is inferred from Git HEAD, which would miss dirty sources, external tools, and binaries built from another checkout.
Metric interpretation
Metrics come from Pimsim's
Chip::getSimulationReport
and EnergyCounter.
- Latency mode measures time to complete one inference.
- Throughput-mode latency is measurement duration / completed iterations,
equal to
1000 / throughputbefore rounding. It is not end-to-end pipelined inference latency; multiplying it by depth does not establish that quantity. - Average power is total modeled static + dynamic chip energy / elapsed time.
- Energy per inference is total energy / completed iterations (one in latency
mode). Plots convert pJ to mJ with
1e-9; no pipeline-depth or batch-size division is applied.
Fast mode measures throughput over the last two convergence windows but power
and energy over the entire run up to early stop. Energy includes startup and
work in flight, so it need not equal average power / reported throughput.
Use --no-fast for full-duration measurements. Consult the
correctness notes before interpreting failed rows
as performance comparisons.