Raptor/docs/evaluation/methodology.md

Evaluation methodology

The documentation index links execution guides, model provenance, reproduction details, and correctness experiments.

Canonical result datasets and output roots

The authoritative default aggregate CSVs are:

• Comparison results
• Ablation results

They are intentionally checked in beside their scripts. Default benchmark runs update the corresponding canonical CSV; default plotting reads its current contents. Plot generation does not run benchmarks or establish publication provenance. Preserve the model/configuration/source/build revisions, invocation, reference inputs, and original per-case reports when publishing measurements.

An explicit custom --out-dir PATH keeps aggregate CSVs under that experiment root. Model artifacts still default to validation/networks/pimcomp_models; changing aggregate placement does not change artifact or shared-reference paths. Per-variant ablation summaries remain <artifact-root>/ablation/<variant>/results_comparison.csv. Use --input PATH to plot a custom aggregate. Plot destinations can be selected with --output-dir PATH. Cleanup of default model artifacts preserves the two canonical datasets.

Functional validity and performance availability

Functional PASS/FAIL and performance availability are independent. A numerically correct program can fail performance simulation or complete no inference before the deadline. Missing, failed, skipped, unsupported, nonfinite, or nonpositive per-inference measurements become NA in aggregates, not zero. With zero completed outputs, interval, throughput, and energy per inference are unavailable; measured average chip power can still be reported. Functional status is retained.

Floating-point correctness requires both elementwise absolute/relative tolerance and a relative L2 limit for each nonzero reference output. The L2 limit prevents small reference outputs from accepting all-zero results under the absolute floor. It cannot substitute for the elementwise check. Zero references use the absolute check; integer outputs require exact agreement. Nonfinite outputs are rejected. Validation reports record both maximum absolute and relative L2 errors. Report reuse includes the validation-policy identity.

Removing normalization operations can amplify floating-point sensitivity and change the computation. Such variants must use their own native reference. Agreement in a global norm alone does not establish elementwise correctness. Independent reference implementations and higher-precision calculations can help diagnose rounding sensitivity without relaxing the recorded pass criteria.

Ablation percentages require positive finite values for both the variant and its matching baseline. Tiny positive measurements retain significant digits instead of being rounded to zero in the comparison CSV. An extreme valid decrease can round to -100.00%; that value alone is not evidence of failure. Reciprocal output interval and throughput cannot both have that decrease. The ablation report uses each disabled feature as its comparison baseline. Planning gain uses no-plan; throughput synchronization gain uses no-sync. Throughput gain measures the relative increase. Latency, power, and energy gains measure relative reductions. Both programs' functional statuses remain separate.

Report reuse and implementation revisions

The comparison guide describes fingerprint validation and failed-report reuse. Fingerprints cover model/configuration contents and experiment settings, but do not identify compiler/tool implementations, built binaries, or manually changed reference artifacts. Reuse is not proof of unchanged tool behavior: rerun the relevant compiler after those change, and record build/source revisions for publication. No revision cache is inferred from Git HEAD, which would miss dirty sources, external tools, and binaries built from another checkout.

Metric interpretation

Metrics come from Pimsim's Chip::getSimulationReport and EnergyCounter.

Pimsim-nn omits vsoftmax before instruction fetch. Reported performance and energy exclude its work. The PIM queue estimator uses the same omission. Functional validation executes Softmax and compares the same full model with its native reference. This timing approximation does not establish hardware support for Softmax or include its execution energy.

• Latency mode measures time to complete one inference.
• Throughput-mode latency is measurement duration / completed iterations, equal to 1000 / throughput before rounding. It is not end-to-end pipelined inference latency; multiplying it by depth does not establish that quantity.
• Average power is total modeled static + dynamic chip energy / elapsed time.
• Energy per inference is total energy / completed iterations (one in latency mode). Plots convert pJ to mJ with 1e-9; no pipeline-depth or batch-size division is applied.

Existing average_energy_pj, raptor_energy_pj, and validation CSV energy columns retain their names for compatibility. They contain energy per inference.

Fast mode measures throughput over the last two convergence windows but power and energy over the entire run up to early stop. Energy includes startup and work in flight, so it need not equal average power / reported throughput. If convergence fails, comparison and ablation reports retain the simulator's full-duration fallback throughput. This measurement includes transient execution. It does not establish steady-state convergence. The per-case JSON records its measurement type and convergence status. Missing or invalid measurements remain NA. Use --no-fast for full-duration measurements. Consult the correctness notes before interpreting failed rows as performance comparisons.