add plot generation scripts adjust validation tools add up-to-date results
This commit is contained in:
NiccoloN
@@ -0,0 +1,75 @@
|
||||
# Evaluation methodology
|
||||
|
||||
The [documentation index](../README.md) links execution guides, model provenance,
|
||||
reproduction details, and correctness experiments.
|
||||
|
||||
## Canonical result datasets and output roots
|
||||
|
||||
The authoritative default aggregate CSVs are:
|
||||
|
||||
- [Comparison results](../../validation/tools/pim/pimcomp/compare/results_comparison.csv)
|
||||
- [Ablation results](../../validation/tools/pim/ablation/results_ablation.csv)
|
||||
|
||||
They are intentionally checked in beside their scripts. Default benchmark runs
|
||||
update the corresponding canonical CSV; default plotting reads its current
|
||||
contents. Plot generation does not run benchmarks or establish publication
|
||||
provenance. Preserve the model/configuration/source/build revisions, invocation,
|
||||
reference inputs, and original per-case reports when publishing measurements.
|
||||
|
||||
An explicit custom `--out-dir PATH` keeps aggregate CSVs under that experiment
|
||||
root. Model artifacts still default to `validation/networks/pimcomp_models`;
|
||||
changing aggregate placement does not change artifact or shared-reference paths.
|
||||
Per-variant ablation summaries remain `<artifact-root>/ablation/<variant>/results_comparison.csv`.
|
||||
Use `--input PATH` to plot a custom aggregate. Plot destinations can be selected
|
||||
with `--output-dir PATH`. Cleanup of default model artifacts preserves the two
|
||||
canonical datasets.
|
||||
|
||||
## Functional validity and performance availability
|
||||
|
||||
Functional PASS/FAIL and performance availability are independent. A numerically
|
||||
correct program can fail performance simulation or complete no inference before
|
||||
the deadline. Missing, failed, skipped, unsupported, nonfinite, or nonpositive
|
||||
per-inference measurements become `NA` in aggregates, not zero. With zero
|
||||
completed outputs, interval, throughput, and energy per inference are unavailable;
|
||||
measured average chip power can still be reported. Functional status is retained.
|
||||
|
||||
Ablation percentages require positive finite values for both the variant and
|
||||
its matching baseline. Tiny positive measurements retain significant digits
|
||||
instead of being rounded to zero in the comparison CSV. An extreme valid
|
||||
decrease can round to `-100.00%`; that value alone is not evidence of failure.
|
||||
Reciprocal output interval and throughput cannot both have that decrease.
|
||||
The ablation plotter warns and displays `N/A` for such invalid paired percentages,
|
||||
while leaving the CSV and functional status unchanged.
|
||||
|
||||
## Report reuse and implementation revisions
|
||||
|
||||
The [comparison guide](../../validation/tools/pim/pimcomp/compare/README.md)
|
||||
describes fingerprint validation and failed-report reuse. Fingerprints cover
|
||||
model/configuration contents and experiment settings, but do not identify
|
||||
compiler/tool implementations, built binaries, or manually changed reference
|
||||
artifacts. Reuse is not proof of unchanged tool behavior: rerun the relevant
|
||||
compiler after those change, and record build/source revisions for publication.
|
||||
No revision cache is inferred from Git HEAD, which would miss dirty sources,
|
||||
external tools, and binaries built from another checkout.
|
||||
|
||||
## Metric interpretation
|
||||
|
||||
Metrics come from Pimsim's
|
||||
[`Chip::getSimulationReport`](../../backend-simulators/pim/pimsim-nn/src/chip/Chip.cpp)
|
||||
and [`EnergyCounter`](../../backend-simulators/pim/pimsim-nn/src/utils/EnergyCounter.cpp).
|
||||
|
||||
- Latency mode measures time to complete one inference.
|
||||
- Throughput-mode latency is measurement duration / completed iterations,
|
||||
equal to `1000 / throughput` before rounding. It is not end-to-end pipelined
|
||||
inference latency; multiplying it by depth does not establish that quantity.
|
||||
- Average power is total modeled static + dynamic chip energy / elapsed time.
|
||||
- Energy per inference is total energy / completed iterations (one in latency
|
||||
mode). Plots convert pJ to mJ with `1e-9`; no pipeline-depth or batch-size
|
||||
division is applied.
|
||||
|
||||
Fast mode measures throughput over the last two convergence windows but power
|
||||
and energy over the entire run up to early stop. Energy includes startup and
|
||||
work in flight, so it need not equal average power / reported throughput.
|
||||
Use `--no-fast` for full-duration measurements. Consult the
|
||||
[correctness notes](../../validation/tools/pim/pimcomp/correctness/README.md) before interpreting failed rows
|
||||
as performance comparisons.
|
||||
@@ -0,0 +1,111 @@
|
||||
# Pimcomp reproduction reference
|
||||
|
||||
The [model README](../../validation/networks/pimcomp_models/README.md) owns provenance, checksums, hardware profiles, and
|
||||
numeric-artifact limitations. The [comparison guide](../../validation/tools/pim/pimcomp/compare/README.md)
|
||||
owns suite commands, reports, reuse rules, and metric interpretation.
|
||||
|
||||
## Frontend acceptance versus scheduled computation
|
||||
|
||||
The current [frontend mapping](../../third_party/PIMCOMP-NN/frontend/frontend.py)
|
||||
accepts exactly these ONNX node types:
|
||||
|
||||
```text
|
||||
Add, AveragePool, BatchNormalization, Clip, Concat, Conv, Dropout, Flatten,
|
||||
Gather, Gemm, GlobalAveragePool, LRN, MatMul, MaxPool, Mul, Pad, Relu, Reshape,
|
||||
Shape, Sigmoid, Softmax, Squeeze, Sub, Sum, Tanh, Transpose, Unsqueeze
|
||||
```
|
||||
|
||||
`Constant` is consumed as metadata rather than emitted as a backend node.
|
||||
Other types cause `operation: <type> not considered` and stop conversion at
|
||||
the first occurrence. In particular, YOLO11n's Split and Resize are unsupported;
|
||||
the emitted prefix cannot represent a complete inference.
|
||||
|
||||
The latency scheduler's explicit no-consider set, also used by hierarchy
|
||||
mapping and GA post-operation accounting, is:
|
||||
|
||||
```text
|
||||
Input, BatchNormalization, Clip, Dropout, Flatten, LRN, MatMul, Reshape,
|
||||
Softmax, Squeeze, Transpose
|
||||
```
|
||||
|
||||
See [ElementPipelineSchedule.cpp](../../third_party/PIMCOMP-NN/backend/ElementPipelineSchedule.cpp),
|
||||
[HierarchyMapping.cpp](../../third_party/PIMCOMP-NN/backend/HierarchyMapping.cpp),
|
||||
and [GeneticAlgorithm.cpp](../../third_party/PIMCOMP-NN/backend/GeneticAlgorithm.cpp).
|
||||
These accepted nodes receive no standalone latency instructions while remaining
|
||||
in the backend graph. Acceptance therefore does not establish timed computation.
|
||||
|
||||
Before scheduling, the frontend may fuse BatchNorm/Relu/Tanh/Sigmoid into
|
||||
Conv/Gemm, turn Reshape–Transpose–Reshape into channel shuffle, remove a specific
|
||||
Shape–Gather–Unsqueeze–Concat shape chain, and merge Pad into its consumer.
|
||||
Those transformations differ from both frontend rejection and leaving a node
|
||||
unscheduled. In particular, the [BatchNorm correctness gap](../../validation/tools/pim/pimcomp/correctness/README.md#resnet-batchnorm-gap)
|
||||
shows why a fusion label is not proof of numeric equivalence.
|
||||
|
||||
Pimsim accepts ISA instructions rather than ONNX operators. Its
|
||||
[instruction decoder](../../backend-simulators/pim/pimsim-nn/src/isa/Instruction.cpp)
|
||||
rejects `vsoftmax` (opcode 21) and unknown opcodes; unsupported operations must
|
||||
not be approximated in the oracle. This is why the dedicated GoogLeNet artifact
|
||||
removes LRN/Softmax and the YOLO artifact removes Softmax. YOLO retains its other
|
||||
operations, including MatMul/Transpose and detection decoding, for Raptor.
|
||||
|
||||
## Standalone frontend/backend reproduction
|
||||
|
||||
Build Pimcomp with CMake and an OpenMP-capable C++ compiler:
|
||||
|
||||
```bash
|
||||
cmake -S third_party/PIMCOMP-NN -B third_party/PIMCOMP-NN/build
|
||||
cmake --build third_party/PIMCOMP-NN/build --target PIMCOMP-NN
|
||||
```
|
||||
|
||||
The backend resolves `../config.json`, `../models/JSON/<name>.json`, and
|
||||
`../output/` relative to its working directory, not its executable. The frontend
|
||||
rewrites its input ONNX batch dimension in place. Use an isolated workspace and
|
||||
a model copy, as the comparison driver does, to preserve source/config files.
|
||||
From the Raptor root, after installing the [frontend dependencies](../../third_party/PIMCOMP-NN/README.md):
|
||||
|
||||
```bash
|
||||
RAPTOR_ROOT=$PWD
|
||||
PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN"
|
||||
PIMCOMP_RUN=$(mktemp -d)
|
||||
mkdir -p "$PIMCOMP_RUN/models/JSON" "$PIMCOMP_RUN/output" "$PIMCOMP_RUN/work"
|
||||
cp validation/pimsim_configs/pimcomp/arch-a/latency_config.json "$PIMCOMP_RUN/config.json"
|
||||
cp validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx "$PIMCOMP_RUN/vgg8.onnx"
|
||||
.venv/bin/python "$PIMCOMP/frontend/frontend.py" \
|
||||
--model_path "$PIMCOMP_RUN/vgg8.onnx" \
|
||||
--save_path "$PIMCOMP_RUN/models/JSON/vgg8.json"
|
||||
(
|
||||
cd "$PIMCOMP_RUN/work"
|
||||
OMP_NUM_THREADS=1 "$PIMCOMP/build/PIMCOMP-NN" \
|
||||
-m=vgg8 -r=GA -p=element -o=YES -v=YES -s=YES
|
||||
)
|
||||
```
|
||||
|
||||
Outputs remain under `$PIMCOMP_RUN/output`. `-v=YES` emits verification data;
|
||||
`-s=YES` emits `SimulationInfo.gz`; `-o=YES` saves the evaluation report.
|
||||
Use `-p=batch` for throughput scheduling, with the matching throughput config
|
||||
when running Pimsim. The paper's batch experiment used 128 samples; the current
|
||||
comparison runner's functional batch default is 64. Element mode represents one
|
||||
inference. Select Arch-B/C by copying that profile's config; Arch-C's rectangular
|
||||
crossbars remain unsupported by Raptor.
|
||||
|
||||
For the bundled Model Zoo graphs, copy `models/JSON/resnet18.json`,
|
||||
`resnet34.json`, or `googlenet.json` from Pimcomp to the workspace and change
|
||||
`-m` accordingly. That reproduces bundled graph scheduling; use the suite
|
||||
runner to regenerate graphs from the exact checked-in performance ONNX files.
|
||||
|
||||
## Genetic algorithm and reproducibility limits
|
||||
|
||||
[`GeneticAlgorithm.h`](../../third_party/PIMCOMP-NN/backend/GeneticAlgorithm.h)
|
||||
sets population 200 and at most 1000 iterations. `-r=GA` selects it; standalone
|
||||
backend default replication is `balance`, whereas the suite explicitly uses GA.
|
||||
GA initialization calls `srand(1)`. Fitness evaluation uses OpenMP with static
|
||||
work distribution; `OMP_NUM_THREADS` bounds its parallelism (use 1 for serial
|
||||
reproduction), otherwise the OpenMP runtime chooses the thread count.
|
||||
|
||||
The fixed GA seed does not make every backend stage deterministic:
|
||||
`ElementPipelineSchedule.cpp` separately calls `srand(time(NULL))`. Keep model
|
||||
hashes, source/build revisions, profile, mode, and thread count with experiment
|
||||
results. Compare correctness
|
||||
against the original ONNX reference, including the known batch and BatchNorm
|
||||
failures; neither omitted operators nor prefilled reference intermediates repair
|
||||
those programs.
|
||||
Reference in New Issue
Block a user