add plot generation scripts adjust validation tools add up-to-date results
This commit is contained in:
@@ -0,0 +1,111 @@
|
||||
# Pimcomp reproduction reference
|
||||
|
||||
The [model README](../../validation/networks/pimcomp_models/README.md) owns provenance, checksums, hardware profiles, and
|
||||
numeric-artifact limitations. The [comparison guide](../../validation/tools/pim/pimcomp/compare/README.md)
|
||||
owns suite commands, reports, reuse rules, and metric interpretation.
|
||||
|
||||
## Frontend acceptance versus scheduled computation
|
||||
|
||||
The current [frontend mapping](../../third_party/PIMCOMP-NN/frontend/frontend.py)
|
||||
accepts exactly these ONNX node types:
|
||||
|
||||
```text
|
||||
Add, AveragePool, BatchNormalization, Clip, Concat, Conv, Dropout, Flatten,
|
||||
Gather, Gemm, GlobalAveragePool, LRN, MatMul, MaxPool, Mul, Pad, Relu, Reshape,
|
||||
Shape, Sigmoid, Softmax, Squeeze, Sub, Sum, Tanh, Transpose, Unsqueeze
|
||||
```
|
||||
|
||||
`Constant` is consumed as metadata rather than emitted as a backend node.
|
||||
Other types cause `operation: <type> not considered` and stop conversion at
|
||||
the first occurrence. In particular, YOLO11n's Split and Resize are unsupported;
|
||||
the emitted prefix cannot represent a complete inference.
|
||||
|
||||
The latency scheduler's explicit no-consider set, also used by hierarchy
|
||||
mapping and GA post-operation accounting, is:
|
||||
|
||||
```text
|
||||
Input, BatchNormalization, Clip, Dropout, Flatten, LRN, MatMul, Reshape,
|
||||
Softmax, Squeeze, Transpose
|
||||
```
|
||||
|
||||
See [ElementPipelineSchedule.cpp](../../third_party/PIMCOMP-NN/backend/ElementPipelineSchedule.cpp),
|
||||
[HierarchyMapping.cpp](../../third_party/PIMCOMP-NN/backend/HierarchyMapping.cpp),
|
||||
and [GeneticAlgorithm.cpp](../../third_party/PIMCOMP-NN/backend/GeneticAlgorithm.cpp).
|
||||
These accepted nodes receive no standalone latency instructions while remaining
|
||||
in the backend graph. Acceptance therefore does not establish timed computation.
|
||||
|
||||
Before scheduling, the frontend may fuse BatchNorm/Relu/Tanh/Sigmoid into
|
||||
Conv/Gemm, turn Reshape–Transpose–Reshape into channel shuffle, remove a specific
|
||||
Shape–Gather–Unsqueeze–Concat shape chain, and merge Pad into its consumer.
|
||||
Those transformations differ from both frontend rejection and leaving a node
|
||||
unscheduled. In particular, the [BatchNorm correctness gap](../../validation/tools/pim/pimcomp/correctness/README.md#resnet-batchnorm-gap)
|
||||
shows why a fusion label is not proof of numeric equivalence.
|
||||
|
||||
Pimsim accepts ISA instructions rather than ONNX operators. Its
|
||||
[instruction decoder](../../backend-simulators/pim/pimsim-nn/src/isa/Instruction.cpp)
|
||||
rejects `vsoftmax` (opcode 21) and unknown opcodes; unsupported operations must
|
||||
not be approximated in the oracle. This is why the dedicated GoogLeNet artifact
|
||||
removes LRN/Softmax and the YOLO artifact removes Softmax. YOLO retains its other
|
||||
operations, including MatMul/Transpose and detection decoding, for Raptor.
|
||||
|
||||
## Standalone frontend/backend reproduction
|
||||
|
||||
Build Pimcomp with CMake and an OpenMP-capable C++ compiler:
|
||||
|
||||
```bash
|
||||
cmake -S third_party/PIMCOMP-NN -B third_party/PIMCOMP-NN/build
|
||||
cmake --build third_party/PIMCOMP-NN/build --target PIMCOMP-NN
|
||||
```
|
||||
|
||||
The backend resolves `../config.json`, `../models/JSON/<name>.json`, and
|
||||
`../output/` relative to its working directory, not its executable. The frontend
|
||||
rewrites its input ONNX batch dimension in place. Use an isolated workspace and
|
||||
a model copy, as the comparison driver does, to preserve source/config files.
|
||||
From the Raptor root, after installing the [frontend dependencies](../../third_party/PIMCOMP-NN/README.md):
|
||||
|
||||
```bash
|
||||
RAPTOR_ROOT=$PWD
|
||||
PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN"
|
||||
PIMCOMP_RUN=$(mktemp -d)
|
||||
mkdir -p "$PIMCOMP_RUN/models/JSON" "$PIMCOMP_RUN/output" "$PIMCOMP_RUN/work"
|
||||
cp validation/pimsim_configs/pimcomp/arch-a/latency_config.json "$PIMCOMP_RUN/config.json"
|
||||
cp validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx "$PIMCOMP_RUN/vgg8.onnx"
|
||||
.venv/bin/python "$PIMCOMP/frontend/frontend.py" \
|
||||
--model_path "$PIMCOMP_RUN/vgg8.onnx" \
|
||||
--save_path "$PIMCOMP_RUN/models/JSON/vgg8.json"
|
||||
(
|
||||
cd "$PIMCOMP_RUN/work"
|
||||
OMP_NUM_THREADS=1 "$PIMCOMP/build/PIMCOMP-NN" \
|
||||
-m=vgg8 -r=GA -p=element -o=YES -v=YES -s=YES
|
||||
)
|
||||
```
|
||||
|
||||
Outputs remain under `$PIMCOMP_RUN/output`. `-v=YES` emits verification data;
|
||||
`-s=YES` emits `SimulationInfo.gz`; `-o=YES` saves the evaluation report.
|
||||
Use `-p=batch` for throughput scheduling, with the matching throughput config
|
||||
when running Pimsim. The paper's batch experiment used 128 samples; the current
|
||||
comparison runner's functional batch default is 64. Element mode represents one
|
||||
inference. Select Arch-B/C by copying that profile's config; Arch-C's rectangular
|
||||
crossbars remain unsupported by Raptor.
|
||||
|
||||
For the bundled Model Zoo graphs, copy `models/JSON/resnet18.json`,
|
||||
`resnet34.json`, or `googlenet.json` from Pimcomp to the workspace and change
|
||||
`-m` accordingly. That reproduces bundled graph scheduling; use the suite
|
||||
runner to regenerate graphs from the exact checked-in performance ONNX files.
|
||||
|
||||
## Genetic algorithm and reproducibility limits
|
||||
|
||||
[`GeneticAlgorithm.h`](../../third_party/PIMCOMP-NN/backend/GeneticAlgorithm.h)
|
||||
sets population 200 and at most 1000 iterations. `-r=GA` selects it; standalone
|
||||
backend default replication is `balance`, whereas the suite explicitly uses GA.
|
||||
GA initialization calls `srand(1)`. Fitness evaluation uses OpenMP with static
|
||||
work distribution; `OMP_NUM_THREADS` bounds its parallelism (use 1 for serial
|
||||
reproduction), otherwise the OpenMP runtime chooses the thread count.
|
||||
|
||||
The fixed GA seed does not make every backend stage deterministic:
|
||||
`ElementPipelineSchedule.cpp` separately calls `srand(time(NULL))`. Keep model
|
||||
hashes, source/build revisions, profile, mode, and thread count with experiment
|
||||
results. Compare correctness
|
||||
against the original ONNX reference, including the known batch and BatchNorm
|
||||
failures; neither omitted operators nor prefilled reference intermediates repair
|
||||
those programs.
|
||||
Reference in New Issue
Block a user