nnicolosi/Raptor
1
0
Fork 0
Code Issues Pull Requests Actions 7 Packages Projects Releases Wiki Activity

more complete pimcomp comparison scripts
Validate Operations / validate-operations (push) Waiting to run
Details

Browse Source 
update pimsim-nn submodule
This commit is contained in:
NiccoloN
2026-08-06 21:49:54 +02:00
parent 4acd3b0c81
commit 2e76164aed
20 changed files with 1626 additions and 548 deletions
Download Patch File Download Diff File Expand all files Collapse all files
validation/networks/pimcomp_models/README.md
+55 -30
View File
@@ -5,8 +5,12 @@ This directory contains the four networks evaluated in
VGG-8, ResNet-18, ResNet-34, and GoogLeNet. It also contains YOLO11n as an
additional compiler comparison model.
See the runner-generated [results.csv](results.csv) for the current latency
and energy results.
See the runner-generated [results.csv](results.csv) for the current comparison
results. Rows are retained separately for each model, architecture, mode, and
pipeline. It records separate `PASS`/`FAIL` functional-validation fields for
the Raptor and PIMCOMP artifacts; rows without a generated report contain `NA`.
Running the runner with `--arch arch-b` or `--arch arch-c` appends those
architecture rows without replacing the existing `arch-a` entries.
## Models and provenance
@@ -18,11 +22,11 @@ and energy results.
| `vgg8/` | VGG-8 reconstruction | `1x1x28x28` | Reconstruction of the [PIMCOMP VGG-8 benchmark](https://arxiv.org/html/2411.09159#S8.SS1), with six convolution and two fully connected layers. |
| `yolo11n/` | YOLO11n detection | `1x3x640x640` | Derived from the canonical local model at `../yolo11n/depth_51/yolo11n_depth_51.onnx`, exported from [Ultralytics YOLO11n](https://github.com/ultralytics/ultralytics/blob/main/docs/en/models/yolo11.md). |
`googlenet/googlenet-12-latency.onnx` is the explicit pimsim-nn-ready GoogLeNet model.
`googlenet/googlenet-12-pimsim-nn.onnx` is the explicit pimsim-nn-ready GoogLeNet model.
It removes the two LRN nodes and terminal Softmax from the original model,
so that the comparison covers only operations scheduled by PIMCOMP and supported by pimsim-nn.
`yolo11n/yolo11n-latency.onnx` is the explicit pimsim-nn-ready YOLO11n model.
`yolo11n/yolo11n-pimsim-nn.onnx` is the explicit pimsim-nn-ready YOLO11n model.
It removes the Softmax nodes from the original model,
so that the compiled artifact can be simulated in pimsim-nn.
@@ -98,9 +102,9 @@ Current SHA-256 checksums:
788088b908e233d924c7c26b997e89ee861290c7bc56783a306e8201d79aac8f resnet18/resnet18-v1-7.onnx
c3231061d081bdd47884137b02134f85142752a39e87263c529cd14ed242b096 resnet34/resnet34-v1-7.onnx
c99c507058eaf41de8723408fdda7db8325cb57f0a89f2ee07a716d6e963e14e googlenet/googlenet-12.onnx
a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982 googlenet/googlenet-12-latency.onnx
a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982 googlenet/googlenet-12-pimsim-nn.onnx
396cdea21e5e7d02c3f26f14d22ef20975171702493f5c5e79b8e0d896e541ef vgg8/vgg8-mnist-reconstructed.onnx
229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f yolo11n/yolo11n-latency.onnx
229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f yolo11n/yolo11n-pimsim-nn.onnx
```
## Paper hardware profiles
@@ -109,8 +113,8 @@ The files in
[`../../pimsim_configs/pimcomp/`](../../pimsim_configs/pimcomp/)
encode Table V's explicit resource parameters.
Each profile subdirectory contains pre-generated latency and throughput
`pimsim-nn` configs plus its matching mesh; comparison and validation select
these checked-in artifacts without generating configs at runtime.
`pimsim-nn` configs plus its matching mesh; comparison and validation reference
these canonical artifacts directly.
| Config | Cores | Crossbars/core | Crossbar | Cell | PIMCOMP layout |
|------------------------------|------------------:|---------------:|------------|------:|-----------------|
@@ -210,36 +214,57 @@ compiles both instruction streams, runs both through `pimsim-nn`, runs
functional validation through `pim-simulator`, and writes Markdown and JSON
reports.
To reproduce the complete Arch-A latency experiment, use the model-by-model
runner. It verifies the paper GA settings, builds Raptor and the existing
`third_party/PIMCOMP-NN/build` tree, then runs the `element`/batch-1 comparison
for one model at a time and regenerates `results.csv` from the JSON reports:
To reproduce all configured architectures and both latency/throughput modes,
use the model-by-model runner. It verifies the paper GA settings, builds Raptor
and the existing `third_party/PIMCOMP-NN/build` tree, then runs the comparisons
in parallel and regenerates `results.csv` from the JSON reports:
```bash
.venv/bin/python validation/tools/pimcomp/run_pimcomp_paper_latency.py
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
```
Each model directory reuses regular validation's ignored `inputs/`, `outputs/`,
`runner/`, `raptor/`, and `simulation/` paths. PIMCOMP-only artifacts and
`comparison_report.{md,json}` live under `pimcomp/`. The frontend regenerates
Use `--arch arch-a --mode latency` for only the Arch-A latency experiment.
Each model directory has a shared ignored `common/` directory containing
`inputs/`, `outputs/`, and the native `runner/`.
Pimsim-nn configs and network meshes remain canonical under
`validation/pimsim_configs/pimcomp/` and are referenced in place.
Architecture- and pipeline-specific `raptor/`, `simulation/`, and PIMCOMP
artifacts remain under each comparison directory; PIMCOMP outputs are prepared
once per model/architecture/mode and linked into the other pipeline directories;
`comparison_report.{md,json}`
live under its `pimcomp/`. The frontend regenerates
one isolated `models/JSON/` graph because PIMCOMP requires that relative
layout; it is removed after a successful backend run and the shared submodule
model directory is never modified. Models requiring BatchNormalization folding
also receive a prepared ONNX file; other models use the original ONNX directly.
model directory is never modified. The original ONNX model is passed to the
frontend unchanged.
PIMCOMP's source tree and build directory remain unchanged at runtime. Use
`--models vgg8` to run one model, `--resume` after an interruption, `--dry-run`
`--models vgg8` to run one model, `--mode throughput` to select one mode,
`--pipeline 4` to select one throughput pipeline, `--only raptor` or
`--only pimcomp` to reuse the other compiler's existing artifacts, `--dry-run`
to inspect every command, or `--out-dir PATH` to keep results outside
`validation/`. The runner continues after a failed model so all reports are
produced.
`validation/`. Use `--clean` to remove generated comparison artifacts and
summaries. Selecting a subset replaces only those comparison rows and
recomputes the aggregate `results.csv`; missing shared inputs, outputs, or the
reference runner are generated even for an isolated run. Use `--jobs 4` to cap
parallel comparisons. The per-stage timeout is unlimited by default; pass a
positive `--timeout-seconds` value to impose one.
PIMCOMP receives the original ONNX model, and its frontend applies native
BatchNormalization fusion when the graph matches its supported Conv/Gemm pattern.
The runner continues after a failed model so all reports are produced.
The known PIMCOMP batch-scheduling correctness issue and a reproducible
reference-intermediate prefill experiment are documented in
[`validation/tools/pim/pimcomp/correctness/README.md`](../../tools/pim/pimcomp/correctness/README.md).
Arch-A low-latency example:
```bash
RAPTOR_ROOT=$PWD
"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pimcomp/compare_raptor_pimcomp.py" \
"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp.py" \
--model "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/resnet34-v1-7.onnx" \
--out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34" \
--out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/arch-a/latency" \
--pimcomp-config "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" \
--core-count 168 \
--crossbar-count 96 \
@@ -251,7 +276,7 @@ RAPTOR_ROOT=$PWD
```
Use the same command with
`yolo11n/yolo11n-latency.onnx` to probe YOLO11n. Released PIMCOMP-NN cannot
`yolo11n/yolo11n-pimsim-nn.onnx` to probe YOLO11n. Released PIMCOMP-NN cannot
compile it: the frontend stops at `/model.2/Split`, and it also has no mapping
for YOLO11n's two nearest-neighbor `Resize` nodes. Treating the emitted prefix
as YOLO11n would produce a misleading latency, so no PIMCOMP number is
@@ -297,7 +322,7 @@ either latency-only artifact for semantic validation.
Current Raptor status:
- VGG-8, ResNet-18, fixed-batch ResNet-34, and GoogLeNet compile on Arch-A.
- Use `googlenet-12-latency.onnx` for the paper-matched latency comparison.
- Use `googlenet-12-pimsim-nn.onnx` for the paper-matched latency comparison.
It removes the two LRN nodes and terminal softmax that PIMCOMP does not
schedule.
- Raptor currently accepts one square `--crossbar-size`; Arch-C's rectangular
@@ -319,8 +344,8 @@ rsync -azL validation/networks/pimcomp_models/ \
"monolith:$REMOTE_REPO/validation/networks/pimcomp_models/"
rsync -az validation/pimsim_configs/pimcomp/ \
"monolith:$REMOTE_REPO/validation/pimsim_configs/pimcomp/"
rsync -az validation/tools/pimcomp/ \
"monolith:$REMOTE_REPO/validation/tools/pimcomp/"
rsync -az validation/tools/pim/ \
"monolith:$REMOTE_REPO/validation/tools/pim/"
rsync -az --exclude=.git --exclude=build --exclude=output \
third_party/PIMCOMP-NN/ \
"monolith:$REMOTE_REPO/third_party/PIMCOMP-NN/"
@@ -334,10 +359,10 @@ cd /home/gmagnani/Project/Raptor
# One-time setup if the repository virtual environment is absent.
python3 -m venv .venv
.venv/bin/python -m pip install numpy onnx onnxruntime onnxsim colorama
.venv/bin/python -m pip install numpy onnx onnxruntime colorama
# Run every latency comparison serially.
.venv/bin/python validation/tools/pimcomp/run_pimcomp_paper_latency.py
# Run every configured comparison in parallel.
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
```
Copy reports back without transferring large compiler artifacts: