more complete pimcomp comparison scripts
Validate Operations / validate-operations (push) Waiting to run
Validate Operations / validate-operations (push) Waiting to run
update pimsim-nn submodule
This commit is contained in:
NiccoloN
@@ -5,8 +5,12 @@ This directory contains the four networks evaluated in
|
||||
VGG-8, ResNet-18, ResNet-34, and GoogLeNet. It also contains YOLO11n as an
|
||||
additional compiler comparison model.
|
||||
|
||||
See the runner-generated [results.csv](results.csv) for the current latency
|
||||
and energy results.
|
||||
See the runner-generated [results.csv](results.csv) for the current comparison
|
||||
results. Rows are retained separately for each model, architecture, mode, and
|
||||
pipeline. It records separate `PASS`/`FAIL` functional-validation fields for
|
||||
the Raptor and PIMCOMP artifacts; rows without a generated report contain `NA`.
|
||||
Running the runner with `--arch arch-b` or `--arch arch-c` appends those
|
||||
architecture rows without replacing the existing `arch-a` entries.
|
||||
|
||||
## Models and provenance
|
||||
|
||||
@@ -18,11 +22,11 @@ and energy results.
|
||||
| `vgg8/` | VGG-8 reconstruction | `1x1x28x28` | Reconstruction of the [PIMCOMP VGG-8 benchmark](https://arxiv.org/html/2411.09159#S8.SS1), with six convolution and two fully connected layers. |
|
||||
| `yolo11n/` | YOLO11n detection | `1x3x640x640` | Derived from the canonical local model at `../yolo11n/depth_51/yolo11n_depth_51.onnx`, exported from [Ultralytics YOLO11n](https://github.com/ultralytics/ultralytics/blob/main/docs/en/models/yolo11.md). |
|
||||
|
||||
`googlenet/googlenet-12-latency.onnx` is the explicit pimsim-nn-ready GoogLeNet model.
|
||||
`googlenet/googlenet-12-pimsim-nn.onnx` is the explicit pimsim-nn-ready GoogLeNet model.
|
||||
It removes the two LRN nodes and terminal Softmax from the original model,
|
||||
so that the comparison covers only operations scheduled by PIMCOMP and supported by pimsim-nn.
|
||||
|
||||
`yolo11n/yolo11n-latency.onnx` is the explicit pimsim-nn-ready YOLO11n model.
|
||||
`yolo11n/yolo11n-pimsim-nn.onnx` is the explicit pimsim-nn-ready YOLO11n model.
|
||||
It removes the Softmax nodes from the original model,
|
||||
so that the compiled artifact can be simulated in pimsim-nn.
|
||||
|
||||
@@ -98,9 +102,9 @@ Current SHA-256 checksums:
|
||||
788088b908e233d924c7c26b997e89ee861290c7bc56783a306e8201d79aac8f resnet18/resnet18-v1-7.onnx
|
||||
c3231061d081bdd47884137b02134f85142752a39e87263c529cd14ed242b096 resnet34/resnet34-v1-7.onnx
|
||||
c99c507058eaf41de8723408fdda7db8325cb57f0a89f2ee07a716d6e963e14e googlenet/googlenet-12.onnx
|
||||
a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982 googlenet/googlenet-12-latency.onnx
|
||||
a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982 googlenet/googlenet-12-pimsim-nn.onnx
|
||||
396cdea21e5e7d02c3f26f14d22ef20975171702493f5c5e79b8e0d896e541ef vgg8/vgg8-mnist-reconstructed.onnx
|
||||
229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f yolo11n/yolo11n-latency.onnx
|
||||
229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f yolo11n/yolo11n-pimsim-nn.onnx
|
||||
```
|
||||
|
||||
## Paper hardware profiles
|
||||
@@ -109,8 +113,8 @@ The files in
|
||||
[`../../pimsim_configs/pimcomp/`](../../pimsim_configs/pimcomp/)
|
||||
encode Table V's explicit resource parameters.
|
||||
Each profile subdirectory contains pre-generated latency and throughput
|
||||
`pimsim-nn` configs plus its matching mesh; comparison and validation select
|
||||
these checked-in artifacts without generating configs at runtime.
|
||||
`pimsim-nn` configs plus its matching mesh; comparison and validation reference
|
||||
these canonical artifacts directly.
|
||||
|
||||
| Config | Cores | Crossbars/core | Crossbar | Cell | PIMCOMP layout |
|
||||
|------------------------------|------------------:|---------------:|------------|------:|-----------------|
|
||||
@@ -210,36 +214,57 @@ compiles both instruction streams, runs both through `pimsim-nn`, runs
|
||||
functional validation through `pim-simulator`, and writes Markdown and JSON
|
||||
reports.
|
||||
|
||||
To reproduce the complete Arch-A latency experiment, use the model-by-model
|
||||
runner. It verifies the paper GA settings, builds Raptor and the existing
|
||||
`third_party/PIMCOMP-NN/build` tree, then runs the `element`/batch-1 comparison
|
||||
for one model at a time and regenerates `results.csv` from the JSON reports:
|
||||
To reproduce all configured architectures and both latency/throughput modes,
|
||||
use the model-by-model runner. It verifies the paper GA settings, builds Raptor
|
||||
and the existing `third_party/PIMCOMP-NN/build` tree, then runs the comparisons
|
||||
in parallel and regenerates `results.csv` from the JSON reports:
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/tools/pimcomp/run_pimcomp_paper_latency.py
|
||||
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
|
||||
```
|
||||
|
||||
Each model directory reuses regular validation's ignored `inputs/`, `outputs/`,
|
||||
`runner/`, `raptor/`, and `simulation/` paths. PIMCOMP-only artifacts and
|
||||
`comparison_report.{md,json}` live under `pimcomp/`. The frontend regenerates
|
||||
Use `--arch arch-a --mode latency` for only the Arch-A latency experiment.
|
||||
|
||||
Each model directory has a shared ignored `common/` directory containing
|
||||
`inputs/`, `outputs/`, and the native `runner/`.
|
||||
Pimsim-nn configs and network meshes remain canonical under
|
||||
`validation/pimsim_configs/pimcomp/` and are referenced in place.
|
||||
Architecture- and pipeline-specific `raptor/`, `simulation/`, and PIMCOMP
|
||||
artifacts remain under each comparison directory; PIMCOMP outputs are prepared
|
||||
once per model/architecture/mode and linked into the other pipeline directories;
|
||||
`comparison_report.{md,json}`
|
||||
live under its `pimcomp/`. The frontend regenerates
|
||||
one isolated `models/JSON/` graph because PIMCOMP requires that relative
|
||||
layout; it is removed after a successful backend run and the shared submodule
|
||||
model directory is never modified. Models requiring BatchNormalization folding
|
||||
also receive a prepared ONNX file; other models use the original ONNX directly.
|
||||
model directory is never modified. The original ONNX model is passed to the
|
||||
frontend unchanged.
|
||||
PIMCOMP's source tree and build directory remain unchanged at runtime. Use
|
||||
`--models vgg8` to run one model, `--resume` after an interruption, `--dry-run`
|
||||
`--models vgg8` to run one model, `--mode throughput` to select one mode,
|
||||
`--pipeline 4` to select one throughput pipeline, `--only raptor` or
|
||||
`--only pimcomp` to reuse the other compiler's existing artifacts, `--dry-run`
|
||||
to inspect every command, or `--out-dir PATH` to keep results outside
|
||||
`validation/`. The runner continues after a failed model so all reports are
|
||||
produced.
|
||||
`validation/`. Use `--clean` to remove generated comparison artifacts and
|
||||
summaries. Selecting a subset replaces only those comparison rows and
|
||||
recomputes the aggregate `results.csv`; missing shared inputs, outputs, or the
|
||||
reference runner are generated even for an isolated run. Use `--jobs 4` to cap
|
||||
parallel comparisons. The per-stage timeout is unlimited by default; pass a
|
||||
positive `--timeout-seconds` value to impose one.
|
||||
PIMCOMP receives the original ONNX model, and its frontend applies native
|
||||
BatchNormalization fusion when the graph matches its supported Conv/Gemm pattern.
|
||||
The runner continues after a failed model so all reports are produced.
|
||||
|
||||
The known PIMCOMP batch-scheduling correctness issue and a reproducible
|
||||
reference-intermediate prefill experiment are documented in
|
||||
[`validation/tools/pim/pimcomp/correctness/README.md`](../../tools/pim/pimcomp/correctness/README.md).
|
||||
|
||||
Arch-A low-latency example:
|
||||
|
||||
```bash
|
||||
RAPTOR_ROOT=$PWD
|
||||
|
||||
"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pimcomp/compare_raptor_pimcomp.py" \
|
||||
"$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp.py" \
|
||||
--model "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/resnet34-v1-7.onnx" \
|
||||
--out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34" \
|
||||
--out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/arch-a/latency" \
|
||||
--pimcomp-config "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" \
|
||||
--core-count 168 \
|
||||
--crossbar-count 96 \
|
||||
@@ -251,7 +276,7 @@ RAPTOR_ROOT=$PWD
|
||||
```
|
||||
|
||||
Use the same command with
|
||||
`yolo11n/yolo11n-latency.onnx` to probe YOLO11n. Released PIMCOMP-NN cannot
|
||||
`yolo11n/yolo11n-pimsim-nn.onnx` to probe YOLO11n. Released PIMCOMP-NN cannot
|
||||
compile it: the frontend stops at `/model.2/Split`, and it also has no mapping
|
||||
for YOLO11n's two nearest-neighbor `Resize` nodes. Treating the emitted prefix
|
||||
as YOLO11n would produce a misleading latency, so no PIMCOMP number is
|
||||
@@ -297,7 +322,7 @@ either latency-only artifact for semantic validation.
|
||||
Current Raptor status:
|
||||
|
||||
- VGG-8, ResNet-18, fixed-batch ResNet-34, and GoogLeNet compile on Arch-A.
|
||||
- Use `googlenet-12-latency.onnx` for the paper-matched latency comparison.
|
||||
- Use `googlenet-12-pimsim-nn.onnx` for the paper-matched latency comparison.
|
||||
It removes the two LRN nodes and terminal softmax that PIMCOMP does not
|
||||
schedule.
|
||||
- Raptor currently accepts one square `--crossbar-size`; Arch-C's rectangular
|
||||
@@ -319,8 +344,8 @@ rsync -azL validation/networks/pimcomp_models/ \
|
||||
"monolith:$REMOTE_REPO/validation/networks/pimcomp_models/"
|
||||
rsync -az validation/pimsim_configs/pimcomp/ \
|
||||
"monolith:$REMOTE_REPO/validation/pimsim_configs/pimcomp/"
|
||||
rsync -az validation/tools/pimcomp/ \
|
||||
"monolith:$REMOTE_REPO/validation/tools/pimcomp/"
|
||||
rsync -az validation/tools/pim/ \
|
||||
"monolith:$REMOTE_REPO/validation/tools/pim/"
|
||||
rsync -az --exclude=.git --exclude=build --exclude=output \
|
||||
third_party/PIMCOMP-NN/ \
|
||||
"monolith:$REMOTE_REPO/third_party/PIMCOMP-NN/"
|
||||
@@ -334,10 +359,10 @@ cd /home/gmagnani/Project/Raptor
|
||||
|
||||
# One-time setup if the repository virtual environment is absent.
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install numpy onnx onnxruntime onnxsim colorama
|
||||
.venv/bin/python -m pip install numpy onnx onnxruntime colorama
|
||||
|
||||
# Run every latency comparison serially.
|
||||
.venv/bin/python validation/tools/pimcomp/run_pimcomp_paper_latency.py
|
||||
# Run every configured comparison in parallel.
|
||||
.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py
|
||||
```
|
||||
|
||||
Copy reports back without transferring large compiler artifacts:
|
||||
|
||||
@@ -1,5 +1,41 @@
|
||||
model,raptor_latency_ms,pimcomp_latency_ms,raptor_energy_pj,pimcomp_energy_pj,faster_compiler,speedup
|
||||
vgg8,1.465778,7.985074,477298145.040001,1597904071.120000,raptor,5.45
|
||||
resnet18,28.099952,58.853733,8781611766.119984,13983168748.119974,raptor,2.09
|
||||
resnet34,45.781486,91.607980,14962940227.679951,22722922369.680016,raptor,2.00
|
||||
googlenet,13.371204,62.923463,6117835798.919991,14547526780.240000,raptor,4.71
|
||||
model,arch,mode,raptor_pipeline,pimcomp_pipeline,raptor_functional_validation,pimcomp_functional_validation,raptor_throughput_samples_s,pimcomp_throughput_samples_s,raptor_latency_ms,pimcomp_latency_ms,raptor_power_mw,pimcomp_power_mw,raptor_energy_pj,pimcomp_energy_pj,better_compiler,speedup
|
||||
vgg8,arch-a,latency,1,element,PASS,PASS,NA,NA,1.465778,7.985074,325.627854,200.111367,477298145.040001,1597904071.120000,raptor,5.45
|
||||
vgg8,arch-b,latency,1,element,PASS,PASS,NA,NA,1.438869,7.152125,304.673458,173.768633,438385194.040001,1242814988.120001,raptor,4.97
|
||||
vgg8,arch-c,latency,1,element,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
vgg8,arch-a,throughput,2,batch,PASS,PASS,1480.000000,1380.000000,0.674197,0.725548,456.064662,475.694103,307477506.200000,345138738.300000,raptor,1.07
|
||||
vgg8,arch-b,throughput,2,batch,PASS,PASS,1160.000000,1080.000000,0.861527,0.921878,333.045167,408.540967,286927542.600000,376624916.800000,raptor,1.07
|
||||
vgg8,arch-c,throughput,2,batch,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
vgg8,arch-a,throughput,4,batch,PASS,PASS,2160.000000,1380.000000,0.462342,0.725548,446.528150,475.694103,206448818.800000,345138738.300000,raptor,1.57
|
||||
vgg8,arch-c,throughput,4,batch,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
vgg8,arch-a,throughput,8,batch,PASS,PASS,831.000000,1380.000000,1.202894,0.725548,331.341726,475.694103,398569128.700000,345138738.300000,pimcomp,1.66
|
||||
vgg8,arch-c,throughput,8,batch,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
resnet18,arch-a,latency,1,element,PASS,PASS,NA,NA,28.099951,58.855175,312.513413,237.590194,8781611597.119984,13983412446.119972,raptor,2.09
|
||||
resnet18,arch-b,latency,1,element,PASS,PASS,NA,NA,34.938563,65.084209,254.439797,200.986958,8889760875.119959,13081077194.119965,raptor,1.86
|
||||
resnet18,arch-c,latency,1,element,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
resnet18,arch-a,throughput,2,batch,PASS,FAIL,20.000000,76.000000,50.000000,13.149606,263.148477,479.788072,13157423870.000000,6309024249.000000,pimcomp,3.80
|
||||
resnet18,arch-b,throughput,2,batch,PASS,FAIL,26.300000,83.200000,38.016529,12.020906,250.553072,483.029728,9525158116.000000,5806454916.000000,pimcomp,3.16
|
||||
resnet18,arch-c,throughput,2,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
resnet18,arch-a,throughput,4,batch,PASS,FAIL,31.700000,76.000000,31.578947,13.149606,336.949886,479.788072,10640522720.000000,6309024249.000000,pimcomp,2.40
|
||||
resnet18,arch-c,throughput,4,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
resnet18,arch-a,throughput,8,batch,PASS,FAIL,46.400000,76.000000,21.566110,13.149606,319.292391,479.788072,6885894954.000000,6309024249.000000,pimcomp,1.64
|
||||
resnet18,arch-c,throughput,8,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
resnet34,arch-a,latency,1,element,PASS,PASS,NA,NA,45.781484,91.608751,326.833876,248.044564,14962939889.679951,22723052668.680016,raptor,2.00
|
||||
resnet34,arch-b,latency,1,element,PASS,PASS,NA,NA,72.119607,94.519582,239.192092,215.513439,17250439680.679901,20370240175.680019,raptor,1.31
|
||||
resnet34,arch-c,latency,1,element,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
resnet34,arch-a,throughput,2,batch,FAIL,FAIL,NA,40.800000,NA,24.522761,NA,506.131154,NA,12411733160.000000,NA,NA
|
||||
resnet34,arch-b,throughput,2,batch,PASS,FAIL,11.600000,43.300000,86.250000,23.076923,260.049856,489.500435,22429300100.000000,11296163880.000000,pimcomp,3.73
|
||||
resnet34,arch-c,throughput,2,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
resnet34,arch-a,throughput,4,batch,FAIL,FAIL,NA,40.800000,NA,24.522761,NA,506.131154,NA,12411733160.000000,NA,NA
|
||||
resnet34,arch-c,throughput,4,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
resnet34,arch-a,throughput,8,batch,PASS,FAIL,24.300000,40.800000,41.176471,24.522761,271.814969,506.131154,11192381060.000000,12411733160.000000,pimcomp,1.68
|
||||
resnet34,arch-c,throughput,8,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
googlenet,arch-a,latency,1,element,PASS,PASS,NA,NA,13.032305,62.923369,465.072002,231.194088,6060960174.919998,14547510894.240002,raptor,4.83
|
||||
googlenet,arch-b,latency,1,element,PASS,PASS,NA,NA,16.086822,37.935747,378.951815,242.039077,6096130396.919978,9181933205.239973,raptor,2.36
|
||||
googlenet,arch-c,latency,1,element,FAIL,PASS,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
googlenet,arch-a,throughput,2,batch,PASS,FAIL,55.000000,66.200000,18.181818,15.094340,346.696726,434.959232,6303576840.000000,6565422375.000000,pimcomp,1.20
|
||||
googlenet,arch-b,throughput,2,batch,PASS,FAIL,55.000000,72.000000,18.181818,13.888889,314.645370,417.992095,5720824912.000000,5805445765.000000,pimcomp,1.31
|
||||
googlenet,arch-c,throughput,2,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
googlenet,arch-a,throughput,4,batch,PASS,FAIL,111.000000,66.200000,9.012016,15.094340,373.195227,434.959232,3363241363.000000,6565422375.000000,raptor,1.68
|
||||
googlenet,arch-c,throughput,4,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
googlenet,arch-a,throughput,8,batch,PASS,FAIL,61.300000,66.200000,16.310680,15.094340,341.592923,434.959232,5571612719.000000,6565422375.000000,pimcomp,1.08
|
||||
googlenet,arch-c,throughput,8,batch,FAIL,FAIL,NA,NA,NA,NA,NA,NA,NA,NA,NA,NA
|
||||
|
||||
|
@@ -2,5 +2,5 @@ Operation,Result,Compile,Host mem,Cores mem,Cores,Xbars,Latency,Power,Energy
|
||||
vgg8-mnist-reconstructed,PASS,1.009 s,1.37 MiB,3.14 MiB,141,761,1.465778 ms,325.627854 mW,477298145.040001 pJ
|
||||
resnet18-v1-7,PASS,11.548 s,9.89 MiB,40.24 MiB,168,7676,28.099952 ms,312.513408 mW,8781611766.119984 pJ
|
||||
resnet34-v1-7,PASS,28.495 s,9.90 MiB,48.89 MiB,168,15292,45.781486 ms,326.833870 mW,14962940227.679951 pJ
|
||||
googlenet-12-latency,PASS,6.573 s,10.74 MiB,22.41 MiB,168,7176,13.371204 ms,457.538139 mW,6117835798.919991 pJ
|
||||
yolo11n-latency,FAIL,58.572 s,82.55 MiB,185.68 MiB,168,6484,885.264931 ms,189.218985 mW,167508931321.001465 pJ
|
||||
googlenet-12-pimsim-nn,PASS,6.573 s,10.74 MiB,22.41 MiB,168,7176,13.371204 ms,457.538139 mW,6117835798.919991 pJ
|
||||
yolo11n-pimsim-nn,FAIL,58.572 s,82.55 MiB,185.68 MiB,168,6484,885.264931 ms,189.218985 mW,167508931321.001465 pJ
|
||||
|
||||
|
Reference in New Issue
Block a user