# PIMCOMP comparison models This directory contains the four networks evaluated in [PIMCOMP: An End-to-End DNN Compiler for Processing-In-Memory Accelerators](https://arxiv.org/pdf/2411.09159): VGG-8, ResNet-18, ResNet-34, and GoogLeNet. It also contains YOLO11n as an additional compiler comparison model. See the runner-generated [results.csv](results.csv) for the current latency and energy results. ## Models and provenance | Directory | Model | Input | Provenance | |--------------|----------------------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `resnet18/` | ResNet-18 v1 | `1x3x224x224` | Symlink to the complete [ONNX Model Zoo `resnet18-v1-7`](https://huggingface.co/onnxmodelzoo/resnet18-v1-7) model already present at `../resnet18/depth_68/resnet18_depth_68.onnx`. | | `resnet34/` | ResNet-34 v1 | `1x3x224x224` | [ONNX Model Zoo `resnet34-v1-7`](https://huggingface.co/onnxmodelzoo/resnet34-v1-7), with its symbolic batch fixed to 1 as PIMCOMP's frontend does. | | `googlenet/` | GoogLeNet | `1x3x224x224` | Unmodified [ONNX Model Zoo `googlenet-12`](https://huggingface.co/onnxmodelzoo/googlenet-12). | | `vgg8/` | VGG-8 reconstruction | `1x1x28x28` | Reconstruction of the [PIMCOMP VGG-8 benchmark](https://arxiv.org/html/2411.09159#S8.SS1), with six convolution and two fully connected layers. | | `yolo11n/` | YOLO11n detection | `1x3x640x640` | Derived from the canonical local model at `../yolo11n/depth_51/yolo11n_depth_51.onnx`, exported from [Ultralytics YOLO11n](https://github.com/ultralytics/ultralytics/blob/main/docs/en/models/yolo11.md). | `googlenet/googlenet-12-latency.onnx` is the explicit pimsim-nn-ready GoogLeNet model. It removes the two LRN nodes and terminal Softmax from the original model, so that the comparison covers only operations scheduled by PIMCOMP and supported by pimsim-nn. `yolo11n/yolo11n-latency.onnx` is the explicit pimsim-nn-ready YOLO11n model. It removes the Softmax nodes from the original model, so that the compiled artifact can be simulated in pimsim-nn. ## Unsupported and ignored operations PIMCOMP's frontend accepts exactly these ONNX operations: ```text Add, AveragePool, BatchNormalization, Clip, Concat, Conv, Dropout, Flatten, Gather, Gemm, GlobalAveragePool, LRN, MatMul, MaxPool, Mul, Pad, Relu, Reshape, Shape, Sigmoid, Softmax, Squeeze, Sub, Sum, Tanh, Transpose, Unsqueeze ``` `Constant` is consumed as frontend metadata rather than emitted as a PIMCOMP node. Every other ONNX operation is unsupported: the frontend prints `operation: not considered` and stops at the first occurrence. Thus the complete unsupported set is the complement of the allowlist above for the model's ONNX opset. In particular, YOLO11n contains unsupported `Split` and `Resize` nodes. PIMCOMP's low-latency scheduler, hierarchy mapper, and genetic algorithm use this complete explicit no-consider set: ```text Input, BatchNormalization, Clip, Dropout, Flatten, LRN, MatMul, Reshape, Softmax, Squeeze, Transpose ``` Those nodes do not receive scheduled latency instructions when they remain in the backend graph. Before that point the frontend may fuse BatchNormalization and activation nodes into Conv/Gemm, convert a Reshape-Transpose-Reshape channel-shuffle pattern to `OP_SHUFFLE`, remove a specific Shape-Gather-Unsqueeze-Concat shape chain, and merge Pad into its consumer. These transformations do not make an otherwise standalone ignored operation timed. `pimsim-nn` consumes PIM ISA instructions. It supports every named opcode in the shared serialized range except `vsoftmax` (opcode 21), which is rejected explicitly in both JSON and binary input. It silently ignores no opcode; unknown names and numbers are errors. These boundaries explain the dedicated artifacts: - GoogLeNet's two LRN nodes and terminal Softmax perform real computation but are ignored by PIMCOMP, so the common latency artifact removes them. Its inference Dropout and shape-only Reshape can remain without adding compute. - YOLO11n's latency artifact bypasses exactly its two Softmax nodes so it can run in `pimsim-nn`. Every other node, including MatMul, Transpose, and the final detection-decoding tail, remains present and timed by Raptor. No PIMCOMP latency is reported because its frontend stops at `Split` and also lacks `Resize`; compiling that prefix would not represent YOLO11n. The authoritative lists are in [`frontend.py`](../../../third_party/PIMCOMP-NN/frontend/frontend.py), [`ElementPipelineSchedule.cpp`](../../../third_party/PIMCOMP-NN/backend/ElementPipelineSchedule.cpp), [`ISA.h`](../../../backend-simulators/pim/pimsim-nn/src/isa/ISA.h), and [`Instruction.cpp`](../../../backend-simulators/pim/pimsim-nn/src/isa/Instruction.cpp). The PIMCOMP authors did not publish the ONNX checkpoints used by the paper. Running PIMCOMP's frontend on the three Model Zoo files above produces JSON graphs exactly equal to PIMCOMP-NN's bundled `resnet18.json`, `resnet34.json`, and `googlenet.json`. There is no VGG-8 artifact in the ONNX Model Zoo or any PIMCOMP-NN revision. The included VGG-8 therefore has deterministic random weights and is suitable for compiler and simulator comparison, not paper-accuracy reproduction. The paper also says that VGG-8 and ResNet-18 were trained on MNIST, while the published PIMCOMP graphs and ResNet Model Zoo artifacts use ImageNet shapes. Current SHA-256 checksums: ```text 788088b908e233d924c7c26b997e89ee861290c7bc56783a306e8201d79aac8f resnet18/resnet18-v1-7.onnx c3231061d081bdd47884137b02134f85142752a39e87263c529cd14ed242b096 resnet34/resnet34-v1-7.onnx c99c507058eaf41de8723408fdda7db8325cb57f0a89f2ee07a716d6e963e14e googlenet/googlenet-12.onnx a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982 googlenet/googlenet-12-latency.onnx 396cdea21e5e7d02c3f26f14d22ef20975171702493f5c5e79b8e0d896e541ef vgg8/vgg8-mnist-reconstructed.onnx 229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f yolo11n/yolo11n-latency.onnx ``` ## Paper hardware profiles The files in [`../../pimsim_configs/pimcomp/`](../../pimsim_configs/pimcomp/) encode Table V's explicit resource parameters. Each profile subdirectory contains pre-generated latency and throughput `pimsim-nn` configs plus its matching mesh; comparison and validation select these checked-in artifacts without generating configs at runtime. | Config | Cores | Crossbars/core | Crossbar | Cell | PIMCOMP layout | |------------------------------|------------------:|---------------:|------------|------:|-----------------| | `arch-a/latency_config.json` | 168 | 96 | `128x128` | 2-bit | `12x14` | | `arch-b/latency_config.json` | 138 | 128 | `128x128` | 2-bit | `6x23` | | `arch-c/latency_config.json` | 64 (16 chips x 4) | 8 | `512x1024` | 2-bit | flattened `8x8` | `adc_count` is 16, matching the paper's 16-bit fixed-point weight precision. The paper does not give a two-dimensional core topology for Arch-A/B, so the factorizations above preserve core count but cannot reproduce unpublished NoC placement details. Released PIMCOMP-NN has no chip-count field; Arch-C is therefore flattened to 64 cores and does not model chip boundaries. The remaining latency and power values come from PIMCOMP-NN's released default configuration. Consequently, instruction/resource comparisons are reproducible, but absolute paper power and energy numbers are not. ## Build and validate the ONNX files From the Raptor repository root: ```bash .venv/bin/python -m pip install -r requirements.txt cmake --build ./build_release cmake --build third_party/PIMCOMP-NN/build --target PIMCOMP-NN .venv/bin/python -c \ 'from pathlib import Path; import onnx; [onnx.checker.check_model(onnx.load(p)) for p in Path("validation/networks/pimcomp_models").glob("*/*.onnx")]' ``` Do not build either project with `ninja` directly. ## Compile with PIMCOMP PIMCOMP-NN reads `third_party/PIMCOMP-NN/config.json` directly. Back it up, select one paper profile, and restore it when the shell exits: ```bash RAPTOR_ROOT=$PWD PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN" PIMCOMP_CONFIGS="$RAPTOR_ROOT/validation/pimsim_configs/pimcomp" CONFIG_BACKUP=$(mktemp) cp "$PIMCOMP/config.json" "$CONFIG_BACKUP" trap 'cp "$CONFIG_BACKUP" "$PIMCOMP/config.json"' EXIT cp "$PIMCOMP_CONFIGS/arch-a/latency_config.json" "$PIMCOMP/config.json" ``` The Model Zoo files map exactly to PIMCOMP's bundled model names, so compile them directly: ```bash cd "$PIMCOMP/build" # High-throughput mode; the paper evaluates batches of 128 samples. ./PIMCOMP-NN -m=resnet18 -r=balance -p=batch -o=YES -v=YES -s=YES ./PIMCOMP-NN -m=resnet34 -r=balance -p=batch -o=YES -v=YES -s=YES ./PIMCOMP-NN -m=googlenet -r=balance -p=batch -o=YES -v=YES -s=YES # Low-latency mode; the paper uses batch size 1. ./PIMCOMP-NN -m=resnet18 -r=balance -p=element -o=YES -v=YES -s=YES ./PIMCOMP-NN -m=resnet34 -r=balance -p=element -o=YES -v=YES -s=YES ./PIMCOMP-NN -m=googlenet -r=balance -p=element -o=YES -v=YES -s=YES ``` VGG-8 first needs PIMCOMP's JSON frontend. Use a temporary ONNX copy because the released frontend rewrites the input batch dimension in place: ```bash cd /path/to/Raptor cp validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx /tmp/vgg8-pimcomp.onnx .venv/bin/python third_party/PIMCOMP-NN/frontend/frontend.py \ --model_path /tmp/vgg8-pimcomp.onnx \ --save_path third_party/PIMCOMP-NN/models/JSON/vgg8_paper_reconstructed.json cd third_party/PIMCOMP-NN/build ./PIMCOMP-NN -m=vgg8_paper_reconstructed -r=balance -p=batch -o=YES -v=YES -s=YES ./PIMCOMP-NN -m=vgg8_paper_reconstructed -r=balance -p=element -o=YES -v=YES -s=YES ``` Repeat after selecting `arch-b/latency_config.json` and `arch-c/latency_config.json`. All four models were compiled successfully in both modes with all three configs. The released random placement code occasionally segfaults; an unchanged retry succeeded in the observed cases. The paper's optimizer uses a genetic algorithm with population 200 and up to 1000 iterations. The checked-out PIMCOMP submodule already has both paper settings in `backend/GeneticAlgorithm.h`; select them with `-r=GA`. Fitness evaluation uses OpenMP and bounded bandwidth timelines. Set `OMP_NUM_THREADS` to control its parallelism; otherwise OpenMP uses the available CPUs. The GA uses the fixed seed `1`, so repeated serial and parallel runs are reproducible. ## Compare Raptor and PIMCOMP The comparison driver uses one random input and one native ONNX-MLIR reference, compiles both instruction streams, runs both through `pimsim-nn`, runs functional validation through `pim-simulator`, and writes Markdown and JSON reports. To reproduce the complete Arch-A latency experiment, use the model-by-model runner. It verifies the paper GA settings, builds Raptor and the existing `third_party/PIMCOMP-NN/build` tree, then runs the `element`/batch-1 comparison for one model at a time and regenerates `results.csv` from the JSON reports: ```bash .venv/bin/python validation/tools/pimcomp/run_pimcomp_paper_latency.py ``` Each model directory reuses regular validation's ignored `inputs/`, `outputs/`, `runner/`, `raptor/`, and `simulation/` paths. PIMCOMP-only artifacts and `comparison_report.{md,json}` live under `pimcomp/`. The frontend regenerates one isolated `models/JSON/` graph because PIMCOMP requires that relative layout; it is removed after a successful backend run and the shared submodule model directory is never modified. Models requiring BatchNormalization folding also receive a prepared ONNX file; other models use the original ONNX directly. PIMCOMP's source tree and build directory remain unchanged at runtime. Use `--models vgg8` to run one model, `--resume` after an interruption, `--dry-run` to inspect every command, or `--out-dir PATH` to keep results outside `validation/`. The runner continues after a failed model so all reports are produced. Arch-A low-latency example: ```bash RAPTOR_ROOT=$PWD "$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pimcomp/compare_raptor_pimcomp.py" \ --model "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/resnet34-v1-7.onnx" \ --out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34" \ --pimcomp-config "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" \ --core-count 168 \ --crossbar-count 96 \ --crossbar-size 128 \ --mesh-rows 12 \ --mesh-cols 14 \ --pimsim-mode latency \ --pimcomp-pipeline element ``` Use the same command with `yolo11n/yolo11n-latency.onnx` to probe YOLO11n. Released PIMCOMP-NN cannot compile it: the frontend stops at `/model.2/Split`, and it also has no mapping for YOLO11n's two nearest-neighbor `Resize` nodes. Treating the emitted prefix as YOLO11n would produce a misleading latency, so no PIMCOMP number is reported for this model. For Arch-A high throughput, use `--pimsim-mode throughput --pimcomp-pipeline batch`. For Arch-B, use 138 cores, 128 crossbars, a `6x23` mesh, and `validation/pimsim_configs/pimcomp/arch-b/latency_config.json`. If only semantic and instruction comparison is required, add `--skip-pimsim-nn`. A VGG-8 run with the same Arch-A LL settings passed both semantic validations with maximum output differences below `5e-10`. The comparison runner enables `--fail-on-error`, so a failed compiler, simulation, or semantic validation makes the command fail while preserving the generated report. ### Numeric precision and simulator artifacts The functional and non-functional simulators intentionally consume different artifacts: - Raptor and PIMCOMP are validated against the native ONNX-MLIR reference as FP32 programs in the Rust simulator. Raptor's emitted program is already FP32. The PIMCOMP-to-Rust export expands its element-addressed storage and byte-sized transfers to FP32, emits `setbw 32, 32`, and keeps vector `imm_len` fields as element counts. - PIMCOMP's original `SimulationInfo.gz` is copied unchanged for `pimsim-nn`. PIMCOMP hardcodes `setbw 8, 8` and one byte per element without performing numerical quantization; this artifact is used only for latency estimation. - Raptor's original FP32 artifact remains unchanged for functional validation. A separate `raptor/pimsim_nn/` view uses `setbw 8, 8` and scales its byte-addressed storage and transfer sizes from four bytes to one byte per element. Vector `imm_len` fields remain element counts. Like PIMCOMP's artifact, this view is not numerically valid and is used only for a fair non-functional comparison. The ISA defines vector lengths in elements, while `ld`, `st`, `lldi`, `lmv`, `send`, `recv`, addresses, and non-vector offsets are byte-based. Do not use either latency-only artifact for semantic validation. Current Raptor status: - VGG-8, ResNet-18, fixed-batch ResNet-34, and GoogLeNet compile on Arch-A. - Use `googlenet-12-latency.onnx` for the paper-matched latency comparison. It removes the two LRN nodes and terminal softmax that PIMCOMP does not schedule. - Raptor currently accepts one square `--crossbar-size`; Arch-C's rectangular `512x1024` arrays can therefore be compiled by PIMCOMP but not compared exactly with Raptor. Do not change the hardware profile to bypass either limitation; that would no longer be a paper-matched comparison. ## Monolith fallback The local `monolith` SSH alias points to the high-memory host. Copy only this suite and the comparison driver; `-L` materializes the ResNet-18 symlink because the canonical `resnet18/depth_68/resnet18_depth_68.onnx` file may not exist remotely: ```bash REMOTE_REPO=/home/gmagnani/Project/Raptor rsync -azL validation/networks/pimcomp_models/ \ "monolith:$REMOTE_REPO/validation/networks/pimcomp_models/" rsync -az validation/pimsim_configs/pimcomp/ \ "monolith:$REMOTE_REPO/validation/pimsim_configs/pimcomp/" rsync -az validation/tools/pimcomp/ \ "monolith:$REMOTE_REPO/validation/tools/pimcomp/" rsync -az --exclude=.git --exclude=build --exclude=output \ third_party/PIMCOMP-NN/ \ "monolith:$REMOTE_REPO/third_party/PIMCOMP-NN/" ``` Then use the same commands over SSH: ```bash ssh monolith cd /home/gmagnani/Project/Raptor # One-time setup if the repository virtual environment is absent. python3 -m venv .venv .venv/bin/python -m pip install numpy onnx onnxruntime onnxsim colorama # Run every latency comparison serially. .venv/bin/python validation/tools/pimcomp/run_pimcomp_paper_latency.py ``` Copy reports back without transferring large compiler artifacts: ```bash rsync -az --include='*/' --include='comparison_report.*' --exclude='*' \ "monolith:$REMOTE_REPO/validation/networks/pimcomp_models/" \ validation/networks/pimcomp_models/ ```