# PIMCOMP comparison models This directory contains the four networks evaluated in [PIMCOMP: An End-to-End DNN Compiler for Processing-In-Memory Accelerators](https://arxiv.org/pdf/2411.09159): VGG-8, ResNet-18, ResNet-34, and GoogLeNet. It also contains YOLO11n as an additional compiler comparison model. See the runner-generated [results.csv](results.csv) for the current comparison results. Rows are retained separately for each model, architecture, mode, and pipeline. It records separate `PASS`/`FAIL` functional-validation fields for the Raptor and PIMCOMP artifacts; rows without a generated report contain `NA`. Running the runner with `--arch arch-b` or `--arch arch-c` appends those architecture rows without replacing the existing `arch-a` entries. ## Models and provenance | Directory | Model | Input | Provenance | |--------------|----------------------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `resnet18/` | ResNet-18 v1 | `1x3x224x224` | Symlink to the complete [ONNX Model Zoo `resnet18-v1-7`](https://huggingface.co/onnxmodelzoo/resnet18-v1-7) model already present at `../resnet18/depth_68/resnet18_depth_68.onnx`. | | `resnet34/` | ResNet-34 v1 | `1x3x224x224` | [ONNX Model Zoo `resnet34-v1-7`](https://huggingface.co/onnxmodelzoo/resnet34-v1-7), with its symbolic batch fixed to 1 as PIMCOMP's frontend does. | | `googlenet/` | GoogLeNet | `1x3x224x224` | Unmodified [ONNX Model Zoo `googlenet-12`](https://huggingface.co/onnxmodelzoo/googlenet-12). | | `vgg8/` | VGG-8 reconstruction | `1x1x28x28` | Reconstruction of the [PIMCOMP VGG-8 benchmark](https://arxiv.org/html/2411.09159#S8.SS1), with six convolution and two fully connected layers. | | `yolo11n/` | YOLO11n detection | `1x3x640x640` | Derived from the canonical local model at `../yolo11n/depth_51/yolo11n_depth_51.onnx`, exported from [Ultralytics YOLO11n](https://github.com/ultralytics/ultralytics/blob/main/docs/en/models/yolo11.md). | `googlenet/googlenet-12-pimsim-nn.onnx` is the explicit pimsim-nn-ready GoogLeNet model. It removes the two LRN nodes and terminal Softmax from the original model, so that the comparison covers only operations scheduled by PIMCOMP and supported by pimsim-nn. `yolo11n/yolo11n-pimsim-nn.onnx` is the explicit pimsim-nn-ready YOLO11n model. It removes the Softmax nodes from the original model, so that the compiled artifact can be simulated in pimsim-nn. ## Unsupported and ignored operations PIMCOMP's frontend accepts exactly these ONNX operations: ```text Add, AveragePool, BatchNormalization, Clip, Concat, Conv, Dropout, Flatten, Gather, Gemm, GlobalAveragePool, LRN, MatMul, MaxPool, Mul, Pad, Relu, Reshape, Shape, Sigmoid, Softmax, Squeeze, Sub, Sum, Tanh, Transpose, Unsqueeze ``` `Constant` is consumed as frontend metadata rather than emitted as a PIMCOMP node. Every other ONNX operation is unsupported: the frontend prints `operation: not considered` and stops at the first occurrence. Thus the complete unsupported set is the complement of the allowlist above for the model's ONNX opset. In particular, YOLO11n contains unsupported `Split` and `Resize` nodes. PIMCOMP's low-latency scheduler, hierarchy mapper, and genetic algorithm use this complete explicit no-consider set: ```text Input, BatchNormalization, Clip, Dropout, Flatten, LRN, MatMul, Reshape, Softmax, Squeeze, Transpose ``` Those nodes do not receive scheduled latency instructions when they remain in the backend graph. Before that point the frontend may fuse BatchNormalization and activation nodes into Conv/Gemm, convert a Reshape-Transpose-Reshape channel-shuffle pattern to `OP_SHUFFLE`, remove a specific Shape-Gather-Unsqueeze-Concat shape chain, and merge Pad into its consumer. These transformations do not make an otherwise standalone ignored operation timed. `pimsim-nn` consumes PIM ISA instructions. It supports every named opcode in the shared serialized range except `vsoftmax` (opcode 21), which is rejected explicitly in both JSON and binary input. It silently ignores no opcode; unknown names and numbers are errors. These boundaries explain the dedicated artifacts: - GoogLeNet's two LRN nodes and terminal Softmax perform real computation but are ignored by PIMCOMP, so the common latency artifact removes them. Its inference Dropout and shape-only Reshape can remain without adding compute. - YOLO11n's latency artifact bypasses exactly its two Softmax nodes so it can run in `pimsim-nn`. Every other node, including MatMul, Transpose, and the final detection-decoding tail, remains present and timed by Raptor. No PIMCOMP latency is reported because its frontend stops at `Split` and also lacks `Resize`; compiling that prefix would not represent YOLO11n. The authoritative lists are in [`frontend.py`](../../../third_party/PIMCOMP-NN/frontend/frontend.py), [`ElementPipelineSchedule.cpp`](../../../third_party/PIMCOMP-NN/backend/ElementPipelineSchedule.cpp), [`ISA.h`](../../../backend-simulators/pim/pimsim-nn/src/isa/ISA.h), and [`Instruction.cpp`](../../../backend-simulators/pim/pimsim-nn/src/isa/Instruction.cpp). The PIMCOMP authors did not publish the ONNX checkpoints used by the paper. Running PIMCOMP's frontend on the three Model Zoo files above produces JSON graphs exactly equal to PIMCOMP-NN's bundled `resnet18.json`, `resnet34.json`, and `googlenet.json`. There is no VGG-8 artifact in the ONNX Model Zoo or any PIMCOMP-NN revision. The included VGG-8 therefore has deterministic random weights and is suitable for compiler and simulator comparison, not paper-accuracy reproduction. The paper also says that VGG-8 and ResNet-18 were trained on MNIST, while the published PIMCOMP graphs and ResNet Model Zoo artifacts use ImageNet shapes. Current SHA-256 checksums: ```text 788088b908e233d924c7c26b997e89ee861290c7bc56783a306e8201d79aac8f resnet18/resnet18-v1-7.onnx c3231061d081bdd47884137b02134f85142752a39e87263c529cd14ed242b096 resnet34/resnet34-v1-7.onnx c99c507058eaf41de8723408fdda7db8325cb57f0a89f2ee07a716d6e963e14e googlenet/googlenet-12.onnx a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982 googlenet/googlenet-12-pimsim-nn.onnx 396cdea21e5e7d02c3f26f14d22ef20975171702493f5c5e79b8e0d896e541ef vgg8/vgg8-mnist-reconstructed.onnx 229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f yolo11n/yolo11n-pimsim-nn.onnx ``` ## Paper hardware profiles The files in [`../../pimsim_configs/pimcomp/`](../../pimsim_configs/pimcomp/) encode Table V's explicit resource parameters. Each profile subdirectory contains pre-generated latency and throughput `pimsim-nn` configs plus its matching mesh; comparison and validation reference these canonical artifacts directly. | Config | Cores | Crossbars/core | Crossbar | Cell | PIMCOMP layout | |------------------------------|------------------:|---------------:|------------|------:|-----------------| | `arch-a/latency_config.json` | 168 | 96 | `128x128` | 2-bit | `12x14` | | `arch-b/latency_config.json` | 138 | 128 | `128x128` | 2-bit | `6x23` | | `arch-c/latency_config.json` | 64 (16 chips x 4) | 8 | `512x1024` | 2-bit | flattened `8x8` | `adc_count` is 16, matching the paper's 16-bit fixed-point weight precision. The paper does not give a two-dimensional core topology for Arch-A/B, so the factorizations above preserve core count but cannot reproduce unpublished NoC placement details. Released PIMCOMP-NN has no chip-count field; Arch-C is therefore flattened to 64 cores and does not model chip boundaries. The remaining latency and power values come from PIMCOMP-NN's released default configuration. Consequently, instruction/resource comparisons are reproducible, but absolute paper power and energy numbers are not. ## Build and validate the ONNX files From the Raptor repository root: ```bash .venv/bin/python -m pip install -r requirements.txt cmake --build ./build_release cmake --build third_party/PIMCOMP-NN/build --target PIMCOMP-NN .venv/bin/python -c \ 'from pathlib import Path; import onnx; [onnx.checker.check_model(onnx.load(p)) for p in Path("validation/networks/pimcomp_models").glob("*/*.onnx")]' ``` Do not build either project with `ninja` directly. ## Compile with PIMCOMP PIMCOMP-NN reads `third_party/PIMCOMP-NN/config.json` directly. Back it up, select one paper profile, and restore it when the shell exits: ```bash RAPTOR_ROOT=$PWD PIMCOMP="$RAPTOR_ROOT/third_party/PIMCOMP-NN" PIMCOMP_CONFIGS="$RAPTOR_ROOT/validation/pimsim_configs/pimcomp" CONFIG_BACKUP=$(mktemp) cp "$PIMCOMP/config.json" "$CONFIG_BACKUP" trap 'cp "$CONFIG_BACKUP" "$PIMCOMP/config.json"' EXIT cp "$PIMCOMP_CONFIGS/arch-a/latency_config.json" "$PIMCOMP/config.json" ``` The Model Zoo files map exactly to PIMCOMP's bundled model names, so compile them directly: ```bash cd "$PIMCOMP/build" # High-throughput mode; the paper evaluates batches of 128 samples. ./PIMCOMP-NN -m=resnet18 -r=balance -p=batch -o=YES -v=YES -s=YES ./PIMCOMP-NN -m=resnet34 -r=balance -p=batch -o=YES -v=YES -s=YES ./PIMCOMP-NN -m=googlenet -r=balance -p=batch -o=YES -v=YES -s=YES # Low-latency mode; the paper uses batch size 1. ./PIMCOMP-NN -m=resnet18 -r=balance -p=element -o=YES -v=YES -s=YES ./PIMCOMP-NN -m=resnet34 -r=balance -p=element -o=YES -v=YES -s=YES ./PIMCOMP-NN -m=googlenet -r=balance -p=element -o=YES -v=YES -s=YES ``` VGG-8 first needs PIMCOMP's JSON frontend. Use a temporary ONNX copy because the released frontend rewrites the input batch dimension in place: ```bash cd /path/to/Raptor cp validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx /tmp/vgg8-pimcomp.onnx .venv/bin/python third_party/PIMCOMP-NN/frontend/frontend.py \ --model_path /tmp/vgg8-pimcomp.onnx \ --save_path third_party/PIMCOMP-NN/models/JSON/vgg8_paper_reconstructed.json cd third_party/PIMCOMP-NN/build ./PIMCOMP-NN -m=vgg8_paper_reconstructed -r=balance -p=batch -o=YES -v=YES -s=YES ./PIMCOMP-NN -m=vgg8_paper_reconstructed -r=balance -p=element -o=YES -v=YES -s=YES ``` Repeat after selecting `arch-b/latency_config.json` and `arch-c/latency_config.json`. All four models were compiled successfully in both modes with all three configs. The released random placement code occasionally segfaults; an unchanged retry succeeded in the observed cases. The paper's optimizer uses a genetic algorithm with population 200 and up to 1000 iterations. The checked-out PIMCOMP submodule already has both paper settings in `backend/GeneticAlgorithm.h`; select them with `-r=GA`. Fitness evaluation uses OpenMP and bounded bandwidth timelines. Set `OMP_NUM_THREADS` to control its parallelism; otherwise OpenMP uses the available CPUs. The GA uses the fixed seed `1`, so repeated serial and parallel runs are reproducible. ## Compare Raptor and PIMCOMP The comparison driver uses one random input and one native ONNX-MLIR reference, compiles both instruction streams, runs both through `pimsim-nn`, runs functional validation through `pim-simulator`, and writes Markdown and JSON reports. To reproduce all configured architectures and both latency/throughput modes, use the model-by-model runner. It verifies the paper GA settings, expects Raptor and the existing `third_party/PIMCOMP-NN/build` tree to already be built, then runs the comparisons in parallel and regenerates `results.csv` from the JSON reports: ```bash .venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py ``` Use `--arch arch-a --mode latency` for only the Arch-A latency experiment. Each model directory has a shared ignored `common/` directory containing `inputs/`, `outputs/`, and the native `runner/`. Pimsim-nn configs and network meshes remain canonical under `validation/pimsim_configs/pimcomp/` and are referenced in place. Architecture- and pipeline-specific `raptor/`, `simulation/`, and PIMCOMP artifacts remain under each comparison directory; PIMCOMP outputs are prepared once per model/architecture/mode and linked into the other pipeline directories; `comparison_report.{md,json}` live under its `pimcomp/`. The frontend regenerates one isolated `models/JSON/` graph because PIMCOMP requires that relative layout; it is removed after a successful backend run and the shared submodule model directory is never modified. The original ONNX model is passed to the frontend unchanged. PIMCOMP's source tree and build directory remain unchanged at runtime. Use `--models vgg8` to run one model, `--mode throughput` to select one mode, `--pipeline 4` to select one throughput pipeline, `--only raptor` or `--only pimcomp` to reuse the other compiler's existing artifacts, `--dry-run` to inspect every command, or `--out-dir PATH` to keep results outside `validation/`. Use `--clean` to remove generated comparison artifacts and summaries. Selecting a subset replaces only those comparison rows and recomputes the aggregate `results.csv`; missing shared inputs, outputs, or the reference runner are generated even for an isolated run. Use `--jobs 4` to cap parallel comparisons. The per-stage timeout is unlimited by default; pass a positive `--timeout-seconds` value to impose one. PIMCOMP receives the original ONNX model, and its frontend applies native BatchNormalization fusion when the graph matches its supported Conv/Gemm pattern. The runner continues after a failed model so all reports are produced. The known PIMCOMP batch-scheduling correctness issue and a reproducible reference-intermediate prefill experiment are documented in [`validation/tools/pim/pimcomp/correctness/README.md`](../../tools/pim/pimcomp/correctness/README.md). Arch-A low-latency example: ```bash RAPTOR_ROOT=$PWD "$RAPTOR_ROOT/.venv/bin/python" "$RAPTOR_ROOT/validation/tools/pim/pimcomp/compare/compare_raptor_pimcomp.py" \ --model "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/resnet34-v1-7.onnx" \ --out-dir "$RAPTOR_ROOT/validation/networks/pimcomp_models/resnet34/arch-a/latency" \ --pimcomp-config "$RAPTOR_ROOT/validation/pimsim_configs/pimcomp/arch-a/latency_config.json" \ --core-count 168 \ --crossbar-count 96 \ --crossbar-size 128 \ --mesh-rows 12 \ --mesh-cols 14 \ --pimsim-mode latency \ --pimcomp-pipeline element ``` Use the same command with `yolo11n/yolo11n-pimsim-nn.onnx` to probe YOLO11n. Released PIMCOMP-NN cannot compile it: the frontend stops at `/model.2/Split`, and it also has no mapping for YOLO11n's two nearest-neighbor `Resize` nodes. Treating the emitted prefix as YOLO11n would produce a misleading latency, so no PIMCOMP number is reported for this model. For Arch-A high throughput, use `--pimsim-mode throughput --pimcomp-pipeline batch`. For Arch-B, use 138 cores, 128 crossbars, a `6x23` mesh, and `validation/pimsim_configs/pimcomp/arch-b/latency_config.json`. If only semantic and instruction comparison is required, add `--skip-pimsim-nn`. A VGG-8 run with the same Arch-A LL settings passed both semantic validations with maximum output differences below `5e-10`. The comparison runner enables `--fail-on-error`, so a failed compiler, simulation, or semantic validation makes the command fail while preserving the generated report. ### Numeric precision and simulator artifacts The functional and non-functional simulators intentionally consume different artifacts: - Raptor and PIMCOMP are validated against the native ONNX-MLIR reference as FP32 programs in the Rust simulator. Raptor's emitted program is already FP32. The PIMCOMP-to-Rust export expands its element-addressed storage and byte-sized transfers to FP32, emits `setbw 32, 32`, and keeps vector `imm_len` fields as element counts. - PIMCOMP's original `SimulationInfo.gz` is copied unchanged for `pimsim-nn`. PIMCOMP hardcodes `setbw 8, 8` and one byte per element without performing numerical quantization; this artifact is used only for latency estimation. - Raptor's original FP32 artifact remains unchanged for functional validation. A separate `raptor/pimsim_nn/` view uses `setbw 8, 8` and scales its byte-addressed storage and transfer sizes from four bytes to one byte per element. Vector `imm_len` fields remain element counts. Like PIMCOMP's artifact, this view is not numerically valid and is used only for a fair non-functional comparison. The ISA defines vector lengths in elements, while `ld`, `st`, `lldi`, `lmv`, `send`, `recv`, addresses, and non-vector offsets are byte-based. Do not use either latency-only artifact for semantic validation. Current Raptor status: - VGG-8, ResNet-18, fixed-batch ResNet-34, and GoogLeNet compile on Arch-A. - Use `googlenet-12-pimsim-nn.onnx` for the paper-matched latency comparison. It removes the two LRN nodes and terminal softmax that PIMCOMP does not schedule. - Raptor currently accepts one square `--crossbar-size`; Arch-C's rectangular `512x1024` arrays can therefore be compiled by PIMCOMP but not compared exactly with Raptor. Do not change the hardware profile to bypass either limitation; that would no longer be a paper-matched comparison. ## Monolith fallback The local `monolith` SSH alias points to the high-memory host. Copy only this suite and the comparison driver; `-L` materializes the ResNet-18 symlink because the canonical `resnet18/depth_68/resnet18_depth_68.onnx` file may not exist remotely: ```bash REMOTE_REPO=/home/gmagnani/Project/Raptor rsync -azL validation/networks/pimcomp_models/ \ "monolith:$REMOTE_REPO/validation/networks/pimcomp_models/" rsync -az validation/pimsim_configs/pimcomp/ \ "monolith:$REMOTE_REPO/validation/pimsim_configs/pimcomp/" rsync -az validation/tools/pim/ \ "monolith:$REMOTE_REPO/validation/tools/pim/" rsync -az --exclude=.git --exclude=build --exclude=output \ third_party/PIMCOMP-NN/ \ "monolith:$REMOTE_REPO/third_party/PIMCOMP-NN/" ``` Then use the same commands over SSH: ```bash ssh monolith cd /home/gmagnani/Project/Raptor # One-time setup if the repository virtual environment is absent. python3 -m venv .venv .venv/bin/python -m pip install numpy onnx onnxruntime colorama # Run every configured comparison in parallel. .venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_paper_latency.py ``` Copy reports back without transferring large compiler artifacts: ```bash rsync -az --include='*/' --include='comparison_report.*' --exclude='*' \ "monolith:$REMOTE_REPO/validation/networks/pimcomp_models/" \ validation/networks/pimcomp_models/ ```