add ablation study

normalize names and artifact paths
2026-08-20 17:58:02 +02:00
parent add20e56eb
commit b009e1ff08
67 changed files with 1573 additions and 993 deletions
@@ -0,0 +1,62 @@
+# Pimcomp model comparison
+
+These scripts compare Raptor with Pimcomp on the models in
+`validation/networks/pimcomp_models`.
+
+`run_pimcomp_models.py` runs the supported model suite across the configured
+architectures and simulation modes. `compare_raptor_pimcomp_model.py` is the
+lower-level one-model comparison used by the suite runner.
+
+## Suite runner
+
+Run all five models, Arch-A and Arch-B, latency, and throughput:
+
+```bash
+.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py
+```
+
+### Options
+
+| Option | Description and default |
+|---|---|
+| `-h`, `--help` | Show help and exit. |
+| `--out-dir PATH` | Suite root; artifacts are placed below `<out-dir>/<model>/artifacts/`. Default: `validation/networks/pimcomp_models`. |
+| `--common-dir PATH` | Shared root for per-model reference artifacts. Default: `<out-dir>/<model>/artifacts/common`. |
+| `--models MODEL [...]` | Models to run: `vgg8`, `resnet18`, `resnet34`, `googlenet`, `yolo11n`. Default: all five; pass a subset to select specific models. |
+| `--archs ARCH [...]` | Pimcomp hardware profiles to run. Default: `arch-a arch-b`. |
+| `--mode {latency,throughput} [...]` | Simulation modes. Default: both. |
+| `--only {raptor,pimcomp}` | Re-run only one compiler side and reuse the other side's existing report. Default: run both sides. |
+| `--raptor-only` | Run Raptor without compiling, validating, or simulating Pimcomp. Default: off. |
+| `--pipeline {1,2,4,8}` | Select one Raptor pipeline. Default: latency pipeline 1 and throughput pipelines 2, 4, and 8. |
+| `--pimsim-time-ms INT` | Throughput convergence deadline. Default: `1000`. |
+| `--batch-size INT` | Functional throughput batch size. Default: `128`. |
+| `--timeout-seconds FLOAT` | Per-stage timeout; `0` means unlimited. Default: `0`. |
+| `-j INT`, `--jobs INT` | Parallel comparison workers. Default: `4`. |
+| `--clean` | Remove generated comparison artifacts and summaries, then exit. Default: off. |
+| `--ablation-variant NAME` | Put generated artifacts below `<mode>[/pipelineN]/ablation/NAME` and write the summary below `ablation/NAME`. Default: none. |
+| `--dry-run` | Print commands without modifying files. Default: off. |
+| `--no-fast` | Disable fast throughput convergence. Default: off. |
+| `--raptor-extra-arg=ARG` | Extra Raptor compiler argument; repeat for multiple arguments. Default: none. |
+
+Arguments beginning with `--` must use the equals form when passed through:
+
+```bash
+.venv/bin/python validation/tools/pim/pimcomp/compare/run_pimcomp_models.py \
+  --models vgg8 \
+  --raptor-extra-arg=--pim-disable-synchronization
+```
+
+The runner writes `results_comparison.csv` under the selected result root. It reuses
+shared model inputs, outputs, and reference runners, and reuses Pimcomp
+artifacts between pipelines in the same model/architecture/mode group. With
+`--raptor-only`, only Raptor results are generated and Pimcomp is not invoked.
+All generated files are kept below each model's `artifacts/` directory;
+`--clean` also removes any `.lock` files left by an interrupted reference
+generation.
+
+## One-model comparator
+
+`compare_raptor_pimcomp_model.py` compares one ONNX model. Its required
+arguments are `--model PATH` and `--out-dir PATH`; use `--help` for the full
+lower-level interface. The suite runner supplies the model, hardware profile,
+simulation mode, and reuse paths automatically.