nnicolosi/Raptor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

minor validation tool fixes
Validate Operations / validate-operations (push) Failing after 3h12m5s
Details

Browse Source
This commit is contained in:
NiccoloN
2026-10-16 17:57:55 +02:00
parent 0c182fdea5
commit 9053cc5e8b
3 changed files with 33 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
validation/tools/pim/ablation/README.md
+26 -14
View File
@@ -23,9 +23,19 @@ latency and throughput pipeline 4.
| `no-sync-no-plan` | Both options. |
Every variant runs Raptor only. Pimcomp is not compiled, validated, or
simulated. Reference inputs and outputs are generated once under the shared
common-artifact root and reused by every variant. Ctrl+C terminates the active
variant and all of its worker jobs.
simulated; its validation column is `SKIPPED`, not `FAIL`. Reference inputs and
outputs are generated once under the shared common-artifact root and reused by
every variant. Ctrl+C terminates the active variant and all of its worker jobs.
Removing synchronization deliberately removes host-buffer lifetime and pipeline
restart guarantees. The `no-sync` variants can therefore fail numerical
validation, especially across repeated throughput inputs. Those failures remain
`FAIL` and cause a nonzero exit; a completed ablation is not proof that every
variant is correct. Baseline and `no-plan` must still pass numerical validation.
The combined CSV retains `functional_validation` for every row. Performance
percentages from failing variants describe incorrect programs, not valid
speedups. Throughput deadline averages likewise do not establish a converged
steady-state period.
The percentage baseline is `baseline`, so its available values are `+0.00%`.
Every other
@@ -38,7 +48,7 @@ values mean the metric is higher; negative values mean it is lower.
| Option | Description and default |
|---|---|
| `-h`, `--help` | Show help and exit. |
| `--out-dir PATH` | Suite root. Default: `validation/networks/pimcomp_models`. Model artifacts are stored below `<out-dir>/<model>/artifacts`; disabled variants use `<arch>/<mode>[/pipelineN]/ablation/<variant>`. Variant summaries remain under `<out-dir>/ablation/<variant>/`. |
| `--out-dir PATH` | Suite root. Default: `validation/networks/pimcomp_models`. Model artifacts are stored below `<out-dir>/<model>/artifacts/<arch>/<mode>[/pipelineN]/ablation/<variant>`. Variant summaries remain under `<out-dir>/ablation/<variant>/`. |
| `--models MODEL [...]` | Models to run. Default: `vgg8 resnet18 resnet34 googlenet`; include `yolo11n` explicitly when needed. |
| `--variant NAME` | Select a variant from the table above; repeat for multiple variants. Default: all variants. The feature-full baseline and percentage reference are added automatically when needed. |
| `--dry-run` | Print the suite-runner commands that would run without modifying files. Default: off. |
@@ -54,13 +64,15 @@ forwarded, the wrapper uses its default latency and throughput/pipeline-4
case set. Supplying either option overrides that default and is passed through
as one suite-runner invocation per variant.
The feature-full `baseline` variant uses the normal `run_pimcomp_models.py`
artifact paths and is rerun with the same selected cases and forwarded options
as the disabled variants. The three disabled variants are stored below each model's
`artifacts/<arch>/<mode>[/pipelineN]/ablation/` directory. Shared reference
artifacts remain under each model's `artifacts/common` directory. Transient
per-variant comparison summaries are removed after aggregation. The combined
table is written to `<out-dir>/results_ablation.csv`; it contains only the variant, case
identifiers, and signed percentage `latency`, `throughput`, `power`, and
`energy` columns. These are Raptor metrics;
Pimcomp metrics are omitted because the ablation invokes Raptor only.
The feature-full `baseline` variant is rerun with the same selected cases and
forwarded options as the disabled variants. All four variants are stored below
each model's `artifacts/<arch>/<mode>[/pipelineN]/ablation/` directory. Shared
reference artifacts remain under each model's `artifacts/common` directory.
Normal comparison artifacts and summaries are preserved. Compact per-variant
comparison summaries remain under
`<out-dir>/ablation/<variant>/results_comparison.csv` after aggregation.
The combined table is written to `<out-dir>/results_ablation.csv`; it contains the
variant, case identifiers, actual `functional_validation` status, and signed
percentage `latency`, `throughput`, `power`, and `energy` columns. These are
Raptor metrics; Pimcomp metrics are omitted because the ablation invokes Raptor
only.