Merge branch 'TestRottoConDeadLock' of chef.heaplab.deib.polimi.it:nnicolosi/Raptor into TestRottoConDeadLock

.gitmodules

@@ -6,4 +6,4 @@
 	url = https://github.com/HEAPLab/pimsim-nn.git
 [submodule "third_party/PIMCOMP-NN"]
 	path = third_party/PIMCOMP-NN
-	url = https://github.com/sunxt99/PIMCOMP-NN
+	url = https://github.com/HEAPLab/PIMCOMP-NN.git

AGENTS.md

@@ -15,6 +15,7 @@ Before modifying the relevant subsystem, read:
 * The debug build is very slow, so use it only on small fast tests such as operation validations, not on network validations
 * Always prepend rtk to shell commands if missing and if rtk is available
 * Always use the repository Python virtual environment (`.venv/bin/python` and its bundled tools) for Python commands
+* Run the complete operations validation (`--operations-dir validation/operations`) outside the sandbox so parallel worker processes can start. This does not apply to one-at-a-time network validation
 
 # Core engineering philosophy
 

README.md

@@ -154,108 +154,10 @@ This writes PIM artifacts under `/tmp/raptor/pim/`.
 
 ## Validation
 
-Functional validation lives in `validation/`. It compiles ONNX models, builds a
+Functional validation compiles ONNX models, compares native ONNX-MLIR and PIM
-native ONNX-MLIR reference runner, generates random inputs, runs Raptor, runs
+simulator outputs, and optionally reports latency and power. See
-the Rust functional simulator, and compares outputs.
+[`validation/README.md`](validation/README.md) for prerequisites, usage,
-
+options, artifacts, and results.
-Python dependencies used by the validation scripts are `numpy`, `onnx`, and
-`colorama`. The simulator requires the Rust toolchain.
-
-Per-operation validation from the repository root:
-
-```bash
-.venv/bin/python validation/validate.py \
-  --raptor-path build_release/Release/bin/onnx-mlir \
-  --onnx-include-dir onnx-mlir/include \
-  --operations-dir validation/operations \
-  --verbose \
-  --raptor-extra-arg=--pim-detect-communication-deadlock \
-  --raptor-extra-arg=--pim-export-spatial-dataflow=none
-```
-
-Validate one network or a subset by pointing `--operations-dir` at any directory
-containing `.onnx` files:
-
-```bash
-.venv/bin/python validation/validate.py \
-  --raptor-path build_release/Release/bin/onnx-mlir \
-  --onnx-include-dir onnx-mlir/include \
-  --operations-dir validation/networks/yolo11n/depth_04 \
-  --verbose \
-  --raptor-extra-arg=--pim-detect-communication-deadlock \
-  --raptor-extra-arg=--pim-export-spatial-dataflow=none
-```
-
-Useful validation options:
-- `--simulator-dir <path>` - override the auto-detected functional
-  `backend-simulators/pim/pim-simulator` path.
-- `--pimcomp-config <arch-a|arch-b|arch-c>` - select the checked-in
-  non-functional simulation profile. Arch-A is the default; the hardware
-  argument defaults match it (`168` cores, `96` crossbars/core, `128x128`
-  crossbars).
-  Selecting another profile also requires its matching hardware arguments.
-- `--non-functional-simulator-build-dir <path>` - override the auto-detected
-  non-functional simulator build directory.
-- `--skip-non-functional-simulation` - skip non-functional simulation. If
-  explicit core/crossbar arguments do not match the selected profile, validation
-  reports the mismatch and skips only non-functional simulation.
-- `--threshold <float>` - maximum allowed per-element output difference.
-- `--seed <int>` - RNG seed for generated inputs.
-- `--command-timeout-seconds <float>` - timeout for compiler, runner, and
-  simulator subprocesses.
-- `--verbose` - print subprocess logs and average PIM pass timings.
-- `--clean` - remove generated validation artifacts and exit.
-
-Each validation run writes artifacts in the model workspace, for example under
-`validation/operations/gemm/small/`:
-- `inputs/` - generated input CSV files.
-- `outputs/` - native ONNX-MLIR reference outputs.
-- `raptor/` - compiler artifacts, including `*.onnx.mlir`, dialect dumps under
-  `dialects/`, reports under `reports/`, and final PIM artifacts under `pim/`.
-- `runner/` - generated reference runner source, build tree, and shared library.
-- `simulation/out.bin` - raw functional simulation output used for comparison.
-
-By default, validation also runs non-functional simulation. Per-model `Latency`
-is shown in the result table alongside `Power`, with measured, failed, and
-skipped counts plus total latency in the summary. Use these validation results
-for performance comparisons; the simulation backend does not need to be invoked
-separately. The selected pre-generated configs and meshes live under
-`validation/pimsim_configs/pimcomp/`.
-
-The compiler currently dumps dialect snapshots such as `spatial0.mlir`,
-`spatial1_graph.mlir`, `spatial2_trivial_merged.mlir`,
-`spatial3_scheduled_no_comm.mlir`, `spatial4_scheduled.mlir`, `pim0.mlir`, `pim1_buff.mlir`,
-`pim2_folded.mlir`, and `pim3_memory_planned.mlir` when an output directory is
-available.
-
-To rerun the functional simulator manually with tracing after validation has
-produced a `raptor/pim/` directory:
-
-```bash
-cd backend-simulators/pim/pim-simulator
-cargo run --no-default-features --features tracing --release \
-  --package pim-simulator --bin pim-simulator -- \
-  -f /path/to/workspace/raptor/pim \
-  -o /path/to/workspace/simulation/out.bin \
-  -d <addr0>,<size0>,<addr1>,<size1>,...
-```
-
-With `--features tracing`, the simulator writes per-core traces as
-`TraceCore0`, `TraceCore1`, ... next to `out.bin`. The validator normally
-computes the `-d` ranges from `raptor/pim/config.json` and model output shapes.
-
-Available validation networks under `validation/networks/`: `vgg16`,
-`yolo11n`, `yolo11nv2`.
-
-Available operation suites under `validation/operations/`: `add`, `concat`,
-`conv`, `div`, `gather`, `gemm`, `gemv`, `matmul`, `mul`, `pool`,
-`reduce_mean`, `relu`, `reshape`, `resize`, `sigmoid`, `softmax`, `split`.
-
-Generated operation tests can be regenerated with:
-
-```bash
-.venv/bin/python validation/operations/gen_tests.py
-```
 
 ## Build
 

validation/README.md

@@ -0,0 +1,228 @@
+# Raptor Validation
+
+`validate.py` validates every ONNX model below a selected directory. For each
+model it can:
+
+1. compile an ONNX-MLIR reference library and runner;
+2. generate deterministic random inputs;
+3. compile PIM artifacts with Raptor;
+4. run the reference implementation and functional PIM simulator;
+5. compare their outputs;
+6. run `pimsim-nn` to report latency and power.
+
+Run the script from the repository root with the repository Python environment.
+
+## Prerequisites
+
+- A built Raptor compiler, normally
+  `build_release/Release/bin/onnx-mlir`.
+- ONNX-MLIR runtime headers, normally `onnx-mlir/include`.
+- The `numpy`, `onnx`, and `colorama` packages installed in `.venv`.
+- The Rust toolchain used by the functional simulator.
+- The Rust functional simulator under
+  `backend-simulators/pim/pim-simulator`, unless overridden.
+- A built `pimsim-nn` under
+  `backend-simulators/pim/pimsim-nn/build`, unless non-functional simulation is
+  skipped or its path is overridden.
+
+## Basic usage
+
+Validate the complete operation suite:
+
+```bash
+.venv/bin/python validation/validate.py \
+  --raptor-path build_release/Release/bin/onnx-mlir \
+  --onnx-include-dir onnx-mlir/include \
+  --operations-dir validation/operations \
+  --verbose \
+  --raptor-extra-arg=--pim-detect-communication-deadlock \
+  --raptor-extra-arg=--pim-export-spatial-dataflow=none
+```
+
+Validate one operation category or case:
+
+```bash
+.venv/bin/python validation/validate.py \
+  --raptor-path build_release/Release/bin/onnx-mlir \
+  --onnx-include-dir onnx-mlir/include \
+  --operations-dir validation/operations/gemm/small
+```
+
+Validate a network or network slice:
+
+```bash
+.venv/bin/python validation/validate.py \
+  --raptor-path build_release/Release/bin/onnx-mlir \
+  --onnx-include-dir onnx-mlir/include \
+  --operations-dir validation/networks/yolo11n/depth_04
+```
+
+`--operations-dir` may point to any directory tree containing `.onnx` files.
+The script discovers them recursively.
+
+## Validation modes
+
+The default mode performs the complete workflow.
+
+Use `--compile-only` to build the reference runner and PIM artifacts without
+executing either implementation:
+
+```bash
+.venv/bin/python validation/validate.py \
+  --raptor-path build_release/Release/bin/onnx-mlir \
+  --onnx-include-dir onnx-mlir/include \
+  --operations-dir validation/operations/gemm/small \
+  --compile-only
+```
+
+Use `--run-only` to reuse those artifacts and perform input generation,
+reference execution, simulation, and comparison:
+
+```bash
+.venv/bin/python validation/validate.py \
+  --raptor-path build_release/Release/bin/onnx-mlir \
+  --onnx-include-dir onnx-mlir/include \
+  --operations-dir validation/operations/gemm/small \
+  --run-only
+```
+
+`--compile-only` and `--run-only` are mutually exclusive. Run-only mode fails
+with a diagnostic if its required compiled artifacts are missing.
+
+## Parallel execution and output
+
+Models run in parallel using all available CPUs by default. Set the worker
+count with `-j` or `--jobs`:
+
+```bash
+.venv/bin/python validation/validate.py \
+  --raptor-path build_release/Release/bin/onnx-mlir \
+  --onnx-include-dir onnx-mlir/include \
+  --operations-dir validation/operations \
+  --jobs 8
+```
+
+## Options
+
+| Option | Description |
+|---|---|
+| `-h`, `--help` | Print command help and exit. |
+| `--raptor-path PATH` | Raptor compiler binary. Required unless `--clean` is used. |
+| `--onnx-include-dir PATH` | ONNX-MLIR runtime include directory. Required unless `--clean` is used. |
+| `--operations-dir PATH` | Directory tree containing models. Defaults to `validation/operations`. |
+| `--simulator-dir PATH` | Functional `pim-simulator` crate directory. Defaults to the in-tree simulator. |
+| `--non-functional-simulator-build-dir PATH` | `pimsim-nn` build directory. Defaults to the in-tree build. |
+| `--pimcomp-config {arch-a,arch-b,arch-c}` | Non-functional hardware/timing profile. Defaults to `arch-a`. |
+| `--skip-non-functional-simulation` | Skip `pimsim-nn` latency and power measurement. |
+| `--threshold FLOAT` | Absolute output-comparison tolerance. Defaults to `1e-3`. |
+| `--relative-threshold FLOAT` | Relative output-comparison tolerance. Defaults to `1e-5`. |
+| `--seed INT` | Seed for generated inputs. Defaults to `0`. |
+| `--crossbar-size INT` | Crossbar dimensions passed to Raptor. Defaults to the Arch-A value, `128`. |
+| `--crossbar-count INT` | Crossbars per core passed to Raptor. Defaults to the Arch-A value, `96`. |
+| `--core-count INT` | PIM core count passed to Raptor. Defaults to the Arch-A value, `168`. |
+| `--raptor-extra-arg=ARG` | Additional Raptor compiler argument. Repeat for multiple arguments. |
+| `--command-timeout-seconds FLOAT` | Timeout for each compiler, runner, and simulator subprocess. Defaults to `1000000.0`. |
+| `-j INT`, `--jobs INT` | Parallel validation workers. Defaults to all available CPUs and must be at least one. |
+| `--clean` | Remove generated validation artifacts and exit. |
+| `--compile-only` | Compile reference and PIM artifacts without execution or comparison. |
+| `--run-only` | Reuse compiled artifacts and perform execution, simulation, and comparison. |
+| `--verbose` | Print passing per-stage and subprocess logs, plus average PIM pass timings. |
+
+Arguments beginning with `--` that are passed through to Raptor should use the
+equals form:
+
+```bash
+--raptor-extra-arg=--pim-detect-communication-deadlock
+```
+
+## Hardware profiles and non-functional simulation
+
+The selected PIMCOMP profile must match `--core-count`, `--crossbar-count`, and
+`--crossbar-size`. A mismatch disables only non-functional simulation and
+prints the incompatible values; functional validation still runs.
+
+The checked-in profiles are under
+`validation/pimsim_configs/pimcomp/<profile>/latency_config.json`.
+
+Use `--skip-non-functional-simulation` when latency and power are not required.
+The summary reports non-functional results as measured, failed, unsupported, or
+skipped.
+
+Overall PASS/FAIL is determined by compilation and functional output
+comparison. A non-functional simulation failure remains visible as `ERROR` in
+the latency and power columns but does not change a functional PASS.
+
+`pimsim-nn` does not currently implement the `vsoftmax` instruction. When its
+explicit unsupported-op diagnostic is encountered, Softmax validations retain
+their functional PASS and show `UNSUPPORTED` in both non-functional columns.
+Other `pimsim-nn` failures remain `ERROR`.
+
+## Generated artifacts
+
+Artifacts are written beside each model:
+
+| Path | Contents |
+|---|---|
+| `inputs/` | Generated input CSV files. |
+| `outputs/` | ONNX-MLIR reference output CSV files. |
+| `raptor/` | Exported MLIR, dialect snapshots, reports, and final `pim/` artifacts. |
+| `runner/` | Generated reference runner source, build tree, and shared library. |
+| `simulation/out.bin` | Functional simulator output used for comparison. |
+
+The `raptor/` directory may include `spatial0.mlir`,
+`spatial1_graph.mlir`, `spatial2_trivial_merged.mlir`,
+`spatial3_scheduled_no_comm.mlir`, `spatial4_scheduled.mlir`, `pim0.mlir`,
+`pim1_buff.mlir`, `pim2_folded.mlir`, and `pim3_memory_planned.mlir`.
+
+Remove these artifacts for every discovered model with:
+
+```bash
+.venv/bin/python validation/validate.py \
+  --operations-dir validation/operations/gemm/small \
+  --clean
+```
+
+`--clean` does not require `--raptor-path` or `--onnx-include-dir`.
+
+## Available suites
+
+Checked-in validation networks under `validation/networks/` include `vgg16`,
+`yolo11n`, and `yolo11nv2`.
+
+The generated operation inventory is documented in
+[`operations/README.md`](operations/README.md). Regenerate its models with:
+
+```bash
+.venv/bin/python validation/operations/gen_tests.py
+```
+
+## Manual functional simulator tracing
+
+After validation has produced a `raptor/pim/` directory, rerun the functional
+simulator with tracing from its crate directory:
+
+```bash
+cd backend-simulators/pim/pim-simulator
+cargo run --no-default-features --features tracing --release \
+  --package pim-simulator --bin pim-simulator -- \
+  -f /path/to/workspace/raptor/pim \
+  -o /path/to/workspace/simulation/out.bin \
+  -d <addr0>,<size0>,<addr1>,<size1>,...
+```
+
+Tracing writes `TraceCore0`, `TraceCore1`, and so on beside `out.bin`. The
+validator normally derives the `-d` address and byte ranges from
+`raptor/pim/config.json` and the model output shapes.
+
+## Results and exit status
+
+The final table reports functional pass/fail state and non-functional latency
+and power. The summary includes pass/fail totals, non-functional simulation
+counts, total measured latency, and average PIM pass timings when `--verbose`
+is enabled.
+
+- Exit status `0`: all discovered models passed, or cleanup completed.
+- Exit status `1`: validation failed, the model directory was invalid, or no
+  `.onnx` models were found.
+- Exit status `2`: command-line arguments were invalid or required arguments
+  were missing.

validation/raptor_validation/validate_one.py

@@ -2,6 +2,7 @@ import json
 import os
 import re
 import shutil
+import subprocess
 import sys
 import numpy as np
 from dataclasses import dataclass, field
@@ -47,12 +48,18 @@ MODE_STAGE_TITLES = {
 
 PIMSIM_DONE = "DONE"
 PIMSIM_FAILED = "ERROR"
+PIMSIM_UNSUPPORTED = "UNSUPPORTED"
 PIMSIM_SKIPPED = "SKIP"
 PIMSIM_NOT_RUN = "-"
+PIMSIM_UNSUPPORTED_VSOFTMAX = "pimsim-nn does not support binary opcode vsoftmax"
 PIMSIM_LATENCY_RE = re.compile(r"\blatency:\s+([0-9.eE+-]+)\s+ms")
 PIMSIM_POWER_RE = re.compile(r"\baverage power:\s+([0-9.eE+-]+)\s+mW")
 
 
+class PimSimUnsupportedError(RuntimeError):
+    pass
+
+
 def sanitize_output_name(name):
     return "".join(ch if ch.isalnum() or ch in "_.-" else "_" for ch in name[:255])
 
@@ -158,9 +165,9 @@ class ProgressReporter:
         if self.enabled:
             self._clear()
         if color:
-            print(color + message + Style.RESET_ALL)
+            print(color + message + Style.RESET_ALL, flush=True)
         else:
-            print(message)
+            print(message, flush=True)
         self._render()
 
     def set_stage(self, model_index, model_total, model_name, stage_name):
@@ -245,13 +252,19 @@ def pimcomp_compatibility_errors(config_path, *, core_count, crossbar_count, cro
 
 
 def run_pimsim_nn(pimsim_nn_build_dir, pim_dir, config_path, reporter=None, timeout_sec=None):
-    output = run_command(
+    try:
-        [pimsim_nn_build_dir / "ChipTest", pim_dir, config_path, "false"],
+        output = run_command(
-        cwd=pimsim_nn_build_dir,
+            [pimsim_nn_build_dir / "ChipTest", pim_dir, config_path, "false"],
-        reporter=reporter,
+            cwd=pimsim_nn_build_dir,
-        timeout_sec=timeout_sec,
+            reporter=reporter,
-        capture_output=True,
+            timeout_sec=timeout_sec,
-    )
+            capture_output=True,
+        )
+    except subprocess.CalledProcessError as exc:
+        error_output = exc.output.decode("utf-8", errors="replace") if isinstance(exc.output, bytes) else str(exc.output)
+        if PIMSIM_UNSUPPORTED_VSOFTMAX in error_output:
+            raise PimSimUnsupportedError(PIMSIM_UNSUPPORTED_VSOFTMAX) from exc
+        raise
     latency_match = PIMSIM_LATENCY_RE.search(output)
     power_match = PIMSIM_POWER_RE.search(output)
     if not latency_match or not power_match:
@@ -544,6 +557,9 @@ def validate_network(network_onnx_path, raptor_path, onnx_include_dir,
                     reporter,
                     f"Latency: {pimsim_latency_ms:.6f} ms, "
                     f"Power: {pimsim_power_mw:.6f} mW")
+            except PimSimUnsupportedError as exc:
+                pimsim_status = PIMSIM_UNSUPPORTED
+                print_info(reporter, str(exc))
             except Exception as exc:
                 pimsim_status = PIMSIM_FAILED
                 reporter.suspend()
@@ -559,7 +575,6 @@ def validate_network(network_onnx_path, raptor_path, onnx_include_dir,
             print_info(reporter, "pimsim-nn non-functional simulation skipped")
         reporter.advance()
 
-        passed = passed and pimsim_status != PIMSIM_FAILED
         reporter.record_result(passed)
         status = Fore.GREEN + "PASS" + Style.RESET_ALL if passed else Fore.RED + "FAIL" + Style.RESET_ALL
         reporter.log(Style.BRIGHT + f"Result: {status}" + Style.RESET_ALL)

validation/validate.py

@@ -1,10 +1,15 @@
 #!/usr/bin/env python3
 
 import argparse
+import os
 import signal
 import subprocess
 import sys
+from concurrent.futures import ProcessPoolExecutor, as_completed
+from contextlib import nullcontext, redirect_stderr, redirect_stdout
+from itertools import groupby
 from pathlib import Path
+from tempfile import TemporaryDirectory
 from colorama import Style, Fore
 from raptor_validation.validate_one import (
     MODE_COMPILE_ONLY,
@@ -15,6 +20,7 @@ from raptor_validation.validate_one import (
     PIMSIM_FAILED,
     PIMSIM_NOT_RUN,
     PIMSIM_SKIPPED,
+    PIMSIM_UNSUPPORTED,
     ProgressReporter,
     ValidationResult,
     clean_workspace_artifacts,
@@ -28,6 +34,56 @@ DEFAULT_PIMCOMP_CONFIG = "arch-a"
 PIMCOMP_CONFIG_CHOICES = ("arch-a", "arch-b", "arch-c")
 
 
+def run_validation_job(job):
+    models, options = job
+    completed = []
+    for index, onnx_path, rel, log_path in models:
+        reporter = ProgressReporter(
+            options["model_total"],
+            stages_per_model=len(MODE_STAGE_TITLES[options["mode"]]),
+            enabled=False,
+            verbose=options["verbose"],
+        )
+
+        def validate():
+            try:
+                return validate_network(
+                    onnx_path,
+                    reporter=reporter,
+                    model_index=index,
+                    model_total=options["model_total"],
+                    verbose=options["verbose"],
+                    mode=options["mode"],
+                    **options["validation_kwargs"],
+                )
+            except Exception as exc:
+                print_validation_error(reporter, rel, exc)
+                return ValidationResult(False, pimsim_status=PIMSIM_NOT_RUN)
+            finally:
+                reporter.finish()
+
+        sys.stdout.flush()
+        sys.stderr.flush()
+        if log_path is None:
+            result = validate()
+        else:
+            saved_stdout = os.dup(1)
+            saved_stderr = os.dup(2)
+            try:
+                with open(log_path, "w", encoding="utf-8", buffering=1) as log:
+                    os.dup2(log.fileno(), 1)
+                    os.dup2(log.fileno(), 2)
+                    with redirect_stdout(log), redirect_stderr(log):
+                        result = validate()
+            finally:
+                os.dup2(saved_stdout, 1)
+                os.dup2(saved_stderr, 2)
+                os.close(saved_stdout)
+                os.close(saved_stderr)
+        completed.append((str(rel), result, str(log_path) if log_path else None))
+    return completed
+
+
 def format_return_status(returncode):
     if returncode < 0:
         signal_num = -returncode
@@ -111,6 +167,8 @@ def main():
                     help="Additional argument to pass through to the Raptor compiler. Repeat as needed.")
     ap.add_argument("--command-timeout-seconds", type=float, default=1000000.0,
                     help="Per-subprocess timeout in seconds for compiler, runner, and simulation commands.")
+    ap.add_argument("-j", "--jobs", type=int, default=os.cpu_count() or 1,
+                    help="Number of model validations to run in parallel (default: all available CPUs).")
     ap.add_argument("--clean", action="store_true",
                     help="Remove generated validation artifacts under each model workspace and exit.")
     mode_group = ap.add_mutually_exclusive_group()
@@ -123,6 +181,8 @@ def main():
     ap.add_argument("--verbose", action="store_true",
                     help="Print per-stage progress and subprocess logs for passing validations too.")
     a = ap.parse_args()
+    if a.jobs < 1:
+        ap.error("--jobs must be at least 1")
 
     operations_dir = Path(a.operations_dir).resolve() if a.operations_dir else script_dir / "operations"
     simulator_dir = Path(a.simulator_dir).resolve() if a.simulator_dir else (
@@ -161,6 +221,7 @@ def main():
 
     print(Style.BRIGHT + f"Found {len(onnx_files)} ONNX file(s) to validate." + Style.RESET_ALL)
     print(f"Operations root: {operations_dir}")
+    print(f"Max parallel jobs: {a.jobs}")
     print("=" * 72)
 
     mode = MODE_FULL
@@ -193,43 +254,79 @@ def main():
     pass_timing_counts = {label: 0 for _, label in PIM_PASS_LABELS}
     total_timing_sum = 0.0
     timed_benchmark_count = 0
-    reporter = ProgressReporter(len(onnx_files), stages_per_model=len(MODE_STAGE_TITLES[mode]), verbose=a.verbose)
+    reporter = ProgressReporter(len(onnx_files), stages_per_model=1, verbose=a.verbose)
-    for index, onnx_path in enumerate(onnx_files, start=1):
+    validation_kwargs = {
-        rel = onnx_path.relative_to(operations_dir)
+        "raptor_path": a.raptor_path,
-        try:
+        "onnx_include_dir": a.onnx_include_dir,
-            result = validate_network(
+        "simulator_dir": simulator_dir,
-                onnx_path, a.raptor_path, a.onnx_include_dir, simulator_dir,
+        "crossbar_size": a.crossbar_size,
-                crossbar_size=a.crossbar_size, crossbar_count=a.crossbar_count, core_count=a.core_count,
+        "crossbar_count": a.crossbar_count,
-                raptor_extra_args=a.raptor_extra_arg,
+        "core_count": a.core_count,
-                pimsim_nn_build_dir=pimsim_nn_build_dir,
+        "raptor_extra_args": a.raptor_extra_arg,
-                pimsim_config_path=selected_pimsim_config,
+        "pimsim_nn_build_dir": pimsim_nn_build_dir,
-                command_timeout_seconds=a.command_timeout_seconds,
+        "pimsim_config_path": selected_pimsim_config,
-                threshold=a.threshold,
+        "command_timeout_seconds": a.command_timeout_seconds,
-                rtol=a.relative_threshold,
+        "threshold": a.threshold,
-                seed=a.seed,
+        "rtol": a.relative_threshold,
-                reporter=reporter,
+        "seed": a.seed,
-                model_index=index,
+    }
-                model_total=len(onnx_files),
+    indexed_files = list(enumerate(onnx_files, start=1))
-                verbose=a.verbose,
+    workspace_groups = [
-                mode=mode,
+        list(group)
+        for _, group in groupby(indexed_files, key=lambda indexed_path: indexed_path[1].parent)
+    ]
+    print_directly = min(a.jobs, len(workspace_groups)) == 1
+    with (nullcontext(None) if print_directly else TemporaryDirectory(prefix="raptor-validation-")) as log_dir:
+        jobs = [
+            (
+                [
+                    (
+                        index,
+                        onnx_path,
+                        onnx_path.relative_to(operations_dir),
+                        Path(log_dir) / f"{index}.log" if log_dir else None,
+                    )
+                    for index, onnx_path in workspace_group
+                ],
+                {
+                    "model_total": len(onnx_files),
+                    "mode": mode,
+                    "verbose": a.verbose,
+                    "validation_kwargs": validation_kwargs,
+                },
             )
-            results[str(rel)] = result
+            for workspace_group in workspace_groups
-            if result.pim_pass_timings:
+        ]
-                benchmark_total = 0.0
+        with (nullcontext(None) if print_directly else ProcessPoolExecutor(max_workers=a.jobs)) as executor:
-                for label, duration in result.pim_pass_timings.items():
+            completed_groups = (
-                    pass_timing_sums[label] += duration
+                map(run_validation_job, jobs)
-                    pass_timing_counts[label] += 1
+                if print_directly
-                    benchmark_total += duration
+                else (future.result() for future in as_completed(
-                total_timing_sum += benchmark_total
+                    executor.submit(run_validation_job, job) for job in jobs
-                timed_benchmark_count += 1
+                ))
-        except subprocess.CalledProcessError as exc:
+            )
-            results[str(rel)] = ValidationResult(False, pimsim_status=PIMSIM_NOT_RUN)
+            for completed_group in completed_groups:
-            print_validation_error(reporter, rel, exc)
+                for rel, result, log_path in completed_group:
-        except Exception as exc:
+                    if log_path:
-            results[str(rel)] = ValidationResult(False, pimsim_status=PIMSIM_NOT_RUN)
+                        reporter.suspend()
-            print_validation_error(reporter, rel, exc)
+                        output = Path(log_path).read_text(encoding="utf-8", errors="replace")
+                        if output:
+                            print(output, end="" if output.endswith("\n") else "\n")
+                        reporter.resume()
+                    reporter.advance()
+                    reporter.record_result(result.passed)
+                    results[rel] = result
+                    if result.pim_pass_timings:
+                        benchmark_total = 0.0
+                        for label, duration in result.pim_pass_timings.items():
+                            pass_timing_sums[label] += duration
+                            pass_timing_counts[label] += 1
+                            benchmark_total += duration
+                        total_timing_sum += benchmark_total
+                        timed_benchmark_count += 1
 
     reporter.finish()
+    results = dict(sorted(results.items()))
 
     # Summary
     n_passed = sum(1 for result in results.values() if result.passed)
@@ -251,7 +348,7 @@ def main():
     print(separator)
     print(
         f"| {'Operation'.ljust(path_width)} | {'Result'.ljust(status_width)} | "
-        f"{'Latency'.ljust(latency_width)} | {'Power'.ljust(power_width)} |"
+        f"{'Latency'.rjust(latency_width)} | {'Power'.rjust(power_width)} |"
     )
     print(separator)
     for rel, result in results.items():
@@ -260,8 +357,8 @@ def main():
             Fore.RED + plain_status.ljust(status_width) + Style.RESET_ALL
         latency, power = formatted_metrics[rel]
         print(
-            f"| {rel.ljust(path_width)} | {status} | {latency.ljust(latency_width)} | "
+            f"| {rel.ljust(path_width)} | {status} | {latency.rjust(latency_width)} | "
-            f"{power.ljust(power_width)} |")
+            f"{power.rjust(power_width)} |")
     print(separator)
     print("\n" + Style.BRIGHT + Fore.CYAN + "Summary" + Style.RESET_ALL)
     print(Style.BRIGHT + f"Passed: {n_passed}" + Style.RESET_ALL)
@@ -278,10 +375,14 @@ def main():
         result.pimsim_status in (PIMSIM_SKIPPED, PIMSIM_NOT_RUN)
         for result in results.values()
     )
+    pimsim_unsupported = sum(
+        result.pimsim_status == PIMSIM_UNSUPPORTED for result in results.values()
+    )
     print(
         Style.BRIGHT
         + f"pimsim-nn: {len(measured_latencies)} measured, "
-        f"{pimsim_failed} failed, {pimsim_skipped} skipped"
+        f"{pimsim_failed} failed, {pimsim_unsupported} unsupported, "
+        f"{pimsim_skipped} skipped"
         + Style.RESET_ALL
     )
     if measured_latencies: