better compile times for throughput scheduling
Validate Operations / validate-operations (push) Has been cancelled
Validate Operations / validate-operations (push) Has been cancelled
fix failing no-plan throughput ablation runs update results
This commit is contained in:
@@ -11,11 +11,11 @@ operations, standalone commands, configuration layout, and GA/OpenMP controls.
|
||||
|
||||
| Directory | Model / input | Source |
|
||||
|---|---|---|
|
||||
| `resnet18/` | ResNet-18 v1, `1x3x224x224` | [ONNX Model Zoo resnet18-v1-7](https://huggingface.co/onnxmodelzoo/resnet18-v1-7); symlink to `../resnet18/depth_68/resnet18_depth_68.onnx`. |
|
||||
| `resnet18/` | ResNet-18 v1, `1x3x224x224` | [ONNX Model Zoo resnet18-v1-7](https://huggingface.co/onnxmodelzoo/resnet18-v1-7); symlink to `../../resnet18/depth_68/resnet18_depth_68.onnx`. |
|
||||
| `resnet34/` | ResNet-34 v1, `1x3x224x224` | [ONNX Model Zoo resnet34-v1-7](https://huggingface.co/onnxmodelzoo/resnet34-v1-7); symbolic batch fixed to 1. |
|
||||
| `googlenet/` | GoogLeNet, `1x3x224x224` | [ONNX Model Zoo googlenet-12](https://huggingface.co/onnxmodelzoo/googlenet-12), with only its two LRN nodes bypassed. |
|
||||
| `vgg8/` | VGG-8 reconstruction, `1x1x28x28` | Six convolution and two fully connected layers with deterministic random weights. |
|
||||
| `yolo11n/` | YOLO11n detection, `1x3x640x640` | Uses `../yolo11n/depth_51/yolo11n_depth_51.onnx`; this directory retains the historical Softmax-free variant. |
|
||||
| `yolo11n/` | YOLO11n detection, `1x3x640x640` | [Ultralytics YOLO11n](https://docs.ultralytics.com/models/yolo11/); symlink to `../../yolo11n/depth_51/yolo11n_depth_51.onnx`. |
|
||||
|
||||
The paper's checkpoints were not published. The three Model Zoo networks
|
||||
produce frontend graphs matching Pimcomp's bundled JSON graphs. VGG-8 is
|
||||
@@ -29,15 +29,9 @@ Its terminal Softmax, output `prob_1`, weights, and other operations are retaine
|
||||
The native reference uses this same graph. VGG8 and both ResNet graphs have no
|
||||
removed operators; ResNet-34 fixes the symbolic batch size to one.
|
||||
|
||||
Historical performance models change the original computation:
|
||||
|
||||
- `googlenet/googlenet-12-pimsim-nn.onnx` removes two LRN nodes and terminal
|
||||
Softmax, which Pimcomp does not schedule.
|
||||
- `yolo11n/yolo11n-pimsim-nn.onnx` removes two internal Softmax nodes. It remains
|
||||
available for reproducing historical measurements.
|
||||
|
||||
The current YOLO comparison uses `../yolo11n/depth_51/yolo11n_depth_51.onnx`,
|
||||
The current YOLO comparison uses `yolo11n/yolo11n.onnx`,
|
||||
including both Softmax operations. Its native reference uses the same full model.
|
||||
The checked-in ONNX graph uses opset 18. The original exporter version and checkpoint hash are not recorded.
|
||||
The functional simulator executes Softmax. Pimsim-nn skips `vsoftmax` before
|
||||
fetch, and Raptor's PIM queue estimator omits the same instruction cost.
|
||||
Timing and energy exclude Softmax work; they do not model hardware Softmax support.
|
||||
@@ -50,9 +44,8 @@ Model SHA-256 checksums:
|
||||
c3231061d081bdd47884137b02134f85142752a39e87263c529cd14ed242b096 resnet34/resnet34-v1-7.onnx
|
||||
c99c507058eaf41de8723408fdda7db8325cb57f0a89f2ee07a716d6e963e14e googlenet/googlenet-12.onnx
|
||||
c960c195953423388733cd0e60712cf425ca1f8917e264f5cf98e028752c61cf googlenet/googlenet-12-no-lrn.onnx
|
||||
a26f9e33901c573e60c34a3f0abbb4744fff83e4e0f21b18fc66e20395e72982 googlenet/googlenet-12-pimsim-nn.onnx
|
||||
396cdea21e5e7d02c3f26f14d22ef20975171702493f5c5e79b8e0d896e541ef vgg8/vgg8-mnist-reconstructed.onnx
|
||||
229f3975af8933d39aee8d9031d969bff074b69c33e304a78abb35ff0c5f445f yolo11n/yolo11n-pimsim-nn.onnx
|
||||
5e7bd85dc60b252674f8d7e1ad24f2317d00dfb8c524826536698462de04620d yolo11n/yolo11n.onnx
|
||||
```
|
||||
|
||||
## Hardware profiles
|
||||
|
||||
Binary file not shown.
Binary file not shown.
@@ -0,0 +1 @@
|
||||
../../yolo11n/depth_51/yolo11n_depth_51.onnx
|
||||
@@ -11,7 +11,7 @@ MODELS = {
|
||||
"resnet18": SUITE / "resnet18/resnet18-v1-7.onnx",
|
||||
"resnet34": SUITE / "resnet34/resnet34-v1-7.onnx",
|
||||
"googlenet": SUITE / "googlenet/googlenet-12-no-lrn.onnx",
|
||||
"yolo11n": REPO / "validation/networks/yolo11n/depth_51/yolo11n_depth_51.onnx",
|
||||
"yolo11n": SUITE / "yolo11n/yolo11n.onnx",
|
||||
}
|
||||
MODEL_NAMES = tuple(MODELS)
|
||||
DEFAULT_MODELS = MODEL_NAMES
|
||||
|
||||
@@ -511,7 +511,9 @@ def test_comparison_uses_same_model_for_reference_and_execution(tmp_path, monkey
|
||||
|
||||
|
||||
def test_yolo_uses_full_model_without_moving_comparison_artifacts():
|
||||
assert suite.MODELS["yolo11n"] == suite.REPO / "validation/networks/yolo11n/depth_51/yolo11n_depth_51.onnx"
|
||||
assert suite.MODELS["yolo11n"] == suite.SUITE / "yolo11n/yolo11n.onnx"
|
||||
assert suite.MODELS["yolo11n"].is_symlink()
|
||||
assert suite.MODELS["yolo11n"].resolve() == suite.REPO / "validation/networks/yolo11n/depth_51/yolo11n_depth_51.onnx"
|
||||
assert suite.model_dir(None, "yolo11n") == suite.SUITE / "yolo11n"
|
||||
assert suite.common_dir(None, "yolo11n") == suite.SUITE / "yolo11n/artifacts/common-full-model"
|
||||
|
||||
@@ -756,3 +758,43 @@ def test_ablation_aggregate_keeps_pass_with_zero_performance(tmp_path):
|
||||
assert rows[0]["functional_validation"] == "PASS"
|
||||
assert rows[0]["baseline_functional_validation"] == "PASS"
|
||||
assert all(rows[0][f"{metric}_gain_percent"] == "NA" for _, metric in ablation.PERCENTAGE_FIELDS)
|
||||
|
||||
|
||||
def test_completed_no_sync_ablation_preserves_numerical_failure():
|
||||
report = {"pimsim_mode": "throughput", "raptor_extra_args": ["--pim-disable-synchronization"],
|
||||
"raptor_validation": {"status": "done", "passed": False, "max_diffs": {"output": 1}},
|
||||
"raptor_performance": {"throughput": 100, "output_count": 8},
|
||||
"steps": [{"name": "Functional validation Raptor", "status": "passed"}],
|
||||
"failures": []}
|
||||
assert suite.completed_synchronization_ablation(report)
|
||||
assert not suite.comparison_passed(report)
|
||||
assert suite.functional_validation_status(report["raptor_validation"]) == "FAIL"
|
||||
for field, value in (("pimsim_mode", "latency"), ("raptor_extra_args", []),
|
||||
("raptor_validation", {"status": "failed", "passed": False}),
|
||||
("raptor_validation", {"status": "done", "passed": False}),
|
||||
("raptor_performance", {"error": "SIGSEGV", "throughput": 100}),
|
||||
("raptor_performance", {"throughput": 100, "output_count": 0}),
|
||||
("steps", [{"status": "failed"}]),
|
||||
("failures", [{"stage": "Compile Raptor Pim", "error": "failed"}])):
|
||||
assert not suite.completed_synchronization_ablation({**report, field: value})
|
||||
|
||||
|
||||
@pytest.mark.parametrize("process_failed", [False, True])
|
||||
@pytest.mark.parametrize("variant,expected", [("no-sync", 0), ("no-plan", 1), ("baseline", 1)])
|
||||
def test_ablation_runner_distinguishes_invalid_outputs_from_failed_execution(tmp_path, monkeypatch, variant, expected, process_failed):
|
||||
report = {"pimsim_mode": "throughput", "raptor_extra_args": ["--pim-disable-synchronization"],
|
||||
"raptor_validation": {"status": "done", "passed": False, "max_diffs": {"output": 1}},
|
||||
"raptor_performance": {"throughput": 100, "output_count": 8},
|
||||
"steps": [{"status": "passed"}], "failures": []}
|
||||
output = suite.result_dir(tmp_path, "vgg8", "arch-a", "throughput", 4, variant) / "pimcomp/comparison_report.json"
|
||||
output.parent.mkdir(parents=True)
|
||||
output.write_text(json.dumps(report))
|
||||
csv = tmp_path / "results.csv"
|
||||
csv.write_text("summary\n")
|
||||
monkeypatch.setattr(suite, "run", lambda *a, **k: 0)
|
||||
monkeypatch.setattr(suite, "run_comparison_jobs", lambda *a: (["arch-a/vgg8/throughput/pipeline4"] if process_failed else [], 1))
|
||||
monkeypatch.setattr(suite, "write_results_csv", lambda *a: csv)
|
||||
monkeypatch.setattr(sys, "argv", ["run_pimcomp_models.py", "--out-dir", str(tmp_path),
|
||||
"--models", "vgg8", "--archs", "arch-a", "--mode", "throughput", "--pipeline", "4",
|
||||
"--raptor-only", "--ablation-variant", variant, "--jobs", "1"])
|
||||
assert suite.main() == (1 if process_failed else expected)
|
||||
|
||||
@@ -59,8 +59,10 @@ Both programs' functional statuses are recorded as `functional_validation`
|
||||
and `baseline_functional_validation`. The latter belongs to the disabled feature.
|
||||
|
||||
Disabling synchronization removes host-buffer lifetime and pipeline restart
|
||||
guarantees, so those variants may fail numerical validation and exit nonzero.
|
||||
Baseline and `no-plan` must pass. Percentages involving failing programs remain
|
||||
guarantees, so those variants may fail numerical validation. A completed
|
||||
`no-sync` measurement retains `FAIL` in the CSV and is reported as diagnostic
|
||||
in the runner summary. Compile, execution, missing-output, and measurement
|
||||
failures still make the run fail. Baseline and `no-plan` must pass. Percentages involving failing programs remain
|
||||
diagnostic comparisons, not valid speedups.
|
||||
|
||||
When fast-mode convergence fails, positive fallback throughput remains available.
|
||||
|
||||
@@ -1,65 +1,25 @@
|
||||
variant,arch,model,mode,raptor_pipeline,functional_validation,latency,throughput,power,energy
|
||||
baseline,arch-a,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-a,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,-0.00%
|
||||
no-plan,arch-a,googlenet,latency,1,PASS,+11513.29%,NA,-69.42%,+3452.69%
|
||||
no-sync-no-plan,arch-a,googlenet,latency,1,PASS,+11512.99%,NA,-69.42%,+3452.59%
|
||||
baseline,arch-a,googlenet,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-a,googlenet,throughput,4,FAIL,+26.17%,-20.74%,+9.97%,+18.50%
|
||||
no-plan,arch-a,googlenet,throughput,4,PASS,-100.00%,-100.00%,-62.73%,-100.00%
|
||||
no-sync-no-plan,arch-a,googlenet,throughput,4,FAIL,+4094.63%,-97.62%,-59.86%,+1311.01%
|
||||
baseline,arch-a,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-a,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-plan,arch-a,resnet18,latency,1,PASS,+3191.30%,NA,-60.97%,+1184.24%
|
||||
no-sync-no-plan,arch-a,resnet18,latency,1,PASS,+3191.30%,NA,-60.97%,+1184.24%
|
||||
baseline,arch-a,resnet18,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-a,resnet18,throughput,4,FAIL,+110.65%,-52.55%,+43.25%,+177.48%
|
||||
no-plan,arch-a,resnet18,throughput,4,PASS,+1598.93%,-94.12%,-51.04%,+664.87%
|
||||
no-sync-no-plan,arch-a,resnet18,throughput,4,FAIL,+1174.21%,-92.16%,-45.06%,+543.74%
|
||||
baseline,arch-a,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-a,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-plan,arch-a,resnet34,latency,1,PASS,+2361.37%,NA,-56.53%,+969.99%
|
||||
no-sync-no-plan,arch-a,resnet34,latency,1,PASS,+2361.37%,NA,-56.53%,+969.99%
|
||||
baseline,arch-a,resnet34,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-a,resnet34,throughput,4,FAIL,+33.65%,-25.21%,+31.76%,+15.56%
|
||||
no-plan,arch-a,resnet34,throughput,4,PASS,+1726.82%,-94.52%,-54.50%,+455.98%
|
||||
no-sync-no-plan,arch-a,resnet34,throughput,4,FAIL,+813.41%,-89.04%,-49.77%,+206.84%
|
||||
baseline,arch-a,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-a,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-plan,arch-a,vgg8,latency,1,PASS,+1800.66%,NA,-44.65%,+953.39%
|
||||
no-sync-no-plan,arch-a,vgg8,latency,1,PASS,+1800.66%,NA,-44.65%,+953.39%
|
||||
baseline,arch-a,vgg8,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-a,vgg8,throughput,4,PASS,+2.33%,-1.72%,+18.49%,-0.98%
|
||||
no-plan,arch-a,vgg8,throughput,4,PASS,+1251.16%,-92.58%,-56.68%,+577.04%
|
||||
no-sync-no-plan,arch-a,vgg8,throughput,4,PASS,+1260.47%,-92.63%,-53.36%,+467.83%
|
||||
baseline,arch-b,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-b,googlenet,latency,1,PASS,+0.00%,NA,+0.00%,-0.00%
|
||||
no-plan,arch-b,googlenet,latency,1,PASS,+9849.64%,NA,-70.51%,+2833.89%
|
||||
no-sync-no-plan,arch-b,googlenet,latency,1,PASS,+9849.55%,NA,-70.51%,+2833.89%
|
||||
baseline,arch-b,googlenet,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-b,googlenet,throughput,4,FAIL,+11.45%,-10.29%,+21.82%,+19.27%
|
||||
no-plan,arch-b,googlenet,throughput,4,PASS,-100.00%,-100.00%,-63.39%,-100.00%
|
||||
no-sync-no-plan,arch-b,googlenet,throughput,4,FAIL,+3348.28%,-97.10%,-61.58%,+1026.09%
|
||||
baseline,arch-b,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-b,resnet18,latency,1,PASS,+0.00%,NA,+0.00%,-0.00%
|
||||
no-plan,arch-b,resnet18,latency,1,PASS,+2935.63%,NA,-63.12%,+1019.81%
|
||||
no-sync-no-plan,arch-b,resnet18,latency,1,PASS,+2936.09%,NA,-63.12%,+1019.97%
|
||||
baseline,arch-b,resnet18,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-b,resnet18,throughput,4,FAIL,+8.03%,-7.44%,+47.59%,+40.61%
|
||||
no-plan,arch-b,resnet18,throughput,4,PASS,+1511.85%,-93.80%,-52.93%,+560.80%
|
||||
no-sync-no-plan,arch-b,resnet18,throughput,4,FAIL,+867.12%,-89.67%,-45.41%,+359.83%
|
||||
baseline,arch-b,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-b,resnet34,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-plan,arch-b,resnet34,latency,1,PASS,+2551.16%,NA,-62.51%,+893.70%
|
||||
no-sync-no-plan,arch-b,resnet34,latency,1,PASS,+2550.91%,NA,-62.51%,+893.62%
|
||||
baseline,arch-b,resnet34,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-b,resnet34,throughput,4,FAIL,+28.25%,-22.13%,+49.76%,+57.30%
|
||||
no-plan,arch-b,resnet34,throughput,4,PASS,+2427.17%,-96.05%,-49.58%,+938.62%
|
||||
no-sync-no-plan,arch-b,resnet34,throughput,4,FAIL,+531.79%,-84.19%,-42.32%,+197.08%
|
||||
baseline,arch-b,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-sync,arch-b,vgg8,latency,1,PASS,+0.00%,NA,+0.00%,+0.00%
|
||||
no-plan,arch-b,vgg8,latency,1,PASS,+1784.97%,NA,-49.50%,+849.73%
|
||||
no-sync-no-plan,arch-b,vgg8,latency,1,PASS,+1784.97%,NA,-49.50%,+849.73%
|
||||
baseline,arch-b,vgg8,throughput,4,PASS,+0.00%,+0.00%,+0.00%,+0.00%
|
||||
no-sync,arch-b,vgg8,throughput,4,PASS,+23.91%,-19.82%,+25.67%,+23.93%
|
||||
no-plan,arch-b,vgg8,throughput,4,PASS,+1158.70%,-92.07%,-58.95%,+408.33%
|
||||
no-sync-no-plan,arch-b,vgg8,throughput,4,PASS,+1172.83%,-92.17%,-55.83%,+327.00%
|
||||
comparison,arch,model,mode,raptor_pipeline,baseline_variant,baseline_functional_validation,functional_validation,latency_gain_percent,throughput_gain_percent,power_gain_percent,energy_gain_percent
|
||||
plan,arch-a,googlenet,latency,1,no-plan,PASS,PASS,+99.14%,NA,-226.98%,+97.19%
|
||||
plan,arch-a,googlenet,throughput,4,no-plan,PASS,PASS,NA,NA,-281.83%,NA
|
||||
sync,arch-a,googlenet,throughput,4,no-sync,FAIL,PASS,-5.17%,-4.96%,+12.23%,-1.65%
|
||||
plan,arch-a,resnet18,latency,1,no-plan,PASS,PASS,+96.96%,NA,-156.23%,+92.21%
|
||||
plan,arch-a,resnet18,throughput,4,no-plan,PASS,PASS,+95.17%,+1970.00%,-157.48%,+85.39%
|
||||
sync,arch-a,resnet18,throughput,4,no-sync,FAIL,PASS,+7.15%,+7.81%,+14.70%,+7.92%
|
||||
plan,arch-a,resnet34,latency,1,no-plan,PASS,PASS,+95.94%,NA,-130.05%,+90.65%
|
||||
plan,arch-a,resnet34,throughput,4,no-plan,PASS,PASS,+96.00%,+2400.00%,-189.18%,+88.43%
|
||||
sync,arch-a,resnet34,throughput,4,no-sync,FAIL,PASS,-9.11%,-8.42%,+7.32%,-1.02%
|
||||
plan,arch-a,vgg8,latency,1,no-plan,PASS,PASS,+94.75%,NA,-80.65%,+90.51%
|
||||
plan,arch-a,vgg8,throughput,4,no-plan,PASS,PASS,+92.77%,+1279.31%,-170.46%,+82.49%
|
||||
sync,arch-a,vgg8,throughput,4,no-sync,FAIL,PASS,+19.18%,+24.03%,+21.29%,+18.62%
|
||||
plan,arch-b,googlenet,latency,1,no-plan,PASS,PASS,+98.99%,NA,-239.10%,+96.59%
|
||||
plan,arch-b,googlenet,throughput,4,no-plan,PASS,PASS,NA,NA,-306.99%,NA
|
||||
sync,arch-b,googlenet,throughput,4,no-sync,FAIL,PASS,-1.28%,-1.63%,+9.99%,-4.29%
|
||||
plan,arch-b,resnet18,latency,1,no-plan,PASS,PASS,+96.71%,NA,-171.16%,+91.07%
|
||||
plan,arch-b,resnet18,throughput,4,no-plan,PASS,PASS,+95.46%,+2102.50%,-200.52%,+83.26%
|
||||
sync,arch-b,resnet18,throughput,4,no-sync,FAIL,PASS,+2.22%,+2.32%,+8.26%,-7.88%
|
||||
plan,arch-b,resnet34,latency,1,no-plan,PASS,PASS,+96.23%,NA,-166.76%,+89.94%
|
||||
plan,arch-b,resnet34,throughput,4,no-plan,PASS,PASS,+97.58%,+4030.00%,-168.71%,+91.79%
|
||||
sync,arch-b,resnet34,throughput,4,no-sync,FAIL,PASS,-0.80%,-0.96%,+10.61%,-13.51%
|
||||
plan,arch-b,vgg8,latency,1,no-plan,PASS,PASS,+94.68%,NA,-98.04%,+89.47%
|
||||
plan,arch-b,vgg8,throughput,4,no-plan,PASS,PASS,+93.70%,+1484.16%,-205.54%,+81.37%
|
||||
sync,arch-b,vgg8,throughput,4,no-sync,FAIL,PASS,+17.07%,+20.30%,+19.03%,+9.63%
|
||||
|
||||
|
@@ -1,41 +1,41 @@
|
||||
arch,model,mode,raptor_pipeline,pimcomp_pipeline,raptor_functional_validation,pimcomp_functional_validation,raptor_latency_ms,pimcomp_latency_ms,raptor_throughput_samples_s,pimcomp_throughput_samples_s,raptor_power_mw,pimcomp_power_mw,raptor_energy_pj,pimcomp_energy_pj,better_compiler,speedup
|
||||
arch-a,vgg8,latency,1,element,PASS,PASS,1.51833,7.98507,NA,NA,321.046,200.111,4.87455e+08,1.5979e+09,raptor,5.26
|
||||
arch-a,vgg8,throughput,2,batch,PASS,PASS,0.906321,1.0364,1100,965,429.311,482.597,4.07972e+08,5.41692e+08,raptor,1.14
|
||||
arch-a,vgg8,throughput,4,batch,PASS,PASS,0.778242,1.0364,1280,965,457.039,482.597,4.15762e+08,5.41692e+08,raptor,1.33
|
||||
arch-a,vgg8,throughput,8,batch,PASS,PASS,0.778094,1.0364,1290,965,464.071,482.597,3.96424e+08,5.41692e+08,raptor,1.34
|
||||
arch-a,resnet18,latency,1,element,PASS,FAIL,14.0332,58.1775,NA,NA,455.67,238.389,6.3945e+09,1.38689e+10,raptor,4.15
|
||||
arch-a,resnet18,throughput,2,batch,PASS,FAIL,11.7314,13.4585,85.2,74.3,503.111,481.604,6.62807e+09,6.56345e+09,raptor,1.15
|
||||
arch-a,resnet18,throughput,4,batch,PASS,FAIL,19.2072,13.4585,52.1,74.3,380.1,481.604,7.95703e+09,6.56345e+09,pimcomp,1.43
|
||||
arch-a,resnet18,throughput,8,batch,PASS,FAIL,14.1578,13.4585,70.6,74.3,453.416,481.604,6.77909e+09,6.56345e+09,pimcomp,1.05
|
||||
arch-a,resnet34,latency,1,element,PASS,FAIL,29.7186,66.7114,NA,NA,411.806,277.485,1.22383e+10,1.85114e+10,raptor,2.24
|
||||
arch-a,resnet34,throughput,2,batch,PASS,FAIL,23.2705,23.5817,43,42.4,471.717,515.68,1.3055e+10,1.22836e+10,raptor,1.01
|
||||
arch-a,resnet34,throughput,4,batch,PASS,FAIL,27.468,23.5817,36.4,42.4,422.223,515.68,1.48991e+10,1.22836e+10,pimcomp,1.16
|
||||
arch-a,resnet34,throughput,8,batch,PASS,FAIL,39.8392,23.5817,25.1,42.4,352.28,515.68,1.70627e+10,1.22836e+10,pimcomp,1.69
|
||||
arch-a,googlenet,latency,1,element,PASS,PASS,9.77772,62.9248,NA,NA,563.864,231.193,5.5133e+09,1.45477e+10,raptor,6.44
|
||||
arch-a,googlenet,throughput,2,batch,PASS,FAIL,8.41545,18.2643,119,54.8,610.746,435.107,5.77746e+09,8.16479e+09,raptor,2.17
|
||||
arch-a,googlenet,throughput,4,batch,PASS,FAIL,10.7292,18.2643,93.2,54.8,528.026,435.107,6.59289e+09,8.16479e+09,raptor,1.70
|
||||
arch-a,googlenet,throughput,8,batch,PASS,FAIL,10.1219,18.2643,98.8,54.8,523.808,435.107,6.52174e+09,8.16479e+09,raptor,1.80
|
||||
arch-a,yolo11n,latency,1,element,PASS,FAIL,147.49,NA,NA,NA,290.174,NA,4.27977e+10,NA,NA,NA
|
||||
arch-a,yolo11n,throughput,2,batch,SKIPPED,FAIL,102.337,NA,9.77,NA,337.882,NA,3.83081e+10,NA,NA,NA
|
||||
arch-a,yolo11n,throughput,4,batch,SKIPPED,FAIL,87.9103,NA,11.4,NA,358.201,NA,3.87298e+10,NA,NA,NA
|
||||
arch-a,yolo11n,throughput,8,batch,SKIPPED,FAIL,82.9508,NA,12.1,NA,366.725,NA,3.85347e+10,NA,NA,NA
|
||||
arch-a,vgg8,throughput,2,batch,PASS,FAIL,0.625998,1.0364,1600,965,523.278,482.597,4.17419e+08,5.41692e+08,NA,1.66
|
||||
arch-a,vgg8,throughput,4,batch,PASS,FAIL,0.625998,1.0364,1600,965,523.278,482.597,4.17419e+08,5.41692e+08,NA,1.66
|
||||
arch-a,vgg8,throughput,8,batch,PASS,FAIL,0.62635,1.0364,1600,965,519.182,482.597,4.14242e+08,5.41692e+08,NA,1.66
|
||||
arch-a,resnet18,latency,1,element,PASS,FAIL,14.0332,58.1775,NA,NA,455.67,238.389,6.3945e+09,1.38689e+10,NA,4.15
|
||||
arch-a,resnet18,throughput,2,batch,PASS,FAIL,11.0904,13.4585,90.2,74.3,520.244,481.604,6.55167e+09,6.56345e+09,NA,1.21
|
||||
arch-a,resnet18,throughput,4,batch,PASS,FAIL,12.0822,13.4585,82.8,74.3,487.576,481.604,6.91662e+09,6.56345e+09,NA,1.11
|
||||
arch-a,resnet18,throughput,8,batch,PASS,FAIL,11.9279,13.4585,83.8,74.3,499.354,481.604,6.84729e+09,6.56345e+09,NA,1.13
|
||||
arch-a,resnet34,latency,1,element,PASS,FAIL,29.7189,66.7114,NA,NA,411.804,277.485,1.22384e+10,1.85114e+10,NA,2.24
|
||||
arch-a,resnet34,throughput,2,batch,PASS,FAIL,21.8196,23.5817,45.8,42.4,477.108,515.68,1.57899e+10,1.22836e+10,NA,1.08
|
||||
arch-a,resnet34,throughput,4,batch,PASS,FAIL,20,23.5817,50,42.4,548.372,515.68,1.09674e+10,1.22836e+10,NA,1.18
|
||||
arch-a,resnet34,throughput,8,batch,PASS,FAIL,20.6363,23.5817,48.5,42.4,504.854,515.68,1.23725e+10,1.22836e+10,NA,1.14
|
||||
arch-a,googlenet,latency,1,element,PASS,FAIL,9.77731,62.9248,NA,NA,563.88,231.193,5.51323e+09,1.45477e+10,NA,6.44
|
||||
arch-a,googlenet,throughput,2,batch,PASS,FAIL,7.34567,18.2643,136,54.8,679.351,435.107,5.5053e+09,8.16479e+09,NA,2.48
|
||||
arch-a,googlenet,throughput,4,batch,PASS,FAIL,7.43728,18.2643,134,54.8,674.025,435.107,5.60809e+09,8.16479e+09,NA,2.45
|
||||
arch-a,googlenet,throughput,8,batch,PASS,FAIL,9.01886,18.2643,111,54.8,595.582,435.107,5.71605e+09,8.16479e+09,NA,2.03
|
||||
arch-a,yolo11n,latency,1,element,PASS,FAIL,149.46,NA,NA,NA,288.579,NA,4.31311e+10,NA,NA,NA
|
||||
arch-a,yolo11n,throughput,2,batch,PASS,FAIL,94.946,NA,10.5,NA,346.466,NA,3.75539e+10,NA,NA,NA
|
||||
arch-a,yolo11n,throughput,4,batch,PASS,FAIL,80.1193,NA,12.5,NA,377.408,NA,3.8811e+10,NA,NA,NA
|
||||
arch-a,yolo11n,throughput,8,batch,PASS,FAIL,99.5488,NA,10,NA,339.331,NA,4.27113e+10,NA,NA,NA
|
||||
arch-b,vgg8,latency,1,element,PASS,PASS,1.5333,7.15212,NA,NA,292.559,173.769,4.48582e+08,1.24281e+09,raptor,4.66
|
||||
arch-b,vgg8,throughput,2,batch,PASS,PASS,0.685556,1.07282,1460,932,456.044,409.47,4.03694e+08,4.39289e+08,raptor,1.57
|
||||
arch-b,vgg8,throughput,4,batch,PASS,PASS,0.673192,1.07282,1490,932,448.898,409.47,4.13527e+08,4.39289e+08,raptor,1.60
|
||||
arch-b,vgg8,throughput,8,batch,PASS,PASS,0.751106,1.07282,1330,932,425.742,409.47,3.94458e+08,4.39289e+08,raptor,1.43
|
||||
arch-b,resnet18,latency,1,element,PASS,FAIL,15.3758,64.3818,NA,NA,400.661,201.663,6.16048e+09,1.29834e+10,raptor,4.19
|
||||
arch-b,resnet18,throughput,2,batch,PASS,FAIL,12.1214,13.434,82.5,74.4,461.407,484.804,6.29532e+09,6.56966e+09,raptor,1.11
|
||||
arch-b,resnet18,throughput,4,batch,PASS,FAIL,11.3601,13.434,88,74.4,479.333,484.804,6.98745e+09,6.56966e+09,raptor,1.18
|
||||
arch-b,resnet18,throughput,8,batch,PASS,FAIL,13.1216,13.434,76.2,74.4,441.248,484.804,6.38271e+09,6.56966e+09,raptor,1.02
|
||||
arch-b,resnet34,latency,1,element,PASS,FAIL,27.9541,107.2,NA,NA,397.206,206.476,1.11035e+10,2.21342e+10,raptor,3.83
|
||||
arch-b,resnet34,throughput,2,batch,PASS,FAIL,22.282,22.8454,44.9,43.8,438.238,493.332,1.42187e+10,1.13922e+10,raptor,1.03
|
||||
arch-b,resnet34,throughput,4,batch,PASS,FAIL,29.0906,22.8454,34.4,43.8,364.956,493.332,1.63285e+10,1.13922e+10,pimcomp,1.27
|
||||
arch-b,resnet34,throughput,8,batch,PASS,FAIL,47.68,22.8454,21,43.8,289.821,493.332,1.63464e+10,1.13922e+10,pimcomp,2.09
|
||||
arch-b,googlenet,latency,1,element,PASS,PASS,11.2212,37.9367,NA,NA,483.266,242.037,5.42282e+09,9.18206e+09,raptor,3.38
|
||||
arch-b,googlenet,throughput,2,batch,PASS,FAIL,9.21209,21.3902,109,46.8,545.488,417.866,5.5934e+09,9.01473e+09,raptor,2.33
|
||||
arch-b,googlenet,throughput,4,batch,PASS,FAIL,8.56194,21.3902,117,46.8,555.264,417.866,5.86312e+09,9.01473e+09,raptor,2.50
|
||||
arch-b,googlenet,throughput,8,batch,PASS,FAIL,10.8652,21.3902,92,46.8,483.477,417.866,6.12683e+09,9.01473e+09,raptor,1.97
|
||||
arch-b,yolo11n,latency,1,element,PASS,FAIL,168.144,NA,NA,NA,245.314,NA,4.1248e+10,NA,NA,NA
|
||||
arch-b,yolo11n,throughput,2,batch,SKIPPED,FAIL,104.494,NA,9.57,NA,302.121,NA,3.58015e+10,NA,NA,NA
|
||||
arch-b,yolo11n,throughput,4,batch,SKIPPED,FAIL,108.247,NA,9.24,NA,297.484,NA,3.76297e+10,NA,NA,NA
|
||||
arch-b,yolo11n,throughput,8,batch,SKIPPED,FAIL,100.129,NA,9.99,NA,308.833,NA,3.71808e+10,NA,NA,NA
|
||||
arch-b,vgg8,throughput,2,batch,PASS,FAIL,0.624729,1.07282,1600,932,489.899,409.47,3.93036e+08,4.39289e+08,NA,1.72
|
||||
arch-b,vgg8,throughput,4,batch,PASS,FAIL,0.62331,1.07282,1600,932,490.955,409.47,3.93418e+08,4.39289e+08,NA,1.72
|
||||
arch-b,vgg8,throughput,8,batch,PASS,FAIL,0.62331,1.07282,1600,932,490.955,409.47,3.93418e+08,4.39289e+08,NA,1.72
|
||||
arch-b,resnet18,latency,1,element,PASS,FAIL,15.3758,64.3818,NA,NA,400.661,201.663,6.16048e+09,1.29834e+10,NA,4.19
|
||||
arch-b,resnet18,throughput,2,batch,PASS,FAIL,12.9113,13.434,77.5,74.4,438.931,484.804,6.47794e+09,6.56966e+09,NA,1.04
|
||||
arch-b,resnet18,throughput,4,batch,PASS,FAIL,11.3505,13.434,88.1,74.4,478.606,484.804,6.66517e+09,6.56966e+09,NA,1.18
|
||||
arch-b,resnet18,throughput,8,batch,PASS,FAIL,11.3685,13.434,88,74.4,478.33,484.804,6.51513e+09,6.56966e+09,NA,1.18
|
||||
arch-b,resnet34,latency,1,element,PASS,FAIL,27.9541,107.2,NA,NA,397.206,206.476,1.11035e+10,2.21342e+10,NA,3.83
|
||||
arch-b,resnet34,throughput,2,batch,PASS,FAIL,22.4967,22.8454,44.5,43.8,440.06,493.332,1.49623e+10,1.13922e+10,NA,1.02
|
||||
arch-b,resnet34,throughput,4,batch,PASS,FAIL,24.1838,22.8454,41.3,43.8,426.129,493.332,1.30195e+10,1.13922e+10,NA,1.06
|
||||
arch-b,resnet34,throughput,8,batch,PASS,FAIL,24.1516,22.8454,41.4,43.8,420.745,493.332,1.39184e+10,1.13922e+10,NA,1.06
|
||||
arch-b,googlenet,latency,1,element,PASS,FAIL,11.2206,37.9367,NA,NA,483.283,242.037,5.42274e+09,9.18206e+09,NA,3.38
|
||||
arch-b,googlenet,throughput,2,batch,PASS,FAIL,9.08396,21.3902,110,46.8,550.292,417.866,5.70509e+09,9.01473e+09,NA,2.35
|
||||
arch-b,googlenet,throughput,4,batch,PASS,FAIL,8.23423,21.3902,121,46.8,597.327,417.866,5.62683e+09,9.01473e+09,NA,2.59
|
||||
arch-b,googlenet,throughput,8,batch,PASS,FAIL,7.78944,21.3902,128,46.8,620.65,417.866,5.47039e+09,9.01473e+09,NA,2.74
|
||||
arch-b,yolo11n,latency,1,element,PASS,FAIL,171.598,NA,NA,NA,243.177,NA,4.17288e+10,NA,NA,NA
|
||||
arch-b,yolo11n,throughput,2,batch,PASS,FAIL,99.5318,NA,10,NA,309.102,NA,3.5219e+10,NA,NA,NA
|
||||
arch-b,yolo11n,throughput,4,batch,PASS,FAIL,85.3497,NA,11.7,NA,326.712,NA,3.58691e+10,NA,NA,NA
|
||||
arch-b,yolo11n,throughput,8,batch,PASS,FAIL,97.9941,NA,10.2,NA,308.041,NA,3.5011e+10,NA,NA,NA
|
||||
|
||||
|
@@ -282,6 +282,23 @@ def comparison_passed(report: dict) -> bool:
|
||||
return result.get("status") == "done" and bool(result.get("passed"))
|
||||
|
||||
|
||||
def completed_synchronization_ablation(report: dict) -> bool:
|
||||
"""A completed no-sync measurement may have intentionally invalid outputs."""
|
||||
validation = report.get("raptor_validation") or {}
|
||||
performance = report.get("raptor_performance") or {}
|
||||
return (
|
||||
report.get("pimsim_mode") == "throughput"
|
||||
and "--pim-disable-synchronization" in report.get("raptor_extra_args", [])
|
||||
and validation.get("status") == "done" and validation.get("passed") is False
|
||||
and bool(validation.get("max_diffs"))
|
||||
and not report.get("failures")
|
||||
and bool(report.get("steps"))
|
||||
and all(step.get("status") == "passed" for step in report["steps"])
|
||||
and not performance.get("error")
|
||||
and performance_values(performance)["throughput"] is not None
|
||||
)
|
||||
|
||||
|
||||
def functional_validation_status(result: dict | None) -> str:
|
||||
if result is None:
|
||||
return "NA"
|
||||
@@ -869,6 +886,7 @@ def main() -> int:
|
||||
assert results_path is not None
|
||||
print_stage(results_path.name, STAGE_COLORS["Compare outputs"])
|
||||
print(results_path.read_text(encoding="utf-8"), end="")
|
||||
diagnostic_failures = []
|
||||
for arch in args.archs:
|
||||
for name in args.models:
|
||||
for mode, pipeline, _ in comparisons_by_arch[arch]:
|
||||
@@ -876,15 +894,19 @@ def main() -> int:
|
||||
report_path = result_dir(
|
||||
out_dir, name, arch, mode, pipeline, args.ablation_variant
|
||||
) / "pimcomp/comparison_report.json"
|
||||
if not report_path.exists() or not comparison_passed(
|
||||
json.loads(report_path.read_text(encoding="utf-8")),
|
||||
):
|
||||
if label not in failed:
|
||||
report = json.loads(report_path.read_text(encoding="utf-8")) if report_path.exists() else {}
|
||||
if not comparison_passed(report):
|
||||
if (label not in failed and args.ablation_variant == "no-sync"
|
||||
and completed_synchronization_ablation(report)):
|
||||
diagnostic_failures.append(label)
|
||||
elif label not in failed:
|
||||
failed.append(label)
|
||||
print(Style.BRIGHT + Fore.CYAN + "Summary" + Style.RESET_ALL)
|
||||
total_jobs = sum(len(args.models) * len(comparisons) for comparisons in comparisons_by_arch.values())
|
||||
print(Style.BRIGHT + f"Passed: {total_jobs - len(failed)}" + Style.RESET_ALL)
|
||||
print(Style.BRIGHT + f"Passed: {total_jobs - len(failed) - len(diagnostic_failures)}" + Style.RESET_ALL)
|
||||
print(Style.BRIGHT + f"Failed: {len(failed)}" + Style.RESET_ALL)
|
||||
if diagnostic_failures:
|
||||
print("No-sync numerical failures (diagnostic only): " + ", ".join(diagnostic_failures))
|
||||
print(Style.BRIGHT + f"Results: {results_path}" + Style.RESET_ALL)
|
||||
if failed:
|
||||
print(
|
||||
|
||||
@@ -35,13 +35,15 @@ The corresponding scheduling call is disabled in
|
||||
Rust validation starts those buffers at zero, exposing the missing computation.
|
||||
|
||||
```bash
|
||||
.venv/bin/python validation/tools/pim/pimcomp/correctness/prefill/run_experiment.py
|
||||
.venv/bin/python validation/tools/pim/pimcomp/correctness/prefill/run_experiment.py \
|
||||
/path/to/comparison /path/to/matching-model.onnx
|
||||
```
|
||||
|
||||
The default uses GoogLeNet Arch-A throughput/pipeline2. Optional positional
|
||||
arguments are comparison directory, model, work directory, and shared reference
|
||||
directory; the default work directory is `correctness/prefill/` below the
|
||||
comparison artifacts.
|
||||
The comparison directory and its matching model are required. Optional positional
|
||||
arguments are work directory and shared reference directory. The default work
|
||||
directory is `correctness/prefill/` below the comparison artifacts.
|
||||
Historical Softmax-free models can be recovered from Git for historical artifacts.
|
||||
Current GoogLeNet retains Softmax; prefill alone does not correct omitted Softmax execution.
|
||||
|
||||
The experiment compares the original memory image with one prefilled using
|
||||
[`batch_memory.py`](prefill/batch_memory.py). In the recorded case, 20 of 39
|
||||
|
||||
@@ -16,8 +16,6 @@ from batch_memory import prefill_batch_memory
|
||||
|
||||
SCRIPT_DIR = Path(__file__).resolve().parent
|
||||
REPO_ROOT = SCRIPT_DIR.parents[5]
|
||||
DEFAULT_COMPARISON_DIR = REPO_ROOT / "validation/networks/pimcomp_models/googlenet/artifacts/arch-a/throughput/pipeline2"
|
||||
DEFAULT_MODEL = REPO_ROOT / "validation/networks/pimcomp_models/googlenet/googlenet-12-pimsim-nn.onnx"
|
||||
DEFAULT_COMMON_DIR = REPO_ROOT / "validation/networks/pimcomp_models/googlenet/artifacts/common"
|
||||
SIMULATOR_MANIFEST = REPO_ROOT / "backend-simulators/pim/pim-simulator/Cargo.toml"
|
||||
|
||||
@@ -78,8 +76,8 @@ def compare_outputs(
|
||||
|
||||
def main() -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("comparison_dir", nargs="?", type=Path, default=DEFAULT_COMPARISON_DIR)
|
||||
parser.add_argument("model", nargs="?", type=Path, default=DEFAULT_MODEL)
|
||||
parser.add_argument("comparison_dir", type=Path)
|
||||
parser.add_argument("model", type=Path, help="ONNX model used to generate the comparison artifacts")
|
||||
parser.add_argument("work_dir", nargs="?", type=Path)
|
||||
parser.add_argument("common_dir", nargs="?", type=Path, default=DEFAULT_COMMON_DIR)
|
||||
args = parser.parse_args()
|
||||
|
||||
Reference in New Issue
Block a user