better throughput scheduling with steady state cost model (WIP)
This commit is contained in:
@@ -11,11 +11,13 @@ Boolean switches default to off unless stated otherwise.
|
||||
| `--maccel=PIM` | Select the PIM accelerator. |
|
||||
| `--EmitSpatial`, `--EmitPim`, `--EmitPimBufferized`, `--EmitPimCodegen` | Stop at a pipeline stage; default: codegen. |
|
||||
| `--core-count=N`, `--crossbar-size=N`, `--crossbar-count=N` | Required hardware resources; crossbars are square. |
|
||||
| `--pipeline=N` | Throughput stages; default: 1 (latency scheduling). |
|
||||
| `--pipeline=N` | Static queue-aware throughput selection for final codegen with N > 1; intermediate targets retain the original schedule. Default: 1 (latency PEFT). |
|
||||
| `--pim-target-config=PATH` | Target cost/topology configuration; resources must match hardware flags. Default: built-in model. |
|
||||
| `--pim-memory-report=summary\|none` | Memory report at `reports/memory_report.txt`; default: `summary`. |
|
||||
| `--pim-only-codegen` | Run the codegen tail on bufferized Pim IR. |
|
||||
| `--pim-emit-json` | Also emit per-core JSON instructions. |
|
||||
| `--pim-dump-ir` | Write diagnostic IR snapshots with elided large constants; default: off. |
|
||||
| `--pim-periodic-diagnostics` | Write the optional pre-codegen recurrence report; default: off. |
|
||||
| `--pim-export-spatial-dataflow=none\|spatial1\|spatial2\|spatial3\|spatial4\|all` | Export graph, trivially merged, scheduled, and realized CSV snapshots under `reports/`; default: `none`. |
|
||||
| `--pim-conv-lowering=STRATEGY` | `auto` (default), `legacy`, `depthwise`, `packed-im2col`, `streamed-patch`, `streamed-packed`, `output-channel-tiled`, `input-k-tiled`, or `tiled-2d`. |
|
||||
| `--pim-conv-im2col-max-elements=N` | Global im2col limit before streaming; default: 1048576. |
|
||||
|
||||
+288
-15
@@ -1,7 +1,8 @@
|
||||
# Spatial scheduling
|
||||
|
||||
`ScheduleAndRealizeSpatial` constructs a shared `ComputeGraph`, then selects
|
||||
latency PEFT or pipeline scheduling. Pipeline mode does not run PEFT.
|
||||
`prepareSpatialSchedules` constructs the shared `ComputeGraph`. Latency mode
|
||||
uses PEFT. For `--pipeline > 1`, the PIM compiler selects among bounded pipeline
|
||||
candidates with static queue-aware instruction analysis. Pipeline mode does not run PEFT.
|
||||
|
||||
See [implementation notes](scheduling-implementation.md) for fragment-view eligibility,
|
||||
receive forwarding, transfer rounding, cost-model boundaries, and recovery limits.
|
||||
@@ -57,19 +58,14 @@ paths exclude core-order constraints; completion includes them. Stage spans
|
||||
exclude cross-stage routing, restart/host synchronization, and contention.
|
||||
Simulator latency and measured throughput remain the performance gates.
|
||||
|
||||
Pipeline scheduling retains the original ordered-packing/dependency-tail
|
||||
heuristic using nominal movement volumes. A second search uses execution
|
||||
estimates and aggregate serialized shared-memory service, minimizing their
|
||||
maximum. Its DP keeps at most eight compute/memory trade-offs per state;
|
||||
four final alternatives and the initial partition undergo the existing core
|
||||
allocation, topology placement, and capacity verification. The phase-1 winner
|
||||
remains a fallback. A second bounded search accounts for shared publications.
|
||||
Selection retains the fallback unless the modeled bottleneck improves. Selection also requires that
|
||||
shared-memory service plus maximum stage span does not increase. The same
|
||||
check applies to shared-memory service plus maximum per-core compute work.
|
||||
These guards limit compute imbalance; their sums do not predict simulator
|
||||
periods. Sharing-aware alternatives must also preserve maximum stage span,
|
||||
because placement-dependent baseline loads remain approximate. PEFT is unchanged.
|
||||
Pipeline candidate generation retains the original ordered-packing/dependency-tail
|
||||
heuristic and the memory-aware and sharing-aware searches. The partition DP
|
||||
keeps at most eight compute/memory trade-offs per state. Four final alternatives
|
||||
and the initial partition undergo core allocation, topology placement, and
|
||||
capacity verification. The previous compute/memory guards identify candidate
|
||||
zero, the original fallback. They do not reject legal alternatives before queue
|
||||
analysis. All candidates preserve candidate zero's non-empty pipeline stages.
|
||||
PEFT is unchanged.
|
||||
|
||||
`ComputeGraph` retains physical publication and consumer identities separately
|
||||
from aggregated dependency edges. Cross-stage demands charge one store per
|
||||
@@ -92,6 +88,283 @@ Placement ranks maximum per-core compute/incident-transfer demand, then stage
|
||||
completion and compute load; within 1% demand, stage span breaks the tie.
|
||||
These are placement heuristics, not a steady-state timing model.
|
||||
|
||||
## Periodic dependency diagnostics
|
||||
|
||||
`--pim-periodic-diagnostics` writes `reports/pipeline_periodic.json` after deferred
|
||||
boundary planning. This diagnostic does not change schedule selection or IR.
|
||||
It consumes the cached compute-instance costs, materialized core order, physical
|
||||
transfer families, publication store flags, and allocated READY/FREE registers.
|
||||
Compute runs remain atomic; their lanes and loop iterations are not expanded.
|
||||
Compact runs require one physical core; multi-core classes use per-lane steps.
|
||||
One event sequence represents each physical core. The model retains one planned
|
||||
load per physical consumer transfer and one store per shared publication.
|
||||
Later load folding or strided transfer splitting can change request granularity.
|
||||
|
||||
Edges express `t(v,k) >= t(u,k-d) + w`. Within-generation edges have distance
|
||||
zero. Each core's completion-to-entry edge has distance one. Reader-entry FREE
|
||||
signals therefore release the preceding generation through that edge. Signals
|
||||
join their exact planned WAIT count. Dissemination barriers, restart permissions,
|
||||
and the downstream release tree retain their core identities and order.
|
||||
The diagnostic checks its event-register summary against the emitted program
|
||||
in the synchronization tests; it does not replace deadlock verification.
|
||||
|
||||
The injected target describes direct-transfer completion and link, local-memory,
|
||||
shared-memory, and signal timing. The current adapter specifies peer permission
|
||||
before a source read. SEND completion follows the read and payload enqueue;
|
||||
RECV completion also requires payload delivery and a destination write.
|
||||
Unsupported or incomplete target timing produces an `unavailable` report.
|
||||
|
||||
The analysis brackets the maximum cycle ratio within one nanosecond. It tests
|
||||
edge delays `w - II*d`, sweeping the distance-zero DAG between recurrence
|
||||
relaxations. At most `R+1` sweeps detect a positive cycle, where `R` counts
|
||||
positive-distance edges. Bisection uses at most 64 tests. Timing outside the
|
||||
supported numeric range produces an explicit unavailable result. A predecessor
|
||||
cycle supplies a witness, including delay, iteration distance, cores, and links.
|
||||
Detailed witness output is limited to 128 edges; totals cover the entire cycle.
|
||||
|
||||
The reported interval estimate is the maximum of the modeled recurrence,
|
||||
per-core compute work, and aggregate serialized shared-memory service. Memory
|
||||
service is never assigned a fixed global request order. Taking the maximum
|
||||
does not add the same service twice. The within-generation dependency span is
|
||||
reported separately and is not an initiation interval.
|
||||
|
||||
These are conditional estimates for the abstract execution model, not exact
|
||||
simulator predictions or guaranteed physical lower bounds. Serial compute costs
|
||||
can overestimate overlapping work. Baseline memory accesses retain their
|
||||
aggregate service bound, but their positions are not modeled. Communication
|
||||
introduced after deferred realization, including final output collection, is
|
||||
also absent. Instruction issue, local contention, network injection queues, and
|
||||
memory arbitration remain unmodeled. Missing effects are listed in the report;
|
||||
`selection_eligible` remains false. This plan-level diagnostic does not rank
|
||||
production candidates. Production uses the instruction-level queue model below.
|
||||
|
||||
For `V` events, `E` edges, and `R` recurrence edges, construction requires
|
||||
`O(T + F + S log S)` time and `O(V + E + S)` storage, for existing compute
|
||||
instances `T`, physical transfers `F`, and synchronization events `S`.
|
||||
Analysis requires `O(L (R+1) (V+E))` time and `O(V+E)` storage, where `L <= 64`.
|
||||
Here `R` is at most the number of active physical cores. There is no dense
|
||||
all-core dependency matrix or inference-generation unrolling.
|
||||
For `C` analyzed candidates, time scales by `C`; scratch storage can be reused.
|
||||
The current diagnostic analyzes only the selected schedule (`C = 1`).
|
||||
|
||||
### Post-codegen instruction diagnostic
|
||||
|
||||
`raptor-periodic` provides an explicit, read-only diagnostic over an emitted
|
||||
binary artifact. It accepts native or oracle-exported artifacts. Use the
|
||||
oracle-exported artifact when comparing against Pimsim performance results:
|
||||
|
||||
```bash
|
||||
build_release/Release/bin/raptor-periodic /tmp/model/pimsim_nn \
|
||||
--pim-target-config=validation/pimsim_configs/pimcomp/arch-a/throughput_config_1000ms.json \
|
||||
--core-count=168 --crossbar-count=96 --crossbar-size=128 > /tmp/periodic.json
|
||||
```
|
||||
|
||||
The diagnostic uses physical instruction files and their array-group map.
|
||||
Each file owns one physical core's work, including all lanes and static loop
|
||||
iterations already emitted there. It does not expand compiler loops or batches.
|
||||
It replaces Spatial compute costs rather than adding instruction costs to them.
|
||||
The lightweight compiler report remains separate and does not invoke this tool.
|
||||
|
||||
The supported execution contract requires ROB capacity one. Non-scalar
|
||||
instructions therefore retain their serial execution order. Local compute and
|
||||
blocking memory runs become weighted chunks between communication events.
|
||||
Every emitted global request contributes its actual size, rounding, and count.
|
||||
Inputs, constants, split transfers, folded loads, and output collection are
|
||||
included through their emitted instructions. Their positions relative to
|
||||
communication remain explicit. The global service bound remains separate;
|
||||
the model does not impose an order on different cores' global requests.
|
||||
|
||||
Timing queries reuse the injected memory, link, and matrix definitions.
|
||||
PIM instruction semantics belong to the compiler target adapter. Vector
|
||||
writeback follows the oracle's actual width rules. Matrix writeback uses the
|
||||
physical array-group width. Local-memory timing uses its own configured period.
|
||||
The oracle passes `vector_latency_cycle` directly to a nanosecond wait.
|
||||
This parameter does not scale with the processor period; no separate vector
|
||||
clock exists in the supported oracle configuration.
|
||||
In the current oracle, global loads return zero network payload bytes.
|
||||
Global stores retain their outgoing payload cost.
|
||||
|
||||
The source switch has one unbounded FIFO for outgoing requests. Enqueue does
|
||||
not wait for earlier requests. FIFO dispatch waits for their completion.
|
||||
Direct SEND requires permission delivery before the source read. Its payload
|
||||
enqueue event coincides with source-read completion. SEND completes after
|
||||
enqueue, independently of FIFO dispatch or payload delivery. RECV requires delivery and
|
||||
destination write. Permission responses join both peer readiness and the
|
||||
receiver's outgoing FIFO. SYNC delivery can precede its local completion.
|
||||
The model retains WAIT counts, restart events, and cross-generation FIFO state.
|
||||
It does not replace the compiler's synchronization or deadlock verification.
|
||||
|
||||
`--model-network-queues=false` and `--model-issue=false` support timing
|
||||
comparisons. The latter omits scalar issue and retirement delays while retaining
|
||||
SYNC's instruction-defined execution time. Completion notifications can join
|
||||
the current edge before the clock process runs. The model uses minimum
|
||||
retirement delays; it does not reproduce SystemC clock-phase ordering.
|
||||
Scalar issue is charged serially, so these estimates are not guaranteed
|
||||
physical lower bounds. The base recurrence estimate omits global arbitration,
|
||||
frontend buffering, and initial bootstrap. Production does not rank candidates
|
||||
with this recurrence-only estimate.
|
||||
Reports with at most 128 events include `timing_graph` for focused timing checks.
|
||||
Its dependencies distinguish payload enqueue from FIFO dispatch and delivery.
|
||||
The instruction test accepts `--pimsim /path/to/ChipTest` for optional fast-mode
|
||||
timing comparisons. These cover both hardware profiles and distinct processor,
|
||||
local-memory, and vector latency settings.
|
||||
|
||||
For `N` emitted instructions, `F` direct transfers, `S` synchronization events,
|
||||
and `P` active cores, construction takes `O(N + (F+S) log(F+S))` time.
|
||||
The indexed graph uses `O(F+S+P)` space and at most `2P` recurrence edges.
|
||||
Global requests are counted during the same scan without per-request graph
|
||||
nodes. One instruction file is memory-mapped at a time. Maximum-cycle-ratio
|
||||
analysis uses the same bounded algorithm as the lightweight diagnostic.
|
||||
|
||||
`--model-memory-controller` adds a separate `memory_controller` estimate.
|
||||
Each global request retains its readiness, transport, service, and completion
|
||||
dependencies. The controller serves requests in predicted arrival order.
|
||||
Read and write requests share that FIFO. Zero-size requests also join it.
|
||||
Source switches remain occupied until transport completion. The model does not
|
||||
assign a fixed global request order or consume simulator trace order.
|
||||
|
||||
The analysis executes the indexed periodic graph with timestamped tokens.
|
||||
FIFO tokens retain dependencies across generations; the graph is not copied.
|
||||
A request arrival reserves the controller after earlier arrivals, and its
|
||||
completion enables dependent events. Independent cores and switches remain
|
||||
concurrent. The maximum-cycle-ratio solver and its report remain unchanged.
|
||||
|
||||
The queue estimate compares adjacent two-generation windows with the oracle's
|
||||
one-percent convergence threshold. It reports the mean interval across both
|
||||
windows. Per-core queue waits use the corresponding four completed body
|
||||
generations. The maximum request wait includes bootstrap. Event-index order
|
||||
resolves simultaneous readiness. A second run reverses that order and reports
|
||||
`alternative_tie_interval_ns`. These two orders are sensitivity checks, not
|
||||
bounds over every possible tie order. Clock phases and frontend overlap remain
|
||||
approximate. The standalone report remains read-only. Production selection uses the same queue model through its C++ API.
|
||||
|
||||
For `R` global requests, the queue mode adds `O(R)` events and dependencies.
|
||||
For `V` resulting events, `E` dependencies, and generation bound `K`, execution
|
||||
takes `O(K(E + V log V))` time. Storage is `O(V + E + B)`, where `B` counts
|
||||
pending tokens. The event heap contains at most one event per node. FIFO token
|
||||
storage retains pending generations, rather than instruction-stream copies.
|
||||
The diagnostic limits each event to 64 generations and processed events to
|
||||
`16(V + E)`. It reports `unavailable` if either bound prevents convergence.
|
||||
The two tie-order runs reuse the same graph. Production uses this queue execution directly and omits maximum-cycle-ratio analysis.
|
||||
The standalone frontend retains the recurrence witness. Neither analysis changes latency PEFT.
|
||||
|
||||
## Default throughput selection
|
||||
|
||||
Full PIM code generation performs these steps for `--pipeline > 1`.
|
||||
`--pim-only-codegen` retains the physical schedule already present in its input.
|
||||
|
||||
The normal compilation workflow is:
|
||||
|
||||
1. Run the ONNX frontend, layout planning, common lowering, and graph preparation once.
|
||||
2. Generate at most eleven verified physical schedules from the existing searches.
|
||||
3. Remove duplicate placements, including their core order and stage identities.
|
||||
4. Clone the prepared module for each candidate using MLIR's lossless `IRMapping`.
|
||||
5. Remap each schedule's compute identities to the clone.
|
||||
6. Realize communication and lower the clone through verification and instruction selection.
|
||||
7. Generate binary instruction records in memory, without weights, exports, or candidate files.
|
||||
8. Run both static controller-arbitration tie policies on that instruction program.
|
||||
9. Select an eligible candidate and emit only its normal program.
|
||||
|
||||
Immutable attributes, including constants and weights, remain shared in the MLIR
|
||||
context. Candidate operations, values, communication plans, and synchronization
|
||||
state remain independent. Trials never mutate the prepared checkpoint. Candidates
|
||||
run sequentially. At most one best lowered module survives between trials.
|
||||
The retained module follows the conservative selection rule throughout evaluation.
|
||||
A close alternative does not evict candidate zero. Unsupported timing contracts
|
||||
and synchronization ablation generate only the original schedule and lower it
|
||||
once. The compiler checks this before collecting alternatives. The frontend is
|
||||
not repeated.
|
||||
|
||||
Intermediate targets (`--EmitSpatial`, `--EmitPim`, and `--EmitPimBufferized`)
|
||||
retain the original schedule. They do not perform queue-aware selection or run
|
||||
transformations beyond the requested stage. Their schedule can therefore differ
|
||||
from final code generation. Full PIM code generation uses queue-aware selection.
|
||||
|
||||
The PIM compiler owns instruction timing and controller behavior. Spatial owns
|
||||
legal candidate generation and schedule identity remapping. `PimPeriodicAnalysis`
|
||||
is the single instruction-level timing implementation. `raptor-periodic` is its
|
||||
read-only diagnostic frontend. The compiler invokes neither that executable nor
|
||||
an external selection script.
|
||||
|
||||
`SchedulingTarget::prepareScheduleCost` is an optional whole-schedule cost query.
|
||||
It accepts a verified target-lowered module and returns an opaque program identity
|
||||
and a deferred static evaluation. Spatial compares the resulting interval
|
||||
estimates and applies the target's reported uncertainty. Missing uncertainty
|
||||
makes an estimate ineligible. The target owns instruction extraction,
|
||||
protocol timing, contention, and unavailable-result reasons. Preparation and
|
||||
evaluation do not change the candidate. Equal program identities avoid repeated
|
||||
queue analysis. An absent query retains the original schedule.
|
||||
If generation yields one distinct schedule, no comparison is required. The
|
||||
compiler realizes and lowers it once, without a trial clone, duplicate
|
||||
instruction generation, or queue analysis. The selection report records this
|
||||
case separately from unavailable estimates. Final code generation is unchanged.
|
||||
|
||||
PIM transport details reside in `PimInstructionTiming`. The optional plan-level
|
||||
diagnostic also belongs to the PIM compiler. Spatial supplies its physical plans
|
||||
through an injected reporting callback; it does not encode the PIM handshake.
|
||||
The instruction emitter remains the source of register setup, address-dependent
|
||||
instruction suppression, loop expansion, and physical array-group identity.
|
||||
Candidate evaluation uses this emitter in memory after local-memory planning.
|
||||
It does not reconstruct these rules from scalar Spatial operation costs.
|
||||
|
||||
Production projects native fp32 transfer sizes and vector widths to the existing
|
||||
oracle int8 timing contract. It preserves instruction counts, matrix precision,
|
||||
vector element counts, synchronization, and transfer identities. This projection
|
||||
does not change the emitted native program. Unsupported widths, targets without
|
||||
complete timing, and unsupported instruction semantics produce explicit fallback
|
||||
reasons. Synchronization ablation retains the original schedule.
|
||||
|
||||
Both arbitration tie runs must converge within the existing generation and event
|
||||
budgets. Their estimates are sensitivity checks, not formal bounds. Duplicate
|
||||
instruction programs are analyzed once. The candidate with the lowest larger
|
||||
estimate must separate from the original schedule. Close alternatives do not
|
||||
cancel a gain that separates from the original. Comparison uses `[min(ties)/1.021, max(ties)/0.979]`. The fixed 2.1% prediction uncertainty
|
||||
covers offline validation error with upward rounding. It is a selection margin,
|
||||
not a fitted instruction latency or a guaranteed error bound. Overlapping ranges
|
||||
with the original retain candidate zero. Missing original estimates also retain candidate zero.
|
||||
Unavailable alternatives cannot displace an eligible original. This search finds
|
||||
the best distinguishable candidate in its bounded set; it does not establish
|
||||
global optimality.
|
||||
|
||||
`reports/pipeline_selection.json` records candidate identities, estimates,
|
||||
selection reasons, and generation, lowering, instruction-generation, queue-analysis,
|
||||
and finalization times. The selected verified lowered module and its captured
|
||||
schedule report are reused without repeating materialization or communication
|
||||
realization. Memory reports come from normal final code generation.
|
||||
With `--enable-timing`, candidate summaries also contain per-pass lowering times
|
||||
and operation/value counts before bufferization and instruction generation.
|
||||
Instruction-generation summaries separate preparation from per-core emission.
|
||||
Explicit Spatial exports, IR snapshots,
|
||||
and plan-level periodic diagnostics reconstruct the selected Spatial schedule
|
||||
because those outputs require pre-lowering operations. This optional work does
|
||||
not change the retained instruction program. Candidate instructions and rejected
|
||||
IR are released immediately. No temporary compilation directories are created.
|
||||
`--pim-dump-ir` explicitly enables diagnostic IR snapshots;
|
||||
these snapshots elide large constants and cannot serve as lossless checkpoints.
|
||||
|
||||
### Static analysis, diagnostics, and external validation
|
||||
|
||||
Production decisions use only the input program, compiler options, candidate
|
||||
instructions and plans, and configured target resources and timing. The static
|
||||
model executes deterministic abstract events. It does not execute the generated
|
||||
program. It reads no benchmark CSV, profiling trace, measured latency cache, or
|
||||
reference artifact. Identical static inputs produce the same decision regardless
|
||||
of historical measurements. Throughput deferred-demand ordering uses IR order,
|
||||
rather than allocation addresses. Latency PEFT and its demand ordering are unchanged.
|
||||
|
||||
Optional diagnostics expose recurrence witnesses, instruction timing, and resource
|
||||
service. Offline Pimsim fast-mode measurements validate these predictions and
|
||||
candidate rankings. Simulator results never enter production selection. Timing
|
||||
corrections must follow verified hardware behavior or configured parameters.
|
||||
|
||||
For `B <= 11` schedules, `T` compute instances, and `P` cores, captured schedules
|
||||
use `O(B(T+P))` storage. Placement deduplication takes `O(B²T)` time. Each distinct
|
||||
finalist requires downstream lowering and one instruction scan. Queue execution
|
||||
uses the bounded complexity stated above. Peak storage includes the prepared
|
||||
module, one best lowered module, one trial, and one queue graph. It does not scale
|
||||
with retained complete candidate programs. Maximum-cycle-ratio diagnostics do not
|
||||
run in the candidate evaluation loop.
|
||||
|
||||
## Capacity and recovery
|
||||
|
||||
Distinct-weight union / per-core capacity is a lower bound; an oversized task
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
#include "llvm/Support/raw_os_ostream.h"
|
||||
#include "llvm/Support/CommandLine.h"
|
||||
|
||||
#include <fstream>
|
||||
|
||||
@@ -6,6 +7,12 @@
|
||||
#include "src/Accelerators/PIM/Common/Support/FileSystemUtils.hpp"
|
||||
|
||||
namespace onnx_mlir {
|
||||
namespace {
|
||||
llvm::cl::opt<bool> dumpIr("pim-dump-ir", llvm::cl::init(false),
|
||||
llvm::cl::desc("Write PIM compiler IR snapshots (large constants are elided)"));
|
||||
}
|
||||
|
||||
bool isPimIrDumpEnabled() { return dumpIr; }
|
||||
|
||||
std::fstream openDialectDumpFileWithExtension(const std::string& name, llvm::StringRef destination, llvm::StringRef extension) {
|
||||
std::string outputDir = getOutputDir();
|
||||
@@ -18,6 +25,7 @@ std::fstream openDialectDumpFileWithExtension(const std::string& name, llvm::Str
|
||||
}
|
||||
|
||||
void dumpModule(mlir::ModuleOp moduleOp, const std::string& name, bool assumeVerified) {
|
||||
if (!dumpIr) return;
|
||||
std::fstream file = openDialectDumpFileWithExtension(name, "/dialects", "mlir");
|
||||
if (!file.is_open())
|
||||
return;
|
||||
|
||||
@@ -11,6 +11,7 @@ namespace onnx_mlir {
|
||||
/// Emits a MLIR snapshot under the current compiler output
|
||||
/// directory for pass-level debugging.
|
||||
void dumpModule(mlir::ModuleOp moduleOp, const std::string& name, bool assumeVerified = false);
|
||||
bool isPimIrDumpEnabled();
|
||||
|
||||
/// Opens a file under the same dialect dump directory used by dumpModule.
|
||||
std::fstream openDialectDumpFileWithExtension(const std::string& name,llvm::StringRef destination = "/dialects", llvm::StringRef extension = "mlir");
|
||||
|
||||
@@ -5,7 +5,14 @@
|
||||
|
||||
namespace onnx_mlir {
|
||||
|
||||
namespace { thread_local bool suppressOutput = false; }
|
||||
ScopedCompilerOutputSuppression::ScopedCompilerOutputSuppression() : previous(suppressOutput) {
|
||||
suppressOutput = true;
|
||||
}
|
||||
ScopedCompilerOutputSuppression::~ScopedCompilerOutputSuppression() { suppressOutput = previous; }
|
||||
|
||||
std::string getOutputDir() {
|
||||
if (suppressOutput) return {};
|
||||
if (outputBaseName.empty() || outputBaseName == "-")
|
||||
return {};
|
||||
|
||||
|
||||
@@ -8,6 +8,15 @@ namespace onnx_mlir {
|
||||
/// current compiler invocation.
|
||||
std::string getOutputDir();
|
||||
|
||||
// Candidate lowering must not emit reports or temporary artifacts.
|
||||
class ScopedCompilerOutputSuppression {
|
||||
public:
|
||||
ScopedCompilerOutputSuppression();
|
||||
~ScopedCompilerOutputSuppression();
|
||||
private:
|
||||
bool previous;
|
||||
};
|
||||
|
||||
void createDirectory(const std::string& directory);
|
||||
|
||||
} // namespace onnx_mlir
|
||||
|
||||
@@ -15,6 +15,9 @@ add_pim_library(OMPimCompilerOptions
|
||||
|
||||
add_pim_library(OMPimCompilerUtils
|
||||
PimCompilerUtils.cpp
|
||||
PimScheduleDiagnostic.cpp
|
||||
PimPeriodicAnalysis.cpp
|
||||
PimPipelineSelection.cpp
|
||||
PimArtifactWriter.cpp
|
||||
PimCodeGen.cpp
|
||||
PimCoreProgram.cpp
|
||||
@@ -42,3 +45,8 @@ add_pim_library(OMPimCompilerUtils
|
||||
ACCEL_INCLUDE_DIRS PRIVATE
|
||||
${PIM_ACCEL_INCLUDE_DIRS}
|
||||
)
|
||||
|
||||
add_onnx_mlir_executable(raptor-periodic NO_INSTALL
|
||||
PimPeriodicDiagnostic.cpp
|
||||
LINK_LIBS PRIVATE OMPimCompilerUtils
|
||||
)
|
||||
|
||||
@@ -22,6 +22,7 @@
|
||||
#include <absl/types/compare.h>
|
||||
#include <algorithm>
|
||||
#include <cassert>
|
||||
#include <chrono>
|
||||
#include <cstdint>
|
||||
#include <fstream>
|
||||
#include <limits>
|
||||
@@ -41,6 +42,7 @@
|
||||
#include "src/Accelerators/PIM/Common/Support/FileSystemUtils.hpp"
|
||||
#include "src/Accelerators/PIM/Compiler/PimArtifactWriter.hpp"
|
||||
#include "src/Accelerators/PIM/Compiler/PimBinaryFormat.hpp"
|
||||
#include "src/Accelerators/PIM/Compiler/PimPeriodicAnalysis.hpp"
|
||||
#include "src/Accelerators/PIM/Compiler/PimCodeGen.hpp"
|
||||
#include "src/Accelerators/PIM/Compiler/PimCompilerOptions.hpp"
|
||||
#include "src/Accelerators/PIM/Compiler/PimCoreProgram.hpp"
|
||||
@@ -893,6 +895,7 @@ struct CoreEmissionResult {
|
||||
|
||||
OnnxMlirCompilerErrorCodes status = CompilerSuccess;
|
||||
MemoryReportRow reportRow;
|
||||
std::string instructions;
|
||||
llvm::SmallVector<ResolvedWeightView, 8> usedWeights;
|
||||
llvm::SmallVector<DiagnosticRecord, kMaxStoredCodegenDiagnostics> diagnostics;
|
||||
size_t diagnosticCount = 0;
|
||||
@@ -1147,8 +1150,10 @@ static OnnxMlirCompilerErrorCodes emitEmptyCoreArtifacts(StringRef outputDirPath
|
||||
return CompilerSuccess;
|
||||
}
|
||||
|
||||
OnnxMlirCompilerErrorCodes onnx_mlir::compileToPimCode(ModuleOp& moduleOp, std::string& outputDirPath) {
|
||||
if (!outputDirPath.empty()) {
|
||||
OnnxMlirCompilerErrorCodes onnx_mlir::compileToPimCode(ModuleOp& moduleOp, std::string& outputDirPath,
|
||||
PimInstructionProgram *instructions, PimCodegenTiming *timing) {
|
||||
auto start = std::chrono::steady_clock::now();
|
||||
if (!instructions && !outputDirPath.empty()) {
|
||||
if (auto error = sys::fs::create_directory(outputDirPath)) {
|
||||
errs() << "Error creating output directory: " << outputDirPath << ": " << error.message() << '\n';
|
||||
return InvalidOutputFileAccess;
|
||||
@@ -1164,8 +1169,9 @@ OnnxMlirCompilerErrorCodes onnx_mlir::compileToPimCode(ModuleOp& moduleOp, std::
|
||||
memory.hostMem.allocateHost(moduleOp, funcOp);
|
||||
memory.reportHost();
|
||||
|
||||
if (auto err = writeMemoryBinary(moduleOp, funcOp, memory, outputDirPath))
|
||||
return err;
|
||||
if (!instructions)
|
||||
if (auto err = writeMemoryBinary(moduleOp, funcOp, memory, outputDirPath))
|
||||
return err;
|
||||
|
||||
json::Object xbarsPerArrayGroup;
|
||||
size_t maxCoreId = 0;
|
||||
@@ -1295,15 +1301,22 @@ OnnxMlirCompilerErrorCodes onnx_mlir::compileToPimCode(ModuleOp& moduleOp, std::
|
||||
|
||||
std::error_code errorCode;
|
||||
auto outputCorePath = outputDirPath + "/core_" + std::to_string(job.physicalCoreId) + ".pim";
|
||||
raw_fd_ostream coreBinaryStream(outputCorePath, errorCode, sys::fs::OF_None);
|
||||
if (errorCode) {
|
||||
errs() << "Error while opening core file `" << outputCorePath << "`: " << errorCode.message() << '\n';
|
||||
result.status = InvalidOutputFileAccess;
|
||||
return result;
|
||||
llvm::SmallVector<char, 0> binary;
|
||||
raw_svector_ostream memoryStream(binary);
|
||||
std::unique_ptr<raw_fd_ostream> fileStream;
|
||||
if (!instructions) {
|
||||
fileStream = std::make_unique<raw_fd_ostream>(outputCorePath, errorCode, sys::fs::OF_None);
|
||||
if (errorCode) {
|
||||
errs() << "Error opening core file: " << outputCorePath << ": " << errorCode.message() << '\n';
|
||||
result.status = InvalidOutputFileAccess;
|
||||
return result;
|
||||
}
|
||||
}
|
||||
raw_pwrite_stream &coreBinaryStream = instructions
|
||||
? static_cast<raw_pwrite_stream &>(memoryStream) : *fileStream;
|
||||
|
||||
std::unique_ptr<raw_fd_ostream> coreJsonStream;
|
||||
if (pimEmitJson.getValue()) {
|
||||
if (!instructions && pimEmitJson.getValue()) {
|
||||
std::string outputCoreJsonPath = outputDirPath + "/core_" + std::to_string(job.physicalCoreId) + ".json";
|
||||
errorCode = std::error_code();
|
||||
coreJsonStream = std::make_unique<raw_fd_ostream>(outputCoreJsonPath, errorCode);
|
||||
@@ -1319,10 +1332,11 @@ OnnxMlirCompilerErrorCodes onnx_mlir::compileToPimCode(ModuleOp& moduleOp, std::
|
||||
PimCodeGen coreCodeGen(jobMemory, instructionWriter, coreJsonStream.get());
|
||||
auto finalizeInstructions = [&]() {
|
||||
bool succeeded = mlir::succeeded(instructionWriter.finalize());
|
||||
coreBinaryStream.close();
|
||||
if (coreBinaryStream.has_error()) {
|
||||
coreBinaryStream.clear_error();
|
||||
succeeded = false;
|
||||
if (fileStream) {
|
||||
fileStream->close();
|
||||
if (fileStream->has_error()) { fileStream->clear_error(); succeeded = false; }
|
||||
} else {
|
||||
result.instructions.assign(binary.begin(), binary.end());
|
||||
}
|
||||
return succeeded;
|
||||
};
|
||||
@@ -1376,8 +1390,14 @@ OnnxMlirCompilerErrorCodes onnx_mlir::compileToPimCode(ModuleOp& moduleOp, std::
|
||||
};
|
||||
|
||||
std::vector<CoreEmissionResult> jobResults(jobs.size());
|
||||
auto emissionStart = std::chrono::steady_clock::now();
|
||||
if (timing)
|
||||
timing->preparationSeconds = std::chrono::duration<double>(emissionStart - start).count();
|
||||
mlir::parallelFor(
|
||||
moduleOp.getContext(), 0, jobs.size(), [&](size_t index) { jobResults[index] = emitJob(jobs[index]); });
|
||||
if (timing)
|
||||
timing->emissionSeconds = std::chrono::duration<double>(
|
||||
std::chrono::steady_clock::now() - emissionStart).count();
|
||||
|
||||
pim::CappedDiagnosticReporter diagnostics;
|
||||
Operation* summaryAnchor = nullptr;
|
||||
@@ -1407,14 +1427,33 @@ OnnxMlirCompilerErrorCodes onnx_mlir::compileToPimCode(ModuleOp& moduleOp, std::
|
||||
physicalCoreId < activePhysicalCores.size(); ++physicalCoreId) {
|
||||
if (activePhysicalCores[physicalCoreId])
|
||||
continue;
|
||||
if (auto err =
|
||||
emitEmptyCoreArtifacts(outputDirPath, physicalCoreId))
|
||||
return err;
|
||||
if (!instructions)
|
||||
if (auto err = emitEmptyCoreArtifacts(outputDirPath, physicalCoreId))
|
||||
return err;
|
||||
xbarsPerArrayGroup["core" + std::to_string(physicalCoreId)] =
|
||||
json::Array {};
|
||||
memory.recordCoreReport(physicalCoreId, MemoryReportRow {});
|
||||
}
|
||||
|
||||
if (instructions) {
|
||||
instructions->instructions.resize(maxPhysicalCoreId + 1);
|
||||
instructions->arrayGroups.resize(maxPhysicalCoreId + 1);
|
||||
for (size_t core = 0; core <= maxPhysicalCoreId; ++core) {
|
||||
instructions->arrayGroups[core].clear();
|
||||
llvm::SmallVector<char> empty;
|
||||
raw_svector_ostream stream(empty);
|
||||
pim_binary::writeHeader(stream);
|
||||
instructions->instructions[core].assign(empty.begin(), empty.end());
|
||||
}
|
||||
for (size_t i = 0; i < jobs.size(); ++i) {
|
||||
size_t core = jobs[i].physicalCoreId;
|
||||
instructions->instructions[core] = std::move(jobResults[i].instructions);
|
||||
for (const auto &weight : jobResults[i].usedWeights)
|
||||
instructions->arrayGroups[core].push_back(weight.shape[1] / crossbarSize.getValue());
|
||||
}
|
||||
return CompilerSuccess;
|
||||
}
|
||||
|
||||
llvm::SmallVector<WeightFileRequest, 8> weightRequests;
|
||||
weightRequests.reserve(jobs.size());
|
||||
for (size_t jobIndex = 0; jobIndex < jobs.size(); ++jobIndex) {
|
||||
|
||||
@@ -25,6 +25,7 @@
|
||||
namespace onnx_mlir {
|
||||
|
||||
struct CompiledCoreProgram;
|
||||
struct PimInstructionProgram;
|
||||
class PimInstructionWriter;
|
||||
|
||||
struct MemEntry {
|
||||
@@ -226,7 +227,13 @@ public:
|
||||
|
||||
};
|
||||
|
||||
OnnxMlirCompilerErrorCodes compileToPimCode(mlir::ModuleOp& moduleOpRef, std::string& outputDirName);
|
||||
struct PimCodegenTiming {
|
||||
double preparationSeconds = 0;
|
||||
double emissionSeconds = 0;
|
||||
};
|
||||
|
||||
OnnxMlirCompilerErrorCodes compileToPimCode(mlir::ModuleOp& moduleOpRef, std::string& outputDirName,
|
||||
PimInstructionProgram *instructions = nullptr, PimCodegenTiming *timing = nullptr);
|
||||
|
||||
} // namespace onnx_mlir
|
||||
|
||||
|
||||
@@ -89,6 +89,10 @@ llvm::cl::opt<bool> pimReportConvLowering("pim-report-conv-lowering",
|
||||
llvm::cl::init(true),
|
||||
llvm::cl::cat(OnnxMlirOptions));
|
||||
|
||||
llvm::cl::opt<bool> pimPeriodicDiagnostics("pim-periodic-diagnostics",
|
||||
llvm::cl::desc("Emit the optional pre-codegen recurrence timing report"),
|
||||
llvm::cl::init(false), llvm::cl::cat(OnnxMlirOptions));
|
||||
|
||||
llvm::cl::opt<bool> pimEmitJson("pim-emit-json",
|
||||
llvm::cl::desc("Also emit per-core JSON instruction files alongside binary .pim files"),
|
||||
llvm::cl::init(false),
|
||||
|
||||
@@ -56,6 +56,7 @@ extern llvm::cl::opt<PimSpatialDataflowExportType> pimExportSpatialDataflow;
|
||||
|
||||
extern llvm::cl::opt<bool> pimOnlyCodegen;
|
||||
extern llvm::cl::opt<bool> pimEmitJson;
|
||||
extern llvm::cl::opt<bool> pimPeriodicDiagnostics;
|
||||
extern llvm::cl::opt<bool> pimReportConvLowering;
|
||||
extern llvm::cl::opt<bool> pimDetectCommunicationDeadlock;
|
||||
extern llvm::cl::opt<bool> pimVerifyBufferizationCopyFreedom;
|
||||
|
||||
@@ -14,6 +14,9 @@
|
||||
|
||||
#include "src/Accelerators/PIM/Common/PimCommon.hpp"
|
||||
#include "src/Accelerators/PIM/Compiler/PimCompilerOptions.hpp"
|
||||
#include "PimPipelineSelection.hpp"
|
||||
#include "PimInstructionTiming.hpp"
|
||||
#include "PimPeriodicAnalysis.hpp"
|
||||
#include "src/Accelerators/PIM/Compiler/PimCompilerUtils.hpp"
|
||||
#include "src/Accelerators/PIM/Conversion/ONNXToSpatial/ONNXToSpatialOptions.hpp"
|
||||
#include "src/Accelerators/PIM/Dialect/Pim/PimOps.hpp"
|
||||
@@ -174,8 +177,9 @@ std::pair<size_t, size_t> getConfigPair(const llvm::json::Object& object,
|
||||
}
|
||||
|
||||
void loadPimInterProcessorLatencies(
|
||||
spatial::SchedulingTarget& target,
|
||||
const llvm::json::Object& network) {
|
||||
PimInstructionTiming& target,
|
||||
const llvm::json::Object& network,
|
||||
std::optional<int64_t> memorySwitch) {
|
||||
std::optional<llvm::StringRef> filename =
|
||||
network.getString("net_config_file_path");
|
||||
if (!filename)
|
||||
@@ -240,10 +244,29 @@ void loadPimInterProcessorLatencies(
|
||||
? 0
|
||||
: (latencySum + static_cast<Cost>(pairCount) - 1)
|
||||
/ static_cast<Cost>(pairCount);
|
||||
if (memorySwitch) {
|
||||
const auto *memoryRow = latencies->getObject(std::to_string(*memorySwitch));
|
||||
for (size_t core = 0; core < target.processorCount; ++core) {
|
||||
const auto *row = latencies->getObject(std::to_string(core));
|
||||
auto request = row ? row->getNumber(std::to_string(*memorySwitch)) : std::nullopt;
|
||||
auto response = memoryRow ? memoryRow->getNumber(std::to_string(core)) : std::nullopt;
|
||||
if (!request || !response || !std::isfinite(*request)
|
||||
|| !std::isfinite(*response) || *request < 0 || *response < 0
|
||||
|| *request >= static_cast<double>(std::numeric_limits<Cost>::max())
|
||||
|| *response >= static_cast<double>(std::numeric_limits<Cost>::max())) {
|
||||
target.sharedMemoryRequestLatencyNs.clear();
|
||||
target.sharedMemoryResponseLatencyNs.clear();
|
||||
break;
|
||||
}
|
||||
target.sharedMemoryRequestLatencyNs.push_back(std::ceil(*request));
|
||||
target.sharedMemoryResponseLatencyNs.push_back(std::ceil(*response));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
spatial::SchedulingTarget getPimSchedulingTarget() {
|
||||
spatial::SchedulingTarget target = getDefaultPimSchedulingTarget();
|
||||
PimInstructionTiming loadPimInstructionTiming() {
|
||||
PimInstructionTiming target;
|
||||
static_cast<spatial::SchedulingTarget &>(target) = getDefaultPimSchedulingTarget();
|
||||
if (pimTargetConfig.empty())
|
||||
return target;
|
||||
|
||||
@@ -298,12 +321,19 @@ spatial::SchedulingTarget getPimSchedulingTarget() {
|
||||
|| target.matrixColumns != crossbarSize.getValue())
|
||||
llvm::report_fatal_error("Pim target config resources do not match --core-count, "
|
||||
"--crossbar-count, and --crossbar-size");
|
||||
loadPimInterProcessorLatencies(target, network);
|
||||
loadPimInterProcessorLatencies(target, network,
|
||||
core.getInteger("global_memory_switch_id"));
|
||||
|
||||
target.processorPeriodNs =
|
||||
getConfigCost(core, "period", target.processorPeriodNs);
|
||||
target.signalIssueNs = target.processorPeriodNs;
|
||||
target.controlMessageBytes = 1;
|
||||
target.sharedMemoryReadReturnsPayload = false;
|
||||
target.peerPermissionThenSourceRead = true;
|
||||
target.localMemoryWidthBytes =
|
||||
getConfigCost(localMemory, "data_width", target.localMemoryWidthBytes);
|
||||
target.localMemoryPeriodNs =
|
||||
getConfigCost(localMemory, "period", target.localMemoryPeriodNs);
|
||||
target.localMemoryReadLatencyCycles =
|
||||
getConfigCost(localMemory, "read_latency_cycle", target.localMemoryReadLatencyCycles);
|
||||
target.localMemoryWriteLatencyCycles =
|
||||
@@ -345,6 +375,65 @@ spatial::SchedulingTarget getPimSchedulingTarget() {
|
||||
|
||||
} // namespace
|
||||
|
||||
PimInstructionTiming getPimInstructionTiming() {
|
||||
return loadPimInstructionTiming();
|
||||
}
|
||||
|
||||
spatial::SchedulingTarget getPimSchedulingTarget() {
|
||||
auto timing = getPimInstructionTiming();
|
||||
auto target = static_cast<spatial::SchedulingTarget>(timing);
|
||||
configurePimScheduleCosts(target, std::move(timing));
|
||||
return target;
|
||||
}
|
||||
|
||||
llvm::Expected<int64_t> getPimPeriodicTimingContract() {
|
||||
auto buffer = llvm::MemoryBuffer::getFile(pimTargetConfig);
|
||||
if (!buffer) return llvm::createStringError("queue selection requires --pim-target-config with instruction timing");
|
||||
auto parsed = llvm::json::parse(buffer.get()->getBuffer());
|
||||
if (!parsed) return parsed.takeError();
|
||||
const auto *root = parsed->getAsObject();
|
||||
const auto *chip = root ? root->getObject("chip_config") : nullptr;
|
||||
const auto *core = chip ? chip->getObject("core_config") : nullptr;
|
||||
const auto *matrix = core ? core->getObject("matrix_config") : nullptr;
|
||||
if (!matrix || core->getInteger("rob_size").value_or(1) != 1)
|
||||
return llvm::createStringError("queue selection requires a target with ROB capacity one and matrix timing");
|
||||
auto bits = matrix->getInteger("cell_precision").value_or(2);
|
||||
if (bits <= 0) return llvm::createStringError("matrix cell precision must be positive");
|
||||
return bits;
|
||||
}
|
||||
|
||||
void addPimLoweringPasses(PassManager &pm,
|
||||
const spatial::SpatialTargetResources &targetResources, PimEmissionTargetType target,
|
||||
bool reportProgress) {
|
||||
auto message = [&](llvm::StringRef text) {
|
||||
if (reportProgress) pm.addPass(createMessagePass(text.str()));
|
||||
};
|
||||
if (target >= EmitPim) {
|
||||
pm.addPass(createSpatialToPimPass(targetResources));
|
||||
message("Spatial lowered to Pim");
|
||||
}
|
||||
|
||||
if (target >= EmitPimBufferized) {
|
||||
pm.addPass(createPimBufferizationPreparationPass(pimVerifyBufferizationCopyFreedom.getValue()));
|
||||
pm.addPass(createPimOneShotBufferizationPass());
|
||||
pm.addPass(createPimMemoryNormalizationPass());
|
||||
pm.addPass(createPimBufferizationVerificationPass());
|
||||
message("Pim bufferized");
|
||||
}
|
||||
|
||||
if (target >= EmitPimCodegen) {
|
||||
pm.addPass(mlir::createLowerAffinePass());
|
||||
pm.addPass(createPimHostConstantFoldingPass());
|
||||
message("Pim host constants folded");
|
||||
pm.addPass(createPimInstructionSelectionPass());
|
||||
message("Pim instructions selected");
|
||||
pm.addPass(createPimLocalMemoryPlanningPass());
|
||||
message("Pim local memory planned");
|
||||
pm.addPass(createPimVerificationPass(targetResources, pimDetectCommunicationDeadlock.getValue()));
|
||||
message("Pim verified");
|
||||
}
|
||||
}
|
||||
|
||||
void addPassesPim(OwningOpRef<ModuleOp>& module,
|
||||
PassManager& pm,
|
||||
EmissionTargetType& emissionTarget,
|
||||
@@ -374,34 +463,22 @@ void addPassesPim(OwningOpRef<ModuleOp>& module,
|
||||
pm.addPass(createLowerSpatialPlansPass(targetResources, planningOptions, exportStage));
|
||||
pm.addPass(createTrivialGraphComputeMergePass(
|
||||
schedulingTarget.residentWeightCapacity, exportStage));
|
||||
pm.addPass(spatial::createScheduleAndRealizeSpatialPass(
|
||||
schedulingTarget, exportStage, pipelineStages.getValue()));
|
||||
if (pipelineStages.getValue() > 1 && pimEmissionTarget >= EmitPimCodegen) {
|
||||
pm.addPass(createPimPipelineSelectionPass(schedulingTarget, targetResources,
|
||||
pipelineStages.getValue(), exportStage));
|
||||
pm.addPass(createEmitPimCodePass());
|
||||
return;
|
||||
} else {
|
||||
if (pipelineStages.getValue() > 1)
|
||||
pm.addPass(createMessagePass("Queue-aware pipeline: intermediate emission retains the original schedule without backend evaluation"));
|
||||
pm.addPass(spatial::createScheduleAndRealizeSpatialPass(
|
||||
schedulingTarget, exportStage, pipelineStages.getValue(), pimPeriodicDiagnostics.getValue()));
|
||||
}
|
||||
pm.addPass(createMessagePass("Onnx lowered to Spatial"));
|
||||
}
|
||||
|
||||
if (pimEmissionTarget >= EmitPim) {
|
||||
pm.addPass(createSpatialToPimPass(targetResources));
|
||||
pm.addPass(createMessagePass("Spatial lowered to Pim"));
|
||||
}
|
||||
|
||||
if (pimEmissionTarget >= EmitPimBufferized) {
|
||||
pm.addPass(createPimBufferizationPreparationPass(pimVerifyBufferizationCopyFreedom.getValue()));
|
||||
pm.addPass(createPimOneShotBufferizationPass());
|
||||
pm.addPass(createPimMemoryNormalizationPass());
|
||||
pm.addPass(createPimBufferizationVerificationPass());
|
||||
pm.addPass(createMessagePass("Pim bufferized"));
|
||||
}
|
||||
|
||||
addPimLoweringPasses(pm, targetResources, pimEmissionTarget);
|
||||
if (pimEmissionTarget >= EmitPimCodegen) {
|
||||
pm.addPass(mlir::createLowerAffinePass());
|
||||
pm.addPass(createPimHostConstantFoldingPass());
|
||||
pm.addPass(createMessagePass("Pim host constants folded"));
|
||||
pm.addPass(createPimInstructionSelectionPass());
|
||||
pm.addPass(createMessagePass("Pim instructions selected"));
|
||||
pm.addPass(createPimLocalMemoryPlanningPass());
|
||||
pm.addPass(createMessagePass("Pim local memory planned"));
|
||||
pm.addPass(createPimVerificationPass(targetResources, pimDetectCommunicationDeadlock.getValue()));
|
||||
pm.addPass(createMessagePass("Pim verified"));
|
||||
pm.addPass(createEmitPimCodePass());
|
||||
pm.addPass(createMessagePass("Pim code emitted"));
|
||||
}
|
||||
|
||||
@@ -1,6 +1,8 @@
|
||||
#pragma once
|
||||
|
||||
#include "PimCompilerOptions.hpp"
|
||||
#include "mlir/IR/BuiltinOps.h"
|
||||
#include "llvm/Support/Error.h"
|
||||
#include "mlir/Pass/PassManager.h"
|
||||
#include "mlir/Pass/PassRegistry.h"
|
||||
|
||||
@@ -8,6 +10,16 @@
|
||||
|
||||
namespace onnx_mlir {
|
||||
|
||||
namespace spatial { struct SchedulingTarget; struct SpatialTargetResources; }
|
||||
spatial::SchedulingTarget getPimSchedulingTarget();
|
||||
struct PimInstructionTiming;
|
||||
PimInstructionTiming getPimInstructionTiming();
|
||||
llvm::Expected<int64_t> getPimPeriodicTimingContract();
|
||||
|
||||
void addPimLoweringPasses(mlir::PassManager &pm,
|
||||
const spatial::SpatialTargetResources &resources, PimEmissionTargetType target,
|
||||
bool reportProgress = true);
|
||||
|
||||
void addPassesPim(mlir::OwningOpRef<mlir::ModuleOp>& module,
|
||||
mlir::PassManager& pm,
|
||||
EmissionTargetType& emissionTarget,
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
#pragma once
|
||||
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/SchedulingTarget.hpp"
|
||||
|
||||
namespace onnx_mlir {
|
||||
|
||||
// PIM transport and instruction timing. Spatial consumes only the base costs.
|
||||
struct PimInstructionTiming : spatial::SchedulingTarget {
|
||||
bool peerPermissionThenSourceRead = false;
|
||||
Cost signalIssueNs = 0;
|
||||
Cost controlMessageBytes = 0;
|
||||
bool sharedMemoryReadReturnsPayload = true;
|
||||
std::vector<Cost> sharedMemoryRequestLatencyNs;
|
||||
std::vector<Cost> sharedMemoryResponseLatencyNs;
|
||||
|
||||
bool hasPeriodicTransferTiming() const {
|
||||
return processorCount && sharedMemoryWidthBytes && controlMessageBytes
|
||||
&& peerPermissionThenSourceRead
|
||||
&& sharedMemoryRequestLatencyNs.size() == processorCount
|
||||
&& sharedMemoryResponseLatencyNs.size() == processorCount
|
||||
&& interProcessorLatencyNs.size() == processorCount * processorCount
|
||||
&& localMemoryWidthBytes && transferWidthBytes;
|
||||
}
|
||||
|
||||
Cost getLinkServiceNs(Cost bytes, size_t source, size_t destination) const {
|
||||
Cost flits = bytes / transferWidthBytes + (bytes % transferWidthBytes != 0);
|
||||
return checkedMultiply(flits,
|
||||
getInterProcessorLatencyNs(source, destination));
|
||||
}
|
||||
|
||||
Cost getSharedMemoryLinkServiceNs(Cost bytes, size_t core, bool response) const {
|
||||
Cost flits = bytes / transferWidthBytes + (bytes % transferWidthBytes != 0);
|
||||
return checkedMultiply(flits, response ? sharedMemoryResponseLatencyNs[core]
|
||||
: sharedMemoryRequestLatencyNs[core]);
|
||||
}
|
||||
|
||||
};
|
||||
|
||||
} // namespace onnx_mlir
|
||||
@@ -0,0 +1,671 @@
|
||||
#include "PimPeriodicAnalysis.hpp"
|
||||
#include "PimInstructionTiming.hpp"
|
||||
#include "PimCodeGen.hpp"
|
||||
#include "PimCompilerUtils.hpp"
|
||||
#include "PimCompilerOptions.hpp"
|
||||
#include "PimScheduleDiagnostic.hpp"
|
||||
#include "llvm/Support/SHA256.h"
|
||||
#include "llvm/ADT/StringExtras.h"
|
||||
#include <chrono>
|
||||
#include "PimBinaryFormat.hpp"
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PeriodicDependencies.hpp"
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/SchedulingTarget.hpp"
|
||||
|
||||
#include "llvm/ADT/SmallVector.h"
|
||||
#include <array>
|
||||
#include <map>
|
||||
#include <limits>
|
||||
#include <queue>
|
||||
#include <cmath>
|
||||
|
||||
using namespace onnx_mlir;
|
||||
using namespace onnx_mlir::spatial;
|
||||
using Opcode = pim_binary::Opcode;
|
||||
namespace {
|
||||
using Key = std::pair<int64_t, int64_t>;
|
||||
struct Event { int64_t core; uint32_t pc; llvm::StringRef kind; };
|
||||
struct Send { size_t enqueue, requestArrival, payloadArrival; Cost bytes; };
|
||||
struct Receive { size_t complete, permissionArrival, permissionDispatch; Cost bytes; };
|
||||
struct Request { size_t begin, arrival; };
|
||||
struct Notice { size_t arrival; int64_t source; };
|
||||
struct Wait { size_t end; int64_t expected; };
|
||||
struct Core {
|
||||
size_t entry, tail, firstNetwork, lastNetwork;
|
||||
Cost pending = 0, work = 0, instructions = 0;
|
||||
std::array<Cost, 7> service{};
|
||||
bool blockingNetworkInChunk = false;
|
||||
};
|
||||
constexpr size_t absent = std::numeric_limits<size_t>::max();
|
||||
|
||||
llvm::json::Object estimateQueuedPeriod(const std::vector<Event> &events,
|
||||
const std::vector<PeriodicDependency> &edges,
|
||||
const std::vector<std::pair<size_t, Cost>> &services, const std::map<int64_t, Core> &cores,
|
||||
bool reverseTies) {
|
||||
struct Node {
|
||||
llvm::SmallVector<size_t, 2> incoming, outgoing;
|
||||
size_t completed = 0, readyInputs = 0;
|
||||
bool busy = false;
|
||||
};
|
||||
struct Pending {
|
||||
Cost time;
|
||||
size_t node;
|
||||
bool arrival;
|
||||
};
|
||||
struct Tokens {
|
||||
llvm::SmallVector<Cost, 1> values;
|
||||
size_t head = 0;
|
||||
};
|
||||
auto later = [reverseTies](const Pending &a, const Pending &b) {
|
||||
if (a.time != b.time) return a.time > b.time;
|
||||
return reverseTies ? a.node < b.node : a.node > b.node;
|
||||
};
|
||||
std::vector<Node> nodes(events.size());
|
||||
std::vector<Tokens> tokens(edges.size());
|
||||
std::vector<Cost> service(events.size(), 0);
|
||||
std::vector<bool> memoryAccess(events.size(), false);
|
||||
std::vector<bool> entries(events.size(), false);
|
||||
for (auto [node, cost] : services) {
|
||||
service[node] = cost;
|
||||
memoryAccess[node] = true;
|
||||
}
|
||||
for (const auto &[core, state] : cores) entries[state.entry] = true;
|
||||
for (auto [index, edge] : llvm::enumerate(edges)) {
|
||||
nodes[edge.source].outgoing.push_back(index);
|
||||
nodes[edge.target].incoming.push_back(index);
|
||||
tokens[index].values.assign(edge.distance, 0);
|
||||
nodes[edge.target].readyInputs += edge.distance != 0;
|
||||
}
|
||||
std::priority_queue<Pending, std::vector<Pending>, decltype(later)> pending(later);
|
||||
auto schedule = [&](size_t node) {
|
||||
auto &state = nodes[node];
|
||||
if (state.busy || state.readyInputs != state.incoming.size()) return;
|
||||
Cost time = 0;
|
||||
for (size_t edge : state.incoming) {
|
||||
auto &queue = tokens[edge];
|
||||
time = std::max(time, queue.values[queue.head++]);
|
||||
if (queue.head == queue.values.size()) {
|
||||
queue.values.clear(); queue.head = 0; --state.readyInputs;
|
||||
} else if (queue.head * 2 >= queue.values.size()) {
|
||||
queue.values.erase(queue.values.begin(), queue.values.begin() + queue.head);
|
||||
queue.head = 0;
|
||||
}
|
||||
}
|
||||
state.busy = true;
|
||||
pending.push({time - service[node], node, memoryAccess[node]});
|
||||
};
|
||||
for (size_t node = 0; node < nodes.size(); ++node) schedule(node);
|
||||
constexpr size_t maximumRounds = 64;
|
||||
std::array<Cost, maximumRounds> roundEnd{};
|
||||
std::array<size_t, maximumRounds> roundCount{};
|
||||
std::map<int64_t, std::array<Cost, maximumRounds>> queueWait;
|
||||
Cost available = 0;
|
||||
Cost maximumQueueWait = 0;
|
||||
size_t maximumQueueNode = 0;
|
||||
size_t processed = 0;
|
||||
const size_t budget = 16 * (events.size() + edges.size());
|
||||
llvm::json::Object result{{"status", "unavailable"}};
|
||||
while (!pending.empty() && processed++ < budget) {
|
||||
auto next = pending.top(); pending.pop();
|
||||
auto &node = nodes[next.node];
|
||||
if (node.completed >= maximumRounds) {
|
||||
result["reason"] = "queue diagnostic exceeded its per-event generation bound";
|
||||
return result;
|
||||
}
|
||||
if (next.arrival) {
|
||||
Cost begin = std::max(available, next.time);
|
||||
if (begin - next.time > maximumQueueWait) {
|
||||
maximumQueueWait = begin - next.time;
|
||||
maximumQueueNode = next.node;
|
||||
}
|
||||
queueWait[events[next.node].core][node.completed] += begin - next.time;
|
||||
available = begin + service[next.node];
|
||||
pending.push({available, next.node, false});
|
||||
continue;
|
||||
}
|
||||
++node.completed;
|
||||
node.busy = false;
|
||||
if (entries[next.node]) {
|
||||
size_t round = node.completed - 1;
|
||||
roundEnd[round] = std::max(roundEnd[round], next.time);
|
||||
if (++roundCount[round] == cores.size() && round >= 4 && round % 2 == 0) {
|
||||
Cost first = roundEnd[round - 2] - roundEnd[round - 4];
|
||||
Cost second = roundEnd[round] - roundEnd[round - 2];
|
||||
Cost maximum = std::max(first, second);
|
||||
if (maximum && std::abs(static_cast<double>(first) - second) <= maximum * 0.01) {
|
||||
result["status"] = "estimated";
|
||||
result["interval_ns"] = (roundEnd[round] - roundEnd[round - 4]) / 4.0;
|
||||
result["completed_generations"] = static_cast<int64_t>(round);
|
||||
result["processed_events"] = static_cast<int64_t>(processed);
|
||||
result["maximum_queue_wait_ns_including_bootstrap"] = static_cast<int64_t>(maximumQueueWait);
|
||||
if (maximumQueueWait) {
|
||||
result["maximum_queue_core"] = events[maximumQueueNode].core;
|
||||
result["maximum_queue_pc"] = events[maximumQueueNode].pc;
|
||||
}
|
||||
llvm::json::Array waits;
|
||||
for (const auto &[core, values] : queueWait) {
|
||||
Cost wait = 0;
|
||||
for (size_t generation = round - 4; generation < round; ++generation)
|
||||
wait += values[generation];
|
||||
waits.push_back(llvm::json::Object{{"core", core}, {"queue_wait_ns", wait / 4.0}});
|
||||
}
|
||||
result["per_core"] = std::move(waits);
|
||||
result["tie_policy"] = reverseTies ? "reverse event-index order for simultaneous readiness"
|
||||
: "event-index order for simultaneous readiness";
|
||||
return result;
|
||||
}
|
||||
}
|
||||
}
|
||||
for (size_t edge : node.outgoing) {
|
||||
if (tokens[edge].values.empty()) ++nodes[edges[edge].target].readyInputs;
|
||||
tokens[edge].values.push_back(next.time + edges[edge].delayNs);
|
||||
schedule(edges[edge].target);
|
||||
}
|
||||
schedule(next.node);
|
||||
}
|
||||
result["reason"] = "queue diagnostic did not converge within its event budget";
|
||||
return result;
|
||||
}
|
||||
|
||||
class Analysis {
|
||||
public:
|
||||
PimInstructionTiming target;
|
||||
PimPeriodicOptions options;
|
||||
bool queues = true, issue = true, memoryQueue = false;
|
||||
Cost cellBits;
|
||||
std::vector<Event> events;
|
||||
std::vector<PeriodicDependency> edges;
|
||||
std::map<int64_t, Core> cores;
|
||||
SharedMemoryDemand memory;
|
||||
std::vector<std::pair<size_t, Cost>> memoryServices;
|
||||
std::array<Cost, pim_binary::kOpcodeCount> opcodeService{}, opcodeCounts{};
|
||||
std::string error;
|
||||
|
||||
bool readCore(int64_t core, llvm::StringRef data, const llvm::json::Array &groups);
|
||||
llvm::json::Object queueReport();
|
||||
bool connect();
|
||||
llvm::json::Object report();
|
||||
|
||||
private:
|
||||
std::map<Key, std::vector<Send>> sends;
|
||||
std::map<Key, std::vector<Receive>> receives;
|
||||
std::map<Key, std::vector<Notice>> signals;
|
||||
std::map<Key, Wait> waits;
|
||||
size_t event(int64_t core, uint32_t pc, llvm::StringRef kind) {
|
||||
events.push_back({core, pc, kind});
|
||||
return events.size() - 1;
|
||||
}
|
||||
size_t append(int64_t core, uint32_t pc, llvm::StringRef kind, Cost delay = 0) {
|
||||
size_t next = event(core, pc, kind);
|
||||
Core &state = cores.at(core);
|
||||
edges.push_back({state.tail, next, delay, 0});
|
||||
state.tail = next;
|
||||
return next;
|
||||
}
|
||||
void flush(int64_t core, uint32_t pc) {
|
||||
Core &state = cores.at(core);
|
||||
if (state.pending) {
|
||||
append(core, pc, "local-work", state.pending);
|
||||
state.work += state.pending;
|
||||
state.pending = 0;
|
||||
}
|
||||
if (state.blockingNetworkInChunk)
|
||||
state.lastNetwork = state.tail;
|
||||
state.blockingNetworkInChunk = false;
|
||||
}
|
||||
Request network(int64_t core, uint32_t pc, size_t ready,
|
||||
Cost latency, Cost hold = 0) {
|
||||
Core &state = cores.at(core);
|
||||
size_t begin = event(core, pc, "fifo-dispatch"), arrival = event(core, pc, "delivery");
|
||||
size_t end = hold ? event(core, pc, "switch-release") : arrival;
|
||||
edges.push_back({ready, begin, 0, 0});
|
||||
if (queues && state.lastNetwork != absent)
|
||||
edges.push_back({state.lastNetwork, begin, 0, 0});
|
||||
edges.push_back({begin, arrival, latency, 0});
|
||||
if (hold)
|
||||
edges.push_back({arrival, end, hold, 0});
|
||||
if (state.firstNetwork == absent)
|
||||
state.firstNetwork = begin;
|
||||
state.lastNetwork = end;
|
||||
return {begin, arrival};
|
||||
}
|
||||
Cost local(Cost bytes, bool write) const {
|
||||
return target.getLocalMemoryServiceNs(bytes, write);
|
||||
}
|
||||
Cost retirement(bool matrix = false) const {
|
||||
// Completion can join the current clock edge before ClockDomain runs.
|
||||
// ROB retirement needs another edge; matrix also has AG/commit registers.
|
||||
return issue ? (matrix ? 3 : 1) * target.processorPeriodNs : 0;
|
||||
}
|
||||
bool communication(int64_t core, uint32_t pc, Opcode op,
|
||||
int32_t peer, int32_t reg, int32_t count, Cost bytes);
|
||||
};
|
||||
|
||||
bool Analysis::communication(int64_t core, uint32_t pc, Opcode op,
|
||||
int32_t peer, int32_t reg, int32_t count, Cost bytes) {
|
||||
flush(core, pc);
|
||||
size_t begin = append(core, pc, "communication");
|
||||
if (op == Opcode::sync) {
|
||||
if (peer < 0 || peer >= static_cast<int64_t>(target.processorCount)
|
||||
|| reg < 0 || reg >= static_cast<int64_t>(target.synchronizationRegisterCount)) {
|
||||
error = "SYNC has an invalid core or event register";
|
||||
return false;
|
||||
}
|
||||
auto request = network(core, pc, begin,
|
||||
target.getLinkServiceNs(target.controlMessageBytes, core, peer));
|
||||
signals[{peer, reg}].push_back({request.arrival, core});
|
||||
append(core, pc, "sync-complete", target.signalIssueNs + retirement());
|
||||
cores.at(core).work += target.signalIssueNs + retirement();
|
||||
cores.at(core).service[6] += target.signalIssueNs + retirement();
|
||||
opcodeService[static_cast<unsigned>(op)] += target.signalIssueNs + retirement();
|
||||
} else if (op == Opcode::wait) {
|
||||
size_t end = append(core, pc, "wait-complete", retirement());
|
||||
if (reg < 0 || reg >= static_cast<int64_t>(target.synchronizationRegisterCount)
|
||||
|| count <= 0 || !waits.emplace(Key{core, reg}, Wait{end, count}).second) {
|
||||
error = "WAIT requires one positive exact count per register per generation";
|
||||
return false;
|
||||
}
|
||||
cores.at(core).work += retirement();
|
||||
cores.at(core).service[6] += retirement();
|
||||
opcodeService[static_cast<unsigned>(op)] += retirement();
|
||||
} else {
|
||||
if (peer < 0 || peer >= static_cast<int64_t>(target.processorCount)) {
|
||||
error = "direct transfer has an invalid peer";
|
||||
return false;
|
||||
}
|
||||
size_t complete = event(core, pc, op == Opcode::send ? "send-complete" : "receive-complete");
|
||||
edges.push_back({begin, complete, 0, 0});
|
||||
cores.at(core).tail = complete;
|
||||
if (op == Opcode::send) {
|
||||
auto request = network(core, pc, begin,
|
||||
target.getLinkServiceNs(target.controlMessageBytes, core, peer), 1);
|
||||
// SwitchSocket::send enqueues immediately in an unbounded FIFO.
|
||||
size_t enqueue = event(core, pc, "payload-enqueue");
|
||||
edges.push_back({enqueue, complete, retirement(), 0});
|
||||
auto payload = network(core, pc, enqueue, target.getLinkServiceNs(bytes, core, peer));
|
||||
sends[{core, peer}].push_back({enqueue, request.arrival, payload.arrival, bytes});
|
||||
} else {
|
||||
auto permission = network(core, pc, begin,
|
||||
target.getLinkServiceNs(target.controlMessageBytes, core, peer));
|
||||
receives[{peer, core}].push_back({complete, permission.arrival, permission.begin, bytes});
|
||||
}
|
||||
cores.at(core).work += local(bytes, op == Opcode::recv) + retirement();
|
||||
cores.at(core).service[2] += local(bytes, op == Opcode::recv) + retirement();
|
||||
opcodeService[static_cast<unsigned>(op)] += local(bytes, op == Opcode::recv) + retirement();
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
bool Analysis::readCore(int64_t core, llvm::StringRef data, const llvm::json::Array &groups) {
|
||||
if (data.size() < 12 || data.take_front(4) != "PIMB"
|
||||
|| llvm::support::endian::read32le(data.data() + 4) != 1) {
|
||||
error = "invalid binary instruction header"; return false;
|
||||
}
|
||||
uint32_t count = llvm::support::endian::read32le(data.data() + 8);
|
||||
if (data.size() != 12ull + 20ull * count) {
|
||||
error = "binary instruction count does not match file size"; return false;
|
||||
}
|
||||
if (!count) return true;
|
||||
size_t entry = event(core, 0, "body-entry");
|
||||
cores.emplace(core, Core{entry, entry, absent, absent});
|
||||
Core &state = cores.at(core);
|
||||
Cost inputBits = 0, outputBits = 0;
|
||||
for (uint32_t pc = 0; pc < count; ++pc) {
|
||||
const char *record = data.data() + 12 + 20ull * pc;
|
||||
unsigned rawOpcode = static_cast<uint8_t>(record[0]);
|
||||
if (rawOpcode > static_cast<unsigned>(Opcode::sync)) { error = "unsupported instruction opcode"; return false; }
|
||||
Opcode op = static_cast<Opcode>(rawOpcode);
|
||||
++opcodeCounts[rawOpcode];
|
||||
int32_t arg = llvm::support::endian::read32le(record + 4);
|
||||
int32_t a = llvm::support::endian::read32le(record + 8);
|
||||
int32_t b = llvm::support::endian::read32le(record + 12);
|
||||
int32_t length = llvm::support::endian::read32le(record + 16);
|
||||
++state.instructions;
|
||||
if (op <= Opcode::setbw) {
|
||||
if (op == Opcode::sld) { error = "scalar memory loads are unsupported by this diagnostic"; return false; }
|
||||
if (op == Opcode::setbw) {
|
||||
if (a <= 0 || b <= 0) { error = "invalid SETBW"; return false; }
|
||||
if (options.oraclePrecision && (a != 32 || b != 32)) {
|
||||
error = "oracle timing projection requires 32-bit native vector widths"; return false;
|
||||
}
|
||||
inputBits = options.oraclePrecision ? 8 : a;
|
||||
outputBits = options.oraclePrecision ? 8 : b;
|
||||
}
|
||||
// shortcut: serialize scalar issue; model frontend buffering before selection.
|
||||
Cost cost = issue ? target.processorPeriodNs : 0;
|
||||
state.pending += cost; state.service[5] += cost;
|
||||
opcodeService[static_cast<unsigned>(op)] += cost;
|
||||
continue;
|
||||
}
|
||||
if (length < 0 || op > Opcode::sync || op == Opcode::vsoftmax) {
|
||||
error = "unsupported instruction opcode or negative transfer length"; return false;
|
||||
}
|
||||
Cost bytes = length;
|
||||
if (options.oraclePrecision && (op == Opcode::ld || op == Opcode::st
|
||||
|| op == Opcode::lldi || op == Opcode::lmv || op == Opcode::send || op == Opcode::recv)) {
|
||||
if (length % 4) { error = "oracle timing projection requires element-aligned transfers"; return false; }
|
||||
bytes /= 4;
|
||||
}
|
||||
Cost in = (inputBits + 7) / 8, out = (outputBits + 7) / 8;
|
||||
if (op <= Opcode::vrsl && (!inputBits || !outputBits)) {
|
||||
error = "compute instruction precedes SETBW"; return false;
|
||||
}
|
||||
if (op >= Opcode::send) {
|
||||
if (!communication(core, pc, op, arg, a, b, bytes)) return false;
|
||||
continue;
|
||||
}
|
||||
if (op == Opcode::ld || op == Opcode::st) {
|
||||
bool write = op == Opcode::st;
|
||||
auto demand = target.getSharedMemoryDemand(bytes, 1, write);
|
||||
memory += demand;
|
||||
Cost completion = (write ? 0 : local(bytes, true)) + retirement();
|
||||
Cost cost = (write ? local(bytes, false) : 0) + demand.serviceNs + completion
|
||||
+ target.getSharedMemoryLinkServiceNs(write ? bytes : target.controlMessageBytes, core, false)
|
||||
+ target.getSharedMemoryLinkServiceNs(!write && target.sharedMemoryReadReturnsPayload ? bytes : 0, core, true);
|
||||
state.service[write ? 4 : 3] += cost;
|
||||
opcodeService[static_cast<unsigned>(op)] += cost;
|
||||
if (memoryQueue) {
|
||||
flush(core, pc);
|
||||
if (write) append(core, pc, "store-source-read", local(bytes, false));
|
||||
auto request = network(core, pc, state.tail,
|
||||
target.getSharedMemoryLinkServiceNs(write ? bytes : target.controlMessageBytes, core, false));
|
||||
size_t done = event(core, pc, "memory-service-complete");
|
||||
edges.push_back({request.arrival, done, demand.serviceNs, 0});
|
||||
memoryServices.emplace_back(done, demand.serviceNs);
|
||||
Cost reply = target.getSharedMemoryLinkServiceNs(
|
||||
!write && target.sharedMemoryReadReturnsPayload ? bytes : 0, core, true);
|
||||
size_t response = done;
|
||||
if (reply) {
|
||||
response = event(core, pc, "memory-response");
|
||||
edges.push_back({done, response, reply, 0});
|
||||
}
|
||||
state.lastNetwork = response;
|
||||
size_t complete = event(core, pc, "memory-complete");
|
||||
edges.push_back({response, complete, completion, 0});
|
||||
state.tail = complete;
|
||||
state.work += cost;
|
||||
continue;
|
||||
}
|
||||
if (!state.blockingNetworkInChunk) {
|
||||
flush(core, pc);
|
||||
if (write) {
|
||||
Cost read = local(bytes, false);
|
||||
append(core, pc, "store-source-read", read);
|
||||
state.work += read;
|
||||
cost -= read;
|
||||
}
|
||||
size_t begin = append(core, pc, "memory-queue-join");
|
||||
if (queues && state.lastNetwork != absent)
|
||||
edges.push_back({state.lastNetwork, begin, 0, 0});
|
||||
if (state.firstNetwork == absent)
|
||||
state.firstNetwork = begin;
|
||||
state.blockingNetworkInChunk = true;
|
||||
}
|
||||
// ROB=1 serializes these blocking requests. No external dependency joins
|
||||
// inside a chunk; retain their exact service sum without per-request nodes.
|
||||
state.pending += cost;
|
||||
continue;
|
||||
}
|
||||
Cost cost = 0;
|
||||
unsigned resource = 2;
|
||||
if (op == Opcode::mvmul) {
|
||||
if (arg <= 0 || length < 0 || static_cast<size_t>(length) >= groups.size()
|
||||
|| !groups[length].getAsInteger() || *groups[length].getAsInteger() <= 0) {
|
||||
error = "matrix instruction references an invalid array group"; return false;
|
||||
}
|
||||
Cost output = (target.matrixColumns * cellBits / arg) * *groups[length].getAsInteger() * out;
|
||||
cost = local(target.matrixRows * in * target.matrixInputReadMultiplier, false)
|
||||
+ target.getMatrixComputeLatencyNs(inputBits) + local(output, true);
|
||||
cost += retirement(true); resource = 0;
|
||||
} else if (op <= Opcode::vrsl) {
|
||||
bool binary = op >= Opcode::vvadd && op <= Opcode::vvsra;
|
||||
bool outputWidth = op == Opcode::vvmul || op == Opcode::vvsll || op == Opcode::vvsra || op == Opcode::vtanh || op == Opcode::vsigm || op == Opcode::vrsu || op == Opcode::vrsl;
|
||||
// The oracle's VectorUnit uses input width for VAVG/VVDMUL writeback.
|
||||
// It waits vector_latency_cycle nanoseconds, independently of core period.
|
||||
cost = local(bytes * in, false) * (binary ? 2 : 1)
|
||||
+ ((bytes + target.vectorWidth - 1) / target.vectorWidth) * target.vectorLatencyCycles
|
||||
+ local(bytes * (outputWidth ? out : in), true) + retirement();
|
||||
resource = 1;
|
||||
} else {
|
||||
cost = local(bytes, true) + (op == Opcode::lmv ? local(bytes, false) : 0) + retirement();
|
||||
}
|
||||
state.pending += cost;
|
||||
state.service[resource] += cost;
|
||||
opcodeService[static_cast<unsigned>(op)] += cost;
|
||||
}
|
||||
flush(core, count);
|
||||
edges.push_back({state.tail, state.entry, 0, 1});
|
||||
if (queues && state.firstNetwork != absent)
|
||||
edges.push_back({state.lastNetwork, state.firstNetwork, 0, 1});
|
||||
return true;
|
||||
}
|
||||
|
||||
bool Analysis::connect() {
|
||||
if (sends.size() != receives.size() || signals.size() != waits.size()) {
|
||||
error = "communication endpoint sets differ"; return false;
|
||||
}
|
||||
for (const auto &[key, producers] : sends) {
|
||||
auto found = receives.find(key);
|
||||
if (found == receives.end() || found->second.size() != producers.size()) {
|
||||
error = "direct SEND/RECV FIFO counts differ"; return false;
|
||||
}
|
||||
for (auto [send, receive] : llvm::zip(producers, found->second)) {
|
||||
if (send.bytes != receive.bytes) { error = "direct payload sizes differ"; return false; }
|
||||
// The receiver responds after both its RECV and the sender request arrive.
|
||||
// Its outgoing FIFO may still contain earlier SYNC or transport requests.
|
||||
edges.push_back({send.requestArrival, receive.permissionDispatch, 0, 0});
|
||||
edges.push_back({receive.permissionArrival, send.enqueue, local(send.bytes, false), 0});
|
||||
edges.push_back({send.payloadArrival, receive.complete, local(send.bytes, true) + retirement(), 0});
|
||||
}
|
||||
}
|
||||
for (const auto &[key, pending] : waits) {
|
||||
auto found = signals.find(key);
|
||||
if (found == signals.end() || found->second.size() != static_cast<size_t>(pending.expected)) {
|
||||
error = "event register does not receive its exact static count"; return false;
|
||||
}
|
||||
for (const auto ¬ice : found->second)
|
||||
edges.push_back({notice.arrival, pending.end, retirement(), 0});
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
llvm::json::Object Analysis::report() {
|
||||
llvm::json::Object result;
|
||||
if (cores.empty()) {
|
||||
error = "artifact has no active repeating instruction stream";
|
||||
result["status"] = "unavailable"; result["reason"] = error; return result;
|
||||
}
|
||||
if (!connect()) { result["status"] = "unavailable"; result["reason"] = error; return result; }
|
||||
result["global_memory_service_ns"] = static_cast<int64_t>(memory.serviceNs);
|
||||
result["global_read_bytes"] = static_cast<int64_t>(memory.readBytes);
|
||||
result["global_write_bytes"] = static_cast<int64_t>(memory.writeBytes);
|
||||
result["global_read_requests"] = static_cast<int64_t>(memory.readRequests);
|
||||
result["global_write_requests"] = static_cast<int64_t>(memory.writeRequests);
|
||||
if (!options.detailed) {
|
||||
result["status"] = "estimated";
|
||||
result["memory_controller"] = queueReport();
|
||||
return result;
|
||||
}
|
||||
auto estimate = analyzePeriodicDependencies(events.size(), edges, error);
|
||||
if (mlir::failed(estimate)) { result["status"] = "unavailable"; result["reason"] = error; return result; }
|
||||
result["status"] = "estimated";
|
||||
result["recurrence_lower_ns"] = estimate->recurrenceLowerNs;
|
||||
result["recurrence_upper_ns"] = estimate->recurrenceUpperNs;
|
||||
result["within_generation_span_ns"] = static_cast<int64_t>(estimate->withinGenerationSpanNs);
|
||||
result["interval_estimate_ns"] = std::max(estimate->recurrenceUpperNs, static_cast<double>(memory.serviceNs));
|
||||
result["limiting_bound"] = memory.serviceNs > estimate->recurrenceUpperNs ? "global-memory" : "recurrence";
|
||||
result["events"] = static_cast<int64_t>(events.size());
|
||||
result["edges"] = static_cast<int64_t>(edges.size());
|
||||
// Keep small timing fixtures inspectable without dumping network-size graphs.
|
||||
if (events.size() <= 128) {
|
||||
llvm::json::Array nodes, dependencies;
|
||||
for (const auto &node : events)
|
||||
nodes.push_back(llvm::json::Object{{"core", node.core}, {"pc", node.pc}, {"kind", node.kind}});
|
||||
for (const auto &edge : edges)
|
||||
dependencies.push_back(llvm::json::Object{{"source", static_cast<int64_t>(edge.source)},
|
||||
{"target", static_cast<int64_t>(edge.target)}, {"delay_ns", static_cast<int64_t>(edge.delayNs)},
|
||||
{"distance", edge.distance}});
|
||||
result["timing_graph"] = llvm::json::Object{{"events", std::move(nodes)}, {"dependencies", std::move(dependencies)}};
|
||||
}
|
||||
result["network_queues"] = queues; result["instruction_issue"] = issue;
|
||||
if (memoryQueue) result["memory_controller"] = queueReport();
|
||||
result["resource_service_columns"] = llvm::json::Array{"matrix", "vector", "local_transfer",
|
||||
"global_load", "global_store", "scalar_issue", "synchronization"};
|
||||
Cost directBytes = 0, directRequests = 0;
|
||||
for (const auto &[key, transfers] : sends)
|
||||
for (const auto &transfer : transfers) { directBytes += transfer.bytes; ++directRequests; }
|
||||
result["direct_requests"] = static_cast<int64_t>(directRequests);
|
||||
result["direct_bytes"] = static_cast<int64_t>(directBytes);
|
||||
llvm::json::Object opcodeDemand, instructionCounts;
|
||||
for (size_t op = 0; op < opcodeCounts.size(); ++op)
|
||||
if (opcodeCounts[op]) {
|
||||
opcodeDemand[pim_binary::kOpcodeNames[op]] = static_cast<int64_t>(opcodeService[static_cast<unsigned>(op)]);
|
||||
instructionCounts[pim_binary::kOpcodeNames[op]] = static_cast<int64_t>(opcodeCounts[op]);
|
||||
}
|
||||
result["opcode_service_ns"] = std::move(opcodeDemand);
|
||||
result["instruction_counts"] = std::move(instructionCounts);
|
||||
llvm::json::Array perCore, cycle;
|
||||
Cost maximumCoreWork = 0;
|
||||
for (const auto &[core, state] : cores) {
|
||||
maximumCoreWork = std::max(maximumCoreWork, state.work);
|
||||
llvm::json::Array resource;
|
||||
for (Cost value : state.service) resource.push_back(static_cast<int64_t>(value));
|
||||
perCore.push_back(llvm::json::Object{{"core", core}, {"instructions", static_cast<int64_t>(state.instructions)},
|
||||
{"serial_work_ns", static_cast<int64_t>(state.work)}, {"resource_service_ns", std::move(resource)}});
|
||||
}
|
||||
std::map<int64_t, Cost> cycleCores;
|
||||
Cost cycleDelay = 0, cycleDistance = 0;
|
||||
for (size_t index : estimate->criticalCycle) {
|
||||
const auto &edge = edges[index];
|
||||
const auto &from = events[edge.source], &to = events[edge.target];
|
||||
cycleCores[to.core] += edge.delayNs;
|
||||
cycleDelay += edge.delayNs; cycleDistance += edge.distance;
|
||||
if (cycle.size() < 128)
|
||||
cycle.push_back(llvm::json::Object{{"source_core", from.core}, {"target_core", to.core},
|
||||
{"source_pc", from.pc}, {"target_pc", to.pc}, {"kind", to.kind},
|
||||
{"delay_ns", static_cast<int64_t>(edge.delayNs)}, {"distance", edge.distance}});
|
||||
}
|
||||
llvm::json::Array critical;
|
||||
for (auto [core, delay] : cycleCores)
|
||||
critical.push_back(llvm::json::Object{{"core", core}, {"delay_ns", static_cast<int64_t>(delay)}});
|
||||
result["critical_cycle"] = std::move(cycle); result["critical_cores"] = std::move(critical);
|
||||
result["critical_cycle_edges"] = static_cast<int64_t>(estimate->criticalCycle.size());
|
||||
result["critical_cycle_delay_ns"] = static_cast<int64_t>(cycleDelay);
|
||||
result["critical_cycle_distance"] = static_cast<int64_t>(cycleDistance);
|
||||
result["per_core"] = std::move(perCore);
|
||||
result["maximum_core_serial_work_ns"] = static_cast<int64_t>(maximumCoreWork);
|
||||
result["omitted_effects"] = llvm::json::Array{"global-memory arbitration", "clock-phase rounding and frontend buffering",
|
||||
"scalar issue overlap (charged serially)", "initial inference bootstrap"};
|
||||
return result;
|
||||
}
|
||||
|
||||
llvm::json::Object Analysis::queueReport() {
|
||||
auto queued = estimateQueuedPeriod(events, edges, memoryServices, cores, false);
|
||||
auto alternative = estimateQueuedPeriod(events, edges, memoryServices, cores, true);
|
||||
queued["alternative_tie_status"] = alternative.getString("status").value();
|
||||
if (auto reason = alternative.getString("reason"))
|
||||
queued["alternative_tie_reason"] = reason->str();
|
||||
if (auto interval = alternative.getNumber("interval_ns"))
|
||||
queued["alternative_tie_interval_ns"] = *interval;
|
||||
if (auto interval = queued.getNumber("interval_ns"))
|
||||
queued["memory_service_fraction"] = memory.serviceNs / *interval;
|
||||
queued["arbitration"] = "FIFO by predicted controller arrival";
|
||||
return queued;
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
llvm::json::Object onnx_mlir::analyzePimInstructions(const PimInstructionProgram &program,
|
||||
const PimInstructionTiming &target, int64_t cellBits, PimPeriodicOptions options) {
|
||||
Analysis analysis;
|
||||
analysis.target = target;
|
||||
analysis.cellBits = cellBits;
|
||||
analysis.options = options;
|
||||
analysis.queues = options.networkQueues;
|
||||
analysis.issue = options.instructionIssue;
|
||||
analysis.memoryQueue = options.memoryController;
|
||||
auto unavailable = [](llvm::StringRef reason) {
|
||||
return llvm::json::Object{{"status", "unavailable"}, {"reason", reason.str()}};
|
||||
};
|
||||
if (!target.hasPeriodicTransferTiming() || cellBits <= 0)
|
||||
return unavailable("target has no supported instruction timing contract");
|
||||
if (program.instructions.empty() || program.instructions.size() > target.processorCount
|
||||
|| program.instructions.size() != program.arrayGroups.size())
|
||||
return unavailable("instruction program core count or array groups do not fit the target");
|
||||
for (size_t id = 0; id < program.instructions.size(); ++id) {
|
||||
llvm::json::Array groups;
|
||||
for (int64_t count : program.arrayGroups[id]) groups.push_back(count);
|
||||
if (!analysis.readCore(id, program.instructions[id], groups))
|
||||
return unavailable(analysis.error);
|
||||
}
|
||||
return analysis.report();
|
||||
}
|
||||
|
||||
void onnx_mlir::configurePimScheduleCosts(spatial::SchedulingTarget &target,
|
||||
PimInstructionTiming timing) {
|
||||
auto hardware = std::make_shared<const PimInstructionTiming>(std::move(timing));
|
||||
target.reportPeriodicSchedule = [hardware](const spatial::ComputeGraph &graph,
|
||||
const spatial::ScheduledComputeMaterializationResult &materialization,
|
||||
const spatial::DeferredTransferPlan &transfers,
|
||||
const spatial::DeferredBoundaryPlan &boundaries) {
|
||||
dumpPeriodicScheduleEstimate(graph, materialization, transfers, boundaries, *hardware);
|
||||
};
|
||||
auto contract = getPimPeriodicTimingContract();
|
||||
if (!contract) target.wholeScheduleCostUnavailable = llvm::toString(contract.takeError());
|
||||
if (!hardware->hasPeriodicTransferTiming())
|
||||
target.wholeScheduleCostUnavailable = "target has no supported instruction timing contract";
|
||||
if (pimDisableSynchronization)
|
||||
target.wholeScheduleCostUnavailable = "synchronization ablation has no periodic queue timing contract";
|
||||
if (!target.wholeScheduleCostUnavailable.empty()) return;
|
||||
target.prepareScheduleCost = [hardware, cellBits = *contract](mlir::ModuleOp module,
|
||||
llvm::json::Object &summary) -> mlir::FailureOr<spatial::ScheduleCostQuery> {
|
||||
auto start = std::chrono::steady_clock::now();
|
||||
PimInstructionProgram program;
|
||||
PimCodegenTiming timing;
|
||||
std::string unused;
|
||||
if (compileToPimCode(module, unused, &program, &timing) != CompilerSuccess)
|
||||
return mlir::failure();
|
||||
summary["codegen_preparation_seconds"] = timing.preparationSeconds;
|
||||
summary["codegen_emission_seconds"] = timing.emissionSeconds;
|
||||
summary["instruction_generation_seconds"] = std::chrono::duration<double>(
|
||||
std::chrono::steady_clock::now() - start).count();
|
||||
llvm::SHA256 hash;
|
||||
int64_t instructionBytes = 0;
|
||||
for (size_t core = 0; core < program.instructions.size(); ++core) {
|
||||
instructionBytes += program.instructions[core].size();
|
||||
hash.update(program.instructions[core]);
|
||||
for (int64_t group : program.arrayGroups[core]) hash.update(std::to_string(group) + ",");
|
||||
hash.update(";");
|
||||
}
|
||||
summary["instruction_bytes"] = instructionBytes;
|
||||
summary["instruction_count"] = (instructionBytes
|
||||
- pim_binary::kHeaderSize * program.instructions.size()) / pim_binary::kRecordSize;
|
||||
spatial::ScheduleCostQuery query;
|
||||
query.identity = llvm::toHex(hash.final());
|
||||
query.evaluate = [program = std::move(program), hardware, cellBits](llvm::json::Object &summary) {
|
||||
PimPeriodicOptions options;
|
||||
options.oraclePrecision = true;
|
||||
auto report = analyzePimInstructions(program, *hardware, cellBits, options);
|
||||
for (const char *field : {"global_memory_service_ns", "global_read_bytes", "global_write_bytes",
|
||||
"global_read_requests", "global_write_requests"})
|
||||
if (auto value = report.getInteger(field)) summary[field] = *value;
|
||||
spatial::WholeScheduleCost cost;
|
||||
// Conservative selection margin; not a timing correction or formal bound.
|
||||
cost.relativeUncertainty = 0.021;
|
||||
if (auto *queue = report.getObject("memory_controller")) {
|
||||
if (queue->getString("status") == "estimated"
|
||||
&& queue->getString("alternative_tie_status") == "estimated") {
|
||||
cost.primary = queue->getNumber("interval_ns");
|
||||
cost.alternative = queue->getNumber("alternative_tie_interval_ns");
|
||||
}
|
||||
if (auto reason = queue->getString("reason")) cost.reason = reason->str();
|
||||
if (!cost.eligible() && cost.reason.empty())
|
||||
if (auto reason = queue->getString("alternative_tie_reason"))
|
||||
cost.reason = "alternative arbitration: " + reason->str();
|
||||
} else if (auto reason = report.getString("reason")) cost.reason = reason->str();
|
||||
if (!cost.eligible() && cost.reason.empty())
|
||||
cost.reason = "arbitration sensitivity estimate unavailable or nonconverged";
|
||||
return cost;
|
||||
};
|
||||
return query;
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
#pragma once
|
||||
|
||||
#include "llvm/Support/JSON.h"
|
||||
#include <cstdint>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
namespace onnx_mlir {
|
||||
namespace spatial { struct SchedulingTarget; }
|
||||
struct PimInstructionTiming;
|
||||
void configurePimScheduleCosts(spatial::SchedulingTarget &target, PimInstructionTiming timing);
|
||||
|
||||
// Binary instructions, indexed by physical core. No executable or measured data.
|
||||
struct PimInstructionProgram {
|
||||
std::vector<std::string> instructions;
|
||||
std::vector<std::vector<int64_t>> arrayGroups;
|
||||
};
|
||||
|
||||
struct PimPeriodicOptions {
|
||||
bool networkQueues = true;
|
||||
bool instructionIssue = true;
|
||||
bool memoryController = true;
|
||||
bool detailed = false;
|
||||
// Apply the oracle's fp32-to-int8 timing projection without changing the program.
|
||||
bool oraclePrecision = false;
|
||||
};
|
||||
|
||||
llvm::json::Object analyzePimInstructions(const PimInstructionProgram &program,
|
||||
const PimInstructionTiming &target, int64_t cellBits,
|
||||
PimPeriodicOptions options = {});
|
||||
} // namespace onnx_mlir
|
||||
@@ -0,0 +1,76 @@
|
||||
#include "PimPeriodicAnalysis.hpp"
|
||||
#include "PimInstructionTiming.hpp"
|
||||
#include "PimCompilerOptions.hpp"
|
||||
#include "PimCompilerUtils.hpp"
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/SchedulingTarget.hpp"
|
||||
#include "llvm/Support/CommandLine.h"
|
||||
#include "llvm/Support/FormatVariadic.h"
|
||||
#include "llvm/Support/MemoryBuffer.h"
|
||||
#include <chrono>
|
||||
|
||||
using namespace onnx_mlir;
|
||||
namespace {
|
||||
llvm::cl::opt<std::string> artifact(llvm::cl::Positional, llvm::cl::Required,
|
||||
llvm::cl::desc("PIM binary artifact directory (native or oracle-exported)"));
|
||||
llvm::cl::opt<bool> queues("model-network-queues", llvm::cl::init(true));
|
||||
llvm::cl::opt<bool> issue("model-issue", llvm::cl::init(true));
|
||||
llvm::cl::opt<bool> oraclePrecision("oracle-precision", llvm::cl::init(false),
|
||||
llvm::cl::desc("Project native fp32 instructions to oracle int8 timing"));
|
||||
llvm::cl::opt<bool> memoryQueue("model-memory-controller", llvm::cl::init(false));
|
||||
llvm::Expected<llvm::json::Object> readJson(llvm::StringRef path) {
|
||||
auto buffer = llvm::MemoryBuffer::getFile(path);
|
||||
if (!buffer) return llvm::errorCodeToError(buffer.getError());
|
||||
auto parsed = llvm::json::parse(buffer.get()->getBuffer());
|
||||
if (!parsed) return parsed.takeError();
|
||||
if (!parsed->getAsObject()) return llvm::createStringError("expected a JSON object");
|
||||
return std::move(*parsed->getAsObject());
|
||||
}
|
||||
} // namespace
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
llvm::cl::ParseCommandLineOptions(argc, argv, "Post-codegen periodic PIM diagnostics\n");
|
||||
auto start = std::chrono::steady_clock::now();
|
||||
auto program = readJson(artifact + "/config.json");
|
||||
auto cellBits = getPimPeriodicTimingContract();
|
||||
if (!program || !cellBits) {
|
||||
llvm::errs() << (!program ? llvm::toString(program.takeError()) : llvm::toString(cellBits.takeError())) << '\n';
|
||||
return 1;
|
||||
}
|
||||
auto target = getPimInstructionTiming();
|
||||
PimInstructionProgram instructions;
|
||||
auto *groups = program->getObject("array_group_map");
|
||||
if (!groups) { llvm::errs() << "missing physical array groups\n"; return 1; }
|
||||
auto coreCount = program->getInteger("core_cnt");
|
||||
if (!coreCount || *coreCount <= 0 || *coreCount > static_cast<int64_t>(target.processorCount)) {
|
||||
llvm::errs() << "artifact core count does not fit the target\n"; return 1;
|
||||
}
|
||||
for (int64_t id = 0; id < *coreCount; ++id) {
|
||||
auto path = artifact + "/core_" + std::to_string(id) + ".pim";
|
||||
auto *array = groups->getArray("core" + std::to_string(id));
|
||||
if (!array) { llvm::errs() << "missing physical core array groups\n"; return 1; }
|
||||
auto buffer = llvm::MemoryBuffer::getFile(path);
|
||||
if (!buffer) { llvm::errs() << "cannot read instruction artifact\n"; return 1; }
|
||||
instructions.instructions.push_back(buffer.get()->getBuffer().str());
|
||||
std::vector<int64_t> counts;
|
||||
for (const auto &value : *array) {
|
||||
auto count = value.getAsInteger();
|
||||
if (!count) { llvm::errs() << "invalid array group\n"; return 1; }
|
||||
counts.push_back(*count);
|
||||
}
|
||||
instructions.arrayGroups.push_back(std::move(counts));
|
||||
}
|
||||
PimPeriodicOptions options;
|
||||
options.detailed = true;
|
||||
options.oraclePrecision = oraclePrecision;
|
||||
options.networkQueues = queues;
|
||||
options.instructionIssue = issue;
|
||||
options.memoryController = memoryQueue;
|
||||
auto report = analyzePimInstructions(instructions, target, *cellBits, options);
|
||||
report["diagnostic_only"] = true;
|
||||
report["selection_eligible"] = false;
|
||||
bool available = report.getString("status") == "estimated";
|
||||
report["analysis_wall_ms"] = std::chrono::duration<double, std::milli>(
|
||||
std::chrono::steady_clock::now() - start).count();
|
||||
llvm::outs() << llvm::formatv("{0:2}\n", llvm::json::Value(std::move(report)));
|
||||
return available ? 0 : 1;
|
||||
}
|
||||
@@ -0,0 +1,225 @@
|
||||
#include "PimPipelineSelection.hpp"
|
||||
#include "PimCompilerOptions.hpp"
|
||||
#include "PimCompilerUtils.hpp"
|
||||
#include "src/Accelerators/PIM/Common/Support/DebugDump.hpp"
|
||||
#include "src/Accelerators/PIM/Common/Support/FileSystemUtils.hpp"
|
||||
#include "src/Accelerators/PIM/Common/Support/ReportUtils.hpp"
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/ScheduledSpatialPasses.hpp"
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/SpatialTargetResources.hpp"
|
||||
#include "mlir/Pass/PassManager.h"
|
||||
#include "mlir/Pass/PassInstrumentation.h"
|
||||
#include "src/Compiler/CompilerOptions.hpp"
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ScheduleEvaluation.hpp"
|
||||
#include "llvm/Support/FormatVariadic.h"
|
||||
#include "llvm/Support/JSON.h"
|
||||
#include "llvm/Support/raw_os_ostream.h"
|
||||
#include <algorithm>
|
||||
#include <chrono>
|
||||
|
||||
using namespace mlir;
|
||||
using namespace onnx_mlir;
|
||||
using namespace onnx_mlir::spatial;
|
||||
|
||||
namespace {
|
||||
using Clock = std::chrono::steady_clock;
|
||||
double elapsed(Clock::time_point start) {
|
||||
return std::chrono::duration<double>(Clock::now() - start).count();
|
||||
}
|
||||
|
||||
llvm::json::Object countIR(Operation *root) {
|
||||
int64_t operations = 0, values = 0;
|
||||
root->walk([&](Operation *op) {
|
||||
++operations;
|
||||
values += op->getNumResults();
|
||||
for (Region ®ion : op->getRegions())
|
||||
for (Block &block : region) values += block.getNumArguments();
|
||||
});
|
||||
return llvm::json::Object{{"operations", operations}, {"values", values}};
|
||||
}
|
||||
|
||||
// Enabled with the existing compiler timing option. No instruction expansion.
|
||||
struct CandidateTiming final : PassInstrumentation {
|
||||
llvm::json::Object &summary;
|
||||
Clock::time_point start;
|
||||
explicit CandidateTiming(llvm::json::Object &summary) : summary(summary) {}
|
||||
void runBeforePass(Pass *pass, Operation *op) override {
|
||||
if (pass->getArgument() == "pim-one-shot-bufferization")
|
||||
summary["bufferization_input_ir"] = countIR(op);
|
||||
start = Clock::now();
|
||||
}
|
||||
void runAfterPass(Pass *pass, Operation *) override {
|
||||
auto *phases = summary.getObject("lowering_phases_seconds");
|
||||
if (!phases) {
|
||||
summary["lowering_phases_seconds"] = llvm::json::Object{};
|
||||
phases = summary.getObject("lowering_phases_seconds");
|
||||
}
|
||||
(*phases)[pass->getName()] = elapsed(start);
|
||||
}
|
||||
};
|
||||
|
||||
struct PimPipelineSelectionPass final : PassWrapper<PimPipelineSelectionPass, OperationPass<ModuleOp>> {
|
||||
MLIR_DEFINE_EXPLICIT_INTERNAL_INLINE_TYPE_ID(PimPipelineSelectionPass)
|
||||
PimPipelineSelectionPass(const SchedulingTarget &target, const SpatialTargetResources &resources,
|
||||
size_t stages, SpatialDataflowExportStage exportStage)
|
||||
: target(target), resources(resources), stages(stages), exportStage(exportStage) {}
|
||||
SchedulingTarget target;
|
||||
SpatialTargetResources resources;
|
||||
size_t stages;
|
||||
SpatialDataflowExportStage exportStage;
|
||||
|
||||
StringRef getArgument() const override { return "pim-pipeline-selection"; }
|
||||
StringRef getDescription() const override { return "Select a throughput pipeline using static instruction queue analysis"; }
|
||||
|
||||
LogicalResult lower(ModuleOp module, llvm::json::Object *summary = nullptr) {
|
||||
PassManager pm(module.getContext());
|
||||
addPimLoweringPasses(pm, resources, EmitPimCodegen, /*reportProgress=*/false);
|
||||
if (enableTiming && summary)
|
||||
pm.addInstrumentation(std::make_unique<CandidateTiming>(*summary));
|
||||
return pm.run(module);
|
||||
}
|
||||
|
||||
LogicalResult evaluate(ModuleOp checkpoint, const MergeScheduleResult &schedule,
|
||||
OwningOpRef<ModuleOp> &candidate,
|
||||
std::string &scheduleReport, llvm::json::Object &summary) {
|
||||
auto begin = Clock::now();
|
||||
IRMapping mapping;
|
||||
candidate = cast<ModuleOp>(checkpoint->clone(mapping));
|
||||
auto remapped = remapSpatialSchedule(schedule, mapping);
|
||||
if (failed(realizeSpatialSchedule(*candidate, remapped, target, stages,
|
||||
SpatialDataflowExportStage::None, nullptr, &scheduleReport))
|
||||
|| failed(lower(*candidate, &summary))) return failure();
|
||||
if (enableTiming) summary["codegen_input_ir"] = countIR(candidate->getOperation());
|
||||
summary["lowering_seconds"] = elapsed(begin);
|
||||
return success();
|
||||
}
|
||||
|
||||
void runOnOperation() override {
|
||||
ModuleOp module = getOperation();
|
||||
auto start = Clock::now();
|
||||
std::string unsupported;
|
||||
if (!target.prepareScheduleCost)
|
||||
unsupported = target.wholeScheduleCostUnavailable.empty()
|
||||
? "target has no whole-schedule cost query" : target.wholeScheduleCostUnavailable;
|
||||
auto prepared = prepareSpatialSchedules(module, target, stages, unsupported.empty());
|
||||
if (failed(prepared)) { signalPassFailure(); return; }
|
||||
bool singleCandidate = unsupported.empty() && prepared->candidates.size() == 1;
|
||||
double generationSeconds = elapsed(start);
|
||||
std::vector<WholeScheduleCost> scores(prepared->candidates.size());
|
||||
llvm::json::Array summaries;
|
||||
OwningOpRef<ModuleOp> bestModule;
|
||||
std::string bestScheduleReport;
|
||||
size_t bestIndex = 0;
|
||||
double loweringSeconds = 0, instructionSeconds = 0, queueSeconds = 0;
|
||||
size_t lowered = 0, analyzed = 0, unavailable = 0, duplicates = 0;
|
||||
for (size_t i = 0; i < prepared->candidates.size(); ++i) {
|
||||
llvm::json::Object summary{{"index", static_cast<int64_t>(i)}};
|
||||
if (!unsupported.empty()) {
|
||||
scores[i].reason = unsupported;
|
||||
} else if (singleCandidate) {
|
||||
scores[i].reason = "one distinct generated schedule; no cost comparison required";
|
||||
} else {
|
||||
ScopedCompilerOutputSuppression silence;
|
||||
OwningOpRef<ModuleOp> candidate;
|
||||
std::string scheduleReport;
|
||||
if (failed(evaluate(module, prepared->candidates[i], candidate, scheduleReport, summary))) {
|
||||
if (!i) { signalPassFailure(); return; }
|
||||
scores[i].reason = "candidate lowering failed verification";
|
||||
} else {
|
||||
++lowered;
|
||||
loweringSeconds += *summary.getNumber("lowering_seconds");
|
||||
auto query = target.prepareScheduleCost(*candidate, summary);
|
||||
if (failed(query)) {
|
||||
if (!i) { signalPassFailure(); return; }
|
||||
scores[i].reason = "candidate cost preparation failed verification";
|
||||
++unavailable;
|
||||
summary["eligible"] = false;
|
||||
summary["reason"] = scores[i].reason;
|
||||
summaries.push_back(std::move(summary));
|
||||
continue;
|
||||
}
|
||||
instructionSeconds += summary.getNumber("instruction_generation_seconds").value_or(0);
|
||||
scores[i].identity = query->identity;
|
||||
auto duplicate = std::find_if(scores.begin(), scores.begin() + i,
|
||||
[&](const WholeScheduleCost &other) { return other.identity == scores[i].identity; });
|
||||
if (duplicate != scores.begin() + i) {
|
||||
++duplicates;
|
||||
scores[i].reason = "duplicate emitted instruction program";
|
||||
summary["duplicate_of"] = static_cast<int64_t>(duplicate - scores.begin());
|
||||
} else {
|
||||
auto begin = Clock::now();
|
||||
scores[i] = query->evaluate(summary);
|
||||
scores[i].identity = query->identity;
|
||||
double seconds = elapsed(begin);
|
||||
queueSeconds += seconds;
|
||||
++analyzed;
|
||||
summary["queue_seconds"] = seconds;
|
||||
}
|
||||
std::string partialReason;
|
||||
size_t partialSelection = selectPipelineCandidate(
|
||||
llvm::ArrayRef<WholeScheduleCost>(scores.data(), i + 1), partialReason);
|
||||
if (!bestModule || partialSelection == i) {
|
||||
bestIndex = i;
|
||||
bestModule = std::move(candidate);
|
||||
bestScheduleReport = std::move(scheduleReport);
|
||||
}
|
||||
}
|
||||
}
|
||||
if (!singleCandidate && !scores[i].eligible()
|
||||
&& scores[i].reason != "duplicate emitted instruction program") ++unavailable;
|
||||
summary["eligible"] = scores[i].eligible();
|
||||
summary["identity"] = scores[i].identity;
|
||||
if (scores[i].primary) summary["interval_ns"] = *scores[i].primary;
|
||||
if (scores[i].alternative) summary["alternative_tie_interval_ns"] = *scores[i].alternative;
|
||||
if (scores[i].relativeUncertainty)
|
||||
summary["relative_prediction_uncertainty"] = *scores[i].relativeUncertainty;
|
||||
if (!scores[i].reason.empty()) summary["reason"] = scores[i].reason;
|
||||
summaries.push_back(std::move(summary));
|
||||
}
|
||||
std::string reason;
|
||||
size_t selected = selectPipelineCandidate(scores, reason);
|
||||
if (singleCandidate) reason = scores.front().reason;
|
||||
else if (!unsupported.empty()) reason += ": " + unsupported;
|
||||
else if (!scores.front().eligible() && !scores.front().reason.empty())
|
||||
reason += ": " + scores.front().reason;
|
||||
auto finalizeStart = Clock::now();
|
||||
bool spatialArtifacts = exportStage != SpatialDataflowExportStage::None
|
||||
|| pimPeriodicDiagnostics || isPimIrDumpEnabled();
|
||||
// Explicit Spatial diagnostics require the pre-lowering representation.
|
||||
if (!bestModule || spatialArtifacts) {
|
||||
if (failed(realizeSpatialSchedule(module, prepared->candidates[selected], target,
|
||||
stages, exportStage, pimPeriodicDiagnostics ? &prepared->graph : nullptr))) { signalPassFailure(); return; }
|
||||
} else {
|
||||
auto file = openDialectDumpFileWithExtension("spatial3_scheduled_no_comm", "/reports", "txt");
|
||||
if (file) file << bestScheduleReport;
|
||||
}
|
||||
if (bestModule) {
|
||||
assert(selected == bestIndex && "retained module must be the selected candidate");
|
||||
module->setAttrs((*bestModule)->getAttrs());
|
||||
module.getBodyRegion().takeBody(bestModule->getBodyRegion());
|
||||
} else if (failed(lower(module))) { signalPassFailure(); return; }
|
||||
llvm::errs() << "Queue-aware pipeline: candidate " << selected << " of " << scores.size()
|
||||
<< "; " << reason << "\n";
|
||||
llvm::json::Object report{{"selection", static_cast<int64_t>(selected)}, {"reason", reason},
|
||||
{"static_analysis", true}, {"strategy", "queue-aware"}, {"candidates", std::move(summaries)},
|
||||
{"single_candidate_analysis_skipped", singleCandidate},
|
||||
{"candidate_generation_seconds", generationSeconds}, {"candidate_lowering_seconds", loweringSeconds},
|
||||
{"instruction_generation_seconds", instructionSeconds},
|
||||
{"queue_seconds", queueSeconds}, {"finalization_seconds", elapsed(finalizeStart)}, {"total_seconds", elapsed(start)},
|
||||
{"generated_candidates", static_cast<int64_t>(scores.size())},
|
||||
{"lowered_candidates", static_cast<int64_t>(lowered)},
|
||||
{"reused_lowered_candidate", static_cast<bool>(bestModule)},
|
||||
{"reconstructed_spatial_artifacts", spatialArtifacts && static_cast<bool>(bestModule)},
|
||||
{"analyzed_candidates", static_cast<int64_t>(analyzed)},
|
||||
{"unavailable_estimates", static_cast<int64_t>(unavailable)},
|
||||
{"duplicate_programs", static_cast<int64_t>(duplicates)}, {"temporary_disk_bytes", 0}};
|
||||
auto file = openReportFileWithExtension("pipeline_selection", "json");
|
||||
if (file) { llvm::raw_os_ostream out(file); out << llvm::formatv("{0:2}\n", llvm::json::Value(std::move(report))); }
|
||||
}
|
||||
};
|
||||
} // namespace
|
||||
|
||||
std::unique_ptr<Pass> onnx_mlir::createPimPipelineSelectionPass(
|
||||
const SchedulingTarget &target, const SpatialTargetResources &resources,
|
||||
size_t stages, SpatialDataflowExportStage exportStage) {
|
||||
return std::make_unique<PimPipelineSelectionPass>(target, resources, stages, exportStage);
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
#pragma once
|
||||
#include "mlir/Pass/Pass.h"
|
||||
|
||||
namespace onnx_mlir {
|
||||
namespace spatial { struct SchedulingTarget; struct SpatialTargetResources; enum class SpatialDataflowExportStage; }
|
||||
std::unique_ptr<mlir::Pass> createPimPipelineSelectionPass(
|
||||
const spatial::SchedulingTarget &target, const spatial::SpatialTargetResources &resources,
|
||||
size_t stages, spatial::SpatialDataflowExportStage exportStage);
|
||||
} // namespace onnx_mlir
|
||||
@@ -0,0 +1,467 @@
|
||||
#include "PimScheduleDiagnostic.hpp"
|
||||
#include "PimInstructionTiming.hpp"
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/DeferredTransferPlanning.hpp"
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PeriodicDependencies.hpp"
|
||||
#include "src/Accelerators/PIM/Common/Support/DebugDump.hpp"
|
||||
|
||||
#include "llvm/ADT/MapVector.h"
|
||||
#include "llvm/Support/JSON.h"
|
||||
#include "llvm/Support/raw_os_ostream.h"
|
||||
|
||||
#include <chrono>
|
||||
#include <map>
|
||||
|
||||
namespace onnx_mlir {
|
||||
using namespace spatial;
|
||||
namespace {
|
||||
|
||||
struct Event {
|
||||
int64_t core;
|
||||
llvm::StringRef kind;
|
||||
int64_t identity;
|
||||
};
|
||||
struct DirectEndpoint {
|
||||
size_t begin;
|
||||
size_t end;
|
||||
int64_t core;
|
||||
Cost bytes;
|
||||
};
|
||||
struct Signal {
|
||||
size_t event;
|
||||
int64_t core;
|
||||
};
|
||||
struct Wait {
|
||||
size_t event;
|
||||
int64_t count;
|
||||
};
|
||||
using Register = std::pair<int64_t, unsigned>;
|
||||
|
||||
class PeriodicPlan {
|
||||
public:
|
||||
PeriodicPlan(const PimInstructionTiming &target,
|
||||
const DeferredTransferPlan &transfers)
|
||||
: target(target), transfers(transfers), computeWork(target.processorCount),
|
||||
entry(target.processorCount), tail(target.processorCount),
|
||||
active(target.processorCount) {}
|
||||
|
||||
bool build(const ComputeGraph &graph,
|
||||
const ScheduledComputeMaterializationResult &materialization,
|
||||
const DeferredBoundaryPlan &boundaries);
|
||||
|
||||
const PimInstructionTiming ⌖
|
||||
const DeferredTransferPlan &transfers;
|
||||
std::vector<Event> events;
|
||||
std::vector<PeriodicDependency> edges;
|
||||
SharedMemoryDemand boundaryMemory;
|
||||
std::vector<Cost> computeWork;
|
||||
std::string error;
|
||||
void recordSynchronization(llvm::json::Object &report) const;
|
||||
|
||||
private:
|
||||
size_t event(int64_t core, llvm::StringRef kind, int64_t identity = -1) {
|
||||
events.push_back({core, kind, identity});
|
||||
return events.size() - 1;
|
||||
}
|
||||
size_t append(int64_t core, llvm::StringRef kind, Cost delay = 0,
|
||||
int64_t identity = -1) {
|
||||
size_t next = event(core, kind, identity);
|
||||
edges.push_back({tail[core], next, delay, 0});
|
||||
tail[core] = next;
|
||||
return next;
|
||||
}
|
||||
void signal(int64_t source, int64_t targetCore, unsigned reg,
|
||||
llvm::StringRef kind) {
|
||||
size_t begin = append(source, kind, 0, targetCore);
|
||||
signals[{targetCore, reg}].push_back({begin, source});
|
||||
append(source, "signal-complete", target.signalIssueNs);
|
||||
}
|
||||
void wait(int64_t core, unsigned reg, int64_t count, llvm::StringRef kind) {
|
||||
if (!count)
|
||||
return;
|
||||
size_t next = append(core, kind, 0, reg);
|
||||
if (!waits.try_emplace({core, reg}, Wait{next, count}).second)
|
||||
error = "multiple waits consume one event register in a generation";
|
||||
}
|
||||
void entrySynchronization();
|
||||
void completionSynchronization();
|
||||
bool boundary(const BoundaryProgram &program);
|
||||
bool transfer(const ScheduledTransferSlice &slice, bool send);
|
||||
bool connect();
|
||||
|
||||
std::vector<size_t> entry, tail;
|
||||
std::vector<bool> active;
|
||||
std::map<Register, std::vector<Signal>> signals;
|
||||
std::map<Register, Wait> waits;
|
||||
llvm::MapVector<int64_t, DirectEndpoint> sends, receives;
|
||||
};
|
||||
|
||||
void PeriodicPlan::entrySynchronization() {
|
||||
for (const ScheduledInfo &scheduled : transfers.scheduled)
|
||||
for (int64_t core : scheduled.cores) {
|
||||
auto found = transfers.hostReleaseSignals.find(core);
|
||||
if (found != transfers.hostReleaseSignals.end())
|
||||
for (HostReleaseSignal release : found->second)
|
||||
signal(core, release.writerCore, release.eventRegister, "free-signal");
|
||||
}
|
||||
unsigned rounds = 0;
|
||||
for (size_t distance = 1; distance < transfers.stageZeroCores.size(); distance *= 2)
|
||||
++rounds;
|
||||
unsigned last = transfers.synchronizationRegisterCount - 1;
|
||||
for (auto [rank, core] : llvm::enumerate(transfers.downstreamCores)) {
|
||||
int64_t parent = rank ? transfers.downstreamCores[(rank - 1) / 2]
|
||||
: transfers.stageZeroCores.front();
|
||||
signal(core, parent, rank ? last - 1 : last - rounds, "restart-permission");
|
||||
}
|
||||
}
|
||||
|
||||
void PeriodicPlan::completionSynchronization() {
|
||||
unsigned rounds = 0;
|
||||
for (size_t distance = 1; distance < transfers.stageZeroCores.size(); distance *= 2)
|
||||
++rounds;
|
||||
unsigned last = transfers.synchronizationRegisterCount - 1;
|
||||
for (auto [rank, core] : llvm::enumerate(transfers.stageZeroCores))
|
||||
for (unsigned round = 0; round < rounds; ++round) {
|
||||
int64_t peer = transfers.stageZeroCores[
|
||||
(rank + (size_t{1} << round)) % transfers.stageZeroCores.size()];
|
||||
signal(core, peer, last - round, "barrier-signal");
|
||||
wait(core, last - round, 1, "barrier-wait");
|
||||
}
|
||||
if (!transfers.downstreamCores.empty()) {
|
||||
int64_t leader = transfers.stageZeroCores.front();
|
||||
wait(leader, last - rounds, 1, "restart-permission-wait");
|
||||
signal(leader, transfers.downstreamCores.front(), last, "restart-signal");
|
||||
}
|
||||
for (auto [rank, core] : llvm::enumerate(transfers.downstreamCores)) {
|
||||
wait(core, last, 1, "restart-wait");
|
||||
size_t first = 2 * rank + 1;
|
||||
size_t end = std::min(first + 2, transfers.downstreamCores.size());
|
||||
if (first < end)
|
||||
wait(core, last - 1, end - first, "restart-permission-wait");
|
||||
for (size_t child = first; child < end; ++child)
|
||||
signal(core, transfers.downstreamCores[child], last, "restart-signal");
|
||||
}
|
||||
}
|
||||
|
||||
bool PeriodicPlan::transfer(const ScheduledTransferSlice &slice, bool send) {
|
||||
const ExternalTransferFamily &family = *slice.family;
|
||||
auto type = mlir::dyn_cast<mlir::RankedTensorType>(family.requirement->publicationFragmentType);
|
||||
if (!type || !type.hasStaticShape() || !type.getElementType().isIntOrFloat()) {
|
||||
error = "transfer payload has no static byte size";
|
||||
return false;
|
||||
}
|
||||
Cost bits = std::min<Cost>(type.getElementType().getIntOrFloatBitWidth(),
|
||||
target.computeBitwidth);
|
||||
Cost payloadBits = checkedMultiply(static_cast<Cost>(type.getNumElements()), bits);
|
||||
Cost bytes = payloadBits / 8 + (payloadBits % 8 != 0);
|
||||
for (size_t offset = 0; offset < slice.transferCount; ++offset) {
|
||||
size_t index = slice.familyOffset + offset;
|
||||
int64_t source = family.sourceCores.valueAt(index);
|
||||
int64_t destination = family.targetCores.valueAt(index);
|
||||
int64_t core = send ? source : destination;
|
||||
int64_t channel = family.channelIds.valueAt(index);
|
||||
if (!family.hostRouted) {
|
||||
size_t begin = append(core, send ? "send-begin" : "receive-begin", 0, channel);
|
||||
size_t end = append(core, send ? "send-complete" : "receive-complete", 0, channel);
|
||||
if (!(send ? sends : receives).insert({channel, {begin, end, core, bytes}}).second) {
|
||||
error = "direct channel has multiple physical endpoints";
|
||||
return false;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
unsigned reg = family.eventRegisters.valueAt(index);
|
||||
if (send) {
|
||||
if (family.hostStoreFlags.valueAt(index)) {
|
||||
SharedMemoryDemand demand = target.getSharedMemoryDemand(bytes, 1, true);
|
||||
boundaryMemory += demand;
|
||||
Cost service = checkedAdd(target.getLocalMemoryServiceNs(bytes, false),
|
||||
checkedAdd(target.getSharedMemoryLinkServiceNs(bytes, core, false), demand.serviceNs));
|
||||
append(core, "host-store", service, family.hostOffsets.valueAt(index));
|
||||
}
|
||||
signal(core, destination, reg, "ready-signal");
|
||||
} else {
|
||||
wait(core, reg, family.waitValues.valueAt(index), "ready-wait");
|
||||
SharedMemoryDemand demand = target.getSharedMemoryDemand(bytes, 1, false);
|
||||
boundaryMemory += demand;
|
||||
Cost service = checkedAdd(target.getSharedMemoryLinkServiceNs(
|
||||
target.controlMessageBytes, core, false),
|
||||
checkedAdd(demand.serviceNs, checkedAdd(
|
||||
target.getSharedMemoryLinkServiceNs(
|
||||
target.sharedMemoryReadReturnsPayload ? bytes : 0, core, true),
|
||||
target.getLocalMemoryServiceNs(bytes, true))));
|
||||
append(core, "host-load", service, family.hostOffsets.valueAt(index));
|
||||
}
|
||||
}
|
||||
return true;
|
||||
}
|
||||
|
||||
bool PeriodicPlan::boundary(const BoundaryProgram &program) {
|
||||
for (const BoundaryInstruction &instruction : program.instructions) {
|
||||
if (const auto *run = std::get_if<EmitSendRun>(&instruction)) {
|
||||
for (const auto &slice : run->slices)
|
||||
if (!transfer(slice, true))
|
||||
return false;
|
||||
} else if (const auto *run = std::get_if<EmitReceiveAssemblyRun>(&instruction)) {
|
||||
for (const auto &slice : run->slices)
|
||||
if (!transfer(slice, false))
|
||||
return false;
|
||||
} else if (const auto *reuse = std::get_if<EmitHostReuseWait>(&instruction)) {
|
||||
for (LaneInterval lanes : reuse->lanes.intervals())
|
||||
for (unsigned lane = lanes.begin; lane < lanes.end; ++lane)
|
||||
wait(program.key.first->cores[lane], reuse->eventRegisters.valueAt(lane),
|
||||
reuse->waitValues.valueAt(lane), "free-wait");
|
||||
}
|
||||
}
|
||||
return error.empty();
|
||||
}
|
||||
|
||||
bool PeriodicPlan::connect() {
|
||||
if (sends.size() != receives.size()) {
|
||||
error = "direct channel endpoint counts differ";
|
||||
return false;
|
||||
}
|
||||
for (const auto &[channel, send] : sends) {
|
||||
auto found = receives.find(channel);
|
||||
if (found == receives.end() || found->second.bytes != send.bytes) {
|
||||
error = "direct channel has no matching payload";
|
||||
return false;
|
||||
}
|
||||
const DirectEndpoint &receive = found->second;
|
||||
size_t permission = event(receive.core, "peer-permission", channel);
|
||||
edges.push_back({send.begin, permission,
|
||||
target.getLinkServiceNs(target.controlMessageBytes, send.core, receive.core), 0});
|
||||
edges.push_back({receive.begin, permission, 0, 0});
|
||||
edges.push_back({permission, send.end, checkedAdd(
|
||||
target.getLinkServiceNs(target.controlMessageBytes, receive.core, send.core),
|
||||
target.getLocalMemoryServiceNs(send.bytes, false)), 0});
|
||||
edges.push_back({send.end, receive.end, checkedAdd(
|
||||
target.getLinkServiceNs(send.bytes, send.core, receive.core),
|
||||
target.getLocalMemoryServiceNs(send.bytes, true)), 0});
|
||||
}
|
||||
if (signals.size() != waits.size()) {
|
||||
error = "signal and wait register sets differ";
|
||||
return false;
|
||||
}
|
||||
for (const auto &[key, pending] : waits) {
|
||||
auto found = signals.find(key);
|
||||
if (found == signals.end() || pending.count < 0
|
||||
|| found->second.size() != static_cast<size_t>(pending.count)) {
|
||||
error = "event register does not receive its exact static count";
|
||||
return false;
|
||||
}
|
||||
for (Signal signal : found->second)
|
||||
edges.push_back({signal.event, pending.event,
|
||||
target.getLinkServiceNs(target.controlMessageBytes, signal.core, key.first), 0});
|
||||
}
|
||||
for (size_t core = 0; core < active.size(); ++core)
|
||||
if (active[core])
|
||||
edges.push_back({tail[core], entry[core], 0, 1});
|
||||
return true;
|
||||
}
|
||||
|
||||
void PeriodicPlan::recordSynchronization(llvm::json::Object &report) const {
|
||||
llvm::json::Array registers;
|
||||
for (const auto &[key, pending] : waits) {
|
||||
llvm::json::Array sources;
|
||||
std::map<int64_t, int64_t> counts;
|
||||
auto found = signals.find(key);
|
||||
if (found != signals.end())
|
||||
for (Signal signal : found->second)
|
||||
++counts[signal.core];
|
||||
for (auto [core, count] : counts)
|
||||
sources.push_back(llvm::json::Object{{"core", core}, {"count", count}});
|
||||
registers.push_back(llvm::json::Object{{"core", key.first},
|
||||
{"register", key.second}, {"expected", pending.count},
|
||||
{"kind", events[pending.event].kind}, {"sources", std::move(sources)}});
|
||||
}
|
||||
report["event_registers"] = std::move(registers);
|
||||
}
|
||||
|
||||
bool PeriodicPlan::build(const ComputeGraph &graph,
|
||||
const ScheduledComputeMaterializationResult &materialization,
|
||||
const DeferredBoundaryPlan &boundaries) {
|
||||
if (!target.hasPeriodicTransferTiming()) {
|
||||
error = "target has no complete periodic communication timing contract";
|
||||
return false;
|
||||
}
|
||||
if (transfers.scheduled.empty() || transfers.stageZeroCores.empty()
|
||||
|| !transfers.synchronizationRegisterCount) {
|
||||
error = "periodic timing requires an active repeating pipeline body";
|
||||
return false;
|
||||
}
|
||||
for (auto [index, scheduled] : llvm::enumerate(transfers.scheduled)) {
|
||||
if (!materialization.materializedSchedules[index].runs.empty()
|
||||
&& scheduled.cores.size() != 1) {
|
||||
error = "compact compute runs require one physical core";
|
||||
return false;
|
||||
}
|
||||
for (int64_t core : scheduled.cores) {
|
||||
if (active[core]) {
|
||||
error = "physical core appears in multiple scheduled bodies";
|
||||
return false;
|
||||
}
|
||||
active[core] = true;
|
||||
tail[core] = entry[core] = event(core, "body-entry");
|
||||
}
|
||||
}
|
||||
entrySynchronization();
|
||||
llvm::DenseMap<std::pair<const ScheduledInfo *, unsigned>,
|
||||
const BoundaryProgram *> indexed;
|
||||
for (const BoundaryProgram &program : boundaries.boundaries)
|
||||
indexed[program.key] = &program;
|
||||
for (auto [recordIndex, record] : llvm::enumerate(materialization.materializedSchedules)) {
|
||||
const ScheduledInfo *scheduled = &transfers.scheduled[recordIndex];
|
||||
for (unsigned step = 0; step <= scheduled->stepCount; ++step) {
|
||||
if (const auto *program = indexed.lookup({scheduled, step}))
|
||||
if (!boundary(*program))
|
||||
return false;
|
||||
if (step == scheduled->stepCount)
|
||||
continue;
|
||||
auto appendCompute = [&](const ComputeInstance &instance, int64_t core) {
|
||||
auto found = graph.instanceToIndex.find(instance);
|
||||
if (found == graph.instanceToIndex.end()) {
|
||||
error = "scheduled compute instance is absent from the cost graph";
|
||||
return;
|
||||
}
|
||||
append(core, "compute", graph.nodes[found->second].cost, found->second);
|
||||
computeWork[core] = checkedAdd(computeWork[core], graph.nodes[found->second].cost);
|
||||
};
|
||||
if (!record.runs.empty()) {
|
||||
Cost cost = 0;
|
||||
for (const ComputeInstance &instance : record.runs[step].instances) {
|
||||
auto found = graph.instanceToIndex.find(instance);
|
||||
if (found == graph.instanceToIndex.end()) {
|
||||
error = "scheduled run instance is absent from the cost graph";
|
||||
return false;
|
||||
}
|
||||
cost = checkedAdd(cost, graph.nodes[found->second].cost);
|
||||
}
|
||||
append(scheduled->cores.front(), "compute-run", cost, step);
|
||||
int64_t core = scheduled->cores.front();
|
||||
computeWork[core] = checkedAdd(computeWork[core], cost);
|
||||
} else {
|
||||
for (auto [lane, instance] : llvm::enumerate(record.stepPlans[step].stepTuple.instances))
|
||||
appendCompute(instance, scheduled->cores[lane]);
|
||||
}
|
||||
}
|
||||
}
|
||||
completionSynchronization();
|
||||
return error.empty() && connect();
|
||||
}
|
||||
|
||||
static mlir::FailureOr<PeriodicDependencyEstimate> analyzePlan(PeriodicPlan &plan,
|
||||
const ComputeGraph &graph,
|
||||
const ScheduledComputeMaterializationResult &materialization,
|
||||
const DeferredBoundaryPlan &boundaries) {
|
||||
if (!plan.build(graph, materialization, boundaries))
|
||||
return mlir::failure();
|
||||
return analyzePeriodicDependencies(plan.events.size(), plan.edges, plan.error);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
llvm::json::Object getPeriodicScheduleReport(const ComputeGraph &graph,
|
||||
const ScheduledComputeMaterializationResult &materialization,
|
||||
const DeferredTransferPlan &transfers, const DeferredBoundaryPlan &boundaries,
|
||||
const PimInstructionTiming &target) {
|
||||
auto start = std::chrono::steady_clock::now();
|
||||
PeriodicPlan plan(target, transfers);
|
||||
llvm::json::Object report{{"diagnostic_only", true},
|
||||
{"selection_eligible", false},
|
||||
{"timing_model", "serial compute chunks; uncontended communication; no issue or local contention model"}};
|
||||
report["omitted_effects"] = llvm::json::Array{
|
||||
"placement and timing of baseline memory accesses",
|
||||
"communication introduced after deferred realization, including final output collection",
|
||||
"instruction issue, execution-unit overlap and local-memory contention",
|
||||
"projection materialization, strided transfer splitting and load folding",
|
||||
"network injection queues and global-memory arbitration"};
|
||||
report["native_host_buffer_bytes"] = static_cast<int64_t>(transfers.pipelineHostBufferBytes);
|
||||
auto estimate = analyzePlan(plan, graph, materialization, boundaries);
|
||||
if (mlir::succeeded(estimate)) {
|
||||
plan.recordSynchronization(report);
|
||||
SharedMemoryDemand memory = graph.baselineMemory;
|
||||
memory += plan.boundaryMemory;
|
||||
report["recurrence_lower_ns"] = estimate->recurrenceLowerNs;
|
||||
report["recurrence_upper_ns"] = estimate->recurrenceUpperNs;
|
||||
report["within_generation_span_ns"] = static_cast<int64_t>(estimate->withinGenerationSpanNs);
|
||||
report["global_memory_service_ns"] = static_cast<int64_t>(memory.serviceNs);
|
||||
report["baseline_memory_service_ns"] = static_cast<int64_t>(graph.baselineMemory.serviceNs);
|
||||
report["boundary_memory_service_ns"] = static_cast<int64_t>(plan.boundaryMemory.serviceNs);
|
||||
report["boundary_read_bytes"] = static_cast<int64_t>(plan.boundaryMemory.readBytes);
|
||||
report["boundary_write_bytes"] = static_cast<int64_t>(plan.boundaryMemory.writeBytes);
|
||||
report["boundary_read_requests"] = static_cast<int64_t>(plan.boundaryMemory.readRequests);
|
||||
report["boundary_write_requests"] = static_cast<int64_t>(plan.boundaryMemory.writeRequests);
|
||||
Cost coreWork = *std::max_element(plan.computeWork.begin(), plan.computeWork.end());
|
||||
report["maximum_core_compute_ns"] = static_cast<int64_t>(coreWork);
|
||||
report["interval_estimate_ns"] = std::max({estimate->recurrenceUpperNs,
|
||||
static_cast<double>(memory.serviceNs), static_cast<double>(coreWork)});
|
||||
report["relaxation_rounds"] = static_cast<int64_t>(estimate->relaxationRounds);
|
||||
llvm::json::Array cycle;
|
||||
std::map<llvm::StringRef, int64_t> kinds;
|
||||
std::map<int64_t, Cost> coreDelays;
|
||||
std::map<std::pair<int64_t, int64_t>, Cost> linkDelays;
|
||||
Cost delay = 0, distance = 0;
|
||||
for (size_t index : estimate->criticalCycle) {
|
||||
const auto &edge = plan.edges[index];
|
||||
const Event &source = plan.events[edge.source], &destination = plan.events[edge.target];
|
||||
delay = checkedAdd(delay, edge.delayNs);
|
||||
distance = checkedAdd(distance, static_cast<Cost>(edge.distance));
|
||||
++kinds[destination.kind];
|
||||
coreDelays[destination.core] = checkedAdd(coreDelays[destination.core], edge.delayNs);
|
||||
if (source.core != destination.core)
|
||||
linkDelays[{source.core, destination.core}] = checkedAdd(
|
||||
linkDelays[{source.core, destination.core}], edge.delayNs);
|
||||
if (cycle.size() < 128)
|
||||
cycle.push_back(llvm::json::Object{{"source_core", source.core},
|
||||
{"source_kind", source.kind}, {"target_core", destination.core},
|
||||
{"target_kind", destination.kind}, {"identity", destination.identity},
|
||||
{"delay_ns", static_cast<int64_t>(edge.delayNs)},
|
||||
{"distance", edge.distance}});
|
||||
}
|
||||
llvm::json::Object counts;
|
||||
for (auto [kind, count] : kinds)
|
||||
counts[kind] = count;
|
||||
report["critical_cycle"] = std::move(cycle);
|
||||
report["critical_cycle_event_counts"] = std::move(counts);
|
||||
report["critical_cycle_edges"] = static_cast<int64_t>(estimate->criticalCycle.size());
|
||||
report["critical_cycle_delay_ns"] = static_cast<int64_t>(delay);
|
||||
report["critical_cycle_distance"] = static_cast<int64_t>(distance);
|
||||
llvm::json::Array cores, links;
|
||||
for (auto [core, ns] : coreDelays)
|
||||
cores.push_back(llvm::json::Object{{"core", core}, {"delay_ns", static_cast<int64_t>(ns)}});
|
||||
for (auto [pair, ns] : linkDelays)
|
||||
links.push_back(llvm::json::Object{{"source", pair.first}, {"target", pair.second},
|
||||
{"delay_ns", static_cast<int64_t>(ns)}});
|
||||
report["critical_cycle_cores"] = std::move(cores);
|
||||
report["critical_cycle_links"] = std::move(links);
|
||||
}
|
||||
report["status"] = plan.error.empty() ? "estimated" : "unavailable";
|
||||
if (!plan.error.empty())
|
||||
report["reason"] = plan.error;
|
||||
report["events"] = static_cast<int64_t>(plan.events.size());
|
||||
report["edges"] = static_cast<int64_t>(plan.edges.size());
|
||||
llvm::json::Object eventCounts;
|
||||
std::map<llvm::StringRef, int64_t> counts;
|
||||
for (const Event &event : plan.events)
|
||||
++counts[event.kind];
|
||||
for (auto [kind, count] : counts)
|
||||
eventCounts[kind] = count;
|
||||
report["event_counts"] = std::move(eventCounts);
|
||||
report["analysis_wall_ms"] = std::chrono::duration<double, std::milli>(
|
||||
std::chrono::steady_clock::now() - start).count();
|
||||
return report;
|
||||
}
|
||||
|
||||
void dumpPeriodicScheduleEstimate(const ComputeGraph &graph,
|
||||
const ScheduledComputeMaterializationResult &materialization,
|
||||
const DeferredTransferPlan &transfers, const DeferredBoundaryPlan &boundaries,
|
||||
const PimInstructionTiming &target) {
|
||||
auto file = openDialectDumpFileWithExtension("pipeline_periodic", "/reports", "json");
|
||||
if (!file.is_open())
|
||||
return;
|
||||
auto report = getPeriodicScheduleReport(graph, materialization, transfers,
|
||||
boundaries, target);
|
||||
llvm::raw_os_ostream os(file);
|
||||
os << llvm::formatv("{0:2}\n", llvm::json::Value(std::move(report)));
|
||||
}
|
||||
|
||||
} // namespace onnx_mlir
|
||||
@@ -0,0 +1,22 @@
|
||||
#pragma once
|
||||
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/DeferredBoundaryPlanning.hpp"
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ComputeGraph.hpp"
|
||||
#include "llvm/Support/JSON.h"
|
||||
|
||||
namespace onnx_mlir {
|
||||
namespace spatial { struct ScheduledComputeMaterializationResult; }
|
||||
struct PimInstructionTiming;
|
||||
|
||||
llvm::json::Object getPeriodicScheduleReport(const spatial::ComputeGraph &graph,
|
||||
const spatial::ScheduledComputeMaterializationResult &materialization,
|
||||
const spatial::DeferredTransferPlan &transfers, const spatial::DeferredBoundaryPlan &boundaries,
|
||||
const PimInstructionTiming &target);
|
||||
|
||||
// Read-only diagnostics over the final physical transfer and synchronization plan.
|
||||
void dumpPeriodicScheduleEstimate(const spatial::ComputeGraph &graph,
|
||||
const spatial::ScheduledComputeMaterializationResult &materialization,
|
||||
const spatial::DeferredTransferPlan &transfers, const spatial::DeferredBoundaryPlan &boundaries,
|
||||
const PimInstructionTiming &target);
|
||||
|
||||
} // namespace onnx_mlir
|
||||
@@ -39,6 +39,7 @@ add_pim_library(SpatialOps
|
||||
Passes/Transforms/MergeComputeNodes/Scheduling/PipelineScheduling.cpp
|
||||
Passes/Transforms/MergeComputeNodes/Scheduling/PipelineWorkloadPreparation.cpp
|
||||
Passes/Transforms/MergeComputeNodes/Scheduling/ScheduleEvaluation.cpp
|
||||
Passes/Transforms/MergeComputeNodes/Scheduling/PeriodicDependencies.cpp
|
||||
Passes/Transforms/TrivialGraphComputeMergePass.cpp
|
||||
|
||||
EXCLUDE_FROM_OM_LIBS
|
||||
|
||||
+18
-2
@@ -465,8 +465,20 @@ LogicalResult prepareMultiCpuTupleInput(OpBuilder &, Location loc, Value input,
|
||||
|
||||
LogicalResult materializeDeferredPayloadDemands(OpBuilder &builder, Location loc, Block &body,
|
||||
ArrayRef<DeferredInputPlan> plans, IRMapping &mapper,
|
||||
llvm::SmallPtrSetImpl<Operation *> &absorbed) {
|
||||
llvm::SmallPtrSetImpl<Operation *> &absorbed,
|
||||
bool deterministicOrder) {
|
||||
// Deferred demand order must not depend on allocator addresses.
|
||||
DenseMap<Value, unsigned> valueOrder;
|
||||
if (deterministicOrder) {
|
||||
for (BlockArgument argument : body.getArguments())
|
||||
valueOrder.try_emplace(argument, valueOrder.size());
|
||||
body.walk([&](Operation *op) {
|
||||
for (Value result : op->getResults())
|
||||
valueOrder.try_emplace(result, valueOrder.size());
|
||||
});
|
||||
}
|
||||
for (const DeferredInputPlan &plan : plans) {
|
||||
valueOrder.try_emplace(plan.graphInput, valueOrder.size());
|
||||
if (plan.availableValue) { mapper.map(plan.graphInput, plan.availableValue); continue; }
|
||||
SmallVector<Value> roots;
|
||||
bool needsIdentity = false;
|
||||
@@ -498,7 +510,11 @@ LogicalResult materializeDeferredPayloadDemands(OpBuilder &builder, Location loc
|
||||
}
|
||||
}
|
||||
if (needsIdentity) roots.push_back(plan.graphInput);
|
||||
llvm::sort(roots, [](Value a, Value b) { return a.getAsOpaquePointer() < b.getAsOpaquePointer(); });
|
||||
llvm::sort(roots, [&](Value a, Value b) {
|
||||
if (!deterministicOrder) return a.getAsOpaquePointer() < b.getAsOpaquePointer();
|
||||
assert(valueOrder.count(a) && valueOrder.count(b));
|
||||
return valueOrder.lookup(a) < valueOrder.lookup(b);
|
||||
});
|
||||
roots.erase(std::unique(roots.begin(), roots.end()), roots.end());
|
||||
Value sharedSource;
|
||||
if (!plan.blueprint) {
|
||||
|
||||
+2
-1
@@ -47,7 +47,8 @@ LogicalResult materializeDeferredPayloadDemands(OpBuilder &builder, Location loc
|
||||
Block &graphBody,
|
||||
ArrayRef<DeferredInputPlan> plans,
|
||||
IRMapping &mapper,
|
||||
llvm::SmallPtrSetImpl<Operation *> &absorbed);
|
||||
llvm::SmallPtrSetImpl<Operation *> &absorbed,
|
||||
bool deterministicOrder);
|
||||
|
||||
} // namespace spatial
|
||||
} // namespace onnx_mlir
|
||||
|
||||
+4
-1
@@ -148,7 +148,8 @@ static LogicalResult verifyDominance(func::FuncOp funcOp) {
|
||||
LogicalResult realizeDeferredCommunication(func::FuncOp funcOp,
|
||||
const ScheduledComputeMaterializationResult& materialization,
|
||||
const SchedulingTarget& target,
|
||||
size_t pipelineStages) {
|
||||
size_t pipelineStages,
|
||||
const ComputeGraph *costGraph) {
|
||||
IRRewriter rewriter(funcOp.getContext());
|
||||
eraseUnusedIdentityDeferredCommunications(funcOp, rewriter);
|
||||
|
||||
@@ -172,6 +173,8 @@ LogicalResult realizeDeferredCommunication(func::FuncOp funcOp,
|
||||
*transfers, *schedule, target.synchronizationRegisterCount);
|
||||
if (failed(boundaries))
|
||||
return funcOp.emitOpError("phase 2 failed to build sparse boundary programs");
|
||||
if (pipelineStages > 1 && costGraph && target.reportPeriodicSchedule)
|
||||
target.reportPeriodicSchedule(*costGraph, materialization, *transfers, *boundaries);
|
||||
|
||||
if (failed(retargetDeferredPublications(funcOp, *transfers))
|
||||
|| failed(replaceFinalGraphPublications(funcOp, *transfers)))
|
||||
|
||||
+3
-1
@@ -6,10 +6,12 @@ namespace onnx_mlir::spatial {
|
||||
|
||||
struct ScheduledComputeMaterializationResult;
|
||||
struct SchedulingTarget;
|
||||
struct ComputeGraph;
|
||||
|
||||
mlir::LogicalResult realizeDeferredCommunication(mlir::func::FuncOp funcOp,
|
||||
const ScheduledComputeMaterializationResult& materialization,
|
||||
const SchedulingTarget& target,
|
||||
size_t pipelineStages = 1);
|
||||
size_t pipelineStages = 1,
|
||||
const ComputeGraph *costGraph = nullptr);
|
||||
|
||||
} // namespace onnx_mlir::spatial
|
||||
|
||||
+4
-4
@@ -239,7 +239,7 @@ static LogicalResult materializeResultfulBatchRun(
|
||||
|
||||
Block &source = batch.getBody().front();
|
||||
llvm::SmallPtrSet<Operation *, 32> absorbed;
|
||||
if (failed(materializeDeferredPayloadDemands(builder, bodyLoc, source, inputPlans, mapper, absorbed)))
|
||||
if (failed(materializeDeferredPayloadDemands(builder, bodyLoc, source, inputPlans, mapper, absorbed, !schedule.processorStages.empty())))
|
||||
return failure();
|
||||
for (Operation &op : source.without_terminator())
|
||||
if (!absorbed.contains(&op))
|
||||
@@ -334,7 +334,7 @@ static LogicalResult materializeSingleCpuPeftClass(
|
||||
inputPlans.push_back(std::move(plan));
|
||||
}
|
||||
llvm::SmallPtrSet<Operation *, 32> absorbed;
|
||||
if (failed(materializeDeferredPayloadDemands(rewriter, compute.getLoc(), compute.getBody().front(), inputPlans, mapper, absorbed)))
|
||||
if (failed(materializeDeferredPayloadDemands(rewriter, compute.getLoc(), compute.getBody().front(), inputPlans, mapper, absorbed, !schedule.processorStages.empty())))
|
||||
return failure();
|
||||
cloneComputeBody(rewriter, compute.getBody().front(), mapper, yieldedValues, absorbed);
|
||||
} else {
|
||||
@@ -589,7 +589,7 @@ static LogicalResult materializeMultiCpuPeftClass(
|
||||
}
|
||||
SmallVector<Value> yieldedValues;
|
||||
llvm::SmallPtrSet<Operation *, 32> absorbed;
|
||||
if (failed(materializeDeferredPayloadDemands(rewriter, compute.getLoc(), compute.getBody().front(), inputPlans, mapper, absorbed)))
|
||||
if (failed(materializeDeferredPayloadDemands(rewriter, compute.getLoc(), compute.getBody().front(), inputPlans, mapper, absorbed, !schedule.processorStages.empty())))
|
||||
return failure();
|
||||
cloneComputeBody(rewriter, compute.getBody().front(), mapper, yieldedValues, absorbed);
|
||||
for (Value yielded : yieldedValues) {
|
||||
@@ -668,7 +668,7 @@ static LogicalResult materializeMultiCpuPeftClass(
|
||||
for (unsigned index = 0; index < batch.getNumResults(); ++index)
|
||||
mapper.map(*batch.getOutputArgument(index), iterArgs[index]);
|
||||
llvm::SmallPtrSet<Operation *, 32> absorbed;
|
||||
if (failed(materializeDeferredPayloadDemands(builder, bodyLoc, batch.getBody().front(), inputPlans, mapper, absorbed)))
|
||||
if (failed(materializeDeferredPayloadDemands(builder, bodyLoc, batch.getBody().front(), inputPlans, mapper, absorbed, !schedule.processorStages.empty())))
|
||||
return failure();
|
||||
for (Operation &op : batch.getBody().front().without_terminator())
|
||||
if (!absorbed.contains(&op))
|
||||
|
||||
@@ -79,6 +79,13 @@ void dumpScheduledComputeReport(ModuleOp moduleOp, func::FuncOp funcOp, const Me
|
||||
std::fstream file = openDialectDumpFileWithExtension("spatial3_scheduled_no_comm", "/reports", "txt");
|
||||
if (!file.is_open()) return;
|
||||
llvm::raw_os_ostream os(file);
|
||||
writeScheduledComputeReport(os, moduleOp, funcOp, schedule, peftClassPlans, records);
|
||||
}
|
||||
|
||||
void writeScheduledComputeReport(raw_ostream &os, ModuleOp moduleOp, func::FuncOp funcOp,
|
||||
const MergeScheduleResult &schedule,
|
||||
const llvm::MapVector<size_t, PeftClassPlan> &peftClassPlans,
|
||||
ArrayRef<ScheduledMaterializationRecord> records) {
|
||||
OpPrintingFlags flags;
|
||||
flags.elideLargeElementsAttrs().enableDebugInfo(false, false).assumeVerified();
|
||||
AsmState asmState(moduleOp.getOperation(), flags);
|
||||
|
||||
@@ -5,6 +5,12 @@
|
||||
namespace onnx_mlir {
|
||||
namespace spatial {
|
||||
|
||||
void writeScheduledComputeReport(llvm::raw_ostream &os, ModuleOp moduleOp,
|
||||
func::FuncOp funcOp,
|
||||
const MergeScheduleResult &schedule,
|
||||
const llvm::MapVector<size_t, PeftClassPlan> &peftClassPlans,
|
||||
ArrayRef<ScheduledMaterializationRecord> materializedSchedules);
|
||||
|
||||
void dumpScheduledComputeReport(ModuleOp moduleOp,
|
||||
func::FuncOp funcOp,
|
||||
const MergeScheduleResult &schedule,
|
||||
|
||||
+152
-111
@@ -1,5 +1,6 @@
|
||||
#include "mlir/Pass/Pass.h"
|
||||
|
||||
#include "ScheduledSpatialPasses.hpp"
|
||||
#include "DeferredCommunicationRealization.hpp"
|
||||
#include "ScheduledComputeReport.hpp"
|
||||
#include "ScheduledComputeVerification.hpp"
|
||||
@@ -40,9 +41,9 @@ struct ScheduleAndRealizeSpatialPass final
|
||||
ScheduleAndRealizeSpatialPass() = default;
|
||||
ScheduleAndRealizeSpatialPass(const SchedulingTarget& target,
|
||||
SpatialDataflowExportStage exportStage,
|
||||
size_t pipelineStages)
|
||||
size_t pipelineStages, bool periodicDiagnostics)
|
||||
: target(target), exportStage(exportStage),
|
||||
pipelineStages(pipelineStages), hasTarget(true) {}
|
||||
pipelineStages(pipelineStages), periodicDiagnostics(periodicDiagnostics), hasTarget(true) {}
|
||||
|
||||
StringRef getArgument() const override { return "schedule-and-realize-spatial"; }
|
||||
StringRef getDescription() const override {
|
||||
@@ -64,113 +65,10 @@ struct ScheduleAndRealizeSpatialPass final
|
||||
signalPassFailure();
|
||||
return;
|
||||
}
|
||||
auto entry = requireEntry(moduleOp);
|
||||
if (failed(entry)) {
|
||||
signalPassFailure();
|
||||
return;
|
||||
}
|
||||
func::FuncOp entryFunc = *entry;
|
||||
|
||||
ComputeGraph scheduledGraph;
|
||||
MergeScheduleResult schedule;
|
||||
for (;;) {
|
||||
auto graph = buildComputeGraph(entryFunc, target);
|
||||
if (failed(graph)) {
|
||||
signalPassFailure();
|
||||
return;
|
||||
}
|
||||
scheduledGraph = std::move(*graph);
|
||||
std::string pipelineError;
|
||||
if (pipelineStages > 1) {
|
||||
FailureOr<PipelineWorkloadPreparation> preparation =
|
||||
preparePipelineWorkload(
|
||||
scheduledGraph, pipelineStages, target, pipelineError);
|
||||
if (failed(preparation)) {
|
||||
moduleOp.emitError() << pipelineError;
|
||||
signalPassFailure();
|
||||
return;
|
||||
}
|
||||
if (*preparation == PipelineWorkloadPreparation::Changed)
|
||||
continue;
|
||||
}
|
||||
if (pipelineStages == 1) {
|
||||
MergeSchedulingAnalysis analysis(
|
||||
scheduledGraph, target, entryFunc.getContext(), pipelineError);
|
||||
if (failed(analysis.getResult())) {
|
||||
moduleOp.emitError() << pipelineError;
|
||||
signalPassFailure();
|
||||
return;
|
||||
}
|
||||
schedule = std::move(*analysis.getResult());
|
||||
break;
|
||||
}
|
||||
if (succeeded(runPipelineScheduler(
|
||||
scheduledGraph, schedule, pipelineStages, target, pipelineError)))
|
||||
break;
|
||||
moduleOp.emitError() << pipelineError;
|
||||
signalPassFailure();
|
||||
return;
|
||||
}
|
||||
PatternRewriter rewriter(moduleOp.getContext());
|
||||
FailureOr<ScheduledComputeMaterializationResult> materialization =
|
||||
materializeScheduledCompute(entryFunc, schedule, rewriter);
|
||||
if (failed(materialization)) {
|
||||
signalPassFailure();
|
||||
return;
|
||||
}
|
||||
ScheduledComputeMaterializationResult materializationResult = std::move(*materialization);
|
||||
|
||||
if (failed(verifyMaterializedScheduleMapping(
|
||||
entryFunc, schedule, materializationResult.peftClassPlans,
|
||||
materializationResult.graphComputeToBlockMap,
|
||||
materializationResult.materializedSchedules))
|
||||
|| failed(verifyDeferredTransferPhase1Invariants(entryFunc))
|
||||
|| failed(verifyScheduledMaterializationRecords(materializationResult.materializedSchedules))
|
||||
|| failed(verifyScheduledSpatialInvariants(entryFunc))) {
|
||||
moduleOp.emitError("scheduled Spatial phase verification failed");
|
||||
signalPassFailure();
|
||||
return;
|
||||
}
|
||||
|
||||
dumpModule(moduleOp, "spatial3_scheduled_no_comm", /*assumeVerified=*/true);
|
||||
if (shouldExportSpatialDataflowStage(exportStage, SpatialDataflowExportStage::Spatial3)
|
||||
&& failed(exportSpatialDataflowCsvScheduled(
|
||||
entryFunc, materializationResult.materializedSchedules,
|
||||
"spatial3_scheduled_no_comm", "spatial3"))) {
|
||||
signalPassFailure();
|
||||
return;
|
||||
}
|
||||
dumpScheduledComputeReport(
|
||||
moduleOp, entryFunc, schedule, materializationResult.peftClassPlans,
|
||||
materializationResult.materializedSchedules);
|
||||
|
||||
if (failed(realizeDeferredCommunication(
|
||||
entryFunc, materializationResult, target, pipelineStages))) {
|
||||
moduleOp.emitError("Spatial communication realization failed");
|
||||
signalPassFailure();
|
||||
return;
|
||||
}
|
||||
|
||||
bool deferredRemains = false;
|
||||
entryFunc.walk([&](SpatDeferredCommunicationOp deferred) {
|
||||
if (deferredRemains)
|
||||
return;
|
||||
deferred.emitOpError("realized Spatial graph still contains deferred communication");
|
||||
deferredRemains = true;
|
||||
});
|
||||
if (deferredRemains
|
||||
|| failed(verifyScheduledResultsLive(materializationResult.materializedSchedules))
|
||||
|| failed(verifyScheduledSpatialInvariants(entryFunc))) {
|
||||
moduleOp.emitError("realized Spatial communication verification failed");
|
||||
signalPassFailure();
|
||||
return;
|
||||
}
|
||||
|
||||
dumpModule(moduleOp, "spatial4_scheduled", /*assumeVerified=*/true);
|
||||
if (shouldExportSpatialDataflowStage(exportStage, SpatialDataflowExportStage::Spatial4)
|
||||
&& failed(exportSpatialDataflowCsvScheduled(
|
||||
entryFunc, materializationResult.materializedSchedules,
|
||||
"spatial4_scheduled", "spatial4")))
|
||||
auto prepared = prepareSpatialSchedules(moduleOp, target, pipelineStages, false);
|
||||
if (failed(prepared) || failed(realizeSpatialSchedule(moduleOp,
|
||||
prepared->candidates.front(), target, pipelineStages, exportStage,
|
||||
periodicDiagnostics ? &prepared->graph : nullptr)))
|
||||
signalPassFailure();
|
||||
}
|
||||
|
||||
@@ -178,11 +76,154 @@ private:
|
||||
SchedulingTarget target;
|
||||
SpatialDataflowExportStage exportStage = SpatialDataflowExportStage::None;
|
||||
size_t pipelineStages = 1;
|
||||
bool periodicDiagnostics = false;
|
||||
bool hasTarget = false;
|
||||
};
|
||||
|
||||
} // namespace
|
||||
|
||||
FailureOr<PreparedSpatialSchedules> prepareSpatialSchedules(ModuleOp moduleOp,
|
||||
const SchedulingTarget &target, size_t pipelineStages, bool collectCandidates) {
|
||||
if (!hasValidTarget(target) || !PipelineCoreLayout(target.processorCount, pipelineStages).isValid()
|
||||
|| (pipelineStages > 1 && !target.synchronizationRegisterCount))
|
||||
return moduleOp.emitError("Spatial scheduling requires an explicit valid target and pipeline layout"), failure();
|
||||
auto entry = requireEntry(moduleOp);
|
||||
if (failed(entry)) return failure();
|
||||
auto entryFunc = *entry;
|
||||
PreparedSpatialSchedules prepared;
|
||||
auto &scheduledGraph = prepared.graph;
|
||||
MergeScheduleResult schedule;
|
||||
for (;;) {
|
||||
auto graph = buildComputeGraph(entryFunc, target);
|
||||
if (failed(graph)) {
|
||||
return failure();
|
||||
}
|
||||
scheduledGraph = std::move(*graph);
|
||||
std::string pipelineError;
|
||||
if (pipelineStages > 1) {
|
||||
FailureOr<PipelineWorkloadPreparation> preparation =
|
||||
preparePipelineWorkload(
|
||||
scheduledGraph, pipelineStages, target, pipelineError);
|
||||
if (failed(preparation)) {
|
||||
moduleOp.emitError() << pipelineError;
|
||||
return failure();
|
||||
}
|
||||
if (*preparation == PipelineWorkloadPreparation::Changed)
|
||||
continue;
|
||||
}
|
||||
if (pipelineStages == 1) {
|
||||
MergeSchedulingAnalysis analysis(
|
||||
scheduledGraph, target, entryFunc.getContext(), pipelineError);
|
||||
if (failed(analysis.getResult())) {
|
||||
moduleOp.emitError() << pipelineError;
|
||||
return failure();
|
||||
}
|
||||
schedule = std::move(*analysis.getResult());
|
||||
break;
|
||||
}
|
||||
if (succeeded(runPipelineScheduler(
|
||||
scheduledGraph, schedule, pipelineStages, target, pipelineError, collectCandidates ? &prepared.candidates : nullptr)))
|
||||
break;
|
||||
moduleOp.emitError() << pipelineError;
|
||||
return failure();
|
||||
}
|
||||
|
||||
if (prepared.candidates.empty()) prepared.candidates.push_back(std::move(schedule));
|
||||
return prepared;
|
||||
}
|
||||
|
||||
LogicalResult realizeSpatialSchedule(ModuleOp moduleOp,
|
||||
const MergeScheduleResult &schedule, const SchedulingTarget &target,
|
||||
size_t pipelineStages, SpatialDataflowExportStage exportStage,
|
||||
const ComputeGraph *costGraph, std::string *scheduleReport) {
|
||||
auto entry = requireEntry(moduleOp);
|
||||
if (failed(entry)) return failure();
|
||||
auto entryFunc = *entry;
|
||||
PatternRewriter rewriter(moduleOp.getContext());
|
||||
FailureOr<ScheduledComputeMaterializationResult> materialization =
|
||||
materializeScheduledCompute(entryFunc, schedule, rewriter);
|
||||
if (failed(materialization)) {
|
||||
return failure();
|
||||
}
|
||||
ScheduledComputeMaterializationResult materializationResult = std::move(*materialization);
|
||||
|
||||
if (failed(verifyMaterializedScheduleMapping(
|
||||
entryFunc, schedule, materializationResult.peftClassPlans,
|
||||
materializationResult.graphComputeToBlockMap,
|
||||
materializationResult.materializedSchedules))
|
||||
|| failed(verifyDeferredTransferPhase1Invariants(entryFunc))
|
||||
|| failed(verifyScheduledMaterializationRecords(materializationResult.materializedSchedules))
|
||||
|| failed(verifyScheduledSpatialInvariants(entryFunc))) {
|
||||
moduleOp.emitError("scheduled Spatial phase verification failed");
|
||||
return failure();
|
||||
}
|
||||
|
||||
dumpModule(moduleOp, "spatial3_scheduled_no_comm", /*assumeVerified=*/true);
|
||||
if (shouldExportSpatialDataflowStage(exportStage, SpatialDataflowExportStage::Spatial3)
|
||||
&& failed(exportSpatialDataflowCsvScheduled(
|
||||
entryFunc, materializationResult.materializedSchedules,
|
||||
"spatial3_scheduled_no_comm", "spatial3"))) {
|
||||
return failure();
|
||||
}
|
||||
if (scheduleReport) {
|
||||
llvm::raw_string_ostream os(*scheduleReport);
|
||||
writeScheduledComputeReport(os, moduleOp, entryFunc, schedule,
|
||||
materializationResult.peftClassPlans, materializationResult.materializedSchedules);
|
||||
} else {
|
||||
dumpScheduledComputeReport(
|
||||
moduleOp, entryFunc, schedule, materializationResult.peftClassPlans,
|
||||
materializationResult.materializedSchedules);
|
||||
}
|
||||
|
||||
if (failed(realizeDeferredCommunication(
|
||||
entryFunc, materializationResult, target, pipelineStages, costGraph))) {
|
||||
moduleOp.emitError("Spatial communication realization failed");
|
||||
return failure();
|
||||
}
|
||||
|
||||
bool deferredRemains = false;
|
||||
entryFunc.walk([&](SpatDeferredCommunicationOp deferred) {
|
||||
if (deferredRemains)
|
||||
return;
|
||||
deferred.emitOpError("realized Spatial graph still contains deferred communication");
|
||||
deferredRemains = true;
|
||||
});
|
||||
if (deferredRemains
|
||||
|| failed(verifyScheduledResultsLive(materializationResult.materializedSchedules))
|
||||
|| failed(verifyScheduledSpatialInvariants(entryFunc))) {
|
||||
moduleOp.emitError("realized Spatial communication verification failed");
|
||||
return failure();
|
||||
}
|
||||
|
||||
dumpModule(moduleOp, "spatial4_scheduled", /*assumeVerified=*/true);
|
||||
if (shouldExportSpatialDataflowStage(exportStage, SpatialDataflowExportStage::Spatial4)
|
||||
&& failed(exportSpatialDataflowCsvScheduled(
|
||||
entryFunc, materializationResult.materializedSchedules,
|
||||
"spatial4_scheduled", "spatial4")))
|
||||
return failure();
|
||||
return success();
|
||||
}
|
||||
|
||||
MergeScheduleResult remapSpatialSchedule(const MergeScheduleResult &schedule,
|
||||
const IRMapping &mapping) {
|
||||
MergeScheduleResult result = schedule;
|
||||
auto remap = [&](ComputeInstance instance) {
|
||||
instance.op = mapping.lookup(instance.op);
|
||||
return instance;
|
||||
};
|
||||
for (auto &instance : result.dominanceOrderCompute) instance = remap(instance);
|
||||
result.computeToCpuMap.clear();
|
||||
result.computeToCpuSlotMap.clear();
|
||||
result.computeToAestMap.clear();
|
||||
result.isLastComputeOfCpu.clear();
|
||||
for (auto [instance, cpu] : schedule.computeToCpuMap) result.computeToCpuMap[remap(instance)] = cpu;
|
||||
for (auto [instance, slot] : schedule.computeToCpuSlotMap) result.computeToCpuSlotMap[remap(instance)] = slot;
|
||||
for (auto [instance, start] : schedule.computeToAestMap) result.computeToAestMap[remap(instance)] = start;
|
||||
for (auto instance : schedule.isLastComputeOfCpu) result.isLastComputeOfCpu.insert(remap(instance));
|
||||
for (auto &[cpu, instance] : result.cpuToLastComputeMap) instance = remap(instance);
|
||||
return result;
|
||||
}
|
||||
|
||||
std::unique_ptr<Pass> createScheduleAndRealizeSpatialPass() {
|
||||
return std::make_unique<ScheduleAndRealizeSpatialPass>();
|
||||
}
|
||||
@@ -190,9 +231,9 @@ std::unique_ptr<Pass> createScheduleAndRealizeSpatialPass() {
|
||||
std::unique_ptr<Pass> createScheduleAndRealizeSpatialPass(
|
||||
const SchedulingTarget& target,
|
||||
SpatialDataflowExportStage exportStage,
|
||||
size_t pipelineStages) {
|
||||
size_t pipelineStages, bool periodicDiagnostics) {
|
||||
return std::make_unique<ScheduleAndRealizeSpatialPass>(
|
||||
target, exportStage, pipelineStages);
|
||||
target, exportStage, pipelineStages, periodicDiagnostics);
|
||||
}
|
||||
|
||||
} // namespace spatial
|
||||
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
#pragma once
|
||||
#include "Scheduling/ComputeGraph.hpp"
|
||||
#include "Scheduling/MergeSchedule.hpp"
|
||||
#include "SpatialDataflowCsvExporter.hpp"
|
||||
#include "mlir/IR/BuiltinOps.h"
|
||||
#include "mlir/IR/IRMapping.h"
|
||||
|
||||
namespace onnx_mlir::spatial {
|
||||
struct PreparedSpatialSchedules {
|
||||
ComputeGraph graph;
|
||||
std::vector<MergeScheduleResult> candidates;
|
||||
};
|
||||
|
||||
mlir::FailureOr<PreparedSpatialSchedules> prepareSpatialSchedules(
|
||||
mlir::ModuleOp module, const SchedulingTarget &target,
|
||||
size_t pipelineStages, bool collectCandidates);
|
||||
mlir::LogicalResult realizeSpatialSchedule(mlir::ModuleOp module,
|
||||
const MergeScheduleResult &schedule, const SchedulingTarget &target,
|
||||
size_t pipelineStages, SpatialDataflowExportStage exportStage,
|
||||
const ComputeGraph *costGraph = nullptr, std::string *scheduleReport = nullptr);
|
||||
MergeScheduleResult remapSpatialSchedule(const MergeScheduleResult &schedule,
|
||||
const mlir::IRMapping &mapping);
|
||||
} // namespace onnx_mlir::spatial
|
||||
+1
-23
@@ -184,29 +184,7 @@ struct SchedulerCostModel {
|
||||
}
|
||||
|
||||
Cost getMatrixComputeLatency(Cost inputBitwidth) const {
|
||||
Cost inputTimes = ceilDiv(inputBitwidth, target.matrixInputResolutionBits);
|
||||
Cost inputPasses = ceilDiv(target.matrixRows, target.matrixInputParallelism);
|
||||
Cost outputPasses = ceilDiv(target.matrixColumns, target.matrixOutputParallelism);
|
||||
Cost readCycles = ceilDiv(target.matrixReadLatencyNs, target.matrixPeriodNs);
|
||||
Cost frontStage = target.matrixInputBufferLatencyCycles + target.matrixInputLatencyCycles
|
||||
+ readCycles + target.matrixSampleLatencyCycles;
|
||||
Cost backPipe = std::max(target.matrixOutputLatencyCycles,
|
||||
checkedAdd(target.matrixShiftLatencyCycles,
|
||||
target.matrixBufferLatencyCycles));
|
||||
Cost backStage = checkedAdd(
|
||||
checkedAdd(target.matrixOutputLatencyCycles, target.matrixShiftLatencyCycles),
|
||||
target.matrixBufferLatencyCycles);
|
||||
backStage = checkedAdd(backStage, checkedMultiply(outputPasses - 1, backPipe));
|
||||
Cost totalTimes = checkedMultiply(inputTimes, inputPasses);
|
||||
if (!target.matrixPipeline)
|
||||
return checkedMultiply(
|
||||
checkedMultiply(checkedAdd(frontStage, backStage), totalTimes),
|
||||
target.matrixPeriodNs);
|
||||
Cost stagePipe = std::max(frontStage, backStage);
|
||||
return checkedMultiply(
|
||||
checkedAdd(checkedAdd(frontStage, backStage),
|
||||
checkedMultiply(totalTimes - 1, stagePipe)),
|
||||
target.matrixPeriodNs);
|
||||
return target.getMatrixComputeLatencyNs(inputBitwidth);
|
||||
}
|
||||
|
||||
Cost getWvmmCost(Type inputType, Type outputType) const {
|
||||
|
||||
+141
@@ -0,0 +1,141 @@
|
||||
#include "PeriodicDependencies.hpp"
|
||||
|
||||
#include <limits>
|
||||
|
||||
namespace onnx_mlir::spatial {
|
||||
|
||||
mlir::FailureOr<PeriodicDependencyEstimate> analyzePeriodicDependencies(
|
||||
size_t nodeCount, llvm::ArrayRef<PeriodicDependency> edges,
|
||||
std::string &error) {
|
||||
std::vector<std::vector<size_t>> outgoing(nodeCount);
|
||||
std::vector<size_t> indegree(nodeCount), recurrent;
|
||||
Cost upper = 0;
|
||||
for (auto [index, edge] : llvm::enumerate(edges)) {
|
||||
if (edge.source >= nodeCount || edge.target >= nodeCount) {
|
||||
error = "periodic dependency references an invalid event";
|
||||
return mlir::failure();
|
||||
}
|
||||
if (edge.delayNs > std::numeric_limits<Cost>::max() - upper) {
|
||||
error = "periodic delay sum exceeds the timing range";
|
||||
return mlir::failure();
|
||||
}
|
||||
upper += edge.delayNs;
|
||||
if (edge.distance) {
|
||||
recurrent.push_back(index);
|
||||
} else {
|
||||
outgoing[edge.source].push_back(index);
|
||||
++indegree[edge.target];
|
||||
}
|
||||
}
|
||||
std::vector<size_t> order;
|
||||
for (size_t node = 0; node < nodeCount; ++node)
|
||||
if (!indegree[node])
|
||||
order.push_back(node);
|
||||
for (size_t cursor = 0; cursor < order.size(); ++cursor)
|
||||
for (size_t index : outgoing[order[cursor]])
|
||||
if (!--indegree[edges[index].target])
|
||||
order.push_back(edges[index].target);
|
||||
if (order.size() != nodeCount) {
|
||||
error = "periodic distance-zero dependencies contain a cycle";
|
||||
return mlir::failure();
|
||||
}
|
||||
|
||||
PeriodicDependencyEstimate result;
|
||||
std::vector<Cost> span(nodeCount);
|
||||
for (size_t node : order) {
|
||||
result.withinGenerationSpanNs =
|
||||
std::max(result.withinGenerationSpanNs, span[node]);
|
||||
for (size_t index : outgoing[node]) {
|
||||
const auto &edge = edges[index];
|
||||
span[edge.target] = std::max(span[edge.target],
|
||||
checkedAdd(span[node], edge.delayNs));
|
||||
}
|
||||
}
|
||||
if (recurrent.empty() || !upper)
|
||||
return result;
|
||||
|
||||
const size_t absent = std::numeric_limits<size_t>::max();
|
||||
std::vector<double> potential(nodeCount);
|
||||
std::vector<size_t> predecessor(nodeCount, absent);
|
||||
auto infeasible = [&](double interval) {
|
||||
std::fill(potential.begin(), potential.end(), 0);
|
||||
std::fill(predecessor.begin(), predecessor.end(), absent);
|
||||
auto relax = [&](size_t index) {
|
||||
const auto &edge = edges[index];
|
||||
double value = potential[edge.source] + edge.delayNs
|
||||
- interval * edge.distance;
|
||||
if (value <= potential[edge.target] + 1e-7)
|
||||
return false;
|
||||
potential[edge.target] = value;
|
||||
predecessor[edge.target] = index;
|
||||
return true;
|
||||
};
|
||||
for (size_t node : order)
|
||||
for (size_t index : outgoing[node])
|
||||
relax(index);
|
||||
// Eliminate distance-zero paths by a DAG sweep. A simple residual path
|
||||
// visits at most R recurrent edges, so R+1 rounds detect a positive cycle.
|
||||
for (size_t round = 0; round <= recurrent.size(); ++round) {
|
||||
++result.relaxationRounds;
|
||||
bool changed = false;
|
||||
for (size_t index : recurrent)
|
||||
changed |= relax(index);
|
||||
for (size_t node : order)
|
||||
for (size_t index : outgoing[node])
|
||||
changed |= relax(index);
|
||||
if (!changed)
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
};
|
||||
result.recurrenceUpperNs = upper;
|
||||
for (unsigned iteration = 0;
|
||||
iteration < 64 && result.recurrenceUpperNs - result.recurrenceLowerNs > 1;
|
||||
++iteration) {
|
||||
double middle = (result.recurrenceLowerNs + result.recurrenceUpperNs) / 2;
|
||||
if (middle == result.recurrenceLowerNs || middle == result.recurrenceUpperNs)
|
||||
break;
|
||||
if (infeasible(middle))
|
||||
result.recurrenceLowerNs = middle;
|
||||
else
|
||||
result.recurrenceUpperNs = middle;
|
||||
}
|
||||
if (result.recurrenceUpperNs - result.recurrenceLowerNs > 1) {
|
||||
error = "periodic timing exceeds one-nanosecond floating-point precision";
|
||||
return mlir::failure();
|
||||
}
|
||||
if (!infeasible(result.recurrenceLowerNs))
|
||||
return result;
|
||||
|
||||
// Recover witness cycles from the last longest-path predecessor forest.
|
||||
std::vector<size_t> seen(nodeCount, absent);
|
||||
double best = 0;
|
||||
for (size_t start = 0; start < nodeCount; ++start) {
|
||||
size_t node = start;
|
||||
while (seen[node] == absent && predecessor[node] != absent) {
|
||||
seen[node] = start;
|
||||
node = edges[predecessor[node]].source;
|
||||
}
|
||||
if (seen[node] != start || predecessor[node] == absent)
|
||||
continue;
|
||||
size_t first = node;
|
||||
Cost delay = 0, distance = 0;
|
||||
std::vector<size_t> cycle;
|
||||
do {
|
||||
size_t index = predecessor[node];
|
||||
cycle.push_back(index);
|
||||
delay = checkedAdd(delay, edges[index].delayNs);
|
||||
distance = checkedAdd(distance, static_cast<Cost>(edges[index].distance));
|
||||
node = edges[index].source;
|
||||
} while (node != first);
|
||||
double ratio = distance ? static_cast<double>(delay) / distance : 0;
|
||||
if (ratio > best) {
|
||||
best = ratio;
|
||||
std::reverse(cycle.begin(), cycle.end());
|
||||
result.criticalCycle = std::move(cycle);
|
||||
}
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
} // namespace onnx_mlir::spatial
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
#pragma once
|
||||
|
||||
#include "Utils.hpp"
|
||||
#include "mlir/Support/LogicalResult.h"
|
||||
#include "llvm/ADT/ArrayRef.h"
|
||||
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
namespace onnx_mlir::spatial {
|
||||
|
||||
// t(target, k) >= t(source, k - distance) + delayNs.
|
||||
struct PeriodicDependency {
|
||||
size_t source = 0;
|
||||
size_t target = 0;
|
||||
Cost delayNs = 0;
|
||||
unsigned distance = 0;
|
||||
};
|
||||
|
||||
struct PeriodicDependencyEstimate {
|
||||
double recurrenceLowerNs = 0;
|
||||
double recurrenceUpperNs = 0;
|
||||
Cost withinGenerationSpanNs = 0;
|
||||
std::vector<size_t> criticalCycle;
|
||||
size_t relaxationRounds = 0;
|
||||
};
|
||||
|
||||
// Requires an acyclic distance-zero graph. Brackets the maximum cycle ratio
|
||||
// within one nanosecond; storage is linear, without generation unrolling.
|
||||
mlir::FailureOr<PeriodicDependencyEstimate> analyzePeriodicDependencies(
|
||||
size_t nodeCount, llvm::ArrayRef<PeriodicDependency> edges,
|
||||
std::string &error);
|
||||
|
||||
} // namespace onnx_mlir::spatial
|
||||
+67
-13
@@ -1,6 +1,7 @@
|
||||
#include "llvm/ADT/DenseMap.h"
|
||||
#include "llvm/ADT/SmallBitVector.h"
|
||||
#include "llvm/ADT/STLExtras.h"
|
||||
#include "llvm/Support/raw_ostream.h"
|
||||
|
||||
#include <algorithm>
|
||||
#include <limits>
|
||||
@@ -1079,10 +1080,20 @@ static LogicalResult packPipelineAssignment(const ComputeGraph &graph,
|
||||
}
|
||||
|
||||
|
||||
struct PipelineCandidateCapture {
|
||||
bool enabled = false;
|
||||
std::vector<MergeScheduleResult> packed;
|
||||
|
||||
void consider(const MergeScheduleResult &candidate) {
|
||||
if (enabled)
|
||||
packed.push_back(candidate);
|
||||
}
|
||||
};
|
||||
|
||||
static LogicalResult assignPipelineCores(const ComputeGraph &graph,
|
||||
MergeScheduleResult &schedule, size_t pipelineStages,
|
||||
const SchedulingTarget &physicalTarget, std::string &error, bool memoryAware,
|
||||
bool sharePublications = true) {
|
||||
bool sharePublications = true, PipelineCandidateCapture *capture = nullptr) {
|
||||
PipelineCoreLayout balancedLayout(physicalTarget.processorCount, pipelineStages);
|
||||
if (!balancedLayout.isValid()) {
|
||||
error = "pipeline scheduling received an incompatible physical core layout";
|
||||
@@ -1098,6 +1109,8 @@ static LogicalResult assignPipelineCores(const ComputeGraph &graph,
|
||||
if (failed(packPipelineAssignment(graph, taskModel, assignment, candidate,
|
||||
pipelineStages, physicalTarget, error)))
|
||||
continue;
|
||||
if (capture)
|
||||
capture->consider(candidate);
|
||||
auto maximumStageSpan = [](const MergeScheduleResult &result) {
|
||||
return result.costs.stageSpanNs.empty() ? Cost{0}
|
||||
: *llvm::max_element(result.costs.stageSpanNs);
|
||||
@@ -1162,27 +1175,68 @@ mlir::LogicalResult runPipelineScheduler(const ComputeGraph& graph,
|
||||
MergeScheduleResult& schedule,
|
||||
size_t pipelineStages,
|
||||
const SchedulingTarget& physicalTarget,
|
||||
std::string& error) {
|
||||
std::string& error,
|
||||
std::vector<MergeScheduleResult> *candidates) {
|
||||
PipelineCoreLayout pipelineLayout(
|
||||
physicalTarget.processorCount, pipelineStages);
|
||||
if (pipelineStages <= 1 || !pipelineLayout.isValid()) {
|
||||
error = "pipeline scheduling requires a valid balanced physical core layout";
|
||||
return mlir::failure();
|
||||
}
|
||||
if (failed(assignPipelineCores(graph, schedule, pipelineStages, physicalTarget, error, false, false)))
|
||||
PipelineCandidateCapture capture{candidates != nullptr, {}};
|
||||
if (failed(assignPipelineCores(graph, schedule, pipelineStages, physicalTarget, error, false, false, &capture)))
|
||||
return failure();
|
||||
if (!physicalTarget.sharedMemoryWidthBytes || graph.publicationTransfers.empty())
|
||||
return success();
|
||||
std::string candidateError;
|
||||
// Retain the calibrated phase-1 winner before evaluating sharing-aware cuts.
|
||||
(void)assignPipelineCores(graph, schedule, pipelineStages, physicalTarget, candidateError, true, false);
|
||||
std::vector<size_t> taskStages;
|
||||
for (const auto &node : graph.nodes)
|
||||
taskStages.push_back(schedule.processorStages[
|
||||
schedule.computeToCpuMap.lookup(node.instance)]);
|
||||
schedule.costs.boundaryMemory = getPipelineBoundaryMemoryDemand(
|
||||
graph, taskStages, physicalTarget);
|
||||
(void)assignPipelineCores(graph, schedule, pipelineStages, physicalTarget, candidateError, true, true);
|
||||
if (physicalTarget.sharedMemoryWidthBytes && !graph.publicationTransfers.empty()) {
|
||||
(void)assignPipelineCores(graph, schedule, pipelineStages, physicalTarget, candidateError, true, false, &capture);
|
||||
std::vector<size_t> taskStages;
|
||||
for (const auto &node : graph.nodes)
|
||||
taskStages.push_back(schedule.processorStages[
|
||||
schedule.computeToCpuMap.lookup(node.instance)]);
|
||||
schedule.costs.boundaryMemory = getPipelineBoundaryMemoryDemand(
|
||||
graph, taskStages, physicalTarget);
|
||||
(void)assignPipelineCores(graph, schedule, pipelineStages, physicalTarget, candidateError, true, true, &capture);
|
||||
}
|
||||
if (candidates) {
|
||||
auto activeStages = [&](const MergeScheduleResult &result) {
|
||||
llvm::SmallBitVector stages(pipelineStages);
|
||||
for (const auto &node : graph.nodes)
|
||||
stages.set(result.processorStages[result.computeToCpuMap.lookup(node.instance)]);
|
||||
return stages;
|
||||
};
|
||||
auto samePlacement = [&](const MergeScheduleResult &a, const MergeScheduleResult &b) {
|
||||
for (const auto &node : graph.nodes) {
|
||||
size_t cpu = a.computeToCpuMap.lookup(node.instance);
|
||||
if (cpu != b.computeToCpuMap.lookup(node.instance)
|
||||
|| a.computeToCpuSlotMap.lookup(node.instance) != b.computeToCpuSlotMap.lookup(node.instance)
|
||||
|| a.processorStages[cpu] != b.processorStages[cpu])
|
||||
return false;
|
||||
}
|
||||
return true;
|
||||
};
|
||||
auto originalStages = activeStages(schedule);
|
||||
SmallVector<size_t> alternatives;
|
||||
for (auto [index, candidate] : llvm::enumerate(capture.packed)) {
|
||||
if (activeStages(candidate) != originalStages || samePlacement(candidate, schedule))
|
||||
continue;
|
||||
if (llvm::none_of(alternatives, [&](size_t other) {
|
||||
return samePlacement(candidate, capture.packed[other]);
|
||||
}))
|
||||
alternatives.push_back(index);
|
||||
}
|
||||
candidates->push_back(schedule);
|
||||
for (size_t index : alternatives) {
|
||||
auto &candidate = capture.packed[index];
|
||||
std::vector<size_t> taskStages;
|
||||
for (const auto &node : graph.nodes)
|
||||
taskStages.push_back(candidate.processorStages[
|
||||
candidate.computeToCpuMap.lookup(node.instance)]);
|
||||
candidate.costs.boundaryMemory = getPipelineBoundaryMemoryDemand(
|
||||
graph, taskStages, physicalTarget);
|
||||
candidates->push_back(std::move(candidate));
|
||||
}
|
||||
}
|
||||
return success();
|
||||
}
|
||||
|
||||
|
||||
+2
-1
@@ -124,7 +124,8 @@ mlir::LogicalResult runPipelineScheduler(const ComputeGraph& graph,
|
||||
MergeScheduleResult& schedule,
|
||||
size_t pipelineStages,
|
||||
const SchedulingTarget& physicalTarget,
|
||||
std::string& error);
|
||||
std::string& error,
|
||||
std::vector<MergeScheduleResult> *candidates = nullptr);
|
||||
|
||||
enum class PipelineWorkloadPreparation {
|
||||
Ready,
|
||||
|
||||
+30
@@ -1,10 +1,40 @@
|
||||
#include "ScheduleEvaluation.hpp"
|
||||
#include "MergeSchedule.hpp"
|
||||
|
||||
#include <cmath>
|
||||
#include <queue>
|
||||
|
||||
namespace onnx_mlir::spatial {
|
||||
|
||||
bool WholeScheduleCost::eligible() const {
|
||||
return primary && alternative && std::isfinite(*primary) && std::isfinite(*alternative)
|
||||
&& *primary > 0 && *alternative > 0 && relativeUncertainty
|
||||
&& std::isfinite(*relativeUncertainty) && *relativeUncertainty >= 0
|
||||
&& *relativeUncertainty < 1;
|
||||
}
|
||||
|
||||
size_t selectPipelineCandidate(llvm::ArrayRef<WholeScheduleCost> scores, std::string &reason) {
|
||||
if (scores.empty() || !scores.front().eligible()) {
|
||||
reason = "original estimate unavailable";
|
||||
return 0;
|
||||
}
|
||||
size_t best = 0;
|
||||
auto upper = [&](size_t i) {
|
||||
return std::max(*scores[i].primary, *scores[i].alternative)
|
||||
/ (1 - *scores[i].relativeUncertainty);
|
||||
};
|
||||
for (size_t i = 1; i < scores.size(); ++i)
|
||||
if (scores[i].eligible() && upper(i) < upper(best)) best = i;
|
||||
if (!best) { reason = "original has the lowest conservative estimate"; return 0; }
|
||||
double originalLower = std::min(*scores.front().primary, *scores.front().alternative);
|
||||
if (upper(best) >= originalLower / (1 + *scores.front().relativeUncertainty)) {
|
||||
reason = "candidate and original overlap static prediction uncertainty";
|
||||
return 0;
|
||||
}
|
||||
reason = "candidate improves on original under both arbitration tie policies and prediction uncertainty";
|
||||
return best;
|
||||
}
|
||||
|
||||
void recordScheduleCosts(const ScheduleEvaluation &evaluation,
|
||||
MergeScheduleResult &schedule) {
|
||||
schedule.costs.totalWorkNs = evaluation.totalWork;
|
||||
|
||||
+3
@@ -9,6 +9,9 @@ namespace onnx_mlir::spatial {
|
||||
|
||||
struct MergeScheduleResult;
|
||||
|
||||
size_t selectPipelineCandidate(llvm::ArrayRef<WholeScheduleCost> scores,
|
||||
std::string &reason);
|
||||
|
||||
struct ScheduleEvaluation {
|
||||
std::vector<Time> taskStarts;
|
||||
std::vector<Time> taskEnds;
|
||||
|
||||
+73
@@ -2,10 +2,18 @@
|
||||
|
||||
#include <cassert>
|
||||
#include <cstddef>
|
||||
#include <functional>
|
||||
#include <optional>
|
||||
#include <string>
|
||||
#include <vector>
|
||||
|
||||
#include "mlir/Support/LogicalResult.h"
|
||||
|
||||
#include "Utils.hpp"
|
||||
|
||||
namespace mlir { class ModuleOp; }
|
||||
namespace llvm::json { class Object; }
|
||||
|
||||
namespace onnx_mlir {
|
||||
namespace spatial {
|
||||
|
||||
@@ -26,6 +34,25 @@ struct SharedMemoryDemand {
|
||||
}
|
||||
};
|
||||
|
||||
struct ComputeGraph;
|
||||
struct ScheduledComputeMaterializationResult;
|
||||
struct DeferredTransferPlan;
|
||||
struct DeferredBoundaryPlan;
|
||||
|
||||
struct WholeScheduleCost {
|
||||
// Steady-state intervals in nanoseconds. The alternative tests sensitivity.
|
||||
std::optional<double> primary, alternative;
|
||||
std::string reason;
|
||||
std::string identity;
|
||||
std::optional<double> relativeUncertainty;
|
||||
bool eligible() const;
|
||||
};
|
||||
|
||||
struct ScheduleCostQuery {
|
||||
std::string identity;
|
||||
std::function<WholeScheduleCost(llvm::json::Object &)> evaluate;
|
||||
};
|
||||
|
||||
struct SchedulingTarget {
|
||||
size_t processorCount = 0;
|
||||
size_t residentWeightCapacity = 0;
|
||||
@@ -35,6 +62,7 @@ struct SchedulingTarget {
|
||||
Cost computeBitwidth = 8;
|
||||
Cost processorPeriodNs = 1;
|
||||
Cost localMemoryWidthBytes = 64;
|
||||
Cost localMemoryPeriodNs = 1;
|
||||
Cost localMemoryReadLatencyCycles = 1;
|
||||
Cost localMemoryWriteLatencyCycles = 1;
|
||||
Cost transferWidthBytes = 8;
|
||||
@@ -48,6 +76,22 @@ struct SchedulingTarget {
|
||||
Cost sharedMemoryWriteLatencyNs = 0;
|
||||
bool zeroSplatUsesSharedMemory = true;
|
||||
|
||||
// Input is a verified, target-lowered candidate. Preparation and evaluation
|
||||
// are read-only. An absent query keeps the original schedule.
|
||||
std::function<mlir::FailureOr<ScheduleCostQuery>(mlir::ModuleOp,
|
||||
llvm::json::Object &)> prepareScheduleCost;
|
||||
std::string wholeScheduleCostUnavailable;
|
||||
std::function<void(const ComputeGraph &,
|
||||
const ScheduledComputeMaterializationResult &, const DeferredTransferPlan &,
|
||||
const DeferredBoundaryPlan &)> reportPeriodicSchedule;
|
||||
|
||||
Cost getLocalMemoryServiceNs(Cost bytes, bool write) const {
|
||||
Cost accesses = bytes / localMemoryWidthBytes
|
||||
+ (bytes % localMemoryWidthBytes != 0);
|
||||
return checkedMultiply(accesses, checkedMultiply(localMemoryPeriodNs,
|
||||
write ? localMemoryWriteLatencyCycles : localMemoryReadLatencyCycles));
|
||||
}
|
||||
|
||||
SharedMemoryDemand getSharedMemoryDemand(
|
||||
Cost bytesPerRequest, Cost requests, bool write) const {
|
||||
SharedMemoryDemand demand;
|
||||
@@ -83,6 +127,35 @@ struct SchedulingTarget {
|
||||
Cost matrixInputBufferLatencyCycles = 0;
|
||||
bool matrixPipeline = true;
|
||||
|
||||
Cost getMatrixComputeLatencyNs(Cost inputBitwidth) const {
|
||||
auto divideRoundedUp = [](Cost value, Cost width) {
|
||||
return value / width + (value % width != 0);
|
||||
};
|
||||
Cost inputTimes = divideRoundedUp(inputBitwidth, matrixInputResolutionBits);
|
||||
Cost inputPasses = divideRoundedUp(matrixRows, matrixInputParallelism);
|
||||
Cost outputPasses = divideRoundedUp(matrixColumns, matrixOutputParallelism);
|
||||
Cost readCycles = divideRoundedUp(matrixReadLatencyNs, matrixPeriodNs);
|
||||
Cost frontStage = matrixInputBufferLatencyCycles + matrixInputLatencyCycles
|
||||
+ readCycles + matrixSampleLatencyCycles;
|
||||
Cost backPipe = std::max(matrixOutputLatencyCycles,
|
||||
checkedAdd(matrixShiftLatencyCycles,
|
||||
matrixBufferLatencyCycles));
|
||||
Cost backStage = checkedAdd(
|
||||
checkedAdd(matrixOutputLatencyCycles, matrixShiftLatencyCycles),
|
||||
matrixBufferLatencyCycles);
|
||||
backStage = checkedAdd(backStage, checkedMultiply(outputPasses - 1, backPipe));
|
||||
Cost totalTimes = checkedMultiply(inputTimes, inputPasses);
|
||||
if (!matrixPipeline)
|
||||
return checkedMultiply(
|
||||
checkedMultiply(checkedAdd(frontStage, backStage), totalTimes),
|
||||
matrixPeriodNs);
|
||||
Cost stagePipe = std::max(frontStage, backStage);
|
||||
return checkedMultiply(
|
||||
checkedAdd(checkedAdd(frontStage, backStage),
|
||||
checkedMultiply(totalTimes - 1, stagePipe)),
|
||||
matrixPeriodNs);
|
||||
}
|
||||
|
||||
Cost getInterProcessorLatencyNs(size_t source,
|
||||
size_t destination) const {
|
||||
assert(source < processorCount && destination < processorCount
|
||||
|
||||
@@ -17,7 +17,7 @@ std::unique_ptr<mlir::Pass> createScheduleAndRealizeSpatialPass();
|
||||
std::unique_ptr<mlir::Pass> createScheduleAndRealizeSpatialPass(
|
||||
const SchedulingTarget& target,
|
||||
SpatialDataflowExportStage exportStage,
|
||||
size_t pipelineStages = 1);
|
||||
size_t pipelineStages = 1, bool periodicDiagnostics = false);
|
||||
}
|
||||
|
||||
std::unique_ptr<mlir::Pass> createONNXToSpatialPass();
|
||||
|
||||
@@ -37,6 +37,13 @@ add_pim_unittest(SpatialSchedulingTargetTest
|
||||
OMPimCompilerUtils
|
||||
)
|
||||
|
||||
add_pim_unittest(PeriodicDependenciesTest
|
||||
PeriodicDependenciesTest.cpp
|
||||
|
||||
LINK_LIBS PRIVATE
|
||||
OMPimCompilerUtils
|
||||
)
|
||||
|
||||
add_test(
|
||||
NAME PimHostReuseSynchronizationTest
|
||||
COMMAND "${PYTHON_EXECUTABLE}"
|
||||
@@ -49,3 +56,24 @@ add_test(NAME PimTargetConfigTest
|
||||
COMMAND "${PYTHON_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/test_PimTargetConfig.py"
|
||||
--compiler "$<TARGET_FILE:onnx-mlir>")
|
||||
set_tests_properties(PimTargetConfigTest PROPERTIES LABELS pim-unittest)
|
||||
|
||||
add_test(NAME PimPeriodicInstructionTest
|
||||
COMMAND "${PYTHON_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/test_PimPeriodicDiagnostic.py"
|
||||
--diagnostic "$<TARGET_FILE:raptor-periodic>")
|
||||
set_tests_properties(PimPeriodicInstructionTest PROPERTIES LABELS pim-unittest)
|
||||
add_dependencies(pim-unittest raptor-periodic)
|
||||
|
||||
add_test(NAME PimQueueSelectionTest
|
||||
COMMAND "${PYTHON_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/test_QueueSelection.py"
|
||||
--compiler "$<TARGET_FILE:onnx-mlir>")
|
||||
set_tests_properties(PimQueueSelectionTest PROPERTIES LABELS pim-unittest)
|
||||
|
||||
add_test(NAME PimEmissionTest
|
||||
COMMAND "${PYTHON_EXECUTABLE}" "${CMAKE_CURRENT_SOURCE_DIR}/test_PimEmission.py"
|
||||
--compiler "$<TARGET_FILE:onnx-mlir>")
|
||||
set_tests_properties(PimEmissionTest PROPERTIES LABELS pim-unittest)
|
||||
|
||||
add_pim_unittest(PimPipelineSelectionTest
|
||||
PimPipelineSelectionTest.cpp
|
||||
LINK_LIBS PRIVATE OMPimCompilerUtils
|
||||
)
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/PeriodicDependencies.hpp"
|
||||
|
||||
#include <cassert>
|
||||
#ifdef NDEBUG
|
||||
#error "Periodic dependency tests require assertions"
|
||||
#endif
|
||||
#include <cmath>
|
||||
#include <functional>
|
||||
#include <random>
|
||||
|
||||
using namespace onnx_mlir::spatial;
|
||||
|
||||
static double enumerateCycles(size_t count,
|
||||
const std::vector<PeriodicDependency> &edges) {
|
||||
double maximum = 0;
|
||||
std::vector<bool> visited(count);
|
||||
for (size_t start = 0; start < count; ++start) {
|
||||
std::function<void(size_t, Cost, Cost)> visit =
|
||||
[&](size_t source, Cost delay, Cost distance) {
|
||||
visited[source] = true;
|
||||
for (const auto &edge : edges) {
|
||||
if (edge.source != source)
|
||||
continue;
|
||||
Cost nextDelay = delay + edge.delayNs;
|
||||
Cost nextDistance = distance + edge.distance;
|
||||
if (edge.target == start && nextDistance)
|
||||
maximum = std::max(maximum,
|
||||
static_cast<double>(nextDelay) / nextDistance);
|
||||
else if (!visited[edge.target])
|
||||
visit(edge.target, nextDelay, nextDistance);
|
||||
}
|
||||
visited[source] = false;
|
||||
};
|
||||
visit(start, 0, 0);
|
||||
}
|
||||
return maximum;
|
||||
}
|
||||
|
||||
static void check(size_t count, const std::vector<PeriodicDependency> &edges,
|
||||
double expected) {
|
||||
std::string error;
|
||||
auto result = analyzePeriodicDependencies(count, edges, error);
|
||||
assert(mlir::succeeded(result) && error.empty());
|
||||
assert(result->recurrenceLowerNs <= expected + 1e-6);
|
||||
assert(result->recurrenceUpperNs >= expected - 1e-6);
|
||||
assert(result->recurrenceUpperNs - result->recurrenceLowerNs <= 1.000001);
|
||||
Cost delay = 0, distance = 0;
|
||||
for (size_t index : result->criticalCycle) {
|
||||
delay += edges[index].delayNs;
|
||||
distance += edges[index].distance;
|
||||
}
|
||||
if (expected > 1) {
|
||||
assert(distance > 0);
|
||||
assert(static_cast<double>(delay) / distance >= result->recurrenceLowerNs - 1e-6);
|
||||
}
|
||||
auto repeated = analyzePeriodicDependencies(count, edges, error);
|
||||
assert(repeated->criticalCycle == result->criticalCycle);
|
||||
assert(repeated->recurrenceUpperNs == result->recurrenceUpperNs);
|
||||
}
|
||||
|
||||
int main() {
|
||||
check(3, {{0, 1, 12, 0}, {1, 2, 7, 0}}, 0);
|
||||
check(3, {{0, 1, 12, 1}, {1, 2, 7, 1}}, 0);
|
||||
check(1, {{0, 0, 30, 2}}, 15);
|
||||
// Independent core periods remain parallel.
|
||||
check(2, {{0, 0, 10, 1}, {1, 1, 20, 1}}, 20);
|
||||
// Equal serial demand, different cross-core blocking recurrences.
|
||||
check(4, {{0, 1, 10, 0}, {2, 3, 10, 0},
|
||||
{1, 0, 0, 1}, {3, 2, 0, 1}}, 10);
|
||||
check(4, {{0, 1, 10, 0}, {2, 3, 10, 0},
|
||||
{1, 2, 0, 0}, {3, 0, 0, 1}}, 20);
|
||||
std::string error;
|
||||
assert(mlir::failed(analyzePeriodicDependencies(
|
||||
2, {{0, 1, 1, 0}, {1, 0, 1, 0}}, error)));
|
||||
assert(mlir::failed(analyzePeriodicDependencies(1, {{0, 1, 1, 1}}, error)));
|
||||
assert(mlir::failed(analyzePeriodicDependencies(
|
||||
1, {{0, 0, Cost{1} << 60, 1}}, error)));
|
||||
std::mt19937 random(812);
|
||||
for (unsigned sample = 0; sample < 300; ++sample) {
|
||||
std::vector<PeriodicDependency> edges;
|
||||
for (size_t source = 0; source < 6; ++source)
|
||||
for (size_t target = 0; target < 6; ++target)
|
||||
if (random() % 4 == 0)
|
||||
edges.push_back({source, target, random() % 101,
|
||||
source < target ? 0u : static_cast<unsigned>(1u + random() % 3)});
|
||||
check(6, edges, enumerateCycles(6, edges));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/Scheduling/ScheduleEvaluation.hpp"
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/ScheduledSpatialPasses.hpp"
|
||||
#include "llvm/Support/JSON.h"
|
||||
#include <cassert>
|
||||
#include <limits>
|
||||
using namespace onnx_mlir;
|
||||
using namespace onnx_mlir::spatial;
|
||||
int main() {
|
||||
mlir::MLIRContext context;
|
||||
context.allowUnregisteredDialects();
|
||||
mlir::OwningOpRef<mlir::ModuleOp> checkpoint = mlir::ModuleOp::create(mlir::UnknownLoc::get(&context));
|
||||
mlir::OperationState state(checkpoint->getLoc(), "test.compute");
|
||||
auto *op = mlir::Operation::create(state);
|
||||
checkpoint->getBody()->push_back(op);
|
||||
spatial::ComputeInstance instance{op, 3, 2};
|
||||
spatial::MergeScheduleResult schedule;
|
||||
schedule.dominanceOrderCompute.push_back(instance);
|
||||
schedule.computeToCpuMap[instance] = 7;
|
||||
schedule.computeToCpuSlotMap[instance] = 4;
|
||||
schedule.computeToAestMap[instance] = 42;
|
||||
schedule.isLastComputeOfCpu.insert(instance);
|
||||
schedule.cpuToLastComputeMap[7] = instance;
|
||||
for (int i = 0; i < 3; ++i) {
|
||||
mlir::IRMapping mapping;
|
||||
mlir::OwningOpRef<mlir::ModuleOp> trial = mlir::cast<mlir::ModuleOp>((*checkpoint)->clone(mapping));
|
||||
auto remapped = spatial::remapSpatialSchedule(schedule, mapping);
|
||||
auto copy = remapped.dominanceOrderCompute.front();
|
||||
assert(copy.op != op && copy.laneStart == 3 && copy.laneCount == 2);
|
||||
assert(remapped.computeToCpuMap.lookup(copy) == 7 && remapped.computeToCpuSlotMap.lookup(copy) == 4);
|
||||
assert(remapped.computeToAestMap.lookup(copy) == 42 && remapped.isLastComputeOfCpu.contains(copy));
|
||||
assert(remapped.cpuToLastComputeMap.lookup(7) == copy);
|
||||
copy.op->erase();
|
||||
assert(schedule.dominanceOrderCompute.front().op == op && op->getBlock() == checkpoint->getBody());
|
||||
}
|
||||
|
||||
auto score = [](double value, std::string id) { return WholeScheduleCost{value, value, {}, id, 0.021}; };
|
||||
std::string reason;
|
||||
auto original = score(100, "0"), faster = score(90, "1");
|
||||
auto select = [&](std::initializer_list<WholeScheduleCost> values) {
|
||||
return selectPipelineCandidate(llvm::ArrayRef<WholeScheduleCost>(values), reason);
|
||||
};
|
||||
assert(select({original, faster}) == 1);
|
||||
assert(select({original, score(97, "2")}) == 0);
|
||||
assert(select({original, score(105, "2")}) == 0);
|
||||
auto sensitive = faster; sensitive.alternative = 101;
|
||||
assert(select({original, sensitive}) == 0);
|
||||
assert(select({original, faster, score(91, "2")}) == 1);
|
||||
assert(select({original, faster, score(90, "1")}) == 1);
|
||||
auto unavailable = WholeScheduleCost{};
|
||||
assert(select({unavailable, faster}) == 0);
|
||||
assert(select({original, unavailable}) == 0);
|
||||
auto unknownUncertainty = faster;
|
||||
unknownUncertainty.relativeUncertainty.reset();
|
||||
assert(select({original, unknownUncertainty}) == 0);
|
||||
for (double value : {0., -1., std::numeric_limits<double>::infinity(), std::numeric_limits<double>::quiet_NaN()})
|
||||
assert(select({original, score(value, "2")}) == 0);
|
||||
for (int i = 0; i < 8; ++i) assert(select({original, faster}) == 1);
|
||||
|
||||
// The generic query accepts target-lowered IR without a target dialect API.
|
||||
for (double interval : {90., 97.}) {
|
||||
SchedulingTarget target;
|
||||
assert(!target.prepareScheduleCost);
|
||||
target.prepareScheduleCost = [interval](mlir::ModuleOp module,
|
||||
llvm::json::Object &) -> mlir::FailureOr<ScheduleCostQuery> {
|
||||
assert(module.getBody()->getOperations().size() == 1);
|
||||
ScheduleCostQuery query;
|
||||
query.identity = "static-program";
|
||||
query.evaluate = [interval](llvm::json::Object &) {
|
||||
return WholeScheduleCost{interval, interval, {}, "static-program", 0.021};
|
||||
};
|
||||
return query;
|
||||
};
|
||||
llvm::json::Object summary;
|
||||
auto query = target.prepareScheduleCost(*checkpoint, summary);
|
||||
assert(mlir::succeeded(query) && query->identity == "static-program");
|
||||
auto cost = query->evaluate(summary);
|
||||
assert(cost.eligible() && select({original, cost}) == (interval == 90. ? 1 : 0));
|
||||
assert(checkpoint->getBody()->front().getName().getStringRef() == "test.compute");
|
||||
}
|
||||
}
|
||||
@@ -19,6 +19,8 @@
|
||||
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/DeferredCommunicationScheduling.hpp"
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/DeferredBoundaryPlanning.hpp"
|
||||
#include "src/Accelerators/PIM/Compiler/PimScheduleDiagnostic.hpp"
|
||||
#include "src/Accelerators/PIM/Compiler/PimInstructionTiming.hpp"
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/DeferredTransferPlanning.hpp"
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/ScheduledComputeMaterialization.hpp"
|
||||
#include "src/Accelerators/PIM/Dialect/Spatial/Passes/Transforms/MergeComputeNodes/DeferredCommunicationRealization.hpp"
|
||||
@@ -1030,6 +1032,54 @@ static void checkPublicationMemoryTraffic(mlir::func::FuncOp function,
|
||||
assert(mlir::succeeded(communication));
|
||||
auto boundaries = buildDeferredBoundaryPlan(*shared, *communication, registers);
|
||||
assert(mlir::succeeded(boundaries));
|
||||
std::string before, after;
|
||||
llvm::raw_string_ostream beforeStream(before), afterStream(after);
|
||||
function.print(beforeStream);
|
||||
onnx_mlir::PimInstructionTiming timed;
|
||||
static_cast<SchedulingTarget &>(timed) = target;
|
||||
timed.peerPermissionThenSourceRead = true;
|
||||
timed.signalIssueNs = 1;
|
||||
timed.controlMessageBytes = 1;
|
||||
timed.sharedMemoryRequestLatencyNs.assign(4, 3);
|
||||
timed.sharedMemoryResponseLatencyNs.assign(4, 5);
|
||||
timed.interProcessorLatencyNs.assign(16, 2);
|
||||
for (size_t core = 0; core < 4; ++core)
|
||||
timed.interProcessorLatencyNs[core * 4 + core] = 0;
|
||||
auto report = getPeriodicScheduleReport(graph, *materialized, *shared, *boundaries, timed);
|
||||
assert(report.getString("status") == "estimated");
|
||||
assert(report.getBoolean("diagnostic_only") == true);
|
||||
assert(report.getInteger("boundary_write_bytes") == expected.writeBytes);
|
||||
assert(report.getInteger("boundary_read_bytes") == expected.readBytes);
|
||||
timed.sharedMemoryReadReturnsPayload = false;
|
||||
auto noResponse = getPeriodicScheduleReport(graph, *materialized, *shared, *boundaries, timed);
|
||||
assert(noResponse.getString("status") == "estimated");
|
||||
assert(*noResponse.getNumber("interval_estimate_ns") <= *report.getNumber("interval_estimate_ns"));
|
||||
bool checkedRunOwnership = false;
|
||||
for (size_t i = 0; i < shared->scheduled.size(); ++i)
|
||||
if (!materialized->materializedSchedules[i].runs.empty()) {
|
||||
auto &cores = shared->scheduled[i].cores;
|
||||
assert(cores.size() == 1);
|
||||
cores.push_back((cores.front() + 1) % target.processorCount);
|
||||
auto invalidRun = getPeriodicScheduleReport(graph, *materialized, *shared, *boundaries, timed);
|
||||
assert(invalidRun.getString("status") == "unavailable");
|
||||
assert(invalidRun.getString("reason") == "compact compute runs require one physical core");
|
||||
cores.pop_back();
|
||||
checkedRunOwnership = true;
|
||||
break;
|
||||
}
|
||||
assert(checkedRunOwnership);
|
||||
onnx_mlir::PimInstructionTiming incomplete = timed;
|
||||
incomplete.sharedMemoryRequestLatencyNs.clear();
|
||||
auto unavailable = getPeriodicScheduleReport(graph, *materialized,
|
||||
*shared, *boundaries, incomplete);
|
||||
assert(unavailable.getString("status") == "unavailable");
|
||||
incomplete = timed;
|
||||
incomplete.peerPermissionThenSourceRead = false;
|
||||
unavailable = getPeriodicScheduleReport(graph, *materialized,
|
||||
*shared, *boundaries, incomplete);
|
||||
assert(unavailable.getString("status") == "unavailable");
|
||||
function.print(afterStream);
|
||||
assert(before == after && "periodic diagnostics must not mutate scheduled IR");
|
||||
size_t stores = 0, signals = 0, readyWaits = 0, freeWaits = 0, releases = 0;
|
||||
llvm::DenseSet<int64_t> published;
|
||||
for (const auto &boundary : boundaries->boundaries)
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
// Test-only process guard. The compiler can read static inputs but cannot execute a child program.
|
||||
#define _GNU_SOURCE
|
||||
#include <linux/filter.h>
|
||||
#include <linux/seccomp.h>
|
||||
#include <stddef.h>
|
||||
#include <sys/prctl.h>
|
||||
#include <sys/syscall.h>
|
||||
#include <unistd.h>
|
||||
|
||||
__attribute__((constructor)) static void forbidExecution(void) {
|
||||
struct sock_filter instructions[] = {
|
||||
BPF_STMT(BPF_LD | BPF_W | BPF_ABS, offsetof(struct seccomp_data, nr)),
|
||||
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, SYS_execve, 0, 1),
|
||||
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL_PROCESS),
|
||||
BPF_JUMP(BPF_JMP | BPF_JEQ | BPF_K, SYS_execveat, 0, 1),
|
||||
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_KILL_PROCESS),
|
||||
BPF_STMT(BPF_RET | BPF_K, SECCOMP_RET_ALLOW),
|
||||
};
|
||||
struct sock_fprog program = {sizeof(instructions) / sizeof(instructions[0]), instructions};
|
||||
if (prctl(PR_SET_NO_NEW_PRIVS, 1, 0, 0, 0) || prctl(PR_SET_SECCOMP, SECCOMP_MODE_FILTER, &program))
|
||||
_exit(98);
|
||||
}
|
||||
|
||||
#include <spawn.h>
|
||||
pid_t fork(void) { _exit(99); }
|
||||
pid_t vfork(void) { _exit(99); }
|
||||
int posix_spawn(pid_t *pid, const char *path, const posix_spawn_file_actions_t *actions,
|
||||
const posix_spawnattr_t *attributes, char *const arguments[], char *const environment[]) { _exit(99); }
|
||||
int posix_spawnp(pid_t *pid, const char *path, const posix_spawn_file_actions_t *actions,
|
||||
const posix_spawnattr_t *attributes, char *const arguments[], char *const environment[]) { _exit(99); }
|
||||
|
||||
#include <dlfcn.h>
|
||||
#include <fcntl.h>
|
||||
#include <stdarg.h>
|
||||
#include <stdio.h>
|
||||
#include <string.h>
|
||||
static void rejectHistory(const char *path) {
|
||||
const char *forbidden[] = {"results_comparison.csv", "results_ablation.csv", "fast.json",
|
||||
"selection-frozen.json", "latency.json", ".trace"};
|
||||
for (size_t i = 0; i < sizeof(forbidden) / sizeof(forbidden[0]); ++i)
|
||||
if (strstr(path, forbidden[i])) _exit(97);
|
||||
}
|
||||
#define AUDIT_OPEN(name) \
|
||||
int name(const char *path, int flags, ...) { \
|
||||
rejectHistory(path); \
|
||||
mode_t mode = 0; \
|
||||
if (flags & O_CREAT) { va_list arguments; va_start(arguments, flags); mode = va_arg(arguments, int); va_end(arguments); } \
|
||||
int (*original)(const char *, int, ...) = dlsym(RTLD_NEXT, #name); \
|
||||
return original(path, flags, mode); \
|
||||
}
|
||||
AUDIT_OPEN(open)
|
||||
AUDIT_OPEN(open64)
|
||||
#define AUDIT_FOPEN(name) \
|
||||
FILE *name(const char *path, const char *mode) { \
|
||||
rejectHistory(path); \
|
||||
FILE *(*original)(const char *, const char *) = dlsym(RTLD_NEXT, #name); \
|
||||
return original(path, mode); \
|
||||
}
|
||||
AUDIT_FOPEN(fopen)
|
||||
AUDIT_FOPEN(fopen64)
|
||||
@@ -1,5 +1,5 @@
|
||||
import argparse
|
||||
from collections import defaultdict, deque
|
||||
from collections import Counter, defaultdict, deque
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
@@ -13,6 +13,7 @@ from types import SimpleNamespace
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
sys.path.insert(0, str(REPO / "validation/tools/pim/pimcomp/correctness/synchronization"))
|
||||
import global_memory as sync
|
||||
from raptor_validation.pimsim_nn import export_raptor_pimsim_artifact
|
||||
|
||||
|
||||
def read_events(artifact):
|
||||
@@ -121,6 +122,40 @@ def check_lifetimes(streams):
|
||||
return shared, stalled_reader
|
||||
|
||||
|
||||
def check_periodic_estimate(report, streams):
|
||||
assert report["diagnostic_only"] and report["status"] == "estimated", report
|
||||
assert 0 <= report["recurrence_upper_ns"] - report["recurrence_lower_ns"] <= 1
|
||||
assert report["critical_cycle_distance"] > 0
|
||||
ratio = report["critical_cycle_delay_ns"] / report["critical_cycle_distance"]
|
||||
assert report["recurrence_lower_ns"] <= ratio <= report["recurrence_upper_ns"]
|
||||
signals, waits, memory = Counter(), {}, Counter()
|
||||
# The oracle exporter converts this fp32 fixture's addresses to int8 bytes.
|
||||
host_bytes = report["native_host_buffer_bytes"] // 4
|
||||
for core, events in streams.items():
|
||||
for _, op, *args in events:
|
||||
if op == 32:
|
||||
target, register, _ = args
|
||||
signals[core, target, register] += 1
|
||||
elif op == 31:
|
||||
_, register, count = args
|
||||
assert (core, register) not in waits
|
||||
waits[core, register] = count
|
||||
elif op in (25, 26) and args[0] < host_bytes:
|
||||
memory[op, "requests"] += 1
|
||||
memory[op, "bytes"] += args[1]
|
||||
planned = Counter()
|
||||
for event in report["event_registers"]:
|
||||
key = event["core"], event["register"]
|
||||
assert waits.pop(key) == event["expected"]
|
||||
for source in event["sources"]:
|
||||
planned[source["core"], *key] += source["count"]
|
||||
assert not waits and planned == signals
|
||||
for op, direction in ((25, "read"), (26, "write")):
|
||||
for quantity in ("bytes", "requests"):
|
||||
assert report[f"boundary_{direction}_{quantity}"] == memory[op, quantity], (
|
||||
direction, quantity, report[f"boundary_{direction}_{quantity}"], memory[op, quantity])
|
||||
|
||||
|
||||
def run_functional(artifact, inputs, references, outputs, out, policy, seed, reader=None):
|
||||
out.mkdir()
|
||||
command = [str(sync.RUST_BINARY), "--folder", str(artifact),
|
||||
@@ -158,13 +193,17 @@ def main():
|
||||
"--maccel=PIM", "--EmitPimCodegen", "--crossbar-size=128",
|
||||
f"--crossbar-count={weights}", f"--core-count={cores}", "--pipeline=4",
|
||||
f"--pim-target-config={sync.CONFIG_ROOT / arch / 'throughput_config_1000ms.json'}",
|
||||
"--mlir-disable-threading", "--pim-emit-json", "--pim-detect-communication-deadlock"]
|
||||
"--mlir-disable-threading", "--pim-periodic-diagnostics", "--pim-emit-json", "--pim-detect-communication-deadlock"]
|
||||
compiled = subprocess.run(command, capture_output=True, text=True,
|
||||
timeout=120, env=dict(os.environ, OMP_NUM_THREADS="1"))
|
||||
assert compiled.returncode == 0, compiled.stdout + compiled.stderr
|
||||
artifact = out / "pim"
|
||||
streams = read_events(artifact)
|
||||
shared, reader = check_lifetimes(streams)
|
||||
report = json.loads((out / "reports/pipeline_periodic.json").read_text())
|
||||
oracle = out / "pimsim_nn"
|
||||
export_raptor_pimsim_artifact(artifact, oracle)
|
||||
check_periodic_estimate(report, read_events(oracle))
|
||||
for policy, seed in (("greedy", 0), ("randomized", 1), ("randomized", 7)):
|
||||
run_functional(artifact, inputs, references, outputs,
|
||||
out / f"{policy}-{seed}", policy, seed)
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
"""Intermediate emission must stop before unsupported backend transformations."""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import tempfile
|
||||
|
||||
import onnx
|
||||
from onnx import TensorProto, helper
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--compiler", required=True)
|
||||
args = parser.parse_args()
|
||||
with tempfile.TemporaryDirectory(prefix="raptor-emission-") as temporary:
|
||||
root = Path(temporary)
|
||||
|
||||
def model(name, elements):
|
||||
tensors = [helper.make_tensor_value_info(n, TensorProto.FLOAT, [1, elements])
|
||||
for n in ("x", "y", "z")]
|
||||
graph = helper.make_graph([helper.make_node("Add", ["x", "y"], ["z"])],
|
||||
name, tensors[:2], tensors[2:])
|
||||
value = helper.make_model(graph, opset_imports=[helper.make_opsetid("", 18)])
|
||||
value.ir_version = 10
|
||||
path = root / f"{name}.onnx"
|
||||
onnx.save(value, path)
|
||||
return path
|
||||
|
||||
small = model("small", 16)
|
||||
# This static type is legal in Spatial. PIM's signed byte-size operand cannot encode it.
|
||||
large = model("large", 536870912)
|
||||
for source, targets in ((small, ("EmitSpatial", "EmitPim", "EmitPimBufferized", "EmitPimCodegen")),
|
||||
(large, ("EmitSpatial", "EmitPim"))):
|
||||
for target in targets:
|
||||
out = root / source.stem / target
|
||||
out.mkdir(parents=True)
|
||||
result = subprocess.run([args.compiler, str(source), "-o", str(out / "model"),
|
||||
"--maccel=PIM", f"--{target}", "--pipeline=2", "--core-count=168",
|
||||
"--crossbar-count=96", "--crossbar-size=128", "--mlir-disable-threading", "--pim-dump-ir",
|
||||
f"--pim-target-config={REPO}/validation/pimsim_configs/pimcomp/arch-a/throughput_config_1000ms.json"],
|
||||
capture_output=True, text=True, timeout=30, env=dict(os.environ, OMP_NUM_THREADS="1"))
|
||||
output = result.stdout + result.stderr
|
||||
if source == large and target == "EmitPim":
|
||||
assert result.returncode != 0 and "host input copy byte size is outside representable range" in output, output
|
||||
continue
|
||||
assert result.returncode == 0, output
|
||||
if target == "EmitPimCodegen":
|
||||
assert (out / "pim/config.json").exists()
|
||||
report = json.loads((out / "reports/pipeline_selection.json").read_text())
|
||||
assert report["generated_candidates"] == 1
|
||||
assert report["single_candidate_analysis_skipped"]
|
||||
assert report["analyzed_candidates"] == report["lowered_candidates"] == 0
|
||||
assert report["unavailable_estimates"] == 0
|
||||
assert not report["reused_lowered_candidate"]
|
||||
assert "no cost comparison required" in report["reason"]
|
||||
else:
|
||||
assert "intermediate emission retains the original schedule" in output
|
||||
assert not (out / "reports/pipeline_selection.json").exists()
|
||||
assert not (out / "reports/pipeline_periodic.json").exists()
|
||||
assert not (out / "pim").exists()
|
||||
name = {"EmitSpatial": "spatial4_scheduled", "EmitPim": "pim0",
|
||||
"EmitPimBufferized": "pim1_buff"}[target]
|
||||
ir = (out / f"dialects/{name}.mlir").read_text()
|
||||
if target == "EmitSpatial":
|
||||
assert "spat.scheduled_compute" in ir and "pim.core" not in ir
|
||||
assert not (out / "dialects/pim0.mlir").exists()
|
||||
elif target == "EmitPim":
|
||||
assert "pim.core" in ir and "tensor<" in ir
|
||||
assert not (out / "dialects/pim1_buff.mlir").exists()
|
||||
else:
|
||||
assert "pim.core" in ir and "memref<" in ir
|
||||
out = root / "explicit-diagnostic"
|
||||
out.mkdir()
|
||||
result = subprocess.run([args.compiler, str(small), "-o", str(out / "model"),
|
||||
"--maccel=PIM", "--EmitSpatial", "--pipeline=2", "--core-count=168",
|
||||
"--crossbar-count=96", "--crossbar-size=128", "--mlir-disable-threading",
|
||||
"--pim-periodic-diagnostics",
|
||||
f"--pim-target-config={REPO}/validation/pimsim_configs/pimcomp/arch-a/throughput_config_1000ms.json"],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
assert (out / "reports/pipeline_periodic.json").exists()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,334 @@
|
||||
import argparse
|
||||
import copy
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
import struct
|
||||
import subprocess
|
||||
import tempfile
|
||||
|
||||
|
||||
def earliest_events(report):
|
||||
graph = report['timing_graph']
|
||||
incoming = [0] * len(graph['events'])
|
||||
outgoing = [[] for _ in incoming]
|
||||
for edge in graph['dependencies']:
|
||||
if edge['distance'] == 0:
|
||||
incoming[edge['target']] += 1
|
||||
outgoing[edge['source']].append(edge)
|
||||
ready = [i for i, count in enumerate(incoming) if count == 0]
|
||||
times = [0] * len(incoming)
|
||||
for source in ready:
|
||||
for edge in outgoing[source]:
|
||||
target = edge['target']
|
||||
times[target] = max(times[target], times[source] + edge['delay_ns'])
|
||||
incoming[target] -= 1
|
||||
if incoming[target] == 0:
|
||||
ready.append(target)
|
||||
assert len(ready) == len(times), 'within-generation graph is cyclic'
|
||||
return graph, times
|
||||
|
||||
|
||||
def record(op, arg=0, a=0, b=0, length=0):
|
||||
return struct.pack('<BBBBiiii', op, 0, 0, 0, arg, a, b, length)
|
||||
|
||||
|
||||
def check_controller(run):
|
||||
scalar = record(1)
|
||||
load = lambda n: record(25, length=n)
|
||||
store = lambda n: record(26, length=n)
|
||||
work = record(27, length=256)
|
||||
barrier = lambda peer: [record(32, arg=peer, a=0), record(31, a=0, b=1)]
|
||||
staggered = [barrier(1) + [load(64), work], barrier(0) + [work, load(64)]]
|
||||
burst = [barrier(1) + [work, load(64)], barrier(0) + [work, load(64)]]
|
||||
a = run(staggered, controller=True, issue=True)
|
||||
b = run(burst, controller=True, issue=True)
|
||||
for report in (a, b):
|
||||
assert report['memory_controller']['status'] == 'estimated'
|
||||
assert report['global_read_bytes'] == 128
|
||||
assert report['global_memory_service_ns'] == 160
|
||||
assert a['memory_controller']['interval_ns'] < b['memory_controller']['interval_ns']
|
||||
assert b['memory_controller'] == run(burst, controller=True, issue=True)['memory_controller']
|
||||
mixed = run([[scalar, load(33), scalar], [scalar, store(128), scalar]],
|
||||
controller=True, issue=True)
|
||||
queued = mixed['memory_controller']
|
||||
assert mixed['global_memory_service_ns'] == 280
|
||||
assert queued['interval_ns'] >= 280
|
||||
assert queued['completed_generations'] >= 4
|
||||
assert any(core['queue_wait_ns'] > 0 for core in queued['per_core'])
|
||||
sizes = run([[scalar, load(1), scalar], [scalar, load(65), scalar]], controller=True, issue=True)
|
||||
assert sizes['global_memory_service_ns'] == 160
|
||||
assert sizes['memory_controller']['interval_ns'] >= 160
|
||||
delayed = [barrier(1) + [load(8192)], barrier(0) + [load(64)]]
|
||||
behind = run(delayed, controller=True, issue=True)
|
||||
assert behind['memory_controller']['status'] == 'estimated'
|
||||
reader = next(core for core in behind['memory_controller']['per_core'] if core['core'] == 1)
|
||||
assert reader['queue_wait_ns'] > 0
|
||||
zero = run([barrier(1) + [record(27, length=128), load(0)], barrier(0) + [store(128)]],
|
||||
controller=True, issue=True)
|
||||
reader = next(core for core in zero['memory_controller']['per_core'] if core['core'] == 0)
|
||||
assert reader['queue_wait_ns'] > 0 # Zero service still queues behind an active request.
|
||||
bounded = run([[scalar, scalar], [scalar, load(65536), scalar]], controller=True, issue=True)
|
||||
assert bounded['memory_controller']['status'] == 'unavailable'
|
||||
shared = [[record(31, a=0, b=2), store(128), record(32, arg=1, a=1), record(32, arg=2, a=1)],
|
||||
[record(32, arg=0, a=0), record(31, a=1, b=1), load(128), record(29, arg=2, length=64)],
|
||||
[record(32, arg=0, a=0), record(31, a=1, b=1), load(128), record(30, arg=1, length=64)]]
|
||||
publication = run(shared, controller=True, issue=True)
|
||||
assert publication['memory_controller']['status'] == 'estimated'
|
||||
assert publication['global_write_requests'] == 1 and publication['global_read_requests'] == 2
|
||||
assert publication['direct_requests'] == 1
|
||||
return [staggered, burst, [[scalar, load(33), scalar], [scalar, store(128), scalar]],
|
||||
[[scalar, load(1), scalar], [scalar, load(65), scalar]], delayed, shared]
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('--diagnostic', required=True)
|
||||
parser.add_argument('--pimsim', help='Optional ChipTest executable for matched fast-mode timing checks')
|
||||
args = parser.parse_args()
|
||||
repo = Path(__file__).resolve().parents[2]
|
||||
with tempfile.TemporaryDirectory(prefix='raptor-periodic-isa-') as temporary:
|
||||
root = Path(temporary)
|
||||
for arch in ('arch-a', 'arch-b'):
|
||||
config = json.loads((repo / 'validation/pimsim_configs/pimcomp' / arch /
|
||||
'throughput_config_1000ms.json').read_text())
|
||||
config['chip_config']['core_cnt'] = 3
|
||||
latency = {str(i): {str(j): int(i != j) for j in range(3)} for i in range(3)}
|
||||
for i in range(3):
|
||||
latency[str(i)]['-10'] = 2
|
||||
latency['-10'] = {str(i): 2 for i in range(3)}
|
||||
network = root / 'network.json'
|
||||
network.write_text(json.dumps(dict(latency=latency)))
|
||||
config['chip_config']['network_config']['net_config_file_path'] = str(network)
|
||||
path = root / 'hardware.json'
|
||||
path.write_text(json.dumps(config))
|
||||
hardware = config['chip_config']['core_config']
|
||||
command = [args.diagnostic, str(root), f'--pim-target-config={path}',
|
||||
'--core-count=3', '--crossbar-size=128',
|
||||
f'--crossbar-count={hardware["matrix_config"]["xbar_array_count"]}']
|
||||
|
||||
def run(streams, *, issue=False, queues=True, valid=True, controller=False, oracle=False):
|
||||
for artifact in root.glob('core_*.pim'):
|
||||
artifact.unlink()
|
||||
(root / 'config.json').write_text(json.dumps(dict(core_cnt=len(streams),
|
||||
array_group_map={f'core{i}': [i + 1] for i in range(len(streams))})))
|
||||
for i, instructions in enumerate(streams):
|
||||
(root / f'core_{i}.pim').write_bytes(struct.pack('<4sII', b'PIMB', 1,
|
||||
len(instructions)) + b''.join(instructions))
|
||||
before = {p.name: hashlib.sha256(p.read_bytes()).hexdigest() for p in root.glob('core_*.pim')}
|
||||
result = subprocess.run(command + [f'--model-issue={int(issue)}',
|
||||
f'--model-network-queues={int(queues)}',
|
||||
f'--model-memory-controller={int(controller)}', f'--oracle-precision={int(oracle)}'], capture_output=True, text=True)
|
||||
assert (result.returncode == 0) == valid, result.stdout + result.stderr
|
||||
assert before == {p.name: hashlib.sha256(p.read_bytes()).hexdigest() for p in root.glob('core_*.pim')}
|
||||
if not valid:
|
||||
return result
|
||||
report = json.loads(result.stdout)
|
||||
assert report['diagnostic_only'] and not report['selection_eligible']
|
||||
assert report['status'] == 'estimated'
|
||||
if report['recurrence_lower_ns'] > 1:
|
||||
ratio = report['critical_cycle_delay_ns'] / report['critical_cycle_distance']
|
||||
assert report['recurrence_lower_ns'] <= ratio <= report['recurrence_upper_ns']
|
||||
return report
|
||||
|
||||
bw = record(8, a=8, b=8)
|
||||
load = lambda n: record(25, length=n)
|
||||
store = lambda n: record(26, length=n)
|
||||
send = lambda peer: record(29, arg=peer, length=64)
|
||||
recv = lambda peer: record(30, arg=peer, length=64)
|
||||
sync = lambda peer, reg: record(32, arg=peer, a=reg)
|
||||
wait = lambda reg, n=1: record(31, a=reg, b=n)
|
||||
controller_fixtures = check_controller(run)
|
||||
|
||||
native = run([[record(8, a=32, b=32), load(132), record(10, length=65), store(132)]],
|
||||
issue=True, controller=True, oracle=True)
|
||||
exported = run([[bw, load(33), record(10, length=65), store(33)]], issue=True, controller=True)
|
||||
for key in ('global_read_bytes', 'global_write_bytes', 'global_read_requests',
|
||||
'global_write_requests', 'global_memory_service_ns'):
|
||||
assert native[key] == exported[key], key
|
||||
assert native['memory_controller'] == exported['memory_controller']
|
||||
run([[record(8, a=16, b=16)]], oracle=True, valid=False)
|
||||
run([[load(3)]], oracle=True, valid=False)
|
||||
|
||||
# One scan counts input, constant, and final-output accesses exactly.
|
||||
memory = run([[load(33), load(33), store(33)]])
|
||||
assert memory['global_read_requests'] == 2
|
||||
assert memory['global_read_bytes'] == 66 and memory['global_write_bytes'] == 33
|
||||
assert memory['global_memory_service_ns'] == 260
|
||||
assert memory['per_core'][0]['serial_work_ns'] == 348
|
||||
folded = run([[load(66), store(33)]])
|
||||
assert folded['global_memory_service_ns'] == 220
|
||||
|
||||
scalar = run([[record(1)] * 10], issue=True)
|
||||
assert scalar['per_core'][0]['serial_work_ns'] == 10 * hardware['period']
|
||||
vector = run([[bw, record(10, length=65)]])
|
||||
vector_cost = 2 * 2 * 20 + 3 * 4 + 2 * 30
|
||||
assert vector['per_core'][0]['resource_service_ns'][1] == vector_cost
|
||||
issued = run([[bw, record(10, length=65)]], issue=True)
|
||||
assert issued['per_core'][0]['serial_work_ns'] == vector_cost + 2 * hardware['period']
|
||||
repeated = run([[bw] + [record(10, length=65)] * 10000])
|
||||
assert repeated['per_core'][0]['serial_work_ns'] == 10000 * vector_cost
|
||||
assert repeated['events'] < 10
|
||||
|
||||
matrix = run([[bw, record(9, arg=2)], [bw, record(9, arg=2)]])
|
||||
assert len(matrix['per_core']) == 2
|
||||
assert matrix['per_core'][0]['instructions'] == matrix['per_core'][1]['instructions'] == 2
|
||||
assert matrix['per_core'][0]['resource_service_ns'][0] > 0
|
||||
assert matrix['per_core'][1]['serial_work_ns'] - matrix['per_core'][0]['serial_work_ns'] == 60
|
||||
|
||||
# Equal compute and traffic, different placement of a baseline load.
|
||||
reader = [bw, recv(0), record(9, arg=32), load(1024), send(0)]
|
||||
overlap = run([[send(1), load(1024), recv(1)], reader])
|
||||
serial = run([[load(1024), send(1), recv(1)], reader])
|
||||
assert overlap['global_memory_service_ns'] == serial['global_memory_service_ns']
|
||||
assert overlap['interval_estimate_ns'] < serial['interval_estimate_ns']
|
||||
|
||||
output = run([[send(1)], [recv(0), store(64)]])
|
||||
no_output = run([[send(1)], [recv(0)]])
|
||||
assert output['global_write_requests'] == 1 and output['global_write_bytes'] == 64
|
||||
assert output['direct_requests'] == 1 and output['direct_bytes'] == 64
|
||||
assert output['interval_estimate_ns'] > no_output['interval_estimate_ns']
|
||||
|
||||
# Permission precedes the complete source read; enqueue is immediate.
|
||||
delayed = run([[send(1)], [record(27, length=128), recv(0)]])
|
||||
graph, times = earliest_events(delayed)
|
||||
|
||||
def event_id(core, pc, kind, occurrence=0):
|
||||
return [i for i, event in enumerate(graph['events'])
|
||||
if event == dict(core=core, pc=pc, kind=kind)][occurrence]
|
||||
|
||||
request = event_id(0, 0, 'delivery', 0)
|
||||
permission = event_id(1, 1, 'delivery')
|
||||
enqueue = event_id(0, 0, 'payload-enqueue')
|
||||
delivery = event_id(0, 0, 'delivery', 1)
|
||||
complete = event_id(0, 0, 'send-complete')
|
||||
received = event_id(1, 1, 'receive-complete')
|
||||
assert times[permission] >= max(times[request], 60) + 1
|
||||
assert times[enqueue] == times[permission] + 20
|
||||
assert times[complete] == times[enqueue]
|
||||
assert times[delivery] == times[enqueue] + 2
|
||||
assert times[received] == times[delivery] + 30
|
||||
assert any(edge['source'] == delivery and edge['distance'] == 1
|
||||
for edge in graph['dependencies'])
|
||||
|
||||
# An occupied source FIFO delays request dispatch and permission.
|
||||
busy_routes = copy.deepcopy(latency)
|
||||
busy_routes['0']['2'] = 100
|
||||
network.write_text(json.dumps(dict(latency=busy_routes)))
|
||||
graph, times = earliest_events(run([[sync(2, 0), send(1)], [recv(0)], [wait(0)]]))
|
||||
dispatch = event_id(0, 1, 'fifo-dispatch', 0)
|
||||
enqueue = event_id(0, 1, 'payload-enqueue')
|
||||
complete = event_id(0, 1, 'send-complete')
|
||||
assert times[dispatch] == 100
|
||||
assert times[enqueue] == times[complete] == 122
|
||||
network.write_text(json.dumps(dict(latency=latency)))
|
||||
pairs = run([[send(1), send(1)], [recv(0), recv(0)]])
|
||||
assert pairs['direct_requests'] == 2 and pairs['direct_bytes'] == 128
|
||||
interleaved = run([[wait(0), send(1), sync(1, 1)],
|
||||
[sync(0, 0), recv(0), wait(1)]])
|
||||
assert interleaved['direct_requests'] == 1
|
||||
|
||||
# Exact-count handshakes repeat safely, including shared storage reuse.
|
||||
shared = run([[wait(0, 2), store(128), sync(1, 1), sync(2, 1)],
|
||||
[sync(0, 0), wait(1), load(128)],
|
||||
[sync(0, 0), wait(1), load(128)]])
|
||||
assert shared['global_write_requests'] == 1 and shared['global_read_requests'] == 2
|
||||
assert shared['global_write_bytes'] == 128 and shared['global_read_bytes'] == 256
|
||||
stalled = run([[wait(0, 2), store(128), sync(1, 1), sync(2, 1)],
|
||||
[sync(0, 0), wait(1), load(128)],
|
||||
[bw, sync(0, 0), wait(1), load(128), record(10, length=4096)]])
|
||||
assert stalled['interval_estimate_ns'] > shared['interval_estimate_ns']
|
||||
|
||||
notices = [[sync(1, 0), sync(1, 0), wait(1)], [wait(0, 2), sync(0, 1)]]
|
||||
queued = run(notices)
|
||||
unqueued = run(notices, queues=False)
|
||||
assert queued['interval_estimate_ns'] >= unqueued['interval_estimate_ns']
|
||||
# Delivery starts when SYNC enqueues, before its local completion.
|
||||
slower_clock = copy.deepcopy(config)
|
||||
slower_clock['chip_config']['core_config']['period'] = 8
|
||||
slower_clock['chip_config']['core_config']['local_memory_config'].pop('period')
|
||||
path.write_text(json.dumps(slower_clock))
|
||||
signal = run([[sync(1, 0), wait(1)], [wait(0), sync(0, 1)]])
|
||||
assert signal['interval_estimate_ns'] < 18
|
||||
local = run([[record(27, length=65)]])
|
||||
assert local['per_core'][0]['serial_work_ns'] == 60
|
||||
independent_vector = run([[bw, record(10, length=65)]])
|
||||
assert independent_vector['per_core'][0]['serial_work_ns'] == vector_cost
|
||||
different_memory_clock = copy.deepcopy(slower_clock)
|
||||
different_memory_clock['chip_config']['core_config']['local_memory_config']['period'] = 3
|
||||
different_memory_clock['chip_config']['core_config']['vector_latency_cycle'] = 7
|
||||
path.write_text(json.dumps(different_memory_clock))
|
||||
changed_vector = run([[bw, record(10, length=65)]])
|
||||
assert changed_vector['per_core'][0]['serial_work_ns'] == (vector_cost - 12) * 3 + 21
|
||||
path.write_text(json.dumps(config))
|
||||
|
||||
zero_service = copy.deepcopy(config)
|
||||
zero_service['chip_config']['global_memory_config']['read_latency_cycle'] = 0
|
||||
zero_routes = copy.deepcopy(latency)
|
||||
zero_routes['0']['-10'] = 0
|
||||
zero_routes['0']['1'] = 100
|
||||
network.write_text(json.dumps(dict(latency=zero_routes)))
|
||||
path.write_text(json.dumps(zero_service))
|
||||
prefix = [[load(0), sync(1, 0), load(64), wait(1)], [wait(0), sync(0, 1)]]
|
||||
drained = run(prefix)
|
||||
undrained = run(prefix, queues=False)
|
||||
assert drained['interval_estimate_ns'] > undrained['interval_estimate_ns'] + 20
|
||||
network.write_text(json.dumps(dict(latency=latency)))
|
||||
path.write_text(json.dumps(config))
|
||||
|
||||
if args.pimsim:
|
||||
# Scalar padding avoids the oracle's single-instruction fetch loop.
|
||||
# Golden pairs are (rounded graph estimate, oracle interval), in ns.
|
||||
# Their difference exposes clock-phase/frontend costs, without fitting the model.
|
||||
fixtures = [([[send(1)], [recv(0)]], {}, {}, (56, 57)),
|
||||
([[send(1)], [record(27, length=1024), recv(0)]], {}, {}, (537, 539)),
|
||||
([[send(1)], [recv(0)]], {'memory_period': 3}, {}, (157, 157)),
|
||||
([[send(1)], [recv(0)]], {}, {'peer': 100}, (322, 321)),
|
||||
([[sync(2, 0), send(1)], [recv(0)], [wait(0)]], {}, {'third': 100}, (125, 126)),
|
||||
([[send(1), send(1)], [recv(0), recv(0)]], {}, {}, (111, 112)),
|
||||
([[wait(0), send(1), sync(1, 1)], [sync(0, 0), recv(0), wait(1)]], {}, {}, (60, 65)),
|
||||
([[bw, record(10, length=65)]], {'period': 8, 'memory_period': 3, 'vector': 7}, {}, (473, 488)),
|
||||
([[bw, record(10, length=65)]], {'period': 5, 'memory_period': 2, 'vector': 11}, {}, (333, 340))]
|
||||
fixtures = [(*fixture, False) for fixture in fixtures]
|
||||
controller_expected = [(237, 243), (316, 320), (281, 280), (185, 186), (14145, 14169), (690, 698)]
|
||||
assert len(controller_fixtures) == len(controller_expected)
|
||||
fixtures += [(streams, {}, {}, expected, True) for streams, expected in
|
||||
zip(controller_fixtures, controller_expected)]
|
||||
for streams, clocks, links, expected, controller in fixtures:
|
||||
matched = copy.deepcopy(config)
|
||||
core = matched['chip_config']['core_config']
|
||||
core['period'] = clocks.get('period', 1)
|
||||
core['local_memory_config']['period'] = clocks.get('memory_period', 1)
|
||||
core['vector_latency_cycle'] = clocks.get('vector', 4)
|
||||
matched['sim_config']['sim_time'] = 1
|
||||
path.write_text(json.dumps(matched))
|
||||
routes = copy.deepcopy(latency)
|
||||
routes['0']['1'] = links.get('peer', 1)
|
||||
routes['0']['2'] = links.get('third', 1)
|
||||
network.write_text(json.dumps(dict(latency=routes,
|
||||
energy={s: {d: 0 for d in row} for s, row in routes.items()})))
|
||||
program = streams if controller else [[record(1)] + stream + [record(1)] for stream in streams]
|
||||
prediction = run(program, issue=True, controller=controller)
|
||||
result = subprocess.run([args.pimsim, str(root), str(path), '--gui=false', '--fast'],
|
||||
capture_output=True, text=True, timeout=30)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
assert 'Fast throughput converged' in result.stdout
|
||||
measured = float(re.search(r'average latency:\s+(\S+)', result.stdout)[1]) * 1e6
|
||||
estimate = prediction['memory_controller']['interval_ns'] if controller else prediction['interval_estimate_ns']
|
||||
assert (round(estimate), round(measured)) == expected
|
||||
path.write_text(json.dumps(config))
|
||||
network.write_text(json.dumps(dict(latency=latency)))
|
||||
|
||||
run([[send(1), recv(1)], [send(0), recv(0)]], valid=False)
|
||||
run([[sync(1, 0)], [wait(0, 2)]], valid=False)
|
||||
run([[record(21, length=1)]], valid=False)
|
||||
run([[]], valid=False)
|
||||
unsupported = copy.deepcopy(config)
|
||||
unsupported['chip_config']['core_config']['rob_size'] = 2
|
||||
path.write_text(json.dumps(unsupported))
|
||||
run([[load(33)]], valid=False)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -45,6 +45,25 @@ def main():
|
||||
if not valid:
|
||||
assert "Pim target config" in result.stdout, result.stdout
|
||||
|
||||
# Optional diagnostic routes must not change legacy compilation support.
|
||||
network_path = Path(network["net_config_file_path"])
|
||||
for route in (None, 1e100):
|
||||
candidate = copy.deepcopy(config)
|
||||
candidate["chip_config"]["core_config"]["global_memory_switch_id"] = 999
|
||||
routes = json.loads(network_path.read_text())
|
||||
if route is not None:
|
||||
routes["latency"]["999"] = {}
|
||||
for core in range(168):
|
||||
routes["latency"][str(core)]["999"] = route
|
||||
routes["latency"]["999"][str(core)] = route
|
||||
route_path = root / "routes.json"
|
||||
route_path.write_text(json.dumps(routes))
|
||||
candidate["chip_config"]["network_config"]["net_config_file_path"] = str(route_path)
|
||||
path.write_text(json.dumps(candidate))
|
||||
result = subprocess.run(command, stdout=subprocess.PIPE,
|
||||
stderr=subprocess.STDOUT, text=True)
|
||||
assert result.returncode == 0, result.stdout
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
"""Production selection must use static compiler inputs only."""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import tempfile
|
||||
|
||||
REPO = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def check_selected_program(out):
|
||||
report = json.loads((out / "reports/pipeline_selection.json").read_text())
|
||||
program = json.loads((out / "pim/config.json").read_text())
|
||||
digest = hashlib.sha256()
|
||||
for core in range(program["core_cnt"]):
|
||||
digest.update((out / f"pim/core_{core}.pim").read_bytes())
|
||||
for count in program["array_group_map"][f"core{core}"]:
|
||||
digest.update(f"{count},".encode())
|
||||
digest.update(b";")
|
||||
assert digest.hexdigest().upper() == report["candidates"][report["selection"]]["identity"], \
|
||||
"emitted program differs from the analyzed winner"
|
||||
assert report["reused_lowered_candidate"]
|
||||
for name in ("spatial3_scheduled_no_comm.txt", "memory_report.txt"):
|
||||
assert (out / "reports" / name).stat().st_size > 0, name
|
||||
return report
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--compiler", required=True)
|
||||
args = parser.parse_args()
|
||||
compiler = str(Path(args.compiler).resolve())
|
||||
model = REPO / "validation/networks/pimcomp_models/googlenet/googlenet-12-pimsim-nn.onnx"
|
||||
with tempfile.TemporaryDirectory(prefix="raptor-static-selection-") as temporary:
|
||||
root = Path(temporary)
|
||||
history = root / "validation/tools/pim/pimcomp/compare/results_comparison.csv"
|
||||
history.parent.mkdir(parents=True)
|
||||
profile = root / "profiles/latency.json"
|
||||
profile.parent.mkdir()
|
||||
trap = root / "bin"
|
||||
trap.mkdir()
|
||||
for name in ("ChipTest", "pimsim", "raptor-periodic", "perf"):
|
||||
path = trap / name
|
||||
path.write_text("#!/bin/sh\necho forbidden execution >> '" + str(root / "executed") + "'\nexit 99\n")
|
||||
path.chmod(0o755)
|
||||
guard = root / "static-audit.so"
|
||||
subprocess.run([os.environ.get("CC", "cc"), "-shared", "-fPIC",
|
||||
str(REPO / "test/PIM/StaticSchedulingAudit.c"), "-o", str(guard)], check=True)
|
||||
env = dict(os.environ, LD_PRELOAD=str(guard), PATH=str(trap) + os.pathsep + os.environ["PATH"], OMP_NUM_THREADS="1")
|
||||
decisions = []
|
||||
for iteration in range(3):
|
||||
if iteration == 0:
|
||||
history.write_text("network,throughput\ngooglenet,1e99\n")
|
||||
profile.write_text('{"candidate":999,"measured_latency":1}')
|
||||
elif iteration == 1:
|
||||
history.write_bytes(b"invalid historical data\x00")
|
||||
profile.write_text('{"candidate":0,"measured_latency":1e99}')
|
||||
else:
|
||||
history.unlink()
|
||||
shutil.rmtree(profile.parent)
|
||||
out = root / f"run-{iteration}"
|
||||
out.mkdir()
|
||||
command = [compiler, str(model), "-o", str(out / "model"), "--maccel=PIM",
|
||||
"--EmitPimCodegen", "--core-count=138", "--crossbar-count=128", "--crossbar-size=128",
|
||||
"--pipeline=4", "--mlir-disable-threading", "--pim-detect-communication-deadlock",
|
||||
f"--pim-target-config={REPO}/validation/pimsim_configs/pimcomp/arch-b/throughput_config_1000ms.json"]
|
||||
result = subprocess.run(command, cwd=root, env=env, capture_output=True, text=True, timeout=180)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
report = json.loads((out / "reports/pipeline_selection.json").read_text())
|
||||
assert report["static_analysis"] and report["temporary_disk_bytes"] == 0
|
||||
assert report["analyzed_candidates"] > 1, report
|
||||
check_selected_program(out)
|
||||
assert not report["reconstructed_spatial_artifacts"]
|
||||
assert not (out / "dialects").exists()
|
||||
assert not list(out.glob("candidate*"))
|
||||
decisions.append((report["selection"], [(c["identity"], c.get("interval_ns"), c.get("alternative_tie_interval_ns"))
|
||||
for c in report["candidates"]]))
|
||||
# Keep the test's disk peak to one selected program.
|
||||
shutil.rmtree(out)
|
||||
assert decisions[0] == decisions[1] == decisions[2], "historical files or address allocation changed static selection"
|
||||
# Exercise the same winner with and without explicit Spatial artifacts.
|
||||
vgg = REPO / "validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx"
|
||||
for diagnostics in (False, True):
|
||||
out = root / f"vgg-{diagnostics}"
|
||||
out.mkdir()
|
||||
command = [compiler, str(vgg), "-o", str(out / "model"), "--maccel=PIM", "--EmitPimCodegen",
|
||||
"--pipeline=2", "--core-count=138", "--crossbar-count=128", "--crossbar-size=128",
|
||||
"--mlir-disable-threading", "--pim-detect-communication-deadlock",
|
||||
f"--pim-target-config={REPO}/validation/pimsim_configs/pimcomp/arch-b/throughput_config_1000ms.json"]
|
||||
if diagnostics:
|
||||
command += ["--pim-export-spatial-dataflow=all", "--pim-dump-ir", "--pim-periodic-diagnostics"]
|
||||
result = subprocess.run(command, cwd=root, env=env, capture_output=True, text=True, timeout=90)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
report = check_selected_program(out)
|
||||
assert report["selection"] != 0 and report["reconstructed_spatial_artifacts"] == diagnostics
|
||||
identity = report["candidates"][report["selection"]]["identity"]
|
||||
schedule = (out / "reports/spatial3_scheduled_no_comm.txt").read_bytes()
|
||||
if diagnostics:
|
||||
assert identity == previous_identity and schedule == previous_schedule
|
||||
for stage in ("spatial3_scheduled_no_comm", "spatial4_scheduled"):
|
||||
for suffix in ("nodes.csv", "edges.csv"):
|
||||
assert (out / f"reports/{stage}.{suffix}").stat().st_size > 0
|
||||
assert (out / "reports/pipeline_periodic.json").exists()
|
||||
assert (out / "dialects/spatial4_scheduled.mlir").exists()
|
||||
previous_identity, previous_schedule = identity, schedule
|
||||
shutil.rmtree(out)
|
||||
config_path = REPO / "validation/pimsim_configs/pimcomp/arch-a/throughput_config_1000ms.json"
|
||||
unsupported = json.loads(config_path.read_text())
|
||||
unsupported["chip_config"]["core_config"]["rob_size"] = 2
|
||||
network = unsupported["chip_config"]["network_config"]
|
||||
network["net_config_file_path"] = str((config_path.parent / network["net_config_file_path"]).resolve())
|
||||
hardware = root / "unsupported-target.json"
|
||||
hardware.write_text(json.dumps(unsupported))
|
||||
out = root / "fallback"
|
||||
out.mkdir()
|
||||
command = [compiler, str(REPO / "validation/networks/pimcomp_models/vgg8/vgg8-mnist-reconstructed.onnx"),
|
||||
"-o", str(out / "model"), "--maccel=PIM", "--EmitPimCodegen", "--pipeline=4",
|
||||
"--core-count=168", "--crossbar-count=96", "--crossbar-size=128", "--mlir-disable-threading",
|
||||
f"--pim-target-config={hardware}"]
|
||||
result = subprocess.run(command, cwd=root, env=env, capture_output=True, text=True, timeout=60)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
report = json.loads((out / "reports/pipeline_selection.json").read_text())
|
||||
assert report["selection"] == 0 and report["analyzed_candidates"] == 0
|
||||
assert report["generated_candidates"] == 1 and report["lowered_candidates"] == 0
|
||||
assert "ROB capacity one" in report["reason"] and report["unavailable_estimates"] > 0
|
||||
assert report["temporary_disk_bytes"] == 0 and not list(root.glob("candidate*"))
|
||||
shutil.rmtree(out)
|
||||
out.mkdir()
|
||||
command[-1] = f"--pim-target-config={config_path}"
|
||||
command += ["--pim-disable-synchronization"]
|
||||
result = subprocess.run(command, cwd=root, env=env, capture_output=True, text=True, timeout=60)
|
||||
assert result.returncode == 0, result.stdout + result.stderr
|
||||
report = json.loads((out / "reports/pipeline_selection.json").read_text())
|
||||
assert report["selection"] == 0 and report["analyzed_candidates"] == 0
|
||||
assert report["generated_candidates"] == 1 and report["lowered_candidates"] == 0
|
||||
assert "synchronization ablation" in report["reason"]
|
||||
assert not (root / "executed").exists(), "production compiler executed a diagnostic or simulator"
|
||||
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -3,39 +3,39 @@ arch-a,vgg8,latency,1,element,PASS,PASS,1.52,7.99,NA,NA,321.05,200.11,487455139.
|
||||
arch-a,vgg8,throughput,2,batch,PASS,PASS,0.91,1.04,1100.00,965.00,429.31,482.60,407972136.20,541691691.00,raptor,1.14
|
||||
arch-a,vgg8,throughput,4,batch,PASS,PASS,0.78,1.04,1280.00,965.00,457.04,482.60,415761605.20,541691691.00,raptor,1.33
|
||||
arch-a,vgg8,throughput,8,batch,PASS,PASS,0.78,1.04,1290.00,965.00,464.07,482.60,396424007.70,541691691.00,raptor,1.34
|
||||
arch-a,resnet18,latency,1,element,PASS,FAIL,14.03,58.18,NA,NA,455.67,238.39,6394498463.12,13868866808.12,raptor,4.15
|
||||
arch-a,resnet18,latency,1,element,PASS,FAIL,14.03,58.18,NA,NA,455.67,238.39,6394478521.12,13868866808.12,raptor,4.15
|
||||
arch-a,resnet18,throughput,2,batch,PASS,FAIL,11.73,13.46,85.20,74.30,503.11,481.60,6628073894.00,6563451004.00,raptor,1.15
|
||||
arch-a,resnet18,throughput,4,batch,PASS,FAIL,19.21,13.46,52.10,74.30,380.10,481.60,7957033516.00,6563451004.00,pimcomp,1.43
|
||||
arch-a,resnet18,throughput,8,batch,PASS,FAIL,19.93,13.46,50.20,74.30,361.80,481.60,7763790072.00,6563451004.00,pimcomp,1.48
|
||||
arch-a,resnet18,throughput,4,batch,PASS,FAIL,19.21,13.46,52.10,74.30,380.10,481.60,7957033431.00,6563451004.00,pimcomp,1.43
|
||||
arch-a,resnet18,throughput,8,batch,PASS,FAIL,19.93,13.46,50.20,74.30,361.80,481.60,7763795670.00,6563451004.00,pimcomp,1.48
|
||||
arch-a,resnet34,latency,1,element,PASS,FAIL,29.72,66.71,NA,NA,411.81,277.49,12238321706.68,18511425563.68,raptor,2.24
|
||||
arch-a,resnet34,throughput,2,batch,PASS,FAIL,22.26,23.58,44.90,42.40,473.84,515.68,16417303180.00,12283627410.00,raptor,1.06
|
||||
arch-a,resnet34,throughput,4,batch,PASS,FAIL,27.37,23.58,36.50,42.40,415.90,515.68,17019112320.00,12283627410.00,pimcomp,1.16
|
||||
arch-a,resnet34,throughput,8,batch,PASS,FAIL,41.04,23.58,24.40,42.40,346.15,515.68,17351854700.00,12283627410.00,pimcomp,1.74
|
||||
arch-a,googlenet,latency,1,element,PASS,PASS,9.78,62.92,NA,NA,563.88,231.19,5513234559.92,14547744283.24,raptor,6.44
|
||||
arch-a,googlenet,throughput,2,batch,PASS,FAIL,8.42,18.26,119.00,54.80,610.74,435.11,5777550058.00,8164787803.00,raptor,2.17
|
||||
arch-a,googlenet,throughput,4,batch,PASS,FAIL,10.73,18.26,93.20,54.80,528.02,435.11,6592912716.00,8164787803.00,raptor,1.70
|
||||
arch-a,googlenet,throughput,8,batch,PASS,FAIL,11.03,18.26,90.70,54.80,505.94,435.11,6622163369.00,8164787803.00,raptor,1.66
|
||||
arch-a,resnet34,throughput,2,batch,PASS,FAIL,23.27,23.58,43.00,42.40,471.72,515.68,13055037600.00,12283627410.00,raptor,1.01
|
||||
arch-a,resnet34,throughput,4,batch,PASS,FAIL,27.47,23.58,36.40,42.40,422.22,515.68,14899055210.00,12283627410.00,pimcomp,1.16
|
||||
arch-a,resnet34,throughput,8,batch,PASS,FAIL,39.84,23.58,25.10,42.40,352.28,515.68,17062658890.00,12283627410.00,pimcomp,1.69
|
||||
arch-a,googlenet,latency,1,element,PASS,PASS,9.78,62.92,NA,NA,563.87,231.19,5513262106.92,14547744283.24,raptor,6.44
|
||||
arch-a,googlenet,throughput,2,batch,PASS,FAIL,8.42,18.26,119.00,54.80,610.75,435.11,5777456331.00,8164787803.00,raptor,2.17
|
||||
arch-a,googlenet,throughput,4,batch,PASS,FAIL,10.73,18.26,93.20,54.80,528.03,435.11,6592892213.00,8164787803.00,raptor,1.70
|
||||
arch-a,googlenet,throughput,8,batch,PASS,FAIL,10.12,18.26,98.80,54.80,523.81,435.11,6521737843.00,8164787803.00,raptor,1.80
|
||||
arch-a,yolo11n,latency,1,element,PASS,FAIL,147.49,NA,NA,NA,290.17,NA,42797722129.00,NA,NA,NA
|
||||
arch-a,yolo11n,throughput,2,batch,PASS,FAIL,120.93,NA,8.27,NA,313.20,NA,41830757130.00,NA,NA,NA
|
||||
arch-a,yolo11n,throughput,2,batch,PASS,FAIL,102.34,NA,9.77,NA,337.88,NA,38308060150.00,NA,NA,NA
|
||||
arch-a,yolo11n,throughput,4,batch,PASS,FAIL,87.91,NA,11.40,NA,358.20,NA,38729846260.00,NA,NA,NA
|
||||
arch-a,yolo11n,throughput,8,batch,PASS,FAIL,104.07,NA,9.61,NA,330.76,NA,44002712680.00,NA,NA,NA
|
||||
arch-a,yolo11n,throughput,8,batch,PASS,FAIL,83.68,NA,11.90,NA,365.64,NA,38623589740.00,NA,NA,NA
|
||||
arch-b,vgg8,latency,1,element,PASS,PASS,1.53,7.15,NA,NA,292.56,173.77,448581679.04,1242814988.12,raptor,4.66
|
||||
arch-b,vgg8,throughput,2,batch,PASS,PASS,1.18,1.07,846.00,932.00,327.05,409.47,429183159.10,439288835.80,pimcomp,1.10
|
||||
arch-b,vgg8,throughput,2,batch,PASS,PASS,0.69,1.07,1460.00,932.00,456.04,409.47,403694238.30,439288835.80,raptor,1.57
|
||||
arch-b,vgg8,throughput,4,batch,PASS,PASS,0.67,1.07,1490.00,932.00,448.90,409.47,413526781.30,439288835.80,raptor,1.60
|
||||
arch-b,vgg8,throughput,8,batch,PASS,PASS,0.75,1.07,1330.00,932.00,425.74,409.47,394457860.80,439288835.80,raptor,1.43
|
||||
arch-b,resnet18,latency,1,element,PASS,FAIL,15.38,64.38,NA,NA,400.67,201.66,6160441550.12,12983432281.12,raptor,4.19
|
||||
arch-b,resnet18,throughput,2,batch,PASS,FAIL,13.08,13.43,76.40,74.40,440.79,484.80,6399942695.00,6569655551.00,raptor,1.03
|
||||
arch-b,resnet18,throughput,2,batch,PASS,FAIL,12.12,13.43,82.50,74.40,461.41,484.80,6295321671.00,6569655551.00,raptor,1.11
|
||||
arch-b,resnet18,throughput,4,batch,PASS,FAIL,12.68,13.43,78.90,74.40,451.19,484.80,7096979122.00,6569655551.00,raptor,1.06
|
||||
arch-b,resnet18,throughput,8,batch,PASS,FAIL,20.59,13.43,48.60,74.40,321.62,484.80,7163273595.00,6569655551.00,pimcomp,1.53
|
||||
arch-b,resnet18,throughput,8,batch,PASS,FAIL,20.59,13.43,48.60,74.40,321.63,484.80,7162570047.00,6569655551.00,pimcomp,1.53
|
||||
arch-b,resnet34,latency,1,element,PASS,FAIL,27.95,107.20,NA,NA,397.21,206.48,11103517952.68,22134174654.68,raptor,3.83
|
||||
arch-b,resnet34,throughput,2,batch,PASS,FAIL,22.28,22.85,44.90,43.80,438.24,493.33,14218683390.00,11392215660.00,raptor,1.03
|
||||
arch-b,resnet34,throughput,4,batch,PASS,FAIL,39.61,22.85,25.20,43.80,315.01,493.33,14629161660.00,11392215660.00,pimcomp,1.74
|
||||
arch-b,resnet34,throughput,8,batch,PASS,FAIL,49.34,22.85,20.30,43.80,285.76,493.33,16591020470.00,11392215660.00,pimcomp,2.16
|
||||
arch-b,googlenet,latency,1,element,PASS,PASS,11.22,37.94,NA,NA,483.27,242.04,5422799605.92,9182062336.24,raptor,3.38
|
||||
arch-b,googlenet,throughput,2,batch,PASS,FAIL,9.21,21.39,109.00,46.80,545.82,417.87,5591022849.00,9014726090.00,raptor,2.33
|
||||
arch-b,googlenet,throughput,4,batch,PASS,FAIL,14.23,21.39,70.30,46.80,411.90,417.87,6718342521.00,9014726090.00,raptor,1.50
|
||||
arch-b,googlenet,throughput,8,batch,PASS,FAIL,16.19,21.39,61.80,46.80,355.70,417.87,6726459859.00,9014726090.00,raptor,1.32
|
||||
arch-b,resnet34,throughput,4,batch,PASS,FAIL,29.09,22.85,34.40,43.80,364.96,493.33,16328513230.00,11392215660.00,pimcomp,1.27
|
||||
arch-b,resnet34,throughput,8,batch,PASS,FAIL,47.68,22.85,21.00,43.80,289.82,493.33,16346419570.00,11392215660.00,pimcomp,2.09
|
||||
arch-b,googlenet,latency,1,element,PASS,PASS,11.22,37.94,NA,NA,483.27,242.04,5422781257.92,9182062336.24,raptor,3.38
|
||||
arch-b,googlenet,throughput,2,batch,PASS,FAIL,9.21,21.39,109.00,46.80,545.49,417.87,5593397699.00,9014726090.00,raptor,2.33
|
||||
arch-b,googlenet,throughput,4,batch,PASS,FAIL,8.56,21.39,117.00,46.80,555.26,417.87,5863121048.00,9014726090.00,raptor,2.50
|
||||
arch-b,googlenet,throughput,8,batch,PASS,FAIL,16.19,21.39,61.80,46.80,355.70,417.87,6726458739.00,9014726090.00,raptor,1.32
|
||||
arch-b,yolo11n,latency,1,element,PASS,FAIL,168.14,NA,NA,NA,245.31,NA,41248047514.00,NA,NA,NA
|
||||
arch-b,yolo11n,throughput,2,batch,PASS,FAIL,118.15,NA,8.46,NA,287.59,NA,37062981240.00,NA,NA,NA
|
||||
arch-b,yolo11n,throughput,4,batch,PASS,FAIL,94.18,NA,10.60,NA,316.45,NA,37424491250.00,NA,NA,NA
|
||||
arch-b,yolo11n,throughput,8,batch,PASS,FAIL,142.86,NA,7.00,NA,290.68,NA,41525990280.00,NA,NA,NA
|
||||
arch-b,yolo11n,throughput,2,batch,PASS,FAIL,104.49,NA,9.57,NA,302.12,NA,35801517850.00,NA,NA,NA
|
||||
arch-b,yolo11n,throughput,4,batch,PASS,FAIL,108.25,NA,9.24,NA,297.48,NA,37629699620.00,NA,NA,NA
|
||||
arch-b,yolo11n,throughput,8,batch,PASS,FAIL,100.13,NA,9.99,NA,308.83,NA,37180785770.00,NA,NA,NA
|
||||
|
||||
|
Reference in New Issue
Block a user