rework documentation
Validate Operations / validate-operations (push) Failing after 3h10m55s

add plot generation scripts
adjust validation tools
add up-to-date results
This commit is contained in:
NiccoloN
2026-10-19 16:34:01 +02:00
parent 9053cc5e8b
commit 034c197260
33 changed files with 3077 additions and 1798 deletions
+131
View File
@@ -0,0 +1,131 @@
# Graph Explorer architecture and API
The [README](README.md) covers installation, input headers, commands, and tests.
This reference describes the contracts maintained by the implementation.
## Ingestion and identity
[`discovery.py`](raptor_graph_explorer/discovery.py) recursively pairs reports
by their full filename prefix, including ZIP inputs, and rejects unsafe archive
paths. That prefix is `report_id`; unrelated reports never share endpoints.
[`ingest.py`](raptor_graph_explorer/ingest.py) streams CSVs in batches of 2,000
rows through temporary SQLite staging tables. Node IDs, operation IDs, lanes,
cores, and channels remain separate fields. Known node prefixes are `gc`,
`gcb`, `sc`, `scb`; unknown ones retain `unknown:<prefix>`. Explicit operation
and lane columns are authoritative. Blank operation IDs, and blank batch lanes,
may use a parseable source ID with a warning. Explicit edge lanes override
endpoint lanes, with contradictory values diagnosed. Missing numbers stay
SQL NULL; malformed rows are diagnosed and omitted, never coerced to zero.
Extra columns survive in `attributes_json`.
Duplicate node IDs retain the first source row and produce diagnostics. Bulk
joins resolve edges within their report; unknown endpoints are omitted with
bounded examples. Staging tables are dropped even on error. The ingestion
connection closes before indexes, aggregates, motifs, and diagnostics are
built. Secondary indexes are created after raw insertion. CSVs without extra
columns reuse an empty JSON representation.
Raw edge IDs use report ID plus original CSV row number; aggregate IDs use
report ID and grouping keys. Input filesystem paths and browser layout do not
define identity. The manifest stores versions and counts, not duplicate raw
rows or machine-specific input paths. Database schema compatibility is checked
before serving; incompatible outputs must be rebuilt.
## Aggregation and exact lane mappings
Operation nodes group by `(report_id, op_id)`, retaining kind counts, lane/core
cardinalities and ranges, and SSA summaries. Operation edges group by the
source/target operation pair, retaining counts, weights, tensor types, channels,
and exact mapping classification. Core aggregation has a distinct `unassigned`
group for missing cores. Node-kind aggregation is a small diagnostic view.
[`mappings.py`](raptor_graph_explorer/mappings.py) uses all normalized lane
pairs, in the following precedence order:
| Class | Meaning |
|---|---|
| `scalar_to_scalar` | Both lanes absent for every pair. |
| `scalar_to_batch` / `batch_to_scalar` | One side entirely absent, the other entirely present; range, cardinality, and contiguity recorded. |
| `identity` | One-to-one relation with equal lanes. |
| `constant_offset` | One-to-one with one nonzero target-minus-source offset. |
| `permutation` | Remaining one-to-one relation with equal cardinalities. |
| `broadcast` / `fan_in` | One source to many targets / many sources to one target. |
| `stencil` | Multiple sources share exactly the same target-offset set, containing 2–64 offsets. |
| `irregular` | Complete lane data matching none of the preceding forms. |
| `unknown` | Empty or partially missing relation outside the scalar/batch cases. |
SQL computes degrees/cardinalities; stencil comparison streams one operation
pair at a time. No sampling determines classification. Mapping rows remain
paginated; matrices show exact sparse points up to the configured threshold,
then bounded density bins.
## Display projection and layout
[`projection.py`](raptor_graph_explorer/projection.py) rebuilds the entire
display from the current expanded-operation set. Expanding replaces an
operation aggregate with its raw nodes, including isolated nodes. An aggregate
edge survives only while both endpoint operations are collapsed. Otherwise its
raw edges replace it: each endpoint is raw if its operation is expanded and
aggregate if collapsed. The complete set determines endpoints, so expansion
order cannot leave stale endpoints or duplicate aggregate/raw representations.
Raw view expands every operation, preserving each valid source edge once.
Operation ranks use the complete operation graph, including isolated nodes.
Collapsed operations occupy one row; expanded lanes use compact rows in numeric
order, followed by deterministic lane-less rows. Sparse lane numbers leave no
empty rows, and nodes sharing a lane get centered deterministic horizontal
offsets. Operation/raw views use stable server coordinates restored by Reset
layout. Core/node-kind views use ELK. Views flow top-to-bottom.
The [viewer](raptor_graph_explorer/static/index.html) first requests an
operation aggregate before applying the report's view selection. Raw selections
use raw details rather than aggregate-only endpoints. Operation labels use the
first SSA name, falling back to source identity; label collision handling and
bounded node sizes keep dense views usable. Dependencies are pinned in that
file (Graphology, Sigma.js, ELK); first browser use requires CDN access.
## Structural motifs
[`motifs.py`](raptor_graph_explorer/motifs.py) loads only the operation aggregate
into NetworkX. It condenses SCCs into a DAG, attaches synthetic source/sink
nodes, and computes dominators and reverse post-dominators. Branching entries
are paired with their immediate post-dominator; regions retain nodes reachable
before that exit which can reach it, rejecting external entrances/exits.
One shortest representative path per branch avoids enumerating all paths.
`exact_diamond` has two direct one-node branches; `rhomboid` has disjoint
branch interiors; `split_rejoin` covers other valid single-entry/single-exit
regions. Identical memberships are deduplicated; nested motifs remain.
SCC members map back to operation IDs, with deterministic representative
operations for multi-operation SCC endpoints/paths and `contains_cycle` markers.
Screen geometry never determines motif membership.
## Read-only API
All routes below use GET; see [`api.py`](raptor_graph_explorer/api.py).
Responses include `Cache-Control: no-store` to avoid mixing viewer revisions.
| Route | Result / relevant query parameters |
|---|---|
| `/api/manifest` | Tool/schema versions and report counts. |
| `/api/reports` | Available reports. |
| `/api/reports/{report_id}/summary` | Counts, mapping classes, diagnostics. |
| `/api/reports/{report_id}/graph/{level}` | Complete `operation`, `core`, or `node_kind` aggregate. |
| `/api/reports/{report_id}/display-graph` | Deterministic projection; `view=operation` or `raw`, repeated `expanded_op_id`, `expand_all`. |
| `/api/reports/{report_id}/motifs` | Structural motifs. |
| `/api/aggregate-nodes/{aggregate_id}` | Aggregate details; operation SSA names capped at 100. |
| `/api/aggregate-edges/{aggregate_edge_id}` | Aggregate edge details. |
| `/api/aggregate-edges/{aggregate_edge_id}/mapping` | Exact raw lane rows; `limit`, `offset`. Operation edges only. |
| `/api/aggregate-edges/{aggregate_edge_id}/matrix` | Exact sparse points or density bins. Operation edges only. |
| `/api/reports/{report_id}/raw-nodes` | Raw node page; `limit`, `offset`. |
| `/api/reports/{report_id}/raw-edges` | Raw edge page; `limit`, `offset`. |
| `/api/reports/{report_id}/subgraph` | `aggregate_id`, `direction=in/out/both`, `depth`, optional `expand_lanes`. |
`create_app` defaults: page size 100 with a maximum of 500; expansion cap 5,000;
exact matrix threshold 2,000 points; density grid 64×64. Subgraph depth defaults
to 1 and caps at 5. Oversized raw/expanded projections return HTTP 400 without a
partial graph. The projection cap counts raw nodes plus incident raw edges;
collapsed aggregate nodes/edges are additional. Aggregate endpoints have no
pagination or expansion cap, so very large operation graphs remain a scaling
limit. Raw CSV graphs are never loaded wholesale into NetworkX.
+73 -184
View File
@@ -1,228 +1,117 @@
# Raptor Graph Explorer
Raptor Graph Explorer is an external Python 3.13 tool that preprocesses Raptor's existing Spatial dataflow CSV reports into SQLite and serves an interactive browser viewer. It does not modify Raptor, compiler passes, MLIR, or CSV generation. The CSV files remain the source of truth.
Python 3.13+ tool for exploring Raptor Spatial dataflow CSVs in a browser.
It streams reports into SQLite, aggregates operations/cores/node kinds, and
provides bounded raw-lane inspection. Compiler output remains unchanged.
Raw lane graphs can contain hundreds of thousands or millions of rows. Sending those rows to a browser makes layout, rendering, and even JSON serialization impractical. The explorer therefore loads an operation aggregate by default and retrieves raw rows only through bounded pages or a capped, explicitly requested lane expansion.
## Install and run
## Architecture
```text
CSV pairs / directories / ZIP archives
-> recursive discovery and safe ZIP extraction
-> streaming CSV validation and SQLite staging
-> normalized raw_nodes and raw_edges
-> operation, core, and node-kind aggregates
-> exact lane mappings
-> SCC condensation and structural split/rejoin motifs
-> graph.sqlite + manifest.json
-> read-only FastAPI API
-> Sigma.js WebGL viewer with stable operation/lane layout
```
Node identifiers, operation IDs, lanes, cores, and channels are stored separately. In particular, graph lanes, physical fragment slots, scheduled lanes, physical cores, and communication channels are not treated as interchangeable identities.
## Installation
From the Raptor repository root:
From the repository root, create a Python 3.13+ environment if `.venv` does not
exist:
```bash
python3.13 -m venv .venv
source .venv/bin/activate
python3.13 -m pip install -e tools/raptor_graph_explorer[dev]
```
The installed command is `raptor-graph-explorer`. The equivalent module entry point is `python3.13 -m raptor_graph_explorer`.
## Commands
Build a persistent output:
For a new or existing environment, verify the version before installing. Select
a compatible environment if this check fails; do not reuse an older Python:
```bash
raptor-graph-explorer build \
validation/networks/vgg16/depth_07/raptor/reports \
--output /tmp/vgg_graph
.venv/bin/python -c 'import sys; assert sys.version_info >= (3, 13), "Python 3.13+ required"'
.venv/bin/python -m pip install -e 'tools/raptor_graph_explorer[dev]'
.venv/bin/raptor-graph-explorer build /path/to/raptor/reports --output /tmp/raptor_graph
.venv/bin/raptor-graph-explorer serve /tmp/raptor_graph --port 8765
```
Use repeated inputs or pass several paths after `build`. Inputs can be report directories, directories containing report subdirectories, ZIP archives, or explicit `.nodes.csv` / `.edges.csv` files. `--strict` rejects any data-integrity diagnostic. Without it, valid pairs and unambiguous rows are retained. `--overwrite` replaces an existing output.
Open `http://127.0.0.1:8765`. The server is read-only, has no authentication,
and is intended for local use. Browser dependencies load from pinned CDNs and
require network access on first use; preprocessing and inspection work offline.
Serve an existing output:
Other commands:
```bash
raptor-graph-explorer serve /tmp/vgg_graph --port 8765
# Build and serve a directory, CSV pair, or ZIP archive.
.venv/bin/raptor-graph-explorer run /path/to/reports.zip --output /tmp/raptor_graph
# Print counts, mappings, motifs, and diagnostics.
.venv/bin/raptor-graph-explorer inspect /tmp/raptor_graph
```
Build and immediately serve:
`build` and `run` accept multiple input paths and discover reports recursively.
`--overwrite` replaces an existing output. `build --strict` rejects
data-integrity errors; otherwise valid pairs and unambiguous rows are retained.
Outputs are `graph.sqlite` and `manifest.json`. Rebuild outputs if the tool
reports an unsupported schema version.
```bash
raptor-graph-explorer run \
/mnt/data/conv_relu_conv.zip \
--output /tmp/conv_relu_conv_graph \
--overwrite
```
See [architecture and API guarantees](ARCHITECTURE.md) for identity, expansion,
mapping/motif analysis, ingestion constraints, and the endpoint reference.
Print counts, mapping classes, SCCs, motifs, and diagnostics:
## Input contract
```bash
raptor-graph-explorer inspect /tmp/vgg_graph
```
The server defaults to `127.0.0.1:8765`. It has no authentication and is intended for local read-only use.
## Supported CSV reports
Files are paired using the complete prefix before `.nodes.csv` and `.edges.csv`. That prefix is the stable `report_id`; `spatial3_scheduled_no_comm.nodes.csv` therefore pairs only with `spatial3_scheduled_no_comm.edges.csv`.
Node columns:
Pair files by the complete prefix before `.nodes.csv` and `.edges.csv`.
Required headers:
```csv
Id,op_id,lane,core,ssa_name
```
Edge columns:
```csv
Source,Target,Weight,Type,stage,source_lane,target_lane,channel_id
```
Known node prefixes are `gc`, `gcb`, `sc`, and `scb`. Unknown prefixes are retained as `unknown:<prefix>`. Explicit CSV `op_id` and lane columns are authoritative; a blank value may fall back to a parseable node ID and produces a diagnostic. Explicit edge lanes take precedence over endpoint lanes. Missing numeric values remain SQL `NULL`; malformed numeric values are never changed to zero. Additional CSV columns are preserved in `attributes_json`.
Explicit operation IDs and lanes take precedence over parsed node IDs. Missing
numeric values remain missing; malformed values produce diagnostics. Extra
columns are preserved. Edges resolve only within their report; unknown
endpoints are diagnosed and omitted. Unsafe ZIP paths are rejected.
Edges are resolved only against nodes from the same report. Unknown endpoints omit that edge and produce a bounded diagnostic. ZIP paths are checked before extraction and traversal entries are rejected.
Graph lanes, physical fragment slots, scheduled lanes, cores, and channels are
distinct identities. Spatial3 reports contain explicit communication only;
isolated nodes are valid and no dependencies are inferred from SSA text.
## Output
## Views and analysis
```text
OUTPUT_DIRECTORY/
├── graph.sqlite
└── manifest.json
```
- **Operation:** aggregates by operation and supports lane expansion, exact
mapping details, and structural motifs.
- **Core / Node kind:** summarize placement and operation kinds.
- **Raw instances:** displays source nodes and valid edges, including isolated
nodes; spatial4 reports select this view automatically.
`graph.sqlite` contains reports, normalized raw nodes and edges, aggregate nodes and edges, motifs, motif membership, diagnostics, and focused indexes. `manifest.json` contains schema/tool versions and compact per-report counts. It never duplicates raw graph rows or stores machine-specific input paths.
The viewer first loads an operation aggregate. Search, filters, expansion,
selection details, and layout controls support focused inspection. Raw and
expanded graphs cap raw nodes plus incident raw edges at 5,000; exceeding it returns an
error rather than a partial graph. Use an aggregate view or increase
`create_app(..., expansion_cap=...)` when the machine can handle it.
Schema version 1 is checked whenever an output is served. Unsupported or missing versions fail at startup with a clear error.
Lane mappings use all normalized pairs, without sampling. Classes include
scalar/batch relations, identity, constant offset, permutation, broadcast,
fan-in, stencil, irregular, and unknown. Small matrices show exact sparse
points; large ones use a bounded 64×64 density view.
## Aggregate views
Motifs identify structural single-entry/single-exit split/rejoin regions on the
operation graph's SCC condensation. `exact_diamond`, `rhomboid`, and
`split_rejoin` describe progressively broader forms; a diamond-shaped drawing
alone is not evidence of a motif. Cycles remain marked.
### Operation
## API and limits
Nodes group by `(report_id, op_id)` and retain node-kind counts, instance count, lane and core cardinalities/ranges, and an SSA summary. Edges group by `(report_id, source_op_id, target_op_id)` and retain raw count, weight statistics, tensor histogram, distinct channels, and exact mapping classification. This is the default view and the only graph used for motif analysis.
Read-only FastAPI endpoints expose manifests, report summaries, aggregate
graphs/details, mappings, matrices, motifs, and raw pages. See
[`api.py`](raptor_graph_explorer/api.py) for routes and `create_app` limits.
Raw pages default to 100 rows and cap at 500; subgraph depth caps at five.
Aggregate graph endpoints return the complete aggregate, so very large
operation counts can still be expensive.
### Core
Assigned nodes group by `(report_id, core)`. Missing cores use a separate `unassigned` group, never core zero. Edges retain raw count, total weight, channel count, operation-pair count, and tensor histogram.
### Node kind
Nodes group by `(report_id, node_kind)`. This deliberately small diagnostic view shows how graph/scalar/batch/scheduled kinds connect; it does not replace operation aggregation.
### Raw instances
Raw instances displays every `raw_nodes` row exactly once and every valid `raw_edges` row exactly once. Nodes are queried independently from edges, so degree-zero nodes remain visible. A spatial4 report starts in this view automatically; Operation, Core, and Node kind remain selectable.
Spatial3 CSVs record explicit communication only and may legitimately contain many isolated nodes. The explorer never fabricates dependencies from semicolon-separated SSA names or other node text.
## Exact lane mappings
Classifications use all normalized source/target lane pairs for an operation edge. They are evaluated in this order:
- `scalar_to_scalar`: every pair has two absent lanes.
- `scalar_to_batch`: every source lane is absent and every target lane is present; target range, cardinality, and contiguity are recorded.
- `batch_to_scalar`: the corresponding source-lane case.
- `identity`: a one-to-one relation with equal source and target lanes.
- `constant_offset`: a one-to-one relation with one nonzero `target - source` offset.
- `permutation`: a one-to-one relation that is neither identity nor constant offset.
- `broadcast`: one distinct source lane maps to multiple target lanes.
- `fan_in`: multiple source lanes map to one distinct target lane.
- `stencil`: more than one source has exactly the same bounded target-offset set; the complete sorted set is recorded.
- `irregular`: complete lane data matches none of the exact forms.
- `unknown`: lane data is partially missing outside the scalar/batch cases.
No sampling is used. SQL computes cardinalities and degrees; stencil offsets stream one operation edge at a time. Raw mapping rows remain paginated, and the matrix endpoint returns exact sparse points only below its threshold. Larger relations become a bounded 64×64 density matrix.
## Structural motifs
Motifs are not inferred from rendered geometry. For each operation graph the tool:
1. computes strongly connected components and condenses them into a DAG;
2. attaches synthetic source and sink nodes;
3. computes dominators and reverse-graph post-dominators;
4. considers entries with at least two outgoing branches and their immediate post-dominator exit;
5. retains nodes reachable before the exit that can also reach the exit;
6. rejects unexpected entrances to, or exits from, internal nodes;
7. records one shortest representative path per branch without enumerating all paths;
8. maps condensed components back to operation aggregate IDs.
`exact_diamond` is a two-branch direct reconvergence with one internal condensed node per branch. `rhomboid` has clean disjoint branch interiors. `split_rejoin` is the general valid single-entry/single-exit region. Identical memberships are deduplicated while meaningful nested motifs are preserved. SCC membership marks `contains_cycle`; cycles are never analyzed as if the raw operation graph were a DAG.
## Viewer and API
The viewer initially fetches an aggregate operation graph, so a previously selected raw view cannot make a new report exceed the display safety cap before the first render. Browser responses disable caching so HTML and JavaScript from different tool revisions cannot be mixed. Sigma renders the graph with WebGL. Controls cover report/view selection, text and tensor search, mapping filters, self edges, relayout, fitting, and motif selection. Edges use a fixed width. Raw nodes and edges have their own selection details and never call aggregate-only detail or mapping endpoints. Nodes without an SSA name fall back to their source identity. Mapping panels remain available for operation aggregate edges.
Operation expansion is a deterministic projection of the source graph. Expand selected, Expand all operations, Collapse selected operation, and Collapse all rebuild the complete display from the current expanded-operation set. An expanded operation's raw nodes replace its aggregate node, including isolated raw nodes. An aggregate edge is retained only when both endpoint operations are collapsed; otherwise its raw edges replace it with endpoints calculated from the complete expansion set. Expansion order therefore cannot leave stale endpoints or simultaneous aggregate/raw representations.
Operation ranks are calculated from the complete operation graph, including isolated operations, and every view flows top-to-bottom. A collapsed operation reserves one displayed row. Expanded nodes use compact rows for distinct lanes in numeric order, followed by deterministic lane-less rows, so sparse lane numbers create no empty geometric space. Raw nodes sharing a lane receive centered deterministic horizontal offsets. Operation and raw views use stable server coordinates, and `Reset layout` restores them; core and node-kind views use ELK, and `Rerun layout` runs it again. Operation labels use the first SSA name; Sigma's collision grid thins normal labels, and a label is skipped when its measured screen rectangle intersects another visible node. Selected or hovered nodes keep the original white bubble with black text. Selecting a motif frames its member nodes on the first click. Aggregate-node sizes remain bounded.
Browser dependencies are pinned in one place, `static/index.html`:
- Graphology 0.25.4;
- Sigma.js 2.4.0;
- ELK.js 0.11.1.
They load from CDNs, so the browser viewer needs network access on first use. Python preprocessing, inspection, and API tests do not.
Read-only endpoints include:
```text
/api/manifest
/api/reports
/api/reports/{report_id}/summary
/api/reports/{report_id}/graph/{level}
/api/reports/{report_id}/display-graph
/api/reports/{report_id}/motifs
/api/aggregate-nodes/{aggregate_id}
/api/aggregate-edges/{aggregate_edge_id}
/api/aggregate-edges/{aggregate_edge_id}/mapping
/api/aggregate-edges/{aggregate_edge_id}/matrix
/api/reports/{report_id}/raw-nodes
/api/reports/{report_id}/raw-edges
/api/reports/{report_id}/subgraph
```
Raw pages default to 100 and cannot exceed 500. Subgraph depth cannot exceed five. Raw and expanded display graphs default to a combined 5,000-node/edge safety cap. The API returns no partial graph when it is exceeded and tells the user to select an aggregate view or increase the configured `expansion_cap`. These limits are configurable when embedding `create_app`.
## Performance behavior
- CSV readers stream rows and insert them in bounded batches.
- Temporary SQLite staging tables are function-scoped and dropped immediately; bulk joins resolve endpoints without per-edge node queries.
- CSVs without additional columns use a constant empty JSON representation.
- Ingestion closes its write connection before derived indexes, aggregation, motifs, and diagnostics reopen the database.
- Secondary indexes are created after raw insertion and focus on serving and aggregation queries.
- Raw edges are never retained as a Python object graph or loaded into NetworkX.
- Only aggregate operation nodes/edges enter NetworkX.
- Mapping statistics are grouped in SQL; exact stencil comparison streams one operation pair.
- API raw pages, display projections, expansions, and matrices have hard limits.
- Stable IDs derive only from report IDs, grouping keys, and original CSV row numbers.
The normal suite contains a generated multi-lane smoke report. Set `RAPTOR_GRAPH_SLOW=1` to run the opt-in case with 100 operations, 1,000 lanes, and 100,000 raw edges:
## Tests and troubleshooting
```bash
RAPTOR_GRAPH_SLOW=1 python3.13 -m pytest tools/raptor_graph_explorer/tests/test_aggregation.py -m slow
.venv/bin/python -m pytest tools/raptor_graph_explorer/tests
RAPTOR_GRAPH_SLOW=1 .venv/bin/python -m pytest \
tools/raptor_graph_explorer/tests/test_aggregation.py -m slow
```
## Current limitations
The opt-in test generates 100 operations, 1,000 lanes, and 100,000 raw edges.
- The first browser load requires access to the pinned CDNs; offline vendoring is not included.
- Aggregate graph endpoints return the complete selected aggregate level. Raw graphs remain bounded, but an unusually huge operation aggregate may eventually need aggregate pagination.
- Motif representative paths operate at SCC-condensation granularity and choose a deterministic representative operation when an entry, exit, or branch SCC contains multiple operations.
- The API is local and read-only; authentication, write APIs, and a graph database server are intentionally out of scope.
## Troubleshooting
- **No complete report pairs:** check that both files share the exact prefix and end in `.nodes.csv` / `.edges.csv`.
- **Strict build failed:** run without `--strict`, then inspect the diagnostics panel or `diagnostics` table for bounded examples.
- **Output already exists:** pass `--overwrite` only when replacement is intended.
- **Unsupported schema:** rebuild the output with the installed tool version; old databases are not silently reinterpreted.
- **Raw/expanded display returns 400:** use Operation, Core, or Node kind view, or embed the app with a larger `expansion_cap` when the local machine can safely handle it.
- **Viewer is blank but APIs work:** verify browser access to the three pinned CDN URLs and inspect the browser console.
- **No motifs:** this is a valid result; motifs require a structural single-entry/single-exit split and rejoin, not a diamond-like screen shape.
No Raptor build or compiler modification is required.
- **No complete pairs:** check matching prefixes and both CSV suffixes.
- **Strict build failed:** inspect diagnostics from a build without `--strict`.
- **Raw/expanded display returns 400:** use aggregation or adjust the cap.
- **Blank viewer with working APIs:** check browser CDN access and console.
- **No motifs:** valid when no structural split/rejoin region exists.