This commit is contained in:
ilgeco
@@ -0,0 +1,226 @@
|
||||
# Raptor Graph Explorer
|
||||
|
||||
Raptor Graph Explorer is an external Python 3.13 tool that preprocesses Raptor's existing Spatial dataflow CSV reports into SQLite and serves an interactive browser viewer. It does not modify Raptor, compiler passes, MLIR, or CSV generation. The CSV files remain the source of truth.
|
||||
|
||||
Raw lane graphs can contain hundreds of thousands or millions of rows. Sending those rows to a browser makes layout, rendering, and even JSON serialization impractical. The explorer therefore loads an operation aggregate by default and retrieves raw rows only through bounded pages or a capped, explicitly requested lane expansion.
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
CSV pairs / directories / ZIP archives
|
||||
-> recursive discovery and safe ZIP extraction
|
||||
-> streaming CSV validation and SQLite staging
|
||||
-> normalized raw_nodes and raw_edges
|
||||
-> operation, core, and node-kind aggregates
|
||||
-> exact lane mappings
|
||||
-> SCC condensation and structural split/rejoin motifs
|
||||
-> graph.sqlite + manifest.json
|
||||
-> read-only FastAPI API
|
||||
-> Sigma.js WebGL viewer with stable operation/lane layout
|
||||
```
|
||||
|
||||
Node identifiers, operation IDs, lanes, cores, and channels are stored separately. In particular, graph lanes, physical fragment slots, scheduled lanes, physical cores, and communication channels are not treated as interchangeable identities.
|
||||
|
||||
## Installation
|
||||
|
||||
From the Raptor repository root:
|
||||
|
||||
```bash
|
||||
python3.13 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
python3.13 -m pip install -e tools/raptor_graph_explorer[dev]
|
||||
```
|
||||
|
||||
The installed command is `raptor-graph-explorer`. The equivalent module entry point is `python3.13 -m raptor_graph_explorer`.
|
||||
|
||||
## Commands
|
||||
|
||||
Build a persistent output:
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer build \
|
||||
validation/networks/vgg16/depth_07/raptor/reports \
|
||||
--output /tmp/vgg_graph
|
||||
```
|
||||
|
||||
Use repeated inputs or pass several paths after `build`. Inputs can be report directories, directories containing report subdirectories, ZIP archives, or explicit `.nodes.csv` / `.edges.csv` files. `--strict` rejects any data-integrity diagnostic. Without it, valid pairs and unambiguous rows are retained. `--overwrite` replaces an existing output.
|
||||
|
||||
Serve an existing output:
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer serve /tmp/vgg_graph --port 8765
|
||||
```
|
||||
|
||||
Build and immediately serve:
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer run \
|
||||
/mnt/data/conv_relu_conv.zip \
|
||||
--output /tmp/conv_relu_conv_graph \
|
||||
--overwrite
|
||||
```
|
||||
|
||||
Print counts, mapping classes, SCCs, motifs, and diagnostics:
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer inspect /tmp/vgg_graph
|
||||
```
|
||||
|
||||
The server defaults to `127.0.0.1:8765`. It has no authentication and is intended for local read-only use.
|
||||
|
||||
## Supported CSV reports
|
||||
|
||||
Files are paired using the complete prefix before `.nodes.csv` and `.edges.csv`. That prefix is the stable `report_id`; `spatial2_scheduled_no_comm.nodes.csv` therefore pairs only with `spatial2_scheduled_no_comm.edges.csv`.
|
||||
|
||||
Node columns:
|
||||
|
||||
```csv
|
||||
Id,op_id,lane,core,ssa_name
|
||||
```
|
||||
|
||||
Edge columns:
|
||||
|
||||
```csv
|
||||
Source,Target,Weight,Type,stage,source_lane,target_lane,channel_id
|
||||
```
|
||||
|
||||
Known node prefixes are `gc`, `gcb`, `sc`, and `scb`. Unknown prefixes are retained as `unknown:<prefix>`. Explicit CSV `op_id` and lane columns are authoritative; a blank value may fall back to a parseable node ID and produces a diagnostic. Explicit edge lanes take precedence over endpoint lanes. Missing numeric values remain SQL `NULL`; malformed numeric values are never changed to zero. Additional CSV columns are preserved in `attributes_json`.
|
||||
|
||||
Edges are resolved only against nodes from the same report. Unknown endpoints omit that edge and produce a bounded diagnostic. ZIP paths are checked before extraction and traversal entries are rejected.
|
||||
|
||||
## Output
|
||||
|
||||
```text
|
||||
OUTPUT_DIRECTORY/
|
||||
├── graph.sqlite
|
||||
└── manifest.json
|
||||
```
|
||||
|
||||
`graph.sqlite` contains reports, normalized raw nodes and edges, aggregate nodes and edges, motifs, motif membership, diagnostics, and focused indexes. `manifest.json` contains schema/tool versions and compact per-report counts. It never duplicates raw graph rows or stores machine-specific input paths.
|
||||
|
||||
Schema version 1 is checked whenever an output is served. Unsupported or missing versions fail at startup with a clear error.
|
||||
|
||||
## Aggregate views
|
||||
|
||||
### Operation
|
||||
|
||||
Nodes group by `(report_id, op_id)` and retain node-kind counts, instance count, lane and core cardinalities/ranges, and an SSA summary. Edges group by `(report_id, source_op_id, target_op_id)` and retain raw count, weight statistics, tensor histogram, distinct channels, and exact mapping classification. This is the default view and the only graph used for motif analysis.
|
||||
|
||||
### Core
|
||||
|
||||
Assigned nodes group by `(report_id, core)`. Missing cores use a separate `unassigned` group, never core zero. Edges retain raw count, total weight, channel count, operation-pair count, and tensor histogram.
|
||||
|
||||
### Node kind
|
||||
|
||||
Nodes group by `(report_id, node_kind)`. This deliberately small diagnostic view shows how graph/scalar/batch/scheduled kinds connect; it does not replace operation aggregation.
|
||||
|
||||
### Raw instances
|
||||
|
||||
Raw instances displays every `raw_nodes` row exactly once and every valid `raw_edges` row exactly once. Nodes are queried independently from edges, so degree-zero nodes remain visible. A spatial3 report starts in this view automatically; Operation, Core, and Node kind remain selectable.
|
||||
|
||||
Spatial3 CSVs record explicit communication only and may legitimately contain many isolated nodes. The explorer never fabricates dependencies from semicolon-separated SSA names or other node text.
|
||||
|
||||
## Exact lane mappings
|
||||
|
||||
Classifications use all normalized source/target lane pairs for an operation edge. They are evaluated in this order:
|
||||
|
||||
- `scalar_to_scalar`: every pair has two absent lanes.
|
||||
- `scalar_to_batch`: every source lane is absent and every target lane is present; target range, cardinality, and contiguity are recorded.
|
||||
- `batch_to_scalar`: the corresponding source-lane case.
|
||||
- `identity`: a one-to-one relation with equal source and target lanes.
|
||||
- `constant_offset`: a one-to-one relation with one nonzero `target - source` offset.
|
||||
- `permutation`: a one-to-one relation that is neither identity nor constant offset.
|
||||
- `broadcast`: one distinct source lane maps to multiple target lanes.
|
||||
- `fan_in`: multiple source lanes map to one distinct target lane.
|
||||
- `stencil`: more than one source has exactly the same bounded target-offset set; the complete sorted set is recorded.
|
||||
- `irregular`: complete lane data matches none of the exact forms.
|
||||
- `unknown`: lane data is partially missing outside the scalar/batch cases.
|
||||
|
||||
No sampling is used. SQL computes cardinalities and degrees; stencil offsets stream one operation edge at a time. Raw mapping rows remain paginated, and the matrix endpoint returns exact sparse points only below its threshold. Larger relations become a bounded 64×64 density matrix.
|
||||
|
||||
## Structural motifs
|
||||
|
||||
Motifs are not inferred from rendered geometry. For each operation graph the tool:
|
||||
|
||||
1. computes strongly connected components and condenses them into a DAG;
|
||||
2. attaches synthetic source and sink nodes;
|
||||
3. computes dominators and reverse-graph post-dominators;
|
||||
4. considers entries with at least two outgoing branches and their immediate post-dominator exit;
|
||||
5. retains nodes reachable before the exit that can also reach the exit;
|
||||
6. rejects unexpected entrances to, or exits from, internal nodes;
|
||||
7. records one shortest representative path per branch without enumerating all paths;
|
||||
8. maps condensed components back to operation aggregate IDs.
|
||||
|
||||
`exact_diamond` is a two-branch direct reconvergence with one internal condensed node per branch. `rhomboid` has clean disjoint branch interiors. `split_rejoin` is the general valid single-entry/single-exit region. Identical memberships are deduplicated while meaningful nested motifs are preserved. SCC membership marks `contains_cycle`; cycles are never analyzed as if the raw operation graph were a DAG.
|
||||
|
||||
## Viewer and API
|
||||
|
||||
Except for spatial3, the viewer initially fetches an aggregate operation graph unless the browser has a saved view choice. Sigma renders the graph with WebGL. Controls cover report/view/metric selection, text and tensor search, mapping filters, self edges, relayout, fitting, and motif selection. Raw nodes and edges have their own selection details and never call aggregate-only detail or mapping endpoints. Mapping panels remain available for operation aggregate edges.
|
||||
|
||||
Operation expansion is a deterministic projection of the source graph. Expand selected, Expand all operations, Collapse selected operation, and Collapse all rebuild the complete display from the current expanded-operation set. An expanded operation's raw nodes replace its aggregate node, including isolated raw nodes. An aggregate edge is retained only when both endpoint operations are collapsed; otherwise its raw edges replace it with endpoints calculated from the complete expansion set. Expansion order therefore cannot leave stale endpoints or simultaneous aggregate/raw representations.
|
||||
|
||||
Operation ranks are calculated from the complete operation graph, including isolated operations. Collapsed nodes use stable operation anchors. Expanded nodes are sorted by lane and node ID; lane numbers form perpendicular rows, equal lanes align across adjacent operations, and lane-less nodes use distinct deterministic scalar rows. This same model is used by Rerun layout, so unrelated operations do not jump during expansion or collapse and disconnected nodes never pile up at `(0, 0)`.
|
||||
|
||||
Browser dependencies are pinned in one place, `static/index.html`:
|
||||
|
||||
- Graphology 0.25.4;
|
||||
- Sigma.js 2.4.0;
|
||||
- ELK.js 0.11.1.
|
||||
|
||||
They load from CDNs, so the browser viewer needs network access on first use. Python preprocessing, inspection, and API tests do not.
|
||||
|
||||
Read-only endpoints include:
|
||||
|
||||
```text
|
||||
/api/manifest
|
||||
/api/reports
|
||||
/api/reports/{report_id}/summary
|
||||
/api/reports/{report_id}/graph/{level}
|
||||
/api/reports/{report_id}/display-graph
|
||||
/api/reports/{report_id}/motifs
|
||||
/api/aggregate-nodes/{aggregate_id}
|
||||
/api/aggregate-edges/{aggregate_edge_id}
|
||||
/api/aggregate-edges/{aggregate_edge_id}/mapping
|
||||
/api/aggregate-edges/{aggregate_edge_id}/matrix
|
||||
/api/reports/{report_id}/raw-nodes
|
||||
/api/reports/{report_id}/raw-edges
|
||||
/api/reports/{report_id}/subgraph
|
||||
```
|
||||
|
||||
Raw pages default to 100 and cannot exceed 500. Subgraph depth cannot exceed five. Raw and expanded display graphs default to a combined 5,000-node/edge safety cap. The API returns no partial graph when it is exceeded and tells the user to select an aggregate view or increase the configured `expansion_cap`. These limits are configurable when embedding `create_app`.
|
||||
|
||||
## Performance behavior
|
||||
|
||||
- CSV readers stream rows and insert them in bounded batches.
|
||||
- Temporary SQLite staging and bulk joins resolve endpoints; ingestion performs no per-edge node query.
|
||||
- Secondary indexes are created after raw insertion.
|
||||
- Raw edges are never retained as a Python object graph or loaded into NetworkX.
|
||||
- Only aggregate operation nodes/edges enter NetworkX.
|
||||
- Mapping statistics are grouped in SQL; exact stencil comparison streams one operation pair.
|
||||
- API raw pages, display projections, expansions, and matrices have hard limits.
|
||||
- Stable IDs derive only from report IDs, grouping keys, and original CSV row numbers.
|
||||
|
||||
The normal suite contains a generated multi-lane smoke report. Set `RAPTOR_GRAPH_SLOW=1` to run the opt-in case with 100 operations, 1,000 lanes, and 100,000 raw edges:
|
||||
|
||||
```bash
|
||||
RAPTOR_GRAPH_SLOW=1 python3.13 -m pytest tools/raptor_graph_explorer/tests/test_aggregation.py -m slow
|
||||
```
|
||||
|
||||
## Current limitations
|
||||
|
||||
- The first browser load requires access to the pinned CDNs; offline vendoring is not included.
|
||||
- Aggregate graph endpoints return the complete selected aggregate level. Raw graphs remain bounded, but an unusually huge operation aggregate may eventually need aggregate pagination.
|
||||
- Motif representative paths operate at SCC-condensation granularity and choose a deterministic representative operation when an entry, exit, or branch SCC contains multiple operations.
|
||||
- The API is local and read-only; authentication, write APIs, and a graph database server are intentionally out of scope.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **No complete report pairs:** check that both files share the exact prefix and end in `.nodes.csv` / `.edges.csv`.
|
||||
- **Strict build failed:** run without `--strict`, then inspect the diagnostics panel or `diagnostics` table for bounded examples.
|
||||
- **Output already exists:** pass `--overwrite` only when replacement is intended.
|
||||
- **Unsupported schema:** rebuild the output with the installed tool version; old databases are not silently reinterpreted.
|
||||
- **Raw/expanded display returns 400:** use Operation, Core, or Node kind view, or embed the app with a larger `expansion_cap` when the local machine can safely handle it.
|
||||
- **Viewer is blank but APIs work:** verify browser access to the three pinned CDN URLs and inspect the browser console.
|
||||
- **No motifs:** this is a valid result; motifs require a structural single-entry/single-exit split and rejoin, not a diamond-like screen shape.
|
||||
|
||||
No Raptor build or compiler modification is required.
|
||||
Reference in New Issue
Block a user