This commit is contained in:
ilgeco
@@ -0,0 +1,226 @@
|
||||
# Raptor Graph Explorer
|
||||
|
||||
Raptor Graph Explorer is an external Python 3.13 tool that preprocesses Raptor's existing Spatial dataflow CSV reports into SQLite and serves an interactive browser viewer. It does not modify Raptor, compiler passes, MLIR, or CSV generation. The CSV files remain the source of truth.
|
||||
|
||||
Raw lane graphs can contain hundreds of thousands or millions of rows. Sending those rows to a browser makes layout, rendering, and even JSON serialization impractical. The explorer therefore loads an operation aggregate by default and retrieves raw rows only through bounded pages or a capped, explicitly requested lane expansion.
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
CSV pairs / directories / ZIP archives
|
||||
-> recursive discovery and safe ZIP extraction
|
||||
-> streaming CSV validation and SQLite staging
|
||||
-> normalized raw_nodes and raw_edges
|
||||
-> operation, core, and node-kind aggregates
|
||||
-> exact lane mappings
|
||||
-> SCC condensation and structural split/rejoin motifs
|
||||
-> graph.sqlite + manifest.json
|
||||
-> read-only FastAPI API
|
||||
-> Sigma.js WebGL viewer with stable operation/lane layout
|
||||
```
|
||||
|
||||
Node identifiers, operation IDs, lanes, cores, and channels are stored separately. In particular, graph lanes, physical fragment slots, scheduled lanes, physical cores, and communication channels are not treated as interchangeable identities.
|
||||
|
||||
## Installation
|
||||
|
||||
From the Raptor repository root:
|
||||
|
||||
```bash
|
||||
python3.13 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
python3.13 -m pip install -e tools/raptor_graph_explorer[dev]
|
||||
```
|
||||
|
||||
The installed command is `raptor-graph-explorer`. The equivalent module entry point is `python3.13 -m raptor_graph_explorer`.
|
||||
|
||||
## Commands
|
||||
|
||||
Build a persistent output:
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer build \
|
||||
validation/networks/vgg16/depth_07/raptor/reports \
|
||||
--output /tmp/vgg_graph
|
||||
```
|
||||
|
||||
Use repeated inputs or pass several paths after `build`. Inputs can be report directories, directories containing report subdirectories, ZIP archives, or explicit `.nodes.csv` / `.edges.csv` files. `--strict` rejects any data-integrity diagnostic. Without it, valid pairs and unambiguous rows are retained. `--overwrite` replaces an existing output.
|
||||
|
||||
Serve an existing output:
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer serve /tmp/vgg_graph --port 8765
|
||||
```
|
||||
|
||||
Build and immediately serve:
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer run \
|
||||
/mnt/data/conv_relu_conv.zip \
|
||||
--output /tmp/conv_relu_conv_graph \
|
||||
--overwrite
|
||||
```
|
||||
|
||||
Print counts, mapping classes, SCCs, motifs, and diagnostics:
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer inspect /tmp/vgg_graph
|
||||
```
|
||||
|
||||
The server defaults to `127.0.0.1:8765`. It has no authentication and is intended for local read-only use.
|
||||
|
||||
## Supported CSV reports
|
||||
|
||||
Files are paired using the complete prefix before `.nodes.csv` and `.edges.csv`. That prefix is the stable `report_id`; `spatial2_scheduled_no_comm.nodes.csv` therefore pairs only with `spatial2_scheduled_no_comm.edges.csv`.
|
||||
|
||||
Node columns:
|
||||
|
||||
```csv
|
||||
Id,op_id,lane,core,ssa_name
|
||||
```
|
||||
|
||||
Edge columns:
|
||||
|
||||
```csv
|
||||
Source,Target,Weight,Type,stage,source_lane,target_lane,channel_id
|
||||
```
|
||||
|
||||
Known node prefixes are `gc`, `gcb`, `sc`, and `scb`. Unknown prefixes are retained as `unknown:<prefix>`. Explicit CSV `op_id` and lane columns are authoritative; a blank value may fall back to a parseable node ID and produces a diagnostic. Explicit edge lanes take precedence over endpoint lanes. Missing numeric values remain SQL `NULL`; malformed numeric values are never changed to zero. Additional CSV columns are preserved in `attributes_json`.
|
||||
|
||||
Edges are resolved only against nodes from the same report. Unknown endpoints omit that edge and produce a bounded diagnostic. ZIP paths are checked before extraction and traversal entries are rejected.
|
||||
|
||||
## Output
|
||||
|
||||
```text
|
||||
OUTPUT_DIRECTORY/
|
||||
├── graph.sqlite
|
||||
└── manifest.json
|
||||
```
|
||||
|
||||
`graph.sqlite` contains reports, normalized raw nodes and edges, aggregate nodes and edges, motifs, motif membership, diagnostics, and focused indexes. `manifest.json` contains schema/tool versions and compact per-report counts. It never duplicates raw graph rows or stores machine-specific input paths.
|
||||
|
||||
Schema version 1 is checked whenever an output is served. Unsupported or missing versions fail at startup with a clear error.
|
||||
|
||||
## Aggregate views
|
||||
|
||||
### Operation
|
||||
|
||||
Nodes group by `(report_id, op_id)` and retain node-kind counts, instance count, lane and core cardinalities/ranges, and an SSA summary. Edges group by `(report_id, source_op_id, target_op_id)` and retain raw count, weight statistics, tensor histogram, distinct channels, and exact mapping classification. This is the default view and the only graph used for motif analysis.
|
||||
|
||||
### Core
|
||||
|
||||
Assigned nodes group by `(report_id, core)`. Missing cores use a separate `unassigned` group, never core zero. Edges retain raw count, total weight, channel count, operation-pair count, and tensor histogram.
|
||||
|
||||
### Node kind
|
||||
|
||||
Nodes group by `(report_id, node_kind)`. This deliberately small diagnostic view shows how graph/scalar/batch/scheduled kinds connect; it does not replace operation aggregation.
|
||||
|
||||
### Raw instances
|
||||
|
||||
Raw instances displays every `raw_nodes` row exactly once and every valid `raw_edges` row exactly once. Nodes are queried independently from edges, so degree-zero nodes remain visible. A spatial3 report starts in this view automatically; Operation, Core, and Node kind remain selectable.
|
||||
|
||||
Spatial3 CSVs record explicit communication only and may legitimately contain many isolated nodes. The explorer never fabricates dependencies from semicolon-separated SSA names or other node text.
|
||||
|
||||
## Exact lane mappings
|
||||
|
||||
Classifications use all normalized source/target lane pairs for an operation edge. They are evaluated in this order:
|
||||
|
||||
- `scalar_to_scalar`: every pair has two absent lanes.
|
||||
- `scalar_to_batch`: every source lane is absent and every target lane is present; target range, cardinality, and contiguity are recorded.
|
||||
- `batch_to_scalar`: the corresponding source-lane case.
|
||||
- `identity`: a one-to-one relation with equal source and target lanes.
|
||||
- `constant_offset`: a one-to-one relation with one nonzero `target - source` offset.
|
||||
- `permutation`: a one-to-one relation that is neither identity nor constant offset.
|
||||
- `broadcast`: one distinct source lane maps to multiple target lanes.
|
||||
- `fan_in`: multiple source lanes map to one distinct target lane.
|
||||
- `stencil`: more than one source has exactly the same bounded target-offset set; the complete sorted set is recorded.
|
||||
- `irregular`: complete lane data matches none of the exact forms.
|
||||
- `unknown`: lane data is partially missing outside the scalar/batch cases.
|
||||
|
||||
No sampling is used. SQL computes cardinalities and degrees; stencil offsets stream one operation edge at a time. Raw mapping rows remain paginated, and the matrix endpoint returns exact sparse points only below its threshold. Larger relations become a bounded 64×64 density matrix.
|
||||
|
||||
## Structural motifs
|
||||
|
||||
Motifs are not inferred from rendered geometry. For each operation graph the tool:
|
||||
|
||||
1. computes strongly connected components and condenses them into a DAG;
|
||||
2. attaches synthetic source and sink nodes;
|
||||
3. computes dominators and reverse-graph post-dominators;
|
||||
4. considers entries with at least two outgoing branches and their immediate post-dominator exit;
|
||||
5. retains nodes reachable before the exit that can also reach the exit;
|
||||
6. rejects unexpected entrances to, or exits from, internal nodes;
|
||||
7. records one shortest representative path per branch without enumerating all paths;
|
||||
8. maps condensed components back to operation aggregate IDs.
|
||||
|
||||
`exact_diamond` is a two-branch direct reconvergence with one internal condensed node per branch. `rhomboid` has clean disjoint branch interiors. `split_rejoin` is the general valid single-entry/single-exit region. Identical memberships are deduplicated while meaningful nested motifs are preserved. SCC membership marks `contains_cycle`; cycles are never analyzed as if the raw operation graph were a DAG.
|
||||
|
||||
## Viewer and API
|
||||
|
||||
Except for spatial3, the viewer initially fetches an aggregate operation graph unless the browser has a saved view choice. Sigma renders the graph with WebGL. Controls cover report/view/metric selection, text and tensor search, mapping filters, self edges, relayout, fitting, and motif selection. Raw nodes and edges have their own selection details and never call aggregate-only detail or mapping endpoints. Mapping panels remain available for operation aggregate edges.
|
||||
|
||||
Operation expansion is a deterministic projection of the source graph. Expand selected, Expand all operations, Collapse selected operation, and Collapse all rebuild the complete display from the current expanded-operation set. An expanded operation's raw nodes replace its aggregate node, including isolated raw nodes. An aggregate edge is retained only when both endpoint operations are collapsed; otherwise its raw edges replace it with endpoints calculated from the complete expansion set. Expansion order therefore cannot leave stale endpoints or simultaneous aggregate/raw representations.
|
||||
|
||||
Operation ranks are calculated from the complete operation graph, including isolated operations. Collapsed nodes use stable operation anchors. Expanded nodes are sorted by lane and node ID; lane numbers form perpendicular rows, equal lanes align across adjacent operations, and lane-less nodes use distinct deterministic scalar rows. This same model is used by Rerun layout, so unrelated operations do not jump during expansion or collapse and disconnected nodes never pile up at `(0, 0)`.
|
||||
|
||||
Browser dependencies are pinned in one place, `static/index.html`:
|
||||
|
||||
- Graphology 0.25.4;
|
||||
- Sigma.js 2.4.0;
|
||||
- ELK.js 0.11.1.
|
||||
|
||||
They load from CDNs, so the browser viewer needs network access on first use. Python preprocessing, inspection, and API tests do not.
|
||||
|
||||
Read-only endpoints include:
|
||||
|
||||
```text
|
||||
/api/manifest
|
||||
/api/reports
|
||||
/api/reports/{report_id}/summary
|
||||
/api/reports/{report_id}/graph/{level}
|
||||
/api/reports/{report_id}/display-graph
|
||||
/api/reports/{report_id}/motifs
|
||||
/api/aggregate-nodes/{aggregate_id}
|
||||
/api/aggregate-edges/{aggregate_edge_id}
|
||||
/api/aggregate-edges/{aggregate_edge_id}/mapping
|
||||
/api/aggregate-edges/{aggregate_edge_id}/matrix
|
||||
/api/reports/{report_id}/raw-nodes
|
||||
/api/reports/{report_id}/raw-edges
|
||||
/api/reports/{report_id}/subgraph
|
||||
```
|
||||
|
||||
Raw pages default to 100 and cannot exceed 500. Subgraph depth cannot exceed five. Raw and expanded display graphs default to a combined 5,000-node/edge safety cap. The API returns no partial graph when it is exceeded and tells the user to select an aggregate view or increase the configured `expansion_cap`. These limits are configurable when embedding `create_app`.
|
||||
|
||||
## Performance behavior
|
||||
|
||||
- CSV readers stream rows and insert them in bounded batches.
|
||||
- Temporary SQLite staging and bulk joins resolve endpoints; ingestion performs no per-edge node query.
|
||||
- Secondary indexes are created after raw insertion.
|
||||
- Raw edges are never retained as a Python object graph or loaded into NetworkX.
|
||||
- Only aggregate operation nodes/edges enter NetworkX.
|
||||
- Mapping statistics are grouped in SQL; exact stencil comparison streams one operation pair.
|
||||
- API raw pages, display projections, expansions, and matrices have hard limits.
|
||||
- Stable IDs derive only from report IDs, grouping keys, and original CSV row numbers.
|
||||
|
||||
The normal suite contains a generated multi-lane smoke report. Set `RAPTOR_GRAPH_SLOW=1` to run the opt-in case with 100 operations, 1,000 lanes, and 100,000 raw edges:
|
||||
|
||||
```bash
|
||||
RAPTOR_GRAPH_SLOW=1 python3.13 -m pytest tools/raptor_graph_explorer/tests/test_aggregation.py -m slow
|
||||
```
|
||||
|
||||
## Current limitations
|
||||
|
||||
- The first browser load requires access to the pinned CDNs; offline vendoring is not included.
|
||||
- Aggregate graph endpoints return the complete selected aggregate level. Raw graphs remain bounded, but an unusually huge operation aggregate may eventually need aggregate pagination.
|
||||
- Motif representative paths operate at SCC-condensation granularity and choose a deterministic representative operation when an entry, exit, or branch SCC contains multiple operations.
|
||||
- The API is local and read-only; authentication, write APIs, and a graph database server are intentionally out of scope.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **No complete report pairs:** check that both files share the exact prefix and end in `.nodes.csv` / `.edges.csv`.
|
||||
- **Strict build failed:** run without `--strict`, then inspect the diagnostics panel or `diagnostics` table for bounded examples.
|
||||
- **Output already exists:** pass `--overwrite` only when replacement is intended.
|
||||
- **Unsupported schema:** rebuild the output with the installed tool version; old databases are not silently reinterpreted.
|
||||
- **Raw/expanded display returns 400:** use Operation, Core, or Node kind view, or embed the app with a larger `expansion_cap` when the local machine can safely handle it.
|
||||
- **Viewer is blank but APIs work:** verify browser access to the three pinned CDN URLs and inspect the browser console.
|
||||
- **No motifs:** this is a valid result; motifs require a structural single-entry/single-exit split and rejoin, not a diamond-like screen shape.
|
||||
|
||||
No Raptor build or compiler modification is required.
|
||||
@@ -0,0 +1,23 @@
|
||||
[build-system]
|
||||
requires = ["setuptools>=75"]
|
||||
build-backend = "setuptools.build_meta"
|
||||
|
||||
[project]
|
||||
name = "raptor-graph-explorer"
|
||||
version = "0.1.0"
|
||||
description = "Preprocess and interactively explore Raptor graph CSV reports"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.13"
|
||||
dependencies = ["fastapi>=0.115,<0.116", "networkx>=3.4,<4", "uvicorn>=0.34,<0.35"]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = ["anyio>=4.6,<4.10", "httpx>=0.28,<0.29", "pytest>=8.3,<9"]
|
||||
|
||||
[project.scripts]
|
||||
raptor-graph-explorer = "raptor_graph_explorer.cli:main"
|
||||
|
||||
[tool.setuptools.package-data]
|
||||
raptor_graph_explorer = ["static/*"]
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
markers = ["slow: generated large-report smoke tests"]
|
||||
@@ -0,0 +1,240 @@
|
||||
Metadata-Version: 2.4
|
||||
Name: raptor-graph-explorer
|
||||
Version: 0.1.0
|
||||
Summary: Preprocess and interactively explore Raptor graph CSV reports
|
||||
Requires-Python: >=3.13
|
||||
Description-Content-Type: text/markdown
|
||||
Requires-Dist: fastapi<0.116,>=0.115
|
||||
Requires-Dist: networkx<4,>=3.4
|
||||
Requires-Dist: uvicorn<0.35,>=0.34
|
||||
Provides-Extra: dev
|
||||
Requires-Dist: anyio<4.10,>=4.6; extra == "dev"
|
||||
Requires-Dist: httpx<0.29,>=0.28; extra == "dev"
|
||||
Requires-Dist: pytest<9,>=8.3; extra == "dev"
|
||||
|
||||
# Raptor Graph Explorer
|
||||
|
||||
Raptor Graph Explorer is an external Python 3.13 tool that preprocesses Raptor's existing Spatial dataflow CSV reports into SQLite and serves an interactive browser viewer. It does not modify Raptor, compiler passes, MLIR, or CSV generation. The CSV files remain the source of truth.
|
||||
|
||||
Raw lane graphs can contain hundreds of thousands or millions of rows. Sending those rows to a browser makes layout, rendering, and even JSON serialization impractical. The explorer therefore loads an operation aggregate by default and retrieves raw rows only through bounded pages or a capped, explicitly requested lane expansion.
|
||||
|
||||
## Architecture
|
||||
|
||||
```text
|
||||
CSV pairs / directories / ZIP archives
|
||||
-> recursive discovery and safe ZIP extraction
|
||||
-> streaming CSV validation and SQLite staging
|
||||
-> normalized raw_nodes and raw_edges
|
||||
-> operation, core, and node-kind aggregates
|
||||
-> exact lane mappings
|
||||
-> SCC condensation and structural split/rejoin motifs
|
||||
-> graph.sqlite + manifest.json
|
||||
-> read-only FastAPI API
|
||||
-> Sigma.js WebGL viewer with stable operation/lane layout
|
||||
```
|
||||
|
||||
Node identifiers, operation IDs, lanes, cores, and channels are stored separately. In particular, graph lanes, physical fragment slots, scheduled lanes, physical cores, and communication channels are not treated as interchangeable identities.
|
||||
|
||||
## Installation
|
||||
|
||||
From the Raptor repository root:
|
||||
|
||||
```bash
|
||||
python3.13 -m venv .venv
|
||||
source .venv/bin/activate
|
||||
python3.13 -m pip install -e tools/raptor_graph_explorer[dev]
|
||||
```
|
||||
|
||||
The installed command is `raptor-graph-explorer`. The equivalent module entry point is `python3.13 -m raptor_graph_explorer`.
|
||||
|
||||
## Commands
|
||||
|
||||
Build a persistent output:
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer build \
|
||||
validation/networks/vgg16/depth_07/raptor/reports \
|
||||
--output /tmp/vgg_graph
|
||||
```
|
||||
|
||||
Use repeated inputs or pass several paths after `build`. Inputs can be report directories, directories containing report subdirectories, ZIP archives, or explicit `.nodes.csv` / `.edges.csv` files. `--strict` rejects any data-integrity diagnostic. Without it, valid pairs and unambiguous rows are retained. `--overwrite` replaces an existing output.
|
||||
|
||||
Serve an existing output:
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer serve /tmp/vgg_graph --port 8765
|
||||
```
|
||||
|
||||
Build and immediately serve:
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer run \
|
||||
/mnt/data/conv_relu_conv.zip \
|
||||
--output /tmp/conv_relu_conv_graph \
|
||||
--overwrite
|
||||
```
|
||||
|
||||
Print counts, mapping classes, SCCs, motifs, and diagnostics:
|
||||
|
||||
```bash
|
||||
raptor-graph-explorer inspect /tmp/vgg_graph
|
||||
```
|
||||
|
||||
The server defaults to `127.0.0.1:8765`. It has no authentication and is intended for local read-only use.
|
||||
|
||||
## Supported CSV reports
|
||||
|
||||
Files are paired using the complete prefix before `.nodes.csv` and `.edges.csv`. That prefix is the stable `report_id`; `spatial2_scheduled_no_comm.nodes.csv` therefore pairs only with `spatial2_scheduled_no_comm.edges.csv`.
|
||||
|
||||
Node columns:
|
||||
|
||||
```csv
|
||||
Id,op_id,lane,core,ssa_name
|
||||
```
|
||||
|
||||
Edge columns:
|
||||
|
||||
```csv
|
||||
Source,Target,Weight,Type,stage,source_lane,target_lane,channel_id
|
||||
```
|
||||
|
||||
Known node prefixes are `gc`, `gcb`, `sc`, and `scb`. Unknown prefixes are retained as `unknown:<prefix>`. Explicit CSV `op_id` and lane columns are authoritative; a blank value may fall back to a parseable node ID and produces a diagnostic. Explicit edge lanes take precedence over endpoint lanes. Missing numeric values remain SQL `NULL`; malformed numeric values are never changed to zero. Additional CSV columns are preserved in `attributes_json`.
|
||||
|
||||
Edges are resolved only against nodes from the same report. Unknown endpoints omit that edge and produce a bounded diagnostic. ZIP paths are checked before extraction and traversal entries are rejected.
|
||||
|
||||
## Output
|
||||
|
||||
```text
|
||||
OUTPUT_DIRECTORY/
|
||||
├── graph.sqlite
|
||||
└── manifest.json
|
||||
```
|
||||
|
||||
`graph.sqlite` contains reports, normalized raw nodes and edges, aggregate nodes and edges, motifs, motif membership, diagnostics, and focused indexes. `manifest.json` contains schema/tool versions and compact per-report counts. It never duplicates raw graph rows or stores machine-specific input paths.
|
||||
|
||||
Schema version 1 is checked whenever an output is served. Unsupported or missing versions fail at startup with a clear error.
|
||||
|
||||
## Aggregate views
|
||||
|
||||
### Operation
|
||||
|
||||
Nodes group by `(report_id, op_id)` and retain node-kind counts, instance count, lane and core cardinalities/ranges, and an SSA summary. Edges group by `(report_id, source_op_id, target_op_id)` and retain raw count, weight statistics, tensor histogram, distinct channels, and exact mapping classification. This is the default view and the only graph used for motif analysis.
|
||||
|
||||
### Core
|
||||
|
||||
Assigned nodes group by `(report_id, core)`. Missing cores use a separate `unassigned` group, never core zero. Edges retain raw count, total weight, channel count, operation-pair count, and tensor histogram.
|
||||
|
||||
### Node kind
|
||||
|
||||
Nodes group by `(report_id, node_kind)`. This deliberately small diagnostic view shows how graph/scalar/batch/scheduled kinds connect; it does not replace operation aggregation.
|
||||
|
||||
### Raw instances
|
||||
|
||||
Raw instances displays every `raw_nodes` row exactly once and every valid `raw_edges` row exactly once. Nodes are queried independently from edges, so degree-zero nodes remain visible. A spatial3 report starts in this view automatically; Operation, Core, and Node kind remain selectable.
|
||||
|
||||
Spatial3 CSVs record explicit communication only and may legitimately contain many isolated nodes. The explorer never fabricates dependencies from semicolon-separated SSA names or other node text.
|
||||
|
||||
## Exact lane mappings
|
||||
|
||||
Classifications use all normalized source/target lane pairs for an operation edge. They are evaluated in this order:
|
||||
|
||||
- `scalar_to_scalar`: every pair has two absent lanes.
|
||||
- `scalar_to_batch`: every source lane is absent and every target lane is present; target range, cardinality, and contiguity are recorded.
|
||||
- `batch_to_scalar`: the corresponding source-lane case.
|
||||
- `identity`: a one-to-one relation with equal source and target lanes.
|
||||
- `constant_offset`: a one-to-one relation with one nonzero `target - source` offset.
|
||||
- `permutation`: a one-to-one relation that is neither identity nor constant offset.
|
||||
- `broadcast`: one distinct source lane maps to multiple target lanes.
|
||||
- `fan_in`: multiple source lanes map to one distinct target lane.
|
||||
- `stencil`: more than one source has exactly the same bounded target-offset set; the complete sorted set is recorded.
|
||||
- `irregular`: complete lane data matches none of the exact forms.
|
||||
- `unknown`: lane data is partially missing outside the scalar/batch cases.
|
||||
|
||||
No sampling is used. SQL computes cardinalities and degrees; stencil offsets stream one operation edge at a time. Raw mapping rows remain paginated, and the matrix endpoint returns exact sparse points only below its threshold. Larger relations become a bounded 64×64 density matrix.
|
||||
|
||||
## Structural motifs
|
||||
|
||||
Motifs are not inferred from rendered geometry. For each operation graph the tool:
|
||||
|
||||
1. computes strongly connected components and condenses them into a DAG;
|
||||
2. attaches synthetic source and sink nodes;
|
||||
3. computes dominators and reverse-graph post-dominators;
|
||||
4. considers entries with at least two outgoing branches and their immediate post-dominator exit;
|
||||
5. retains nodes reachable before the exit that can also reach the exit;
|
||||
6. rejects unexpected entrances to, or exits from, internal nodes;
|
||||
7. records one shortest representative path per branch without enumerating all paths;
|
||||
8. maps condensed components back to operation aggregate IDs.
|
||||
|
||||
`exact_diamond` is a two-branch direct reconvergence with one internal condensed node per branch. `rhomboid` has clean disjoint branch interiors. `split_rejoin` is the general valid single-entry/single-exit region. Identical memberships are deduplicated while meaningful nested motifs are preserved. SCC membership marks `contains_cycle`; cycles are never analyzed as if the raw operation graph were a DAG.
|
||||
|
||||
## Viewer and API
|
||||
|
||||
Except for spatial3, the viewer initially fetches an aggregate operation graph unless the browser has a saved view choice. Sigma renders the graph with WebGL. Controls cover report/view/metric selection, text and tensor search, mapping filters, self edges, relayout, fitting, and motif selection. Raw nodes and edges have their own selection details and never call aggregate-only detail or mapping endpoints. Mapping panels remain available for operation aggregate edges.
|
||||
|
||||
Operation expansion is a deterministic projection of the source graph. Expand selected, Expand all operations, Collapse selected operation, and Collapse all rebuild the complete display from the current expanded-operation set. An expanded operation's raw nodes replace its aggregate node, including isolated raw nodes. An aggregate edge is retained only when both endpoint operations are collapsed; otherwise its raw edges replace it with endpoints calculated from the complete expansion set. Expansion order therefore cannot leave stale endpoints or simultaneous aggregate/raw representations.
|
||||
|
||||
Operation ranks are calculated from the complete operation graph, including isolated operations. Collapsed nodes use stable operation anchors. Expanded nodes are sorted by lane and node ID; lane numbers form perpendicular rows, equal lanes align across adjacent operations, and lane-less nodes use distinct deterministic scalar rows. This same model is used by Rerun layout, so unrelated operations do not jump during expansion or collapse and disconnected nodes never pile up at `(0, 0)`.
|
||||
|
||||
Browser dependencies are pinned in one place, `static/index.html`:
|
||||
|
||||
- Graphology 0.25.4;
|
||||
- Sigma.js 2.4.0;
|
||||
- ELK.js 0.11.1.
|
||||
|
||||
They load from CDNs, so the browser viewer needs network access on first use. Python preprocessing, inspection, and API tests do not.
|
||||
|
||||
Read-only endpoints include:
|
||||
|
||||
```text
|
||||
/api/manifest
|
||||
/api/reports
|
||||
/api/reports/{report_id}/summary
|
||||
/api/reports/{report_id}/graph/{level}
|
||||
/api/reports/{report_id}/display-graph
|
||||
/api/reports/{report_id}/motifs
|
||||
/api/aggregate-nodes/{aggregate_id}
|
||||
/api/aggregate-edges/{aggregate_edge_id}
|
||||
/api/aggregate-edges/{aggregate_edge_id}/mapping
|
||||
/api/aggregate-edges/{aggregate_edge_id}/matrix
|
||||
/api/reports/{report_id}/raw-nodes
|
||||
/api/reports/{report_id}/raw-edges
|
||||
/api/reports/{report_id}/subgraph
|
||||
```
|
||||
|
||||
Raw pages default to 100 and cannot exceed 500. Subgraph depth cannot exceed five. Raw and expanded display graphs default to a combined 5,000-node/edge safety cap. The API returns no partial graph when it is exceeded and tells the user to select an aggregate view or increase the configured `expansion_cap`. These limits are configurable when embedding `create_app`.
|
||||
|
||||
## Performance behavior
|
||||
|
||||
- CSV readers stream rows and insert them in bounded batches.
|
||||
- Temporary SQLite staging and bulk joins resolve endpoints; ingestion performs no per-edge node query.
|
||||
- Secondary indexes are created after raw insertion.
|
||||
- Raw edges are never retained as a Python object graph or loaded into NetworkX.
|
||||
- Only aggregate operation nodes/edges enter NetworkX.
|
||||
- Mapping statistics are grouped in SQL; exact stencil comparison streams one operation pair.
|
||||
- API raw pages, display projections, expansions, and matrices have hard limits.
|
||||
- Stable IDs derive only from report IDs, grouping keys, and original CSV row numbers.
|
||||
|
||||
The normal suite contains a generated multi-lane smoke report. Set `RAPTOR_GRAPH_SLOW=1` to run the opt-in case with 100 operations, 1,000 lanes, and 100,000 raw edges:
|
||||
|
||||
```bash
|
||||
RAPTOR_GRAPH_SLOW=1 python3.13 -m pytest tools/raptor_graph_explorer/tests/test_aggregation.py -m slow
|
||||
```
|
||||
|
||||
## Current limitations
|
||||
|
||||
- The first browser load requires access to the pinned CDNs; offline vendoring is not included.
|
||||
- Aggregate graph endpoints return the complete selected aggregate level. Raw graphs remain bounded, but an unusually huge operation aggregate may eventually need aggregate pagination.
|
||||
- Motif representative paths operate at SCC-condensation granularity and choose a deterministic representative operation when an entry, exit, or branch SCC contains multiple operations.
|
||||
- The API is local and read-only; authentication, write APIs, and a graph database server are intentionally out of scope.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **No complete report pairs:** check that both files share the exact prefix and end in `.nodes.csv` / `.edges.csv`.
|
||||
- **Strict build failed:** run without `--strict`, then inspect the diagnostics panel or `diagnostics` table for bounded examples.
|
||||
- **Output already exists:** pass `--overwrite` only when replacement is intended.
|
||||
- **Unsupported schema:** rebuild the output with the installed tool version; old databases are not silently reinterpreted.
|
||||
- **Raw/expanded display returns 400:** use Operation, Core, or Node kind view, or embed the app with a larger `expansion_cap` when the local machine can safely handle it.
|
||||
- **Viewer is blank but APIs work:** verify browser access to the three pinned CDN URLs and inspect the browser console.
|
||||
- **No motifs:** this is a valid result; motifs require a structural single-entry/single-exit split and rejoin, not a diamond-like screen shape.
|
||||
|
||||
No Raptor build or compiler modification is required.
|
||||
@@ -0,0 +1,31 @@
|
||||
README.md
|
||||
pyproject.toml
|
||||
raptor_graph_explorer/__init__.py
|
||||
raptor_graph_explorer/__main__.py
|
||||
raptor_graph_explorer/aggregate.py
|
||||
raptor_graph_explorer/api.py
|
||||
raptor_graph_explorer/cli.py
|
||||
raptor_graph_explorer/database.py
|
||||
raptor_graph_explorer/discovery.py
|
||||
raptor_graph_explorer/ingest.py
|
||||
raptor_graph_explorer/mappings.py
|
||||
raptor_graph_explorer/motifs.py
|
||||
raptor_graph_explorer/projection.py
|
||||
raptor_graph_explorer/schema.py
|
||||
raptor_graph_explorer.egg-info/PKG-INFO
|
||||
raptor_graph_explorer.egg-info/SOURCES.txt
|
||||
raptor_graph_explorer.egg-info/dependency_links.txt
|
||||
raptor_graph_explorer.egg-info/entry_points.txt
|
||||
raptor_graph_explorer.egg-info/requires.txt
|
||||
raptor_graph_explorer.egg-info/top_level.txt
|
||||
raptor_graph_explorer/static/app.js
|
||||
raptor_graph_explorer/static/index.html
|
||||
raptor_graph_explorer/static/styles.css
|
||||
tests/test_aggregation.py
|
||||
tests/test_api.py
|
||||
tests/test_cli.py
|
||||
tests/test_discovery.py
|
||||
tests/test_ingest.py
|
||||
tests/test_mappings.py
|
||||
tests/test_motifs.py
|
||||
tests/test_projection.py
|
||||
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1,2 @@
|
||||
[console_scripts]
|
||||
raptor-graph-explorer = raptor_graph_explorer.cli:main
|
||||
@@ -0,0 +1,8 @@
|
||||
fastapi<0.116,>=0.115
|
||||
networkx<4,>=3.4
|
||||
uvicorn<0.35,>=0.34
|
||||
|
||||
[dev]
|
||||
anyio<4.10,>=4.6
|
||||
httpx<0.29,>=0.28
|
||||
pytest<9,>=8.3
|
||||
@@ -0,0 +1 @@
|
||||
raptor_graph_explorer
|
||||
@@ -0,0 +1,160 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from collections import defaultdict
|
||||
|
||||
from .mappings import classify_mapping_sql
|
||||
|
||||
|
||||
def aggregate_node_id(level: str, report_id: str, key: str | int) -> str:
|
||||
return f"{level}:{report_id}:{key}"
|
||||
|
||||
|
||||
def aggregate_edge_id(level: str, report_id: str, source: str, target: str) -> str:
|
||||
return f"{level}:{report_id}:{source}->{target}"
|
||||
|
||||
|
||||
def _json(value) -> str:
|
||||
return json.dumps(value, sort_keys=True, separators=(",", ":"))
|
||||
|
||||
|
||||
def _operation_nodes(connection: sqlite3.Connection, report_id: str) -> int:
|
||||
kinds: dict[int, dict[str, int]] = defaultdict(dict)
|
||||
for op_id, kind, count in connection.execute("""
|
||||
SELECT op_id,node_kind,COUNT(*) FROM raw_nodes WHERE report_id=?
|
||||
GROUP BY op_id,node_kind ORDER BY op_id,node_kind
|
||||
""", (report_id,)):
|
||||
kinds[op_id][kind] = count
|
||||
ssa: dict[int, list[str]] = defaultdict(list)
|
||||
for op_id, name in connection.execute("""
|
||||
SELECT DISTINCT op_id,ssa_name FROM raw_nodes
|
||||
WHERE report_id=? AND ssa_name<>'' ORDER BY op_id,ssa_name
|
||||
""", (report_id,)):
|
||||
if len(ssa[op_id]) < 10:
|
||||
ssa[op_id].append(name)
|
||||
rows = []
|
||||
for row in connection.execute("""
|
||||
SELECT op_id,COUNT(*),COUNT(DISTINCT lane),MIN(lane),MAX(lane),
|
||||
COUNT(DISTINCT core),MIN(core),MAX(core),COUNT(DISTINCT NULLIF(ssa_name,''))
|
||||
FROM raw_nodes WHERE report_id=? GROUP BY op_id ORDER BY op_id
|
||||
""", (report_id,)):
|
||||
op_id = row[0]
|
||||
rows.append((aggregate_node_id("operation", report_id, op_id), report_id, "operation", str(op_id),
|
||||
op_id, None, None, _json(kinds[op_id]), *row[1:], _json(ssa[op_id])))
|
||||
connection.executemany("""
|
||||
INSERT INTO aggregate_nodes(
|
||||
aggregate_id,report_id,level,group_key,op_id,core,node_kind,node_kind_counts,
|
||||
instance_count,lane_count,minimum_lane,maximum_lane,core_count,minimum_core,maximum_core,
|
||||
ssa_name_count,ssa_summary) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
|
||||
""", rows)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def _edge_histograms(connection: sqlite3.Connection, query: str, params: tuple) -> dict[tuple, dict[str, int]]:
|
||||
histograms: dict[tuple, dict[str, int]] = defaultdict(dict)
|
||||
for *key, tensor_type, count in connection.execute(query, params):
|
||||
histograms[tuple(key)][tensor_type] = count
|
||||
return histograms
|
||||
|
||||
|
||||
def _operation_edges(connection: sqlite3.Connection, report_id: str) -> int:
|
||||
histograms = _edge_histograms(connection, """
|
||||
SELECT source_op_id,target_op_id,tensor_type,COUNT(*) FROM raw_edges WHERE report_id=?
|
||||
GROUP BY source_op_id,target_op_id,tensor_type ORDER BY source_op_id,target_op_id,tensor_type
|
||||
""", (report_id,))
|
||||
rows = []
|
||||
for row in connection.execute("""
|
||||
SELECT source_op_id,target_op_id,COUNT(*),COALESCE(SUM(weight),0),MIN(weight),MAX(weight),
|
||||
COUNT(DISTINCT tensor_type),COUNT(DISTINCT channel_id)
|
||||
FROM raw_edges WHERE report_id=? GROUP BY source_op_id,target_op_id ORDER BY source_op_id,target_op_id
|
||||
""", (report_id,)):
|
||||
source, target = row[0], row[1]
|
||||
mapping_kind, metadata = classify_mapping_sql(connection, report_id, source, target)
|
||||
source_id = aggregate_node_id("operation", report_id, source)
|
||||
target_id = aggregate_node_id("operation", report_id, target)
|
||||
rows.append((aggregate_edge_id("operation", report_id, str(source), str(target)), report_id, "operation",
|
||||
source_id, target_id, source, target, *row[2:7], _json(histograms[(source, target)]), row[7], 1,
|
||||
mapping_kind, _json(metadata)))
|
||||
connection.executemany("""
|
||||
INSERT INTO aggregate_edges(
|
||||
aggregate_edge_id,report_id,level,source_aggregate_id,target_aggregate_id,source_op_id,target_op_id,
|
||||
raw_edge_count,total_weight,minimum_weight,maximum_weight,distinct_tensor_type_count,
|
||||
tensor_type_histogram,distinct_channel_count,operation_pair_count,mapping_kind,mapping_metadata)
|
||||
VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
|
||||
""", rows)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def _group_nodes(connection: sqlite3.Connection, report_id: str, level: str, expression: str,
|
||||
value_column: str) -> int:
|
||||
rows = []
|
||||
query = f"""
|
||||
SELECT {expression} group_key,COUNT(*),COUNT(DISTINCT lane),MIN(lane),MAX(lane),
|
||||
COUNT(DISTINCT core),MIN(core),MAX(core),COUNT(DISTINCT NULLIF(ssa_name,''))
|
||||
FROM raw_nodes WHERE report_id=? GROUP BY group_key ORDER BY group_key
|
||||
"""
|
||||
for row in connection.execute(query, (report_id,)):
|
||||
key = str(row[0])
|
||||
core = None if level != "core" or key == "unassigned" else int(key)
|
||||
kind = key if level == "node_kind" else None
|
||||
rows.append((aggregate_node_id(level, report_id, key), report_id, level, key, None, core, kind, "{}",
|
||||
*row[1:], "[]"))
|
||||
connection.executemany("""
|
||||
INSERT INTO aggregate_nodes(
|
||||
aggregate_id,report_id,level,group_key,op_id,core,node_kind,node_kind_counts,
|
||||
instance_count,lane_count,minimum_lane,maximum_lane,core_count,minimum_core,maximum_core,
|
||||
ssa_name_count,ssa_summary) VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
|
||||
""", rows)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def _group_edges(connection: sqlite3.Connection, report_id: str, level: str,
|
||||
source_expression: str, target_expression: str) -> int:
|
||||
joins = """
|
||||
JOIN raw_nodes s ON s.report_id=e.report_id AND s.node_id=e.source_node_id
|
||||
JOIN raw_nodes t ON t.report_id=e.report_id AND t.node_id=e.target_node_id
|
||||
"""
|
||||
histogram_query = f"""
|
||||
SELECT {source_expression} source_key,{target_expression} target_key,e.tensor_type,COUNT(*)
|
||||
FROM raw_edges e {joins} WHERE e.report_id=?
|
||||
GROUP BY source_key,target_key,e.tensor_type ORDER BY source_key,target_key,e.tensor_type
|
||||
"""
|
||||
histograms = _edge_histograms(connection, histogram_query, (report_id,))
|
||||
query = f"""
|
||||
SELECT {source_expression} source_key,{target_expression} target_key,COUNT(*),COALESCE(SUM(e.weight),0),
|
||||
MIN(e.weight),MAX(e.weight),COUNT(DISTINCT e.tensor_type),COUNT(DISTINCT e.channel_id),
|
||||
COUNT(DISTINCT printf('%d:%d',e.source_op_id,e.target_op_id))
|
||||
FROM raw_edges e {joins} WHERE e.report_id=?
|
||||
GROUP BY source_key,target_key ORDER BY source_key,target_key
|
||||
"""
|
||||
rows = []
|
||||
for row in connection.execute(query, (report_id,)):
|
||||
source, target = str(row[0]), str(row[1])
|
||||
source_id = aggregate_node_id(level, report_id, source)
|
||||
target_id = aggregate_node_id(level, report_id, target)
|
||||
rows.append((aggregate_edge_id(level, report_id, source, target), report_id, level, source_id, target_id,
|
||||
None, None, *row[2:7], _json(histograms[(row[0], row[1])]), row[7], row[8], None, "{}"))
|
||||
connection.executemany("""
|
||||
INSERT INTO aggregate_edges(
|
||||
aggregate_edge_id,report_id,level,source_aggregate_id,target_aggregate_id,source_op_id,target_op_id,
|
||||
raw_edge_count,total_weight,minimum_weight,maximum_weight,distinct_tensor_type_count,
|
||||
tensor_type_histogram,distinct_channel_count,operation_pair_count,mapping_kind,mapping_metadata)
|
||||
VALUES(?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)
|
||||
""", rows)
|
||||
return len(rows)
|
||||
|
||||
|
||||
def aggregate_report(connection: sqlite3.Connection, report_id: str) -> None:
|
||||
operation_nodes = _operation_nodes(connection, report_id)
|
||||
operation_edges = _operation_edges(connection, report_id)
|
||||
core_nodes = _group_nodes(connection, report_id, "core", "COALESCE(CAST(core AS TEXT),'unassigned')", "core")
|
||||
core_edges = _group_edges(connection, report_id, "core",
|
||||
"COALESCE(CAST(s.core AS TEXT),'unassigned')",
|
||||
"COALESCE(CAST(t.core AS TEXT),'unassigned')")
|
||||
kind_nodes = _group_nodes(connection, report_id, "node_kind", "node_kind", "node_kind")
|
||||
kind_edges = _group_edges(connection, report_id, "node_kind", "s.node_kind", "t.node_kind")
|
||||
connection.execute("""
|
||||
UPDATE reports SET operation_node_count=?,operation_edge_count=?,core_node_count=?,core_edge_count=?,
|
||||
node_kind_node_count=?,node_kind_edge_count=? WHERE report_id=?
|
||||
""", (operation_nodes, operation_edges, core_nodes, core_edges, kind_nodes, kind_edges, report_id))
|
||||
@@ -0,0 +1,239 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import Depends, FastAPI, HTTPException, Query
|
||||
from fastapi.staticfiles import StaticFiles
|
||||
|
||||
from .database import connect_readonly
|
||||
from .projection import display_graph
|
||||
|
||||
JSON_COLUMNS = {"attributes_json", "node_kind_counts", "ssa_summary", "tensor_type_histogram", "mapping_metadata",
|
||||
"internal_aggregate_ids", "branch_successor_aggregate_ids", "representative_paths",
|
||||
"examples_json"}
|
||||
|
||||
|
||||
def _dict(row: sqlite3.Row) -> dict:
|
||||
result = dict(row)
|
||||
for key in JSON_COLUMNS & result.keys():
|
||||
result[key] = json.loads(result[key])
|
||||
return result
|
||||
|
||||
|
||||
def _page(limit: int, offset: int, maximum: int) -> None:
|
||||
if limit < 1 or limit > maximum:
|
||||
raise HTTPException(400, f"limit must be between 1 and {maximum}")
|
||||
if offset < 0:
|
||||
raise HTTPException(400, "offset must be non-negative")
|
||||
|
||||
|
||||
def create_app(output_directory: str | Path, *, max_page_size: int = 500,
|
||||
expansion_cap: int = 5_000, matrix_bins: int = 64,
|
||||
exact_matrix_points: int = 2_000) -> FastAPI:
|
||||
output = Path(output_directory)
|
||||
database = output / "graph.sqlite"
|
||||
manifest_path = output / "manifest.json"
|
||||
if not database.is_file() or not manifest_path.is_file():
|
||||
raise ValueError(f"not a preprocessed graph output: {output}")
|
||||
# Validate eagerly so startup errors are actionable.
|
||||
connect_readonly(database).close()
|
||||
manifest = json.loads(manifest_path.read_text())
|
||||
app = FastAPI(title="Raptor Graph Explorer", version=manifest.get("tool_version", "unknown"))
|
||||
|
||||
def connection():
|
||||
db = connect_readonly(database)
|
||||
try:
|
||||
yield db
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
@app.get("/api/manifest")
|
||||
def get_manifest():
|
||||
return manifest
|
||||
|
||||
@app.get("/api/reports")
|
||||
def reports(db=Depends(connection)):
|
||||
return [_dict(row) for row in db.execute("SELECT * FROM reports ORDER BY report_id")]
|
||||
|
||||
@app.get("/api/reports/{report_id}/summary")
|
||||
def report_summary(report_id: str, db=Depends(connection)):
|
||||
row = db.execute("SELECT * FROM reports WHERE report_id=?", (report_id,)).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(404, "report not found")
|
||||
diagnostics = [_dict(item) for item in db.execute(
|
||||
"SELECT * FROM diagnostics WHERE report_id=? OR report_id IS NULL ORDER BY diagnostic_id", (report_id,))]
|
||||
mapping_classes = {item[0]: item[1] for item in db.execute("""
|
||||
SELECT mapping_kind,COUNT(*) FROM aggregate_edges
|
||||
WHERE report_id=? AND level='operation' GROUP BY mapping_kind ORDER BY mapping_kind
|
||||
""", (report_id,))}
|
||||
return {"report": _dict(row), "mapping_classes": mapping_classes, "diagnostics": diagnostics}
|
||||
|
||||
@app.get("/api/reports/{report_id}/graph/{level}")
|
||||
def graph(report_id: str, level: str, db=Depends(connection)):
|
||||
level = level.replace("-", "_")
|
||||
if level not in {"operation", "core", "node_kind"}:
|
||||
raise HTTPException(404, "unknown aggregation level")
|
||||
nodes = [_dict(row) for row in db.execute(
|
||||
"SELECT * FROM aggregate_nodes WHERE report_id=? AND level=? ORDER BY aggregate_id", (report_id, level))]
|
||||
if not nodes and not db.execute("SELECT 1 FROM reports WHERE report_id=?", (report_id,)).fetchone():
|
||||
raise HTTPException(404, "report not found")
|
||||
edges = [_dict(row) for row in db.execute(
|
||||
"SELECT * FROM aggregate_edges WHERE report_id=? AND level=? ORDER BY aggregate_edge_id",
|
||||
(report_id, level))]
|
||||
return {"level": level, "nodes": nodes, "edges": edges}
|
||||
|
||||
@app.get("/api/reports/{report_id}/display-graph")
|
||||
def projected_graph(report_id: str, view: str = Query("operation", pattern="^(operation|raw)$"),
|
||||
expanded_op_id: list[int] = Query(default=[]), expand_all: bool = False,
|
||||
db=Depends(connection)):
|
||||
try:
|
||||
return display_graph(db, report_id, view=view, expanded_operation_ids=expanded_op_id,
|
||||
expand_all=expand_all, cap=expansion_cap)
|
||||
except LookupError as exc:
|
||||
raise HTTPException(404, str(exc)) from exc
|
||||
except ValueError as exc:
|
||||
raise HTTPException(400, str(exc)) from exc
|
||||
|
||||
@app.get("/api/reports/{report_id}/motifs")
|
||||
def motifs(report_id: str, db=Depends(connection)):
|
||||
return [_dict(row) for row in db.execute("SELECT * FROM motifs WHERE report_id=? ORDER BY motif_id",
|
||||
(report_id,))]
|
||||
|
||||
@app.get("/api/aggregate-nodes/{aggregate_id:path}")
|
||||
def aggregate_node(aggregate_id: str, db=Depends(connection)):
|
||||
row = db.execute("SELECT * FROM aggregate_nodes WHERE aggregate_id=?", (aggregate_id,)).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(404, "aggregate node not found")
|
||||
result = _dict(row)
|
||||
if result["level"] == "operation":
|
||||
result["ssa_names"] = [item[0] for item in db.execute("""
|
||||
SELECT DISTINCT ssa_name FROM raw_nodes WHERE report_id=? AND op_id=? AND ssa_name<>''
|
||||
ORDER BY ssa_name LIMIT 100
|
||||
""", (result["report_id"], result["op_id"]))]
|
||||
return result
|
||||
|
||||
@app.get("/api/aggregate-edges/{aggregate_edge_id:path}/mapping")
|
||||
def mapping(aggregate_edge_id: str, limit: int = 100, offset: int = 0, db=Depends(connection)):
|
||||
_page(limit, offset, max_page_size)
|
||||
edge = db.execute("SELECT * FROM aggregate_edges WHERE aggregate_edge_id=? AND level='operation'",
|
||||
(aggregate_edge_id,)).fetchone()
|
||||
if not edge:
|
||||
raise HTTPException(404, "operation aggregate edge not found")
|
||||
total = db.execute("""
|
||||
SELECT COUNT(*) FROM raw_edges WHERE report_id=? AND source_op_id=? AND target_op_id=?
|
||||
""", (edge["report_id"], edge["source_op_id"], edge["target_op_id"])).fetchone()[0]
|
||||
rows = [_dict(row) for row in db.execute("""
|
||||
SELECT edge_id,source_node_id,target_node_id,source_lane,target_lane,weight,tensor_type,channel_id
|
||||
FROM raw_edges WHERE report_id=? AND source_op_id=? AND target_op_id=?
|
||||
ORDER BY edge_id LIMIT ? OFFSET ?
|
||||
""", (edge["report_id"], edge["source_op_id"], edge["target_op_id"], limit, offset))]
|
||||
return {"total": total, "limit": limit, "offset": offset, "items": rows}
|
||||
|
||||
@app.get("/api/aggregate-edges/{aggregate_edge_id:path}/matrix")
|
||||
def matrix(aggregate_edge_id: str, db=Depends(connection)):
|
||||
edge = db.execute("SELECT * FROM aggregate_edges WHERE aggregate_edge_id=? AND level='operation'",
|
||||
(aggregate_edge_id,)).fetchone()
|
||||
if not edge:
|
||||
raise HTTPException(404, "operation aggregate edge not found")
|
||||
params = (edge["report_id"], edge["source_op_id"], edge["target_op_id"])
|
||||
extent = db.execute("""
|
||||
SELECT MIN(source_lane),MAX(source_lane),MIN(target_lane),MAX(target_lane),
|
||||
COUNT(DISTINCT printf('%d:%d',source_lane,target_lane)),
|
||||
SUM(source_lane IS NULL OR target_lane IS NULL)
|
||||
FROM raw_edges WHERE report_id=? AND source_op_id=? AND target_op_id=?
|
||||
""", params).fetchone()
|
||||
result = {"source_extent": [extent[0], extent[1]], "target_extent": [extent[2], extent[3]],
|
||||
"null_lane_count": extent[5] or 0}
|
||||
if extent[4] <= exact_matrix_points:
|
||||
result.update(mode="exact", points=[{"source": row[0], "target": row[1], "count": row[2]}
|
||||
for row in db.execute("""
|
||||
SELECT source_lane,target_lane,COUNT(*) FROM raw_edges
|
||||
WHERE report_id=? AND source_op_id=? AND target_op_id=?
|
||||
AND source_lane IS NOT NULL AND target_lane IS NOT NULL
|
||||
GROUP BY source_lane,target_lane ORDER BY source_lane,target_lane
|
||||
""", params)])
|
||||
return result
|
||||
source_span = max(1, extent[1] - extent[0] + 1)
|
||||
target_span = max(1, extent[3] - extent[2] + 1)
|
||||
bins = [list(row) for row in db.execute("""
|
||||
SELECT MIN(?, CAST((source_lane-?)*?/? AS INTEGER)),
|
||||
MIN(?, CAST((target_lane-?)*?/? AS INTEGER)),COUNT(*)
|
||||
FROM raw_edges WHERE report_id=? AND source_op_id=? AND target_op_id=?
|
||||
AND source_lane IS NOT NULL AND target_lane IS NOT NULL
|
||||
GROUP BY 1,2 ORDER BY 1,2
|
||||
""", (matrix_bins - 1, extent[0], matrix_bins, source_span,
|
||||
matrix_bins - 1, extent[2], matrix_bins, target_span, *params))]
|
||||
result.update(mode="binned", dimensions=[matrix_bins, matrix_bins], bins=bins)
|
||||
return result
|
||||
|
||||
@app.get("/api/aggregate-edges/{aggregate_edge_id:path}")
|
||||
def aggregate_edge(aggregate_edge_id: str, db=Depends(connection)):
|
||||
row = db.execute("SELECT * FROM aggregate_edges WHERE aggregate_edge_id=?", (aggregate_edge_id,)).fetchone()
|
||||
if not row:
|
||||
raise HTTPException(404, "aggregate edge not found")
|
||||
return _dict(row)
|
||||
|
||||
def raw_page(table: str, report_id: str, limit: int, offset: int, db: sqlite3.Connection):
|
||||
_page(limit, offset, max_page_size)
|
||||
total = db.execute(f"SELECT COUNT(*) FROM {table} WHERE report_id=?", (report_id,)).fetchone()[0]
|
||||
items = [_dict(row) for row in db.execute(
|
||||
f"SELECT * FROM {table} WHERE report_id=? ORDER BY rowid LIMIT ? OFFSET ?",
|
||||
(report_id, limit, offset))]
|
||||
return {"total": total, "limit": limit, "offset": offset, "items": items}
|
||||
|
||||
@app.get("/api/reports/{report_id}/raw-nodes")
|
||||
def raw_nodes(report_id: str, limit: int = 100, offset: int = 0, db=Depends(connection)):
|
||||
return raw_page("raw_nodes", report_id, limit, offset, db)
|
||||
|
||||
@app.get("/api/reports/{report_id}/raw-edges")
|
||||
def raw_edges(report_id: str, limit: int = 100, offset: int = 0, db=Depends(connection)):
|
||||
return raw_page("raw_edges", report_id, limit, offset, db)
|
||||
|
||||
@app.get("/api/reports/{report_id}/subgraph")
|
||||
def subgraph(report_id: str, aggregate_id: str, direction: str = Query("both", pattern="^(in|out|both)$"),
|
||||
depth: int = Query(1, ge=0, le=5), expand_lanes: bool = False, db=Depends(connection)):
|
||||
selected = db.execute("""
|
||||
SELECT op_id FROM aggregate_nodes WHERE report_id=? AND level='operation' AND aggregate_id=?
|
||||
""", (report_id, aggregate_id)).fetchone()
|
||||
if not selected:
|
||||
raise HTTPException(404, "selected operation aggregate not found")
|
||||
aggregate_edges = list(db.execute("""
|
||||
SELECT * FROM aggregate_edges WHERE report_id=? AND level='operation' ORDER BY aggregate_edge_id
|
||||
""", (report_id,)))
|
||||
visited, frontier = {aggregate_id}, {aggregate_id}
|
||||
for _ in range(depth):
|
||||
following = set()
|
||||
for edge in aggregate_edges:
|
||||
if direction in {"out", "both"} and edge["source_aggregate_id"] in frontier:
|
||||
following.add(edge["target_aggregate_id"])
|
||||
if direction in {"in", "both"} and edge["target_aggregate_id"] in frontier:
|
||||
following.add(edge["source_aggregate_id"])
|
||||
frontier = following - visited
|
||||
visited |= frontier
|
||||
placeholders = ",".join("?" for _ in visited)
|
||||
nodes = [_dict(row) for row in db.execute(
|
||||
f"SELECT * FROM aggregate_nodes WHERE aggregate_id IN ({placeholders}) ORDER BY aggregate_id", tuple(visited))]
|
||||
edges = [_dict(edge) for edge in aggregate_edges
|
||||
if edge["source_aggregate_id"] in visited and edge["target_aggregate_id"] in visited]
|
||||
result = {"nodes": nodes, "edges": edges, "expanded_raw_nodes": [], "expanded_raw_edges": []}
|
||||
if expand_lanes:
|
||||
op_id = selected[0]
|
||||
node_count = db.execute("SELECT COUNT(*) FROM raw_nodes WHERE report_id=? AND op_id=?",
|
||||
(report_id, op_id)).fetchone()[0]
|
||||
edge_count = db.execute("""
|
||||
SELECT COUNT(*) FROM raw_edges WHERE report_id=? AND (source_op_id=? OR target_op_id=?)
|
||||
""", (report_id, op_id, op_id)).fetchone()[0]
|
||||
if node_count + edge_count > expansion_cap:
|
||||
raise HTTPException(400, f"lane expansion has {node_count} nodes and {edge_count} edges; safety cap is {expansion_cap}")
|
||||
result["expanded_raw_nodes"] = [_dict(row) for row in db.execute(
|
||||
"SELECT * FROM raw_nodes WHERE report_id=? AND op_id=? ORDER BY node_id", (report_id, op_id))]
|
||||
result["expanded_raw_edges"] = [_dict(row) for row in db.execute("""
|
||||
SELECT * FROM raw_edges WHERE report_id=? AND (source_op_id=? OR target_op_id=?) ORDER BY edge_id
|
||||
""", (report_id, op_id, op_id))]
|
||||
return result
|
||||
|
||||
static = Path(__file__).with_name("static")
|
||||
app.mount("/", StaticFiles(directory=static, html=True), name="static")
|
||||
return app
|
||||
@@ -0,0 +1,141 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import shutil
|
||||
import sqlite3
|
||||
import tempfile
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from . import __version__
|
||||
from .aggregate import aggregate_report
|
||||
from .database import connect_readonly, create_database, create_indexes, store_diagnostics
|
||||
from .discovery import discover_reports
|
||||
from .ingest import ingest_report
|
||||
from .motifs import analyze_report
|
||||
from .schema import BuildError, DiagnosticCollector, SCHEMA_VERSION
|
||||
|
||||
|
||||
def _manifest(connection: sqlite3.Connection, diagnostics: DiagnosticCollector) -> dict:
|
||||
columns = ("report_id", "stage", "raw_node_count", "raw_edge_count", "operation_node_count",
|
||||
"operation_edge_count", "core_node_count", "core_edge_count", "scc_count", "motif_count")
|
||||
reports = [dict(zip(columns, row)) for row in connection.execute(
|
||||
f"SELECT {','.join(columns)} FROM reports ORDER BY report_id")]
|
||||
return {
|
||||
"schema_version": SCHEMA_VERSION,
|
||||
"tool_version": __version__,
|
||||
"created_at": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"),
|
||||
"reports": reports,
|
||||
"diagnostic_count": sum(item.count for item in diagnostics.items),
|
||||
}
|
||||
|
||||
|
||||
def build_output(inputs: list[str | Path], output_directory: str | Path, *, strict: bool = False,
|
||||
overwrite: bool = False) -> dict:
|
||||
output = Path(output_directory)
|
||||
if output.exists() and any(output.iterdir() if output.is_dir() else [output]):
|
||||
if not overwrite:
|
||||
raise BuildError(f"output exists and is not empty: {output}; use --overwrite")
|
||||
if output.is_dir():
|
||||
shutil.rmtree(output)
|
||||
else:
|
||||
output.unlink()
|
||||
output.mkdir(parents=True, exist_ok=True)
|
||||
diagnostics = DiagnosticCollector()
|
||||
with tempfile.TemporaryDirectory(prefix="raptor-graph-explorer-") as temporary:
|
||||
pairs, _ = discover_reports(inputs, Path(temporary), diagnostics)
|
||||
if not pairs:
|
||||
raise BuildError("no complete report pairs discovered")
|
||||
connection = create_database(output / "graph.sqlite")
|
||||
try:
|
||||
with connection:
|
||||
for pair in pairs:
|
||||
ingest_report(connection, pair, diagnostics)
|
||||
create_indexes(connection)
|
||||
for pair in pairs:
|
||||
aggregate_report(connection, pair.report_id)
|
||||
analyze_report(connection, pair.report_id)
|
||||
store_diagnostics(connection, diagnostics.items)
|
||||
manifest = _manifest(connection, diagnostics)
|
||||
(output / "manifest.json").write_text(json.dumps(manifest, indent=2, sort_keys=True) + "\n")
|
||||
connection.execute("PRAGMA optimize")
|
||||
connection.execute("PRAGMA wal_checkpoint(TRUNCATE)")
|
||||
connection.execute("PRAGMA journal_mode=DELETE")
|
||||
finally:
|
||||
connection.close()
|
||||
if strict and diagnostics.error_count:
|
||||
raise BuildError(f"strict build rejected {diagnostics.error_count} data-integrity error(s); see diagnostics")
|
||||
return manifest
|
||||
|
||||
|
||||
def inspect_output(output_directory: str | Path) -> str:
|
||||
output = Path(output_directory)
|
||||
manifest = json.loads((output / "manifest.json").read_text())
|
||||
connection = connect_readonly(output / "graph.sqlite")
|
||||
lines = [f"Raptor Graph Explorer schema {manifest['schema_version']}, tool {manifest['tool_version']}"]
|
||||
try:
|
||||
for report in manifest["reports"]:
|
||||
report_id = report["report_id"]
|
||||
mappings = ", ".join(f"{kind}={count}" for kind, count in connection.execute("""
|
||||
SELECT mapping_kind,COUNT(*) FROM aggregate_edges
|
||||
WHERE report_id=? AND level='operation' GROUP BY mapping_kind ORDER BY mapping_kind
|
||||
""", (report_id,))) or "none"
|
||||
motifs = ", ".join(f"{kind}={count}" for kind, count in connection.execute(
|
||||
"SELECT motif_kind,COUNT(*) FROM motifs WHERE report_id=? GROUP BY motif_kind ORDER BY motif_kind",
|
||||
(report_id,))) or "none"
|
||||
lines.extend([
|
||||
f"{report_id} ({report['stage']}): raw {report['raw_node_count']} nodes / {report['raw_edge_count']} edges",
|
||||
f" operation {report['operation_node_count']} / {report['operation_edge_count']}; core {report['core_node_count']} / {report['core_edge_count']}; SCCs {report['scc_count']}",
|
||||
f" mappings: {mappings}", f" motifs: {motifs}",
|
||||
])
|
||||
lines.append(f"diagnostics: {manifest['diagnostic_count']}")
|
||||
finally:
|
||||
connection.close()
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def parser() -> argparse.ArgumentParser:
|
||||
root = argparse.ArgumentParser(prog="raptor-graph-explorer")
|
||||
commands = root.add_subparsers(dest="command", required=True)
|
||||
build = commands.add_parser("build", help="preprocess CSV reports")
|
||||
build.add_argument("inputs", nargs="+")
|
||||
build.add_argument("--output", required=True)
|
||||
build.add_argument("--strict", action="store_true")
|
||||
build.add_argument("--overwrite", action="store_true")
|
||||
serve = commands.add_parser("serve", help="serve an existing output")
|
||||
serve.add_argument("output")
|
||||
serve.add_argument("--host", default="127.0.0.1")
|
||||
serve.add_argument("--port", type=int, default=8765)
|
||||
run = commands.add_parser("run", help="build and serve")
|
||||
run.add_argument("inputs", nargs="+")
|
||||
run.add_argument("--output", required=True)
|
||||
run.add_argument("--host", default="127.0.0.1")
|
||||
run.add_argument("--port", type=int, default=8765)
|
||||
run.add_argument("--overwrite", action="store_true")
|
||||
inspect = commands.add_parser("inspect", help="print a preprocessed output summary")
|
||||
inspect.add_argument("output")
|
||||
return root
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
args = parser().parse_args(argv)
|
||||
try:
|
||||
if args.command == "build":
|
||||
manifest = build_output(args.inputs, args.output, strict=args.strict, overwrite=args.overwrite)
|
||||
print(f"built {len(manifest['reports'])} report(s) in {args.output}")
|
||||
elif args.command == "inspect":
|
||||
print(inspect_output(args.output))
|
||||
else:
|
||||
if args.command == "run":
|
||||
build_output(args.inputs, args.output, overwrite=args.overwrite)
|
||||
output = args.output
|
||||
else:
|
||||
output = args.output
|
||||
from uvicorn import run as uvicorn_run
|
||||
from .api import create_app
|
||||
uvicorn_run(create_app(output), host=args.host, port=args.port)
|
||||
except (BuildError, OSError, ValueError, json.JSONDecodeError, sqlite3.Error) as exc:
|
||||
print(f"error: {exc}")
|
||||
return 2
|
||||
return 0
|
||||
@@ -0,0 +1,110 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
from .schema import Diagnostic, SCHEMA_VERSION
|
||||
|
||||
DDL = """
|
||||
PRAGMA foreign_keys=ON;
|
||||
CREATE TABLE metadata(key TEXT PRIMARY KEY, value TEXT NOT NULL);
|
||||
CREATE TABLE reports(
|
||||
report_id TEXT PRIMARY KEY, stage TEXT NOT NULL, raw_node_count INTEGER NOT NULL DEFAULT 0,
|
||||
raw_edge_count INTEGER NOT NULL DEFAULT 0, operation_node_count INTEGER NOT NULL DEFAULT 0,
|
||||
operation_edge_count INTEGER NOT NULL DEFAULT 0, core_node_count INTEGER NOT NULL DEFAULT 0,
|
||||
core_edge_count INTEGER NOT NULL DEFAULT 0, node_kind_node_count INTEGER NOT NULL DEFAULT 0,
|
||||
node_kind_edge_count INTEGER NOT NULL DEFAULT 0, scc_count INTEGER NOT NULL DEFAULT 0,
|
||||
motif_count INTEGER NOT NULL DEFAULT 0
|
||||
);
|
||||
CREATE TABLE raw_nodes(
|
||||
report_id TEXT NOT NULL, stage TEXT NOT NULL, node_id TEXT NOT NULL, node_kind TEXT NOT NULL,
|
||||
op_id INTEGER NOT NULL, lane INTEGER, core INTEGER, ssa_name TEXT NOT NULL, attributes_json TEXT NOT NULL,
|
||||
PRIMARY KEY(report_id,node_id), FOREIGN KEY(report_id) REFERENCES reports(report_id)
|
||||
);
|
||||
CREATE TABLE raw_edges(
|
||||
edge_id TEXT PRIMARY KEY, report_id TEXT NOT NULL, stage TEXT NOT NULL,
|
||||
source_node_id TEXT NOT NULL, target_node_id TEXT NOT NULL, source_op_id INTEGER NOT NULL,
|
||||
target_op_id INTEGER NOT NULL, source_lane INTEGER, target_lane INTEGER, weight INTEGER,
|
||||
tensor_type TEXT NOT NULL, channel_id INTEGER, attributes_json TEXT NOT NULL,
|
||||
FOREIGN KEY(report_id) REFERENCES reports(report_id)
|
||||
);
|
||||
CREATE TABLE aggregate_nodes(
|
||||
aggregate_id TEXT PRIMARY KEY, report_id TEXT NOT NULL, level TEXT NOT NULL, group_key TEXT NOT NULL,
|
||||
op_id INTEGER, core INTEGER, node_kind TEXT, node_kind_counts TEXT NOT NULL DEFAULT '{}',
|
||||
instance_count INTEGER NOT NULL, lane_count INTEGER NOT NULL DEFAULT 0, minimum_lane INTEGER,
|
||||
maximum_lane INTEGER, core_count INTEGER NOT NULL DEFAULT 0, minimum_core INTEGER, maximum_core INTEGER,
|
||||
ssa_name_count INTEGER NOT NULL DEFAULT 0, ssa_summary TEXT NOT NULL DEFAULT '[]',
|
||||
UNIQUE(report_id,level,group_key)
|
||||
);
|
||||
CREATE TABLE aggregate_edges(
|
||||
aggregate_edge_id TEXT PRIMARY KEY, report_id TEXT NOT NULL, level TEXT NOT NULL,
|
||||
source_aggregate_id TEXT NOT NULL, target_aggregate_id TEXT NOT NULL,
|
||||
source_op_id INTEGER, target_op_id INTEGER, raw_edge_count INTEGER NOT NULL,
|
||||
total_weight INTEGER NOT NULL, minimum_weight INTEGER, maximum_weight INTEGER,
|
||||
distinct_tensor_type_count INTEGER NOT NULL, tensor_type_histogram TEXT NOT NULL,
|
||||
distinct_channel_count INTEGER NOT NULL, operation_pair_count INTEGER NOT NULL DEFAULT 0,
|
||||
mapping_kind TEXT, mapping_metadata TEXT NOT NULL DEFAULT '{}',
|
||||
UNIQUE(report_id,level,source_aggregate_id,target_aggregate_id)
|
||||
);
|
||||
CREATE TABLE motifs(
|
||||
motif_id TEXT PRIMARY KEY, report_id TEXT NOT NULL, entry_aggregate_id TEXT NOT NULL,
|
||||
exit_aggregate_id TEXT NOT NULL, internal_aggregate_ids TEXT NOT NULL,
|
||||
branch_successor_aggregate_ids TEXT NOT NULL, representative_paths TEXT NOT NULL,
|
||||
node_count INTEGER NOT NULL, edge_count INTEGER NOT NULL, contains_cycle INTEGER NOT NULL,
|
||||
motif_kind TEXT NOT NULL, confidence REAL NOT NULL
|
||||
);
|
||||
CREATE TABLE motif_membership(
|
||||
motif_id TEXT NOT NULL, aggregate_id TEXT NOT NULL, role TEXT NOT NULL,
|
||||
PRIMARY KEY(motif_id,aggregate_id), FOREIGN KEY(motif_id) REFERENCES motifs(motif_id)
|
||||
);
|
||||
CREATE TABLE diagnostics(
|
||||
diagnostic_id INTEGER PRIMARY KEY, report_id TEXT, severity TEXT NOT NULL, code TEXT NOT NULL,
|
||||
message TEXT NOT NULL, occurrence_count INTEGER NOT NULL, examples_json TEXT NOT NULL
|
||||
);
|
||||
"""
|
||||
|
||||
INDEX_DDL = """
|
||||
CREATE INDEX raw_nodes_report_op ON raw_nodes(report_id,op_id);
|
||||
CREATE INDEX raw_nodes_report_lane ON raw_nodes(report_id,op_id,lane);
|
||||
CREATE INDEX raw_nodes_report_core ON raw_nodes(report_id,core);
|
||||
CREATE INDEX raw_edges_source_node ON raw_edges(report_id,source_node_id);
|
||||
CREATE INDEX raw_edges_target_node ON raw_edges(report_id,target_node_id);
|
||||
CREATE INDEX raw_edges_source_op ON raw_edges(report_id,source_op_id,target_op_id);
|
||||
CREATE INDEX raw_edges_target_op ON raw_edges(report_id,target_op_id,source_op_id);
|
||||
CREATE INDEX raw_edges_mapping ON raw_edges(report_id,source_op_id,target_op_id,source_lane,target_lane);
|
||||
CREATE INDEX raw_edges_channel ON raw_edges(report_id,channel_id);
|
||||
CREATE INDEX aggregate_nodes_lookup ON aggregate_nodes(report_id,level,op_id,core);
|
||||
CREATE INDEX aggregate_edges_lookup ON aggregate_edges(report_id,level);
|
||||
CREATE INDEX motifs_report ON motifs(report_id);
|
||||
"""
|
||||
|
||||
|
||||
def create_database(path: Path) -> sqlite3.Connection:
|
||||
connection = sqlite3.connect(path)
|
||||
connection.row_factory = sqlite3.Row
|
||||
connection.executescript("PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL;" + DDL)
|
||||
connection.execute("INSERT INTO metadata VALUES('schema_version',?)", (str(SCHEMA_VERSION),))
|
||||
return connection
|
||||
|
||||
|
||||
def create_indexes(connection: sqlite3.Connection) -> None:
|
||||
connection.executescript(INDEX_DDL)
|
||||
|
||||
|
||||
def store_diagnostics(connection: sqlite3.Connection, diagnostics: list[Diagnostic]) -> None:
|
||||
connection.executemany(
|
||||
"INSERT INTO diagnostics(report_id,severity,code,message,occurrence_count,examples_json) VALUES(?,?,?,?,?,?)",
|
||||
[(d.report_id, d.severity, d.code, d.message, d.count, json.dumps(d.examples, separators=(",", ":")))
|
||||
for d in diagnostics],
|
||||
)
|
||||
|
||||
|
||||
def connect_readonly(path: Path) -> sqlite3.Connection:
|
||||
connection = sqlite3.connect(f"file:{path}?mode=ro", uri=True, check_same_thread=False)
|
||||
connection.row_factory = sqlite3.Row
|
||||
row = connection.execute("SELECT value FROM metadata WHERE key='schema_version'").fetchone()
|
||||
if not row or int(row[0]) != SCHEMA_VERSION:
|
||||
connection.close()
|
||||
raise ValueError(f"unsupported database schema version: {row[0] if row else 'missing'}")
|
||||
return connection
|
||||
@@ -0,0 +1,72 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
from .schema import DiagnosticCollector, ReportPair
|
||||
|
||||
|
||||
def _safe_extract(archive: Path, destination: Path) -> None:
|
||||
root = destination.resolve()
|
||||
with zipfile.ZipFile(archive) as zipped:
|
||||
for member in zipped.infolist():
|
||||
target = (destination / member.filename).resolve()
|
||||
if target != root and root not in target.parents:
|
||||
raise ValueError(f"ZIP entry escapes extraction directory: {member.filename}")
|
||||
if member.is_dir():
|
||||
target.mkdir(parents=True, exist_ok=True)
|
||||
continue
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
with zipped.open(member) as source, target.open("wb") as output:
|
||||
shutil.copyfileobj(source, output)
|
||||
|
||||
|
||||
def _collect_path(path: Path, extraction_root: Path, files: set[Path], diagnostics: DiagnosticCollector) -> None:
|
||||
if not path.exists():
|
||||
diagnostics.add("input_not_found", "input path does not exist", example=str(path))
|
||||
elif path.is_dir():
|
||||
files.update(p for p in path.rglob("*.csv") if p.is_file())
|
||||
elif path.suffix.lower() == ".zip":
|
||||
target = extraction_root / f"archive-{len(list(extraction_root.iterdir())):04d}"
|
||||
target.mkdir(parents=True)
|
||||
try:
|
||||
_safe_extract(path, target)
|
||||
except (OSError, zipfile.BadZipFile, ValueError) as exc:
|
||||
diagnostics.add("invalid_zip", str(exc), example=str(path))
|
||||
else:
|
||||
files.update(p for p in target.rglob("*.csv") if p.is_file())
|
||||
elif path.name.endswith((".nodes.csv", ".edges.csv")):
|
||||
files.add(path)
|
||||
else:
|
||||
diagnostics.add("unsupported_input", "input is not a directory, ZIP, or report CSV", example=str(path))
|
||||
|
||||
|
||||
def discover_reports(inputs: list[str | Path], extraction_root: Path,
|
||||
diagnostics: DiagnosticCollector | None = None) -> tuple[list[ReportPair], DiagnosticCollector]:
|
||||
diagnostics = diagnostics or DiagnosticCollector()
|
||||
extraction_root.mkdir(parents=True, exist_ok=True)
|
||||
files: set[Path] = set()
|
||||
for raw in inputs:
|
||||
_collect_path(Path(raw), extraction_root, files, diagnostics)
|
||||
|
||||
grouped: dict[tuple[Path, str], dict[str, Path]] = {}
|
||||
for path in sorted(files, key=lambda p: str(p)):
|
||||
suffix = ".nodes.csv" if path.name.endswith(".nodes.csv") else ".edges.csv" if path.name.endswith(".edges.csv") else None
|
||||
if suffix:
|
||||
grouped.setdefault((path.parent, path.name[:-len(suffix)]), {})[suffix] = path
|
||||
|
||||
pairs: list[ReportPair] = []
|
||||
seen_ids: set[str] = set()
|
||||
for (_, report_id), parts in sorted(grouped.items(), key=lambda item: (item[0][1], str(item[0][0]))):
|
||||
missing = [name for name in (".nodes.csv", ".edges.csv") if name not in parts]
|
||||
if missing:
|
||||
diagnostics.add("incomplete_pair", f"missing {', '.join(missing)}", report_id)
|
||||
continue
|
||||
if report_id in seen_ids:
|
||||
diagnostics.add("duplicate_report_id", "report_id occurs in more than one input directory", report_id,
|
||||
str(parts[".nodes.csv"]))
|
||||
continue
|
||||
seen_ids.add(report_id)
|
||||
pairs.append(ReportPair(report_id, parts[".nodes.csv"], parts[".edges.csv"]))
|
||||
return pairs, diagnostics
|
||||
@@ -0,0 +1,220 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import json
|
||||
import re
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
from .schema import DiagnosticCollector, EDGE_COLUMNS, NODE_COLUMNS, NODE_KINDS, ReportPair
|
||||
|
||||
BATCH_SIZE = 2_000
|
||||
|
||||
|
||||
def infer_stage(report_id: str) -> str:
|
||||
match = re.match(r"^(spatial\d+)(?:_|$)", report_id)
|
||||
return match.group(1) if match else report_id
|
||||
|
||||
|
||||
def _integer(value: str | None, field: str) -> int | None:
|
||||
if value is None or not value.strip():
|
||||
return None
|
||||
try:
|
||||
return int(value)
|
||||
except ValueError as exc:
|
||||
raise ValueError(f"malformed integer in {field}: {value!r}") from exc
|
||||
|
||||
|
||||
def _id_parts(node_id: str) -> tuple[str, int | None, int | None]:
|
||||
parts = node_id.split(":")
|
||||
prefix = parts[0]
|
||||
try:
|
||||
op_id = int(parts[1]) if len(parts) > 1 else None
|
||||
lane = int(parts[2]) if len(parts) > 2 else None
|
||||
except ValueError:
|
||||
op_id = lane = None
|
||||
return prefix, op_id, lane
|
||||
|
||||
|
||||
def _extra(row: dict[str, str | None], known: set[str]) -> str:
|
||||
return json.dumps({key: value for key, value in row.items() if key not in known},
|
||||
sort_keys=True, separators=(",", ":"))
|
||||
|
||||
|
||||
def _batches(rows, size: int = BATCH_SIZE):
|
||||
batch = []
|
||||
for row in rows:
|
||||
batch.append(row)
|
||||
if len(batch) == size:
|
||||
yield batch
|
||||
batch = []
|
||||
if batch:
|
||||
yield batch
|
||||
|
||||
|
||||
def _check_columns(reader: csv.DictReader, required: set[str], kind: str, report_id: str,
|
||||
diagnostics: DiagnosticCollector) -> bool:
|
||||
missing = sorted(required - set(reader.fieldnames or ()))
|
||||
if missing:
|
||||
diagnostics.add("missing_required_column", f"{kind} CSV missing columns: {', '.join(missing)}", report_id)
|
||||
return False
|
||||
return True
|
||||
|
||||
|
||||
def _read_nodes(connection: sqlite3.Connection, pair: ReportPair, stage: str,
|
||||
diagnostics: DiagnosticCollector) -> int:
|
||||
connection.executescript("""
|
||||
DROP TABLE IF EXISTS temp.ingest_nodes;
|
||||
CREATE TEMP TABLE ingest_nodes(
|
||||
row_number INTEGER, node_id TEXT, node_kind TEXT, op_id INTEGER, lane INTEGER,
|
||||
core INTEGER, ssa_name TEXT, attributes_json TEXT
|
||||
);
|
||||
""")
|
||||
valid = 0
|
||||
with pair.nodes_path.open(newline="", encoding="utf-8-sig") as stream:
|
||||
reader = csv.DictReader(stream)
|
||||
if not _check_columns(reader, NODE_COLUMNS, "node", pair.report_id, diagnostics):
|
||||
return 0
|
||||
|
||||
def normalized():
|
||||
nonlocal valid
|
||||
for row_number, row in enumerate(reader, 2):
|
||||
node_id = (row.get("Id") or "").strip()
|
||||
if not node_id:
|
||||
diagnostics.add("invalid_node", "node Id is empty", pair.report_id, f"row {row_number}")
|
||||
continue
|
||||
prefix, id_op, id_lane = _id_parts(node_id)
|
||||
try:
|
||||
op_id = _integer(row.get("op_id"), "op_id")
|
||||
lane = _integer(row.get("lane"), "lane")
|
||||
core = _integer(row.get("core"), "core")
|
||||
except ValueError as exc:
|
||||
diagnostics.add("malformed_integer", str(exc), pair.report_id, f"row {row_number}")
|
||||
continue
|
||||
if op_id is None:
|
||||
if id_op is None:
|
||||
diagnostics.add("invalid_node", "missing op_id and unusable node ID fallback", pair.report_id,
|
||||
f"row {row_number}: {node_id}")
|
||||
continue
|
||||
op_id = id_op
|
||||
diagnostics.add("id_fallback", "op_id parsed from node ID", pair.report_id, node_id,
|
||||
severity="warning")
|
||||
if lane is None and prefix in {"gcb", "scb"} and id_lane is not None:
|
||||
lane = id_lane
|
||||
diagnostics.add("id_fallback", "lane parsed from node ID", pair.report_id, node_id,
|
||||
severity="warning")
|
||||
valid += 1
|
||||
yield (row_number, node_id, NODE_KINDS.get(prefix, f"unknown:{prefix}"), op_id, lane, core,
|
||||
row.get("ssa_name") or "", _extra(row, NODE_COLUMNS))
|
||||
|
||||
for batch in _batches(normalized()):
|
||||
connection.executemany("INSERT INTO ingest_nodes VALUES(?,?,?,?,?,?,?,?)", batch)
|
||||
|
||||
duplicate = connection.execute(
|
||||
"SELECT COALESCE(SUM(c-1),0) FROM (SELECT COUNT(*) c FROM ingest_nodes GROUP BY node_id HAVING c>1)"
|
||||
).fetchone()[0]
|
||||
if duplicate:
|
||||
examples = [row[0] for row in connection.execute(
|
||||
"SELECT node_id FROM ingest_nodes GROUP BY node_id HAVING COUNT(*)>1 ORDER BY node_id LIMIT 5")]
|
||||
diagnostics.add("duplicate_node_id", "duplicate node ID; first row retained", pair.report_id,
|
||||
", ".join(examples), duplicate)
|
||||
connection.execute("""
|
||||
INSERT INTO raw_nodes(report_id,stage,node_id,node_kind,op_id,lane,core,ssa_name,attributes_json)
|
||||
SELECT ?,?,n.node_id,n.node_kind,n.op_id,n.lane,n.core,n.ssa_name,n.attributes_json
|
||||
FROM ingest_nodes n
|
||||
JOIN (SELECT node_id,MIN(row_number) first_row FROM ingest_nodes GROUP BY node_id) first
|
||||
ON first.node_id=n.node_id AND first.first_row=n.row_number
|
||||
ORDER BY n.row_number
|
||||
""", (pair.report_id, stage))
|
||||
return valid - duplicate
|
||||
|
||||
|
||||
def _read_edges(connection: sqlite3.Connection, pair: ReportPair, stage: str,
|
||||
diagnostics: DiagnosticCollector) -> int:
|
||||
connection.executescript("""
|
||||
DROP TABLE IF EXISTS temp.ingest_edges;
|
||||
CREATE TEMP TABLE ingest_edges(
|
||||
row_number INTEGER, stage TEXT, source_node_id TEXT, target_node_id TEXT,
|
||||
explicit_source_lane INTEGER, explicit_target_lane INTEGER, weight INTEGER,
|
||||
tensor_type TEXT, channel_id INTEGER, attributes_json TEXT
|
||||
);
|
||||
""")
|
||||
with pair.edges_path.open(newline="", encoding="utf-8-sig") as stream:
|
||||
reader = csv.DictReader(stream)
|
||||
if not _check_columns(reader, EDGE_COLUMNS, "edge", pair.report_id, diagnostics):
|
||||
return 0
|
||||
|
||||
def normalized():
|
||||
for row_number, row in enumerate(reader, 2):
|
||||
source = (row.get("Source") or "").strip()
|
||||
target = (row.get("Target") or "").strip()
|
||||
if not source or not target:
|
||||
diagnostics.add("invalid_edge", "edge Source or Target is empty", pair.report_id,
|
||||
f"row {row_number}")
|
||||
continue
|
||||
try:
|
||||
source_lane = _integer(row.get("source_lane"), "source_lane")
|
||||
target_lane = _integer(row.get("target_lane"), "target_lane")
|
||||
weight = _integer(row.get("Weight"), "Weight")
|
||||
channel_id = _integer(row.get("channel_id"), "channel_id")
|
||||
except ValueError as exc:
|
||||
diagnostics.add("malformed_integer", str(exc), pair.report_id, f"row {row_number}")
|
||||
continue
|
||||
yield (row_number, (row.get("stage") or stage).strip() or stage, source, target,
|
||||
source_lane, target_lane, weight, row.get("Type") or "", channel_id,
|
||||
_extra(row, EDGE_COLUMNS))
|
||||
|
||||
for batch in _batches(normalized()):
|
||||
connection.executemany("INSERT INTO ingest_edges VALUES(?,?,?,?,?,?,?,?,?,?)", batch)
|
||||
|
||||
unknown_count = connection.execute("""
|
||||
SELECT COUNT(*) FROM ingest_edges e
|
||||
LEFT JOIN raw_nodes s ON s.report_id=? AND s.node_id=e.source_node_id
|
||||
LEFT JOIN raw_nodes t ON t.report_id=? AND t.node_id=e.target_node_id
|
||||
WHERE s.node_id IS NULL OR t.node_id IS NULL
|
||||
""", (pair.report_id, pair.report_id)).fetchone()[0]
|
||||
if unknown_count:
|
||||
examples = [f"{row[0]}->{row[1]}" for row in connection.execute("""
|
||||
SELECT e.source_node_id,e.target_node_id FROM ingest_edges e
|
||||
LEFT JOIN raw_nodes s ON s.report_id=? AND s.node_id=e.source_node_id
|
||||
LEFT JOIN raw_nodes t ON t.report_id=? AND t.node_id=e.target_node_id
|
||||
WHERE s.node_id IS NULL OR t.node_id IS NULL ORDER BY e.row_number LIMIT 5
|
||||
""", (pair.report_id, pair.report_id))]
|
||||
diagnostics.add("unknown_endpoint", "edge endpoint is absent from the matching report; edge omitted",
|
||||
pair.report_id, ", ".join(examples), unknown_count)
|
||||
|
||||
contradictions = connection.execute("""
|
||||
SELECT COUNT(*) FROM ingest_edges e
|
||||
JOIN raw_nodes s ON s.report_id=? AND s.node_id=e.source_node_id
|
||||
JOIN raw_nodes t ON t.report_id=? AND t.node_id=e.target_node_id
|
||||
WHERE (e.explicit_source_lane IS NOT NULL AND s.lane IS NOT NULL AND e.explicit_source_lane<>s.lane)
|
||||
OR (e.explicit_target_lane IS NOT NULL AND t.lane IS NOT NULL AND e.explicit_target_lane<>t.lane)
|
||||
""", (pair.report_id, pair.report_id)).fetchone()[0]
|
||||
if contradictions:
|
||||
diagnostics.add("contradictory_lane", "explicit edge lane contradicts endpoint lane; explicit lane retained",
|
||||
pair.report_id, count=contradictions)
|
||||
|
||||
connection.execute("""
|
||||
INSERT INTO raw_edges(
|
||||
edge_id,report_id,stage,source_node_id,target_node_id,source_op_id,target_op_id,
|
||||
source_lane,target_lane,weight,tensor_type,channel_id,attributes_json)
|
||||
SELECT printf('%s:e:%012d',?,e.row_number),?,e.stage,e.source_node_id,e.target_node_id,s.op_id,t.op_id,
|
||||
COALESCE(e.explicit_source_lane,s.lane),COALESCE(e.explicit_target_lane,t.lane),
|
||||
e.weight,e.tensor_type,e.channel_id,e.attributes_json
|
||||
FROM ingest_edges e
|
||||
JOIN raw_nodes s ON s.report_id=? AND s.node_id=e.source_node_id
|
||||
JOIN raw_nodes t ON t.report_id=? AND t.node_id=e.target_node_id
|
||||
ORDER BY e.row_number
|
||||
""", (pair.report_id, pair.report_id, pair.report_id, pair.report_id))
|
||||
return connection.execute("SELECT COUNT(*) FROM raw_edges WHERE report_id=?", (pair.report_id,)).fetchone()[0]
|
||||
|
||||
|
||||
def ingest_report(connection: sqlite3.Connection, pair: ReportPair, diagnostics: DiagnosticCollector) -> None:
|
||||
stage = infer_stage(pair.report_id)
|
||||
connection.execute("INSERT INTO reports(report_id,stage) VALUES(?,?)", (pair.report_id, stage))
|
||||
node_count = _read_nodes(connection, pair, stage, diagnostics)
|
||||
edge_count = _read_edges(connection, pair, stage, diagnostics) if node_count else 0
|
||||
if not node_count:
|
||||
diagnostics.add("empty_report", "report contains no valid nodes", pair.report_id)
|
||||
connection.execute("UPDATE reports SET raw_node_count=?,raw_edge_count=? WHERE report_id=?",
|
||||
(node_count, edge_count, pair.report_id))
|
||||
@@ -0,0 +1,135 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import sqlite3
|
||||
from collections import defaultdict
|
||||
|
||||
MAX_STENCIL_OFFSETS = 64
|
||||
|
||||
|
||||
def _lane_stats(values: set[int]) -> dict:
|
||||
minimum, maximum = min(values), max(values)
|
||||
return {
|
||||
"lane_count": len(values),
|
||||
"minimum_lane": minimum,
|
||||
"maximum_lane": maximum,
|
||||
"contiguous": len(values) == maximum - minimum + 1,
|
||||
}
|
||||
|
||||
|
||||
def classify_lane_pairs(pairs) -> tuple[str, dict]:
|
||||
relation = set(pairs)
|
||||
if relation and all(source is None and target is None for source, target in relation):
|
||||
return "scalar_to_scalar", {}
|
||||
if relation and all(source is None and target is not None for source, target in relation):
|
||||
return "scalar_to_batch", {"target": _lane_stats({target for _, target in relation})}
|
||||
if relation and all(source is not None and target is None for source, target in relation):
|
||||
return "batch_to_scalar", {"source": _lane_stats({source for source, _ in relation})}
|
||||
if not relation or any(source is None or target is None for source, target in relation):
|
||||
return "unknown", {}
|
||||
|
||||
source_targets: dict[int, set[int]] = defaultdict(set)
|
||||
target_sources: dict[int, set[int]] = defaultdict(set)
|
||||
for source, target in relation:
|
||||
source_targets[source].add(target)
|
||||
target_sources[target].add(source)
|
||||
one_to_one = max(map(len, source_targets.values())) == max(map(len, target_sources.values())) == 1
|
||||
if one_to_one:
|
||||
if all(source == next(iter(targets)) for source, targets in source_targets.items()):
|
||||
return "identity", {}
|
||||
offsets = {next(iter(targets)) - source for source, targets in source_targets.items()}
|
||||
if len(offsets) == 1:
|
||||
return "constant_offset", {"offset": offsets.pop()}
|
||||
if len(source_targets) == len(target_sources):
|
||||
return "permutation", {}
|
||||
if len(source_targets) == 1 and len(target_sources) > 1:
|
||||
return "broadcast", {"source_lane": next(iter(source_targets))}
|
||||
if len(target_sources) == 1 and len(source_targets) > 1:
|
||||
return "fan_in", {"target_lane": next(iter(target_sources))}
|
||||
offset_sets = {tuple(sorted(target - source for target in targets))
|
||||
for source, targets in source_targets.items()}
|
||||
if len(source_targets) > 1 and len(offset_sets) == 1:
|
||||
offsets = next(iter(offset_sets))
|
||||
if 1 < len(offsets) <= MAX_STENCIL_OFFSETS:
|
||||
return "stencil", {"offsets": list(offsets)}
|
||||
return "irregular", {}
|
||||
|
||||
|
||||
def classify_mapping_sql(connection: sqlite3.Connection, report_id: str,
|
||||
source_op_id: int, target_op_id: int) -> tuple[str, dict]:
|
||||
where = "report_id=? AND source_op_id=? AND target_op_id=?"
|
||||
params = (report_id, source_op_id, target_op_id)
|
||||
categories = connection.execute(f"""
|
||||
SELECT COUNT(*) total,
|
||||
SUM(source_lane IS NULL AND target_lane IS NULL) both_null,
|
||||
SUM(source_lane IS NULL AND target_lane IS NOT NULL) source_null,
|
||||
SUM(source_lane IS NOT NULL AND target_lane IS NULL) target_null,
|
||||
SUM(source_lane IS NOT NULL AND target_lane IS NOT NULL) both_set
|
||||
FROM raw_edges WHERE {where}
|
||||
""", params).fetchone()
|
||||
total, both_null, source_null, target_null, both_set = map(lambda value: value or 0, categories)
|
||||
if total == both_null:
|
||||
return "scalar_to_scalar", {}
|
||||
if total == source_null:
|
||||
row = connection.execute(
|
||||
f"SELECT COUNT(DISTINCT target_lane),MIN(target_lane),MAX(target_lane) FROM raw_edges WHERE {where}",
|
||||
params).fetchone()
|
||||
return "scalar_to_batch", {"target": {"lane_count": row[0], "minimum_lane": row[1],
|
||||
"maximum_lane": row[2], "contiguous": row[0] == row[2] - row[1] + 1}}
|
||||
if total == target_null:
|
||||
row = connection.execute(
|
||||
f"SELECT COUNT(DISTINCT source_lane),MIN(source_lane),MAX(source_lane) FROM raw_edges WHERE {where}",
|
||||
params).fetchone()
|
||||
return "batch_to_scalar", {"source": {"lane_count": row[0], "minimum_lane": row[1],
|
||||
"maximum_lane": row[2], "contiguous": row[0] == row[2] - row[1] + 1}}
|
||||
if total != both_set:
|
||||
return "unknown", {}
|
||||
|
||||
stats = connection.execute(f"""
|
||||
SELECT COUNT(DISTINCT source_lane),COUNT(DISTINCT target_lane),
|
||||
SUM(source_lane<>target_lane),MIN(target_lane-source_lane),MAX(target_lane-source_lane)
|
||||
FROM raw_edges WHERE {where}
|
||||
""", params).fetchone()
|
||||
source_count, target_count, non_identity, min_offset, max_offset = stats
|
||||
max_out = connection.execute(
|
||||
f"SELECT MAX(c) FROM (SELECT COUNT(DISTINCT target_lane)c FROM raw_edges WHERE {where} GROUP BY source_lane)",
|
||||
params).fetchone()[0]
|
||||
max_in = connection.execute(
|
||||
f"SELECT MAX(c) FROM (SELECT COUNT(DISTINCT source_lane)c FROM raw_edges WHERE {where} GROUP BY target_lane)",
|
||||
params).fetchone()[0]
|
||||
if max_out == max_in == 1:
|
||||
if not non_identity:
|
||||
return "identity", {}
|
||||
if min_offset == max_offset:
|
||||
return "constant_offset", {"offset": min_offset}
|
||||
if source_count == target_count:
|
||||
return "permutation", {}
|
||||
if source_count == 1 and target_count > 1:
|
||||
lane = connection.execute(f"SELECT MIN(source_lane) FROM raw_edges WHERE {where}", params).fetchone()[0]
|
||||
return "broadcast", {"source_lane": lane}
|
||||
if target_count == 1 and source_count > 1:
|
||||
lane = connection.execute(f"SELECT MIN(target_lane) FROM raw_edges WHERE {where}", params).fetchone()[0]
|
||||
return "fan_in", {"target_lane": lane}
|
||||
|
||||
reference = None
|
||||
offsets: list[int] = []
|
||||
current = None
|
||||
for source, target in connection.execute(
|
||||
f"SELECT DISTINCT source_lane,target_lane FROM raw_edges WHERE {where} ORDER BY source_lane,target_lane",
|
||||
params):
|
||||
if source != current and current is not None:
|
||||
candidate = tuple(offsets)
|
||||
if reference is None:
|
||||
reference = candidate
|
||||
elif candidate != reference:
|
||||
return "irregular", {}
|
||||
offsets = []
|
||||
current = source
|
||||
offsets.append(target - source)
|
||||
if len(offsets) > MAX_STENCIL_OFFSETS:
|
||||
return "irregular", {}
|
||||
candidate = tuple(offsets)
|
||||
if reference is None:
|
||||
reference = candidate
|
||||
if source_count > 1 and candidate == reference and 1 < len(reference) <= MAX_STENCIL_OFFSETS:
|
||||
return "stencil", {"offsets": list(reference)}
|
||||
return "irregular", {}
|
||||
@@ -0,0 +1,140 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from dataclasses import dataclass
|
||||
|
||||
import networkx as nx
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Motif:
|
||||
entry: str
|
||||
exit: str
|
||||
internal: tuple[str, ...]
|
||||
branches: tuple[str, ...]
|
||||
paths: tuple[tuple[str, ...], ...]
|
||||
node_count: int
|
||||
edge_count: int
|
||||
contains_cycle: bool
|
||||
kind: str
|
||||
confidence: float
|
||||
|
||||
|
||||
def _condense(graph: nx.DiGraph):
|
||||
components = sorted((tuple(sorted(component)) for component in nx.strongly_connected_components(graph)),
|
||||
key=lambda component: component)
|
||||
member_to_component = {member: index for index, component in enumerate(components) for member in component}
|
||||
dag = nx.DiGraph()
|
||||
dag.add_nodes_from(range(len(components)))
|
||||
dag.add_edges_from((member_to_component[source], member_to_component[target])
|
||||
for source, target in graph.edges if member_to_component[source] != member_to_component[target])
|
||||
return dag, components, member_to_component
|
||||
|
||||
|
||||
def _reachable_without_crossing(dag: nx.DiGraph, entry: int, exit_node: int) -> set[int]:
|
||||
seen = {entry}
|
||||
stack = [entry]
|
||||
while stack:
|
||||
node = stack.pop()
|
||||
for successor in dag.successors(node):
|
||||
if successor == exit_node or successor in seen:
|
||||
continue
|
||||
seen.add(successor)
|
||||
stack.append(successor)
|
||||
return seen
|
||||
|
||||
|
||||
def _kind(dag: nx.DiGraph, entry: int, exit_node: int, internal: set[int], branches: list[int]) -> str:
|
||||
if (len(branches) == 2 and internal == set(branches)
|
||||
and all(dag.has_edge(branch, exit_node) for branch in branches)):
|
||||
return "exact_diamond"
|
||||
branch_regions = []
|
||||
region = internal | {exit_node}
|
||||
for branch in branches:
|
||||
branch_regions.append(({node for node in nx.descendants(dag, branch) | {branch} if node in region}) - {exit_node})
|
||||
if all(left.isdisjoint(right) for index, left in enumerate(branch_regions) for right in branch_regions[index + 1:]):
|
||||
return "rhomboid"
|
||||
return "split_rejoin"
|
||||
|
||||
|
||||
def _dominates(dominators: dict, entry: int, node: int) -> bool:
|
||||
while node in dominators and node != dominators[node]:
|
||||
if node == entry:
|
||||
return True
|
||||
node = dominators[node]
|
||||
return node == entry
|
||||
|
||||
|
||||
def detect_motifs(graph: nx.DiGraph) -> tuple[list[Motif], int]:
|
||||
if not graph:
|
||||
return [], 0
|
||||
dag, components, _ = _condense(graph)
|
||||
source, sink = "__source__", "__sink__"
|
||||
dag.add_node(source)
|
||||
dag.add_node(sink)
|
||||
dag.add_edges_from((source, node) for node in list(dag) if node not in {source, sink} and dag.in_degree(node) == 0)
|
||||
dag.add_edges_from((node, sink) for node in list(dag) if node not in {source, sink} and dag.out_degree(node) == 0)
|
||||
dominators = nx.immediate_dominators(dag, source)
|
||||
post_dominators = nx.immediate_dominators(dag.reverse(copy=False), sink)
|
||||
motifs = []
|
||||
seen = set()
|
||||
for entry in sorted(node for node in dag if isinstance(node, int) and dag.out_degree(node) >= 2):
|
||||
exit_node = post_dominators.get(entry)
|
||||
if not isinstance(exit_node, int) or exit_node == entry:
|
||||
continue
|
||||
branches = sorted(successor for successor in dag.successors(entry)
|
||||
if successor == exit_node or nx.has_path(dag, successor, exit_node))
|
||||
if len(branches) < 2:
|
||||
continue
|
||||
reachable = _reachable_without_crossing(dag, entry, exit_node)
|
||||
internal = (reachable & (nx.ancestors(dag, exit_node) | {exit_node})) - {entry, exit_node}
|
||||
region = internal | {entry, exit_node}
|
||||
if any(not _dominates(dominators, entry, node) for node in internal):
|
||||
continue
|
||||
if any(predecessor not in region for node in internal for predecessor in dag.predecessors(node)):
|
||||
continue
|
||||
if any(successor not in region for node in internal for successor in dag.successors(node)):
|
||||
continue
|
||||
expanded = sorted(member for component in region for member in components[component])
|
||||
entry_aggregate = components[entry][0]
|
||||
exit_aggregate = components[exit_node][0]
|
||||
internal_aggregates = tuple(node for node in expanded if node not in {entry_aggregate, exit_aggregate})
|
||||
key = (entry_aggregate, exit_aggregate, internal_aggregates)
|
||||
if key in seen:
|
||||
continue
|
||||
seen.add(key)
|
||||
branch_aggregates = tuple(components[branch][0] for branch in branches)
|
||||
paths = tuple(tuple(components[node][0] for node in [entry] + nx.shortest_path(dag, branch, exit_node))
|
||||
for branch in branches)
|
||||
subgraph = graph.subgraph(expanded)
|
||||
cycle = any(len(components[component]) > 1 for component in region) or any(graph.has_edge(node, node) for node in expanded)
|
||||
kind = _kind(dag, entry, exit_node, internal, branches)
|
||||
motifs.append(Motif(entry_aggregate, exit_aggregate, internal_aggregates, branch_aggregates, paths,
|
||||
len(expanded), subgraph.number_of_edges(), cycle, kind,
|
||||
1.0 if kind != "split_rejoin" else 0.9))
|
||||
motifs.sort(key=lambda motif: (motif.entry, motif.exit, motif.internal))
|
||||
return motifs, len(components)
|
||||
|
||||
|
||||
def analyze_report(connection: sqlite3.Connection, report_id: str) -> None:
|
||||
graph = nx.DiGraph()
|
||||
graph.add_nodes_from(row[0] for row in connection.execute(
|
||||
"SELECT aggregate_id FROM aggregate_nodes WHERE report_id=? AND level='operation' ORDER BY aggregate_id",
|
||||
(report_id,)))
|
||||
graph.add_edges_from(connection.execute("""
|
||||
SELECT source_aggregate_id,target_aggregate_id FROM aggregate_edges
|
||||
WHERE report_id=? AND level='operation' ORDER BY aggregate_edge_id
|
||||
""", (report_id,)))
|
||||
motifs, scc_count = detect_motifs(graph)
|
||||
for index, motif in enumerate(motifs, 1):
|
||||
motif_id = f"motif:{report_id}:{index:04d}"
|
||||
connection.execute("INSERT INTO motifs VALUES(?,?,?,?,?,?,?,?,?,?,?,?)", (
|
||||
motif_id, report_id, motif.entry, motif.exit, json.dumps(motif.internal), json.dumps(motif.branches),
|
||||
json.dumps(motif.paths), motif.node_count, motif.edge_count, int(motif.contains_cycle),
|
||||
motif.kind, motif.confidence))
|
||||
roles = [(motif_id, motif.entry, "entry"), (motif_id, motif.exit, "exit")]
|
||||
roles.extend((motif_id, node, "internal") for node in motif.internal)
|
||||
connection.executemany("INSERT INTO motif_membership VALUES(?,?,?)", roles)
|
||||
connection.execute("UPDATE reports SET scc_count=?,motif_count=? WHERE report_id=?",
|
||||
(scc_count, len(motifs), report_id))
|
||||
@@ -0,0 +1,178 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from collections import defaultdict
|
||||
|
||||
import networkx as nx
|
||||
|
||||
JSON_COLUMNS = {
|
||||
"attributes_json", "mapping_metadata", "node_kind_counts", "ssa_summary",
|
||||
"tensor_type_histogram",
|
||||
}
|
||||
LANE_SPACING = 42
|
||||
OPERATION_SPACING = 240
|
||||
|
||||
|
||||
class DisplayGraphTooLarge(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
def _record(row: sqlite3.Row) -> dict:
|
||||
result = dict(row)
|
||||
for key in JSON_COLUMNS & result.keys():
|
||||
result[key] = json.loads(result[key])
|
||||
return result
|
||||
|
||||
|
||||
def _operation_anchors(nodes: list[dict], edges: list[dict]) -> dict[int, tuple[float, float]]:
|
||||
graph = nx.DiGraph()
|
||||
graph.add_nodes_from(node["op_id"] for node in nodes)
|
||||
graph.add_edges_from((edge["source_op_id"], edge["target_op_id"]) for edge in edges)
|
||||
condensed = nx.condensation(graph)
|
||||
ranks: dict[int, int] = {}
|
||||
for component in nx.topological_sort(condensed):
|
||||
ranks[component] = max((ranks[parent] + 1 for parent in condensed.predecessors(component)), default=0)
|
||||
operation_ranks = {op_id: ranks[condensed.graph["mapping"][op_id]] for op_id in graph}
|
||||
by_rank: dict[int, list[dict]] = defaultdict(list)
|
||||
for node in nodes:
|
||||
by_rank[operation_ranks[node["op_id"]]].append(node)
|
||||
|
||||
anchors = {}
|
||||
for rank, ranked_nodes in sorted(by_rank.items()):
|
||||
ranked_nodes.sort(key=lambda item: item["op_id"])
|
||||
cursor = 0
|
||||
for node in ranked_nodes:
|
||||
minimum = node["minimum_lane"] if node["minimum_lane"] is not None else 0
|
||||
maximum = node["maximum_lane"] if node["maximum_lane"] is not None else minimum - 1
|
||||
lane_rows = max(0, maximum - minimum + 1)
|
||||
scalar_rows = max(0, node["instance_count"] - node["lane_count"])
|
||||
anchor_y = 0 if len(ranked_nodes) == 1 else cursor - minimum * LANE_SPACING
|
||||
anchors[node["op_id"]] = (rank * OPERATION_SPACING, anchor_y)
|
||||
cursor += max(1, lane_rows + scalar_rows) * LANE_SPACING + 2 * LANE_SPACING
|
||||
return anchors
|
||||
|
||||
|
||||
def _where_ids(column: str, ids: list[int]) -> tuple[str, list[int]]:
|
||||
return f"{column} IN ({','.join('?' for _ in ids)})", ids
|
||||
|
||||
|
||||
def _raw_nodes(db: sqlite3.Connection, report_id: str, expanded: set[int], all_expanded: bool) -> list[dict]:
|
||||
if all_expanded:
|
||||
rows = db.execute("SELECT * FROM raw_nodes WHERE report_id=? ORDER BY op_id,lane IS NULL,lane,node_id",
|
||||
(report_id,))
|
||||
return [_record(row) for row in rows]
|
||||
result = []
|
||||
ids = sorted(expanded)
|
||||
for offset in range(0, len(ids), 800):
|
||||
where, params = _where_ids("op_id", ids[offset:offset + 800])
|
||||
result.extend(_record(row) for row in db.execute(
|
||||
f"SELECT * FROM raw_nodes WHERE report_id=? AND {where}", (report_id, *params)))
|
||||
return sorted(result, key=lambda row: (row["op_id"], row["lane"] is None,
|
||||
row["lane"] if row["lane"] is not None else 0, row["node_id"]))
|
||||
|
||||
|
||||
def _raw_edges(db: sqlite3.Connection, report_id: str, expanded: set[int], all_expanded: bool) -> list[dict]:
|
||||
if all_expanded:
|
||||
return [_record(row) for row in db.execute(
|
||||
"SELECT * FROM raw_edges WHERE report_id=? ORDER BY edge_id", (report_id,))]
|
||||
result: dict[str, dict] = {}
|
||||
ids = sorted(expanded)
|
||||
for offset in range(0, len(ids), 400):
|
||||
chunk = ids[offset:offset + 400]
|
||||
source_where, source_params = _where_ids("source_op_id", chunk)
|
||||
target_where, target_params = _where_ids("target_op_id", chunk)
|
||||
for row in db.execute(
|
||||
f"SELECT * FROM raw_edges WHERE report_id=? AND ({source_where} OR {target_where})",
|
||||
(report_id, *source_params, *target_params)):
|
||||
item = _record(row)
|
||||
result[item["edge_id"]] = item
|
||||
return [result[key] for key in sorted(result)]
|
||||
|
||||
|
||||
def display_graph(db: sqlite3.Connection, report_id: str, *, view: str = "operation",
|
||||
expanded_operation_ids=(), expand_all: bool = False, cap: int = 5_000) -> dict:
|
||||
if view not in {"operation", "raw"}:
|
||||
raise ValueError("view must be operation or raw")
|
||||
if not db.execute("SELECT 1 FROM reports WHERE report_id=?", (report_id,)).fetchone():
|
||||
raise LookupError("report not found")
|
||||
|
||||
operation_nodes = [_record(row) for row in db.execute(
|
||||
"SELECT * FROM aggregate_nodes WHERE report_id=? AND level='operation' ORDER BY aggregate_id",
|
||||
(report_id,))]
|
||||
operation_edges = [_record(row) for row in db.execute(
|
||||
"SELECT * FROM aggregate_edges WHERE report_id=? AND level='operation' ORDER BY aggregate_edge_id",
|
||||
(report_id,))]
|
||||
known = {node["op_id"] for node in operation_nodes}
|
||||
aggregate_ids = {node["op_id"]: node["aggregate_id"] for node in operation_nodes}
|
||||
expanded = known if view == "raw" or expand_all else set(expanded_operation_ids)
|
||||
unknown = expanded - known
|
||||
if unknown:
|
||||
raise ValueError(f"unknown operation IDs: {', '.join(map(str, sorted(unknown)))}")
|
||||
|
||||
raw_node_count = sum(node["instance_count"] for node in operation_nodes if node["op_id"] in expanded)
|
||||
raw_edge_count = sum(edge["raw_edge_count"] for edge in operation_edges
|
||||
if edge["source_op_id"] in expanded or edge["target_op_id"] in expanded)
|
||||
if raw_node_count + raw_edge_count > cap:
|
||||
raise DisplayGraphTooLarge(
|
||||
f"raw/expanded graph requires {raw_node_count} raw nodes and {raw_edge_count} raw edges; "
|
||||
f"safety cap is {cap}. Use an aggregate view or increase the configured expansion cap."
|
||||
)
|
||||
|
||||
anchors = _operation_anchors(operation_nodes, operation_edges)
|
||||
nodes = []
|
||||
for node in operation_nodes:
|
||||
if node["op_id"] not in expanded:
|
||||
x, y = anchors[node["op_id"]]
|
||||
nodes.append({**node, "display_id": node["aggregate_id"], "representation": "aggregate",
|
||||
"x": x, "y": y})
|
||||
|
||||
raw_nodes = _raw_nodes(db, report_id, expanded, expanded == known)
|
||||
lane_occurrences: dict[tuple[int, int], int] = defaultdict(int)
|
||||
scalar_indices: dict[int, int] = defaultdict(int)
|
||||
maximum_lanes = {node["op_id"]: node["maximum_lane"] for node in operation_nodes}
|
||||
for node in raw_nodes:
|
||||
anchor_x, anchor_y = anchors[node["op_id"]]
|
||||
if node["lane"] is None:
|
||||
index = scalar_indices[node["op_id"]]
|
||||
scalar_indices[node["op_id"]] += 1
|
||||
row = (maximum_lanes[node["op_id"]] if maximum_lanes[node["op_id"]] is not None else -1) + 1 + index
|
||||
x, y = anchor_x, anchor_y + row * LANE_SPACING
|
||||
else:
|
||||
key = (node["op_id"], node["lane"])
|
||||
x = anchor_x + lane_occurrences[key] * 12
|
||||
y = anchor_y + node["lane"] * LANE_SPACING
|
||||
lane_occurrences[key] += 1
|
||||
nodes.append({**node, "display_id": node["node_id"], "representation": "raw", "x": x, "y": y})
|
||||
|
||||
edges = []
|
||||
for edge in operation_edges:
|
||||
if edge["source_op_id"] not in expanded and edge["target_op_id"] not in expanded:
|
||||
edges.append({**edge, "display_id": edge["aggregate_edge_id"], "representation": "aggregate",
|
||||
"source": edge["source_aggregate_id"], "target": edge["target_aggregate_id"]})
|
||||
for edge in _raw_edges(db, report_id, expanded, expanded == known):
|
||||
source = edge["source_node_id"] if edge["source_op_id"] in expanded else aggregate_ids[edge["source_op_id"]]
|
||||
target = edge["target_node_id"] if edge["target_op_id"] in expanded else aggregate_ids[edge["target_op_id"]]
|
||||
edges.append({**edge, "display_id": edge["edge_id"], "representation": "raw",
|
||||
"source": source, "target": target})
|
||||
|
||||
nodes.sort(key=lambda item: item["display_id"])
|
||||
edges.sort(key=lambda item: item["display_id"])
|
||||
degree = {node["display_id"]: 0 for node in nodes}
|
||||
for edge in edges:
|
||||
degree[edge["source"]] += 1
|
||||
degree[edge["target"]] += 1
|
||||
isolated = sum(value == 0 for value in degree.values())
|
||||
raw_node_count = sum(node["representation"] == "raw" for node in nodes)
|
||||
raw_edge_count = sum(edge["representation"] == "raw" for edge in edges)
|
||||
return {
|
||||
"view": view,
|
||||
"expanded_operation_ids": sorted(expanded),
|
||||
"nodes": nodes,
|
||||
"edges": edges,
|
||||
"counts": {
|
||||
"nodes": len(nodes), "edges": len(edges), "isolated_nodes": isolated,
|
||||
"raw_nodes": raw_node_count, "aggregate_nodes": len(nodes) - raw_node_count,
|
||||
"raw_edges": raw_edge_count, "aggregate_edges": len(edges) - raw_edge_count,
|
||||
},
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
|
||||
SCHEMA_VERSION = 1
|
||||
NODE_COLUMNS = {"Id", "op_id", "lane", "core", "ssa_name"}
|
||||
EDGE_COLUMNS = {"Source", "Target", "Weight", "Type", "stage", "source_lane", "target_lane", "channel_id"}
|
||||
NODE_KINDS = {
|
||||
"gc": "graph_compute",
|
||||
"gcb": "graph_compute_batch_lane",
|
||||
"sc": "scheduled_compute",
|
||||
"scb": "scheduled_compute_batch_lane",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ReportPair:
|
||||
report_id: str
|
||||
nodes_path: Path
|
||||
edges_path: Path
|
||||
|
||||
|
||||
@dataclass
|
||||
class Diagnostic:
|
||||
code: str
|
||||
message: str
|
||||
report_id: str | None = None
|
||||
severity: str = "error"
|
||||
count: int = 1
|
||||
examples: list[str] = field(default_factory=list)
|
||||
|
||||
|
||||
class DiagnosticCollector:
|
||||
def __init__(self, example_cap: int = 5):
|
||||
self.example_cap = example_cap
|
||||
self._items: dict[tuple[str | None, str, str, str], Diagnostic] = {}
|
||||
|
||||
def add(self, code: str, message: str, report_id: str | None = None, example: str | None = None,
|
||||
count: int = 1, severity: str = "error") -> None:
|
||||
key = (report_id, code, message, severity)
|
||||
item = self._items.setdefault(key, Diagnostic(code, message, report_id, severity, 0, []))
|
||||
item.count += count
|
||||
if example and len(item.examples) < self.example_cap and example not in item.examples:
|
||||
item.examples.append(example)
|
||||
|
||||
def extend(self, diagnostics: list[Diagnostic]) -> None:
|
||||
for item in diagnostics:
|
||||
examples = item.examples or [None]
|
||||
self.add(item.code, item.message, item.report_id, examples[0], item.count, item.severity)
|
||||
for example in examples[1:]:
|
||||
self.add(item.code, item.message, item.report_id, example, 0, item.severity)
|
||||
|
||||
@property
|
||||
def items(self) -> list[Diagnostic]:
|
||||
return sorted(self._items.values(), key=lambda d: (d.report_id or "", d.code, d.message))
|
||||
|
||||
@property
|
||||
def error_count(self) -> int:
|
||||
return sum(d.count for d in self._items.values() if d.severity == "error")
|
||||
|
||||
|
||||
class BuildError(RuntimeError):
|
||||
pass
|
||||
@@ -0,0 +1,232 @@
|
||||
/* Pinned dependencies are declared once in index.html. */
|
||||
const $ = id => document.getElementById(id);
|
||||
const state = {
|
||||
reports: [], data: null, graph: null, renderer: null, motifs: [], motif: null,
|
||||
expandedOperationIds: new Set(), mappingOffset: 0,
|
||||
};
|
||||
const colors = {operation: "#55a6e8", core: "#68c58c", node_kind: "#d994e8", raw: "#f8b84e"};
|
||||
|
||||
async function api(path) {
|
||||
const response = await fetch(path);
|
||||
if (!response.ok) throw new Error((await response.json().catch(() => ({}))).detail || `${response.status} ${response.statusText}`);
|
||||
return response.json();
|
||||
}
|
||||
function html(value) { const div = document.createElement("div"); div.textContent = value ?? "—"; return div.innerHTML; }
|
||||
function definition(object, keys) {
|
||||
return `<dl>${keys.filter(key => object[key] !== undefined).map(key => `<dt>${html(key.replaceAll("_", " "))}</dt><dd>${html(typeof object[key] === "object" ? JSON.stringify(object[key]) : object[key])}</dd>`).join("")}</dl>`;
|
||||
}
|
||||
function status(message, error = false) { $("status").textContent = message; $("status").className = error ? "error" : ""; }
|
||||
function selectedReport() { return state.reports.find(report => report.report_id === $("report").value); }
|
||||
function defaultViewForReport(report) {
|
||||
if (report?.stage === "spatial3") return "raw";
|
||||
const saved = localStorage.getItem("raptor-graph-view");
|
||||
return ["operation", "raw", "core", "node_kind"].includes(saved) ? saved : "operation";
|
||||
}
|
||||
|
||||
async function loadReports() {
|
||||
state.reports = await api("/api/reports");
|
||||
if (!state.reports.length) throw new Error("No reports were preprocessed.");
|
||||
$("report").innerHTML = state.reports.map(report => `<option>${html(report.report_id)}</option>`).join("");
|
||||
$("level").value = defaultViewForReport(selectedReport());
|
||||
await loadGraph();
|
||||
}
|
||||
function aggregateGraph(data) {
|
||||
return {
|
||||
view: data.level,
|
||||
nodes: data.nodes.map(node => ({...node, display_id: node.aggregate_id, representation: "aggregate", x: 0, y: 0})),
|
||||
edges: data.edges.map(edge => ({...edge, display_id: edge.aggregate_edge_id, representation: "aggregate", source: edge.source_aggregate_id, target: edge.target_aggregate_id})),
|
||||
};
|
||||
}
|
||||
async function requestDisplay(expandAll = false) {
|
||||
const report = encodeURIComponent($("report").value);
|
||||
const params = new URLSearchParams({view: $("level").value});
|
||||
if (expandAll) params.set("expand_all", "true");
|
||||
else for (const opId of [...state.expandedOperationIds].sort((a, b) => a - b)) params.append("expanded_op_id", opId);
|
||||
return api(`/api/reports/${report}/display-graph?${params}`);
|
||||
}
|
||||
async function loadGraph(resetExpansion = true) {
|
||||
const report = $("report").value, level = $("level").value;
|
||||
if (resetExpansion) state.expandedOperationIds.clear();
|
||||
state.motif = null;
|
||||
status(`Loading ${report} / ${level}…`);
|
||||
try {
|
||||
const graphRequest = ["operation", "raw"].includes(level)
|
||||
? requestDisplay()
|
||||
: api(`/api/reports/${encodeURIComponent(report)}/graph/${level}`).then(aggregateGraph);
|
||||
[state.data, state.motifs] = await Promise.all([
|
||||
graphRequest,
|
||||
["operation", "raw"].includes(level) ? api(`/api/reports/${encodeURIComponent(report)}/motifs`) : [],
|
||||
]);
|
||||
state.expandedOperationIds = new Set(state.data.expanded_operation_ids || []);
|
||||
await renderGraph();
|
||||
await loadSummary();
|
||||
renderMotifs();
|
||||
updateExpansionButtons();
|
||||
updateStatus();
|
||||
} catch (error) { status(error.message, true); }
|
||||
}
|
||||
function updateStatus() {
|
||||
if (state.data.counts) {
|
||||
const counts = state.data.counts;
|
||||
status(`${counts.nodes} displayed nodes, ${counts.edges} displayed edges, ${counts.isolated_nodes} isolated nodes`);
|
||||
} else status(`${state.data.nodes.length} aggregate nodes, ${state.data.edges.length} aggregate edges`);
|
||||
}
|
||||
function nodeLabel(node) {
|
||||
if (node.representation === "raw") return node.lane === null ? node.node_id : `${node.node_id} · lane ${node.lane}`;
|
||||
return node.op_id !== null ? `op ${node.op_id}` : node.group_key;
|
||||
}
|
||||
function edgeSize(edge) {
|
||||
if (edge.representation === "raw") return 1;
|
||||
const value = edge[$("metric").value] || 0;
|
||||
return 1 + Math.log2(1 + Math.max(0, value)) / 3;
|
||||
}
|
||||
async function renderGraph() {
|
||||
if (state.renderer) state.renderer.kill();
|
||||
const graph = new graphology.MultiDirectedGraph();
|
||||
for (const node of state.data.nodes) graph.addNode(node.display_id, {
|
||||
label: nodeLabel(node), x: node.x, y: node.y,
|
||||
size: node.representation === "raw" ? 3 : 4 + Math.log2(1 + node.instance_count),
|
||||
color: node.representation === "raw" ? colors.raw : colors[node.level], raw: node,
|
||||
searchable: JSON.stringify(node).toLowerCase(),
|
||||
});
|
||||
for (const edge of state.data.edges) if (graph.hasNode(edge.source) && graph.hasNode(edge.target)) {
|
||||
graph.addDirectedEdgeWithKey(edge.display_id, edge.source, edge.target, {
|
||||
size: edgeSize(edge), color: edge.representation === "raw" ? "#e0a94f" : "#718096",
|
||||
type: "arrow", raw: edge,
|
||||
});
|
||||
}
|
||||
state.graph = graph;
|
||||
await runLayout();
|
||||
state.renderer = new Sigma(graph, $("graph"), {defaultEdgeType: "arrow", enableEdgeEvents: true, nodeReducer: reduceNode, edgeReducer: reduceEdge});
|
||||
state.renderer.on("clickNode", event => showNode(event.node));
|
||||
state.renderer.on("clickEdge", event => showEdge(event.edge));
|
||||
applyFilters();
|
||||
}
|
||||
async function runLayout() {
|
||||
if (["operation", "raw"].includes($("level").value)) {
|
||||
for (const id of state.graph.nodes()) {
|
||||
const node = state.graph.getNodeAttribute(id, "raw");
|
||||
state.graph.mergeNodeAttributes(id, {x: node.x, y: node.y});
|
||||
}
|
||||
} else {
|
||||
const layout = await new ELK().layout({
|
||||
id: "root",
|
||||
layoutOptions: {"elk.algorithm": "layered", "elk.direction": "RIGHT", "elk.spacing.nodeNode": "35", "elk.layered.spacing.nodeNodeBetweenLayers": "65"},
|
||||
children: state.graph.nodes().map(id => ({id, width: 28, height: 28})),
|
||||
edges: state.graph.edges().map(id => ({id, sources: [state.graph.source(id)], targets: [state.graph.target(id)]})),
|
||||
});
|
||||
for (const node of layout.children || []) if (state.graph.hasNode(node.id)) state.graph.mergeNodeAttributes(node.id, {x: node.x || 0, y: -(node.y || 0)});
|
||||
}
|
||||
state.renderer?.refresh();
|
||||
}
|
||||
function motifIdentity(node) {
|
||||
return node.representation === "raw" ? `operation:${node.report_id}:${node.op_id}` : node.aggregate_id;
|
||||
}
|
||||
function reduceNode(id, attributes) {
|
||||
const result = {...attributes}, search = $("search").value.trim().toLowerCase();
|
||||
if (search && !attributes.searchable.includes(search)) result.hidden = true;
|
||||
if (state.motif) {
|
||||
const members = new Set([state.motif.entry_aggregate_id, state.motif.exit_aggregate_id, ...state.motif.internal_aggregate_ids]);
|
||||
const identity = motifIdentity(attributes.raw);
|
||||
if (!members.has(identity)) { result.color = "#263341"; result.label = ""; }
|
||||
else if (identity === state.motif.entry_aggregate_id) { result.color = "#54d67b"; result.size *= 1.5; }
|
||||
else if (identity === state.motif.exit_aggregate_id) { result.color = "#ff6f6f"; result.size *= 1.5; }
|
||||
}
|
||||
return result;
|
||||
}
|
||||
function reduceEdge(id, attributes) {
|
||||
const result = {...attributes}, edge = attributes.raw || {};
|
||||
if (!$("selfEdges").checked && edge.source === edge.target) result.hidden = true;
|
||||
if ($("mapping").value && edge.mapping_kind !== $("mapping").value) result.hidden = true;
|
||||
if ($("tensor").value && !JSON.stringify(edge.tensor_type_histogram || edge.tensor_type || "").toLowerCase().includes($("tensor").value.toLowerCase())) result.hidden = true;
|
||||
return result;
|
||||
}
|
||||
function applyFilters() { state.renderer?.refresh(); }
|
||||
function clearMapping() { $("matrix").hidden = true; $("mappingRows").innerHTML = ""; }
|
||||
|
||||
async function showNode(id) {
|
||||
const local = state.graph.getNodeAttribute(id, "raw");
|
||||
clearMapping();
|
||||
if (local.representation === "raw") {
|
||||
$("details").innerHTML = definition(local, ["node_id", "node_kind", "op_id", "lane", "core", "ssa_name", "stage", "attributes_json"])
|
||||
+ ($("level").value === "operation" ? '<button id="collapseSelected">Collapse selected operation</button>' : "");
|
||||
if ($("collapseSelected")) $("collapseSelected").onclick = () => collapseOperation(local.op_id);
|
||||
return;
|
||||
}
|
||||
const node = await api(`/api/aggregate-nodes/${encodeURIComponent(id)}`);
|
||||
$("details").innerHTML = definition(node, ["aggregate_id", "op_id", "core", "node_kind_counts", "instance_count", "lane_count", "minimum_lane", "maximum_lane", "core_count", "minimum_core", "maximum_core", "ssa_name_count", "ssa_names"])
|
||||
+ (node.level === "operation" ? '<button id="expandSelected">Expand selected operation</button>' : "");
|
||||
if ($("expandSelected")) $("expandSelected").onclick = () => expandOperation(node.op_id);
|
||||
}
|
||||
async function rebuildProjection(expandAll = false) {
|
||||
try {
|
||||
state.data = await requestDisplay(expandAll);
|
||||
state.expandedOperationIds = new Set(state.data.expanded_operation_ids);
|
||||
await renderGraph();
|
||||
updateExpansionButtons();
|
||||
updateStatus();
|
||||
} catch (error) { status(error.message, true); }
|
||||
}
|
||||
function expandOperation(opId) { state.expandedOperationIds.add(opId); return rebuildProjection(); }
|
||||
function collapseOperation(opId) { state.expandedOperationIds.delete(opId); return rebuildProjection(); }
|
||||
function expandAllOperations() { return rebuildProjection(true); }
|
||||
function collapseAllOperations() { state.expandedOperationIds.clear(); return rebuildProjection(); }
|
||||
function updateExpansionButtons() {
|
||||
const operation = $("level").value === "operation";
|
||||
$("expandAll").hidden = !operation;
|
||||
$("collapseAll").hidden = !operation || state.expandedOperationIds.size === 0;
|
||||
}
|
||||
|
||||
async function showEdge(id) {
|
||||
const local = state.graph.getEdgeAttribute(id, "raw");
|
||||
clearMapping();
|
||||
if (local.representation === "raw") {
|
||||
$("details").innerHTML = definition(local, ["edge_id", "source_node_id", "target_node_id", "source_op_id", "target_op_id", "source_lane", "target_lane", "weight", "tensor_type", "channel_id", "stage", "attributes_json"]);
|
||||
return;
|
||||
}
|
||||
const edge = await api(`/api/aggregate-edges/${encodeURIComponent(id)}`);
|
||||
$("details").innerHTML = definition(edge, ["aggregate_edge_id", "source_aggregate_id", "target_aggregate_id", "raw_edge_count", "total_weight", "minimum_weight", "maximum_weight", "tensor_type_histogram", "distinct_channel_count", "mapping_kind", "mapping_metadata"]);
|
||||
if (edge.level === "operation") {
|
||||
state.mappingOffset = 0;
|
||||
await Promise.all([drawMatrix(id), loadMappingRows(id)]);
|
||||
}
|
||||
}
|
||||
async function drawMatrix(id) {
|
||||
const matrix = await api(`/api/aggregate-edges/${encodeURIComponent(id)}/matrix`), canvas = $("matrix"), context = canvas.getContext("2d");
|
||||
canvas.hidden = false; context.fillStyle = "#0b1119"; context.fillRect(0, 0, canvas.width, canvas.height);
|
||||
const points = matrix.mode === "exact" ? matrix.points.map(point => [point.source, point.target, point.count]) : matrix.bins;
|
||||
const sx = matrix.mode === "exact" ? matrix.source_extent : [0, matrix.dimensions[0] - 1], sy = matrix.mode === "exact" ? matrix.target_extent : [0, matrix.dimensions[1] - 1], maximum = Math.max(1, ...points.map(point => point[2]));
|
||||
for (const [source, target, count] of points) {
|
||||
if (source === null || target === null) continue;
|
||||
const x = (source - sx[0]) / Math.max(1, sx[1] - sx[0]) * (canvas.width - 8) + 4, y = canvas.height - 4 - (target - sy[0]) / Math.max(1, sy[1] - sy[0]) * (canvas.height - 8);
|
||||
context.fillStyle = `rgba(74,190,255,${.2 + .8 * Math.log1p(count) / Math.log1p(maximum)})`; context.fillRect(x - 2, y - 2, 5, 5);
|
||||
}
|
||||
context.fillStyle = "#c7d2df"; context.fillText(`${matrix.mode}; source ${matrix.source_extent}, target ${matrix.target_extent}`, 8, 14);
|
||||
}
|
||||
async function loadMappingRows(id) {
|
||||
const page = await api(`/api/aggregate-edges/${encodeURIComponent(id)}/mapping?limit=25&offset=${state.mappingOffset}`);
|
||||
$("mappingRows").innerHTML = `<h3>Raw mapping rows (${page.total ? `${page.offset + 1}–${Math.min(page.total, page.offset + page.items.length)}` : "0"} of ${page.total})</h3><table><tr><th>source</th><th>target</th><th>lanes</th><th>weight</th></tr>${page.items.map(row => `<tr><td>${html(row.source_node_id)}</td><td>${html(row.target_node_id)}</td><td>${html(row.source_lane)} → ${html(row.target_lane)}</td><td>${html(row.weight)}</td></tr>`).join("")}</table><div class="pager"><button id="prev" ${page.offset === 0 ? "disabled" : ""}>Previous</button><button id="next" ${page.offset + page.items.length >= page.total ? "disabled" : ""}>Next</button></div>`;
|
||||
$("prev").onclick = () => { state.mappingOffset = Math.max(0, state.mappingOffset - 25); loadMappingRows(id); };
|
||||
$("next").onclick = () => { state.mappingOffset += 25; loadMappingRows(id); };
|
||||
}
|
||||
function renderMotifs() {
|
||||
$("motifs").innerHTML = state.motifs.length ? state.motifs.map((motif, index) => `<button class="motif" data-index="${index}">${html(motif.motif_kind)}: ${html(motif.entry_aggregate_id)} → ${html(motif.exit_aggregate_id)} (${motif.node_count})</button>`).join("") : "No motifs detected.";
|
||||
document.querySelectorAll(".motif").forEach(button => button.onclick = () => selectMotif(state.motifs[button.dataset.index]));
|
||||
}
|
||||
function selectMotif(motif) {
|
||||
state.motif = motif; applyFilters();
|
||||
$("details").innerHTML = '<p class="muted">Structural SCC-condensed split/rejoin; rendered geometry is not evidence.</p>' + definition(motif, ["motif_kind", "entry_aggregate_id", "exit_aggregate_id", "internal_aggregate_ids", "branch_successor_aggregate_ids", "representative_paths", "node_count", "edge_count", "contains_cycle", "confidence"]);
|
||||
}
|
||||
async function loadSummary() {
|
||||
const summary = await api(`/api/reports/${encodeURIComponent($("report").value)}/summary`), report = summary.report;
|
||||
$("summary").innerHTML = definition(report, ["raw_node_count", "raw_edge_count", "operation_node_count", "operation_edge_count", "core_node_count", "core_edge_count", "scc_count", "motif_count"]) + `<strong>Mappings</strong><pre>${html(JSON.stringify(summary.mapping_classes, null, 2))}</pre><strong>Diagnostics</strong><pre>${html(summary.diagnostics.length ? JSON.stringify(summary.diagnostics, null, 2) : "none")}</pre>`;
|
||||
$("mapping").innerHTML = '<option value="">All mappings</option>' + Object.keys(summary.mapping_classes).map(kind => `<option>${html(kind)}</option>`).join("");
|
||||
}
|
||||
|
||||
$("report").onchange = () => { $("level").value = defaultViewForReport(selectedReport()); loadGraph(); };
|
||||
$("level").onchange = () => { localStorage.setItem("raptor-graph-view", $("level").value); loadGraph(); };
|
||||
$("metric").onchange = renderGraph;
|
||||
for (const id of ["search", "mapping", "tensor", "selfEdges"]) $(id).addEventListener(id === "selfEdges" ? "change" : "input", applyFilters);
|
||||
$("layout").onclick = runLayout; $("fit").onclick = () => state.renderer?.getCamera().animatedReset();
|
||||
$("expandAll").onclick = expandAllOperations; $("collapseAll").onclick = collapseAllOperations;
|
||||
loadReports().catch(error => status(error.message, true));
|
||||
@@ -0,0 +1,31 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
|
||||
<title>Raptor Graph Explorer</title>
|
||||
<!-- All browser dependency versions are pinned here and documented in README.md. -->
|
||||
<script src="https://cdnjs.cloudflare.com/ajax/libs/graphology/0.25.4/graphology.umd.min.js"></script>
|
||||
<script src="https://cdnjs.cloudflare.com/ajax/libs/sigma.js/2.4.0/sigma.min.js"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/elkjs@0.11.1/lib/elk.bundled.js"></script>
|
||||
<link rel="stylesheet" href="/styles.css">
|
||||
</head>
|
||||
<body>
|
||||
<header><h1>Raptor Graph Explorer</h1><span id="status" role="status">Loading…</span></header>
|
||||
<main>
|
||||
<aside class="controls" aria-label="Graph controls">
|
||||
<label>Report<select id="report"></select></label>
|
||||
<label>Aggregation/view<select id="level"><option value="operation">Operation</option><option value="raw">Raw instances</option><option value="core">Core</option><option value="node_kind">Node kind</option></select></label>
|
||||
<label>Edge metric<select id="metric"><option value="raw_edge_count">Count</option><option value="total_weight">Total weight</option></select></label>
|
||||
<label>Search<input id="search" type="search" placeholder="ID, op, core, SSA"></label>
|
||||
<label>Mapping<select id="mapping"><option value="">All mappings</option></select></label>
|
||||
<label>Tensor type<input id="tensor" type="search" placeholder="substring"></label>
|
||||
<label class="check"><input id="selfEdges" type="checkbox">Show self edges</label>
|
||||
<div class="buttons"><button id="expandAll">Expand all operations</button><button id="collapseAll" hidden>Collapse all</button><button id="layout">Rerun layout</button><button id="fit">Fit graph</button></div>
|
||||
<h2>Motifs</h2><div id="motifs" class="scroll">No motifs detected.</div>
|
||||
<h2>Report statistics and diagnostics</h2><div id="summary" class="scroll"></div>
|
||||
</aside>
|
||||
<section id="graph" aria-label="Interactive directed graph"></section>
|
||||
<aside class="details" aria-label="Selection details"><h2>Details</h2><div id="details">Select a node, edge, or motif.</div><canvas id="matrix" width="320" height="220" aria-label="Lane mapping heatmap" hidden></canvas><div id="mappingRows"></div></aside>
|
||||
</main>
|
||||
<script src="/app.js"></script>
|
||||
</body></html>
|
||||
@@ -0,0 +1 @@
|
||||
:root{color-scheme:dark;font:14px/1.4 system-ui,sans-serif;background:#10151d;color:#e7edf5}*{box-sizing:border-box}body{margin:0;overflow:hidden}header{height:52px;display:flex;align-items:center;gap:20px;padding:0 16px;border-bottom:1px solid #334155;background:#17202c}h1{font-size:18px;margin:0}h2{font-size:14px;margin:16px 0 6px;color:#9fc5f8}main{display:grid;grid-template-columns:250px minmax(300px,1fr) 350px;height:calc(100vh - 52px)}aside{padding:12px;overflow:auto;background:#17202c}.controls{border-right:1px solid #334155}.details{border-left:1px solid #334155}#graph{background:#0d131b;min-width:0}label{display:block;margin-bottom:10px;color:#b9c6d6}select,input,button{width:100%;margin-top:3px;padding:7px;border:1px solid #44546a;border-radius:4px;background:#0f1722;color:inherit}button{cursor:pointer;background:#263b55}button:hover,button:focus{background:#355274;outline:2px solid #70a5d8}.check{display:flex;gap:8px;align-items:center}.check input{width:auto;margin:0}.buttons{display:grid;grid-template-columns:1fr 1fr;gap:6px}.scroll{max-height:220px;overflow:auto;font-size:12px}.motif{text-align:left;margin:3px 0}dl{display:grid;grid-template-columns:max-content 1fr;gap:4px 9px;margin:4px 0}dt{color:#91a5bd}dd{margin:0;overflow-wrap:anywhere}pre{white-space:pre-wrap;overflow-wrap:anywhere;background:#0e1620;padding:8px;border-radius:4px;font-size:11px}.error{color:#ff9a9a}.muted{color:#94a3b8}.pager{display:flex;gap:6px}.pager button{width:auto}#matrix{width:100%;height:auto;background:#0b1119;border:1px solid #334155;margin-top:10px}table{width:100%;font-size:11px;border-collapse:collapse}th,td{padding:3px;border-bottom:1px solid #2d3b4c;text-align:left;overflow-wrap:anywhere}@media(max-width:900px){main{grid-template-columns:210px 1fr}.details{position:absolute;right:0;top:52px;width:320px;height:calc(100vh - 52px);box-shadow:-5px 0 20px #0008}}
|
||||
@@ -0,0 +1,35 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import shutil
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
TOOL_ROOT = Path(__file__).parents[1]
|
||||
sys.path.insert(0, str(TOOL_ROOT))
|
||||
|
||||
from raptor_graph_explorer.cli import build_output
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fixture_dir() -> Path:
|
||||
return Path(__file__).with_name("fixtures")
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def built_output(tmp_path: Path, fixture_dir: Path) -> Path:
|
||||
reports = tmp_path / "reports"
|
||||
reports.mkdir()
|
||||
for source in fixture_dir.glob("demo_*.csv"):
|
||||
shutil.copy2(source, reports / source.name)
|
||||
output = tmp_path / "built"
|
||||
build_output([reports], output)
|
||||
return output
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def regression_output(tmp_path: Path, fixture_dir: Path) -> Path:
|
||||
output = tmp_path / "regressions"
|
||||
build_output([fixture_dir / "regressions"], output)
|
||||
return output
|
||||
@@ -0,0 +1,10 @@
|
||||
Source,Target,Weight,Type,stage,source_lane,target_lane,channel_id,edge_extra
|
||||
gc:0,gcb:1:0,16,tensor<4xf32>,spatial1,,0,,a
|
||||
gc:0,gcb:1:1,16,tensor<4xf32>,spatial1,,1,,a
|
||||
gcb:1:0,gcb:2:0,16,tensor<4xf32>,spatial1,0,0,,b
|
||||
gcb:1:1,gcb:2:1,16,tensor<4xf32>,spatial1,1,1,,b
|
||||
gcb:2:0,gcb:3:0,8,tensor<2xf32>,spatial1,,,7,c
|
||||
gcb:2:1,gcb:3:1,8,tensor<2xf32>,spatial1,,,8,c
|
||||
gcb:3:0,gc:4,8,tensor<2xf32>,spatial1,0,,,d
|
||||
gcb:3:1,gc:4,8,tensor<2xf32>,spatial1,1,,,d
|
||||
gc:4,mystery:5,,tensor<1xf32>,spatial1,,,,e
|
||||
|
@@ -0,0 +1,10 @@
|
||||
Id,op_id,lane,core,ssa_name,node_extra
|
||||
gc:0,0,,,%0,root
|
||||
gcb:1:0,1,0,,%1#0,lane
|
||||
gcb:1:1,1,1,,%1#1,lane
|
||||
gcb:2:0,2,0,,%2#0,lane
|
||||
gcb:2:1,2,1,,%2#1,lane
|
||||
gcb:3:0,3,0,,%3#0,lane
|
||||
gcb:3:1,3,1,,%3#1,lane
|
||||
gc:4,4,,,%4,sink
|
||||
mystery:5,5,,,%5,unknown
|
||||
|
@@ -0,0 +1,3 @@
|
||||
Source,Target,Weight,Type,stage,source_lane,target_lane,channel_id
|
||||
scb:0:0,sc:1,4,tensor<1xf32>,spatial3,0,,21
|
||||
scb:0:1,sc:1,4,tensor<1xf32>,spatial3,1,,22
|
||||
|
@@ -0,0 +1,4 @@
|
||||
Id,op_id,lane,core,ssa_name
|
||||
scb:0:0,0,0,10,%0
|
||||
scb:0:1,0,1,11,%0
|
||||
sc:1,1,,12,%1
|
||||
|
@@ -0,0 +1 @@
|
||||
Source,Target,Weight,Type,stage,source_lane,target_lane,channel_id
|
||||
|
@@ -0,0 +1,4 @@
|
||||
Id,op_id,lane,core,ssa_name
|
||||
gc:0,0,,,%0
|
||||
gcb:1:0,1,0,,%1
|
||||
gcb:1:1,1,1,,%1
|
||||
|
+65
@@ -0,0 +1,65 @@
|
||||
Source,Target,Weight,Type,stage,source_lane,target_lane,channel_id
|
||||
gcb:1:0,gcb:2:0,8192,tensor<16x1x8x1x16xf32>,spatial1,0,0,
|
||||
gcb:1:1,gcb:2:1,8192,tensor<16x1x8x1x16xf32>,spatial1,1,1,
|
||||
gcb:1:2,gcb:2:2,8192,tensor<16x1x8x1x16xf32>,spatial1,2,2,
|
||||
gcb:1:3,gcb:2:3,8192,tensor<16x1x8x1x16xf32>,spatial1,3,3,
|
||||
gcb:1:4,gcb:2:4,8192,tensor<16x1x8x1x16xf32>,spatial1,4,4,
|
||||
gcb:1:5,gcb:2:5,8192,tensor<16x1x8x1x16xf32>,spatial1,5,5,
|
||||
gcb:1:6,gcb:2:6,8192,tensor<16x1x8x1x16xf32>,spatial1,6,6,
|
||||
gcb:1:7,gcb:2:7,8192,tensor<16x1x8x1x16xf32>,spatial1,7,7,
|
||||
gcb:1:8,gcb:2:8,8192,tensor<16x1x8x1x16xf32>,spatial1,8,8,
|
||||
gcb:1:9,gcb:2:9,8192,tensor<16x1x8x1x16xf32>,spatial1,9,9,
|
||||
gcb:1:10,gcb:2:10,8192,tensor<16x1x8x1x16xf32>,spatial1,10,10,
|
||||
gcb:1:11,gcb:2:11,8192,tensor<16x1x8x1x16xf32>,spatial1,11,11,
|
||||
gcb:1:12,gcb:2:12,8192,tensor<16x1x8x1x16xf32>,spatial1,12,12,
|
||||
gcb:1:13,gcb:2:13,8192,tensor<16x1x8x1x16xf32>,spatial1,13,13,
|
||||
gcb:1:14,gcb:2:14,8192,tensor<16x1x8x1x16xf32>,spatial1,14,14,
|
||||
gcb:1:15,gcb:2:15,8192,tensor<16x1x8x1x16xf32>,spatial1,15,15,
|
||||
gcb:2:0,gcb:3:0,8192,tensor<16x1x8x1x16xf32>,spatial1,0,0,
|
||||
gcb:2:1,gcb:3:1,8192,tensor<16x1x8x1x16xf32>,spatial1,1,1,
|
||||
gcb:2:2,gcb:3:2,8192,tensor<16x1x8x1x16xf32>,spatial1,2,2,
|
||||
gcb:2:3,gcb:3:3,8192,tensor<16x1x8x1x16xf32>,spatial1,3,3,
|
||||
gcb:2:4,gcb:3:4,8192,tensor<16x1x8x1x16xf32>,spatial1,4,4,
|
||||
gcb:2:5,gcb:3:5,8192,tensor<16x1x8x1x16xf32>,spatial1,5,5,
|
||||
gcb:2:6,gcb:3:6,8192,tensor<16x1x8x1x16xf32>,spatial1,6,6,
|
||||
gcb:2:7,gcb:3:7,8192,tensor<16x1x8x1x16xf32>,spatial1,7,7,
|
||||
gcb:2:8,gcb:3:8,8192,tensor<16x1x8x1x16xf32>,spatial1,8,8,
|
||||
gcb:2:9,gcb:3:9,8192,tensor<16x1x8x1x16xf32>,spatial1,9,9,
|
||||
gcb:2:10,gcb:3:10,8192,tensor<16x1x8x1x16xf32>,spatial1,10,10,
|
||||
gcb:2:11,gcb:3:11,8192,tensor<16x1x8x1x16xf32>,spatial1,11,11,
|
||||
gcb:2:12,gcb:3:12,8192,tensor<16x1x8x1x16xf32>,spatial1,12,12,
|
||||
gcb:2:13,gcb:3:13,8192,tensor<16x1x8x1x16xf32>,spatial1,13,13,
|
||||
gcb:2:14,gcb:3:14,8192,tensor<16x1x8x1x16xf32>,spatial1,14,14,
|
||||
gcb:2:15,gcb:3:15,8192,tensor<16x1x8x1x16xf32>,spatial1,15,15,
|
||||
gcb:3:0,gcb:4:0,8192,tensor<16x1x8x1x16xf32>,spatial1,0,0,
|
||||
gcb:3:1,gcb:4:1,8192,tensor<16x1x8x1x16xf32>,spatial1,1,1,
|
||||
gcb:3:2,gcb:4:2,8192,tensor<16x1x8x1x16xf32>,spatial1,2,2,
|
||||
gcb:3:3,gcb:4:3,8192,tensor<16x1x8x1x16xf32>,spatial1,3,3,
|
||||
gcb:3:4,gcb:4:4,8192,tensor<16x1x8x1x16xf32>,spatial1,4,4,
|
||||
gcb:3:5,gcb:4:5,8192,tensor<16x1x8x1x16xf32>,spatial1,5,5,
|
||||
gcb:3:6,gcb:4:6,8192,tensor<16x1x8x1x16xf32>,spatial1,6,6,
|
||||
gcb:3:7,gcb:4:7,8192,tensor<16x1x8x1x16xf32>,spatial1,7,7,
|
||||
gcb:3:8,gcb:4:8,8192,tensor<16x1x8x1x16xf32>,spatial1,8,8,
|
||||
gcb:3:9,gcb:4:9,8192,tensor<16x1x8x1x16xf32>,spatial1,9,9,
|
||||
gcb:3:10,gcb:4:10,8192,tensor<16x1x8x1x16xf32>,spatial1,10,10,
|
||||
gcb:3:11,gcb:4:11,8192,tensor<16x1x8x1x16xf32>,spatial1,11,11,
|
||||
gcb:3:12,gcb:4:12,8192,tensor<16x1x8x1x16xf32>,spatial1,12,12,
|
||||
gcb:3:13,gcb:4:13,8192,tensor<16x1x8x1x16xf32>,spatial1,13,13,
|
||||
gcb:3:14,gcb:4:14,8192,tensor<16x1x8x1x16xf32>,spatial1,14,14,
|
||||
gcb:3:15,gcb:4:15,8192,tensor<16x1x8x1x16xf32>,spatial1,15,15,
|
||||
gc:0,gcb:1:0,3888,tensor<1x3x18x18xf32>,spatial1,,0,
|
||||
gc:0,gcb:1:1,3888,tensor<1x3x18x18xf32>,spatial1,,1,
|
||||
gc:0,gcb:1:2,3888,tensor<1x3x18x18xf32>,spatial1,,2,
|
||||
gc:0,gcb:1:3,3888,tensor<1x3x18x18xf32>,spatial1,,3,
|
||||
gc:0,gcb:1:4,3888,tensor<1x3x18x18xf32>,spatial1,,4,
|
||||
gc:0,gcb:1:5,3888,tensor<1x3x18x18xf32>,spatial1,,5,
|
||||
gc:0,gcb:1:6,3888,tensor<1x3x18x18xf32>,spatial1,,6,
|
||||
gc:0,gcb:1:7,3888,tensor<1x3x18x18xf32>,spatial1,,7,
|
||||
gc:0,gcb:1:8,3888,tensor<1x3x18x18xf32>,spatial1,,8,
|
||||
gc:0,gcb:1:9,3888,tensor<1x3x18x18xf32>,spatial1,,9,
|
||||
gc:0,gcb:1:10,3888,tensor<1x3x18x18xf32>,spatial1,,10,
|
||||
gc:0,gcb:1:11,3888,tensor<1x3x18x18xf32>,spatial1,,11,
|
||||
gc:0,gcb:1:12,3888,tensor<1x3x18x18xf32>,spatial1,,12,
|
||||
gc:0,gcb:1:13,3888,tensor<1x3x18x18xf32>,spatial1,,13,
|
||||
gc:0,gcb:1:14,3888,tensor<1x3x18x18xf32>,spatial1,,14,
|
||||
gc:0,gcb:1:15,3888,tensor<1x3x18x18xf32>,spatial1,,15,
|
||||
|
+66
@@ -0,0 +1,66 @@
|
||||
Id,op_id,lane,core,ssa_name
|
||||
gc:0,0,,,%0
|
||||
gcb:1:0,1,0,,%1
|
||||
gcb:1:1,1,1,,%1
|
||||
gcb:1:2,1,2,,%1
|
||||
gcb:1:3,1,3,,%1
|
||||
gcb:1:4,1,4,,%1
|
||||
gcb:1:5,1,5,,%1
|
||||
gcb:1:6,1,6,,%1
|
||||
gcb:1:7,1,7,,%1
|
||||
gcb:1:8,1,8,,%1
|
||||
gcb:1:9,1,9,,%1
|
||||
gcb:1:10,1,10,,%1
|
||||
gcb:1:11,1,11,,%1
|
||||
gcb:1:12,1,12,,%1
|
||||
gcb:1:13,1,13,,%1
|
||||
gcb:1:14,1,14,,%1
|
||||
gcb:1:15,1,15,,%1
|
||||
gcb:2:0,2,0,,%2
|
||||
gcb:2:1,2,1,,%2
|
||||
gcb:2:2,2,2,,%2
|
||||
gcb:2:3,2,3,,%2
|
||||
gcb:2:4,2,4,,%2
|
||||
gcb:2:5,2,5,,%2
|
||||
gcb:2:6,2,6,,%2
|
||||
gcb:2:7,2,7,,%2
|
||||
gcb:2:8,2,8,,%2
|
||||
gcb:2:9,2,9,,%2
|
||||
gcb:2:10,2,10,,%2
|
||||
gcb:2:11,2,11,,%2
|
||||
gcb:2:12,2,12,,%2
|
||||
gcb:2:13,2,13,,%2
|
||||
gcb:2:14,2,14,,%2
|
||||
gcb:2:15,2,15,,%2
|
||||
gcb:3:0,3,0,,%3
|
||||
gcb:3:1,3,1,,%3
|
||||
gcb:3:2,3,2,,%3
|
||||
gcb:3:3,3,3,,%3
|
||||
gcb:3:4,3,4,,%3
|
||||
gcb:3:5,3,5,,%3
|
||||
gcb:3:6,3,6,,%3
|
||||
gcb:3:7,3,7,,%3
|
||||
gcb:3:8,3,8,,%3
|
||||
gcb:3:9,3,9,,%3
|
||||
gcb:3:10,3,10,,%3
|
||||
gcb:3:11,3,11,,%3
|
||||
gcb:3:12,3,12,,%3
|
||||
gcb:3:13,3,13,,%3
|
||||
gcb:3:14,3,14,,%3
|
||||
gcb:3:15,3,15,,%3
|
||||
gcb:4:0,4,0,,%4
|
||||
gcb:4:1,4,1,,%4
|
||||
gcb:4:2,4,2,,%4
|
||||
gcb:4:3,4,3,,%4
|
||||
gcb:4:4,4,4,,%4
|
||||
gcb:4:5,4,5,,%4
|
||||
gcb:4:6,4,6,,%4
|
||||
gcb:4:7,4,7,,%4
|
||||
gcb:4:8,4,8,,%4
|
||||
gcb:4:9,4,9,,%4
|
||||
gcb:4:10,4,10,,%4
|
||||
gcb:4:11,4,11,,%4
|
||||
gcb:4:12,4,12,,%4
|
||||
gcb:4:13,4,13,,%4
|
||||
gcb:4:14,4,14,,%4
|
||||
gcb:4:15,4,15,,%4
|
||||
|
+2
@@ -0,0 +1,2 @@
|
||||
Source,Target,Weight,Type,stage,source_lane,target_lane,channel_id
|
||||
scb:1:5,sc:0,512,tensor<1x8x1x16xf32>,spatial3,5,,0
|
||||
|
+17
@@ -0,0 +1,17 @@
|
||||
Id,op_id,lane,core,ssa_name
|
||||
sc:0,0,,500,%0#0;%0#1;%0#2;%0#3;%0#4
|
||||
scb:1:0,1,0,380,%1#0;%1#1;%1#2;%1#3
|
||||
scb:1:1,1,1,419,%1#0;%1#1;%1#2;%1#3
|
||||
scb:1:2,1,2,420,%1#0;%1#1;%1#2;%1#3
|
||||
scb:1:3,1,3,421,%1#0;%1#1;%1#2;%1#3
|
||||
scb:1:4,1,4,459,%1#0;%1#1;%1#2;%1#3
|
||||
scb:1:5,1,5,460,%1#0;%1#1;%1#2;%1#3
|
||||
scb:1:6,1,6,461,%1#0;%1#1;%1#2;%1#3
|
||||
scb:1:7,1,7,498,%1#0;%1#1;%1#2;%1#3
|
||||
scb:1:8,1,8,499,%1#0;%1#1;%1#2;%1#3
|
||||
scb:1:9,1,9,501,%1#0;%1#1;%1#2;%1#3
|
||||
scb:1:10,1,10,502,%1#0;%1#1;%1#2;%1#3
|
||||
scb:1:11,1,11,539,%1#0;%1#1;%1#2;%1#3
|
||||
scb:1:12,1,12,540,%1#0;%1#1;%1#2;%1#3
|
||||
scb:1:13,1,13,541,%1#0;%1#1;%1#2;%1#3
|
||||
scb:1:14,1,14,580,%1#0;%1#1;%1#2;%1#3
|
||||
|
@@ -0,0 +1,57 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import os
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from raptor_graph_explorer.cli import build_output
|
||||
|
||||
|
||||
def test_operation_core_and_node_kind_aggregation(built_output: Path):
|
||||
db = sqlite3.connect(built_output / "graph.sqlite")
|
||||
report = db.execute("SELECT * FROM reports WHERE report_id='demo_graph'").fetchone()
|
||||
assert report[2:8] == (9, 9, 6, 5, 1, 1)
|
||||
op = db.execute("SELECT instance_count,lane_count,minimum_lane,maximum_lane FROM aggregate_nodes WHERE aggregate_id='operation:demo_graph:1'").fetchone()
|
||||
assert op == (2, 2, 0, 1)
|
||||
assert db.execute("SELECT COUNT(*) FROM aggregate_nodes WHERE report_id='demo_graph' AND level='node_kind'").fetchone()[0] == 3
|
||||
core = db.execute("SELECT group_key FROM aggregate_nodes WHERE report_id='demo_scheduled' AND level='core' ORDER BY group_key").fetchall()
|
||||
assert core == [("10",), ("11",), ("12",)]
|
||||
db.close()
|
||||
|
||||
|
||||
def test_manifest_generation(built_output: Path):
|
||||
import json
|
||||
manifest = json.loads((built_output / "manifest.json").read_text())
|
||||
assert manifest["schema_version"] == 1
|
||||
assert [report["report_id"] for report in manifest["reports"]] == ["demo_graph", "demo_scheduled"]
|
||||
assert all("input" not in report for report in manifest["reports"])
|
||||
|
||||
|
||||
def _generated_report(directory: Path, operations=10, lanes=50, edges=500):
|
||||
directory.mkdir()
|
||||
with (directory / "generated.nodes.csv").open("w", newline="") as stream:
|
||||
writer=csv.writer(stream);writer.writerow(["Id","op_id","lane","core","ssa_name"])
|
||||
for op in range(operations):
|
||||
for lane in range(lanes):writer.writerow([f"gcb:{op}:{lane}",op,lane,"",f"%{op}"])
|
||||
with (directory / "generated.edges.csv").open("w", newline="") as stream:
|
||||
writer=csv.writer(stream);writer.writerow(["Source","Target","Weight","Type","stage","source_lane","target_lane","channel_id"])
|
||||
for index in range(edges):
|
||||
op=index%(operations-1);lane=index%lanes;writer.writerow([f"gcb:{op}:{lane}",f"gcb:{op+1}:{lane}",4,"tensor<1xf32>","generated",lane,lane,""])
|
||||
|
||||
|
||||
def test_generated_performance_shape(tmp_path: Path):
|
||||
_generated_report(tmp_path / "reports")
|
||||
manifest=build_output([tmp_path / "reports"],tmp_path / "out")
|
||||
assert manifest["reports"][0]["raw_node_count"] == 500
|
||||
assert manifest["reports"][0]["raw_edge_count"] == 500
|
||||
|
||||
|
||||
@pytest.mark.slow
|
||||
@pytest.mark.skipif(not os.getenv("RAPTOR_GRAPH_SLOW"), reason="set RAPTOR_GRAPH_SLOW=1")
|
||||
def test_100k_edge_smoke(tmp_path: Path):
|
||||
_generated_report(tmp_path / "reports",operations=100,lanes=1000,edges=100_000)
|
||||
manifest=build_output([tmp_path / "reports"],tmp_path / "out")
|
||||
assert manifest["reports"][0]["raw_edge_count"] == 100_000
|
||||
@@ -0,0 +1,45 @@
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
import pytest
|
||||
|
||||
from raptor_graph_explorer.api import create_app
|
||||
|
||||
|
||||
def test_manifest_graph_motifs_and_pagination(built_output: Path):
|
||||
client=TestClient(create_app(built_output,max_page_size=2))
|
||||
assert "Raptor Graph Explorer" in client.get("/").text
|
||||
assert "runLayout" in client.get("/app.js").text
|
||||
assert client.get("/api/manifest").status_code==200
|
||||
assert len(client.get("/api/reports").json())==2
|
||||
graph=client.get("/api/reports/demo_graph/graph/operation").json()
|
||||
assert len(graph["nodes"])==6 and len(graph["edges"])==5
|
||||
assert client.get("/api/reports/demo_graph/motifs").status_code==200
|
||||
page=client.get("/api/reports/demo_graph/raw-nodes?limit=2").json()
|
||||
assert page["total"]==9 and len(page["items"])==2
|
||||
assert client.get("/api/reports/demo_graph/raw-nodes?limit=3").status_code==400
|
||||
|
||||
|
||||
def test_edge_mapping_matrix_and_binning(built_output: Path):
|
||||
client=TestClient(create_app(built_output,exact_matrix_points=1))
|
||||
edge="operation:demo_graph:1->2"
|
||||
detail=client.get(f"/api/aggregate-edges/{edge}").json()
|
||||
assert detail["mapping_kind"]=="identity"
|
||||
assert client.get(f"/api/aggregate-edges/{edge}/mapping").json()["total"]==2
|
||||
matrix=client.get(f"/api/aggregate-edges/{edge}/matrix").json()
|
||||
assert matrix["mode"]=="binned" and matrix["dimensions"]==[64,64]
|
||||
|
||||
|
||||
def test_expansion_safety_cap(built_output: Path):
|
||||
client=TestClient(create_app(built_output,expansion_cap=2))
|
||||
selected="operation:demo_graph:1"
|
||||
response=client.get(f"/api/reports/demo_graph/subgraph?aggregate_id={selected}&expand_lanes=true")
|
||||
assert response.status_code==400 and "safety cap" in response.json()["detail"]
|
||||
|
||||
|
||||
def test_unsupported_database_schema_is_rejected(built_output: Path):
|
||||
import sqlite3
|
||||
db=sqlite3.connect(built_output/"graph.sqlite")
|
||||
db.execute("UPDATE metadata SET value='999' WHERE key='schema_version'");db.commit();db.close()
|
||||
with pytest.raises(ValueError,match="unsupported database schema"):
|
||||
create_app(built_output)
|
||||
@@ -0,0 +1,13 @@
|
||||
from pathlib import Path
|
||||
|
||||
from raptor_graph_explorer.cli import main
|
||||
|
||||
|
||||
def test_build_inspect_and_exit_codes(tmp_path: Path, fixture_dir: Path, capsys):
|
||||
output=tmp_path/"output"
|
||||
assert main(["build",str(fixture_dir),"--output",str(output)])==0
|
||||
assert main(["inspect",str(output)])==0
|
||||
captured=capsys.readouterr().out
|
||||
assert "demo_graph" in captured and "mappings:" in captured
|
||||
assert main(["build",str(fixture_dir),"--output",str(output)])==2
|
||||
assert main(["build",str(tmp_path/"missing"),"--output",str(tmp_path/"bad")])==2
|
||||
@@ -0,0 +1,36 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import zipfile
|
||||
from pathlib import Path
|
||||
|
||||
from raptor_graph_explorer.discovery import discover_reports
|
||||
|
||||
|
||||
def test_discovers_complete_pairs_and_reports_incomplete(tmp_path: Path, fixture_dir: Path):
|
||||
nested = tmp_path / "nested"
|
||||
nested.mkdir()
|
||||
for path in fixture_dir.glob("demo_graph.*.csv"):
|
||||
(nested / path.name).write_bytes(path.read_bytes())
|
||||
(tmp_path / "lonely.nodes.csv").write_text("Id,op_id,lane,core,ssa_name\n")
|
||||
pairs, diagnostics = discover_reports([tmp_path], tmp_path / "extract")
|
||||
assert [pair.report_id for pair in pairs] == ["demo_graph"]
|
||||
assert any(item.code == "incomplete_pair" for item in diagnostics.items)
|
||||
|
||||
|
||||
def test_zip_and_explicit_repeated_inputs(tmp_path: Path, fixture_dir: Path):
|
||||
archive = tmp_path / "reports.zip"
|
||||
with zipfile.ZipFile(archive, "w") as zipped:
|
||||
for path in fixture_dir.glob("demo_graph.*.csv"):
|
||||
zipped.write(path, f"sub/{path.name}")
|
||||
pairs, _ = discover_reports([archive], tmp_path / "extract")
|
||||
assert len(pairs) == 1 and pairs[0].report_id == "demo_graph"
|
||||
|
||||
|
||||
def test_zip_traversal_is_rejected(tmp_path: Path):
|
||||
archive = tmp_path / "bad.zip"
|
||||
with zipfile.ZipFile(archive, "w") as zipped:
|
||||
zipped.writestr("../escape.nodes.csv", "bad")
|
||||
pairs, diagnostics = discover_reports([archive], tmp_path / "extract")
|
||||
assert not pairs
|
||||
assert diagnostics.items[0].code == "invalid_zip"
|
||||
assert not (tmp_path / "escape.nodes.csv").exists()
|
||||
@@ -0,0 +1,58 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import sqlite3
|
||||
from pathlib import Path
|
||||
|
||||
from raptor_graph_explorer.cli import build_output
|
||||
|
||||
|
||||
def test_normalization_unknown_columns_prefix_and_lane_fallback(built_output: Path):
|
||||
db = sqlite3.connect(built_output / "graph.sqlite")
|
||||
db.row_factory = sqlite3.Row
|
||||
unknown = db.execute("SELECT * FROM raw_nodes WHERE node_id='mystery:5'").fetchone()
|
||||
assert unknown["node_kind"] == "unknown:mystery"
|
||||
assert json.loads(unknown["attributes_json"]) == {"node_extra": "unknown"}
|
||||
edge = db.execute("SELECT * FROM raw_edges WHERE source_node_id='gcb:2:0'").fetchone()
|
||||
assert (edge["source_lane"], edge["target_lane"]) == (0, 0)
|
||||
assert json.loads(edge["attributes_json"])["edge_extra"] == "c"
|
||||
db.close()
|
||||
|
||||
|
||||
def test_malformed_rows_unknown_endpoint_and_strict_mode(tmp_path: Path):
|
||||
reports = tmp_path / "reports"; reports.mkdir()
|
||||
(reports / "bad.nodes.csv").write_text(
|
||||
"Id,op_id,lane,core,ssa_name\ngc:0,0,,,%0\ngc:1,nope,,,%1\n")
|
||||
(reports / "bad.edges.csv").write_text(
|
||||
"Source,Target,Weight,Type,stage,source_lane,target_lane,channel_id\n"
|
||||
"gc:0,gc:missing,1,tensor<1xf32>,bad,,,\n")
|
||||
manifest = build_output([reports], tmp_path / "out")
|
||||
assert manifest["reports"][0]["raw_node_count"] == 1
|
||||
assert manifest["reports"][0]["raw_edge_count"] == 0
|
||||
assert manifest["diagnostic_count"] == 2
|
||||
try:
|
||||
build_output([reports], tmp_path / "strict", strict=True)
|
||||
except RuntimeError as exc:
|
||||
assert "strict build" in str(exc)
|
||||
else:
|
||||
raise AssertionError("strict mode accepted invalid rows")
|
||||
|
||||
|
||||
def test_missing_schema_column_is_diagnosed(tmp_path: Path):
|
||||
reports = tmp_path / "reports"; reports.mkdir()
|
||||
(reports / "x.nodes.csv").write_text("Id,op_id,lane,core\ngc:0,0,,,\n")
|
||||
(reports / "x.edges.csv").write_text("Source,Target,Weight,Type,stage,source_lane,target_lane,channel_id\n")
|
||||
manifest = build_output([reports], tmp_path / "out")
|
||||
assert manifest["reports"][0]["raw_node_count"] == 0
|
||||
assert manifest["diagnostic_count"] == 2
|
||||
|
||||
|
||||
def test_explicit_edge_lane_precedes_endpoint_lane(tmp_path: Path):
|
||||
reports=tmp_path/"reports";reports.mkdir()
|
||||
(reports/"x.nodes.csv").write_text("Id,op_id,lane,core,ssa_name\ngcb:0:0,0,0,,%0\ngcb:1:0,1,0,,%1\n")
|
||||
(reports/"x.edges.csv").write_text("Source,Target,Weight,Type,stage,source_lane,target_lane,channel_id\ngcb:0:0,gcb:1:0,1,tensor<1xf32>,x,9,8,\n")
|
||||
output=tmp_path/"out";build_output([reports],output)
|
||||
db=sqlite3.connect(output/"graph.sqlite")
|
||||
assert db.execute("SELECT source_lane,target_lane FROM raw_edges").fetchone()==(9,8)
|
||||
assert db.execute("SELECT occurrence_count FROM diagnostics WHERE code='contradictory_lane'").fetchone()==(1,)
|
||||
db.close()
|
||||
@@ -0,0 +1,22 @@
|
||||
import pytest
|
||||
|
||||
from raptor_graph_explorer.mappings import classify_lane_pairs
|
||||
|
||||
|
||||
@pytest.mark.parametrize(("pairs","kind"),[
|
||||
([(None,None)],"scalar_to_scalar"), ([(None,0),(None,1)],"scalar_to_batch"),
|
||||
([(0,None),(1,None)],"batch_to_scalar"), ([(0,0),(1,1)],"identity"),
|
||||
([(0,2),(1,3)],"constant_offset"), ([(0,1),(1,0)],"permutation"),
|
||||
([(3,0),(3,1)],"broadcast"), ([(0,3),(1,3)],"fan_in"),
|
||||
([(0,0),(0,1),(1,1),(1,2)],"stencil"),
|
||||
([(0,0),(0,2),(1,1)],"irregular"), ([(None,1),(2,3)],"unknown"),
|
||||
])
|
||||
def test_all_mapping_classes(pairs,kind):
|
||||
assert classify_lane_pairs(pairs)[0] == kind
|
||||
|
||||
|
||||
def test_mapping_metadata_is_exact():
|
||||
assert classify_lane_pairs([(0,2),(1,3)])[1] == {"offset":2}
|
||||
kind,metadata=classify_lane_pairs([(0,0),(0,1),(1,1),(1,2)])
|
||||
assert kind == "stencil" and metadata == {"offsets":[0,1]}
|
||||
assert classify_lane_pairs([(None,2),(None,3)])[1]["target"]["contiguous"] is True
|
||||
@@ -0,0 +1,51 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import networkx as nx
|
||||
|
||||
from raptor_graph_explorer.motifs import detect_motifs
|
||||
|
||||
|
||||
def graph(edges):
|
||||
result=nx.DiGraph();result.add_edges_from(edges);return result
|
||||
|
||||
|
||||
def test_exact_diamond():
|
||||
motifs,scc=detect_motifs(graph([("s","a"),("s","b"),("a","t"),("b","t")]))
|
||||
assert scc == 4 and len(motifs)==1 and motifs[0].kind=="exact_diamond"
|
||||
|
||||
|
||||
def test_longer_rhomboid():
|
||||
motifs,_=detect_motifs(graph([("s","a"),("a","c"),("c","t"),("s","b"),("b","d"),("d","t")]))
|
||||
assert len(motifs)==1 and motifs[0].kind=="rhomboid"
|
||||
|
||||
|
||||
def test_three_branch_split_rejoin():
|
||||
motifs,_=detect_motifs(graph([("s","a"),("s","b"),("s","c"),("a","t"),("b","t"),("c","t")]))
|
||||
assert len(motifs)==1 and len(motifs[0].branches)==3
|
||||
|
||||
|
||||
def test_nested_motifs_are_preserved():
|
||||
edges=[("s","a"),("s","b"),("a","c"),("a","d"),("c","e"),("d","e"),("e","t"),("b","t")]
|
||||
motifs,_=detect_motifs(graph(edges))
|
||||
assert {(motif.entry,motif.exit) for motif in motifs} == {("a","e"),("s","t")}
|
||||
|
||||
|
||||
def test_disconnected_graph():
|
||||
motifs,scc=detect_motifs(graph([("s","a"),("s","b"),("a","t"),("b","t"),("x","y")]))
|
||||
assert len(motifs)==1 and scc==6
|
||||
|
||||
|
||||
def test_cycle_is_condensed():
|
||||
edges=[("s","a"),("s","b"),("a","c"),("c","a"),("c","t"),("b","t")]
|
||||
motifs,_=detect_motifs(graph(edges))
|
||||
assert len(motifs)==1 and motifs[0].contains_cycle
|
||||
|
||||
|
||||
def test_external_entrance_rejected():
|
||||
edges=[("s","a"),("s","b"),("a","t"),("b","t"),("x","a")]
|
||||
assert detect_motifs(graph(edges))[0] == []
|
||||
|
||||
|
||||
def test_external_exit_rejected():
|
||||
edges=[("s","a"),("s","b"),("a","t"),("b","t"),("a","x")]
|
||||
assert detect_motifs(graph(edges))[0] == []
|
||||
@@ -0,0 +1,125 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi.testclient import TestClient
|
||||
import pytest
|
||||
|
||||
from raptor_graph_explorer.api import create_app
|
||||
from raptor_graph_explorer.database import connect_readonly
|
||||
from raptor_graph_explorer.projection import DisplayGraphTooLarge, display_graph
|
||||
|
||||
|
||||
def project(output: Path, report_id: str, expanded=(), *, view="operation", cap=5_000):
|
||||
db = connect_readonly(output / "graph.sqlite")
|
||||
try:
|
||||
return display_graph(db, report_id, view=view, expanded_operation_ids=expanded, cap=cap)
|
||||
finally:
|
||||
db.close()
|
||||
|
||||
|
||||
def test_spatial1_baseline_and_one_operation_expansion(regression_output: Path):
|
||||
db = connect_readonly(regression_output / "graph.sqlite")
|
||||
source_counts = db.execute("""
|
||||
SELECT operation_node_count,operation_edge_count,raw_node_count,raw_edge_count
|
||||
FROM reports WHERE report_id='spatial1_graph'
|
||||
""").fetchone()
|
||||
db.close()
|
||||
assert tuple(source_counts) == (5, 4, 65, 64)
|
||||
baseline = project(regression_output, "spatial1_graph")
|
||||
assert baseline["counts"] == {
|
||||
"nodes": 5, "edges": 4, "isolated_nodes": 0,
|
||||
"raw_nodes": 0, "aggregate_nodes": 5, "raw_edges": 0, "aggregate_edges": 4,
|
||||
}
|
||||
expanded = project(regression_output, "spatial1_graph", [1])
|
||||
node_ids = {node["display_id"] for node in expanded["nodes"]}
|
||||
assert "operation:spatial1_graph:1" not in node_ids
|
||||
assert {f"gcb:1:{lane}" for lane in range(16)} <= node_ids
|
||||
assert {f"operation:spatial1_graph:{op_id}" for op_id in (0, 2, 3, 4)} <= node_ids
|
||||
aggregate_pairs = {(edge["source_op_id"], edge["target_op_id"])
|
||||
for edge in expanded["edges"] if edge["representation"] == "aggregate"}
|
||||
assert aggregate_pairs == {(2, 3), (3, 4)}
|
||||
assert {(edge["source_op_id"], edge["target_op_id"])
|
||||
for edge in expanded["edges"] if edge["representation"] == "raw"} == {(0, 1), (1, 2)}
|
||||
assert len({edge["display_id"] for edge in expanded["edges"]}) == len(expanded["edges"])
|
||||
|
||||
|
||||
def test_expansion_order_is_irrelevant(regression_output: Path):
|
||||
first = project(regression_output, "spatial1_graph", [1, 2])
|
||||
second = project(regression_output, "spatial1_graph", [2, 1])
|
||||
assert first["nodes"] == second["nodes"]
|
||||
assert first["edges"] == second["edges"]
|
||||
|
||||
|
||||
def test_expand_all_spatial1_is_exact_raw_graph(regression_output: Path):
|
||||
graph = project(regression_output, "spatial1_graph", range(5))
|
||||
assert graph["counts"] == {
|
||||
"nodes": 65, "edges": 64, "isolated_nodes": 0,
|
||||
"raw_nodes": 65, "aggregate_nodes": 0, "raw_edges": 64, "aggregate_edges": 0,
|
||||
}
|
||||
node_ids = {node["node_id"] for node in graph["nodes"]}
|
||||
assert all(edge["source"] in node_ids and edge["target"] in node_ids for edge in graph["edges"])
|
||||
identity_edges = [edge for edge in graph["edges"] if edge["source_op_id"] >= 1]
|
||||
assert identity_edges and all(edge["source_lane"] == edge["target_lane"] for edge in identity_edges)
|
||||
|
||||
|
||||
def test_collapse_reconstructs_every_boundary(regression_output: Path):
|
||||
graph = project(regression_output, "spatial1_graph", [0, 1, 3, 4])
|
||||
node_ids = {node["display_id"] for node in graph["nodes"]}
|
||||
assert "operation:spatial1_graph:2" in node_ids
|
||||
assert not any(node.get("op_id") == 2 and node["representation"] == "raw" for node in graph["nodes"])
|
||||
assert all(edge["source"] in node_ids and edge["target"] in node_ids for edge in graph["edges"])
|
||||
boundaries = {(edge["source_op_id"], edge["target_op_id"]): (edge["source"], edge["target"])
|
||||
for edge in graph["edges"]
|
||||
if edge["source_lane"] == 0 or edge["source_lane"] is None and edge["target_lane"] == 0}
|
||||
assert boundaries[(1, 2)] == ("gcb:1:0", "operation:spatial1_graph:2")
|
||||
assert boundaries[(2, 3)] == ("operation:spatial1_graph:2", "gcb:3:0")
|
||||
assert boundaries[(0, 1)] == ("gc:0", "gcb:1:0")
|
||||
assert boundaries[(3, 4)] == ("gcb:3:0", "gcb:4:0")
|
||||
assert all(edge["representation"] == "raw" for edge in graph["edges"])
|
||||
|
||||
|
||||
def test_spatial3_includes_isolated_nodes_and_has_distinct_positions(regression_output: Path):
|
||||
graph = project(regression_output, "spatial3_scheduled", view="raw")
|
||||
assert graph["counts"] == {
|
||||
"nodes": 16, "edges": 1, "isolated_nodes": 14,
|
||||
"raw_nodes": 16, "aggregate_nodes": 0, "raw_edges": 1, "aggregate_edges": 0,
|
||||
}
|
||||
nodes = {node["node_id"]: node for node in graph["nodes"]}
|
||||
assert {"scb:1:0", "scb:1:14", "sc:0"} <= nodes.keys()
|
||||
assert len({(node["x"], node["y"]) for node in graph["nodes"]}) == 16
|
||||
assert [(edge["source"], edge["target"]) for edge in graph["edges"]] == [("scb:1:5", "sc:0")]
|
||||
|
||||
|
||||
def test_header_only_edges_preserve_all_nodes(regression_output: Path):
|
||||
graph = project(regression_output, "empty_edges", view="raw")
|
||||
assert graph["counts"]["nodes"] == 3
|
||||
assert graph["counts"]["edges"] == 0
|
||||
assert graph["counts"]["isolated_nodes"] == 3
|
||||
|
||||
|
||||
def test_projection_cap_is_actionable_and_never_partial(regression_output: Path):
|
||||
with pytest.raises(DisplayGraphTooLarge, match="Use an aggregate view or increase"):
|
||||
project(regression_output, "spatial3_scheduled", view="raw", cap=16)
|
||||
client = TestClient(create_app(regression_output, expansion_cap=16))
|
||||
response = client.get("/api/reports/spatial3_scheduled/display-graph?view=raw")
|
||||
assert response.status_code == 400
|
||||
assert "safety cap" in response.json()["detail"] and "aggregate view" in response.json()["detail"]
|
||||
|
||||
|
||||
def test_display_api_and_static_raw_interactions(regression_output: Path):
|
||||
client = TestClient(create_app(regression_output))
|
||||
assert client.get("/api/reports/spatial3_scheduled/display-graph?view=raw").json()["counts"]["nodes"] == 16
|
||||
index = client.get("/").text
|
||||
script = client.get("/app.js").text
|
||||
assert '<option value="raw">Raw instances</option>' in index
|
||||
assert 'report?.stage === "spatial3"' in script
|
||||
assert 'local.representation === "raw"' in script
|
||||
reducer = script[script.index("function reduceEdge"):script.index("function applyFilters")]
|
||||
assert "state.graph.source(id)" not in reducer
|
||||
assert "edge.source === edge.target" in reducer
|
||||
raw_edge_branch = script.index('if (local.representation === "raw")', script.index("async function showEdge"))
|
||||
aggregate_request = script.index("/api/aggregate-edges/", raw_edge_branch)
|
||||
assert "return;" in script[raw_edge_branch:aggregate_request]
|
||||
aggregate_edge = "operation:spatial1_graph:1->2"
|
||||
assert client.get(f"/api/aggregate-edges/{aggregate_edge}/matrix").status_code == 200
|
||||
Reference in New Issue
Block a user