{"id":5385,"date":"2026-04-11T23:35:13","date_gmt":"2026-04-11T23:35:13","guid":{"rendered":"https:\/\/172-234-197-23.ip.linodeusercontent.com\/?p=5385"},"modified":"2026-04-11T23:35:13","modified_gmt":"2026-04-11T23:35:13","slug":"global-scythe-grpc-stream-latent-swarm-decomposition","status":"publish","type":"post","link":"https:\/\/neurosphere-2.tail52f848.ts.net\/wordpress\/?p=5385","title":{"rendered":"GLOBAL SCYTHE gRPC Stream & Latent Swarm Decomposition"},"content":{"rendered":"\n
\"\"<\/a><\/figure>\n\n\n\n

SCYTHE Intelligence platform evolution, sans potential data pipeline bottlenecks, and the language the system uses to think about data becomes the constraint. We hit that!<\/p>\n\n\n\n

Previous versions of SCYTHE moved data over HTTP REST and WebSocket frames. It worked. But it was always a translation problem: produce JSON, ship JSON, parse JSON, render JSON. Every hop was another opportunity to lose structure, lose timing, lose trust. And in a system designed to reason about adversarial network behavior, structural loss is a threat surface.<\/p>\n\n\n\n

This release changes the data plane from the ground up.<\/p>\n\n\n\n

\n\n\n\n

The Problem We Were Actually Solving<\/h2>\n\n\n\n

By Stage 8, SCYTHE could detect clusters, narrate behavior, simulate activation cascades, and render live RF volumetric fields on a Cesium globe. That’s not a small surface area.<\/p>\n\n\n\n

But three things were grinding against each other:<\/p>\n\n\n\n

    \n
  1. JSON overhead on high-frequency voxel streams. The RF field processor was packing 32\u00b3 float32 arrays into JSON lists and shipping them over WebSocket. At 64\u00b3 resolution, that’s 524,288 bytes of numeric text that had to be parsed on the other end before a single shader could touch it. We were burning cycles on syntax, not signal.<\/li>\n\n\n\n
  2. Token authentication was brittle at WebSocket reconnect. URL-embedded tokens (?token=abc123<\/code>) broke on orchestrator restart, failed on Private Network Access preflight checks, and provided no isolation between instances sharing the same relay. The auth model was bolted on at the protocol boundary \u2014 the worst place for security.<\/li>\n\n\n\n
  3. “Quiet” clusters were invisible. A cluster of 694 nodes sitting dormant showed up as a static counter. The system had no vocabulary for what coordinated inactivity means \u2014 no density decomposition, no temporal ghosting, no probabilistic intent layer. A staging infrastructure in standby looked identical to an abandoned one.<\/li>\n<\/ol>\n\n\n\n
    \n\n\n\n

    The Latent Swarm Decomposition Engine<\/h2>\n\n\n\n

    Before we changed the wire protocol, we changed how the system thinks about clusters.<\/p>\n\n\n\n

    The insight: a node count is a compressed event, not a measurement. 694 nodes sitting quiet isn’t the absence of signal. It’s a coiled spring with a signature.<\/p>\n\n\n\n

    We extended the cluster intelligence layer with what we’re calling the Latent Swarm Decomposition Engine \u2014 a multi-layer autopsy that fires when a cluster is examined rather than just enumerated.<\/p>\n\n\n\n

    Dimensional Density (not a counter)<\/h3>\n\n\n\n

    Instead of reporting:<\/p>\n\n\n\n

    Nodes: 694<\/code><\/pre>\n\n\n\n
    The system now computes:<\/p>\n\n\n\n
    Spatial Density:     0.82   (highly co-located)\nTemporal Activity:   0.03   (dormant)\nASN Entropy:         1.74   (multi-operator blend)\nSignal Coherence:    0.61   (shared infra patterns)<\/code><\/pre>\n\n\n\n
    Low activity + high density = staging posture. Mixed ASN + coherence = coordination layer. These aren’t labels applied after the fact \u2014 they’re derived from the same cluster graph the system already maintains.<\/p>\n\n\n\n
    ASN Stratification<\/h3>\n\n\n\n
    Each cluster decomposes into its ASN membership:<\/p>\n\n\n\n
    AS6849  \u2192 312 nodes  (core infrastructure)\nAS16509 \u2192 148 nodes  (AWS bleed-in)\nAS13335 \u2192  97 nodes  (Cloudflare masking layer)\n+ 3 minor ASNs<\/code><\/pre>\n\n\n\n
    The interpretation engine generates narrative automatically: “Core infrastructure anchored in AS6849 with cloud spillover. Pattern suggests hybrid hosting + traffic obfuscation.” That text isn’t written by a human \u2014 it’s derived from the ASN stratification logic in cluster_swarm_engine.py<\/code>, which runs a 3-tier IP\u2192ASN resolution pipeline (pyasn radix, MaxMind ASN, MaxMind City) with a 10K-entry LRU cache.<\/p>\n\n\n\n
    Temporal Ghosting<\/h3>\n\n\n\n
    Even a “quiet” cluster has temporal structure. The system now surfaces:<\/p>\n\n\n\n
    02:14 UTC \u2192 microburst (11 nodes)\n07:52 UTC \u2192 sync jitter across 200 nodes\n13:08 UTC \u2192 ASN route shift<\/code><\/pre>\n\n\n\n
    This reframes silence as below-threshold coordination rather than absence. The cluster isn’t gone \u2014 it’s between moves.<\/p>\n\n\n\n
    Silence Pressure Metric<\/h3>\n\n\n\n
    Silence Pressure = Node Count \u00d7 Inactivity Duration \u00d7 Coherence<\/code><\/pre>\n\n\n\n
    A cluster with high node count, sustained inactivity, and high phase coherence has elevated silence pressure \u2014 meaning coordinated activation is more probable, not less. The counter became an intelligence surface.<\/p>\n\n\n\n
    Probabilistic Intent Engine<\/h3>\n\n\n\n
    Latent Intent Probabilities:\n  Staging Infrastructure:  61%\n  Traffic Relay Mesh:      22%\n  Abandoned \/ Decaying:     9%\n  Unknown:                  8%<\/code><\/pre>\n\n\n\n
    These scores are computed from six independent basis vectors in narrate_cluster()<\/code>: node concentration, ASN diversity, phase coherence, behavioral type match, temporal activity, and RF fraction. They aren’t hard-coded \u2014 they’re a weighted inference from the live cluster state.<\/p>\n\n\n\n
    \n\n\n\n
    The Architecture Shift: gRPC + Protobuf Data Plane<\/h2>\n\n\n\n
    With the cluster intelligence layer upgraded, we turned to the wire protocol problem.<\/p>\n\n\n\n
    What changed<\/h3>\n\n\n\n
    The SCYTHE stack now has two clearly separated planes:<\/p>\n\n\n\n
    Control Plane   \u2192 port 8765  (WebSocket, orchestrator commands, operator auth)\nData Plane      \u2192 port 50051 (gRPC + Protobuf, high-throughput binary streams)\nLegacy Fallback \u2192 port 8766  (rf_voxel_processor HTTP\/WS, unchanged)<\/code><\/pre>\n\n\n\n
    We extended scythe.proto<\/code> with a new ScytheStreamService<\/code> alongside the existing OrchestratorService<\/code>, HypergraphService<\/code>, and ClusterIntelService<\/code>:<\/p>\n\n\n\n
    service ScytheStreamService {\n  rpc StreamClusters(StreamRequest) returns (stream StreamCluster);\n  rpc StreamRFField(LodHint)        returns (stream RFField);\n  rpc StreamDeltas(StreamRequest)   returns (stream StreamDelta);\n}<\/code><\/pre>\n\n\n\n
    Each message is Protobuf-encoded. RFField.voxels<\/code> is raw bytes<\/code> \u2014 the float32 grid lands directly in the GPU texture upload path with no parsing step.<\/p>\n\n\n\n
    GPU-Native RF Field Synthesis<\/h3>\n\n\n\n
    The voxel field computation moved from a CPU numpy loop to a CUDA-accelerated inverse-distance weighting kernel:<\/p>\n\n\n\n
    # Vectorised on a CUDA meshgrid \u2014 no Python loop over voxels\ngrid = torch.stack(torch.meshgrid(\n    torch.linspace(-1, 1, S, device=device),\n    torch.linspace(-1, 1, S, device=device),\n    torch.linspace(-1, 1, S, device=device),\n    indexing='ij'\n), dim=-1)  # [S, S, S, 3]\n\nfor pos, intensity in zip(pos_t, int_t):\n    diff = grid - pos.view(1, 1, 1, 3)\n    dist2 = (diff * diff).sum(-1)\n    field += intensity \/ (dist2 + 0.01)<\/code><\/pre>\n\n\n\n
    The result is normalised, packed as [sx:u16 LE][sy:u16 LE][sz:u16 LE][float32 LE\u2026]<\/code>, and cached with an ISO-8601 timestamp header (X-Field-Timestamp<\/code>). The gRPC StreamRFField<\/code> servicer deduplicates by that header \u2014 frames are only forwarded when the field actually changed.<\/p>\n\n\n\n
    LOD Demand Signaling<\/h3>\n\n\n\n
    The client sends camera altitude in the LodHint<\/code> message; the server selects:<\/p>\n\n\n\n
    >50 km  \u2192 LOD 0  (16\u00b3,  200ms poll)\n>10 km  \u2192 LOD 1  (32\u00b3,  500ms poll)\n \u226410 km \u2192 LOD 2  (64\u00b3, 1000ms poll)<\/code><\/pre>\n\n\n\n
    LOD downsampling uses scipy.ndimage.zoom<\/code> with order=1<\/code>. The client gets resolution appropriate to its viewing distance \u2014 no wasted bandwidth computing 64\u00b3 fields for a globe-altitude view.<\/p>\n\n\n\n
    Instance-Bound Auth on Streaming RPCs<\/h3>\n\n\n\n
    A subtle but important hardening: every streaming RPC that touches instance-specific data now carries instance_id<\/code> in its request message and enforces it via _check_instance_auth()<\/code>. The old design used Empty<\/code> request types for StreamClusters<\/code> and StreamDeltas<\/code> \u2014 meaning a session bound to one instance could see data from all instances. That’s now closed.<\/p>\n\n\n\n
    \n\n\n\n
    The deck.gl Arbitration Layer<\/h2>\n\n\n\n
    The rf-arbitration-layer.js<\/code> brings volumetric RF rendering to the browser without a WebGPU dependency. It uses a WebGL2 TEXTURE_3D<\/code> path with a custom 64-step raymarching shader:<\/p>\n\n\n\n
    \/\/ Heat-map colour ramp: blue \u2192 cyan \u2192 green \u2192 yellow \u2192 red\nvec3 heatmap(float t) {\n    if (t < 0.25) return mix(vec3(0.0,0.0,1.0), vec3(0.0,1.0,1.0), t*4.0);\n    if (t < 0.50) return mix(vec3(0.0,1.0,1.0), vec3(0.0,1.0,0.0), (t-0.25)*4.0);\n    if (t < 0.75) return mix(vec3(0.0,1.0,0.0), vec3(1.0,1.0,0.0), (t-0.50)*4.0);\n    return            mix(vec3(1.0,1.0,0.0), vec3(1.0,0.0,0.0), (t-0.75)*4.0);\n}<\/code><\/pre>\n\n\n\n
    The computeArbitration()<\/code> kernel runs CPU-side before the first render:<\/p>\n\n\n\n
    signal \u00d7 0.6 + coherence \u00d7 0.3 + (1 \u2212 diversity) \u00d7 0.1<\/code><\/pre>\n\n\n\n
    This gives each voxel an arbitration weight \u2014 not just “how strong is the signal here” but “how contested is this space.” The computeDominance()<\/code> function computes per-voxel difference between two competing RF fields, enabling multi-cluster interference visualization.<\/p>\n\n\n\n
    Cesium camera sync keeps the deck.gl overlay locked to the globe view via camera.changed<\/code> and camera.moveEnd<\/code> events.<\/p>\n\n\n\n
    \n\n\n\n
    What We Caught Before It Shipped<\/h2>\n\n\n\n
    Four design reviews ran in parallel during implementation. Three blocking issues were caught before the code landed:<\/p>\n\n\n\n
    Intent scores iteration bug. cluster_swarm_engine.py<\/code> returns intent_scores<\/code> as a list of {label, basis, score}<\/code> dicts. The gRPC DecomposeCluster<\/code> handler was calling .items()<\/code> on it \u2014 a silent AttributeError<\/code> at runtime. Fixed to proper list iteration.<\/p>\n\n\n\n
    Empty request types bypass instance auth. StreamClusters(Empty)<\/code> and StreamDeltas(Empty)<\/code> had no instance_id<\/code> field, so _check_instance_auth()<\/code> couldn’t be called. Both changed to StreamClusters(StreamRequest)<\/code> \/ StreamDeltas(StreamRequest)<\/code>. Sessions are now properly scoped.<\/p>\n\n\n\n
    \/api\/gpu-field<\/code> unauthenticated compute endpoint. The docstring said “protected by X-Internal-Token” but the implementation had no check. Fixed to validate against SCYTHE_INTERNAL_TOKEN<\/code> env var when set.<\/p>\n\n\n\n
    Wrong data source for cluster streams. StreamClusters<\/code> was fetching raw nodes from \/api\/gravity\/nodes<\/code> and grouping them manually \u2014 bypassing the intelligence layer entirely. Changed to \/api\/clusters\/intel<\/code>, which returns pre-narrated cluster summaries with threat scores, ASN diversity, and phase coherence already computed.<\/p>\n\n\n\n
    \n\n\n\n
    VoxelStreamHub Cluster Publishers<\/h2>\n\n\n\n
    voxel_stream_engine.py<\/code> already defined CH_CLUSTER_NODES (0x02)<\/code> and CH_CLUSTER_DELTA (0x03)<\/code> constants and a pack_nodes()<\/code> serialiser \u2014 but the VoxelStreamHub<\/code> had no publish methods for them. Only publish_rf_field()<\/code> existed.<\/p>\n\n\n\n
    Two new methods complete the channel matrix:<\/p>\n\n\n\n
    async def publish_cluster_nodes(self, nodes: list) -> None:\n    payload = pack_nodes(nodes)\n    await self.publish(CH_CLUSTER_NODES, payload)\n\nasync def publish_cluster_delta(self, node: dict) -> None:\n    payload = pack_nodes([node])\n    await self.publish(CH_CLUSTER_DELTA, payload)<\/code><\/pre>\n\n\n\n
    These are the hooks that will let cluster_swarm_engine.py<\/code> push live cluster updates into the streaming pipeline \u2014 replacing the current REST-poll architecture with a true publish\/subscribe model.<\/p>\n\n\n\n
    \n\n\n\n
    Starting the System<\/h2>\n\n\n\n
    The orchestrator startup command is unchanged:<\/p>\n\n\n\n
    python3 scythe_orchestrator.py \\\n  --host 0.0.0.0 \\\n  --port 5001 \\\n  --ollama-url http:\/\/192.168.1.185:11434 \\\n  --stream-relay-url ws:\/\/192.168.1.185:8765\/ws \\\n  --mcp-ws-url ws:\/\/192.168.1.185:8766\/ws<\/code><\/pre>\n\n\n\n
    The gRPC server now launches automatically as a managed subprocess on port 50051. No additional flag required. To customise the gRPC port:<\/p>\n\n\n\n
      --grpc-port 50051   # optional \u2014 50051 is the default<\/code><\/pre>\n\n\n\n
    To disable the gRPC server entirely (e.g., for a lightweight dev session):<\/p>\n\n\n\n
      --no-grpc<\/code><\/pre>\n\n\n\n
    The --stream-relay-url<\/code> and --mcp-ws-url<\/code> flags continue to configure the WebSocket control plane and the legacy voxel API \u2014 both still present and fully operational. The gRPC data plane adds capability; it doesn’t replace the existing WS layer.<\/p>\n\n\n\n
    \n\n\n\n
    What This Platform Is Becoming<\/h2>\n\n\n\n
    Most network intelligence systems are built around the same primitive: poll an API, parse JSON, render a number. SCYTHE’s architecture is increasingly built around a different primitive: a cluster is a live field, and fields have physics.<\/p>\n\n\n\n
    A cluster doesn’t sit in a database cell. It has density, entropy, coherence, and silence pressure. It occupies space in a 3D RF field that can be raymarched and rendered. It has a temporal ghost \u2014 a record of micro-events that reframes what “quiet” means. It has an activation cascade that can be simulated before it happens.<\/p>\n\n\n\n
    The gRPC data plane, the GPU-native field synthesis, and the deck.gl arbitration layer are all in service of that one idea: intelligence isn’t a number you report. It’s a surface you explore.<\/p>\n\n\n\n
    SCYTHE is developed as an open-source RF intelligence and network visualization platform. The stack runs on Python (FastAPI, Flask, gRPC), PyTorch CUDA, CesiumJS, Three.js, and deck.gl. All intelligence narration is generated by local LLM inference \u2014 no external API calls.<\/p>\n","protected":false},"excerpt":{"rendered":"
    SCYTHE Intelligence platform evolution, sans potential data pipeline bottlenecks, and the language the system uses to think about data becomes the constraint. We hit that! Previous versions of SCYTHE moved data over HTTP REST and WebSocket frames. It worked. But it was always a translation problem: produce JSON, ship JSON, parse JSON, render JSON. Every… <\/p>\n","protected":false},"author":2,"featured_media":5387,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"neve_meta_sidebar":"","neve_meta_container":"","neve_meta_enable_content_width":"","neve_meta_content_width":0,"neve_meta_title_alignment":"","neve_meta_author_avatar":"","neve_post_elements_order":"","neve_meta_disable_header":"","neve_meta_disable_footer":"","neve_meta_disable_title":"","footnotes":""},"categories":[11,13],"tags":[],"class_list":["post-5385","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-signal_scythe","category-the-truben-show"],"_links":{"self":[{"href":"https:\/\/neurosphere-2.tail52f848.ts.net\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/5385","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/neurosphere-2.tail52f848.ts.net\/wordpress\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/neurosphere-2.tail52f848.ts.net\/wordpress\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/neurosphere-2.tail52f848.ts.net\/wordpress\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/neurosphere-2.tail52f848.ts.net\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5385"}],"version-history":[{"count":0,"href":"https:\/\/neurosphere-2.tail52f848.ts.net\/wordpress\/index.php?rest_route=\/wp\/v2\/posts\/5385\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/neurosphere-2.tail52f848.ts.net\/wordpress\/index.php?rest_route=\/wp\/v2\/media\/5387"}],"wp:attachment":[{"href":"https:\/\/neurosphere-2.tail52f848.ts.net\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5385"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/neurosphere-2.tail52f848.ts.net\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5385"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/neurosphere-2.tail52f848.ts.net\/wordpress\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5385"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}