ON THIS PAGE
Overview Visual Architecture Active Memory Concepts 4-Tier Fabric Persistence & WAL Security Detail API Specification Scenario Guide

Superbrain Documentation

Superbrain is a next-generation distributed memory fabric designed for AI agents. It transforms passive storage into a cognitive participant in your agentic workflows.

Traditional databases treat data as static bytes to be retrieved. Superbrain treats data as Active Memoryβ€”with semantic awareness, intentional decay, and microsecond-latency sharing between co-located and distributed agents.

New in v3.0.0: Advanced cognitive triggers and mTLS enrollment for secure multi-agent swarms.

Architecture Deep Dive

Superbrain separates the **Control Plane** (Metadata, Consensus, Routing) from the **Data Plane** (Raw Bytes, mmap, rDMA). This allows for extreme horizontal scale without impacting metadata consistency.

Coordinator Raft Consensus / CA / mDNS Agent SDK AES-256-GCM SDK Local Metadata Cache Fabric Node RAM 1. Allocate (Control) 2. DATA IO (gRPC / SHM)

Coordinator Bypass (Metadata Caching)

Establishing a new pointer requires a ROUND TRIP to the Coordinator. However, Superbrain SDKs implement a local **Pointer Map Cache**. Once a 36-byte UUID is resolved to a node's IP:

  • βœ” Direct I/O: All future Read/Write calls bypass the Coordinator entirely.
  • βœ” 1ms Operations: Eliminates control-plane latency from the data hot-path.
  • βœ” P2P Discovery: SDKs listen to Gossip protocol to update node healthy states locally.

Active Memory Concepts

Active Memory transforms distributed storage into a cognitive participant. Unlike Redis or S3, Superbrain understands the "liveliness" and "intent" of the data it stores.

Forgetting Mechanism

Every write can have a liveliness score (0.0 to 1.0). The node runs a background **Forgetter** loop that scans blocks periodically.

  • βœ” 1.0 (Durable): Never evicted, mirrored to L4.
  • βœ” 0.5 (Normal): Evicted to L3 KV Pool after 10m.
  • βœ” 0.1 (Ephemeral): Fast-decay, wiped after 30s.

Semantic Consistency

When multiple agents write to the same context, Superbrain resolves logical conflicts using **Semantic Locking** held at the Intent level.

The 4-Tier Memory Fabric

Superbrain automatically routes data between these four tiers based on proximity and configuration.

TierLatencyMechanismDurability
L1: SHM Bypass~13ΞΌsmmap /dev/shmVolatile (Node)
L2: Distributed RAM~1.2msgRPC StreamingVolatile (Cluster)
L3: KV Cache Pool~5mszlib CompressionManaged RAM
L4: Durable Storage~50msWAL / Write-BehindPermanent

Quickstart Guide

Get your agent swarm up and running in minutes.

1. Install the SDK

pip install superbrain-sdk
npm install superbrain-distributed-sdk
go get github.com/golightstep/superbrainSdk

2. Basic Memory Share

from superbrain import SuperbrainClient
client = SuperbrainClient("localhost:50050")

# Write to distributed RAM
ptr_id = client.allocate(1024)
client.write(ptr_id, 0, b"Agent State Alpha")
const { SuperbrainClient } = require('superbrain-distributed-sdk');
const client = new SuperbrainClient('localhost:50050');

const ptrId = await client.allocate(1024);
await client.write(ptrId, 0, Buffer.from("Agent State Alpha"));
client := sdk.NewClient("localhost:50050")
ptr, _ := client.Allocate(1024)
client.Write(ptr.ID, 0, []byte("Agent State Alpha"))

Configuration Ability

Tune every aspect of the SDK and Cluster behavior. Superbrain supports configuration via Environment Variables (recommended for containers) or explicit `Config` objects.

SDK Settings

Persistence Storage Tunables

Pass these options inside the SB_PERSISTENCE_CONFIG JSON object.

ParameterDefaultDescription
SB_COORDINATORlocalhost:50050Primary gRPC endpoint. Comma-separate for HA pools.
SB_ENCRYPTION_KEYnull32-byte key for E2EE. Enabling this wraps all IO in AES-GCM.
SB_BYPASS_LOCALtrueIf node IP is 127.0.0.1, use SHM for 15ΞΌs latency bypass.
SB_MTLS_ENABLEfalseForce mutual TLS certification for all agent operations.
ParameterTypeDefaultDescription
pathstring./sb_dataRoot directory for FileStore/WAL.
wal_sync_modestringnonealways for synchronous fsync, none for OS buffering.
max_queue_sizeint10000Bounded write-behind queue size before backpressure kicks in.
compressbooltrueEnable zlib compression for L3 tier blocks.

Persistence & Durability

Superbrain includes an optional persistence layer designed for minimal impact on the RAM fabric's performance. It uses an asynchronous Write-Behind pattern to offload I/O while maintaining in-memory speed.

DURABLE

Write-Ahead Log (WAL)

To prevent data loss in the async queue during a crash, Superbrain records every write task in a local Write-Ahead Log before acknowledging it to the agent's RAM. On restart, the node automatically replays the WAL to recover any unsaved blocks.

Default Behavior: Enabling the filestore provider automatically activates the WAL in the same directory.

Configuration Example

# Start node with Durable Local Storage
./bin/node --persistence-provider=filestore \
           --persistence-config='{"path": "/var/lib/superbrain", "wal_sync_mode": "always"}'

JSON Configuration Backends

Configure the L3 tier by passing provider-specific JSON via the --persistence-config flag.

./bin/node --persistence-provider=filestore \
           --persistence-config='{"path": "/data", "wal_sync_mode": "always"}'
./bin/node --persistence-provider=redis \
           --persistence-config='{"addr": "localhost:6379", "db": 0}'
./bin/node --persistence-provider=postgres \
           --persistence-config='{"dsn": "postgres://user:pass@host:5432/db"}'
Zero-Impact Design: Reads and writes to the memory fabric occur at RAM speed. The persistence engine operates as a "Write-Behind" queue, ensuring disk/DB latency never touches the agent hot-path.

Security Fabric

Superbrain is designed for untrusted environments. It uses a dual-layer security model to protect both communication and data.

ENCRYPTED

Zero-Trust mTLS

Every node and agent must be enrolled in the cluster's private PKI. The Coordinator acts as a Certificate Authority (CA), issuing short-lived certificates upon successful registration.

SDK-LEVEL

End-to-End Encryption (E2EE)

When enabled via SB_ENCRYPTION_KEY, the SDK wraps all memory writes in AES-GCM-256. Infrastructure nodes only see encrypted entropy, ensuring your agent's private reasoning remains private.

client = superbrain.Client(encryption_key=KEY_32_BYTES)
const client = new SuperbrainClient(addr, { encryptionKey: KEY_32_BYTES });
client, _ := sdk.NewClientWithEncryption(key, addr)
Control Plane
ENROLLMENT

Client.Register(agentID)

The first step for any secure agent. This method performs an mTLS handshake, generates a local keypair, and obtains a signed certificate from the Coordinator CA.

# Identity enrollment in the Secure Fabric
client.register("research-agent-01")
// Enrollment is required before any Write/Allocate
await client.register("research-agent-01");
err := client.Register("research-agent-01")
CONSENSUS

Consistency Guard (Raft)

For distributed agents, Superbrain uses a Raft-based consensus protocol to manage Semantic Locks. This prevents logical race conditions where two agents might attempt to "read-modify-write" a shared reasoning state simultaneously.

Note: Semantic locks are held at the Intent level, allowing for high-concurrency writes to different logical goals within the same memory block.

Detailed API Specification

Superbrain provides three primary API tiers. Level 1 is the Raw Pointer API, offering maximum performance and control over byte offsets.

gRPC / SHM
CORE

Client.Allocate(size)

Reserves chunks of RAM across the distributed fabric. Returns a 36-byte UUID pointer.

# πŸ”’ Secure Client (mTLS + E2EE)
client = superbrain.Client(
    coordinators=["coord:50050"],
    encryption_key=os.getenv("SB_KEY")
)

# Allocate 50MB
ptr_id = client.allocate(50 * 1024 * 1024)
print(f"Memory ready: {ptr_id}")
// Allocate 50MB
const ptrId = await client.allocate(50 * 1024 * 1024);
console.log(`Memory ready: ${ptrId}`);
// Allocate 50MB
ptr, err := client.Allocate(50 * 1024 * 1024)
fmt.Printf("Memory ready: %s\n", ptr.ID)

Parameters

sizeuint64Number of bytes to allocate.

Common Exceptions

  • ERR_QUOTA_EXHAUSTED: Cluster has no free RAM in this tier.
  • ERR_COORD_UNREACHABLE: Metadata resolve failed.
Active Memory
COGNITIVE

Client.WriteCognitive(...)

Enriches raw data writes with semantic metadata. This is the foundation of Superbrain's "Cognitive Architecture".

Parameters

summarystringLLM-readable summary of the chunk content.
intentstringThe logical purpose of this write (e.g. "Drafting Plan").
livelinessfloat (0-1)Sets the decay rate. 0.1 = ephemeral, 1.0 = persistent.
Tip: Use low liveliness for intermediate "Chain of Thought" reasoning to save RAM and avoid context bloat.
# Write with metadata and 30-day "half-life"
client.write_cognitive(
    ptr_id, offset=0, data=b"...",
    liveliness=0.8,
    intent="Strategic Planning",
    summary="Agent decision to expand search",
    tag="Critical"
)
await client.writeCognitive(ptrId, 0, data, {
    liveliness: 0.8,
    intent: "Strategic Planning",
    summary: "Agent decision to expand search",
    tag: "Critical"
});
client.WriteCognitive(ptr, 0, data, sdk.CognitiveOptions{
    Liveliness: 0.8,
    Intent:     "Strategic Planning",
    Summary:    "Agent decision to expand search",
    Tag:        "Critical",
})
Pub/Sub
REAL-TIME

Client.SemanticSubscribe(intent)

Registers a reactive trigger that wakes your agent up when any other agent in the fabric writes with a matching semantic intent.

def on_user_goal(notify):
    print(f"Goal Detected: {notify.summary}")

# Block until a 'User Goal' is written to the fabric
client.semantic_subscribe("User Goal", on_user_goal)
client.semanticSubscribe('User Goal', (notify) => {
    console.log(`Goal Detected: ${notify.summary}`);
});
client.SemanticSubscribe("User Goal", func(n *sdk.Notification) {
    fmt.Printf("Goal Detected: %s\n", n.Summary)
})

Real-World Scenarios

Scenario 1: Collaborative Document Research

One agent downloads a 500MB PDF. Instead of sending the PDF to 10 other agents, it writes to Superbrain and shares the 36-byte pointer via a message bus.

Scenario 2: The "Strategic Interrupt"

An execution agent is writing logs. A security agent subscribes to tag="Sensitive". The moment the executor writes an API key by mistake, the security agent is woken up by Superbrain to scrub the memory and flag the breach.

Scenario 3: Cross-Language Handover

A Go ingestion agent writes high-speed telemetry. A Python agent reads the same offsets for ML inference. Superbrain handles the byte-order and mmap alignment automatically.

CRASH RECOVERY

Scenario 4: Durable Recovery After Hard Crash

A Memory Node experiences a hardware power failure while holding critical agent context. Because Level 4 (FileStore) was enabled:

  • βœ” WAL Replay: Upon restart, the node detects the WAL file and replays all uncommitted writes.
  • βœ” Self-Healing: Agents reconnect automatically and find their data intact in RAM (rehydrated from disk).
  • βœ” Zero Code Change: This durability is transparent to the SDK.