StarIntel Server

StarIntel Server

StarIntel Server is the Common Lisp runtime for storing, routing, querying, and recursively enriching StarIntel documents.

It combines:

• CouchDB for durable documents, views, and full-text search.
• RabbitMQ topic exchanges for document and actor traffic.
• Sento actors for local concurrency, supervision boundaries, timers, and message passing.
• A Ningle/Clack/Hunchentoot HTTP API.
• The star-cl document library and StarIntel specification adapters.
• Nix-built binaries, tests, and container images.

This repository is an experimental operator system, not a hardened public SaaS service.

Do not expose the HTTP API or RabbitMQ directly to the public Internet.

The HTTP API currently has no authentication or authorization and sends Access-Control-Allow-Origin: *. Put it behind an authenticated reverse proxy, restrict network access, and treat the Lisp init file as trusted executable code.

Position in the Quasar / StarIntel stack

StarIntel Server is a backend service layer. It is not the Quasar browser UI and it is not replaced by quasar-ui.

The intended deployment split is:

quasar-ui
  browser UI / graph renderer / standalone subset
        |
        | typed commands, projections, capability discovery
        v
quasar
  canonical Common Lisp control plane/runtime
        |
        | StarIntel service APIs and adapters
        v
starintel-server
  persistent ingest / storage / search / routing / RabbitMQ
        |
        +-----------------------------+
        |                             |
        v                             v
star-bbpd                       other actor services
  external recon actors          collectors / analyzers / tools

Responsibilities are intentionally separated:

• quasar-ui owns browser presentation, mobile/PWA behavior, Cytoscape rendering, browser-local standalone workspaces, and browser-safe bounded actions.
• quasar owns the canonical Common Lisp command/control boundary for migrated durable Quasar operations, persistent runtime supervision, privileged local integrations, reconnect/replay, and capability discovery.
• starintel-server owns the persistent StarIntel backend responsibilities implemented here: document ingest, CouchDB persistence/querying, RabbitMQ document/actor routing, HTTP service boundaries, and recursive dataflow.
• star-bbpd and similar repositories are external actor services. BBPD consumes actor-specific RabbitMQ targets, runs Subfinder, Nmap, Httpx, Katana and DNS workflows, and publishes derived StarIntel documents and relations.

A connected Quasar UI may expose controls and results for server/external services, but that does not make those capabilities browser implementations. The UI should discover which runtime/service capabilities are actually available.

Standalone quasar-ui remains valid without this server. That mode is a bounded subset and must not be documented as feature-equivalent to a connected StarIntel deployment.

What the server does

A document normally moves through this pipeline:

HTTP/client/actor
      |
      v
documents.ingest.<dtype>
      |
      v
CouchDB insert + _id/_rev enrichment
      |
      v
documents.new.<dtype>
      |
      +--> local actor via TELL
      +--> remote actor via RabbitMQ topic route
      +--> derived documents and relations
                 |
                 +--> documents.ingest.<dtype>  (durable recursion)
                 `--> documents.new.<dtype>     (event-only fan-out)

Actors can emit more StarIntel documents, relations, targets, and actor events. Those outputs can trigger more actors. This is dataflow recursion: the graph expands through messages rather than recursive function calls.

Quick start: Nix-built Compose stack

Requirements: Nix with flakes, Docker Engine, Docker Compose v2, curl, jq, and openssl.

cp .env.example .env
install -d -m 0700 secrets
openssl rand -base64 32 > secrets/couchdb_password
openssl rand -base64 48 > secrets/couchdb_secret
openssl rand -hex 24 | tr '[:lower:]' '[:upper:]' > secrets/erlang_cookie
openssl rand -base64 32 > secrets/rabbitmq_password
chmod 0600 secrets/*

nix run .#load-images
docker compose up --detach --wait
curl --fail http://127.0.0.1:5000/health

Default local endpoints:

Service	Address
StarIntel HTTP API	http://127.0.0.1:5000
CouchDB	http://127.0.0.1:5984
RabbitMQ AMQP	127.0.0.1:5672
RabbitMQ management	http://127.0.0.1:15672

See DOCKER.md for image builds, secrets, persistence, migration, backup, FTS, and upgrades.

Submit a document

The body must contain a dtype. The route chooses the RabbitMQ routing key but does not currently inject or validate the body dtype.

curl --fail \
  --header 'Content-Type: application/json' \
  --request POST \
  --data '{
    "_id": "example-note",
    "dataset": "demo",
    "dtype": "note",
    "sources": ["manual"],
    "version": "0.8.0",
    "dateAdded": 0,
    "dateUpdated": 0,
    "content": "first document"
  }' \
  http://127.0.0.1:5000/new/document/note

The API acknowledges queue publication, not CouchDB persistence. Read the document after the ingest consumer has processed it:

curl --fail http://127.0.0.1:5000/document/example-note | jq

Build and test

nix build
nix run .#star-unit-tests

With CouchDB and RabbitMQ available:

nix run .#star-integration-tests

Full Nix image, health, FTS, restart, and persistence test:

./scripts/stack-test.sh

See docs/testing.md.

Run from Common Lisp

The pinned Nix build is the supported reproducible path. For interactive development:

nix develop
sbcl --load run.lisp

Build the executable:

nix build
./result/bin/star-server start --init ./example_configs/init.lisp

The executable accepts:

star-server start -i PATH
star-server start --init PATH

The same path can be supplied through STAR_SERVER_INIT_FILE.

Documentation map

Document	Contents
docs/index.org	Documentation index and implementation status
docs/architecture.org	Runtime structure, startup order, concurrency, and repository layout
docs/actors.org	Creating, registering, targeting, scheduling, and operating actors
docs/messaging.org	RabbitMQ exchanges, queues, routing keys, recursion, delivery, and loop control
docs/document-spec.org	StarIntel 0.9 and legacy 0.8 documents, types, relations, IDs, metadata, and provenance
docs/configuration.org	Environment, init files, secrets, advanced examples, and tuning
docs/http-api-docs.org	HTTP endpoint reference and examples
DOCKER.md	Nix-built container stack and operations
docs/testing.md	Unit, integration, and stack tests

Runtime status

The documentation distinguishes three states:

• Active: loaded by source/starintel-gserver.asd and started by star::main.
• Present but not active: code exists in the repository but is not loaded by the ASDF system or is not started by the current startup path.
• Stub/experimental: API or actor code exists but is incomplete.

Important current limits:

• HTTP ingestion does not enforce the strict StarIntel 0.9 schema.
• The server still uses legacy flat 0.8 constructors in parts of the actor code.
• source/actor-systems/user-finder.lisp and user-hunt.lisp are not loaded by the ASDF system.
• The matcher actor framework is experimental; the URL extractor is loaded, but a complete global pattern-dispatch loop is not wired.
• /new/event/:id is a stub.
• *http-api-base-path* and the HTTP certificate/key variables are not applied by start-http-api.
• The second /dataset-size route definition replaces or shadows the first, depending on Ningle route behavior.

These are documented facts, not supported guarantees.

License

See LICENSE.