CSBrainAI

Privacy-first RAG workbench for technical teams — Supabase + pgvector, OpenAI, and Sentry

---

🎯 Overview

CSBrainAI is a privacy-first RAG workbench for technical teams that need internal knowledge assistants. User queries are never stored in raw form — only HMAC-SHA256 hashes and lengths are logged, ensuring complete privacy while maintaining full observability. Answers come with citations and similarity scores, backed by an automated eval harness and operational guardrails.

Key Features

• 🔒 Privacy-First: HMAC-SHA256 hashed queries (no raw PII in logs)
• 🚀 RAG Pipeline: Supabase + pgvector for semantic search
• 🤖 OpenAI Integration: text-embedding-3-small + gpt-4o-mini
• 🛡️ Security: CSP, rate limiting (10 req/min), HSTS
• 📊 Observability: Sentry L5 tool with PII scrubbing
• 🧪 Nightly Evals: Automated quality testing via GitHub Actions
• ⚡ Production-Ready: Type-safe, tested, documented

---

🏗️ Architecture

┌─────────────┐
│   User      │
└──────┬──────┘
       │ query
       ▼
┌─────────────────────────┐
│  Next.js App Router     │
│  - Rate Limiting        │
│  - Security Headers     │
│  - PII Scrubbing        │
└──────┬──────────────────┘
       │
       ▼
┌─────────────────────────┐
│  /api/answer            │
│  1. Hash query (HMAC)   │
│  2. Embed query (OpenAI)│
│  3. Vector search       │
│  4. LLM generation      │
└──────┬──────────────────┘
       │
       ▼
┌─────────────────────────┐
│  Supabase + pgvector    │
│  - IVFFlat index        │
│  - Cosine similarity    │
│  - RLS policies         │
└─────────────────────────┘

---

📦 Tech Stack

Component	Technology	Purpose
Framework	Next.js 16 (App Router)	Full-stack React framework
Language	TypeScript 5	Type safety
Database	Supabase (PostgreSQL)	Vector storage with pgvector
Embeddings	OpenAI text-embedding-3-small	1536-dim vectors
LLM	OpenAI gpt-4o-mini	Answer generation
Observability	Sentry	Error tracking (L5 tool)
Rate Limiting	Upstash Redis / Token Bucket	API protection
Security	Next.js Proxy	CSP, HSTS, headers
CI/CD	GitHub Actions	Nightly evaluations

---

🚀 Quick Demo

Prerequisites:

• Node.js 20+
• Supabase account (for vector storage)
• OpenAI API key
• Sentry account (optional but recommended)

Run the demo:

# 1. Install dependencies
npm install

# 2. Set up environment variables
cp .env.example .env
# Edit .env with your SUPABASE_URL, SUPABASE_ANON_KEY, OPENAI_API_KEY

# 3. Run database migration (in Supabase SQL Editor)
# Copy contents of supabase/migrations/001_rag_schema.sql and execute

# 4. Ingest sample knowledge
npm run ingest

# 5. Start development server
npm run dev

# 6. Test the API (in another terminal)
curl -X POST http://localhost:3000/api/answer \
  -H "Content-Type: application/json" \
  -d '{"query":"What is RAG?"}'

Expected Output:

{
  "answer": "RAG (Retrieval Augmented Generation) is an AI architecture...",
  "citations": [
    {
      "source_url": "file://sample-1.md#chunk-0",
      "content": "RAG (Retrieval Augmented Generation) is...",
      "similarity": 0.87
    }
  ],
  "q_hash": "a3f2b9c1d4e5f6a7b8c9d0e1f2a3b4c5...",
  "q_len": 12,
  "tokensUsed": 456
}

What this proves:

• Privacy-first RAG system (queries are hashed, never stored raw)
• Vector search with Supabase + pgvector
• Structured citations with similarity scores
• Operational guardrails (rate limiting, PII scrubbing, hashed-query telemetry) fit for internal knowledge assistants

Alternative demo (no setup needed):

# Run unit tests to see functionality
npm test

Next Steps:

• See docs/ANSWER-FLOW.md for API architecture
• See TEST_NOTES.md for manual test scenarios

---

🚀 Quick Start

Prerequisites

• Node.js 20+
• Supabase account
• OpenAI API key
• Sentry account (optional but recommended)

1. Clone & Install

git clone https://github.com/RazonIn4K/csbrainai.git
cd csbrainai
npm install

2. Environment Setup

cp .env.example .env

Edit .env with your credentials:

# OpenAI
OPENAI_API_KEY=sk-...

# Supabase
SUPABASE_URL=https://xxx.supabase.co
SUPABASE_ANON_KEY=eyJ...
SUPABASE_SERVICE_ROLE=eyJ...  # For ingestion only

# Security
HASH_SALT=$(openssl rand -hex 32)

# Sentry
SENTRY_DSN=https://xxx@xxx.ingest.sentry.io/xxx
NEXT_PUBLIC_SENTRY_DSN=https://xxx@xxx.ingest.sentry.io/xxx

# Optional: Upstash Redis (for distributed rate limiting)
UPSTASH_REDIS_REST_URL=https://xxx.upstash.io
UPSTASH_REDIS_REST_TOKEN=xxx

3. Database Setup

Run the migration in Supabase SQL Editor or via CLI:

# Copy contents of supabase/migrations/001_rag_schema.sql
# and execute in Supabase SQL Editor

Or using Supabase CLI:

supabase link --project-ref <your-project-ref>
supabase db push

4. Ingest Knowledge

Add your knowledge files to data/knowledge/:

# Example structure:
data/knowledge/
├── faq.md
├── product-docs.md
└── technical-guide.md

Run ingestion:

npm run ingest

Expected output:

🚀 Starting RAG ingestion...
📚 Found 3 knowledge file(s)
...
✅ Ingestion complete!
📊 Summary:
  ├─ Total chunks: 42
  ├─ Processed: 42
  ├─ Skipped (duplicates): 0
  └─ Errors: 0

5. Run Development Server

npm run dev

Open http://localhost:3000 and try asking a question!

6. Run Evaluations

npm run evals

---

📖 Documentation

Comprehensive docs in the docs/ directory:

• SCHEMA.md - Database schema and indexes
• INGEST.md - Ingestion process and chunking
• ANSWER-FLOW.md - Answer API architecture
• PRIVACY.md - Privacy guarantees and PII protection
• GO-LIVE-GATE.md - Production readiness checklist

---

🔐 Security & Privacy

Privacy Guarantees

✅ What we store:

• Query hash (HMAC-SHA256)
• Query length
• Metadata (timestamps, token usage)

❌ What we NEVER store:

• Raw query text
• User identifiers (beyond IP for rate limiting)
• PII of any kind

Security Measures

1. Rate Limiting: 10 requests/min/IP
2. Security Headers: CSP, HSTS, X-Frame-Options
3. PII Scrubbing: Automatic Sentry beforeSend hooks
4. Input Validation: Max query length, type checking
5. HTTPS Only: Enforced via HSTS

See PRIVACY.md for full details.

---

🧪 Testing & Evaluation

Nightly Evaluations

Automated quality testing runs nightly at 2 AM UTC:

# .github/workflows/nightly-evals.yml
- Skips scheduled runs when required evaluation secrets are not configured
- Fails manual dispatches when required evaluation secrets are missing
- Loads 20 test questions from data/evals/test-questions.jsonl
- Calls /api/answer for each question
- Validates response quality, citations, latency
- Uploads results as GitHub Actions artifact
- Fails only on severe regressions (< 50% quality)

Run manually:

npm run evals

Manual Testing

curl -X POST http://localhost:3000/api/answer \
  -H "Content-Type: application/json" \
  -d '{"query": "What is RAG?"}'

---

📊 API Reference

POST /api/answer

Generate an answer using RAG.

Request:

{
  "query": "What is RAG and how does it work?"
}

Response (200):

{
  "answer": "RAG (Retrieval Augmented Generation) is an AI architecture...",
  "citations": [
    {
      "source_url": "file://sample-1.md#chunk-0",
      "content": "RAG (Retrieval Augmented Generation) is...",
      "similarity": 0.87
    }
  ],
  "q_hash": "a3f2b9c1d4e5f6a7b8c9d0e1f2a3b4c5...",
  "q_len": 35,
  "tokensUsed": 456
}

Error (429):

{
  "error": "Too Many Requests",
  "message": "Rate limit exceeded. Please try again later."
}

See ANSWER-FLOW.md for complete API docs.

---

🛠️ Development

Scripts

npm run dev          # Start dev server
npm run build        # Production build
npm run start        # Start production server
npm run lint         # Run ESLint
npm run type-check   # TypeScript type checking
npm run ingest       # Run ingestion script
npm run evals        # Run evaluation suite

Project Structure

csbrainai/
├── app/
│   ├── api/answer/route.ts    # Answer endpoint
│   ├── layout.tsx             # Root layout
│   └── page.tsx               # Home page
├── components/
│   └── AnswerDemo.tsx         # Demo UI
├── data/
│   ├── evals/                 # Evaluation questions
│   └── knowledge/             # Knowledge base files
├── docs/                      # Documentation
├── lib/
│   ├── crypto-utils.ts        # HMAC hashing
│   ├── openai.ts              # OpenAI client
│   ├── rate-limiter.ts        # Rate limiting
│   ├── sentry-utils.ts        # PII scrubbing
│   └── supabase.ts            # Database client
├── scripts/
│   ├── evals-runner.ts        # Evaluation script
│   └── ingest.ts              # Ingestion script
├── supabase/migrations/       # Database migrations
├── middleware.ts              # Security & rate limiting
├── sentry.client.config.ts    # Sentry client
└── sentry.server.config.ts    # Sentry server

---

🚢 Deployment

Vercel (Recommended)

# Install Vercel CLI
npm i -g vercel

# Deploy
vercel --prod

Environment Variables

Set all variables from .env.example in Vercel dashboard:

• Project Settings → Environment Variables

Post-Deployment

1. Run database migration in Supabase
2. Run ingestion: npm run ingest (local or CI)
3. Verify Sentry integration
4. Test API endpoint
5. Check nightly evals workflow

See GO-LIVE-GATE.md for production checklist.

---

🔍 Monitoring

Sentry Dashboard

Monitor in production:

• Error rate (target: < 1%)
• Performance (p95 < 3s)
• Rate limit violations
• PII scrubbing effectiveness

Metrics to Track

• Latency: Median, p95, p99 response times
• Quality: Nightly eval scores
• Cost: OpenAI token usage
• Errors: 4xx/5xx rates

---

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit changes (git commit -m 'Add amazing feature')
4. Push to branch (git push origin feature/amazing-feature)
5. Open a Pull Request

---

📄 License

MIT License - see LICENSE for details.

---

🙏 Acknowledgments

• Supabase for pgvector support
• OpenAI for embeddings and LLM APIs
• Sentry for observability
• Next.js for the framework
• Upstash for serverless Redis

---

📞 Support

• Documentation: docs/
• Issues: GitHub Issues
• Security: Report vulnerabilities through GitHub Security Advisories for this repository.

---

Built by David Ortiz.