Oley v3

Open-Source Web Intelligence & Data Extraction Engine (Firecrawl & Perplexity Alternative)

A self-hostable Web Intelligence & Data Extraction Engine designed for LLMs, RAG pipelines, and agentic workflows. It provides a clean, unified API for scraping, crawling, structured data extraction, web search, screenshots, PDFs, and deep research synthesis.

🇬🇧 English • 🇪🇸 Español • 🇨🇳 中文 • 🇯🇵 日本語

---

📖 Table of Contents

• About Oley
• Key Features
• Quick Start
• Docker Deployment
• Environment Variables
• Unified API Endpoint: POST /api/fire
• Usage Examples
• Specialty & Streaming Routes
• SDK Clients
• Directory Structure
• License

---

🔍 About Oley

Oley is a production-ready, open-source alternative to Firecrawl and Perplexity. It acts as a bridge between the raw web and Large Language Models (LLMs). It solves common challenges in web data harvesting, including client-side JS rendering, rate limiting, anti-bot protections, and parsing unstructured HTML into LLM-ready formats (like clean GFM Markdown or structured JSON).

Why Oley?

• All-in-One Engine: No need for separate microservices for crawling, scraping, searching, or generating screenshots.
• Agent-Friendly: Automatically cleans up noise (ads, tracking scripts, navigation elements) to reduce LLM context token usage.
• Deep Research Capability: Built-in multi-agent planner that acts like Perplexity's deep search, returning structured facts with precise source citations.

---

🚀 Key Features

• Unified /api/fire Endpoint: One simple interface to rule all actions—auto-detects targets from your payload.
• LLM Schema Extraction (Firecrawl Parity): Extract structured JSON data using custom schemas (e.g. products, reviews, pricing) with natural language prompt guidance.
• Deep Research Engine (Perplexity Parity): Expands queries, executes parallel web searches, scrapes relevant sources, and synthesizes citation-backed answers.
• Anti-Bot Stealth: Rotates user-agents, spoofs canvas/timezone fingerprints, connection parameters, and WebGL context using a randomized pool of real GPUs.
• Advanced Scraper Options: Injects custom cookies/headers, clicks/fills inputs, evaluates custom JS snippets, and waits for selectors before extraction.
• GFM Markdown Engine: Produces clean GitHub-Flavored Markdown tables, nested lists, and checkboxes from dynamic HTML pages.
• $O(1)$ LRU Cache: Memory-efficient response cache with TTL configuration and route-specific cache invalidation support.
• Live SSE Streaming: Stream tokens or research milestones in real-time.
• Zero Dependencies SDKs: Pre-packaged clients for TypeScript and Python.

---

⚡ Quick Start

Running Locally (Node 18+)

1. Clone the repository and navigate into it:

   git clone https://github.com/Asno-dev/asno-ai.git oley && cd oley

2. Copy and customize the environment configurations:

   cp .env.example .env

3. Install dependencies and launch the developer environment:

   npm install
   npm run dev

   Your unified server is now running at http://localhost:3000.

---

🐳 Docker Deployment

To spin up the service in a containerized environment (which automatically sets up Chrome/Playwright dependencies):

docker compose up --build

---

🌐 Environment Variables (api/.env)

Customize your server behaviors by defining these keys inside your .env file:

PORT=3000                       # Web server port
OLEY_API_KEY=your-api-key       # Optional: Enforces Bearer token auth
RATE_LIMIT_PER_MIN=60           # Max requests per minute per IP
PUBLIC_URL=http://localhost:3000

# LLM Providers (Required for AI actions, schema extraction & deep research)
OPENAI_API_KEY=your_key
ANTHROPIC_API_KEY=your_key
GOOGLE_API_KEY=your_key
GROQ_API_KEY=your_key
MISTRAL_API_KEY=your_key

# Local LLM Support (Ollama)
OLLAMA_BASE_URL=http://localhost:11434/v1
OLLAMA_MODEL=llama3.1

---

🔥 The Unified API Endpoint: POST /api/fire

Submit requests to /api/fire to automatically route and run any intelligence job.

Parameter Reference

Parameter	Type	Default	Description
url	string	-	Target URL for scraping, screenshot, crawl, or extraction.
query	string	-	Query term for web search or deep research.
action	string	-	Explicit task: scrape, crawl, extract, llmExtract, search, screenshot, pdf, render, research, translate, etc. (Auto-detected if left empty).
formats	array	["markdown"]	Desired outputs: markdown, html, text, metadata, links, images, screenshot, pdf.
renderJs	boolean	false	Enable Playwright JS rendering context.
stealth	boolean	false	Apply Canvas, GPU, and timezone anti-fingerprint blocks.
blockResources	boolean	false	Block css/fonts/images from loading for speed.
waitForSelector	string	-	Wait for CSS selector to appear in DOM before scraping.
actions	array	-	Interactivity script: [{"type": "click", "selector": "#btn"}].
cookies	array	-	Inject context cookies: [{"name": "a", "value": "b", "domain": "..."}].
extractSchema	object	-	Shorthand JSON schema for structured extraction.
prompt	string	-	Instructs the extraction parser in plain English.
cache	boolean	true	Enable $O(1)$ LRU response caching.
ttl	number	300000	Custom cache duration in milliseconds.

---

🛠 Usage Examples

1. Simple Scrape to Markdown (cURL)

curl -X POST http://localhost:3000/api/fire \
  -H "Content-Type: application/json" \
  -d '{"url": "https://github.com/Asno-dev/asno-ai", "formats": ["markdown"]}'

2. Structured LLM Data Extraction

curl -X POST http://localhost:3000/api/fire \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://github.com/Asno-dev/asno-ai",
    "action": "llmExtract",
    "extractSchema": {
      "repositoryName": "string",
      "starCount": "number",
      "author": "string"
    }
  }'

3. Deep Research Synthesis

curl -X POST http://localhost:3000/api/fire \
  -H "Content-Type: application/json" \
  -d '{
    "query": "superconductor news updates",
    "action": "research",
    "depth": "deep"
  }'

4. PowerShell Usage (unified RestMethod)

Invoke-RestMethod -Uri "http://localhost:3000/api/fire" `
  -Method Post `
  -ContentType "application/json" `
  -Body '{"url": "https://github.com/Asno-dev/asno-ai", "formats": ["markdown"], "stealth": true}'

---

📡 Specialty & Streaming Routes

Real-Time Research Streams (POST /api/ai/research/stream)

Allows your clients to trace deep research progress iteratively.

curl -N -X POST http://localhost:3000/api/ai/research/stream \
  -H "Content-Type: application/json" \
  -d '{"query": "solid state battery breakthroughs"}'

Emits live Event Source records (event: start, event: sources_found, event: answer, event: done).

Real-Time Chat Stream (POST /api/ai/chat/stream)

Direct LLM completion streaming.

curl -N -X POST http://localhost:3000/api/ai/chat/stream \
  -H "Content-Type: application/json" \
  -d '{"prompt": "Write a Python script for binary search."}'

Cache Management

Keep scrapers synchronized and discard stale cache entries instantly:

• Check Stats: GET /api/cache/stats
• Invalidate URL: POST /api/cache/invalidate (payload: {"url": "https://url-to-invalidate.com"})
• Flush Cache: POST /api/cache/clear

---

📦 Zero-Dependency SDK Clients

Oley includes clean client SDK wrappers for TS and Python under the sdks/ directory.

TypeScript SDK

import { Oley } from 'oley';

const oley = new Oley({ baseUrl: 'http://localhost:3000' });
const result = await oley.fire({
  url: 'https://example.com',
  formats: ['markdown'],
  stealth: true
});

Python SDK

from oley import Oley

oley = Oley(base_url="http://localhost:3000")
res = oley.fire(
    url="https://example.com",
    formats=["markdown"],
    stealth=True
)

---

🏗 Directory Architecture

api/src/
  index.ts              # System Entry point & browser pool
  routes.ts             # REST router & SSE stream controllers
  openapi.ts            # OpenAPI 3.1.0 generator schema
  core/
    fire.ts             # Unified request engine
    scraper.ts          # Playwright & Turndown GFM compiler
    cache.ts            # O(1) LRU caching layer
    stealth.ts          # Browser anti-fingerprint spoofing
    research.ts         # Multi-query deep research builder
    synthesis.ts        # BM25 ranker & Citation synthesize formatter
    stream.ts           # Server-Sent Events formatter
  extract/
    llm_extract.ts      # LLM JSON-schema mapper
    ai.ts               # Summarization, translations, entities pipelines

---

License

MIT © asno-dev