Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AGENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ Developer reference for agents and contributors. User-facing overview: [README.m
- `web/src/` -- React SPA (`AppRoutes.tsx`, `views/`, `components/`); pipeline UI: `PipelineRunnerFab`, `PipelineContext`
- `alembic/` -- schema migrations

**Local dev:** `./local-run` (Postgres in Docker `wp-pg`, FileService `:8080`, Data `:8091`, AiService `:8092`, ReportService `:8094`, FastAPI `:8001`, BFF `:8090`, Vite `:3000`; default `DATABASE_URL`: `postgres://postgres:dev@127.0.0.1:5432/website_profiling`). See `scripts/local-run.sh`. **Local tests:** `./local-test` runs **three** Python coverage gates (core 100%, reporting 100%, tools 100%) plus web and .NET checks — mirrors CI; Docker CI is separate (see `.github/workflows/ci.yml`). `./local-test browser` for `@pytest.mark.browser` integration tests — see `scripts/local-test.sh`. Mocked browser unit tests: `tests/test_browser_fetcher_unit.py`.
**Local dev:** `./local-run` (Postgres in Docker `wp-pg`, FileService `:8097`, Data `:8091`, AiService `:8092`, ReportService `:8094`, FastAPI `:8096`, BFF `:8090`, Vite `:3000`; default `DATABASE_URL`: `postgres://postgres:dev@127.0.0.1:5432/website_profiling`). See `scripts/local-run.sh`. **Local tests:** `./local-test` runs **three** Python coverage gates (core 100%, reporting 100%, tools 100%) plus web and .NET checks — mirrors CI; Docker CI is separate (see `.github/workflows/ci.yml`). `./local-test browser` for `@pytest.mark.browser` integration tests — see `scripts/local-test.sh`. Mocked browser unit tests: `tests/test_browser_fetcher_unit.py`.

**JavaScript crawl (optional):** Config keys `crawl_render_mode` (`static` | `javascript` | `auto`) and `crawl_js_*` in pipeline config / `pipelineConfigSchema.ts`. JS/auto crawls can capture browser console errors and uncaught exceptions (`crawl_js_capture_console`, stored under `page_analysis.browser`). **Auto mode** uses static-first fetch, pre-parse SPA heuristics (`needs_js_render`), then post-parse low-outlink fallback (`needs_js_render_after_parse`) in `crawler.py`. **Preflight:** `GET /api/crawl/browser-status` (localhost) spawns Python `browser_status()`; Run audit settings/run validation calls it when render mode is `javascript` or `auto`. Browser deps: Playwright from `requirements.txt` (installed by `./local-run setup` and `./local-test`). Runtime needs Chromium on `PATH` or `CHROME_PATH` (Docker sets `CHROME_PATH=/usr/bin/chromium`). Integration tests: `@pytest.mark.browser` — excluded by default in `pytest.ini`; Docker CI runs `tests/test_crawl_fetchers.py` and `tests/test_crawler_browser_e2e.py -m browser`; locally `./local-test browser`.

Expand Down
2 changes: 1 addition & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ This file is the canonical entry point for agents. For full detail see [AGENT.md
- `services/ReportService/` — .NET report build + pipeline orchestration (port 8094). See [services/ReportService/README.md](services/ReportService/README.md)
- `services/AiService/` — .NET AI service (Microsoft.Extensions.AI, chat, enrichment, MCP, **secrets/llm-settings writes**; port 8092). See [services/AiService/README.md](services/AiService/README.md)
- `services/IntegrationsService/` — .NET Google/Bing integrations (GSC/GA4 fetch, OAuth, page-live, keyword reads; port 8093). See [services/IntegrationsService/README.md](services/IntegrationsService/README.md)
- `services/FileService/` — .NET PDF + Excel workbook export (port 8080). HTTP-only via `REPORT_API_URL`; no Postgres. Profiles: `executive|standard|full|premium`. Details: [services/FileService/README.md](services/FileService/README.md). Env: `FILE_SERVICE_URL` (MCP), `REPORT_API_URL` (FileService).
- `services/FileService/` — .NET PDF + Excel workbook export (port 8097). HTTP-only via `REPORT_API_URL`; no Postgres. Profiles: `executive|standard|full|premium`. Details: [services/FileService/README.md](services/FileService/README.md). Env: `FILE_SERVICE_URL` (MCP), `REPORT_API_URL` (FileService).
- `alembic/` — DB migrations
- `services/WebsiteProfiling.slnx` — unified .NET solution (all six services + shared libs)
- `docs/` — documentation index
Expand Down
4 changes: 2 additions & 2 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# syntax=docker/dockerfile:1
# WebsiteProfiling: FastAPI (port 8001) + Python worker + pipeline.
# WebsiteProfiling: FastAPI (port 8096) + Python worker + pipeline.
# Web UI is a separate image: web/Dockerfile (Vite SPA + nginx).
# Build from repository root: docker build -t website-profiling .
# BuildKit cache mounts (default in Docker Desktop) reuse pip/npm downloads across rebuilds.
Expand Down Expand Up @@ -72,6 +72,6 @@ ENV NODE_ENV=production
# Persisted data directory (secrets + shadow config)
RUN mkdir -p /data && chmod +x /app/docker-entrypoint.sh

EXPOSE 8001
EXPOSE 8096

CMD ["/app/docker-entrypoint.sh"]
16 changes: 8 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,12 +149,12 @@ flowchart TB

subgraph BffRoutes["BFF upstream routing"]
direction LR
BFF --> FastAPI["fastapi :8001<br/>Crawl · jobs · pipeline settings"]
BFF --> FastAPI["fastapi :8096<br/>Crawl · jobs · pipeline settings"]
BFF --> Report["report :8094<br/>Report build · orchestration"]
BFF --> Integrations["integrations :8093<br/>Google/Bing · GSC/GA4 · keywords"]
BFF --> Ai["ai :8092<br/>Chat · LLM · secrets · MCP"]
BFF --> Data["data :8091<br/>Report reads · portfolio · issues"]
BFF --> Files["files :8080<br/>PDF · Excel export"]
BFF --> Files["files :8097<br/>PDF · Excel export"]
end

subgraph Pipeline["Background pipeline"]
Expand Down Expand Up @@ -208,7 +208,7 @@ flowchart TB
```
WebsiteProfiling/
├── src/website_profiling/ # Python audit engine (CLI: python -m src)
│ ├── api/ # FastAPI app (uvicorn :8001)
│ ├── api/ # FastAPI app (uvicorn :8096)
│ ├── worker/ # Background pipeline job runner
│ ├── crawl/ # Crawler, fetchers, JS rendering
│ ├── reporting/ # Report builder (Python bridge; native build in ReportService)
Expand All @@ -233,7 +233,7 @@ WebsiteProfiling/
├── services/IntegrationsService/ # .NET Google/Bing integrations — OAuth, GSC/GA4, keywords (port 8093)
├── services/AiService/ # .NET AI — chat, secrets, LLM config, MCP, enrichment (port 8092)
├── services/Data/ # .NET read service — report/portfolio/issue reads (port 8091)
├── services/FileService/ # .NET PDF + Excel workbook export (port 8080)
├── services/FileService/ # .NET PDF + Excel workbook export (port 8097)
├── alembic/versions/ # PostgreSQL schema migrations
├── tests/ # pytest suite + fixtures
├── docs/ # Glossary, MCP, ops, brand assets
Expand Down Expand Up @@ -291,7 +291,7 @@ Build and run the full dev stack from source:
docker compose up --build
```

Services: **postgres**, **fastapi** (`:8001`, internal), **worker**, **report** (`:8094`, internal), **integrations** (`:8093`, internal), **ai** (`:8092`, internal), **data** (`:8091`, internal), **bff** (`:8090`), **web** (`:3000`), **files** (`:8080`, internal).
Services: **postgres**, **fastapi** (`:8096`, internal), **worker**, **report** (`:8094`, internal), **integrations** (`:8093`, internal), **ai** (`:8092`, internal), **data** (`:8091`, internal), **bff** (`:8090`), **web** (`:3000`), **files** (`:8097`, internal).

Open [http://localhost:3000/home](http://localhost:3000/home). The browser talks only to the **BFF** (`:8090`); the BFF proxies to FastAPI (crawl/pipeline), IntegrationsService (Google/Bing), AiService (AI/secrets), Data (report reads), and FileService (PDF/workbook export).

Expand All @@ -308,7 +308,7 @@ Production deployment: `docker-compose.prod.yml` — set `POSTGRES_USER`, `POSTG
./local-prod # Same DB, Vite production build + preview (no hot reload)
```

`./local-run` starts (in order): **FileService** `:8080`, **Data** `:8091`, **AiService** `:8092` (MCP HTTP enabled), **ReportService** `:8094`, **IntegrationsService** `:8093`, **ConfigService** `:8095`, **FastAPI** `:8001` (Python bridge), **BFF** `:8090`, and **Vite** `:3000`. Use `localhost` (not `127.0.0.1`) for pipeline APIs so CORS and cookies match the BFF origin.
`./local-run` starts (in order): **FileService** `:8097`, **Data** `:8091`, **AiService** `:8092` (MCP HTTP enabled), **ReportService** `:8094`, **IntegrationsService** `:8093`, **ConfigService** `:8095`, **FastAPI** `:8096` (Python bridge), **BFF** `:8090`, and **Vite** `:3000`. Use `localhost` (not `127.0.0.1`) for pipeline APIs so CORS and cookies match the BFF origin.

Default local `DATABASE_URL`: `postgres://postgres:dev@127.0.0.1:5432/website_profiling` (Docker Compose dev stack uses `profiling:profiling`).

Expand All @@ -324,8 +324,8 @@ With `./local-run`, each service runs in **Development** and exposes interactive
| **IntegrationsService** | 8093 | [http://localhost:8093/docs](http://localhost:8093/docs) | `/swagger/v1/swagger.json` |
| **ReportService** | 8094 | [http://localhost:8094/docs](http://localhost:8094/docs) | `/swagger/v1/swagger.json` |
| **ConfigService** | 8095 | [http://localhost:8095/docs](http://localhost:8095/docs) | `/swagger/v1/swagger.json` |
| **FileService** | 8080 | [http://localhost:8080/docs](http://localhost:8080/docs) | `/swagger/v1/swagger.json` |
| **Python bridge** (FastAPI) | 8001 | [http://localhost:8001/docs](http://localhost:8001/docs) | `/openapi.json` |
| **FileService** | 8097 | [http://localhost:8097/docs](http://localhost:8097/docs) | `/swagger/v1/swagger.json` |
| **Python bridge** (FastAPI) | 8096 | [http://localhost:8096/docs](http://localhost:8096/docs) | `/openapi.json` |

Swagger UI is disabled when `ASPNETCORE_ENVIRONMENT=Production`. The legacy `web/openapi.json` reflects the old monolithic FastAPI app and is not a complete map of the current C# services.

Expand Down
Loading
Loading