About

What this is

POHBOS is a learning project I built after reading Cloudflare’s engineering write-up on scaling anomaly detection.

The project implements a scoring pipeline that uses the Histogram Based Outlier Scoring to analyze anomalous HTTP behavior. Events enter through an ingestor, move through Kafka-compatible topics, get projected into durable storage, and are evaluated by a detector that maintains each visitor’s trailing window. Eligible windows are scored against a source-specific HBOS baseline, mapped through a policy, and emitted as decisions.

Repository structure

Generally speaking the repository is structured in the following manner

├── services/
│   ├── pipeline/        # Streaming detection pipeline
│   └── analytics/       # Daily risk insights from stored pipeline data
├── infra/               # Airflow orchestration, Redpanda topics, and Postgres migrations
├── tools/
│   └── replay/          # Python CLI for replaying HTTP traffic using eclog dataset
└── apps/
  └── showcase/        # Astro/Svelte documentation and demo site

Pipeline services

services/pipeline is one Go module with multiple service binaries and shared internal packages.

services/pipeline/
├── cmd/ 
|    # Binary/service composition logic including: lifecycle management, configuration and initilization. 
│   ├── ingestor/      
│   ├── detector/ 
│   ├── sink/           
│   ├── baseline-worker/
│   └── queryapi/       
└── internal/
  ├── app/             # service-specific application logic
  ├── broker/          # Kafka-compatible producer and consumer wrappers
  ├── domain/          # event and decision domain types
  ├── envelope/        # stream message envelopes
  ├── extractor/       # trailing-window features and predicates
  ├── hbos/            # histogram model, transforms, validation, scoring
  └── postgres/        # durable storage access

Infrastructure

The development infrastructure stack is managed through Docker Compose. Redpanda provides the Kafka-compatible broker. Postgres acts as the external storage instance that stores events, decisions, policies, baseline runs, and labels.

infra/
├── airflow/
│   ├── dags/
│   │   └── analytics_publish.py # Weekly analytics workflow
│   └── Dockerfile               # Airflow runtime image
├── kafka/
│   └── create_topics.sh # raw events, decisions, DLQ, baseline signals
└── migrations/
  ├── 001_policy_versions.sql
  ├── 002_events.sql
  ├── 003_baseline_runs.sql
  ├── 004_decisions.sql
  └── 005_labels.sql

Replay tool

The replay tool is a Python CLI used to drive realistic traffic into the ingestor. It reads rows from EClog, a public dataset containing HTTP access traffic from a real e-commerce website, maps those rows into the ingestor payload shape, and sends them over HTTP with rate and concurrency controls.

tools/replay/
├── pyproject.toml       # uv project and CLI dependencies
└── src/traffic_replay/
  ├── cli.py           # command entrypoint
  ├── adapters/        # EClog parsing and mapping
  ├── client.py        # ingestor HTTP client
  ├── rate.py          # pacing and rate control
  └── stats.py         # run statistics

Using EClog keeps the pipeline grounded in real access-log behavior, so the replay tool exercises ingestion, windowing, scoring, and backpressure against traffic patterns that hand-written examples would miss.

Analytics service

The analytics service creates daily risk insights for each analyzed source from stored events, detector decisions, and baseline runs.

It summarizes traffic, scoring coverage, risk levels, recommended actions, score distributions, and baseline usage for the showcase.

services/analytics/
├── src/pohbos_analytics/ # Snapshot extraction and inspection CLI
├── dbt/                  # Daily models, tests, and Parquet export
├── data/                 # Local snapshots and DuckDB warehouse
└── tests/                # Python command and extraction tests

See how the analytics pipeline works →

Showcase app

The showcase is the public read surface, what you’re currently reading. It explains the architecture, documents service responsibilities, and renders the pipeline as an interactive system diagram.

apps/showcase/
├── src/pages/           # Astro routes
├── src/content/docs/    # MDX content pages
├── src/layouts/         # Docs and immersive page shells
├── src/features/
│   ├── docs/            # documentation components
│   ├── navigation/      # sidebar, footer, mobile toggle
│   ├── pipeline/        # interactive pipeline visualization
│   └── brand/           # project wordmark
└── src/styles/          # global tokens and base styling

This showcase is still a work in progress. The next pieces are a live SSE stream for decisions moving through the pipeline, and an interactive session crafter for generating visitor behavior against the ingestor.

The showcase also presents the daily risk summary through browser-side DuckDB queries.

Tech stack