thelake

Open source · Apache-2.0

thelake

The Open Source LangSmith / Datadog Alternative Built on DuckDB.

Built for long-term storage of large full payloads — 200x cheaper than traditional observability platforms.

Star on GitHub API docs

Why thelake

Keep the full payload. Drop the indexing tax.

Traditional observability platforms charge you to index everything, then force sampling and short retention. thelake keeps large prompts, responses, tool I/O, and HTTP bodies as open Parquet in a DuckLake — designed so long-term full-fidelity storage stays economical.

Traditional

Index-heavy observability

Heavy indexes, short retention windows, sampled bodies — cost climbs with every byte you want to keep forever.

thelake

Storage-first evidence lake

Storage-first economics on object-store Parquet. Retain complete AI and application payloads for years, then query them with DuckDB SQL when you need them.

What you get

Full payloads. Long retention. Open SQL.

Architecture aimed at 200x lower long-term cost than index-heavy observability stacks — without discarding the bodies you need later.

  1. 01

    Full payloads, no sampling

    OTLP traces and logs into a tenant-bound runtime. Keep prompts, tool calls, HTTP bodies, and outcomes connected — not truncated for an incident window.

  2. 02

    Long-term lake storage

    DuckLake + Parquet on your object store or disk. Built for years of large recordings you can reopen, evaluate, and regress against.

  3. 03

    Query when it matters

    DuckDB SQL on snapshot-correct tables, plus tenant-controlled column promotion when a field becomes hot — without a second proprietary warehouse.

How it works

One durable write path

OTLP in; open SQL and query-compatible APIs out.

  1. OTLP HTTP / gRPC
  2. thelake tenant-bound runtime
  3. DuckLake Parquet + catalog
  4. SQL DuckDB · Grafana query APIs
  • OpenTelemetry OTLP write
  • gRPC ingest
  • DuckDB SQL engine
  • DuckLake catalog
  • Apache Parquet columnar files
  • Grafana datasources
  • Loki query API
  • Tempo query API

Loki- and Tempo-compatible query APIs are available so existing Grafana datasources can read the lake. Product metrics / PromQL are out of scope. OTLP remains the sole write path for traces and logs.

Open source + commercial

Open source lake. Powers Softprobe Agent QA.

thelake is the durable evidence foundation underneath Softprobe Agent QA — self-host the lake, or use it managed inside Softprobe.

thelake

Self-host the Rust runtime: OTLP ingest, DuckLake storage, DuckDB SQL, promotion, and telemetry search. Apache-2.0.

github.com/softprobe/thelake

Softprobe Agent QA

Built on thelake. Softprobe Agent QA adds Explorer, diagnosis, and collaboration on top of the same evidence lake — hosted when you want the product surface without operating thelake yourself.

softprobe.ai

Quick start

Run it locally

Local MinIO + DuckLake Postgres, then build and run. Details live in the repository README.

make setup
make build
export CONFIG_FILE=config.yaml
cargo run --bin thelake
Open the README