OutForDelivery

Kafka · Spark · Delta Lake · dbt · AWS

Real-time parcel delivery operations pipeline: Kafka lifecycle events into Spark Structured Streaming and a Delta lakehouse, with a live SLA-risk dashboard on FastAPI and MapLibre.

Use case

Replicates the operations room of a parcel carrier: live status breakdowns, throughput per hub, a map of parcels moving across the Netherlands, SLA-breach risk flags on a compressed clock, and history marts for on-time rate, failure analysis and weather impact.

Architecture

A simulator produces parcel lifecycle events into Kafka (Redpanda) with Avro schemas enforced by a schema registry. Spark Structured Streaming lands them in Bronze and Silver Delta Lake medallion layers, dbt builds the gold marts on DuckDB locally or Athena on AWS, and DynamoDB serves the live view through FastAPI to a MapLibre and Chart.js dashboard. Storage is MinIO locally and S3 in the cloud; the whole stack runs on Docker Compose at zero cost or deploys with Terraform to AWS, where SNS and Lambda handle SLA alerts.