Data Engineering Dispatch · Est. 2024
A weekly dispatch for senior data engineers, analytics engineers, and platform leads. Iceberg tables. Query engines. Catalog wars. Every partition labeled. Every schema glowing clean.
12,400 subscribers
62% open rate
MEDALLION ARCHITECTURE · LIVE EXPLORER production stack · us-east-1 · last updated 2m ago
INGEST Raw / Bronze TRANSFORM Cleansed / Silver SERVE Aggregated / Gold CATALOG Metadata / Glue ↑ click any node to trace lineage
Raw / Bronze
Cleansed / Silver
Aggregated / Gold
Metadata / Glue
ARCHITECTURE
Lakehouse Reference Stack
SAMPLE ISSUE
Issue #47: Iceberg at Scale
COMMUNITY
Top Threads This Week
FORMAT WARS
Delta vs. Iceberg vs. Hudi
Click Bronze → Silver → Gold nodes to trace data lineage through the medallion architecture.
STACK OVERVIEW
Storage S3 / GCS / ADLS Gen2 object storage, ~$23/TB/mo
Table Format Apache Iceberg v2 row-level deletes, partition evolution
Catalog AWS Glue + Nessie multi-branch schema dev
Transform dbt + Spark 3.5 847 models, 4.2min avg runtime
Query Trino 435 p50: 0.8s on Gold tables
Orchestration Airflow 2.8 340 DAGs, 94% SLA
PARTITION SCHEMA
CREATE TABLE lakehouse.gold.user_events (
event_id UUID ,
user_id BIGINT ,
event_type VARCHAR ,
event_ts TIMESTAMP ,
properties MAP (VARCHAR , VARCHAR )
)
USING iceberg
PARTITIONED BY (
days (event_ts),
bucket (32 , user_id)
)
TBLPROPERTIES (
'write.target-file-size-bytes' = '134217728' ,
'write.distribution-mode' = 'hash'
);
BY THE NUMBERS
0
Subscribers
senior data engineers
0
Issues published
every Monday, no exceptions
0
Discord members
94 online right now
0%
Open rate
industry avg is 21%
COMMUNITY CONTRIBUTORS engineers who write, not just read
PN
Priya Nair
Sr. Data Engineer
Confluent
MW
Marcus Webb
Analytics Eng Lead
DoorDash
SR
Sanjana Rao
Platform Lead
Robinhood
TE
Tom Eriksson
Data Architect
Klarna
AO
Aisha Okonkwo
Staff Data Eng
Stripe
JK
James Kowalski
Platform Lead
Stripe
YOU'VE SEEN THE CONTENT. NOW GET IT WEEKLY.
Every Monday. No fluff. Join 12,400 engineers who read Lakehouse instead of Slack on Monday mornings. Deep dives, benchmarks, hot takes — all written by engineers who ship.