Optimizing Apache Spark & Delta Lake for High Throughput

Apache Spark paired with ACID table formats like Delta Lake or Apache Iceberg forms the backbone of modern Lakehouse architectures. However, out-of-the-box configurations frequently bottleneck on small-file problems, shuffle spills, and partition skew.

1. Tackling the Small File Problem: Auto-Optimize & Compaction

Small files lead to excessive S3/GCS API metadata operations and executor overhead.

ALTER TABLE main_events SET TBLPROPERTIES (
  'delta.autoOptimize.optimizeWrite' = 'true',
  'delta.autoOptimize.autoCompact' = 'true'
);

-- Z-Order by tenant and event timestamp
OPTIMIZE main_events 
ZORDER BY (tenant_id, event_timestamp);

2. Broadcast Joins vs. Salting for Data Skew

When joining a massive fact table with a dimensional table where key distribution is severely skewed, standard SortMergeJoin causes executor OOM errors.

# Force Broadcast Join for Dimension Tables under 100MB
from pyspark.sql.functions import broadcast

df_joined = df_large_fact.join(
    broadcast(df_small_dim),
    on="dim_key",
    how="inner"
)