Most data engineers can write a Spark job. But writing one that’s actually fast? That’s where things get interesting.
I’ve spent years working on large-scale data pipelines, and time and again I see the same performance mistakes show up in Spark jobs — even from experienced engineers. The good news: most of them are easy to fix once you know what to look for.
Here are 5 Spark optimization tricks that have saved me hours of compute time and a lot of frustrated debugging.
1. Stop Triggering Unnecessary Shuffles
Shuffles are Spark’s most expensive operation. When a shuffle happens, data is redistributed across the network — taking time, memory, and I/O. Operations that trigger shuffles: groupBy(), wide join(), distinct(), repartition().
The fix: use broadcast joins for small tables.
from pyspark.sql.functions import broadcast
result = large_df.join(broadcast(small_lookup_df), on="id", how="left")2. Get Your Partition Count Right
Target ~128MB per partition. Too few = idle cores. Too many = scheduling overhead.
df.rdd.getNumPartitions()
df = df.repartition(200)
df = df.coalesce(50)Use coalesce() when reducing, repartition() when increasing or needing even distribution.
3. Cache DataFrames You Use More Than Once
Spark’s lazy evaluation means it recomputes DataFrames from scratch each time an action is triggered — unless you cache them.
df = spark.read.parquet("s3://my-bucket/data/")
df.cache()
count = df.count()
filtered = df.filter(...) # uses cache4. Switch to Kryo Serialization
Kryo can be 2–10x faster than Java’s default serializer. One config line:
spark = SparkSession.builder .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer") .getOrCreate()5. Actually Read the Spark UI
The Spark UI (at http://localhost:4040) shows exactly where time is wasted — skewed partitions, slow stages, missed caches. Check the Stages tab, DAG visualization, and Executors tab. A task taking 5 minutes while 199 others take 10 seconds? That’s partition skew — fix it by salting your join key.
Putting It All Together
Checklist for every new Spark job: avoid shuffle-heavy operations where possible, check partition count, cache reused DataFrames, enable Kryo, and inspect the Spark UI after a test run. Optimizing Spark is systematic — find the bottleneck, understand it, fix it.
— Pushpjeet Cholkar, Data Engineer
Leave a Reply