95% of My PySpark Job Finished in 4 Minutes. The Last Task Took 40. Here's Why.translating…

I had a PySpark job joining a large transactions table with a customer dimension table. Nothing exotic — a standard join, then an aggregation. On paper, it looked like it should scale fine across the cluster. In practice, the job would race through most of its tasks and then…