I keep running into a frustrating issue with PyTorch DDP on a single node (2–8 GPUs): adding GPUs sometimes makes training slower (not scale proportionally), and it is hard to tell what's actually gating the step.

In practice I see:

one rank silently becomes the “worst rank” and gates every step

step time spikes where GPUs look idle, but it’s unclear if the culprit is dataloader stalls, CPU contention, batch/sequence-length imbalance, or NCCL sync

Questions for folks who run multi-GPU training:

What do you suspect first when scaling regresses on a single node?

What signals do you look at to distinguish data vs compute vs comms/sync?

Any repeatable workflow / checklist that gets you to root cause fast?

Context: I am building a small OSS tool that shows live per-rank step timing + stall attribution (always-on; not a replacement for PyTorch Profiler/Nsight). If you have a workload where DDP scaling is weird and you are willing to run a ~10-minute test, I am happy to help interpret results and prioritize support for your setup.

Repo: https://github.com/traceopt-ai/traceml