The problem:

Most distributed training assumes homogeneous GPUs and high-bandwidth interconnects (NVLink/InfiniBand). On heterogeneous fleets over standard internet, tensor/data parallel approaches become communication-bound and fragile.

What Zagora does under the hood:

- Uses pipeline-style parallelism instead of heavy tensor synchronization.

- Passes only boundary activations between stages rather than full parameter sync.

- Assigns layers proportionally to GPU capability to reduce straggler idle time.

- Uses checkpoint-based recovery to tolerate worker crashes.

- Supports adapter-based fine-tuning (e.g., QLoRA) to reduce memory pressure.

Zagora currently supports managed runs (we provision GPUs in-region) and a BYOC mode where users run workers on their own infrastructure.

Limitations:

- Full-parameter fine-tuning is not supported yet.

- It won't beat an NVLink cluster on raw throughput.

- Cross-region training is still latency-sensitive.

- Heterogeneous nodes scheduling is an ongoing tuning problem.

IMPORTANT:

I'm currently running jobs manually, so it may take some time before training starts. However, I will run every submitted job.

Link: app.zagora.ai

I'd be interested in feedback from people who've worked on distributed training at scale.

Happy to answer technical questions.