For example

WebKit http://build.webkit.org/waterfall

Chromium http://build.chromium.org/p/chromium/waterfall

Firefox https://tbpl.mozilla.org/

These systems generally let you submit a patch from version control, review the patch with others and then finally submit the patch to a semi-automated system that applies the patch, builds, runs tests, and then commits the patch if there were no errors.

In an idea world the automated systems would take one patch at a time, build with that patch, run all the tests, if any of them failed reject the patch, then move on to the next patch.

Unfortunately it's never that simple. With 100s of programmers the patches come fast and furious. Faster than they can be built. A complete build cycle with all tests can take hours. Sharding (spliting the build and tests across multiple machines) helps but even that has its limits.

Some systems therefore start a build once the previous build has finished and take all patches that came in since the last build started which can be anywhere from 1 to 30 patches. They run the tests, if there are any failures the system shuts down and someone manually has to look at the last N patches and try to figure out which patch needs to be rejected.

So, what's your experience here? Remember this is specific to LARGE projects. Projects with 100s of programmers. Projects with tens of thousands of tests.

- Have you done anything special to speed things up?

- Have you tried auto-rejecting all patches when one is bad?

- Have you figured out a way to do one patch at a time and still keep up?

- Have you subdivided patches after a failure and still been able to keep up?

- Have you used any special build software that has special features that help this problem?

What are your best tips or practices?