In principle, the hierarchical organization could allow us to make sense of more complex problem spaces and reach a higher functionality, but it seems difficult to find examples of concrete research done in the past regarding this. I've found a few sources:

https://www.teco.edu/~albrecht/neuro/html/node32.html

https://vtechworks.lib.vt.edu/bitstream/handle/10919/27998/etd.pdf?sequence=1&isAllowed=y

A few concrete questions I have:

Are there any tasks where MNNs have shown better performance than large single nets?

Could MNNs be used for multimodal classification, i.e. train each net on a fundamentally different type of data, (text vs image) and feed forward to a higher level intermediary that operates on all the outputs?

From a software engineering perspective, aren't these more fault tolerant and easily isolatable on a distributed system?

Has there been any work into dynamically adapting the topologies of subnetworks using a process like Neural Architecture Search?

Generally, are MNNs practical in any way?

Apologies if these questions seem naive, I've just come into ML and more broadly CS from a biology/neuroscience background and am captivated by the potential interplay.

I really appreciate you taking the time and lending your insight!