I found out that I could cut the startup time (importing/ model loading time) in a project in half, after I started optimizing. (Running a profiler and finding out what takes time and cutting unused imports/stuff from used libraries and doing the work in parallel for time consuming stuff).

This lead to faster bug fixing and testing because less time was spend at startup. So I was able to add features and test them with multiple restarts in half the time without loosing focus on the waiting time, making development way more smooth.

In conclusion, yeah leave out optimization at start, but as soon you have something working, make sure you are not wasting time while developing.