Caching: factorizing / toughening

Created by: valschneider

I've been struggling to reproduce the bugs generated by caching (I did encounter them in the past so I'm not doubting their existence), as such I mostly went through the code and improved what I thought needed to be improved.

I'm not re-enabling caching in this PR, but I will try to convince some people to re-enable it on their side and do some bug-finding for me.