I wouldn’t recommend going with less standoffs than designed. They are included not to made a rugged Case (a 3mm sheet plexi is way more robust than a PCB…) its to minimize the mechanical stress on the PCB. It carries 6 heavy voice cards and you have that encoder and buttons that will constantly be pushed. Albeit OK for a short period of time a PCB won’t stand much bends and will fail relatively fast from loads. So please don’t consider the PCB as supporting structure - itself must be supported.
Next thing its that 12mm clearance is defined for the Voicecards; matches ventilation need and future designs might need all this space and it would be sad if you only could mount 3 Cards because of you saving 0,6mm (which also leads to ugly expensive Spacers) just to have the beauty of having exactly 50mm.