That's not a bad soil, 75% rock. I am currently messing around with potting soil, pumice, wood chips, which is similar to yours. The important thing is that they should not hold water.

Skip the vermiculite. I have never grown cacti or succulents from seed but would assume that they need somewhat drier conditions than most other seed types. Vermiculite holds too much water. And when fully saturated, collapses, further impeding good drainage. So you are correct about its instability :-)

I restrict vermiculite to only a cutting starting medium.....and even then use it rarely.

Vermiculite is not much recommended for growing succulent because it absorbs water like a sponge. It has also lower (rated medium) air porosity (compare to perlite rated as high air porosity). I believe it is OK for seed starting - they need more moisture. I think that quartz sand is quite coarse - if so, it is also good for seed staring. Only if coarse enough it is suitable for growing. It is likely much better to use grit, in 3-6mm size.

I'm thinking it would probably work pretty well for seeds, but yeah, take out the vermiculite and I think you're good.

If I take-out the Vermiculite, which is best for them? Is Ground Tuffa OK? I have this material locally available. Quartz sand is coarse sand which is commonly used in sand-filters for swimming pools, the grains often larger than most cacti seeds, so it has good drainage. Ground Tuffa, however, is a mix of particles ranging from very small pebbles to fine dust.

Any small particles (especially ground) are to be avoided, they impede drainage. Think large, coarse particles for fast drainage.

Sift fufa to get rid of dust and smallest particles - just as perlite or even pumice if they contain dust.