Successful pavers rely on a solid, tightly packed granular base, side-to-side interlocking and secure edge retention. It doesn't look like you have a solid base and for sure you don't have side-to-side interlocking. Gator Dust (which is dry-applied mortar) will not work long term if your base is inadequate as the pavers will shift and the GDust will break up (the same as how the grout of a ceramic tile floor gives way over time on a wood floor if it is slightly bouncy.) If you want to grow plants between pavers, you still need a solid base and must use a pattern that still interlocks while leaving gaps between the stones. If not using special "grass block" type pavers, the most common pattern for this is as in the illustration below. (Don't allow a plant-growing-gap dimension greater than 1/3 of the paver length. Alternatively, you may decide not to grow plants between the pavers and have them be contiguous and solid. You need good edge retention which could be plastic paver edge, or the old standard, tooled concrete with its surface sunk 1" below the surface of the pavers. It will be easier to apply the edge restraint if you install a row border of pavers (with the field pavers being cut where they meet the border. Plan for and prepare all of the cutting so it can be done in one day or less in order to keep rental cost of the brick saw to a minimum.) I suggest an 8" brick shape paver for the edge row. The pavers should be at least 2" thick or they're not designed to support vehicles. Not just any stone works for the base. Something like pea gravel would be unusable. Instead, it could be a combination of layers beginning with crusher run (driveway gravel) on the bottom and sand-like granite or limestone screenings above. (Limestone screenings is what we use in Florida and it's commonly referred to as "paver base." (In warm climates, the crusher run may be eliminated and only a thicker base of the stone screenings used.) The sub-base of the excavated hole is compacted by using a vibratory plate compactor before placing any of the stone. (This machine is run over all the work, handled much as if one was using a lawn mower.) The various stone is compacted in two-inch layers as it is added. First dry, then wet. Above the stone is a 1" layer setting bed of UNcompacted screenings/paver base on which the pavers themselves are set. After the edge restraint is installed, the pavers are compacted into the setting bed with the vibratory plate compactor ... first dry, then wet. Installed like this, it will last. Get all the plants growing at once so as to get the jump on weed infiltration. My own thoughts about using this scheme (growing carpet-like plants in between pavers in a vehicular area) is that it never works out as well as it's envisioned. I'm not saying that it never could, just that usually, it doesn't. The plants need attention and watering and unless this is kept up with, they're going to have too hard of a time. Also, having plants mixed in with pavers increases maintenance. It's not as easy to clean paving where plants are rising above its surface. I think it's better to have paver areas and then separate areas with plantings. Then, the issues of combining these two elements don't exist. In a paver area where no vehicular traffic will occur, it will be somewhat easier to allow plants to grow between the pavers.

I use 6 to 8 inches of crushed concrete as my sub base under patio pavers or flagstone decks if my buyers don't want to pay for a concrete sub base. Then I put 1/2" to 1" of masons sand or torpedo sand, after placing my patio pavers I sprinkle masons sand in between the bricks, lightly water, and repeat the process. You can also use crushed granite or a 3 sack stabilized sand as your sub base. See ya, Kelly

" We would like to xeriscape the yard and have a cactus garden and some sporadic pavers. ... Can we just infill with gravel over the lower paver deck and plant the cactus directly into the gravel?" No .., the roots won't make it through the pavers, and cacti need more than just gravel to live in. However, pavers make an excellent weed control and mulch. It would be a pity to waste the previous owner's hard work setting those things. Re-define your idea to be "paver patio with inserted low water use plantings" and it's a piece of cake. Remove a few pavers where you want to plant something, making sure to leave enough room for stem expansion (2-3 times the width of the estimated mature stem size). Plant. Run drip lines if you are going to be using them, and use the gravel for mulch just in the spot where you removed the pavers to fill the hole level with the patio. You will have to monitor watering carefully, because the pavers act as a mulch and it will be easy to overwater. This picture is over the top, but you see how it's done. A plant or two, then an area with more plants than pavers, as if they are taking over the patio slowly. Mix fluffy low growers in with spiky tall ones. http://habhero.org/wp-content/uploads/2013/11/Cheryl-Mills-IMG_1717-4x6.jpg

The danger is not that the stone can't support the patio. It can. But over time, a sand setting bed could migrate down into the stones, causing uneven settling of the pavers. Is the first picture an area you will be paving? It doesn't look like you can build on top of this without getting to high on the foundation, or without raising the fence. I think if you added an inch and a half or so of paver base on top of the gravel, and mix it in a little before compacting, it would be sufficient to slow or stop the sand migration into the gravel. And then add the sand setting bed. But again ... grade elevation ... need to get that worked out. You don't want to create a nice patio that destroys house wood or invites termites.