Scarlet Oaks do nicely in slightly acidic clay soil. You found out when digging the hole, the fact that clay not only holds on to water very well, it also quickly releases the moisture it contains into voids in the soil. You dug a hole and not only the rain filled it with water, but all the water in the clay soil around the hole kept filling the hole until the water pressure in the hole became close to the same as the water pressure in the soil. Then as evaporation occurs the water which turns into vapor and rises away is replaced with more moisture from the soil, until there is not enough water in the clay soil to replace that which has evaporated. At that point the clay will begin to absorb enough of the water in the hole to once again bring the pressure of the water in the soil to be equal with the water pressure in the hole. Only once that process begins to happen will you be able to notice the water level in the hole beginning to lower. This is the very reason why it is a bad idea to dig a hole in clay, plant the tree, and backfill with a different kind of soil. In that case the backfill soil will remain full of water, not only from rain and your watering but also from being kept filled with water that the virgin clay soil is holding. The backfill soil needs to be the same as the virgin soil which came out of the hole. By doing so the pressures remain more even and the there is much less of a void in to soil situation presented. This is also why it is bad to plant a root ball that is full of potting soil directly into clay. Instead as much as possible of the potting soil need to be washed away from the rootball mass and refilled with the virgin clay soil as what is found around the planting hole. That is why it is best to plant trees when they are in their dormant stage. If for any reason you want to amend the clay soil where your tree is to be planted, there must be amendment fairly evenly mixed into the virgin soil for a very wide spreading area. The more clay ratio in the virgin soil the farther spreading you planting bed needs to be that you amend. The amended area needs to be at least 6 times, if not more the width of the tree's rootball. Also, plant the tree with the rootflare about 3 to 4 inches above the soil level. After the tree is securely planted, and staked if needed, only then back fill the exposed 3-4 inches of raised rootball with a good quality and fairly finely shredded cedar mulch. Just make certain you do not pile any mulch above the topmost part of the rootflare. The trunk should not appear to look more like a post sticking out of the ground. Instead you should be able to just see where the topmost part of the root flare is growing out of the trunk. Finally never pile any mulch all the way against the trunk, make certain to leave all mulch at least 4 or more inches away from the base of the tree's trunk.

Congratulations for having interest in growing many fruit trees. That is good. Trying to grow them in a way that keeps all the trees happy is the next thing. If you dug a hole in the sandy beach shore to make a swimming spot, the water that you put in it would just drain away through the sand. But, if you placed a one foot thick layer of your clay soil on top of the sandy hole, the water would be unable to drain away and the swimming spot would work. Although the water would not be clear.When you water too often, or when heavy rains won't go away, the fruit tree roots in your dug-up clay hold excess water for a l o n g time, which is very unhealthy for many kinds of fruit trees. As an immediate action, I would not put any water on the plants without checking the moisture in the soil a few inches deep. If it is moist below, leave it alone until it is not wet.. For a "fixing" of the problem, when your next Winter is beginning, you could carefully dig up each tree's root ball and replant it ON TOP of the clay and surround it with a mound of good/organic/sandy/rich soil mix. Like a doughnut with a doughnut hole. Make the mound a little higher than the root ball, since it will sink downward as it settles. These mounds have the exact opposite challenge in that the water easily descends through them to reach the flat clay surface below. Keeping this mound moist in hot, dry Summer will take frequent, slow soaking to get them thoroughly watered and thus welcoming to the tree roots' expansion in search of moisture and food while enjoying what they did not have enough of before: air in the soil mix. As the trees grow in size, you could add more soil mix over time to enlarge the mounds. When the trees are ready, they can choose to send their roots downward into the clay below. Building boxed sides or some other kind of material for sides is not needed, though it may be wanted.

I have very heavy soil too, have found that the first year will be different as you adjust the soil. I would add sand along with the compost to help drainage (especially if you get some shade), and start off with unfussy crops. Avoid carrots this year, unless you want to double-dig. How about...things with roots "partly ready-made," like potatoes, garlic, onion sets (not onion seeds), or small-rooted things like beans and peas (their vines help do some of the stabilizing work of a root, plus their root nodules fix soil nitrogen). Be sure to remove all of the potatoes at the end of the season so that they don't pop up all the next year. You could even try a few strawberries if you feel ready for a perennial, they don't need deep roots either and have pretty foliage. If that is a sunny wall, you could also try a cherry tomato vine against a trellis on the wall, if you provide compost and keep it away from the peas and beans, as the extra nitrogen could make it too leafy. This is a more fussy crop but the hot stones would keep it warm at night, and it's very tasty so it might be worth it. Put in a fast cover crop like buckwheat when not actively gardening, and till it under prior to its forming seeds, to boost soil aeriation and nutrients. Have fun! :)

I'm not sure you are all that limited by plant choices. Clay is really not as much of a hindrance to plant establishment as many folks think. The only real caveat is to dig a wide, shallow planing hole and plant high, bringing the soil upto the crown of the plant in a small mound. I would encourage you to do as much amending/improving the soil as possible while it is cleared before planting. So much easier to do what you can to change soil conditions while the area is empty than after the fact. I'd bring as much compost or other organic matter as I could and work it in as best you can. You can also bring in just decent garden soil (check out bulk soil suppliers for this and the compost) and raise up the planting area. FWIW, plants with purplish foliage tend not to hold that color well in much shade. And plants with that coloring also tend to disappear in shade. Try things with yellow or gold foliage or variegated leaves. They really pop in shade and seem to glow. I am not all that familiar with works in z5 so check to see if these are hardy for you: hostas - a given!! epimediums heucheras, heucherellas and tiarellas Aralia 'Sun King' Pieris 'Little Heath' Tradescantia 'Sweet Kate' or 'Blue and Gold' Kerria japonica 'Pleniflora' Aucuba japonica 'Variegata' or 'Mr. Gold Strike' Hydrangea serrata 'Golden Sunlight' Hydrangea quercifolia 'Little Honey' Saxifraga x urbium 'Variegata' (nice groundcover) Lamuim maculatum 'Anne Greenaway' (another excellent groundcover) Carex morrowii 'Aureovariegata' Hakonechloa macra 'Aureola' or 'All Gold' Dicentra spectabilis 'Goldheart' and of course ferns - Japanese painted fern or 'Ghost" fern will add color as well.