what is the depth of soil against foundation that you would be able to put on top gravel? Do you have gutters? 3' isn't much distance for slope and thus doesn't sound like you'll be able to build up very high against the foundation and create the necessary depth for planting. Also, gravel does have a way of migrating upward as your soil is slowly leached through the old driveway so in the long run don't expect the tidiest garden. No reason why you cant build that wall on the existing driveway, assuming your levels work out.

I didn't remove the gravel when I did this, and it turned out great. Later I wished I had a couple buckets of gravel for another project, but otherwise I ran into only one problem: I planted three trees in the bed along with a bunch of perennials. The two young (2 gallon pots)trees did very well. The larger (burlap ball) tree struggled for three years until I finally moved it. It's roots had spread out along the top of the gravel layer and were weirdly fragile. It's doing well in it's "new" location but getting a lot larger a lot faster than I expected. Good thing I moved :)

The ideal thing would be to make lemonade from lemons and use these concrete chunks in some creative garden feature. If that's not possible there are probably places in your area that accept so-called clean fill - the problem is finding them. Where I live, the gravel yard that I purchase gravel and stones from for my garden has a section where you can drop off clean fill. I don't know if every or even many gravel yards have similar policies, but since these are fairly common establishments (there's always a need for gravel in construction and the like pretty much everywhere in the USA) you might want to inquire of some in your locality.

I don't quite understand why a trip to the store has to be a family outing. Father AND mother have to go and bring the kids, so they can put the kids in separate carts, leave the carts in the center of the aisles and then stand one on each side to block it off for everyone else? To say nothing of the screaming, the arguing (no, we have candy at home... you don't want that do you? Do you need a time out? bla, bla, bla) Why not one of them stay home, or in the car with the kids while the other one goes in, gets what they need more efficiently and get out? We've had those signs around here and I just shake my head. More people looking for special treatment. If they don't get a special discount, they want special parking, or a special loan or whatever. Gimme, gimme, gimme. The only thing more laughable is handicapped parking at a home improvement store. JUST ONCE, let me see someone in a wheelchair come out of there toting a 12' sheet of drywall! And I have far more physical limitations than most of the people that I see parked in these "handicapped" spots and no, I don't have a cripple card or plate. (I imagine someone would be totally offended if I called it what it's known as around here.) And around here you can call the police and they'll issue a $200 ticket to someone parked without the handicapped plate or card. The plates aren't given out as freely anymore, which is good. They were so misused. At least the cards technically expire, though seldom is it enforced. Also, technically, around here someone parked in a courtesy spot could be towed. Since it's private property, the property owner can have "unauthorized" vehicles towed off their property. Doubtful that they'd ever do that, but if someone got ornery enough, they could. I give credit to some good friends of mine. He's in his 80s and is in a wheelchair and she's in her 50s. They don't often use handicapped spots. Like they said, he's riding anyway and enjoys the fresh air. More people could learn a lesson there. Naturally in inclement weather they will use the handicapped spots, but also very often he will stay in the vehicle. He doesn't have to go in.