If I were to do it , I would take all the top good portion out. (pile it on the side)Then dig inside the box , a good shovel depth. Then add compost, About 6 to 8 bags. and mix everything real good. Then return whatever you took out. Add about another 4 to 6 bags of compost/manur and mix it all together. Clay when amended with organic matter is good. If you do this. you will have more than enough depth for just about any vegetables. In general, you need a minimum of 10 to 12 inches of good mixture. With a 4'x4' it should not be very hard to do.

Winter is often the busiest gardening time for me, especially if there's heavy labor involved. Also, I always avail myself of late fall markdowns that get planted over the winter. Just don't have the ooomph to battle the bugs & heat of summer like I used to. It helps that I don't plant annuals except for a few window/porch pots, and that 99% of my plantings are flowering shrubs and evergreens. GreenthumbGene - re "not fully understanding your NC garden ... I think you'll find a whole lot of gardeners here, myself included, who had a lot to learn upon landing here. There's some good NC gardening books out there that'll set things straight, along with NC State University's online content. But, I've learned the most right here at Gardening in the Carolinas, so you're in the right place :) Here is a link that might be useful: NC State Urban Horticulture

I have some raised beds in the backyard where I grow tomatoes and herbs. In the fall I pull out the annual plants, weed, water, and spread a couple inches of shredded leaf mulch, The mulch breaks down over winter resulting in a nice loam, easy for turning over and planting in spring. In spring I also dig in compost or rotted manure.

Sounds like you will do well with this...thinking ahead and coming in with prior growing experience is huge. I've grown in my back yard and also at a plot that I visited once a week or so. The management is different but very doable if you plan differently for each location. You will need to arrive each weekend mentally and physically prepared to deal with any problems that you discover came up in the previous week. Don't waste time being overly upset by "bad" things that may have happened while you were away. Maybe you will need to spend a lot of the time dealing with pests, maybe watering, maybe fence failure, maybe storm/wind damage when you first arrive each week. You just never know what will greet you, but don't let it get you down, most is fixable. And often most everything will be fine and you'll be amazed at the changes a week brings. And many problems can often be avoided by upfront planning and a little work...ie. using row covers, mulch, well planned and well constructed fencing, and possibly automated irrigation depending on your climate/weather. Crop selection is also different. Veggies needing timely harvest may be out, but that still leaves lots of things to grow. I've found I can adjust my harvesting ideas and still grow many crops others rule out because they think they must have frequent harvesting. You can pick tomatoes at breaker stage along with the vine ripened ones and a week later the plants and fruit should be fine for a repeat picking. Pick any beans over an inch long and a week later you will return to many perfectly sized ones, some overgrown ones okay for soup, and some tiny ones. Pick any cucumbers you see, again even 1 inch ones, and a weekend later you should have a nice harvest again. Following the same method, zucchini, summer squash, etc. can all work out....some of the baby veggies will be fussy to cook with and some of the larger ones may sometimes be best in the compost, but it can work if you adjust your thinking and your eating and cooking methods. It took me awhile to get over having to throw out a few overripe items and worrying about which tiny veggies to pick and which to leave. But now I love knowing I can go away if I want and still have some beans and cucumbers. Potatoes, beets, carrots, dry beans,onions, winter squash, etc., etc., etc., all do fine with weekly check ups...even less often if mulch is keeping weeds out and water in the plot. Enjoy the land, expect challenges and enjoy learning from them.