Sorry I have been absent from this forum for awhile. I apologize for not reading the original post more carefully as to the size requirements. Just as a note, Loropetalum chinense is available in two sizes other than jumbo. Purple Diamond tops out at four feet. Purple Pixie at about two feet by two feet. Again, I have no idea if they are winter hardy for you, but if so, they are strikingly beautiful shrubs with four season interest: very showy spring flowers with purple foliage, good blue-green summer foliage, brilliant red coloring mixed into the green in fall, and evergreen in winter. IF they are hardy, Camellia sasanqua Hot Flash and Shishigashira will mature at around 3' by 4' and are shrubs of the highest caliber in my opinion. Good, clean, evergreen foliage and lovely fall/winter flowers. Osmanthus Goshiki with evergreen green/cream variegated foliage is a fairly dwarf shrub, topping out at 3 to 4 feet. It grows so slowly, it could easily be kept pruned to any size you like. Osmanthus heterophyllus Variegata gets larger, but again is a very slow grower, so can be kept where you want it sizewize. It just might be my very favorite shade plant with clean, sprakling white variegation on dark green leaves. It just lights up its space planted among white hydrangeas. Again, I have no idea about hardiness for you with any of these shrubs. One holly I would also recommend is Ilex cornuta O'Spring. It has yellow and green variegated shiny leaves and is gorgeous.

I'd do a mix of perennials, you can do Gaultheria procumbens (wintergreen! chew on the berries/leaves, evergreen to semi-evergreen) and the sprawling Phlox divaricata which will flop over if planted on the edge. Phlox divaricata 'May Breeze' is a beautiful off-white/dove grey/baby blue (really strange color) and there are others that come in a bevy of pretty purples too. Microbiota decussata or Russian cyprus seems to take shade. I'll think of others.

Let's just say that evergreen shrubs that are hardy to zone 5, and can handle shade are not a dime a dozen. Adding in other complicating factors, just makes that list shorter. At least people are being honest, instead of making up answers just for the sake of an answer.

Hollys are, I think for the most part, OK in shade. There are such a range of heights and widths. Most of them might get taller than 7 ft. There is one I'd like to have called "Castle Wall" that only gets about 5-7 ft and 3 or so wide. Yews are perhaps my favorite evergreen shrub and can take shade to full sun. They are so beautiful, and so versatile. A word of caution about them, though: They are extremely poisonous. Something like .2% body weight can be deadly (that's pretty toxic!) I have dogs who tend to explore with their mouths, so I have eliminated yews from the roster. I just mention this because you said you had a toddler :) Hey, what about Skip or Cherry Laurel? I think that would be about right in terms of height, and they'd like the filtered light.