Winterberries have red berries on bare branches in the winter.

The birds just ate the last of the fruit on my Winterberries today.

If you decide to try for Winterberries, ... be sure to get a Male/Female pair ... as both are needed for berries.

Red-twigged Dogwood shrubs add red color in the Winter, as well ... and the variegated varieties (Elegantisimma, Ivory Halo, etc.) make for striking silvery toned shrubs in the growing season, as well.

Finally, evergreen shrubs (Yew, Junipers, Boxwood, Dwarf Spruces, etc.) ... will add to your color in the winter months.

The following photo shows my Winterberry and Red-Twigged Dogwood (in the background), with some accompanying evergreens.

{{gwi:240564}}

Perennials that look good in winter include Hellebores, cyclamen, Arum italicum pictum, These plants actually show fresh green leaves all winter. Other plants I love in winter are the grasses, bleached to a linen color, but swaying and rustling in the breeze. My favorite is the huge ravenna grass (Erianthus ravennae) which is almost 14 foot tall and doesn't break down in wind or snow.

Cotoneaster looks good at this time of year, with long-lasting red berries and deep green glossy leaves. It's very variable, so you'd want to select a variety that fits the space you have.

My cotoneaster are sited next to a deep purple heath that's blooming now; most heaths bloom in mid-winter, in all shades of pink, red, and purple. The witch hazels ( Hamamelis x intermedia) are also blooming full force now - Arnold Promise, Jelena and Diane are all stunning.

Since you're in zone 5, some of these may not be hardy for you.

And, I agree with both earlier posters; deciduous holly is great and so are some herbaceous plants as well as shrubs that lose their leaves. Evergreens are great, but too much of a good thing can be very boring.

I am not a yew lover, but there are a host of evergreen shrubs, e.g. boxwood, Japanese holly, heaths and heather, Pieris japonica and P. floribunda, Leucothea, Mountain laurel, Rhododendron. Then there are trees with interesting bark - paper bark maple, kousa dogwood, moosewood and the asian striped bark maples, the birches. There are evergreen ferns notably christmas fern and marginal shield fern. AS I recall there is book "The Garden in Winter"

I like the dried flower heads of Annabelle hydrangea in the winter as well as others mentioned. Vernal witchhazel and one of the less common red varieties of H x intermedia, 'Ruby Glow', are hardier than the other spring blooming witch hazels. It's mentioned on the Fairweather Gardens website.
I also love the branching structure of my pagoda dogwood (Cornus alternifolia) in the winter.

Here is a link that might be useful: Fairweather Gardens witchhazels

Spirea becase of their dense growing habit also provide winter structure. A perennial that acts like a small shrub and has a very nice purply brown winter color is Origanum Hopley's Purple.

Wow, nhbabs, Fairweather Gardens is expensive! Guess I'll have to look around a bit if I want to expand my hamamelis collection.

diggingthedirt, yes Fairweather Gardens is expensive, but the plants are huge and are packed better than any other nursery I've ever received plants from. Remarkable shipping technique--and really huge shrubs--often three feet tall and bushy.