It sounds like your landscape architect did a great job of choosing low-maintenance perennials! But I agree, if you don't love the looks of something, definitely time to give it the boot. :)

I think with the Karl Foerster and coneflower, maybe something low and yellow would look nice. What about some more coreopsis? Always great to repeat plants to give unity to the garden, and I think the yellow would be a nice compliment to the blooms of KF as well as the pink/purple of the coneflower.

I will admit that I have at least seven varieties of coreopsis right now, so I'm obviously biased. :p

Or maybe another smaller grass with a slightly different bloom than KF? Pennisetum Hameln might be nice if you have the room - they are a more rounded grass form, which would be a nice compliment to the spiky KF.

Sporobolus heterolepis 'Tara' is a smaller version of prairie dropseed and might be just the ticket to fill that space and soften your pathway.

I am partial to Purple Love Grass - Eragrostis spectabilis:
{{gwi:232635}}

I'm thinking what you need is a change in texture and shape. How about some Sedum 'autumn Joy' or 'Autumn Fire'?

I like that you're practiciing Xeriscaping, which is great.... but why don't you stop worrying about complimenting your flowers and first eliminate all of the INVASIVE species that you have in your yard already, such as burning bush, and butterfly bush.... You may not notice these bushes sprouting in your yard which is already crowded by grass, but sure enough, in the woods down the street, these bushes will spread, killing native species and animals.
andy

Agree with the sedum rec - "Matron" would be nice with its reddish stems. These sedums are going to be taller than the blue fescue, though, but there are very short sedums, as well - there is a blue-green one and a reddish one, but unfortunately the names escape me.

For shorter plants, how about lavender or ajania (aka gold-and-silver mum)? Or Bergenia - bergnia would look fantastic with the grass, great foliage compliment.

How about a mix of Daylilies - maybe Happy Returns and Rosy Returns or some other pink Daylily. They would do well in a dry spot.

Maybe a Heliopsis like 'Summer sun' which has pretty double yellow flowers.

Prairie Drop seed (Sporobolis heterolepis) is a very drought tolerant and pretty grass.

Bizarro, the OP says "butterfly weed" which I assume is Asclepias tuberosa, the orange milkweed; not "butterfly bush" (Buddleia). Buddleia probably wouldn't survive in a zone 4 winter and is not invasive here in zone 5. Burning bush is definitely invasive here though.