Thanks CMK! That is it :) It's only taken me 5 years to post the pic and the mystery was solved in minutes!

This question is driving me crazy. Since none of us can think of a correct spring blooming perennial, I'm going on the assumption that they were started from seed indoors or in a greenhouse and are ahead of their bloom time. What about lupine? I know the leaves aren't lance-shaped, but if it was a quick look, I suppose they could be mistaken for such. Another is campanula glomerata. What about Agapanthus (hey, it starts with 'A')? The other thing I came up with, in addition to many of those the rest of you suggested, is a primula. Are there any "blue enough to be called blue" that send up a spike?

Thanks to both of you. So good to know. I quite like them. I am not too troubled by them showing up everywhere in my garden but I would not want to think that they are invasive and it would be a bit challenging to thoroughly remove the spent flowers though I could try. Across the road is a forested area and a path with tall wild plants like goldenrod and aster and beyond that a marsh. I don't want to do anything irresponsible. I understand they are short lived. Should I be concerned about them spreading to this natural area across the road?

Yes. Routinely called Lithodora diffusa but actually Glandora prostrata. Plant of the Month: April 2013 http://arthurleej.com/p-o-m-April13.html