This came out of a terrarium, the leaves are just a bit fuzzy, with a very faint tinge of purple, and the purple on the underside of the leaves. If you can help me id this I'd really appreciate it, thanks ! Image link:

Ok so I dug up all the roots. And I saw plenty of smaller ones and I pulled those out as well. I hope I got everything, cuz I am planting some more hedges.

I am also pretty certain it is shasta daisy, Leucanthemum x superbum, one of the toughest and commonest perennials. If you looked up pictures they are probably of shastas in flower. As hortster said, this is not a 'bush' but a herbaceous perennial so in the winter only the basal foliage would be there. In the summer it will put up flowering stalks which will the die back down next winter. That's what makes it a perennial and not a shrub - the stem framework is not permanent. Look at the link to see the plant with the flowering stems rising from it. Here is a link that might be useful: Shasta daisy

I don't know of any foolproof method to get rid of this in one go... It is rather resistant to Roundup, and the running roots tend to keep sprouting up even after the majority of the plant is removed. Probably the only sure method is to keep removing it as it comes up, which will undoubtedly take more than a year or two. I think you will have to learn to live with it if it is a permanent, desired plant in your neighbor's yard.