Funny, this is a common and very expensive/ luxury detail in some areas. Savannah and Charleston this would be considered a major upgrade from any typical driveway material, lumps and all. You could pull it out and reset it our just have a good landscaper reset sunken areas. Removing it seems crazy to me. Get someone to price a full like kind replacement...
Q