I've done exactly that twice. In both cases the trunk of the discarded tree was dug out, and the new tree was planted in the same location. Both new trees grew on healthy and successfully. If you cut the unwanted stump close to the soil line,, you can rent a stump grinder and remove it quite quickly.

You may also consider drastically trimming the orange back and grafting a variety or more of lemons onto it, removing the orange producing wood entirely once the lemons grafts have successfully taken over the system.

Thanks, Imelda and Fireball. Grafting is an interesting thought. I've never done it, and I've heard it's pretty tricky--do you think I've got a decent chance of making it work as a rookie following directions out of a book, or is it a high-level skill?

Having never tried it, I can only refer to what I've read here and in other publications. Look on youtube at the grafting videos for a visual aid. People here report anywhere from complete failure to all grafts surviving so I think that if you tried multiple grafts, you have an excellent chance of some taking. Remember to keep some orange branches intact while waiting on the grafts to take, in case you need to try again.

Is the orange tree worth saving , even for the root stock, or are you replacing because of sickness and deterioation of orang tree.

The orange tree seems to be healthy though the fruit is often small and not very sweet. I mostly want to do this because I could eat a lot more lemons than oranges.