After 3 years in the same pot the medium had compacted and your roots aren't getting enough oxygen. Citrus needs to be repotted every 2 years or so. Are you sure it hasn't been longer than 3 years? It's quite unusual for a seed grown citrus to bear fruit in less than 5 years.