Looks to be in SERIOUS need of water and fertilizer; and cut out the dead wood. Citrus are self thinning, i.e., the tree will drop any fruit it can't support... and from the look of your tree, it can barely support itself, much less fruit.

Get some good citrus fertilizer and apply it according to label directions at the dripline ... at least 3 times per year; follow that with deep watering on a regular schedule; and you should see some major improvement.

Ditto what John said, definitely way too dry. Create a nice well around this little tree beyond the drip line (edge of canopy). Water deeply twice a week, three times if you're in a really hot area (you don't mention where you live). Looks like it's really getting a baking in this corner, so it's going to need some water. Once you've watered really well for about a week, apply a good citrus fertilizer that contains micronutrients as well as the macronutrients. As John said, cut out the dead wood and stems. Your little tree is starving to death.

Patty S.