Sounds like rootstock fruit to me. Nectarines are not as vigorous as their peach sisters so I could easily see this happening. Fruit trees are grafted to a different tree which forms the root system. Are your fruit like hard little peaches?

I've had this tree for 2 years now. It is located outdoors in a full sun spot with good soil that I amended with compost when I was planting it. Since then, other than watering, I haven't really fertilized it. It has a good amount of green growth on it since planted it, but since the one time that it tried to set fruit, it hasn't really blossomed. I guess lack of fertilizing and the fact that its still relatively young might be the reason for no fruiting. I will try fertilizing it, putting it on a regular watering schedule, I hope to see some fruits of my labor!! :) Thanks for all the responses!

this my plant. Sorry for the bad pic. Lol thanks John. It's in a South facing window which gets at least 6 hrs direct light. The fertilizer is one my we got that was made for citrus trees. It gets fed monthly. Thanks for the help and let me know if I need to do anything different.

U have 2 remember that when u replant something outside (or inside) it takes a while 4 it 2 acclimate 2 where u've planted it. The roots 1st have 2 establish themselves, using the tree's energy there. Once the roots r happy with where they r, u'll c new growth in branches n leaves. Citrus likes acidity. U can mulch it with acidic bark. They like 2 stay moist. (Ground n leaves). N they have 2 have some kind of humidity in drier climates. I've seen many comments saying 2 use 5-1-1 fertilizer, a ratio of 3-1-1 is better 4 ur tree until its established, then 5-1-1. I am in Idaho n have a 10 year old kumquat tree n a 2 year old lemon tree. 4 a taller tree prune the bottom branches off n 4 a fuller tree prune the upper branches. Good luck.