Example of the leaves curling

Hard to tell from the picture, but could also be mealy bugs. Either way you could spray the trees with a mild solution of dish washing detergent or coffee to send them on their way. Meally bugs have a waxy coating which the detergent helps to break down, as it also tends to prevent them from being able to breath. If you are scientific, you can spray one branch with coffee, which tend to be toxic to insect, and one branch with mild dish detergent and see which works best. they are tough little guys, so you may have to spray your tree a few times to get them under control. Or, as Tim suggested, Spinosad should definitely work, as that stuff is relatively potent!

A picture would be useful. Olives aren't great houseplants. Could be too hot, too wet or too dark. Needs to be right by a window and relatively cool. Cooler than your heating probably is. Depending on where you live it might be able to live outdoors most, if not all of the year.

Flex, you are not getting many answers because you tagged 'ask the yard doctor'. One of the quirks of this site is that people using the gardenweb site can't comment if you do this. It's only accessible via Houzz and from the US. If you repost and leave off atyd you should get some help. Bottom line is that olives make difficult houseplants. They're seldom happy indoors.