This is not where it sits all day- it is normally on the window next to that one (the one you can sort of see in the background of the picture of the whole plant) along with a bunch of other plants. Because it is with so many others and i only have one fan running in the room, that it is probably getting inadequate airflow. (Also my basil keeps growing a white mouldy film on the side of the pots, once again i think it is from lack of airflow). Trying to find more places to put my plants, but living in an apartment with only so many southern windows and no outdoor space is really challenging! I think there is more new growth on it than i realized, but the rate of leaves dropping off to new growth happening was troubling to me :P

I uploaded some pix of my olive tree. As I suspected, they're not terribly helpful: :) Olive tree And I got a helpful response back from the nursery where I purchased the tree. This is what the guy had to say: Olives, like all fruit trees, make exponentially more flowers than fruit. Only about three percent of the flowers can be expected to develop. A small tree will usually make only about half a dozen fruit. Very dry or windy conditions can exacerbate the situation. Keep your tree well watered during the first few months. Many people incorrectly assume that olives should seldom be watered. While mature, established olive trees are famous for drought resistance, young or newly planted trees should be treated like any other newly planted tree. I can't do anything about the wind, but I think I'm going to water the tree more at least, even though the leaves look fine. Anyway, it doesn't sound like the lost blossoms are anything to be terribly worried about.

Hard to tell from the picture, but could also be mealy bugs. Either way you could spray the trees with a mild solution of dish washing detergent or coffee to send them on their way. Meally bugs have a waxy coating which the detergent helps to break down, as it also tends to prevent them from being able to breath. If you are scientific, you can spray one branch with coffee, which tend to be toxic to insect, and one branch with mild dish detergent and see which works best. they are tough little guys, so you may have to spray your tree a few times to get them under control. Or, as Tim suggested, Spinosad should definitely work, as that stuff is relatively potent!

You don't want a landscape contractor, you want an arborist. The first place to start is with your city, many so cal cities have a city arborist that will visit and consult.