Looks like you got some leaf spot going on (Septoria leaf spot). Most people rip out plants, but since it doesn't effect the fruit/stems you can prune off infected leaves until the plant has too few leaves to set fruit efficiently...if you really really really want to keep the plant, that is. Try to get all the infected plant matter out of your garden as you prune (and end of season). I personally wouldn't compost it, though some do.

Your first photo shows azalea leaf gall... common fungal disease in the early spring on azaleas. Leaves of infected plants develop distorted growth. If only a few plants are affected and on each plant just a few stems/leaves, pick and destroy (meaning garbage not compost) galls, it rarely does enough damage to require chemical control. If the infection was extensive, a fungicide might be helpful but must be sprayed before the new foliage buds begin to open during or immediately after bloom....in my zone my own are past that now. Your second photo is showing damage from azalea lace bug. Here is a link that might be useful: Azalea lace bug

possibly low nitrogen? what fertilizers have you used?

Thank you all for the responses. I went out there this morning to get a closer look. The caterpillars are Bordered Patch (so I will tolerate them). However, I looked at the bottom of the leaves and found some white residue (kinda fuzzy). The link below will have more pictures of it. I haven't found any insects on it (though that doesn't mean they aren't there). But I'm now leaning towards this being a fungal disease. This all started after we got a lot of rain at the end of June and beginning of July. Plus, this is in my backyard, where there is less wind due to the fence. Plus, my zexmenia are planted in a raised bed (excellent drainage), so the border acts as an additional barrier for wind. And, the underside of leaves are affected on the mature leaves while the top side of the leaves appear normal. But then again, I could be wrong. Do these new pictures help anyone identify this problem? Here is a link that might be useful: More pics of my Zexmenia on Flickr