Critter in the soil -- very fuzzy image, but by the shape I'd say pupal cases of moths. Non-feeding stage. To find out who they are, place in a covered clear container. Place where you'll see it daily but NOT in the sun. Eventually, moth will emerge.

I'm only watering if the ground is completely dry. But for the past week or so it's required watering every morning. The fertilizer is supposed to be applied about every 10 days which I'm currently doing so I don't think there's too much fertilizer on the plants.

Too big a plant in too small a container. So what you are seeing is the effects of drought damage to the leaves due to small root death. Remove all the affected leaves from the plant and thin out some of the excess foliage on it. If possible also remove some of the lateral branches (aka suckers) that are not bearing fruit yet to bring the overall size of the plant more into line with the limited size of the container and so reduce the water demands. Then resolve the "and there are times it remains empty for a day or so" problem. It is not unusual for the reservoir of SWC to need filling several times a day even with containers much larger than this one. Dave

The containers will probably improve with more steady feeding. Rain and watering leach fertilizer faster from a container than the soil. The one in ground will eventually get nutrients from the mulch as it decomposes but depending on conditions could probably use some extra fertilizer at times to grow well. If the leaves start having problems again take some more pictures. You should also learn about pruning, trees that are weedy and dense are more vulnerable to fungi that attack leaves.