Sounds like it might be the Carpenterworm. UCD has what to do to control them on their pest page: Fig Carpenterworm Fig Pests Hope this helps. Sue

Wash it with clean water. Fig tree do not tolerate spraying with any chemicals.

I had a terrible problem with tiny brown spots on every fig leaf for the longest time. Some sort of fungus, I suppose. During the Big Freeze, all the leaves fell off my fig, then when it started greening up a few weeks later, I was ready with my spray bottle loaded with copper fungicide. I sprayed the new leaves very aggressively on several occasions, and everything is looking terrific now.

Ficus lyrata is not a slow growing tree if it's happy. I would buy the healthier looking smaller one instead, but suspect you've already decided on the larger one anyway.