Losing all leaves means the change in light and heat was too big too fast. You probably need to move them outside earlier, in May I'd think. If necessary protect from those late freezes by moving back inside overnight or covering.

You need to provide as much heat as possible as soon as possible but make the changes gradual, like over a week or two. You know what was too big a change this yr so try to improve on that.

They can be outside anytime in spring so long as they don't freeze. A good location would be protected from cold winds, as warm as possible, and as much light as possible.

Easier said that done isn't it? Every Spring and Fall, I move my potted trees back and forth, sometimes a couple of times per week. After it looks like it's becoming a futile waste of time, I finally stop until next season. But what makes this do-able is these two hand trucks from Nursery Enterprises for moving large potted plants. The bigger one has allowed me alone to do the work of a team of men--all without damaging my expensive pots. I come home from work, quickly move the potted trees before dark, and I'm done.

http://www.nurseryenterprises.com/15-gallon-tree-dolly.html

http://www.nurseryenterprises.com/large-pot-dolly.html

It is possible to take a cutting from a fig tree that will never produce fruit. Just a thought.