How big is the island? How big is the kitchen? Are all the seats in a row? Or are they in an L shape? Facing each other? How big are your stools going to be? How comfy? Is there a lot of space behind them? Or are they in the walking space?

If it's one of those big islands with the built in round kitchen table? Sure! If it's a big workspace with everyone in a line, two is enough.

If your dining area is casual, I think seating for 3 should be enough at the island. I am providing for 4 at my island in my remodel because I only have a formal dining room and don't want to have to set it up all the time.

We've just decided on four seats after ditching the kitchen table. Our island will have a round eating area at the end, better for conversation. Usually breakfast and weekend lunches it's myself and our two boys eating together. Dinners we all eat together in the dining room because we find it more comfortable. Anyways, we figured four would be best for larger family gatherings because the kitchen is where everyone always gathers.

We are also a family of four and have four seats at our large (8x4) island. When we purchased the house there was only room for three, which was OK, since my son was an infant and sitting in a seat that clipped onto the island. But soon it became a problem that there were only three seats. I removed a base cabinet to provide the extra space for a stool. I did lose that storage space, but I really wanted us all to be able to eat together. Like you, we don't always eat every meal together, but it is really nice to be able to all sit together. My dining room is also directly off the kitchen, but we still don't use it as a quasi-kitchen table.
Having said that, I think the arrangement is important. All in a row is less conducive to conversation. We have two on the end and then two on the side, in an "L" formation. Having more seating is also really nice when you are entertaining; there are more people to hang out with you while you cook and prepare!
Good luck!

I think it would be fine to have 3, as long as eating in the DR is as easy as eating in the kitchen, and as long as you can do everything else in the DR that you used to do at the kitchen table. E.g. -- if you used your old kitchen table to play cards or board games, or read the paper, or have family meetings, is it okay to do all that in the DR? Even though you will rarely all be eating at the same time in the kitchen, maybe your DH is worried about other things that used to take place there...

We are a family of four as well.

We eliminated the kitchen table with our remodel and added a peninsula with 2 seats. Those two seats have been fine most of the time. Occasionally, we've wanted one more (3 total), but we've never wanted 4, so I think 3 is fine.

However, whenever there are all 4 of us at a meal (average 6 nights out of 7), we eat at the DR table. We opened up the wall b/w the DR & Kitchen w/the remodel but even if we had not, we would have still used the DR for family meals. We have found that eating in the DR is much nicer than when we ate in the Kitchen w/a kitchen table and have commented that we wish we had started using the DR full-time even before the remodel!

We have three at the island. Family of four, but kitchen table is right there. So if we're all there, we eat at the table. We could fit four at the island, but it seemed unnecessary.