I agree with cat_ky that the house appears to be stained. The color very well ties together your brick and your stone chimneys as well as your brown roof. Your landscaping is also nice. I would put more money into extending the planting beds even more. Get a professional designer to help you do that (and hide your HVAC compressors in back.) Here are multiple Houzz photos of home like yours. Step back from the front of the house quite a bit so we can see the whole setting. It's possible a beautiful stone walkway and a painted entrance doorway would enhance your already wonderful house. What's wrong with it?

I'd go a nice crisp white and redo the landscaping with lots of lush greenery. Black door. Remove the shutters. White railing in front and maybe add thicker, square columns on the porch.

Personally would NOT want the house to look larger and like the traditional vibe you have now just fine -- actually much prefer it to the "modern farmhouse" fad -- that, too will pass. You have a really nice house and yard and I like the siding color you have with the white windows and dark shutters. White shows dirt and wear and imperfections a lot faster than the really nice natural/neutral siding color you now have. Would go with your original instinct -- removing the shutters only to repaint the existing siding your choice of color. While the shutters are removed, take a long look at the house and take some pictures of it. Did you prefer it with shutters; and if so, ( I likely would) then put them back on.

Do you want to go dark or stay light? The siding looks like a creamy pale yellow, so lots of options for that.