I would post this in design dilemmas as well. When I have a hard time making a decision and there are many choices, I only choose between two at a time. Start with your siding first, A against B, A against C, C against D, etc. After I made a final siding choice, if I couldn't find a trim color that works with it, I'd eliminate that siding color and I'd start the siding process again. It's hard for us to suggest colors without seeing your choices.

The window trim change would be to Craftsman style. Unless you are doing the work yourself it seems a lot of expense for little difference on upstairs windows. It would, however, be extremely easy to do on your front door as you would only need to change the top board. And you could replace the side lights with solid wood inserts, or you could use textured glass that would obscure the view while still letting light in. While the limestone exterior is very light it has a yellow hue. Simply speaking the easiest way to find colors that work with the warm limestone is to choose a nearby color. That was probably why the trim is green. Since you prefer a modern look, I suggest you keep your siding light and use color only in the trim. Below are two combinations that I like. I darkened the roof color in these pictures.

I think I would like to split up the wall creating a top border at least 18" left as is. Then I'd like a reddish warmer color to the lower portion of the wall of bricks. (usual standard red brick color). I think it needs to be broken up with contrast colors. It will ground the home and bring some life to the neutrals. This will break up the height covering so much of the lower half of the home. Or..maybe BETTER yet.... .. remove 12" H in rows of brick, off the top and replace with a 'deeper depth' wood facing (compared to brick depth) .... to create a band of color above the lower brick. Maybe match the wood facing with the vinyl beige or a contrast color for the wood trim added ..... to create a larger space above the brick. Lower brick in a natural red brick color. The windows will not feel so cramped above the brick top edge if you match the beige vinyl color.. I think it will fool the eye. This will give you a needed dimension to a rather flat facade of the home. Adds some architecture and a visual interest.

Friends have the Charcoal Gray on a West facing wall. They have had to have it replaced as it faded within the first year. Has done so again, so they are painting that wall. I'd be tempted to with light on the color.