We had this same problem when we built last year. We have no white on the exterior of the house but have creamy white trim on the inside everywhere except DH's study. When it came time to put shutters on the walk-in closet, we chose to use the stained shutters with white trim and it looks fine. Here's a picture:

KellyC I came in here today to see if anyone had suggestions to your exact dilemma what to do with the brick. Mine is a strange color as well, and painting it was suggested to me as well. I like just needing to pressure clean it now and then better then painting and it needing to be painted again and again. I like the idea of a stain, have you gotten any feed back on a stain, something that would saturate into the brick and stain it darker. Mine too has the random dry brush paint swishes, year 1970. When they used well water to water the lawn it stained the brick a rusty color and I didn't hate that. That I could work with some earthy tones. Also considered the "smear" or however they spell it but I have way too much of it and it might look like a mess when finished.

I think a grayed olive green for the trim, and a deeper shade of the same green on the door would look great. I would look for light fixtures that coordinate with the door hardware. Try SW Oakmoss for the door color, and one of the lighter shades for the trim.

Try to complement the most prominent features, which are the roof and the stone around pool. Below is a link to SW paint visualizer to help. https://www.sherwin-williams.com/homeowners/color/color-tools/colorsnap-mobile I’d look for a muted grey/blue/green, which will work with roof/stone/pool and the surroundings. It would also feel soft and sky like when viewing from inside when shutters are closed.