post photos if you can

Mineral paint technology like Romabio and PermaTint has been used for HUNDREDS of years in Europe on all types of masonry. Tons of info available about this, it's an easy Google so you can get the facts and decide for yourself. Painting brick is a maintenance nightmare only for people who don't know to do it right. I agree. Paint it. The house is not attractive as is. The mix of stone and brick is weird. This house is the result of someone trying to get cute with materials - that never ends well. I *might* try leaving the vertical brick trim as is just to see how it looks because it is well done. You can always paint it later. In general, however, you can get a bucket of Romabio, apply it to a section of the house and if you like it, then leave it and keep going. One color could make the mix of textures more attractive, interesting. If you don't like it, hose it off. It's mineral paint so you can do that. Read and follow the directions to a "T".

We want to change siding in the coming years but can't yet. Was thinking that painting the brick could modernize the outdoor in the meantime. Thoughts? Color suggestions?

I agree. Your house is very nice and the brick is very nice. Please leave it like it is, even if it isnt what you want, since you mention downsizing. New owners may buy it just because its brick and maintenance free. Sometimes, painting brick, can even devaluate your house. I would leave that decision up to who ever ends up buying it eventually. I would definitely work on the landscaping which looks like it has seen better days. Add some large colored pots with some bright colored flowers.