Oh no, the yellow would have to go... I very much agree with PaintColorHelp's thoughts, above. The siding does not complement the brick and I think that is the main issue. I think my personal preference, with your house, would be red siding and grey trim, so I would be looking to paint the siding a red color. Another option could be grey siding with white trim. Some inspiration pics:

You could try it without shutters, if that is allowed. If not, black Or sage green Either would be beautiful :) I'm sorry I don't have specific colours to suggest.

I don’t think having each half be a different color is a good idea. Maybe paint them the same color (yes, consider painting the siding the color of the brick). These homes have yellow brick, but they are more charming (sorry) because they’re one material. Landscaping would do wonders, it always does. Paint the front door a fun color, make the front more inviting. Inspiration for beauty, appeal and colors: LOOOVE the red with the brick. Never would have thought of it, that green tile roof looks spectacular with the brick. A whole street of yellow brick homes.

I like the white siding with it. I would paint the shutters. And the front door. Maybe a pale blue? Green would also be pretty!! Maybe a new garage door. Wood? More lighting? Landscaping I like simple. I would remove everything blocking the front door view. And yes a neat row of boxwoods. I did suggest more than you asked. And obviously you should do what you can. But I suggested these to give you the full pic... Hope was helpful