That is definitely an odd look. If it was me, I would take off the trim completely and just run the baseboard to the edge of the opening. Unfortunately, I think it will still look a bit odd once you do this because the upper part of the opening is unusual. Have you thought of rounding it off and making an arch? This isn't hard, time-consuming, or expensive to do. I just had two done in a kitchen remodel--both alterations of existing doorways. (By the way, none of my archways--new or old--have trim on them and they look fine.)

Much, much better! Love your trim and details with the chair rail/moldings!

Because the brick around the door is sheltered by the overhang you could probably paint it without having to worry about damaging its ability to breathe. However, there is an alternative way to change the color of brick and mortar. I've been fascinated by DYEBRICK since I first discovered the company. The non-toxic, water-based stain soaks in so the texture of the brick is retained. In addition to traditional brick colors they offer LimeLike that can significantly whiten even dark brick. Final color is determined by dilution rate with the fixative. HERE is a photo of one job. The only problem is that people from the U.S. and Canada have to order the product from England.

Hi K Zie! When you say that it is 24" on the interior, do you mean from the inside of the door jamb? Also, what kind of door was on there before--bi-fold, sliding, swing open out? If you took the door trim off, could you use a barn door type sliding door in that space?