From what you described removing it sounds fine. But I'd recommend you bring a pro in to inspect it and patch it up, especially if you need to patch an exterior wall. You'll want it done correctly so no air or critters make their way in.

The apparent contradiction of cooling inadequacy vs. heating sufficiency is not a contradiction at all. It is basic physics and it can not be messed with. If you have the heat transfer rates and capacities/contributions of both heating and cooling appliances arithmetic will make sense of it. I specify contribution because you need to know if they are operating up to their rated capacity. Condition of the ducts also contributes, to both heating and cooling as you pointed out. Of course, and the calculations involving existing construction vs. new construction is sometimes challenging due to unknowns in the old stuff. Rather than estimates of heat transfer using assumptions about the construction, you can do better with fuel use rates. You should be able to get real heat contribution from the furnace and fuel use records (and heating degree days) from the coldest weeks of the year when space heating makes hot water and cooking insignificant. You also need to know the efficiency of the furnace. An energy rater or engineer should be able to use that data to calculate the heat transfer rates of the structure. (BTU in must equal BTU out.) Those rates can also be used for heat transfer in the other direction in the summer. Those calculations can be done today. You have fuel use records and you can get degree days from a nearby weather station. It is somewhat less common to use energy use of the cooling appliance for estimates, but it should work just as well. You need to separate electricity use by installing a separate meter and the contribution of humidity removal will have to be factored in. That means that you'd have to record humidity inside and outside and include that in the calculation. It is likely that you have sufficient capacity for you furnace or it is oversized whereas the cooling capacity is not adequate if your cooling unit is functioning properly. Again, calculations will tell you. How do you fix it? Better insulation on the ducts, structure envelope and better sealing of both will always cost less in the long run than installing a larger appliance. Bringing the ducts indoors is also a good idea as pointed out by mike_home. Don't think of active fans if you consider better attic ventilation. It needs to be passive. it is difficult, however, to draw enough air through an attic to make much of a difference in its temperature. Most of the heat transfer into the living space is due to re-radiation of heat from the underside of the roof deck. That is why radiation barriers were so "hot" for a time. The trouble with them is that they get dirty and lose effectiveness pretty quickly. It was a nice idea, but did not work as expected.

The 9 ft ceilings might be your friend. Look at that level with an eye towards furring down in a way that makes it look like it belongs rather than having been added in. If you try to make it architecturally interesting rather than hiding it, you might make it work. The AHU for minimally-ducted (low static pressure) mini split units are very compact and if you use two or three of them, you may find places to steal enough space for them inside the existing envelope. You might steal the upper reaches of a closet in one room and part of the adjoining room's closet upper reaches for ducting in that room and the next. You may lower the ceiling in an entry area and it can look good. Varying ceiling heights can be pleasing to the eye. I am going to assume that if you don't like the look of the most common, high wall mini splits, you won't like any kind of exposed ducting ;-) There are also ceiling cassette mini splits and floor consoles. Have you seen them?

I think you need to clarify what Charles is talking about. Is the whole attic conditioned? I think it is but definitely need to clarify. I suppose this is obvious - the location of a air handler should be as centralized as possible. The plan you describe is terribly uncentralized. It is part of the problem with a rambling design. The heat loss or gain on that master closet is going to be a problem. It has almost 3 walls exposed and it is seemingly 80 feet from your air handler. You will need/want (my code would say need) a return in your master bedroom. That - by rough sizing - would need to be at least an 8 inch duct and probably 10 or 12. Which means that you have no good way of ducting that back corner. Why are you so restricted on your air handler locations? What climate are you in? I am going to guess that it is pretty warm but you should clarify. Looks like TX to me.