If this is a gas or oil fired heating system, the first thing you need to determine is how the system is supplied with combustion air. Likely the system is a retrofit (old building) and it 'could' be relying on a leaky building for oxygen.

If it has a vented air supply, then you can proceed to tighten it up.

You can not leave foam board exposed (fire spread). It MUST be covered by drywall.

To make things easier, you will want to disassemble as much of the ductwork as possible. This style is easier to take apart than it sounds like.

There is a mess of wiring to deal with and possibly? a water line or drain line--maybe a condensate line for the HVAC? You will need to hire an electrician & clearly explain to him what you plan to do with the walls & ceiling so he can route the wiring accordingly.

If space allows, you may decide to just add a layer of drywall over the existing, shimming exposed studs to the depth of the existing plaster for at least part of it. Some looks to be quite the mess & undetermined is what will be left to deal with after the electrician is finished.

All in all, I'm going to say that working inside of a closet is a good place for a newby DIY to begin. If it turns out way less than perfect, you can paint & close the door on it.

OTOH, I'll add that 'this' closet isn't an ideal place for a beginner. For one thing, it is going to take way longer than you think it will. For another--do you have the tools needed? If not, there is added expense plus the time needed to shop & accumulate them. During which time, the HVAC system won't be operating the way it was designed. So, if you decide to tackle it, this time of year is a good time--moderate weather so you won't absolutely have to have conditioned space.

At a minimum--IF the heater has an outside air combustion supply, you could lightly stuff some rock wool insulation in the holes/ tack up batts, etc. to deaden staircase noise & reduce air infiltration. Then close the door on it for now. Later, when you have acquired some basic tools and a few skills, then tackle it.

I believe that is rock lath which was used in the 50s, the metal is the "cornerbead." The underside of stairs are required to be fireblocked to prevent passage of smoke and fire between stories, so all this needs to be sealed up.

This is accomplished easiest by 1/2" drywall. You would need to cut out some of the rock lath over to some studs that you can screw your drywall to. A reciprocating saw is best, oscillating will work too. Metal blade to cut if mesh behind, but realize the cementious board will dull the blades quick. Tin snips or wire cutters for any exposed mesh. Then piece in same depth with drywall. You can use cheap painters caulk to seal any tiny gaps around the duct if it protrudes out of the drywall.

If you want some sound prevention, put in mineral wool batts or get a cheap roll of fiberglass batts and stuff in anywhere you can reach.

Thank you both for your insights! It sounds like it will be a bit of an undertaking. I think I may start out with some drywall and caulk in areas I can easily fit and attach (plus some batts) without disrupting the HVAC ductwork - depending on how that goes, I may call my handyman in. I appreciate your expertise!