You really need to figure out what is going on before you touch *ANYTHING*. There's no NEUTRAL on this switch PERIOD. If you actually managed to put HOT and NEUTRAL across the switch, the breaker/fuse will blow as soon as you turn it on. The first thing you have to determine is if you have a switch loop or an incoming feed going through the switches to the light and fan. I suspect the prior. Hopefully you've not disconnected the wires from the old switch yet. If you haven't we can help you diagnose things. If you have, you'll need an electrician to unscramble the mess you've made.

First, the green wire on the switch is for grounding, and must be connected to a green or bare copper wire from the wall. If you don't have one of those, you'll either need to rewire, or purchase a switch that doesn't require a ground. White wires (neutrals) should be left alone. There is not enough information provided to tell you about how the other wires should be connected. In a typical set up, one of the black wires in one cable is your supply. The other black and the red will be in a second cable and will go to the fan/light combination. The instructions in the switch package will tell you which wire should be connected to which. There's no guarantee that the new and old switch will follow the same wire color scheme. Make sure that all of your connections are tightly made using approved wire nuts.

I would replace the capacitor if I was you, continuing to do that to your fan can burn out the motor eventually. Note: I cannot edit what I posted a couple posts above (pre houzz merge), what I said about the capacitor was wrong. The capacitor charges constantly when supplied power, it just discharges when the fan is turned on, giving the boost needed to make the fan move from stand still. After that, the fan motor keeps the fan moving. It wouldnt hurt to give it a minute to charge before using after replacing with new. In reality it probably only takes like 10 to 15 seconds. That is what I was going for with that comment. Probably the time walking from the panel to the switch is long enough.

The bulb in this socket draws enough current that the spurious voltages will not occur. I assume that there are two black/white pairs which come from the breaker box. It's easy to see which black is which. First, connect the white on this socket to the twisted whites and touch the black on the test light to each of the blacks and see if any of the blacks are hot. Once you have confirmed that at least one white/black pair is powered, turn off that breaker and make these temporary connections more secure (not just touching live wires). Turn the breaker back on and then turn on and off the likely wall switches. If none of the switches make the test light go off, then you have found a source of power. Repeat the above for all the black wires and see if any of them are switched. Finally, and I'm not sure about the following test, but if one of the black wires never shows power, it must go to another light fixture or receptacle. Connect your test light between the black power source and this unknown black wire. If your test light glows, then that black wire goes to something which draws current (perhaps a ceiling light with an incandescent bulb in it), and you can figure out what that is by observation. If it doesn't glow, then take another test light, put a regular plug on it, and plug it into each of the likely receptacles. If this second test light glows, then the two lights are in series and you know where this unknown wire goes and you can nut it to the power source wire (or to the switched wire, if you want that receptacle switched). One thing I don't know is how to trace out the white wires. If each of them act as a reliable neutral, then maybe it doesn't matter. If it does matter, I don't know how to identify which is which. Sorry for this long response; I hope I'm clear.