Proviso I'm not an hvac professional. But, at base, it sounds like you've got a pressure differential between the inside/outside of the house or possibly the cold attic - the lower floors are pulling in air because air will move to lower-pressure areas. Trying to track down exactly where the air is coming in from may be a bit of a fool's errand, in that if that pressure differential exists, the lower floor will pull in the air from somewhere - outdoors through doors or windows, or through ductwork from elsewhere. Close off one entirely, you may find it just moves elsewhere.

So perhaps focus slightly differently, on what bothers you. Is it the feeling of draftiness due to cold / warm air, affecting comfort? Is temperature okay otherwise? If so, one way to address would potentially be closing partially those vents (with a simple louvre) or redirecting the cold air flow in a different way in the room. (I'm kind of assuming you don't have any other significant issues or complaints about your hvac system).

I have had this issue in a few rooms seasonally (note these are not A/C system vents, my heat is hydronic). We felt the draft and was uncomfortable in certain places. We replaced some vent covers with types that partially close or can be redirected. It does not completely block the air, just keeps it from the feeling of draft. Overall it has resolved the issue (and of course important that we did not want to stop the exchange of fresh air, just moderate it some).

Note, since you've said it happens differently with windy weather outside, I strongly suspect it is differential pressure within the house (wind blows and there's all kinds of effects of faster air/lower pressure around the house).

It's possible there are other ways to address such as improving air flow within the conditioned space (so instead of cold air moving to reduce the pressure imbalance, it's warmer air doing so). But this is a hideously complex subject where the house has to be looked at as a system and I'm not going to guess how to do this or diagnose over internet.

Oh, another small thought - why did you turn off the fresh air system? What is that system? It's possible this is part of the issue, that the fresh air system was partially intended or designed to help deal with/moderate the pressure imbalances. Did this problem start occurring more when you stopped using that system?