the latex will be fine

You probably do not need a primer. I'm not even sure that I would use an oil based coating. A high quality acrylic floor coating would probably be fine. Dutch Boy is certainly not considered high quality, but the issue could have been curing. If you pull tape and do not score the line where you want the tape to break away, you can pull the paint off. The paint will stick better and bond better and become more durable as it cures (up to a month). Paints in bathrooms can take longer to cure because you are showering in there and putting moisture in the air which can sometimes re-wet the paint film. Use the fan, take shorter showers, shower with the door open, etc..whatever you can do to help that paint cure. There are no-skid, non-slip additives you can add to the paint if you are worried the surface will be too slick.

Assuming you're using clear poly, generally oil based poly will bring out the wood color somewhat which is the opposite of what you seem to have observed. The finish will yellow over time but I doubt you would have observed this in a short period of time. I'm not sure what you consider shiny. Satin oil poly is not shiny but if it's too shiny for your taste there are duller oil based finishes. It's important to stir the poly thoroughly because the flatting agent settles to the bottom. You can also pour some of the poly off the top before stirring to get a duller result. I like the oil finish since it's tough, easy to use and dries slowly allowing time to apply uniformly. The downside is that the longer drying time can allow time for dust to settle in the finish.

Thank you. I sanded too hard after applying the poly so needed to refinish. Sanded down, and used a gel stain this time and a two coats of poly over it. Turned out better I think!