Our soil is mostly clay.

Some of the inexpensive "weed barriers" (those you can see through) will allow "weeds" to grow right through them and become more of a problem as time goes by. The heavier duty landscape fabrics that a good landscape contractor would use does not allow "weeds" to root through them, although I have seen plant roots interwoven with the fabric.
The "weeds" growing in the pea gravel are getting nutrients from the dust and organic matter that has accumulated in the pea gravel over time, so you could "top dress" although much better would be to move that pea gravel and put what ever material you are "top dressing" with down and then put the gravel back.
I would, however, replace the pea gravel with some type of vegetative mulch.

you could start saving up sheets of cardboard and newspaper (shiny stuff removed). When you can get fall leaves - shredded lays down neater - then start laying newspaper around the perennials and cardboard in larger areas. Wet it down and then place several inches of leaves over the paper/cardboard. It should smother the grass and weeds between your desired plants. You needs to reapply leaves or other mulch as it breaks down and new seeds blow in to create the next family of weeds. But its easier than dealing with weeds/grass that is tied up in landscape fabric. I guess I would just leave the gravel where it is.

When I do newspaper layering, I use sections that are folded in the way that the newspaper comes, so there a a few sheets in each section. I fan then with overlapping sections in a circular fan around a plant. For trees you can fan cardboard in similar way. The key is having lots of the materials ready at one time, getting a large section done at once so you can get the paper covered adequately. Blocking the light should suppress the plants/grass below the newspaper/cardboard. If you want to use wood chips to keep the paper in place, that works great also. I actually use both chips and leaves, depending on the season.

Rachel

Pull up the weed cloth and pea gravel unless you want to get on your hand and knees and pull every weed.

When I landscaped we more than a few times had a client who wanted his river rock or pea gravel cleaned.

If you want it cleaned, and not have unsightly dead weeds replacing green weeds the only good way to do it is pull cloth edges till the pea gravel is in a pile.
If you want weed cloth which is a waste of time, put down new weed cloth or if it is the real heavy type just put it back in place.
Get a twenty to thirty gallon wash tub, fill it with water and use it to rinse off the worst dirt and weeds. The weeds not pulled by hand will float to the surface and most remain debri wil go out with the water.
Dump half the water after each tub full.

Shovel the clean gravel out of the tub with a flat shovel in a pile and when all is clean just shovel it back in place.

Either way, it looks like I have a workout on my hands. Thanks for your ideas, everyone.

Well, I figured if you have already liked letting fall leaves decompose and feed your plantings, the idea of smothering it all with the wetted cardboard/newspaper and then leaves on top would be in line with what you already do. Lifting out gravel and the cloth seems like so much work. Maybe you are worried you would build up too high if you cover it all? That might be a concern. My "smothered" areas have always flattened flush with the non-smothered nearby regions in time, but then again I have only smothered lawn, not gravel. Good luck to you and pick a cool day(s) to work!