It may be the way your washing the walls. Try using the TSP BUT use two or three buckets when washing. One for the solution, one to rise out the sponge in and wringing it out and then another to rise the sponge before you put it in the TSP solution again.
Try changing out all the buckets when you start a new wall.
It might be that residue from the walls is staying on your cleaning sponge and all your really doing is moving the glue around.
The only other option I can think of is to clean the walls as best you can and then prime them with a stain blocker which if your walls are unpainted plaster you should do anyhow before you paint.

Try vinegar in warm water using one of those green scrubby things, followed by plain water and a cloth or sponge for rinse. A lot of glues used for wallpaper will dissolve in vinegar solution.

Might the patches be skimcoat? In which case, you want it to stay right where it is.
If you've washed them down well, the rest may just be diminishing returns. Paint will stick to a little glue residue. And what do you mean by "show"?

At some point, you just have to put down the razor and the putty knife and trust to a coat of primer, 2 coats of paint and well-placed artwork.

I agree with growlery. I wash thoroughly with hot water and TSP, rinse and I'm done. Primer seems to take care of any problem spots, which are probably not glue anyway; glue dissolves fairly readily.

You may need to hunt around for real TSP (tri sodium phosphate). The 'TSP substitutes' are mostly sodium metasilicate and are not nearly as effective.
Real TSP requires rubber gloves since it will produce chemical burns of allowed to remain on skin.

"Spic 'n Span" is TSP, or at least it used to be.

The thing is that a lot of localities (states?) have outlawed phosphate detergents, since they are very polluting in the water. I know that in New York I can't get real TSP because of that, and hunting around for it (unless I'm willing to drive to other states) isn't going to help.

Googling around I find this from the University of Colorado:
"In the end, local needs for immediate action to curtail eutrophication coupled with scientific, judicial, and popular support resulted in the patchwork of legislation the Industry had feared. By 1985, jurisdictions which had enacted phosphate bans included New York, Michigan, Indiana, Vermont, Minnesota, Dade County, Florida, Akron, Ohio, and Chicago Illinois (footnote 257 in Fleming et al. 1986). Typical are state statutes which limit phosphate content for certain types of detergents and in certain areas. For example in Pennsylvania, the Water and Sewage Phosphate Detergent Act of 1989 and amended in 1992 (PA ST 35 P.S. §§ 722.1 - 722.3) affects "all counties partially or wholly within the Susquehanna River Watershed or in the Lake Erie Watershed." This Act prohibits the manufacture, sale or distribution of any cleaning agents containing any phosphate, except contained incidentally during manufacture. But excluded from this ban are cleaning agents used in dairy, beverage and food processing equipment, in hospitals and health care facilities, in agricultural production, by industries for metal cleaning, in biological and chemical research facilities, and those used in the household for cleaning windows, sinks, counters, stoves, tubs and other food preparation surfaces and plumbing fixtures. Dishwashing detergents are allowed to be up to 8.7% phosphorus by weight."

I guess you can still find phosphates if you look, if you're not in the states mentioned. Look at dishwashing detergents. But if you use detergents for automatic dishwashers, wear gloves! That stuff is extremely alkaline and will burn you in high enough concentrations.

A little floor stripper in water does the trick! The kind you use to strip wax off a floor.

I had this same problem and tried everything that was recommended by my paint guy, nothing worked. In desperation I got out some masking tape and applied a strip over a section of my problem area, peeled back the tape and the glue underneath stuck to the tape. I then switched to duct tape and it worked just fine.

Hot water and lots of white vinegar is the cheapest way to remove wallpaper glue/paste. Keep changing the water or use two pails. One with clean water/vinegar and the second to rinse the rag out with. I used a "sham wow" you know--- the orange cloth advertised on "as seen on tv". It's a ------ but it will work. I am now on my fifth and last room. I wouldn't have wallpaper again if is was put up for free. Good luck.

For Barabara Clayton: What's the dilution of water to white vinegar you used? My bathroom looks terrible after we painted. There must have been glue and invisible paper stuck to the wall and wehn we painted over it everything shows!!