I just finished emptying my big compost bin and found that in the very middle, where the compost was completely digested, there were no worms; around the outside, where there was still some identifiable stuff, there were tons of worms. So, perhaps your compost was completely done, and the worms headed off for more food? I have never heard of worm snakes. I have lots of brown snakes (that's their name, not just a description), and maybe they eat worms, but they are rarely in the compost. Some of the worms in the compost are 6 to 8 inches long, fat, and can move like lightning - maybe you have ubercompost worms.

Without knowing your location, I'll go out on a limb with Pine Sawfly larvae (Diprion similis)?

It does sound like one of the mud dobbers or potters wasps. Mud (dirt) daubers are a species of wasp. The adult wasp manufactures her nest, finds a grub, caterpillar, spider, etc. to paralyze (but not kill), places it in the nest and lays her egg. The wasp larvae is able to feed on the live (but immobile) prey until it is ready to emerge as an adult wasp.

If you can't keep picking them off, get some Bt spray. Works in ~24 hours, very effective.