Suggest that you spray with neem. Its safe to use on any leaf herb, and should kill the buggers. Safer makes a RTU product and its also available in concentrates for mixing with water in sprayers.

Since you have just a few plants, you may find it easier to get rid of the aphids (which is what those little guys are) by the always popular spraying (with water) and squishing method. Keep after them a couple minutes a day for a few days and you'll have them under control.

Neem can be used inside, but you may find the odor objectionable.

I get aphids on my plants from time to time. They come in various colors, from the most common green color, like yours, to black or a mustard-yellow colored one. I always just squish 'em with my fingers and hose off.

I'm most likely to get aphids if there is excessive water in the soil, either from early spring rains, or if I've watered too much. Usually I do that (overwater) if we've had a sudden heat wave, like a 25 degree temp jump in a short period, like 48 hrs, and I've misjudged on watering needs. This means that when you are hosing off the squished aphids, you have to take care not to keep the soil too wet, so I don't hose 'em off every day, or in some way I try to avoid soaking the soil more than the plants' roots prefer. (Most like the surface of the soil dry down 3/4 " or so, that's about 1 knuckle down, at least, before watering.) Basil, in particular, can suddenly take to wilting and dying if overwatered. I've had aphids appear on chives for the same reason.
Hope this helps.

Thanks, everything helped. Now I have a plan of attack and a mind for watering so that they hopefully don't come back.

Neem has not really strong smell, but it can have a smell depending on what emulsifier its mixed with. Some use a Castile soap, while others use alcohol to allow it to mix with water. It does have a residual action being a repellent too.