It sort of sounds like you have an ill-fitting flue pipe and the ash byproducts of the oil are finding there way through the gap. Maybe a few screws to draw the connection tighter or a large stainless steel hose clamp.

How old is your furnace ? White " grit " could be anything.
Without seeing your furnace it's anyone's guess. But i'll
take a guess. It's probably refactor cement coming apart in the fire box and blowing up threw the flue and if that is what it is i wouldn't worry too much about wrapping it up or putting in a few screws or clamps. It means your fire pot is disintegrating. What that means is replace your furnace NOW or have someone in to check it out NOW. If it is a refractor pot it can be rebuilt. This can be caused by a wrong nozzel angle or over firing. Bad and contaminated fuel will cause this but if it is gritty and feels like sand it's refractor cement. The only way you will know is by calling someone in. Do not cover it up or " hide " it. You are probably in trouble. A normal operating furnace will never show this grit. If they find nothing wrong no biggy. You just got piece of mind with a new furnace inspection. Good luck

Was the unit internally cleaned recently?

The furnace is almost 20 years old, but I can think back & remember grit deposits from almost day one. I get it cleaned & serviced every year by my oil guy. I'll make sure to mention this grit situation to him.

If the grit situation turns out to be not so serious, I'd still like to try to control it. Someone mentioned that car muffler tape can be used on the seams. Can regular aluminum tape be used or will its adhesive not hold up to the heat?

Has anyone inspected the chimney ? Muffler tape will work
but the thing is it was never intended to wrap furnace pipes because furnace pipes do not normally spill out grit.
Are your flue pipes galvanized ? If kept up a 20 year old
furnace is not old. They actually made a good furnace in
those days. i wouldn't blame the furnace just yet. there might be something else causing this. I've seen one insident that is very rare might not be your problem but i
was with my journeyman on a call. White grit was on the
flue pipes. took us 8 months to figure it out. The basement
was very damp. Very hard ground water in the area. As the
moist air condensed on the flue pipes it left behind
the calcium from the water. The flue pipe turned white. you
don't say where the furnuce is located. If the grit collects underneath i'd think, Moisture ? It's a stab in the dark. See what your oil guy says. Oh yah. You might not want to do this but taste it. Is it chalky ? If it is chalky it's calcium. Actually if you can write your name in it it is calcium. As i said before see what your oil guy says. After all he is the one you are paying.

When I've seen this problem is when the combustion ash, which is a multi-colored deposit, flakes loose and is carried in the draft and can come out of loose fitting flue pipes.

Baymee. I think you have it. The thing is where i am our
heating oil is filtered. The filtered oil is filtered and
filtered again. Plus there is an additive put in it so it
can't degrade over time. 90 percent of our tanks are outside. Subject to -50. We have heating seasons that last 10 months. I see that you guys use #2 heating oil. Up hear
that is summer diesle.You can't get #2 heating oil up here. It doesn't exist. At 40 below up here that stuff won't come out of a 1/2 inch pipe. Our number 1 heating oil has a carbon count of 6 - 10. number 2 heating oil,diesle has a carbon count of 11 - 14. you guys use #2
oil. You guys don't get a cold winter. Warning to all the other posters. Don't even think of telling me all diesle and heating oil is the same. your number 2 heating oil could be contaminated or not that pure. That is why i thing
you have it, Baymee. i never come across " bad " oil. Up here.

sounds like oil ash, i've seen similar situations many times where the white/multi-colored ash builds up in flue pipes or comes out of leaks in smoke pipe joints when the pipe is sloped and the male end is pointing up. not uncommon at all. should be able to get some high-temp silicone at lowes etc. holds up to 500ºF which should work fine on an oil-burning appliance; smear it on all the joints. i've seen larger boilers burning #2 fuel oil where the ash is carried out in 5 gallon pails yearly (yes some of it is rust flakes etc. from the boiler corroding, but much of it is ash as #2 fuel oil does have a very small ash content filtered or not)