If you do a fully integrated fridge, fine. If you're just doing an obvious wood panel on a fridge where the gasket etc. shows, then no, it's not worth doing.

I think an integrated fridge (or, an integrated-looking fridge, like a built-in with an overlay installed, or even a stand-alone fridge that's inset into an opening in cabinets or into a closet-type space), w/ panels to match either the cabinets or perhaps plain to match the surrounding wall (for example, if the fridge is inset into a drywall-covered closet-like space), is fine.

As long as it looks nice, that's all.

Sure, it's nice to be able to stick stuff on the front of the fridge. But even w/ little kids, it's not necessary.
There's a stereotype about "putting their artwork up on the fridge," but you can use any vertical space for that. Or, don't do it at all.
There's a stereotype about "kids playing with the fridge magnets," but you don't need to do that either. (Often you can't w/ stainless anyway!)

(Heck, you could rig a way to put Homosote on the fridge and have a bulletin-board front if you wanted to.)

Me, I'm a big fan of a white fridge. I have dark cabinets and white counters and woodwork, so I decided that my white appliances are simply an extension of my counters.

HERE ARE SOME IMAGES TO LOOK AT
(I googled "integrated refrigerator")

Look at this one: the covering panels only sort of match the cabinetry; they could be painted wood, or anything.
https://www.houzz.com/products/miele-k-1801-refrigerator-fully-integrated-prvw-vr~816262-K-1801-Refrigerator-Fully-Integrated-refrigerators-new-york

Here's another; the center one has the panels that are flat melamine/wood, and if you had a surrounding drywall closet-type space, you could simply put wood there; the closet frame would minimize the effect of the gasket showing.
http://blog.bostonappliance.net/hs-fs/hub/308869/file-387359087-png/built-in-vs-integrated-vs-overlay.png

And here's an interesting article about the terminology, etc. It might help you frame your thinking, or your questions.
http://blog.bostonappliance.net/built-in-vs-integrated-vs-overlay-refrigerators#.VMJu4WTF9_g

In the first picture here, where a stand-alone fridge has been set into a framed space. Think of what that fridge would look like if it had plain white wood panels on the front--it wouldn't look that bad.

Note that it's the same depth as the opening, though--it doesn't stick out. That might look a little funny--even with paneling to match the cabinets.

http://www.remodelista.com/posts/5-favorites-counter-depth-refrigerators

It would be a fully flush integrated fridge (not counter depth with overlay. Cabs will be white and would like to do a wood paneled fridge instead of stainless but concerned about dirt showing more on the white than on the stainless (I feel like my current stainless fridge never looks clean and there are a few dents in it after 6 years of use.

Our last home had a built in fridge with white wood panels. It was quite nice. Use a quality paint and it cleans fine, never looked dirty. Go for it.

I don't think it's crazy. But how much more does it cost than buying a white refrigerator and just building a cabinetry box around it? And is it worth the extra cost?

It's a 8K+ premium to do a fully integrated paneled fridge vs. a plain white counter depth that fits into a cabinet alcove. That's the cost of a nice range.

I don't think the OP is asking for advice on how to spend her money. I think she is asking about cleaning.

Regarding cleaning, I would just make sure that you have a really good paint job because the refrigerator gets accessed so much. In our last kitchen, we had a white paneled sub zero but it was not real wood--it was Thermafoil--and it was constantly filthy from dirty hands going to the fridge to get something. You will just have to keep after it. And training the kids to use the pulls and wash their hands first might help although I had no success with that!

I think a plain white refrigerator will be easier to clean than a paneled white wood refrigerator. If that doesn't suit then, IME a white wood cabinet doesn't show fingerprints quite as much as a stainless refrigerator, but it's not any easier to clean. The kind of finish will vary how hard it is to keep clean and how well it holds up, IME.

It depends on your household's habits, but I wouldn't personally do a white wood paneled fridge with kids around. White metal, sure.