Fillers from cabinet companies come in 3" and 6" widths and are ripped in the field to the desired size. Partial overlay cabinets usually do not need fillers for clearances because the exposed face frame gives them more clearance against doorways, walls, and adjacent cabinets. Full overlay and frameless cabinets will always need fillers against walls to account for out of squareness and to give clearance to the knobs so they don't bang against the walls. They will also need fillers against tall cabinets for the same reason, and for appliances that have clearance issues. In a small kitchen, some people opt to go without some of these fillers becaue each fraction of an inch is valuable, and then they end up having to use "cabinet stops" on their doors to keep them from banging walls or ajdacent tall cabinets. Using fillers works much better, really. It is up to the KD to understand the cabinet line's clearance needs and to specify the proper fillers in the design. Companies that order cabinets generally do not provide any design services or check over your order to see if you've specified every cabinet with the correct attributes for how it is being used in the design (matching interior, finished sides, etc.) That is the role of a KD, and you are acting as the KD in this instance.

In your design above, you will need a minimum of a 1 1/2" filler for both the top and bottom cabinets against both walls of the upper runs. (I would also rip a filler overlay to the same dimensions so that it ddoesn't appear to have a "recess" next to your cabinet door.) If your moldings on your doorway are especially prominent, you may need to increase that size. Next to your fridge and a wall, depending on the fridge specs, I'd use either 2 1 1/2" fillers or two 3" ones together with the proper supportive side panels for the fridge cabinet above. You will need a minimum of a 1 1/2" filler against the side wall for your pantry, and if it has pull outs, you'll need a 3" one. This directly affects the size of the pantry that you can use if you are planning on that run being only 24" deep. What type of panel you choose to finish off the side of the pantry cabinet will also affect the overall depth of the run as well, so be sure you get those details right before you order.

When beginning a cabinet design, place the fillers first, and since cabinets come in 3" increments, using a 1 1/2" filler against a right and left wall will only take up 3" of cabinet space. Lay out your cabinets, and if you have space left over, then you can increase/decrease the size of your fillers. NEVER go below 1" for wall clearances or 3" if wall clearances are needed for roll out trays.