Your photos didn’t load. Please try adding them in the comments.

I love hydrangea, lavender, roses, peonies, all the old fashioned plants. I just moved here and there are beautiful traditional plantings like spiraea, hostas, and variegated euonymus. Maybe to make more room I could remove the stones.

Good idea to move the stones. How large is the area? I am guessing about 4'x8' assuming that the stones are 1' square. Is this area mostly viewed from close in such as by a walkway or from farther away such as from the road or both? A photo of the wider area including surroundings would be helpful so that this area works well with its surroundings.

If you use mostly perennials and roses, there won't be much late fall - winter interest. Is this an area that you walk by daily such as the path to the door? If so, put in something evergreen, at least flowering evergreen groundcovers. These would include cottage pinks such as Dianthus 'Firewitch' or 'Greystone' or Veronica 'Georgia Blue' or even perennial candytuft/Iberis sempervirens, which isn't strictly a groundcover, but could be planted as several plants close to each other to function like one. Or a broadleafed evergreen such as rhododendron or mountain laurel/Kalmia, both of which have spring flowers, but retain winter leaves.

I might put in a flowering evergreen groundcover, a couple of the smaller panicled hydrangeas such as Bobo or Little Quickfire which will give summer-fall blooms (and I find the dried winter flowers ornamental), and then spring bulbs such as crocus or small daffodils for early spring interest. You could fit several peonies and one hydrangea into the area as well. Because it's a relatively small area, you don't want to use too many different types of plants unless you like the slightly chaotic look of a cottage garden and that style would fit with the rest of your plantings.

Here’s a few pics to show some other areas of the garden and also this particular area which is on the side. Thanks for all the great thoughts! Should I wait until fall or spring if I go with some hydrangeas and roses?

I like to plant in the fall since the soil is warmer in the spring, but the air is often cooler, so it is less stressful for the plants.

Try Rainbow or Coast Leucothoe