Get a ladder and check to see if there is actually a nest there. They may just be hunting bugs that are in the gutter. It would be unusual for them to build there.

Thanks easternmass! I removed it last night and there is still a large intact nest, They have checked out their nest this morning and I and will look for more eggs later today.

I just did this too! They will abandon the nest if you don't take care of it, ASAP!! tonight if you can!. You need to do a nest change. There is good info on the sialis site about nest changes and how to do it. I got as much dried grass as I could and put it in an empty butter container. I tried to line it with pine needles, but I don't know where my female got hers?!! It took a few days, but then the second egg arrived along with a third! She was incubating a few days until a wren, I think, got two of them. Now they built another nest in my other box.. after all that work!! then you have to take care of the ants. They say grease the pole.. I put the liquid ant killer at the base on a plastic lid covered with a larger lid.. I hoped it would distract them from the nest, but I can't tell for sure if it worked since they moved.. but I didn't see any more ants in there. Good luck and keep us posted!

Are you certain that the bird building the new nest is the female bluebird and not a usurping English house sparrow? Typically HOSPs use straw as a nesting material while bluebirds favor fine grasses.