The depth of the soil is the important thing and will vary depending on what you want to grow. We can't tell from the picture how deep the box is. However, if the sides rise much above the soil level light available to the plants will be reduced, as it also will be by proximity to that fence. And you are right that the level of the filling will drop. Can you tell us what the whitish looking material is around the base of the box? Have you put something beneath it? Does it rest on the native soil? If you had wanted to save money you could have made an in-ground bed, amended the soil and and just edged it with your timber. It is not necessary to buy in soil or growing medium unless there is some specific problem with the native earth.