Coat potatoes with olive oil, and sprinkle with salt and pepper. Place on a baking sheet and bake for 45 minutes.
Remove potatoes from oven, and let cool. Chop into 1-inch pieces.
Reduce oven temperature to 350°F/180°C.
Lay half of the potatoes on the bottom of a greased 9x13 baking dish. Sprinkle on half of the bacon, spread half of the sour cream, and sprinkle on half of the cheese. Repeat for a second layer.
Bake for 20 minutes, or until cheese is golden brown and the edges are crisping up.