The formation and development of caves is known as speleogenesis. Caves are formed by various geologic processes and can be variable sizes. These may involve a combination of chemical processes, erosion from water, tectonic forces, microorganisms, pressure, and atmospheric influences. Around here, of course, we have solution caves made by rainwater.
This is a little series of flash videos by NOVA that show how caves form.