The brakes on mountain bikes have evolved over the years. The predominant design is a variation of the cantilever brakes found on most bikes.
A cable running from a lever on the handlebars pulls the two levers on the brakes together. This squeezes the brake pads against the outside of the wheel.
Some bikes, especially those with suspension systems, use disc brakes. These work just like the disc brakes on a car.
The brake lever uses hydraulic fluid to transmit the force from your hand to the brake shoes. The handle presses a small piston that applies pressure to the fluid in the line. At the wheels, a larger piston squeezes the pads onto the disc. Since this piston is larger, the force is multiplied at the wheels. (See a diagram and learn more details about hydraulic force multiplication.)
The handle contains a small device that works something like the master cylinder in your car brakes. It makes sure that there is enough fluid in the reservoir that if the pads wear, or the fluid expands or contracts (as it does when heated or cooled), there will still be enough fluid in the system to actuate the brakes.