Exercise requires energy, and this energy is measured using Calories (or more accurately, kilocalories). Considering 1 pound of fat is the equivalent of 3,500 Calories - you can see that it takes a lot of exercise to burn fat.

In addition to this, the source for energy may come from either fat or muscle glycogen. As the body adapts to each exercise, it becomes more efficient in it's use of Calories - therefore burning less!

The amount of muscle also has an impact on Calories burned. Because muscle is metabolically active - more muscle means more Calories being burned. This explains why strength training is such a good fat loss exercise.