The average person does not realize the effort an equestrian puts into riding. And for this discussion I am only refering to time in the saddle. Riders all spend many hours of physical labor un-mounted, as any horse owner can attest. But why does riding burn calories, after all we are sitting and letting the horse do the work? This epends on your definition and understanding of work. Your body does work to keep you alive, with beating heart, breathing and digesting food to name a few.But we each add to this each day. A minimal activity day might include TV channel surfing, mouse clicking and drinking soda or coffee. Added activity could include "non-purposefull movement" such as toe tapping, finger popping, fidgeting and other ofter nervous habits. One of the current theories of why some people stay so thin has to do with these un-purposed movements. Over time these can and do burn a substansial number of calories.
But I digress. What seems harder - sitting still and unmoving on a bouncing ball or bouncing the ball? Well not really a very fair question but it points out that the act of holding your own position atop a moving animal requires substantial muscular coordination and energy expenditure. This requires calories to be expended. And quite logically the faster and more jarring the gait and the quieter the rider sits, the more isometric muscle activity is needed. (Isometric means muscle use without changing length - like holdng thighs down and pelvis erect during a medium trot.)
So riders do need more fitness to ride well than first meets the eye. Cardiovascular endurance is also important as more dynamically moving horses are ridden and in ever longer tests. This endurance can be gained in the saddle but more easily in traditional endurance activities such as walking briskly, jogging, biking, swimming and eliptical training. And do not forget dancing (esp. Zumba) and even Hula Hoop use. It can be fun and entertaining. So develop more endurance and let riding develop your muscles, then all you need is stratching and work to keep balance throughout your body and life. Have fun!