What sport can you think of which does not require any physical challenge? None. All sports require the player to be physically fit to be able to stand the chance of winning.
Being fit is not limited to the physical, though. Your mind should be fit, too, so you can think sanely how the game should be played without breaking the rules. Not only that, your attire should be fit or apt to the game too. An equestrian would have several tailored sportsman breeches in stock even if he does not compete and just ride for pleasure. If the pair is loose or tight, riding will not be as good.