Can you challenge down a tennis ladder?

Not on a well-run one. Upward only. Let challenges run downward and the strongest player in the band farms the bottom of the standings for safe wins, which is exactly the behavior a ladder exists to prevent. Play a friendly against whoever you like. It does not move the standings.