You're kidding right? Skateboarding is by huge amounts the most challenging "sport" there is.
No doubt there are many, many talented athlets out there. But what do you define as "the most challenging"? Strength, speed, power, agility, coordination, endurance, spatial reasoning, reaction time, mental toughness … the list goes on. It often ends up being a subjective measure. Pretty much every sport/activity at the elite level is near-impossible for the average, un-trained, un-gifted person. That is what makes it elite. There are many, many "fringe" sports most people never heard of that could probably qualify as candidates too.