You like it, cool. But I'd go even further and add that racing is paying to ride a trail not of my choosing, on a date that I don't set, with a big crowd of people who mostly aren't my riding buddies. Sounds like hell. That's my opinion on why more people don't race.
That's why YOU don't like it. That's what I like most about it.
Going somewhere i don't know (albeit usually with some people i DO know) and putting in my best effort - win lose or draw. I've had some wins, some crashes and some events that just plain sucked the life out of me. The date thing, meh, you pick the races you can make it to, de facto agreeing to the date.
If you don't get it, you maybe never will. And that's OK.
Like Oldfart said, racing is racing. It's not about the trail, it's about the competition, and some folks aren't into that.