Yup, the rotors are probably the biggest weak point. I might start with replacing pads/rotors and then if I'm still wanting for more power I might try to sub-in a 4 piston caliper (MT420, 520? Whatever's cheap?) OR go ball-out on a set of Formula Curas (2 or 4... they both get great reviews).
I've been having tons of fun bombing around on more uppie/downie trails than the typical winch and plummet stuff so haven't felt too held-back by the current brakes.