I take my kids to the bmx track too, but I just ride the pump track on my 26" DJ.
Fun, great workout, and I'm on a bike that has familiar reach and mannerisms. You guys at Inter River? They have a pump track yet?
I used to race 24" before kids, but these days I'm a little more realistic about trying to be beat the other adults at my track who tend to be former national champ types on carbon 20" bikes…
If you haven't found out yet, 20" bikes are a really different animal than MTB's and your skills will mostly not translate - 75° head angles, unimaginable rigidity and tiny unstable wheels add up to an alien feeling bike. 24" is more stable, but still a long ways from anything familiar. And 24" is slower than 20" if you're looking to be eventually competitive.
Btw, Danscomp size chart would have you on a Pro XXL.
One option would be to grab a bmx rental from the track for a few sessions and see how it feels? Most tracks have rentals. Like 3 sheets said, be sure to bring long pants and a long sleeve shirt. Helmets are usually available for rent too.
Another option is just race on a 26" pump track/DJ bike. They'll let you in the 24" division. You'll be handicapped by the bigger wheels, but at least you'll be able to ride the damn thing. ;)
Btw, beat your kids badly now. Just smoke em and talk a bunch of smack. It takes them about 2 weeks to get fast as hell and leave you in the dust.