This is where being broke is the advantage! I couldn't afford any "mtb-specific" gear for years (all the money went into the bike) so I just rode with whatever thrift-store shorts and shirts I could find.
I also started riding mountain bikes when Down/Double Down and New World Disorder came out, so the whole mx-inspired look was normalized for me. I was never into racing, so the whole skinsuit/lycra thing was silly to me.
To each their own! I find it funnier to see someone in full matchy-match gear than someone who just threw on some clothes to go for a ride.