I saw an elderly lady go up in loafers and a cardigan passing people. There is no danger at all other than the fact its so easy and popular you get a bunch of out of shape idiots going up the same trail as a bunch of fitness douches.
I didn't wear flip flops, but I also didn't bother lacing up my skate shoes for the slow pace set by my outdoor enthusiast friends. Not exciting.