Well, yes. From a personal point of view the impact is irrelevant. But then it all adds up, and to indulge ourselves into the fallacies of recycling and environmental impact, without actually knowing about the subject, is precisely what makes us fall into those fallacies, and what got us where we are as a climate-changing species. Carbon, in the long run, can't outlast aluminum simply because it's really not recyclable. It doesn't matter if your rims last from now until you can't ride anymore, it will stop there (at a higher environmental impact per unit), while Al can become something else. It's actually the opposite, it will outlast aluminum in a useless form (but you understood me).
Can't we just talk about expensive plastic bikes without injecting our poor understanding of energy consumption issues?