Old sperm killing new sperm?

This is probably a stupid question, but it's been bothering me. I've heard that when a man doesn't for awhile, the old sperm builds up and can actually affect or kill the new, fresh sperm. My DH and I BD'd once during my fertile window after no for almost three weeks. If any sperm were to actually meet up with egg, would they be the healthy, best sperm? The old or affected ones would have died or not been able to make the journey, right? For some reason I'm paranoid that a defect sperm would make it and the baby would be born with some sort of syndrome or something. Please help to ease my mind! TIA!