Why is nondisjunction not a problem during mitosis in adult humans?

If nondisjunction occurs in an adult, you just have two cells that have the incorrect amount of genetic information. Just one cell doing something wrong is generally not a problem, that cell will just die and you won't even notice. But if it occurs in a sperm or egg cell, than EVERY cell in the resulting organism will have the incorrect amount of genetic material and you see big malformations like Downs.