throw the baby out with the bath (bathwater)
To reject all of something because part is faulty.
God knows that there are weaknesses in the program, but if they act too hastily they may cause the baby to be thrown out with the bathwater.
Categories:
Source: A Dictionary of American Idioms