To put my reply in perspective, poison is always in the dosage. At the residual levels that are typically found in oil, neither is within the levels of health concern. You would get more in your system wiping a skin patch with Isopropyl, than from the oil. It is absorbed through the skin as well.
If I were going to drink it, I would choose Ethanol because Isopropyl causes auditory nerve damage and more toxic. Even though Ethanol is food grade, it is also toxic and does significant damage to any number of body systems when consumed in excess.
There is also the hysteria factor, by which it doesn't matter what the facts are, it still scares the shit out of anyone who hasn't studied the nuances enough to realize what levels are toxic. Most of them would be horrified to read the MSDS data on Ethanol, because of how well it has been studied and all the bad things we know that it has done to people. Take a look at what it does to fetuses for instance.
As I've noted tongue in cheek before, a 75% oxygen atmosphere will kill a human being. Too much of a good thing is just enough that it becomes toxic. Some things just take less be toxic than others, but still have their uses. Consider Belladona for instance, a deadly poison used as a heart medicine.