I've never heard this before, but I have heard of people with allergies to the plant. They can smoke it and handle dry buds, but the living plant material makes them break out, some worse than others. Maybe you have some sort of allergy like this. Do you notice any symptoms around other plants?