I don't see it as serious enough that we need to go to, or move them to a different room to avoid second hand smoke, but I had a friend that had a cat, that used to actually come to him when he heard the baggie, and went to the 'smoking room'.
The cat knew that my friend would take a hit, then cat, sitting next to him on the back of the couch, would nose up to him to get a nosefull of smoke.
I asked how he trained the cat to do this, and he said he started doing it to mellow the cat out when he would start walking on the counter and playing in the sink and moving dishes. Friend didn't get my point when I asked him did the cat only start playing with dishes after not being fed for a while.
I didn't see the humor in it as he did, and I guess the cat still went hungry but the weed just occupied his mind elsewhere for a while.