If hes in a pot and you can move him out of the way, and in a place wher you can chedck him all the time and just cut a sack-filled branch off him when hes ready to spurt and trash the rest, well you might be fine.
If not then, well pollen can travel 100's of miles in the wind
peace.