if you smoke cigs, then the answer is yes.
you should only touch your plants when checking them for growth problems.
other then that you shouldnt touch them for no reason.
PS- washing your hands before touching them, decreases the chance of you passing on germs to them