Your statement seems inconsistent. They can't just walk up to you and pat you down for no reason. First, they have to be able to articulate a "reasonable suspicion". If they don't have reasonable suspicion, the pat down is illegal. Anything found on you after said illegal pat down is "fruits of the forbidden tree" and can't be used against you.

Reasonable suspicion is a lower standard than probable cause, but it is a standard nonetheless.