For budworms, you should spray your plants every 3 weeks with BT, a beneficial bacteria, throughout their veg state, and the first half of flowering. That will keep the worms from growing on your plants. And that way, you don't have to spray your ripening buds.
BT will affect the taste of your bud if used too late in the season. It's just fine to wash your plants before harvest. Even my indoor plants are sometimes taken outside on harvest day and sprayed down well with a fine spray, and then dried for a couple of hours and harvested.
HTH
