So my 2 plants have been attacked by caterpillars and I picked off about 20 over the last few days and today I haven't seen any.

Is it a good idea to use the BT now?

I read that it may be too late and that it is better to use before flowering. I am also concerned about the what the BT will do to the taste of the buds. Please if anyone has used this before let me know how it worked for you and if it affected the taste of your buds. I bought ladybugs yesterday and they have set up a few base camps under some of the larger buds. Hopefully they keep em away!!

This is my first real Outdoor growing and any input is appreciated.

You have to get some BT on those right away especial with 20 cats. That is alot for 2 plants. Depending on how much reading you have done already the cats are coming from flying moth/butterfly.
what color are they? you should post a photo. even if you are into the last weeks before harvest i would spray those babies down.