"Bugs" seems a bit generalized, no? I would look for the bug (insect or mite) and signs of the pest itself like webbing or spots of defacation or eggs, etc. A magnifying glass would help.
It is a long way from losing your crop and it may be signs of a past population now gone or in early stages of becoming established. I think those young buds have weeks to go and are still forming.
If it is indeed an established number of insects or mites you can still perhaps isolate them or save the others from infestation. It seems a bit late for soap or NEEM as a treatment since they are flowering now.......... but if you only treat the plants with signs and symptoms, then you can still spray them.
But if it was me- I would first identify the pest to aphid, spider mite, thrips or whatever it may be if signs are still there. I also hate "bugs" and have been no stranger to dealing with a number of different types over the years in a number of different manners (materials and methods) .
I would not panic in that situation since the symptoms (signs) of bugs are still not confirmed. They may be gone now. It depends on knowing which type of bug, no? Then go from there........... It is late in the game but you can still save your crop and you may not need to do anything depending on what you find with a bit of looking under a small amount of magnification (5x or 10x)...............
jus sayin'- best regards and I am new here so consider the source.
you have not lost your crop by a long shot at this stage of the game