Sounds horrifying. There's probably a sea of larvae as well that you're not noticing because they're tiny and green. If you have a magnifying glass and want to be grossed out, try taking a look.
If you can use diatomaceous earth to dust the leaves and stems around the affected areas, I'm pretty confident in its ability to kill them off. I wouldn't apply it to buds. Not that I intended to smoke, in any case.
Good luck, friend. It sounds like you have your work cut out.
