Please help identify, 1/4 - 1/8" green caterpillar looking things

I have been finding these little fuckers on 1 of my plants over the past week. Plant is in week 3-4 of flowering

I have been using Neem oil and removing them with tweezers but wanted to know if they are coming from the inside of the buds (I find them on the buds,not really inside) or if they were just laid there by moths.
Also, what else can I do? are they going to ruin my plant regardless if I can kill them all by the tweezing & smashing method?

Thanks for your input.

Im spraying mine with "safer Caterpillar Killer" Has the bt in it and once the caterpillars ingest it they die within 5 days. I just started spraying haven't really seen any progress yet, but that's what a lot of people swear by

i started using safer caterpillar killer this year also.what i learned is spray every other day because i have some still and i spray alot.the caterpillars live inside the bud so you wont see them often if you have a couple you have allot.they eat the bud from inside out,so go get some killer or they will f your shit up.