Worms in an all 100% organic grow would be fine and wouldn't hurt anything. Although not sure they would do much that some perlite wouldn't do better. Throw a banana peel or piece of apple on top and worms would eat it to survive
If you use chems though it would kill the worms