to be safe i'd take a couple $5 bills and place them at the bottom of the towers ladder poking out from under a rock... if somebody see's money then they will take it and you will know if somebody has been there and seen your crop or not...
I've thought about this as well... I was thinking plants must drown outdoors!! I like your idea of digging deeper and using gravel.... but if you were gonna do a big grow then that would be alot of work carrying alot of gravel into your spot!!