your fine, chances are you will get frost before it is fully ripe, but they can survive a few light frosts, its when you get two really cold nights in a row, like 27 degrees so that the leaves just shrivel up and the bark goes all slimey...usually the buds are for the most part untouched unless you leave them out there too long, they will mold up quick after that.
immature bud is all they seem to sell in most of minnesota anyway, better than no bud right?