I've read that line about using your hand to judge proper light height all over the web and in lots of books and magazines. It's probably not bad advice but it's never sounded quite right to me. After all, my hand can take a heck of a lot more heat than I'd ever want my plants to feel.
Getting the right height is important because light intensity drops off so dramatically with distance. A 400w HID light one foot away that gives 40,000 lumens will only provide 10,000 at two feet and 4,500 at three.
I'm no expert, but this is what I've been able to come up with after scouring every available source I can find and based on my own experience.
First the bad news: there is no one right answer because there are so many different situations. Are you using flourescents, LED, or HID lights? Is your light air-cooled, water-cooled or just a bare bulb? Are you growing seedlings, teens or mature blooming plants? Do you have enough fans or AC?
In most gardens too much light usually isn't the problem. The problem is heat and the amount of light you can have depends on your ability to remove it. If lights didn't produce heat you could probably let your plants touch the bulb without problem. But because light equals heat, proper ventilation and finding the right distance are important.
Things to keep in mind:
The younger a plant is the less heat it can tolerate.
Plants used to lower intensity lights (florescents) will need a little time to harden themselves to more intense HID lights.
Water-cooling is the most efficient way to remove heat but it's complicated and potentially dangerous.
Air-cooling works well but be aware that a reflector hood with 8" ducting will be cooler than the same light with 6" ducting.
Good air movement, air conditioning, CO2, and/or a light mover let you keep lights closer to plants.
Now that I've made it seem complicated let's make it as simple as possible:
Florescent lights should be no closer than 6" from the tops of your plants.
400w air-cooled lights should be no closer than 12"
1000w air-cooled lights should be no closer than 18"
I would use this as a starting point and then adjust the height according to your personal situation. For example, if I've got seedlings under a 400w light 12" is way too close. Young tender plants would wilt or scorch under a relatively powerful light like that so I'd back it off quite a bit, maybe hang it 4 feet above the tops. If it's a 400w bare bulb in a poorly ventilated room even that might be too close. If you veg under florescents and then put your plants under 1000w to bloom you should keep the lights 3-4' above the tops and then slowly lower them over several days to avoid shock. If you have mature blooming plants under 600w HPS lights with 8" ducting and a powerful inline fan then maybe you can keep that light just 12" from the tops, but if your ducting is only 6" or 4" then 15-18" away is probably a better bet.
Ultimately it's the temperature those top leaves get that's important. I like to use an infrared thermometer to see exactly how hot they are but a thermometer at plant level will also work. I like the tops to be no more than 85 degrees with CO2 and no more than 80 without. If your leaves start cupping or curling at the edges then that's also sign you've got too much heat.
