Take off brown leaves at bottom?

Plant is frosty and flowering at week 4, but lots of fan leaves at the bottom are yellow and at the very bottom are brown and curling. I touched the brown leaves, and they instantly fell off. Should I keep doing this and removing the yellow ones?
 
Top