If the leaves are damaged, dead or nearly dead removing them is fine IMO. My school of thought is that if they are healthy leave them if they are not remove them since they will not get better.
If the damage is large them perhaps lack of fan leaves could create issues, however, if you stil...