In some very low risk facilities where cross contact from clothing is difficult, e.g. beverages, it's still the norm for people to wear clothing into rest rooms etc. I don't like it and don't agree with it because even in beverage processing there is still a manufacturing element somewhere.
If you were going to get rid of the nut ban, which I actually strangely enough DO agree with and I'll explain later, then I think you need to sort out your workwear issue. Unless there is ZERO chance of any of your workwear contaminating the product flow or packaging a consumer touches, why the hell are you allowing staff to wear workwear in a canteen? Why is the control to put clothing on not take it off? That's bonkers. It's just so much harder to control someone putting on a coat and making sure it's sufficiently covering workwear than the other way round. Then there's the fact they'll be wearing their hygienic workwear in the toilet as well.
But if you have controls like 90% of food factories in the world where coats are removed BEFORE breaks, then I'd think differently. Nut bans came into lots of factories at a time when allergen controls were pretty weak in most sites and if we're honest, allergen training was too. There is no more risk to an allergenic consumer of a member of staff eating nuts on break than a consumer allergic to sesame when they're eating a sesame seed covered bap. If anything, the risk on the second case is worse as seeds falling off is highly likely. So you could then validate it by doing allergen swabs on someone's hands and workwear after they ate something nut containing for lunch if you wanted to be really rigorous. But to be honest, I've worked in factories audited by GFSI which both do and don't have nut bans and both have been fine.
My ONE caveat to that though is that nuts are loose and easy to create an alarmist contamination incident with. One site I worked in did have a nut ban and the day of BRC we found peanuts not just in one place in the car park but everywhere. We got it cleaned up before the auditor (it was in the announced days) but as a result we did do some bag checks. The fact nuts were banned meant that had we found some (we didn't) we could have disciplined just for possession. So that would have been kind of helpful.