In my previous facility inspectors wore gloves when handling surfaces, and used high-power flashlights at an angle to the surfaces to look for residues as well as wiping them with sanitized, gloved, hands. Residues were obvious on stainless steel and other surfaces using this method.
There may be other resources the inspector calls upon, but the FSIS directive concerning the pre-op inspeciton task states:
IPP are to assess the cleanliness of areas or equipment that, if insanitary, would present the greatest risk of transferring pathogens or other contaminants to product (e.g., direct food contact surfaces that are difficult to clean or that may serve as harborage sites). When assessing pre-op sanitation conditions, IPP are to use professional judgment in determining whether the establishment‟s pre-op measures have resulted in a clean and sanitary environment.
No instructions not to use gloves, but it does say that they're to use their best judgement in how they determine the equipment is good to go.
HOWEVER, if you have a consistent, written policy that product contact equipment is touched only with gloved hands, your IPP has to comply with your rules per Directive 5060.1 which is current as of September, 2016.
This directive instructs inspection program personnel (IPP) to comply fully with the sanitary and hygiene procedures and biosecurity measures put in place by an official meat or poultry establishment, an egg products plant, or an official import inspection establishment that these facilities also require of all of their employees. FSIS previously issued these instructions in FSIS Notice 17-15, FSIS Program Personnel Hygiene and Biosecurity Practices.
*Politely* share that directive with him and make him wear gloves as long as you require it of your other employees, after all, it's your product and reputation, not his. If he still disagrees that you're preventing him from doing his job, it's up to you if it's worth reaching out to your regional office for a final ruling.
Given his statement in your post, I imagine this conversation in general won't buy you any goodwill.