There is some evidence in FDA guidance that would indicate that the FDA could see your situation as a reason to deny inspection. I've found some guidance for the drug/devices industry that mentions this scenario. During unannounced visits FDA inspectors do need to be ready for unforeseen circumstances like employee absences, schedule plant shutdowns, or big facility maintenance projects.
See lines 246 - 250 in my attached document:
DENIAL OF INSPECTION
Examples of potentially reasonable explanations that might result in the drugs or devices not being deemed adulterated under section 501(j) of the FD&C Act include, but are not limited to:
At the beginning of an unannounced inspection, appropriate personnel are not immediately available to accurately answer the FDA investigator’s questions.
I would suggest training all senior staff on the process of speaking with FDA auditors about this circumstance. Something like "Our quality manager is away at this time, and will return at X date. While we do have some insight into the quality program at our facility and participate in it, the quality manager is by far the best suited in our company to assist you during an inspection. For a timely inspection process we'd like to request you return at a later date when our QA manager is present."
This is just my hunch from reading some FDA guidances though, I would love to hear if anyone has any real experience with this scenario.