Assuming here, BRCGS food 8 audit took place. If you turn to the Standard itself, in the back section Part III Audit Protocol, section 2.2 the on site audit, auditors are expected to spend approximately 50% percent of their time "on the floor". Note BRCGS uses a different number, and definition of "on the floor" that some other programs, notabley SQF, so they can't easily be compared. The standard requirements themselves, are color coded to give guidance on which clauses are expected to be assessed "on the floor".
There is the usual debate about reviewing records and documents, but generally reviewing procedures and historical documents in a production area office, is not generally considered "on the floor". Interviewing staff, supervisors and managers in their production area offices generally is, and reviewing active records being produced on the floor generally is as well.
That time split is obviously not always possible, and there is opportunity for the auditor to explain and justify variations.
In the end, audits are a service. They are bound by a number of oversight systems, from accreditation bodies, certification bodies and program owners, most things are not negotiable with the auditor. if you feel your auditor did not follow protocol, you should be bringing this to the attention of the service provider (the certification body) to get the right explanation. You have a copy of the standard you are being certified to, within it, or the CPO website, you can usually find the actual requirements for pretty much everything. If you don;t get satisfaction that way, all CPO's have direct contact to elevate your concern to.
Taking action to protect the industry is the right thing to do, where you feel anyone is not in compliance.