However your mention of a restriction "not having more than one audit in a week" is off. Whether it is a regular certification audit or a customer audit, I wouldn't allow anyone to audit more than once a year unless there is a specific food safety issue that is causing severe issues for the customer in question. If this were to happen more than once a week, I'd have bigger fish to fry than audit compliance.
Agree with everything SHQ is saying, but to clarify this bold part: I think OP is referring to telling your customers that you don't want to schedule yourself more than one audit a week with whomever. Essentially reserving the right to tell a customer, "Sorry, we've already scheduled an audit with another customer or regulator that week, so we will not permit you to also come audit."
For the OP, I would just simply take what criteria you want to enforce, throw it on company letterhead in a notice type format, and most importantly get company leadership to sign off on it. If you want to reserve on-site audits for only your most important customers, add an annual minimum spend requirement (either historical spend or contracted forecast is a smart way to include this, in case a YUGE new customer wants to come see how the sausages are made).
Making them send an agenda/copy of the standard they want to use is smart as well. Back at my old spice plant, the worst customer audit I received was someone coming in from an organic fruit sauce pouching company, declare they were auditing us against the Campbell's standard requirements, and then get mad the company we were co-processing for wasn't making us treat the spices as if they were intended for baby formula. Suffice to say, having the standard ahead of time helps avoid confusion on what they're expecting.
Lastly, once that same spice plant had gotten their first GFSI cert, the customers kept trying to schedule audits. So ownership would send a memo to them detailing that we were going to charge $2,000 per customer audit, limited to 1 day, with no guarantees we would be actively running "their" product. Couple of customers still went for it, but suddenly the rest all agreed that our GFSI audit and cert would suffice. Make them pay for wasting your time and the smaller customers will probably back off.