Hi, I used to be in that boat. We supplied people around the world. Firstly I always tried to push back where I could and was sensible to do so. It didn't always work but one customer did accept going to once a year audits rather than 6 monthly.
Secondly I trained up my team. So it wasn't always me leading the audit. By the end I had four people including me who could competently lead an audit. Good job as well as once three turned up at once(!) We could have turned two away but honestly it was actually easier just to get them all done. Funny though when they all asked for the same training record at once and I ended up ferrying it from room to room lol!
It's worth mapping out your workload as well. You may genuinely need a person whose main job will be compliance with customer standards. In that I'd say ideally you need to pick 2 or 3 of your customers who perhaps have the most stringent standards (or most stringent auditors) and audit against them in key areas internally alongside whatever GFSI you do. That will make audit time a piece of cake but it is time consuming. That's why I put in an additional layer of management in that old role so I had someone whose whole job was pretty much managing that internal audit and external audit process.
But lastly, I actually think you're lucky. I now work in a business where each site only has two audits a year and they're not from customers (it's branded) and I can tell you now, standards are worse. When your customer uses food safety as a tool for negotiation, it does focus the minds!