You might be able to argue that a traceability exercise is an audit. It will cover many areas of a full site visit and does give you a good view of their competence. I forgot to mention that we also ask for a range of supporting information in our SAQ such full HACCP plan including process flow, copies of other audits, quality policy, EH&S and authenticity.
It's true that there's no substitute for a site visit as it gives you a feel for the site and the way it works that an SAQ cannot. However, once you've assessed the risk, you should put a rolling 3-4 year audit plan in place. You don't have to do them all in 1 year, but 4 years is probably the maximum time you can leave. If you have a tight audit document, you should be able to complete each in 1-2 days, so visiting 3 in a week if they're close should be possible. I know it's 12 weeks, but it will also help you build a relationship and it might improve the commercial discussions as you're demonstrating your commitment to them.