Following your pre-requisites and having personnel practices that address the use of facilities and appropriate hand washing afterwards should be sufficient. I wasn't with my current employer when they implemented SQF, but I was informed they were required to have a separated pass through into the restroom for facilities that entered in to the plant. Meaning we had two separate doors, before entering into the bathroom. I can send you pictures if you are interested. We don't do any swab validation / verification, the only verification we do is perform a monthly plant walk through, daily 5s audits, and internal audit of the facility. This has been sufficient so far.
In regards to FSMA, so far I am not aware of any swab verification / validation requirements. I know FSMA is not applicable to packaging in some regards, but be mindful of the Foreign Supplier Verification Program. From seminars I've attended, if you buy raw materials from a foreign supplier, you will be required to have a food safety plan to help ensure your supplier is performing and supplying safe product. You may be requested to send documentation to an FDA as an audit. I believe the owner of the raw material when imported to the US is required to have the appropriate evidence of compliance. If you serve a particular food industry, you may want to look at ensuring your suppliers are providing FDA CFR letters for the materials and perform any microbiological testing of your raw materials.