I would be swabbing the food contact areas post cleaning and disinfection for TVC as an indicator set of organisms, as others say, to show areas of poor hygiene.
Most plants use the retailer codes of practice to decide what to test. In your case, I would argue as hygiene is extremely important that you go for TVCs as you could have presence of bacterial loading which aren't enteros. With some products where loading is huge (e.g. cheese as it contains so many starter cultures deliberately) people look for enteros more but TVC is also not a bad thing to monitor.
When you designed your cleaning processes (if you were in role then) I'd have expected them to be designed using an indicator organism. I.e. can your cleaning practices reduce bacterial loading to a safe level? If you don't have records of this, it's a good idea to do them. It also helps if you want to change chemicals or a process.
I would periodically test door handles and tables as well. As you say, hand test points are also important as they could transfer to food.
I think as you've alluded to, testing only for Listeria spp is going to be hit and miss. Most of the time it won't be there but not necessarily because you've cleaned well. It's a good idea to test for it of course as better to find in the environment than actually in your product but still, it's quite a lagging indicator of the hygiene of your site. It would be important as well to also have a more general verification of hygiene both through visual inspection and trending of swabs.