The essential issues here are cross contamination risks and 'exposure assessment'. From a very quick search and review of available data, both in my opinon would give risk to concern.
Firstly, there are documented outbreaks associated with unpasturised products for E.Coli0157 (UK & Canada) and Salmonella (France) so this might require you to review your target pathogen list. The infectious dose for E. Coli O157 is very low and this should further drive a review of your risk assessment. In terms of Listeria, public health advice to pregnant women is avoidance of unpateurized cheeses which will reduce the risk but this I believe is not acceptable in terms of you assessment of risk.
Washing of knifes and cutting boards will undoubtedly reduce the risks of cross contamination but this means you are depending on this activity to control risks. It will be very difficult for you to establish this as a
CCP. The swabbing data you mention should be considered in light of the fact that you need to know if the unpasteurised product contains pathogens and at what levels before you can use it as validation. This is difficult to do without proper challange testing.
On the other hand we know that significant amounts of unpasteurised cheeses are consumed in countries like France with a low level of associated deaths. However where contamination of a batch of milk for example gets into the supply chain the impacts can be significant. We also know that E Coli O157 outbreaks frequently occur due to cross contamination in the preparation and handing of Ready To Eat products from raw products (meats) and the infectious dose is low.
I think you have a lot to consider in your situation and if it was me I would conduct a very indepth risk assessment using whatever available data you can get your hands on. Personnally I would be seeking to have in place an operational regime which includes separation based on time and an approprate cleaning change over procedure. Just my feeling based on my limited insight into your operation.
George