I'm deeply concerned that the supplier hasn't enacted a public recall on this if it's as you say. The comments on "swarf" specifically 316 stainless make it seem it wasn't just a missed metal check on their part but they've had known contamination.
I agree with Tony the priority is consumer safety and we all know metal detectors have a failure rate, that's why most of us will have some kind of "I've found x pieces, now I stop and investigate...". Depending on the shape of the pieces even ones larger than the ball bearing test piece size you use could get through.
I would be pushing the responsibility and carefully wording everything to the supplier who have not instigated a recall themselves. I'd want a clear answer on why and clearly communicate that you are considering doing so based on what they've shared if they're at all slow or reluctant and WILL be making a claim. Get your legal people on that this morning if that's not already done.
Tony is right in that we are food safety experts and we need to consider the food safety of the product first. It's easy from the outside looking in though to assume you'd know how you'd act but I think this is looking like a recall. There will come a point in this crisis where you either don't know enough from the supplier and you have to make a call despite your attempts, or they finally de-escalate (I feel like this is the least likely option right now) OR, the most likely thing I think you'll see is other customers jump first as you won't be the only one who has used this. If this has gone into an M&S supplier and its been used, then it's quite likely that site knew before the supplier did as they may have picked up the swarf in x-ray and already had to talk to M&S. If it's another UK retailer, they'll be considering pulling it. On that last point, DO remember if you're making retailer branded, this isn't your decision, it's the brand owner and they should be informed.
Your last source of help is one I'd definitely recommend. When you contact your insurers if you haven't already, some of them in the UK work with companies who help assess risk as a fresh pair of eyes / ears on an issue. They won't tell you if you should recall but you can talk through, in confidence, everything that the supplier has gone through with you so they can help you think if anything you might have missed.
But certainly I would not be adding to the problem right now. I'd not be sending any more product into the marketplace containing that ingredient and if you don't get clear answers which make it seem like less of a risk, I'd be inclined to recall. But one thing you need to avoid is the scourge of the rolling recall...
Clear answers I'd ask the supplier right now. Your legal team may have other ideas if you can ask these!
How was the incident found?
What is the source of the metal contamination?
What gives you confidence in the start and end of the contamination window?
How have the lines been decontaminated after the incident if you didn't find it at the time?
What is the most likely date / time for contamination?
What piece sizes have you found? Smallest and largest?
What gives you confidence around other batches you've supplied?
Why was this not detected before we received the sugar?
Have other customers used the product?
Have any customers (manufacturers) found metal on their site due to this?
Have any consumers been impacted?
Are you aware of any other customers of yours considering recall?
I'm sure others can think of other questions to add?