What are you minimum requirements for a supplier? What do you want from a supplier? What do your customers want from you? Use what both you and your customer needs are to determine the supplier risk. Do they meet your standards to be accepted as a supplier. If a new supplier then their risk may be high as they are an unknown entity in your business. Over a period of time you can re-assess the supplier based on how they perform. Existing suppliers you already have data for that can support your risk rating and justification for the rating they have.
Focus on quality performance of the supplier and the materials they supply. As Robin noted you should also consider DIFOT, threats to the supply chain, and other supply chain performance metrics. If you only have one supplier capable of supplying you with a material and you can't source it elsewhere what happens if their factory burns down, do they have contingency plans?
We have a comprehensive questionnaire we send out to suppliers before we consider using them (along with material questionnaires). In a food factory we require as a bare minimum of our suppliers a third party certified HACCP system. Our preference is for a GFSI aligned certification but we understand that's not always practicable.
We typically perform a desktop audit of the supplier - based on the questionnaire and the responses as well as supporting documentation. Based on this review we then give the supplier a rating, High, Medium, Low. We would also decide at this time do we need to audit the suppliers sites be it a distribution warehouse or manufacturing plant ? This determination is based on the review of data, documentation, forms, questionnaires, certificates etc. the risk and what material they are supplying to us. - If it's a material/ingredient we use without further heat steps then we would definitely look at their site to assure us we are at low risk. But in the end no matter how many quality certifications a supplier has, or how well they perform during your audit things can and do wrong. You should also risk assess the suppliers materials and combined with the supplier risk this will help drive how you want to manger the suppliers materials - clear on COA, skip lost testing, spot testing.etc