You need to define metrics that you want to grade your suppliers by.
Many manufacturers have surveys they have their suppliers fill out with questions like:
Do you have a HACCP plan
How many CCPs are in your HACCP plan
Do you have an X Program
Do you have a Y program
How often do you run a traceability/recall exercise?
When was the last recall exercise performed?
Are you certified by a GFSI standard?
If so which one?
Are you audited by any other 3rd party auditor
List auditor companies:_____________
Things like that. Also you want to access how well they supply you with documentation that you request like specs, country of origin statement, etc.
If they perform poorly you can have a few different grades like:
Excellent: Company filled out survey and supplied all appropriate documentation
Good: Company filled out survey, most documentation, easy to contact
Probation: Company didn't fill out survey or a portion of documentation was not received or very difficult to contact.
Fail: Company refused or failed to give appropriate documentation.
Once you have a grade for the company you can use the information they gave you to do a risk assessment on the ingredient.
If they have a HACCP program, cleaning program, are GFSI certified and/or 3rd party audited and received good scores your risk should be lower. If it's a liquid product you may want to send out some for testing for appropriate biological contamination or ask the supplier for test results.
Does any or all of that make sense?