Emily, the risk assessment should be established based on the product, not the type of company you are buying from.
Take for example I am purchasing fresh raspberries. The food safety risks identified are pesticide residue, unapproved pesticides and from certain countries cyclospora. Additionally I have a specification that identifies quality and regulatory attributes.
The option that I have identified for the control of pesticide residue - certification of the on-farm food safety program via a recognized certification program (one that covers the appropriate use of pesticides on farm), with additional targeted testing at the time of receipt.
If I am purchasing direct from a grower, I would require proof of certification.
If I am purchasing from a packing shed, I would have approved the packing shed if their supplier approval program supports mine ( i.e. they need to verify each of their suppliers certification programs).
If I am purchasing from a broker, I would have approved the broker if their supplier approval program supports mine (i.e. they need to verify each of their suppliers' certification programs).
Remember, when you deal with a broker, you are setting the standards and specifications for what you will buy (go back to the meat example, you would specify the grade as: Select, Choice, Grade A etc. and that is what the broker will source, if you specify "tenderloin" they won't ship you ribs...). You have the same ability, and responsibility to tell a broker of raspberries you have a specification that includes the verification of on-farm food safety programs (i.e. GFSI certified or other audits you recognize).
If you decide that you need to purchase outside of approvals - for example the only raspberries available are from farms without verification of their pesticide control programs, then the risk needs to be mitigated at the next stage - that's between your receiving and use of the ingredient. The other option you may have included to control the risk is "test prior to use"; in which case you buy the product, test it, and if your data shows pesticide residue is not an issue, then you have mitigated the risk.