I have been on both sides of the co-manufacturer relationship, and it has always been the responsibility of the co-manufacturer to have their own records for traceability. I have not seen the brand owner have full records of that information.
Typically, I have seen that the co-manufacturer is approved by the brand owner through processes similar to supplier approval - i.e. ensure they have the required quality and food safety systems, which includes traceability at least one step forward and one step back. The brand owner should ensure that the co-man conducts traceability tests to meet their desired frequency (at least once a year). That can be through a mock recall / trace exercise or during a third-party audit.
Even if you are purchasing the materials for the co-manufacturer, it should be their responsibility to track when they use them and store the records. If you find out from a supplier that a recall is needed, you should only have to pass that information along to the co-man and they can tell you what finished goods were produced and where they went.