Interesting question. Can't comment on the regulatory side, but BRC has some specific requirements on control of labelling and you'll therefore want to look at if/how you can ensure you're meeting the requirements of e.g. clause 6.2.3.: Procedures shall be in place to ensure that all products are packed into the correct packaging and correctly labelled.
The question that I suspect an auditor would ask, is: If you know the product is destined for retail, how do you know that it is actually being labelled correctly?
Equally one might be able to construct a counter-argument based on BRC's own definition of Primary Packaging to an extent - since this is "the packaging that constitutes the unit of sale to the consumer or customer", and the label isn't part of that if it isn't applied, is it outside the scope of the points in section 6.2? Not perfect though, as it's included in the outer pack so one could also argue it the other way...
To me it certainly doesn't look clear-cut as a scenario. Is there a reason you're not applying the retail labels yourself, if you know that's where the product is destined?