You should look at your product and view what potential risks are there.
I am assuming that your ambiant packaging location is already of a suitable hygienic design (clean, no dust, maintained).
Then it is the question what kind of risks am I mitigating with my cold storage? Is it for microbiological shelf life, sensorische or for more commercial reasons (e.g. consumers think the product is of higher quality when bought from the fridge than ambiant, thus it is kept cooled).
A hard gouda style cheese may not need to be kept in cold supply chain for its shelf life. A soft cheese is more likely to need a cold supply chain.
If the product should be kept cooled than you could see which risks ambiant storage could bring.
How long is the product in the ambiant room and what will the effect on product temperature be? If packging only takes a short time you could look to validate the proces (take into account hot summer weather!). and decide which procedures you want to put into place. So if packaging also takes 30 minutes from taking it out of cold storage, packaging and putting it back in cold storage. It will likely not impact your core temperature that much.
Keep in mind if you are willing to take the risk of having the product heat up when procedure is not followed.