I just noticed you are in Canada after typing all this. Posting in case it helps you or someone in the future.
According to the FDA, lactose is only an allergen if protein is included. Would depend on the process used to create the lactose.
It is still a milk derivative regardless of the allergen status.
Here is a link to an FDA Q&A that asks this exact question. (Page 18, D.8)
https://www.fda.gov/...163454/download
D.8 Lactose is a milk sugar and ghee is a milk-derived fat. As a manufacturer, do I have to declare milk on the label if I use these ingredients?
- The FD&C Act requires labeling of a food or ingredient that is or contains protein from a major food allergen (section 403(w) and 201(qq)). Ingredients derived from a major food allergen that do not contain proteins are not subject to FDA’s allergen labeling requirements and would not be subject to the labeling requirements described in section 403(w)(1) of the FD&C Act. If your major food allergen-derived ingredient is processed using technology that reliably produces a protein-free ingredient and you can ensure that the ingredient does not contain protein, then you would not have to declare the major food allergen on the label. While lactose is a milk sugar and ghee is a milk-derived fat, we understand that there is commonly residual protein from milk in these ingredients. When that is the case, lactose and ghee must be labeled in accordance with section 403(w)(1) of the FD&C Act. However, the manufacturer may consider the Food Allergen Labeling Exemptions Petition and Notification process if the firm believes that their ingredient qualifies for an exemption from section 403(w) of the FD&C Act