Greetings Shakti,
Yes, you should take it into consideration but in the end it will be scored low in the risk assessment, because it is only a really small possibility for the ingredient to contain allergens ("May contain traces...") and also even if it has traces the end quantity will be negligible as the ingredient is incorporated in your final product.
You can however, if you wish, to back it up with a detection analysis for these traces once in a while or have your supplier provide you with one if he does it for his product. A step further would be to ask your supplier if he performs some fast allergen test after producing something with an allergen and before producing your allergen free product, so as to verify the thorough cleaning of the production line in between (if he does it in separate lines even better) and you can include that info in your R.A..
Regards!