Dear Milos Vasic,
Thks for reply. Yr process is now a little clearer to me. 
I agree with yr comments there are analogies to other systems. In fact variations of yr question, notably for metal detectors, occur in many other threads on this forum.
AFAIK, IMO, there are 3 initial, basic, questions associated with yr post, preferably interpreted within the typical chain of implementing a haccp system. The queries / answers are inevitably debatable since many of the haccp decision procedures involve multiple factors (and opinions) together with fundamental probability logic. Such is the nature of risk assessment.
(1) – Can the process step referred be justified to be pre-defined as a Prerequisite function ? If so, it will, by definition, not be a CCP.
(2) – If not pre-defined as a PRP, is there a significant hazard associated with the process step for which the colour sorting activity is acting as a control measure ? The decision is based on a (validatable) likelihood/severity risk assessment. If the answer is No then the step is not a CCP. Some experts may, now, consider it classifiable as a PRP.
If Yes, it may be a CCP (particularly in the Codex interpretation of a CCP, eg as per the Codex tree). Some experts may, now, consider it classifiable as a PRP. 
(3) –If a Yes answer as per (2), and you are implementing ISO/FSSC-22000, it could also be an OPRP.
Personally, for “traditional” haccp, this generic analysis (Kraft) seems quite impressive IMO –
haccp extraneous material.pdf 61.85KB
253 downloads
Rgds / Charles.C
PS - I would add one slight addition to the attachment, where "likelihood of occurrence" is mentioned, AFAIK, it is typically referred to the situation at the point of consumption (assuming a finished product is discussed).
PPS, Qu2, Qu4 of the tree in attachment, are very slightly modified from the Codex tree. Qu2 in the latter might be (slightly) more definitive for yr current situation.