The method by which the timers work may have a bearing on it - if these are ye olde clockwork-driven things and are responsible for timing processes that are critical to safety or quality then it may be necessary, in the same way that it is necessary to service mechanical wrist watches. Equally it might be possible to show that the variation is small enough to not have any consequence for your process, although I'm not sure I'd fancy trying to sell that to an auditor.
If they're driven by a quartz clock (which seems more likely?) then potentially there is no real need, as IIRC those are expected to only drift by a few parts per million as the frequency with which a quartz crystal oscillates under voltage is fixed, with only minor variation due to temperature. Quartz has a fairly low coefficient of thermal expansion, hence the relatively tiny variation expected.
The same arguably applies to the stopwatches though - are they calibrated, and if not, how do you know they're accurate?
Do you know the background to the checks? I'm curious!