18 replies to this topic

[StevoTheDevo]

We manufacture most orders in a Raw Material blending and packing batch process.
Many orders involve 10 or more batches.

The product is a low risk dry ingredient that is sold to customers for use as an ingredient in processed food of medium-high risk.

Past practice has been to micro test only the first batch manufactured and from those results, prepare a CoA for the entire run's production based on those results.
History has shown no concerning results from that machinery over a 10 year period of use at least once per week.

Clearly the current practice of testing only 1 batch is unacceptable to subsequently extrapolate the results to cover the entire order in the preparation of a Certificate of Analysis/Conformance.

I have proposed a change to test a composite sample from Start, Middle and End of production, but there is some consternation from my senior that this is still not sufficient and that we ought to be testing each batch individually and sending each batch out as a separate batch rather than as one lot.

What is the opinion of others here?

Is a Start, Middle and End testing regime (backed by history) sufficient to produce a CoA, or should it only be a CoC (since we are not testing every batch)?

Edited by StevoTheDevo, 19 January 2009 - 01:49 AM.

[Charles.C]

Dear Stevo,

I am not sure what the exact definition of a COA is ? If it is supposed to genuinely reflect the actual composition of the specific product involved, you are faced with a standard sampling question for every individually packed batch or a somewhat different one if they are mixed together.
I presume coc means certificate of conformity (?) but this would seem to lead to the same basic type question?

The error in any shortened procedure will depend on the variation between batches. More statistics (and more data) involved.

Any professional statisticians here ?

Rgds - Charles.C

Ps - just reread yr post and maybe misunderstood. Is current procedure to mix them all together?. If so, then you need to use standard sampling theory to estimate number of samples for yr target accuracy and then divide into input stream. But the viability will depend on how consistent yr input is. I once had similar (but more complex) problem when mixing rock mineral ores from various sources into one combined lot. Problem is that you may be forced to take worst case scenario (ie std.dev.) as representative estimate. if all the line is very similar maybe not much a problem. This is like sampling a conveyor belt, typically at uniform intervals in statistics theory.

added - of course, if there are big variations between batches, the likelihood of any deviations from a quoted average when ascribed to individual packs will increase (as yr resident experts maybe suspect)

added2 - additionally you will be faced with a different statistical problem depending on what the micro test includes, eg salmonella is not the same as APC

[StevoTheDevo]

Sorry,
CoA = Certificate of Analysis
CoC = Certificate of Conformance

To me the difference being Analysis indicates that testing has been performed and the results of testing are included on the Certificate.

Conformance indicates that testing may have occurred, or the manufacturer has controls in place that ensure the product conforms to the specification. (No result are required on a CoC).

I guess I'm asking 2 questions
1) Is it acceptable for me to merge all the batches of a consecutive run into one batch code?
2a) If so, what testing regime is best to report the results
2b) If not, across a run of 10 batches, can I composite samples for Micro and if so what is the limit, 2 tests of 5 merged batches, 3 tests of 3 merged batches?

It does come down to stats, but the last time I did any stats was in school, 15 years ago.

[StevoTheDevo]

Your first inmpression was correct Charles,
We blend Raw Materials, then pack them, then add more raw materials and blend again.
There is some transfer between batches in that the blender is not completely emptied or cleaned out between batches.
We do not mix each batch together. (Although I can see how you might have read it that way, I might re-word my original post)

[Charles.C]

Dear Stevo..,

Thks yr clarification. It is tricky to describe processes like this without flow charts. Yr definition of COA, COC makes sense but both seem to me to hv fundamentally the same practical requirements unless one possesses a sixth sense for micro. properties enabling an intuitive declaration of conformance.

I think a proper answer to yr question requires actual data so I only make some broad suggestions.

Conceptually, yr query involves the statistician’s “what is the requirement to define a batch or lot”. Sadly my stats are all self-taught but I think the standard reply is that ideally one lot should consist of product manufactured under the same conditions. I presume this is the concept behind lot coding systems also so as to enable proper segregation / damage limitation in the event of a subsequent problem. However the practical reality obviously includes all kinds of permutations, intentional and accidental. This is similar to the usual statistical recommendation to work with large size lots per code for economic sampling reasons but any rejection will then also be a big one (not the statistician’s problem of course).

Clearly if all yr separate batches had identical properties, they could be classified as one combined lot with no detectable difference between the stated average composition and any random samplings. The reality will depend on the data / variation. Unfortunately, the typical accuracy of micro. measurements as compared to, say, composition data, varies from poor to very poor.

So my response to No1 and 2a is you hv to assess the variation in the items you are considering and set a maximum tolerance on the average value, eg X+/- 1% (or less), at 95% confidence level. *Can be done by considering the collection of sublots as one lot (2-stage sampling). As well as the full treatments, there are some approximate formulas available (from the experts) which show how to consider a sampling of this lot as being equivalent to a single sampling (eg ASTM sampling standards). Makes the calculations easier.*

For 2b, my answer is similar for the numerical parameters. I guess at a minimum you will need X individual data points to achieve the required accuracy. But you get an added situation for zero tolerance pathogens. As an example the USFDA typically evaluates salmonella on incoming shipments of cartoned seafood product (ie many sublots) by calling one container “a lot” then taking an appropriate sample size (varies with total quantity) made up (sort of proportionally) from various carton codes within the lot so as to achieve a result with a known confidence level. The required sample sizes hv all been published online (FDA / BAM manual, sampling chapter) and in books and they are not small. The FDA logic is that they hv no idea of history so simply make an assumption of uniform distribution and proceed . From memory the lowest level requires something like 12-15 samples across the lot but these samples can be composited since only a detect/non-detect result is involved (added - and it has been shown that the accuracy of the micro result for the composite method can be approx. the same as using individual samples). However the size of individual actual samples, eg 50g each, usually limits the handleable number per one analysis since added diluent is also involved.

IMEX, yr questions are quite understandable but the precise answers to 1 and 2b are "maybe" but need some evaluation of actual parameters/data. For 2a I guess my answer is “a confidence based one”.

Hopefully there are some super-intuitive people here with easier “rules of thumb”. ?

Rgds / Charles.C

added - Can ignore the comment above between the "*"s if there is no subpacking within 1 batch, eg if you compare to a collection of cartons each containing a number of bags, this wud be primary/secondary units. if no bags then direct sampling is involved so the relevance depends on yr style of presentation.

[AS NUR]

Dear Stevo,

IMO.. First you have to statistic analysis to find is your batches different between batches.. the procedure are :

1. collect data analysis from every batches ( 1 -10), you can collect the data 3 times every batch to see repeatabililty.

2. Analysis variance of the data using ANOVA ( I Think excel have the method), ussualy using confidence level 95%

3. if the result is no difference between sample, you can say if you take sample at any batches there is no difference result analysis at confidence level 95%... so you can choose any batch as sample and analyzed to make COA or COC...

thats my opinion.. hope can help you

[Charles.C]

Dear AS NUR,

Yr suggestion looks useful where specific data is available but I don't quite see how you apply it to a situation like salmonella. Do you know of any examples anywhere ?

@ Stevo.. I'm curious whether you currently find each batch is having "similar" micro. "characteristic " (which are ??) or not based on existing data. IMEX, you typically get substantially different results for things like APC even when you are confident that the material is basically similar due to limitations on the methodology, presumably ANOVA can accept "poor" data like this ?? I remember using it (manually) for other routine scenarios but lot of effort required (which now computerised of course ). (even t-tests which i presume are limited version of same objective require some slog )

@ Stevo..So far, looks like you are going to hv to look up some statistics books.

Rgds / Charles.C

added - one thing I am sure of is that it is wise to minimise any micro.work since it is usually slow, expensive and of limited accuracy for variable data, and requires large sampling numbers for detecting zero tolerance factors (or concluding not present in, say, 90pct of a lot to a 95pct confidence level, - in fact it would be very interesting to know what sampling level is used in typical issued COAs which I suppose could represent yr guide, my guess is it's quite low.)

[StevoTheDevo]

I'm only looking at APC, Yeast and Moulds and Coliforms.

Results for the product in question are pretty varied..
APC results for the past 15 batches.. cfu/g
800
140
300
200
140
390
140
800
700
220
350
340
230
6000
100

This is a VERY simple blend of 5 components of which 3 comprise 93% of the formulation!

Edited by StevoTheDevo, 20 January 2009 - 03:58 AM.

[StevoTheDevo]

We typically only report Micro results at the customer's request. Primarily due to expense.

Oh it should be noted that our spec for the above results is less than 10^5 cfu/g.

Edited by StevoTheDevo, 20 January 2009 - 03:58 AM.

[Charles.C]

Dear Stevo..

OK, so no nasty pathogens involved which makes things a bit statistically easier I guess.

Each value is a single sampling or a composite from different locations within same batch ? Or ?

Are any values from lots which would be expected to be very similar, eg produced similar time span ?

Not unusual variations for APC, ie not so small, but conformance is obviously not an issue for this parameter. If the rest are similar i can appreciate yr desire not to waste analysis money. Are these in-house data, ie you trust them ?

Ever made duplicate measurements on the same sample or analysed additional samples from different parts of same batch ? or tested successive batches ?, ie just to get idea of repeatability of procedure, variation within a lot, between lots, whatever. Always easier to experiment if have internal lab of course.

I'm still uncertain as to what kind of accuracy is assumed when issuing a COA. Is it arbitrary ? I always, unasked, issue "typical" results for all products but without any specific guarantee (except usual pathogens of course). Some buyers require specific data per production code for which we mutually agree on number of samples to be taken from a final lot (defined by time or quantity, or both). Like your APC data, conformance was also not an issue (long history of no ppathogen problems also) so both sides agreeable not to waste money by overkill, ie some mutual trust involved. In yr case for example, via analogy to the nMc type system [another option BTW], might hv agreed to take minimum 3 samples per code which would then be spaced out over the production run. The exact statistical significance of such can be determined (probably not that impressive ) but if all results for a non-hazardous parameter are consistently way below an agreed limit as you show, you hardly care I think. Comes back to the actual case / data as usual.

Only throwing ideas out, my guess is that AS NUR's suggestion should be well suited for data like you present also if you want a rigorous evaluation, but at present I expect the data is insufficient.

It's an interesting post

Rgds / Charles.C

[Charles.C]

Dear Stevo..,

I am answering my own (incorrect) comment in previous post.

but if all results for a non-hazardous parameter are consistently way below an agreed limit as you show, you hardly care I think.

Of course you do care since this is a COA and not a COC and a user (presumably) has the right to cross-check yr result similar to this comment -
 COA.jpg   17.96KB   39 downloads
I had a look round the IT on yr topic and the possible varieties of COA are predictably "legion" depending on the ultimate objective, eg calibration type documents are at one extreme, -
http://www.rocklabs....cates/OxC58.pdf.

Maybe something like ANOVA is good if you hv facilities / experience but I'm guessing many statistically un-clever people (eg me) would initially try something like the basic book formula for determining the necessary minimum sample size to attain a specified accuracy / specified confidence level assuming a group of batches in 1 run represents one lot (requires a trial value of the st.dev. of the lot (take some samples and analyse/compute)). Then randomly (or even uniformly [but this does create some known error from memory]) split that answer across the number of individual batches. Hopefully the answer would be a small number like 3 . Not sure what a result > 10 would mean though (more frequent samples I suppose, plus a new problem ). Presumably this logic might satisfy yr higher-ups. Certainly for mineral composition work i was also able to composite the (crushed) acquired sub-samples but there are rules about whether you can do this for any particular case / analysis which is over my head. Again, all this is relatively easy if yr lab is in-house, otherwise starts getting more complex.

Rgds / Charles.C

Added – since the recommendations / comments seemed relevant, I partially extracted a section of a book on official procedures for determining quality of petroleum products (liquids and solids) into the attached document. My guess is that some of the general conclusions presented were the result of ANOVA type analyses. Illustrates that yr type of problem is well-known across industries.

 sampling_packages_for_average_quality.doc   25.5KB   113 downloads

[StevoTheDevo]

Testing is done externally in a NATA lab so I trust the results and all results provided are as per the old practice of sampling 1 batch per production run (not a composite sample), so if we were to see much variation, this would be the place to see it, with different raw material batches and different operators between each result.

Results for Coliforms have always been <3/g using the MPN method and results for Yeast and Moulds varies between <10 and <100/g. I have never seen higher. Once again, these are an order or more below our acceptable limits.

I'll look into AS NUR's method and hopefully end up with a method to composite sample a day's production into 1 test.

[AS NUR]

Testing is done externally in a NATA lab so I trust the results and all results provided are as per the old practice of sampling 1 batch per production run (not a composite sample), so if we were to see much variation, this would be the place to see it, with different raw material batches and different operators between each result.

Results for Coliforms have always been <3/g using the MPN method and results for Yeast and Moulds varies between <10 and <100/g. I have never seen higher. Once again, these are an order or more below our acceptable limits.

I'll look into AS NUR's method and hopefully end up with a method to composite sample a day's production into 1 test.

Dear stevo..

The one of important factor for validity analysis result is homogenity of sample.. if you can assure that your sample is homogen .. IMO.. you can choose sample from any batches to make COA or COC..

And in term of Micro analysis, here i attach the guidelines for measure uncertainty of micro test.. hope can help you

Attached Files

•  qsop4_uncertainty_of_measurement_testing_.pdf   134.21KB   121 downloads

[AS NUR]

DEar Stevo..

acoording to your data.. can yuo give us some information:

1. How many replicate during micro test ( 2 times or 3 times)

2. How many reproducibilty ( how many person that analysis of sample). and how about the reults?

Or
Can yuo geve us raw data, i'll try to countyour RSD and see what the result is ...

[Charles.C]

Dear AS NUR,

I totally appreciate yr logic over homogeneity but it is rather difficult to believe that a stepwise batch procedure ( including blends) is so consistent in its output to be that perfect (no offence Stevo..). Additionally, on intuitive grounds only, it is difficult to convince a third party that a reliable result for a chain of 10 events is adequately described by a sample from the first one only. (2 might work of course ).
However I agree that the variability between batches may well be much less than the testing error, this is of course one purpose of the ANOVA. One problem is that, if there is no existing data as per yr last post, the analytical cost of proving the desired objective may not be negligible (though APC's are pretty cheap usually ?).

Rgds / Charles.C

added - If you do send a group of these samples out for analysis, I would make sure you use random numbers and include one "ringer" of known result if that possible (I know not so easy where bacteria involved).
IMEX, getting agreement within 10% on a replicate APC is almost impossible on a routine basis, anything from 10 - 100% is normal (and not significant if well within limits of course)( just like previous results).

[StevoTheDevo]

OK I think we're drifting off topic a bit..
We're heading more down the microbiology-statistics path than the sampling statistics path.

At least, I think they're different paths!

I think the homogoneity of the sample taken is representative of that batch, but I recognise that it is not representative of the day's production which is my goal.
At worst, I'd like to have 3 representative samples from the entire day, at best, I'd like to have 1 sample. (The samples would need to be composites from across the day.)

Once that's sorted, then we can look at Micro-statistics.

[Charles.C]

Dear Stevo..

At least, I think they're different paths!

Sorry to disagree but I think you are incorrect

You can of course take as many samples as you wish. The question is how accurate you wish the result to be ? This unfortunately involves statistics which minimally requires having (a) sufficient data and (b) knowing its reliability, © knowing how to combine (a) and (b)

Putting it another way, do you care if someone analyses yr product and gets an APC number 100% different from yr stated COA value (I assume no +/- error is currently given on COA)?. Perhaps not if their result is still “well in spec” with regards to the max 100,000. (ie matches a COC). And perhaps if the receiver has experience with bacterial data, they will be equally happy also. Ultimately, it’s yr choice of course but if you wish to evaluate that choice, the statistical and the microbiological aspects are both involved. If you were only talking about, say, the composition of the blend, my guess (no experience) is that an interpretation / conclusion would be much, much easier.

Regardless, good luck with calculations and very interested to hear if you make a successful conclusion.

Rgds / Charles.C

[AS NUR]

yes.. agree with charles..

You have to see the spand of your data, and how many the uncertainty of your data..
is that enough sample to summarize the population?.. there is the statistical of your sample and statistical of the resul interrelation each other...

hope . i can hear the sound of success from you stevo...

[Charles.C]

Dear Stevo..,

I am definitely not a statistician and you probably know most of the below already but here is one very simplified visualisation of yr problem. This is deduced (by me) from the ASTM manual on quality control of materials (ASTM E122 – 58). Similar maths will be found in all stats. books however ASTM present it in relatively ready-to-use form. This is no replacement for AS NUR’s proposal, just a quick way of getting an idea. I once used a similar approach for designing a sampling plan for container shipments of cartons (to estimate net weight) .

In the crudest way, can view situation as single sampling of a large lot of primary units. (correction for a small lot later)

Formula is n = (3v/e)exp2 (exp2 = squared)

n = necessary size of sample, ie number of sampled objects

v = coefficient of variation in per cent = (100).(sigma)/(Xbar) = the advance estimate of the coefficient of variation of the (desired characteristic of the) material, expressed in percent

sigma = the advance estimate of the standard deviation of the lot

e = 100E/(Xbar) = the allowable sampling error expressed as a per cent of Xbar

Xbar = the expected value of the characterisic being measured (ie the mean)

E = the maximum allowable difference between the estimate to be made and the result of testing (by same method) all the units in the universe

3 = a factor corresponding to a probability of about 3 parts in 1000 that the difference between the sample estimate and the result of testing (by the same method) all the units in the universe is greater than E. The choice of the factor 3 is recommended for general use.

Note – the advantage of using “v” is that it “standardises” the deviation with respect to the average quality characteristic
Note – don’t worry about strange terms like universe, I think this is for statistical thoroughness.

Example (my own) - you are seeking a maximum value of 3 for n
e is maybe +/-10 percent

plugging the two values in gives v = approx sq.root(33) = 5 - 6

I seem to remember reading somewhere that many typical powder mixing devices can achieve “v” numbers of around 3-6 percent with respect to composition unifomity but you probably have a better idea than me for that. Whether the (“true”) bacterial plate count has a similar “v” variation will of course depend on factors like materials all from the same source etc. Frankly hv no idea.

There are all sorts of approximations in the above but it gives you an idea.

One more comment – in the total absence of detailed statistical knowledge, can look at the historical smallest and largest values of the characteristic and estimate the sigma values from (largest – smallest) / 6 (assumes normal curve). The least optimistic book value (ie largest st.dev) is for a hypothesised rectangular distribution where formula is (largest – smallest) / 3.5.
[added - If you have a play with the data in earlier post, can see the result is "v" values in range 20-30 percent, the high value simply reflects their apparent variation. This may simply mean that the (absolute) bacteriological quality substantially varies from lot to lot (eg genuinely different raw materials) or that the testing precision is low or other reasons. The distinction is the potential benefit of ANOVA but I'm sure will require replicate testing, etc to proceed along that route. Alternatively, can perhaps substantially increase the value of "e" in my above formula to acknowledge the acceptable tolerance required ]

For a small lot, the formula is (nL) = (N/(N+n)).n

nL = necessary sample size for finite lot
N = total lot size
n = value from first equation
(ie the necesssary sample size decreases if a significant part of the lot is used)

Rgds / Charles.C

Ps anyone with more knowledge on this topic is quite welcome to point out any errors in above. I'm sure there will be some