Do ISO GUM reported uncertainties always require NIST traceability?
July 11, 2011 10:31 AM Subscribe
ISO GUM acceptable measurements w/o NIST traceability possible? Weight and length measurements always require NIST traceability for reported results (if following ISO 17025)? More inside...
Hi!
I'm suppose to be the uncertainty guy. I have two problems that I face while trying to provide an appropriate uncertainty analysis for a test we will be performing. I need to give a reasonable estimate of uncertainty for a weight and length measurement associated with the test setup.
I have a number of scales, and a number of tape measures but none of these would be considered "recently calibrated" with NIST traceability.
My question is, is NIST traceable calibration always necessary?
For the length measurement, I can take multiple measurements with multiple tape measures and calculate and report a mean and variance of these measurements (Type A analysis).
Similarly, I have a number of dumbbells (non-calibrated weights) that I can make observations on multiple scales and compare the outputs to one another. Again, I can perform a Type A analysis on the uncertainty of these measurements.
Would NIST traceability be required at all? The Test Standard does not report a required accuracy of these measurements. Since that is the case, our policy is to follow ISO GUM practices and report a conservative estimate of accuracy. Thanks for your help!
Hi!
I'm suppose to be the uncertainty guy. I have two problems that I face while trying to provide an appropriate uncertainty analysis for a test we will be performing. I need to give a reasonable estimate of uncertainty for a weight and length measurement associated with the test setup.
I have a number of scales, and a number of tape measures but none of these would be considered "recently calibrated" with NIST traceability.
My question is, is NIST traceable calibration always necessary?
For the length measurement, I can take multiple measurements with multiple tape measures and calculate and report a mean and variance of these measurements (Type A analysis).
Similarly, I have a number of dumbbells (non-calibrated weights) that I can make observations on multiple scales and compare the outputs to one another. Again, I can perform a Type A analysis on the uncertainty of these measurements.
Would NIST traceability be required at all? The Test Standard does not report a required accuracy of these measurements. Since that is the case, our policy is to follow ISO GUM practices and report a conservative estimate of accuracy. Thanks for your help!
You are required to have tracablility ("trace of test") under ISO 17025. No exceptions. It doesn't have to be to NIST necessarily, but you must have a tracable calibration. That's basic to the standard. I don't see how you could get around that. Checking on a bunch of out-of-cal.-period scales isn't going to change that, unfortunatly.
In fact, my ISO body (CALA & SCC) requires two independantly-tracable references, a calibration stantard and a verification standard to check against. We've had cases were NIST/NBS materials were bad (ie not as certified).
If you're not doing a certified method (as it sounds like you are not), you are not required to do anything. If you want to follow or approximate ISO 17025, you're going to have to get certification done.
posted by bonehead at 11:57 AM on July 11, 2011
In fact, my ISO body (CALA & SCC) requires two independantly-tracable references, a calibration stantard and a verification standard to check against. We've had cases were NIST/NBS materials were bad (ie not as certified).
If you're not doing a certified method (as it sounds like you are not), you are not required to do anything. If you want to follow or approximate ISO 17025, you're going to have to get certification done.
posted by bonehead at 11:57 AM on July 11, 2011
Response by poster: @odinsdream - At this point, I am the certification organization for my company. Well, I coordinate the calibration of measurement equipment, and keep up with the stickers on everything. For our pressure and temperature measurements it is fairly straight forward as we have reference standards that are traceable to NIST that we use a comparisons for our other instruments that measure these quantities.
We have some weights for our precision scales that we keep NIST traceable.
"Ultimately, it's very unlikely you can do commercial functions with non-certified equipment of any kind." -- seems like contractors use off-the-shelf tape measures all the time in the construction of buildings. Yet, I'm having a hell of a time finding a tape measure that will even state accuracy.
@bonehead - Yes, we are trying to get ISO 17025 certification. We have been audited by one of our clients this year, and did well in the audit. Most everything checked out, but questions regarding how much training I have received were the bigger highlights of where we are lacking. I admit, I have no formal training other than an EE degree (which really isn't a meteorology degree at all). I just have on the job experience and have read through ISO GUM, ISO 17025, ASME PTC 19.1, and some other handbooks on calculating uncertainty.
My major hang up has been finding a calibrated NIST tape measure... can anyone recommend one? How often would it need to be re-calibrated? Thats another thing I have a hard time with is lots of calibration intervals are left to me...and I'm incline to write down that we will re-calibrate once every 10 years (exaggerated) or something unless the manufacturer states that it needs to be done yearly.
Thanks for your input!
posted by nickerbocker at 12:15 PM on July 11, 2011
We have some weights for our precision scales that we keep NIST traceable.
"Ultimately, it's very unlikely you can do commercial functions with non-certified equipment of any kind." -- seems like contractors use off-the-shelf tape measures all the time in the construction of buildings. Yet, I'm having a hell of a time finding a tape measure that will even state accuracy.
@bonehead - Yes, we are trying to get ISO 17025 certification. We have been audited by one of our clients this year, and did well in the audit. Most everything checked out, but questions regarding how much training I have received were the bigger highlights of where we are lacking. I admit, I have no formal training other than an EE degree (which really isn't a meteorology degree at all). I just have on the job experience and have read through ISO GUM, ISO 17025, ASME PTC 19.1, and some other handbooks on calculating uncertainty.
My major hang up has been finding a calibrated NIST tape measure... can anyone recommend one? How often would it need to be re-calibrated? Thats another thing I have a hard time with is lots of calibration intervals are left to me...and I'm incline to write down that we will re-calibrate once every 10 years (exaggerated) or something unless the manufacturer states that it needs to be done yearly.
Thanks for your input!
posted by nickerbocker at 12:15 PM on July 11, 2011
Who are you using as your certification body? They should have a list on hand of acceptible calibration services to answer exactly this kind of question.
For mass we use Fisher Scientific; it costs $275/scale. I note that NIST does length certification. As prices go, those seem to be about average to me. Yearly certification is what we do, though only for a few key pieces of equipment.
As for training, again, we get courses offered every year through our certification body. They are not cheap, but they are well worth taking at least once. This is especially true if you're trying to set up a new method and certifity a lab that's never been certified before.
Feel free to contact me through my user page. I've maintained an ISO chem/phys test lab for more than a decade and would be happy to give you what help I can.
posted by bonehead at 12:25 PM on July 11, 2011
For mass we use Fisher Scientific; it costs $275/scale. I note that NIST does length certification. As prices go, those seem to be about average to me. Yearly certification is what we do, though only for a few key pieces of equipment.
As for training, again, we get courses offered every year through our certification body. They are not cheap, but they are well worth taking at least once. This is especially true if you're trying to set up a new method and certifity a lab that's never been certified before.
Feel free to contact me through my user page. I've maintained an ISO chem/phys test lab for more than a decade and would be happy to give you what help I can.
posted by bonehead at 12:25 PM on July 11, 2011
One last thing, the people at NIST are very helpful in my experience, and quite friendly. You could do a lot worse than calling one of the people listed on that length standard page and having a chat.
posted by bonehead at 12:36 PM on July 11, 2011
posted by bonehead at 12:36 PM on July 11, 2011
Response by poster: Thanks for your reply. I guess I'm confused by certification body. I guess our certification body would be NIST, as just about everything we have that are considered quality measurement items are traceable to NIST. Does that sound correct?
Thanks for the offer on getting in touch with you. I will probably take you up on that with some other questions.
posted by nickerbocker at 12:40 PM on July 11, 2011
Thanks for the offer on getting in touch with you. I will probably take you up on that with some other questions.
posted by nickerbocker at 12:40 PM on July 11, 2011
Tracability is only one part of ISO 17025.
ISO 17025 is a whole management system that covers everything from having and following documented procedures, using certain suppliers, training, data gathering and retention. To get certified, you establish a certified quality management system in your facility, then you apply for accrediation on "scopes", that is to say specific methods. This is done through a certification body who have delegated authority from ISO and (usually) national government approval. That cert body assesses you periodically and requires things like bi-annual proficiency testing to prove that you can do the methods you have certified. A2LA is one of the US bodies which do this. the NIST one is called NVLAP. I don't know the US system very well, nor your industry, so I can't recommend any particular one to you.
In any case, ISO 17025 is usually a pretty big deal, and can be quite time-consuming and expensive to setup and maintain.
posted by bonehead at 12:52 PM on July 11, 2011
ISO 17025 is a whole management system that covers everything from having and following documented procedures, using certain suppliers, training, data gathering and retention. To get certified, you establish a certified quality management system in your facility, then you apply for accrediation on "scopes", that is to say specific methods. This is done through a certification body who have delegated authority from ISO and (usually) national government approval. That cert body assesses you periodically and requires things like bi-annual proficiency testing to prove that you can do the methods you have certified. A2LA is one of the US bodies which do this. the NIST one is called NVLAP. I don't know the US system very well, nor your industry, so I can't recommend any particular one to you.
In any case, ISO 17025 is usually a pretty big deal, and can be quite time-consuming and expensive to setup and maintain.
posted by bonehead at 12:52 PM on July 11, 2011
I consult on the topic of ISO 17025 (among other things) as a profession.
Questions like this are tricky. Standards like 17025 are often deliberately vague about things like calibration intervals because it depends so much on what you are doing. If you use your balance once a month in a pristine lab versus every day in a muddy hole obviously your calibration needs will vary.
I don't feel like the method you suggest of averaging across several devices or weights cuts it. I don't see there is a justification for the assumption that these several measurements represent any kind of representative sample or that you can treat their errors as random (rendering the significance of a mean analysis dubious).
I suspect you should be looking to get a documented figure from an independent source on the tolerances for all your instruments. Manufacturer's guidelines, independent calibration - everyone who has to deal with standards runs into these questions and there are professionals in business to make it easy. You have to balance the savings of doing DIY verification over the trust and ease a third party certification (with appropriate credentials naturally) represents. Likewise, it is really hard to say how often your equipment needs to be calibrated unless you are subjecting it to routine testing with some sort of reasonable standard.
Off the top of my head, try Starrett for N.I.S.T. traceable tape measures.
A word about ISO 17025 "certification" - although this terminology is used pretty often it's not technically correct - ISO specifically states that "ISO/IEC 17025:2005 is not intended to be used as the basis for certification of laboratories." The term is accreditation, the Wikipedia 17025 article (which is decent) states the distinction thus:
In common with other accreditation standards of the ISO 17000 series (and unlike most ISO standards for management systems), third party auditing and appraisal of the laboratory is normally carried out by the national organisation responsible for accreditation. Laboratories are therefore accredited under ISO/IEC 17025, rather than certified or registered (c.f. ISO 9000 series).
In short, accreditation differs from certification by adding the concept of a third party (Accreditation Body (AB)) attesting to technical competence within a laboratory in addition to its adherence and operation under a documented quality system, specific to a Scope of Accreditation. Accreditation is not a requirement to use the standard as the basis for a quality management system. Although it is not directed specifically at the 17000 series, I feel ISO's statement here is relevant:
In most countries, accreditation is a choice, not an obligation and the fact that a certification body is not accredited does not, by itself, mean that it is not a reputable organization. For example, a certification body operating nationally in a highly specific sector might enjoy such a good reputation that it does not feel there is any advantage for it to go to the expense of being accredited. That said, many certification bodies choose to seek accreditation, even when it is not compulsory, in order to be able to demonstrate an independent confirmation of their competence.
Finally, ISO 17025 is really a standard for calibration and testing laboratories. It doesn't sound as if that is what you are exactly. Are you sure it is the appropriate standard for what your aim is?
posted by nanojath at 1:27 PM on July 11, 2011
Questions like this are tricky. Standards like 17025 are often deliberately vague about things like calibration intervals because it depends so much on what you are doing. If you use your balance once a month in a pristine lab versus every day in a muddy hole obviously your calibration needs will vary.
I don't feel like the method you suggest of averaging across several devices or weights cuts it. I don't see there is a justification for the assumption that these several measurements represent any kind of representative sample or that you can treat their errors as random (rendering the significance of a mean analysis dubious).
I suspect you should be looking to get a documented figure from an independent source on the tolerances for all your instruments. Manufacturer's guidelines, independent calibration - everyone who has to deal with standards runs into these questions and there are professionals in business to make it easy. You have to balance the savings of doing DIY verification over the trust and ease a third party certification (with appropriate credentials naturally) represents. Likewise, it is really hard to say how often your equipment needs to be calibrated unless you are subjecting it to routine testing with some sort of reasonable standard.
Off the top of my head, try Starrett for N.I.S.T. traceable tape measures.
A word about ISO 17025 "certification" - although this terminology is used pretty often it's not technically correct - ISO specifically states that "ISO/IEC 17025:2005 is not intended to be used as the basis for certification of laboratories." The term is accreditation, the Wikipedia 17025 article (which is decent) states the distinction thus:
In common with other accreditation standards of the ISO 17000 series (and unlike most ISO standards for management systems), third party auditing and appraisal of the laboratory is normally carried out by the national organisation responsible for accreditation. Laboratories are therefore accredited under ISO/IEC 17025, rather than certified or registered (c.f. ISO 9000 series).
In short, accreditation differs from certification by adding the concept of a third party (Accreditation Body (AB)) attesting to technical competence within a laboratory in addition to its adherence and operation under a documented quality system, specific to a Scope of Accreditation. Accreditation is not a requirement to use the standard as the basis for a quality management system. Although it is not directed specifically at the 17000 series, I feel ISO's statement here is relevant:
In most countries, accreditation is a choice, not an obligation and the fact that a certification body is not accredited does not, by itself, mean that it is not a reputable organization. For example, a certification body operating nationally in a highly specific sector might enjoy such a good reputation that it does not feel there is any advantage for it to go to the expense of being accredited. That said, many certification bodies choose to seek accreditation, even when it is not compulsory, in order to be able to demonstrate an independent confirmation of their competence.
Finally, ISO 17025 is really a standard for calibration and testing laboratories. It doesn't sound as if that is what you are exactly. Are you sure it is the appropriate standard for what your aim is?
posted by nanojath at 1:27 PM on July 11, 2011
Response by poster: @nanojath/@bonehead - We are a test facility with a small number of engineers and technicians on staff. Most of our work is in failure and forensics investigation that often require testing in order to produce data to support our opinions. We also do a lot of CGA and ASTM standard testing directly for customers. Being so small, we have not been required to have an ISO certified quality management system until just recently when one of our larger corporate clients started requiring it (or else they lose their ISO certification). We choose 17025 because we do testing here. We also do calibration of our own equipment (using reference standards), but we do not offer calibration as a service to the general public.
We currently have no certifying/accreditation body as we are not ISO 17025 accredited, but are working towards this and will hopefully become accredited soon.
@Dr Dracator - har har har. I meant metrology which my web browser does not think is a real word. Ever misspell something before? I have and will probably do it again in the future.
posted by nickerbocker at 1:44 PM on July 11, 2011
We currently have no certifying/accreditation body as we are not ISO 17025 accredited, but are working towards this and will hopefully become accredited soon.
@Dr Dracator - har har har. I meant metrology which my web browser does not think is a real word. Ever misspell something before? I have and will probably do it again in the future.
posted by nickerbocker at 1:44 PM on July 11, 2011
nanojath is absolutely corrent on that point, it's not certification but accreditation. Likewise, they're not audits/auditors but assessment and assessors.
Technically, it's true that it's all optional. In practice however, if you want to maintain accreditation and continue to do be able to do business with a growing number of companies, ISO accrediation is compulsory.
posted by bonehead at 1:54 PM on July 11, 2011
Technically, it's true that it's all optional. In practice however, if you want to maintain accreditation and continue to do be able to do business with a growing number of companies, ISO accrediation is compulsory.
posted by bonehead at 1:54 PM on July 11, 2011
You may want to investigate the question of whether your client(s) need you to be actually accredited by a third party or just needs you to document policies consistent with ISO 17025 or whatever. This is a question I've come upon many times. My belief is that while it is a somewhat common opinion that to be accredited (or certified, or registered depending on the relevant standard) all your vendors must be accredited (or certified etc... I'll just say accredited from now on) it is generally not the case (per the ISO statement I cited above). Of course a client has the right to require it but often on reflection (or consultation with their accreditation provider) organizations will realize they don't really need it from everyone. I mention it only because formal accreditation is a big deal, it's costly and it requires routine renewal which is also costly. It can be a real burden for a small lab.
posted by nanojath at 1:55 PM on July 11, 2011
posted by nanojath at 1:55 PM on July 11, 2011
(Having said that, bonehead is correct that it is becoming more frequently required. It is just a lot easier to look at a stamp of approval from an appropriate 3rd party than to actually review a vendor's policies and documentation and decide whether they are sufficient).
posted by nanojath at 1:56 PM on July 11, 2011
posted by nanojath at 1:56 PM on July 11, 2011
We're seeing ISO accrediation required for just about all analytical services now. The labs are just being forced to suck it up and seek accrediation. As you say, it's al lot easier for a client, who is not technical to just write, "conforms to ISO 17025 standards" on an RFP that to go into great detail about requirements. The only major purchaser who still does things their own way in my business is the EPA.
posted by bonehead at 2:00 PM on July 11, 2011
posted by bonehead at 2:00 PM on July 11, 2011
Response by poster: "I mention it only because formal accreditation is a big deal, it's costly and it requires routine renewal which is also costly. It can be a real burden for a small lab."
Tell me about it... it seems that the requirements do not scale well for small labs. It seems to take the same amount of effort for a small or large lab. But, yeah, we will keep getting audited by our customers if we do not become accredited soon. Some of our long-time solid clients have stopped working with us because we have no accreditation.
posted by nickerbocker at 2:12 PM on July 11, 2011
Tell me about it... it seems that the requirements do not scale well for small labs. It seems to take the same amount of effort for a small or large lab. But, yeah, we will keep getting audited by our customers if we do not become accredited soon. Some of our long-time solid clients have stopped working with us because we have no accreditation.
posted by nickerbocker at 2:12 PM on July 11, 2011
« Older Like melt-your-face Raiders of the Lost Ark hot. | Need my grainy, yoghurty deliciousness, please. Newer »
This thread is closed to new comments.
What is your conservative estimate of accuracy? 5% sounds more than generous, depending upon how stringent the requirements.
posted by pentagoet at 11:46 AM on July 11, 2011