Comments on: Or did my elementary school teachers lie to me?
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me/
Comments on Ask MetaFilter post Or did my elementary school teachers lie to me?Thu, 02 Apr 2009 11:52:01 -0800Thu, 02 Apr 2009 11:52:01 -0800en-ushttp://blogs.law.harvard.edu/tech/rss60Question: Or did my elementary school teachers lie to me?
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me
Which one is right, Google or Excel? <br /><br /> I've discovered that Google's calculator and Excel give slightly different results when converting metric and English weights. For example, when I convert 1 gram to ounces in Excel, I get: 0.0352739<strong>7</strong>2 whereas Google gives me 0.0352739<strong>6</strong>2.<br>
<br>
I just tried converting pounds to ounces in Excel. It's telling me that 1 pound contains 16.000002342941 ounces. Is that an error of some kind in Excel, or am I missing something?post:ask.metafilter.com,2009:site.118451Thu, 02 Apr 2009 11:49:37 -0800JDHarperexcelunitsunitconversionmetriccalculatorgoogleerrorresolvedBy: Smarson
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696426
No sure, but <a href="http://www.metafilter.com/79687/No-crime-is-so-great-as-daring-to-excel-Winston-Churchill#2474834">this post</a> might provide some insight.comment:ask.metafilter.com,2009:site.118451-1696426Thu, 02 Apr 2009 11:52:01 -0800SmarsonBy: Smarson
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696428
Specifically <a href="http://www.metafilter.com/79687/No-crime-is-so-great-as-daring-to-excel-Winston-Churchill#2474227">this comment</a>.comment:ask.metafilter.com,2009:site.118451-1696428Thu, 02 Apr 2009 11:53:18 -0800SmarsonBy: bonehead
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696446
Floating-point precision is well-known problem in Excel: Microsoft details the error in <a href="http://support.microsoft.com/kb/78113">this knowledge base article</a>.<br>
<br>
The official definition of an (US avoirdupois) ounce can be found <a href="http://ts.nist.gov/WeightsAndMeasures/Publications/appxc.cfm#4i">here</a>. Using the exact value of 1 oz = 28.349523125 g, NIST gives the inverse as 0.03527396 g/oz (av.), so I'd say Google is (more) correct.comment:ask.metafilter.com,2009:site.118451-1696446Thu, 02 Apr 2009 12:02:40 -0800boneheadBy: orthogonality
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696447
Simply put, floating point calculations on computers are only arbitrarily precise. Some numbers, like 0.1, can't be represented in binary without loss of precision. So numbers get rounded off. Surprisingly, there are a number of ways to round off; the way that's used often can't be (easily) selected by a user or programmer.comment:ask.metafilter.com,2009:site.118451-1696447Thu, 02 Apr 2009 12:02:51 -0800orthogonalityBy: pombe
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696448
That's got to be an error in excel - there are<a href="http://en.wikipedia.org/wiki/Avoirdupois"> 16 ounces to a pound by definition</a>.comment:ask.metafilter.com,2009:site.118451-1696448Thu, 02 Apr 2009 12:03:48 -0800pombeBy: SAC
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696450
<a href="http://ts.nist.gov/WeightsAndMeasures/Publications/appxc.cfm#4i">This</a> seems to suggest Google is right, and <a href="http://www.gramstoounces.org/">this</a> could give the possible explanation that Google and Excel are using different grams, "Its size can vary from system to system depending on the country." I can't speak to the credibility of the second site, but nist.gov looks legit.comment:ask.metafilter.com,2009:site.118451-1696450Thu, 02 Apr 2009 12:06:21 -0800SACBy: fantabulous timewaster
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696453
It's a poorly-controlled <a href="http://en.wikipedia.org/wiki/Floating_point">floating-point</a> truncation error.comment:ask.metafilter.com,2009:site.118451-1696453Thu, 02 Apr 2009 12:07:03 -0800fantabulous timewasterBy: pombe
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696457
Ah, of course it's a floating point error. I feel kind of silly that I overlooked that.comment:ask.metafilter.com,2009:site.118451-1696457Thu, 02 Apr 2009 12:08:45 -0800pombeBy: equalpants
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696458
Using Wikipedia's <a href="http://en.wikipedia.org/wiki/Gram#Conversion_factors">conversion factor</a>, I get the same thing as Google using the Windows Calculator, which uses arbitrary-precision math.<br>
<br>
Excel is notorious for sloppy floating-point math.comment:ask.metafilter.com,2009:site.118451-1696458Thu, 02 Apr 2009 12:08:57 -0800equalpantsBy: bonehead
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696461
NIST's list (liked to above) isn't just another compilation of conversion tables, it's the American government body that defines what an ounce is in the US. It's officially correct.comment:ask.metafilter.com,2009:site.118451-1696461Thu, 02 Apr 2009 12:11:10 -0800boneheadBy: mjg123
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696462
Well, they're both wrong. What I think is happening is as follows:<br>
Just as there is no decimal expansion which is <b>exactly</b> equal to 1/3 (0.3333333333333333 is never quite right, no matter how many 3's you use). So, there is no binary expansion which is precisely equal to the number of grams in an ounce.<br>
Naturally, there is a <a href="http://en.wikipedia.org/wiki/Floating_point#Accuracy_problems">more complex explanation</a> on wikipedia. But it is enough to say that unless great care is taken, there are usually rounding errors on decimal caluclations with computers. And you will really miss that 0.00000001 of a gram.<br>
I expect google and excel give different answers because they use different implementations of floating-point numbers.comment:ask.metafilter.com,2009:site.118451-1696462Thu, 02 Apr 2009 12:11:12 -0800mjg123By: bonehead
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696472
Also, for those saying that it's just a floating-point error, the problem is more subtle than that. It's not just a normal double-precision floating-point rounding error (which would be predictable under IEEE 754), it's a particularly MS Excel-specific rounding error, and thus rather unpredicatable.comment:ask.metafilter.com,2009:site.118451-1696472Thu, 02 Apr 2009 12:15:33 -0800boneheadBy: equalpants
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696477
<em>Just as there is no decimal expansion which is exactly equal to 1/3 [...]</em><br>
<br>
Come on, man, that's overkill. Sure, Google's answer is "wrong" in that it's not exact, it's rounded to 9 digits. But it's <em>correctly</em> rounded. It is the closest 9-decimal-digit number to the true value. It is <em>more</em> correct than Excel, in a very important sense.comment:ask.metafilter.com,2009:site.118451-1696477Thu, 02 Apr 2009 12:20:10 -0800equalpantsBy: JDHarper
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696489
Ah, I had heard that Excel had problems with floating-point operations before, but I didn't realize what that meant. <br>
<br>
Just to make sure I understand: it sounds like the problem that everyone faces is a loss of precision when you switch from fractions to binary (sort of like the problem of switching from fractions to decimal numbers, like mjg123 noted above).<br>
<br>
And it sounds like the problem specifically with Excel is that they do their rounding differently from everyone else in some way.<br>
<br>
It just seems a bit absurd to me that they didn't get 1lb = 16oz right. I wonder how their conversion formula works.comment:ask.metafilter.com,2009:site.118451-1696489Thu, 02 Apr 2009 12:34:38 -0800JDHarperBy: nickerbocker
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696492
My own 2 cents: in a real world scenario how is the inaccuracy of either Google or Excel relevant in regards to weighing something? The user and instrumentation error are going to far out-weigh the floating point representation and arithmetic error. If you ever weighed something, and then reported that to me as being 16.000002342941, I would probably LMAO because there is no way you have that level of accuracy (never mind precision).<br>
<br>
So my answer is, Google and Excel are "equally good enough." (Awkward statement lol).comment:ask.metafilter.com,2009:site.118451-1696492Thu, 02 Apr 2009 12:36:13 -0800nickerbockerBy: JDHarper
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696502
Yeah, for what I'm doing now, I don't need precision to twenty decimal places, but in some cases <a href="http://www.cnn.com/TECH/space/9909/30/mars.metric.02/">precision really matters</a>.comment:ask.metafilter.com,2009:site.118451-1696502Thu, 02 Apr 2009 12:43:30 -0800JDHarperBy: 0xFCAF
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696510
Failing to convert from metric is not a precision error. They are completely unrelated.comment:ask.metafilter.com,2009:site.118451-1696510Thu, 02 Apr 2009 12:48:21 -08000xFCAFBy: phil
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696512
the error occurred because lockheed martin and nasa used two different units of measurement. the <a href="http://en.wikipedia.org/wiki/Mars_Climate_Orbiter">wikipedia article</a> article covers it in detail.comment:ask.metafilter.com,2009:site.118451-1696512Thu, 02 Apr 2009 12:51:01 -0800philBy: bonehead
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696515
<em>it sounds like the problem that everyone faces is a loss of precision when you switch from fractions to binary</em><br>
<br>
That's half of the problem (and the reverse too-when changing from binary back into decimal). The other half of it are the rounding/truncation errors that happen during the division step(s).<br>
<br>
<em>in a real world scenario how is the inaccuracy of either Google or Excel relevant in regards to weighing something?</em><br>
<br>
Problems happen if you want to use that number to start calculating other things and have to do multiple steps. Precision errors can propagate quickly if you're not careful. If, for example, you're doing an analysis that requires minimization, like a non-linear regression, errors can pile up quite quickly. Starting with 9 or 10 decimals of precision, it's not uncommon to end up with only 5 or 6 by the time the minimization process is complete, hundreds or thousands of operations later.comment:ask.metafilter.com,2009:site.118451-1696515Thu, 02 Apr 2009 12:53:39 -0800boneheadBy: JDHarper
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696519
Oops! Quite right. I misremembered the story and posted it without reading it again. (I thought they had made an error in conversion, not that they hadn't converted at all.)comment:ask.metafilter.com,2009:site.118451-1696519Thu, 02 Apr 2009 12:55:28 -0800JDHarperBy: IAmBroom
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1696997
<em>This seems to suggest Google is right, and this could give the possible explanation that Google and Excel are using different grams, "Its size can vary from system to system depending on the country."</em><br>
<br>
<strong>SAC</strong>, a gram's a gram the world round. <br>
<br>
A gram is an SI unit of measure. Previously, some of the predecessors of SI units (in the original metric system) were defined in slightly different ways, but AFAIK the gram has always been defined as 1/1000 of the mass of the definitive 1 kg platinum mass (kept in a vacuum somewhere).<br>
<br>
Countries that use the SI system have only one gram definition, therefore.<br>
<br>
<br>
And, <strong>nickerbocker</strong>, for whom <em>Google and Excel are "equally good enough."</em>, please don't apply for work at either of my most recent employers. The attitude may be appropriate for shop work, but scientists require mathematical accuracy.comment:ask.metafilter.com,2009:site.118451-1696997Thu, 02 Apr 2009 21:09:55 -0800IAmBroomBy: rodgerd
http://ask.metafilter.com/118451/Or-did-my-elementary-school-teachers-lie-to-me#1697069
<i>Some numbers, like 0.1, can't be represented in binary without loss of precision</i><br>
<br>
This is <a href="http://en.wikipedia.org/wiki/Binary-coded_decimal">not true</a>.comment:ask.metafilter.com,2009:site.118451-1697069Thu, 02 Apr 2009 22:57:03 -0800rodgerd