• @needanke@feddit.org
    cake
    link
    fedilink
    15
    edit-2
    7 days ago

    (I don’t think that was your teachers point at all, but) couldn’t the different formulas have produced different rounding errors due to floating point percision?

      • yeah because excel does rounding stuff automatically for you

        try entering 0.1 + 0.2 - 0.1 - 0.2 == 0.0 in any programming language of your choice and see what happens.

    • @Blackmist@feddit.uk
      link
      fedilink
      English
      147 days ago

      Doubtful, but if anything mine would be more accurate. Fewer calculation steps to lose precision on. I think most spreadsheet software fudges floating point precision anyway. A computer programmer may accept that 0.1+0.2 is not 0.3 but an accountant or mathematician would not be having it.

      I think she was just shit at maths tbh. As a kid you sort of assume all the teachers know more than you about every subject, and that’s not the case at all.