# [PD] finite precision in basic math?

Claude Heiland-Allen claude at mathr.co.uk
Mon Jan 7 13:43:30 CET 2019

```Hi Peter,

On 07/01/2019 12:24, Peter P. wrote:
> Hi,
>
> when I send the numer 0.7 to both inlets of a [* ] object (yes, in
> correct order) the result is printed as 0.49
> If I now subtract [- 0.49] the result is -2.98023e-08
> While this is no major concern for me right now, I am still wondering
> which step in this simple calculation is triggering this. If I remember
> my CS lecture correctly it is the subtraction, right?

it's all the steps!

0.7 = 7/10 is not exactly representable in binary floating point, so you
actually get a nearby fraction with a power of 2 as the denominator:
11744051/16777216 = 0.699999988079071044921875
0.49 = 49/100 is also not exactly representable in binary floating
point, so you actually get:  2055209/4194304 = 0.4900000095367431640625
when you multiply two floating point numbers, the result is rounded to
fit the available precision (for single precision float, that is
24bits): you get 16441671/33554432 = 0.4899999797344207763671875

subtracting the last two numbers (which is again rounded) reveals the
behaviour of floating point that is hidden by Pd only printing 6 digits
at most:

0.7*0.7-0.49 computed as float = -1/33554432 = -2.98023223876953125e-8

other environments print more digits, which shows the behaviour sooner:

ghci>  (0.7 * 0.7::Float)
0.48999998

Claude
--
https://mathr.co.uk

```