[PD] finite precision in basic math?

[Claude Heiland-Allen]

Hi Peter,

On 07/01/2019 12:24, Peter P. wrote:
> Hi,
>
> when I send the numer 0.7 to both inlets of a [* ] object (yes, in
> correct order) the result is printed as 0.49
> If I now subtract [- 0.49] the result is -2.98023e-08
> While this is no major concern for me right now, I am still wondering
> which step in this simple calculation is triggering this. If I remember
> my CS lecture correctly it is the subtraction, right?

it's all the steps!

0.7 = 7/10 is not exactly representable in binary floating point, so you 
actually get a nearby fraction with a power of 2 as the denominator: 
11744051/16777216 = 0.699999988079071044921875
0.49 = 49/100 is also not exactly representable in binary floating 
point, so you actually get:  2055209/4194304 = 0.4900000095367431640625
when you multiply two floating point numbers, the result is rounded to 
fit the available precision (for single precision float, that is 
24bits): you get 16441671/33554432 = 0.4899999797344207763671875

subtracting the last two numbers (which is again rounded) reveals the 
behaviour of floating point that is hidden by Pd only printing 6 digits 
at most:

0.7*0.7-0.49 computed as float = -1/33554432 = -2.98023223876953125e-8

other environments print more digits, which shows the behaviour sooner:

ghci>  (0.7 * 0.7::Float)
0.48999998


Claude
-- 
https://mathr.co.uk