[PD] integer (mis)calculator

[katja]

On Thu, Dec 8, 2011 at 9:13 AM, Joe Newlin <jtnewlin at gmail.com> wrote:
>
> Can someone explain what's going on in the attached example? I'm getting 7.1 minus 7 equals .099. I need to split two-digit integers, so this result is giving me problems.
>

An interesting rounding example. If you resize the numbox you would
see it's not 0.099 but 0.0999999. Six significant decimal digits, that
is also the maximum precision which a numbox can show.

The fraction of 7.1 is 'not terminating' when converted to binary
floating point format. Compare with 1/3 which is not terminating in
decimal floating point format. If you type 7.1 in a numbox or message
box, 7.0999999 is in fact closer to the 32 bit binary float which
represents it. But it would be annoying when you type 7.1 and it
changes to 7.0999999 automatically.

I once compiled Pd with the format specifiers set to 8 significant
decimal digits, and quickly found why 6 decimal digits is the default
instead. Some numbers which I normally use without special
consideration were automatically converted to a representation with
many more digits. Type 0.02 and you get 0.019999999. It's confusing
and it makes object boxes and message boxes larger than you like. On
the other hand, I found that 32 bit floats can indeed represent all
the integers up till 2^24. So it is a pity we can not see or type
these in boxes and save them with patches. Maybe it would be better to
represent the numbers 100000 and higher with 8 decimal digits.

See this IEEE 754 Converter if you want to check the rounding of a
single precision float:

http://www.h-schmidt.net/FloatApplet/IEEE754.html


Katja