Re: Trouble getting 30 decimal places

[Clay Lawrence <clay lawrence gmail com>]

Thanks. I had similar results with Excel and Calc. Actually got one more place of precision with Gnumeric. I was just testing one windoze spreadsheet against the others to see which one gave the best answer.

On Tue, Jul 17, 2012 at 5:37 AM, Jean Brefort <jean brefort free fr> wrote:

It's a precision issue. Gnumeric uses C double to store real numbers and
you can't expect more that 17 decimals. Compiling with
--with-long-double would give a better precision, but this has always
been marked as experimental.

Regards,
Jean

Le lundi 16 juillet 2012 à 19:24 -0400, Clay Lawrence a écrit :

> Can someone else try this and see if they get the same results I do.
> I'm using Gnumeric Spreadsheet 1.10.16
>
>
> Format a cell to 30 decimal places and enter =1/sqrt(2)
> I get 0.707106781186547460000000000000 which rounds after 17 places
> instead of 30.
>
>
> I also went to WolframAlpha and calculated 1/√2 to 30+ places;
> 0.7071067811865475244008443621048490392848359376884740 but after
> keying this in directly Gnumeric still rounds it to
> 0.707106781186547570000000000000 as soon as I hit the enter key.
>
>
> Is there a setting somewhere that I can set to get Gnumeric to use all
> of the 30 of the places it displays instead of rounding to 17 places
> and padding the remainder with zeroes?
>
>
> --
>
> Clay
>
> "There are only 10 kinds of people in the world --
>    Those who understand binary, and those who don't."
>
>
>

> _______________________________________________
> gnumeric-list mailing list
> gnumeric-list gnome org
> https://mail.gnome.org/mailman/listinfo/gnumeric-list

-- 

Clay

"There are only 10 kinds of people in the world --
   Those who understand binary, and those who don't."