Re: complex : needless loss of precision [mostly fixed]
- From: Morten Welinder <mortenw gnome org>
- To: John Denker <jsd av8n com>
- Cc: Gnumeric Spreadsheet List <gnumeric-list gnome org>
- Subject: Re: complex : needless loss of precision [mostly fixed]
- Date: Mon, 8 Apr 2013 15:27:46 -0400
The problem is known. Stuffing two values into a string like this never
was a good idea, but we're stuck with it.
The proper precision to use is the one used by the xml format:
l10 = gnm_log10 (FLT_RADIX);
res->output.decimal_digits = (int)gnm_ceil (GNM_MANT_DIG * l10) +
(l10 == (int)l10 ? 0 : 1);
But using this is not without problems: in a lot of cases you get numbers of
the form 0.1000...0001 even when 0.1 would produce the same value.[*]
I think should define that a NULL format given to complex_to_string should
mean that enough precision should be used to round-trip the values.
Morten
[*] And once that is said, one really has to start worrying over
whether the C library
does the closest-representable-value rounding or just whatever the C standard
allows.
[
Date Prev][
Date Next] [
Thread Prev][
Thread Next]
[
Thread Index]
[
Date Index]
[
Author Index]