Re: complex : needless loss of precision [mostly fixed]



The problem is known.  Stuffing two values into a string like this never
was a good idea, but we're stuck with it.

The proper precision to use is the one used by the xml format:

        l10 = gnm_log10 (FLT_RADIX);
        res->output.decimal_digits = (int)gnm_ceil (GNM_MANT_DIG * l10) +
                (l10 == (int)l10 ? 0 : 1);

But using this is not without problems: in a lot of cases you get numbers of
the form 0.1000...0001 even when 0.1 would produce the same value.[*]

I think should define that a NULL format given to  complex_to_string should
mean that enough precision should be used to round-trip the values.

Morten



[*] And once that is said, one really has to start worrying over
whether the C library
does the closest-representable-value rounding or just whatever the C standard
allows.


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]