[proFit-list] 6.2.14: rounding issue with large numbers?

[pro Fit support]

Looks like that for some reason you target data window must have its number format set to float instead of double precision. Double-click the column head in the data window where your program writes its numbers, and under “format” choose the larger range. Then it should work as you expect.
 
On Nov 12, 2014, at 6:59 AM, Holger Apfel <mail at apfel-edv.de> wrote:

> I've to convert some large numbers from hex to decimal. I'm using a script and so far I've succeeded in converting the hex value into a number within the script and to convert this into a text to put it back into the data sheet.
> 
> But as soon as the value, ex 
> 
> 1.415718971000000e+9 (text within a script)
> 
> is copied into the datasheet with SetCell(x,y,txt) the number is changed into
> 
> 1.415718912000e+9 - which is obviously quite different. As the number represents a time in seconds the last digits are quite important.
> 
> Did I miss something?
> 
> Holger Apfel
> Fischerstr. 3
> 
> 85221 Dachau
> 
> 08131 338997
> 
> 
> 
> 
> _______________________________________________
> proFit-list mailing list
> proFit-list at quantum-soft.com
> http://quantum-soft.com/mailman/listinfo/profit-list_quantum-soft.com