[proFit-list] 6.2.14: rounding issue with large numbers?

[Holger Apfel]

Yes, that resolved the issue!

Thanks a lot.

Holger

Am 12.11.2014 um 14:16 schrieb pro Fit support:

> Looks like that for some reason you target data window must have its number format set to float instead of double precision. Double-click the column head in the data window where your program writes its numbers, and under “format” choose the larger range. Then it should work as you expect.
> 
> On Nov 12, 2014, at 6:59 AM, Holger Apfel <mail at apfel-edv.de> wrote:
> 
>> I've to convert some large numbers from hex to decimal. I'm using a script and so far I've succeeded in converting the hex value into a number within the script and to convert this into a text to put it back into the data sheet.
>> 
>> But as soon as the value, ex 
>> 
>> 1.415718971000000e+9 (text within a script)
>> 
>> is copied into the datasheet with SetCell(x,y,txt) the number is changed into
>> 
>> 1.415718912000e+9 - which is obviously quite different. As the number represents a time in seconds the last digits are quite important.
>> 
>> Did I miss something?
>> 
>> Holger Apfel
>> Fischerstr. 3
>> 
>> 85221 Dachau
>> 
>> 08131 338997
>> 
>> 
>> 
>> 
>> _______________________________________________
>> proFit-list mailing list
>> proFit-list at quantum-soft.com
>> http://quantum-soft.com/mailman/listinfo/profit-list_quantum-soft.com
> 
> 
> 
> 
> 
> _______________________________________________
> proFit-list mailing list
> proFit-list at quantum-soft.com
> http://quantum-soft.com/mailman/listinfo/profit-list_quantum-soft.com

Holger Apfel
Fischerstr. 3

85221 Dachau

08131 338997