Forum Discussion

IvanVonLyubenov's avatar
IvanVonLyubenov
Former Employee
Apr 16, 2020

Calculation issues with prime number calculation

Hi all,

 

I have identified issue with simple calculation.
If you SUM following this exact order (29/21)+(27.5/21)+(21/21)+(21/21*-1)+(29/21*-1)+(3.2/21*-1)+(0.25/21)+(27.5/21*-1)+(0.25/21*-1)+(3.2/21) the result would not be 0, but 0.0000000000000002775557561562890.

 

29 is prime number and most probably it is causing the problem - I guess when it's in the middle of the calculation the result is indefinite number and afterwards it cannot cope with second indefinite number.

 

Additionally, if you switch the places of the numbers the result changes and if you put the positive and the negative numbers next to each other the result will be 0.

Same behavior is experienced when Pivot is created.

 

I am attaching the file with the tests.
Column A are the numbers used,
Column B are the calculated result
Column C are the exact numbers, instead of calculation
Column D are the same numbers, but sorted.
Row 12 are the totals.

 

The pivots below show the results when the numbers are sorted and when they're not.

 

Thank you for your time.

Ivan Lyubenov

E.: v-ivlyub@microsoft.com

M.: +359 882 474 305

4 Replies

Resources