Hi,
To display a number rounded down to a user selected number of decimal places, I wrote some code that gave a surprising answer.
Attempt 1
local x = 0.20629048
local dp = 4
local x_round = round(`x', 10^-`dp')
display "this is x `x' this is x_round `x_round'"
The final line results in the following unexpected output - this is x .20629048 this is x_round .2063000000000001
From the following you would expect x_round above to be calculated as .2063
local x = 0.20629048
local dp = 4
display 10^-`dp'
display round(`x', .0001)
However, I can get around the problem with creating a new macro that evaluates the number of decimal places
Attempt 2
local x = 0.20629048
local dp = 4
local dp_value = 10^-`dp'
local x_round = round(`x', `dp_value')
display "this is x `x' this is x_round `x_round'"
The final line results in the following correct output - this is x .20629048 this is x_round .2063
Why does Attempt 1 fail?
Thanks for taking the time to consider this,
Don Vicendese
To display a number rounded down to a user selected number of decimal places, I wrote some code that gave a surprising answer.
Attempt 1
local x = 0.20629048
local dp = 4
local x_round = round(`x', 10^-`dp')
display "this is x `x' this is x_round `x_round'"
The final line results in the following unexpected output - this is x .20629048 this is x_round .2063000000000001
From the following you would expect x_round above to be calculated as .2063
local x = 0.20629048
local dp = 4
display 10^-`dp'
display round(`x', .0001)
However, I can get around the problem with creating a new macro that evaluates the number of decimal places
Attempt 2
local x = 0.20629048
local dp = 4
local dp_value = 10^-`dp'
local x_round = round(`x', `dp_value')
display "this is x `x' this is x_round `x_round'"
The final line results in the following correct output - this is x .20629048 this is x_round .2063
Why does Attempt 1 fail?
Thanks for taking the time to consider this,
Don Vicendese
Comment