Hi everyone!
I am currently running some diagnostics on a constrained problem.
I am confused as to what the number 1e+x (or 1e-x) means, for instance in:
“
Finite difference test on the gradient of f
The relative difference (1e+2): 9.9999842034846176e-01
”
Could someone please clarify this?
Thank you.
Hi, Samuel! 1e+x and 1e-x describes the step length used in the finite difference formula used to check whether the computation of the gradient is consistent with the function itself. Specifically, we use a four point finite difference scheme to approximate the gradient using the formula:
<grad f(x),dx> = (f(x-2 eps dx) - 8 f(x-eps dx) + 8 f(x+eps dx) - f(x+2 eps dx)) / (12 eps)
where = really means approximately equal to and <.,.> denotes the inner product, which is probably <x,y>=x' y with ' meaning transpose. Anyway, this approximation itself has error and we don’t know apriori how to find the best eps for the approximation. Therefore, we generate a series of tests where eps varies from 1e+2 = 100 to 1e-5 = 0.00001. Generally, we want to see a hump in these results, which means the error starts out somewhat high at 1e+2, becomes low, and then becomes high again by the time we hit 1e-5, but it really depends on the scaling of the problem. Sometimes, it just low for each eps. If it’s always high, then that indicates that there’s a problem with the definition of the gradient. Meaning, there’s likely a mistake in our code that generates the gradient, which will probably break the optimization process. All that said, this is really a tool to help find and diagnose problems, but not a perfectly precise technique.
Anyway, hopefully that helps. Let me know if you’ve additional questions!