BFGSExceptions

From Mallet

Q: "During training, I sometimes get premature termination, as well as exceptions, in particular, the "Could not step in current direction" exception (line 194), and the "sy" is postive exception (line 152).

A: L-BFGS can exihibit this behavior for several reasons:

(1) The Maximizable object's getValue() and getValueGradient() methods are not consistent. This can be verified by running

TestMaximizable.testValueAndGradient (maxable);

Although TestMaximizable may occasionally fail for correct Maximzable objects, if it passes we can move on to other possibilities. (Also note that this test takes a while, so you can try using a subset of features, using TestMaximizable.setNumComponents (n) before calling testValueAndGradient.)

(2) L-BFGS approximates the Hessian using previous calls to getValue() and getValueGradient(). If the parameters of a Maximizable object are changed outside the control of L-BFGS, you should call LimitedMemoryBFGS.reset() to reset its memory. Otherwise, the approximation may be completely off.

(3) L-BFGS has occasionally had trouble converging for peculiar cases of Maximizable objects. This can occasionally occur when the curvature is particularly sharp. One thing to try is to catch the exceptions, call LimitedMemoryBFGS.reset(), then continue optimization with the current parameters.

(4) Also, our L-BFGS implementation doesn't seem to work for functions that aren't convex. If you try to run L-BFGS with a non-convex objective, you usually get an sy exception.