Extended LBFGS

class optim.lbfgs_modified.LBFGS_MOD(params, lr=1, max_iter=20, max_eval=None, tolerance_grad=1e-07, tolerance_change=1e-09, history_size=100, line_search_fn=None, line_search_eps=0.0001)

Extends the original steepest gradient descent of PyTorch [torch.optim.LBFGS] with optional backtracking linesearch. The linesearch implementation is adapted from scipy [scipy.optimize.linesearch] and relies only on the value of the loss function, not derivatives.

Parameters:

• lr – float learning rate

• max_iter – int maximal number of iterations per optimization step

• max_eval – int maximal number of function evaluations per optimization step (default: max_iter * 1.25).

• tolerance_grad – float termination tolerance on first order optimality

• tolerance_change – float termination tolerance on function value/parameter changes.

• history_size – int update history size.

• line_search_fn – str either ‘strong_wolfe’ or None.

• line_search_eps – float minimal step size

step_2c(closure, closure_linesearch)

Performs a single optimization step.

Parameters:

• closure (callable) – A closure that reevaluates the model and returns the loss.

• closure_linesearch (callable) – A closure that reevaluates the model and returns the loss in torch.no_grad context