Extended LBFGS

class optim.lbfgs_modified.LBFGS_MOD(params, lr=1, max_iter=20, max_eval=None, tolerance_grad=1e-07, tolerance_change=1e-09, history_size=100, line_search_fn=None, line_search_eps=0.0001)[source]

Extends the original steepest gradient descent of PyTorch [torch.optim.LBFGS] with optional backtracking linesearch. The linesearch implementation is adapted from scipy [scipy.optimize.linesearch] and relies only on the value of the loss function, not derivatives.

Parameters
  • lr – float learning rate

  • max_iter – int maximal number of iterations per optimization step

  • max_eval – int maximal number of function evaluations per optimization step (default: max_iter * 1.25).

  • tolerance_grad – float termination tolerance on first order optimality

  • tolerance_change – float termination tolerance on function value/parameter changes.

  • history_size – int update history size.

  • line_search_fn – str either ‘strong_wolfe’ or None.

  • line_search_eps – float minimal step size

step_2c(closure, closure_linesearch)[source]

Performs a single optimization step.

Parameters
  • closure (callable) – A closure that reevaluates the model and returns the loss.

  • closure_linesearch (callable) – A closure that reevaluates the model and returns the loss in torch.no_grad context