Extended LBFGS

class optim.lbfgs_modified.LBFGS_MOD(params, lr=1, max_iter=20, max_eval=None, tolerance_grad=1e-07, tolerance_change=1e-09, history_size=100, line_search_fn=None, line_search_eps=0.0001)[source]

Extends the original steepest gradient descent of PyTorch [torch.optim.LBFGS] with optional backtracking linesearch. The linesearch implementation is adapted from scipy [scipy.optimize.linesearch] and relies only on the value of the loss function, not derivatives.

Parameters
  • params (iterable) – iterable of parameters to optimize or dicts defining parameter groups

  • lr (float, optional) – learning rate (default: 1)

  • max_iter (int) – maximal number of iterations per optimization step (default: 20)

  • max_eval (int) – maximal number of function evaluations per optimization step (default: max_iter * 1.25).

  • tolerance_grad (float) – termination tolerance on first order optimality (default: 1e-5).

  • tolerance_change (float) – termination tolerance on function value/parameter changes (default: 1e-9).

  • history_size (int) – update history size (default: 100).

  • line_search_fn (str, optional) – line search algorithm. Supported options "strong_wolfe", "backtracking" (default: None)

  • line_search_eps (float, optional) – minimal step size (default: 1.0e-4)

step_2c(closure, closure_linesearch)[source]

Performs a single optimization step.

Parameters
  • closure (callable) – A closure that reevaluates the model and returns the loss.

  • closure_linesearch (callable, optional) – A closure that reevaluates the model and returns the loss in torch.no_grad context