Documentation

FirstOrderMethodsOptimization_Beck_2017.Chap10.Theorem_10_40

noncomputable def mfistaResidualToOptimal {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] (x z : E) (xStar : E) (n : ) :
E

Helper for Theorem 10.40: the theorem uses the source residual u_n = t_n z^n - (x* + (t_n - 1) x^n).

Instances For

    Helper for Theorem 10.40: the canonical momentum sequence stays above 1.

    Helper for Theorem 10.40: the FISTA momentum recursion satisfies t_(n+1)^2 - t_(n+1) = t_n^2.

    theorem mfistaAlphaMulLf_eq_maxStepsize {Lf : NNReal} {α : } {s : PosReal} {η : ProximalGradientBacktrackingGrowthFactor} (hLf : 0 < Lf) ( : α = max (↑η) (s / Lf)) :
    max (η * Lf) s = α * Lf

    Helper for Theorem 10.40: the textbook backtracking constant α = max {η, s / L_f} is equivalent to the stepsize cap α L_f = max {η L_f, s}.

    theorem mfistaObjectiveEqReal_of_memEffectiveDomain {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} (hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf) {xPoint : E} (hxPoint : xPoint effective_domain g) :
    composite_model_objective (Function.toEReal f) g xPoint = (f xPoint + (g xPoint).toReal)

    Helper for Theorem 10.40: on effective_domain g, the composite objective is the finite real sum f + g.

    theorem mfistaObjectiveGap_eq_coe_sub_toReal_of_memEffectiveDomain {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} (hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf) {xPoint : E} (hxPoint : xPoint effective_domain g) :
    ((composite_model_objective (Function.toEReal f) g xPoint).toReal - FOpt) = composite_model_objective (Function.toEReal f) g xPoint - FOpt

    Helper for Theorem 10.40: every finite objective gap is the cast of its real gap.

    theorem mfistaObjectiveDiff_eq_coe_sub_of_memEffectiveDomain {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} (hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf) {xPoint zPoint : E} (hxPoint : xPoint effective_domain g) (hzPoint : zPoint effective_domain g) :

    Helper for Theorem 10.40: once both composite values are finite, their difference is the canonical EReal coercion of the real difference of their toReal values.

    theorem mfistaToReal_ge_FOpt_of_memEffectiveDomain {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} (hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf) {xPoint : E} (hxPoint : xPoint effective_domain g) :
    FOpt (composite_model_objective (Function.toEReal f) g xPoint).toReal

    Helper for Theorem 10.40: every finite objective value is at least the optimal value FOpt.

    theorem mfistaOptimalPoint_memEffectiveDomain {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} {xStar : E} (hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf) (hxStar : xStar XStar) :
    xStar effective_domain g

    Helper for Theorem 10.40: every optimizer has finite g-value, hence lies in effective_domain g.

    theorem mfistaPositiveIterate_memEffectiveDomainG {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} [hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf] {x y z : E} {L : PosReal} (htraj : hproblem.IsMfistaTrajectory x y z L) (n : ) :
    x (n + 1) effective_domain g

    Helper for Theorem 10.40: every positive MFISTA iterate has finite g-value because its objective is bounded above by the finite prox-point objective.

    theorem mfistaPositiveIterateGapNonneg {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} [hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf] {x y z : E} {L : PosReal} (htraj : hproblem.IsMfistaTrajectory x y z L) (n : ) :
    0 (composite_model_objective (Function.toEReal f) g (x (n + 1))).toReal - FOpt

    Helper for Theorem 10.40: every positive MFISTA objective gap is nonnegative on the real layer.

    theorem mfistaOneDivMomentum_memIcc (n : ) :
    1 / fista_momentum_sequence (n + 1) Set.Icc 0 1

    Helper for Theorem 10.40: the reciprocal of the MFISTA momentum coefficient belongs to Set.Icc 0 1.

    theorem mfistaCombinationPoint_memEffectiveDomain {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} [hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf] {x y z : E} {L : PosReal} {xStar : E} (htraj : hproblem.IsMfistaTrajectory x y z L) (hxStar : xStar XStar) (n : ) :
    have θ := 1 / fista_momentum_sequence (n + 1); have w := θ xStar + (1 - θ) x (n + 1); w effective_domain g

    Helper for Theorem 10.40: the source comparison point (1 / t_(n+1)) • xStar + (1 - 1 / t_(n+1)) • x^(n+1) stays in effective_domain g.

    theorem mfistaCombinationObjectiveUpperBoundReal {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} [hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf] {x y z : E} {L : PosReal} {xStar : E} (htraj : hproblem.IsMfistaTrajectory x y z L) (hxStar : xStar XStar) (n : ) :
    have θ := 1 / fista_momentum_sequence (n + 1); have w := θ xStar + (1 - θ) x (n + 1); (composite_model_objective (Function.toEReal f) g w).toReal (1 - θ) * ((composite_model_objective (Function.toEReal f) g (x (n + 1))).toReal - FOpt) + FOpt

    Helper for Theorem 10.40: convexity bounds the objective of the source comparison point by (1 - 1 / t_(n+1)) v_(n+1) + FOpt on the real layer.

    theorem mfistaConvexLinearizationDefect_nonneg {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} (hfast : IsFastProximalGradientProblem f g XStar FOpt Lf) {xPoint : E} (yI : (interior (effective_domain (Function.toEReal f)))) :
    0 ℓ[Function.toEReal f, xPoint, yI]

    Helper for Theorem 10.40: convexity of the real-valued smooth term makes the local linearization defect of f.toEReal nonnegative at every base point.

    theorem mfistaAcceptedProxGapReal {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} [hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf] {xPoint yPoint : E} {Lbar : PosReal} (hxPoint : xPoint effective_domain g) (haccepts : have xPlus := hproblem.proxPoint Lbar yPoint; f xPlus f yPoint + inner (gradient f yPoint) (xPlus - yPoint) + Lbar / 2 * xPlus - yPoint ^ 2) :
    have yI := interior_effective_domain_point_of_real f yPoint; have xPlus := hproblem.proxPoint Lbar yI; Lbar / 2 * xPoint - xPlus ^ 2 - Lbar / 2 * xPoint - yPoint ^ 2 (composite_model_objective (Function.toEReal f) g xPoint).toReal - (composite_model_objective (Function.toEReal f) g xPlus).toReal

    Helper for Theorem 10.40: after dropping the nonnegative convex linearization defect, the fundamental prox-gradient inequality becomes a real inequality on finite endpoints.

    theorem mfistaPrestepVector_eq_previousResidual {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} [hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf] {x y z : E} {L : PosReal} {xStar : E} (htraj : hproblem.IsMfistaTrajectory x y z L) (n : ) :
    fista_momentum_sequence (n + 1) y (n + 1) - (xStar + (fista_momentum_sequence (n + 1) - 1) x (n + 1)) = mfistaResidualToOptimal x z xStar n

    Helper for Theorem 10.40: the MFISTA extrapolation formula gives the exact predecessor transport from the pre-step vector to the previous residual.

    theorem mfistaStepsizeControl {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} [hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf] {y : E} {L : PosReal} {α : } (hrule : hproblem.SublinearRateStepsizeRule y L α) :
    (∀ (n : ), (L n) (L (n + 1))) ∀ (n : ), (L n) α * Lf

    Helper for Theorem 10.40: the accepted stepsize rule supplies both monotonicity of L_n and the uniform cap L_n ≤ α L_f.

    theorem mfistaInitialEnergyBound {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} [hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf] {x y z : E} {L : PosReal} {α : } {xStar : E} (htraj : hproblem.IsMfistaTrajectory x y z L) (hrule : hproblem.SublinearRateStepsizeRule y L α) (hxStar : xStar XStar) :
    have vR := fun (n : ) => (composite_model_objective (Function.toEReal f) g (x n)).toReal - FOpt; mfistaResidualToOptimal x z xStar 0 ^ 2 + 2 / (L 0) * fista_momentum_sequence 0 ^ 2 * vR 1 x 0 - xStar ^ 2

    Helper for Theorem 10.40: the first source energy is bounded by the initial distance to an optimizer on the exact source surface 2 / L_0.

    theorem mfistaCurrentStepGapUpperBound {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} [hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf] {x y z : E} {L : PosReal} {xStar : E} (htraj : hproblem.IsMfistaTrajectory x y z L) (hxStar : xStar XStar) (n : ) :
    have vR := fun (m : ) => (composite_model_objective (Function.toEReal f) g (x m)).toReal - FOpt; have θ := 1 / fista_momentum_sequence (n + 1); have w := θ xStar + (1 - θ) x (n + 1); (composite_model_objective (Function.toEReal f) g w).toReal - (composite_model_objective (Function.toEReal f) g (z (n + 1))).toReal (1 - θ) * vR (n + 1) - vR (n + 2)

    Helper for Theorem 10.40: the source comparison point at step n + 1 satisfies the exact current-step objective-gap estimate on the real layer.

    theorem mfistaAcceptedStepEnergyBalance {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} [hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf] {x y z : E} {L : PosReal} {α : } {xStar : E} (htraj : hproblem.IsMfistaTrajectory x y z L) (hrule : hproblem.SublinearRateStepsizeRule y L α) (hxStar : xStar XStar) (n : ) :
    have vR := fun (m : ) => (composite_model_objective (Function.toEReal f) g (x m)).toReal - FOpt; mfistaResidualToOptimal x z xStar (n + 1) ^ 2 + 2 / (L (n + 1)) * fista_momentum_sequence (n + 1) ^ 2 * vR (n + 2) mfistaResidualToOptimal x z xStar n ^ 2 + 2 / (L (n + 1)) * fista_momentum_sequence n ^ 2 * vR (n + 1)

    Helper for Theorem 10.40: combining the accepted prox-gap with the current-step objective bound yields the exact shared-denominator Lyapunov balance.

    theorem mfistaEnergyStep {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} [hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf] {x y z : E} {L : PosReal} {α : } {xStar : E} (htraj : hproblem.IsMfistaTrajectory x y z L) (hrule : hproblem.SublinearRateStepsizeRule y L α) (hxStar : xStar XStar) (n : ) :
    have vR := fun (m : ) => (composite_model_objective (Function.toEReal f) g (x m)).toReal - FOpt; have ER := fun (m : ) => mfistaResidualToOptimal x z xStar m ^ 2 + 2 / (L m) * fista_momentum_sequence m ^ 2 * vR (m + 1); ER (n + 1) ER n

    Helper for Theorem 10.40: the exact MFISTA Lyapunov energy contracts in one step.

    theorem mfista_objective_gap_le_two_alpha_Lf_dist_sq_div_sq {E : Type u} [NormedAddCommGroup E] [InnerProductSpace E] [ProperSpace E] {f : E} {g : EEReal} {XStar : Set E} {FOpt : } {Lf : NNReal} [hproblem : IsFastProximalGradientProblem f g XStar FOpt Lf] {x y z : E} {L : PosReal} {α : } {xStar : E} (htraj : hproblem.IsMfistaTrajectory x y z L) (hrule : hproblem.SublinearRateStepsizeRule y L α) (hxStar : xStar XStar) (k : ) (hk : 1 k) :
    composite_model_objective (Function.toEReal f) g (x k) - FOpt (2 * α * Lf * x 0 - xStar ^ 2 / (k + 1) ^ 2)

    Theorem 10.40: under Assumption 10.31, any MFISTA trajectory whose curvature estimates are chosen either by the constant rule L_k = L_f with α = 1 or by backtracking procedure B3 with α = max {η, s / L_f} satisfies the accelerated objective-gap bound F(x^k) - F_opt ≤ 2 α L_f ‖x^0 - x*‖^2 / (k + 1)^2 for every optimizer x* ∈ X^* and every k ≥ 1.