| Lectures | Date | Subjects Covered | Lecture Supplementary Material |
| Lecture 22 | Mar. 4 | KKT optimality conditions and CQs |
Derive the KKT opt conditions using the weakest constraint
qualification: T(F,x)=L(x), i.e. the tangent and linearizing cones are
equal.
Then the KKT conditions comes from the optimality conditions: gradient of f(x) is in the polar of T(F,x). (Follows, since the polar of L(x) is the cone generated by the gradients, by Farkas Lemma.) |
| Lecture 21 | Mar. 2 | Hyperplane separation theorem and applications |
Basic Separation Theorem (proof was on assignment 4); Application to proving the Lemma: K=K^++ iff K is a closed convex cone (ccc) |
| Lecture 19-20 | Feb. 25-27 | Optimality Conditions for Constrained Problems |
(
not done is in red
Special case of Lagrange multiplier theorem for equality constraints
proved using the Implicit Function Theorem and compared to the simplex
method; Extension of Fermat (Geometric optimality conditions); tangent cones; Farkas Lemma |
| Lecture 18 | Feb. 23 | Optimality Conditions for Constrained Problems | Extension of Fermat (Geometric optimality conditions); tangent cones; Farkas Lemma |
| Lecture 16-17 | Feb. 9-11 | Numerical Methods for large-scale nonlinear optimization |
Conjugate Gradient Methods (done in some detail
with an assignment); Inexact Newton Methods (outline);
Derivative Free Methods (outline); Nonlinear Least Squares Problems
( not done is in red Nonlinear Equations (Inexact Newton Methods, Homotopy Methods) |
| Lecture 15 | Feb. 6 | Numerical Methods for large-scale nonlinear optimization | Conjugate Gradient Methods (outline); Inexact Newton Methods (outline); Derivative Free Methods (outline); Nonlinear Least Squares Problems |
| Lecture 14 | Feb. 4 | Trust Region Algorithms |
( not done is in red
the hard case for the TRS.
Solving linear systems of equations/Cholesky factorization paradox files: runAs.m; Asolve.m; runeps.m; Aoneseps.m not done till here). |
| Lecture 13 | Feb. 2 | Trust Region Algorithms | (not done is in red Details on the trust region subproblem algorithm ( a survey paper) including modification of the root finding equation and outline of the hard case. |
| Lecture 10-12 | Jan. 26-30 | Unconstrained Minimization-Rn |
proof of quadratic convergence for Newton's method line search algorithms and trust region methods |
| Lecture 9 | Jan. 23 | Unconstrained Minimization-Rn |
line search algorithms convergence using: sufficient decrease; and
sufficiently long steps
rates of convergence start of quadratic convergence proof for Newton's method |
| Lecture 8 | Jan. 21 | Unconstrained Minimization-Rn |
(not covered is in red
derivation of quasi-Newton updates, though there was an assignment on
finding derivatives wrt matrix variables)
and differentiation of a function
of a matrix variable;
sufficient decrease in line search methods (Wolfe condition I) |
| Lecture 7 | Jan. 19 |
Unconstrained Minimization-Rn
( Supplementary NOTES); |
derivation of Newton's method using quadratic model
quasi-Newton methods and the secant equation |
| Lecture 6 | Jan. 16 |
Unconstrained Minimization-Rn
( Supplementary NOTES); |
deflected/scaled steepest descent (SD); best scaling SD derivation of
Newton's method.
MATLAB example - Newton's method scale free behaviour of Newton's method |
| Lecture 5 | Jan. 14 |
Unconstrained Minimization-Rn
( Supplementary NOTES); |
Definitions: condition numbers; ill-conditioned problems;
convex sets/functions; and,
characterizations of convex functions: using first/second derivatives
and using epigraph; cone of convex functions
convex functions: stationary points; global minimima; convex level sets Overview of algorithms: (i) line search; (ii) trust region Method of Steepest Descent with derivation using Lagrange multipliers |
| Lecture 4 | Jan. 12 |
Unconstrained Minimization-Rn
( Supplementary NOTES); ( Complete Supplementary course notes I; Complete Supplementary course notes II ) |
WWW links to
NEOS,
(
WWW Form for unconstr NMTR);
LP and NLP FAQs
Application: Prove (outline only) the arithmetic-geometric mean (AGM) inequality using unconstrained minimization summary: first and second order necessary/sufficient optimality conditions |
| Lecture 3 | Jan. 9 | Unconstrained Minimization - Rn (WIKI!!) |
directional derivative, curvature, linear model, direction of steepest
descent
first and second order necessary optimality conditions second order sufficient optimality conditions convexity and global minimima, characterizations of convex functions optimality conditions and attainment for a quadratic function on Rn Application: Prove the arithmetic-geometric mean (AGM) inequality using unconstrained minimization |
| Lecture 2 | Jan. 7 | Unconstrained Minimization |
Fundamentals Unconstrained Opt. cont... (Chapter 2 - complete chapter
except for R-Rates of Convergence) pgs 11-17,19-24,26-29
Recognizing Solutions Definitions: Frechet derivative, gradient, Hessian, Taylor's Theorem, order notation (big and little O) Example of data fitting using nonlinear least squares Definitions of local/global/strict minima. |
| Lecture 1 | Jan. 5 | Introduction to Continuous Optimization |
Introduction (Chapter 1, pgs 1-4,6,8)
Examples of applications. Mathematical Formulation: example, level sets Dichotomies: continuous and discrete optimization; local and global optima; linear and nonlinear optimization; convex and nonconvex optimization (new paradigm); stochastic and deterministic optimization Definitions: convexity (sets, functions) |