Supplementary Material, CO466/666 Winter'09

This webpage contains files used during the lectures. Additional material/links are also included. Be aware that this webpage is continually being polished/changed.

Lectures Date Subjects Covered Lecture Supplementary Material
Lecture 22 Mar. 4 KKT optimality conditions and CQs Derive the KKT opt conditions using the weakest constraint qualification: T(F,x)=L(x), i.e. the tangent and linearizing cones are equal.
Then the KKT conditions comes from the optimality conditions: gradient of f(x) is in the polar of T(F,x).
(Follows, since the polar of L(x) is the cone generated by the gradients, by Farkas Lemma.)
Lecture 21 Mar. 2 Hyperplane separation theorem and applications Basic Separation Theorem (proof was on assignment 4);
Application to proving the
Lemma: K=K^++ iff K is a closed convex cone (ccc)
Lecture 19-20 Feb. 25-27 Optimality Conditions for Constrained Problems ( not done is in red Special case of Lagrange multiplier theorem for equality constraints proved using the Implicit Function Theorem and compared to the simplex method;
Extension of Fermat (Geometric optimality conditions); tangent cones; Farkas Lemma
Lecture 18 Feb. 23 Optimality Conditions for Constrained Problems Extension of Fermat (Geometric optimality conditions); tangent cones; Farkas Lemma
Lecture 16-17 Feb. 9-11 Numerical Methods for large-scale nonlinear optimization Conjugate Gradient Methods (done in some detail with an assignment); Inexact Newton Methods (outline); Derivative Free Methods (outline); Nonlinear Least Squares Problems
( not done is in red Nonlinear Equations (Inexact Newton Methods, Homotopy Methods)
Lecture 15 Feb. 6 Numerical Methods for large-scale nonlinear optimization Conjugate Gradient Methods (outline); Inexact Newton Methods (outline); Derivative Free Methods (outline); Nonlinear Least Squares Problems
Lecture 14 Feb. 4 Trust Region Algorithms ( not done is in red the hard case for the TRS.
Solving linear systems of equations/Cholesky factorization
paradox files: runAs.m; Asolve.m; runeps.m; Aoneseps.m not done till here).
Lecture 13 Feb. 2 Trust Region Algorithms (not done is in red Details on the trust region subproblem algorithm ( a survey paper) including modification of the root finding equation and outline of the hard case.
Lecture 10-12 Jan. 26-30 Unconstrained Minimization-Rn proof of quadratic convergence for Newton's method
line search algorithms and trust region methods
Lecture 9 Jan. 23 Unconstrained Minimization-Rn line search algorithms convergence using: sufficient decrease; and sufficiently long steps
rates of convergence
start of quadratic convergence proof for Newton's method
Lecture 8 Jan. 21 Unconstrained Minimization-Rn (not covered is in red derivation of quasi-Newton updates, though there was an assignment on finding derivatives wrt matrix variables) and differentiation of a function of a matrix variable;
sufficient decrease in line search methods (Wolfe condition I)
Lecture 7 Jan. 19 Unconstrained Minimization-Rn
( Supplementary NOTES);
derivation of Newton's method using quadratic model
quasi-Newton methods and the secant equation
Lecture 6 Jan. 16 Unconstrained Minimization-Rn
( Supplementary NOTES);
deflected/scaled steepest descent (SD); best scaling SD derivation of Newton's method.
MATLAB example - Newton's method
scale free behaviour of Newton's method
Lecture 5 Jan. 14 Unconstrained Minimization-Rn
( Supplementary NOTES);
Definitions: condition numbers; ill-conditioned problems; convex sets/functions; and, characterizations of convex functions: using first/second derivatives and using epigraph; cone of convex functions
convex functions: stationary points; global minimima; convex level sets
Overview of algorithms: (i) line search; (ii) trust region
Method of Steepest Descent with derivation using Lagrange multipliers
Lecture 4 Jan. 12 Unconstrained Minimization-Rn
( Supplementary NOTES);
( Complete Supplementary course notes I; Complete Supplementary course notes II )
WWW links to NEOS, ( WWW Form for unconstr NMTR); LP and NLP FAQs
Application: Prove (outline only) the arithmetic-geometric mean (AGM) inequality using unconstrained minimization
summary: first and second order necessary/sufficient optimality conditions
Lecture 3 Jan. 9 Unconstrained Minimization - Rn (WIKI!!) directional derivative, curvature, linear model, direction of steepest descent
first and second order necessary optimality conditions
second order sufficient optimality conditions
convexity and global minimima, characterizations of convex functions
optimality conditions and attainment for a quadratic function on Rn
Application: Prove the arithmetic-geometric mean (AGM) inequality using unconstrained minimization
Lecture 2 Jan. 7 Unconstrained Minimization Fundamentals Unconstrained Opt. cont... (Chapter 2 - complete chapter except for R-Rates of Convergence) pgs 11-17,19-24,26-29
Recognizing Solutions
Definitions: Frechet derivative, gradient, Hessian, Taylor's Theorem, order notation (big and little O)
Example of data fitting using nonlinear least squares
Definitions of local/global/strict minima.
Lecture 1 Jan. 5 Introduction to Continuous Optimization Introduction (Chapter 1, pgs 1-4,6,8)
Examples of applications.
Mathematical Formulation: example, level sets
Dichotomies: continuous and discrete optimization; local and global optima; linear and nonlinear optimization; convex and nonconvex optimization (new paradigm); stochastic and deterministic optimization
Definitions: convexity (sets, functions)