Nonlinear Optimization Notes

A bad habit of mine is stuffing all my knowledge on a scrap paper only to completely forget them after an exam. So in the interests of retaining knowledge, I’ve decided to make a “write-up” for my classes and create digital notes for my own (and perhaps others’) use.

Here are my notes on Comp Sci 726 at UW-Madison, which is a graduate level course on non-linear optimization.

Notes

  1. Preliminaries
  2. Basic Descent Analysis
  3. Nonsmooth Descent Analysis
  4. Projected Gradient Descent
  5. Stochastic Gradient Descent
  6. 2nd Order Methods and Acceleration