Course - Optimisation for Data Science HT25

This course analyses optimisation methods suitable for large-scale data science problems, mainly by deriving results on the rate of convergence under increasing assumptions (smooth, convex, strongly convex) on the objective functions.

The course begins with some optimisation terminology and then covers gradient descent and the proximal method, which can be used to apply steepest descent techniques to regularised problems. Then it covers acceleration techniques such as the heavy ball method, and then moves onto stochastic gradient descent and accelerated techniques in that context. Finally, it covers coordinate descent methods.

• Course Webpage
• Lecture Notes
• From the previous year:
  • Lecture Notes
  • 1, Scope and examples
  • 2, Terminology and prerequisites
  • 3, Method of steepest descent
  • 4, The proximal method
  • 5, Acceleration of gradient methods
  • 6, Stochastic gradient descent
  • 7, Reducing the noise floor in SGD
  • 8, Coordinate descent
  • 9, Practical coordinate descent
  • 10, Outlook (non-examinable)
• Other courses this term: Courses HT25^U

Notes

• Notes - Optimisation for Data Science HT25, Overview of results^U ⭐️

• Notes - Optimisation for Data Science HT25, Motivation and examples^U

• Notes - Optimisation for Data Science HT25, Terminology^U

• Notes - Optimisation for Data Science HT25, Convexity^U

• Notes - Optimisation for Data Science HT25, Subgradients^U

• Notes - Optimisation for Data Science HT25, Steepest descent^U

• Notes - Optimisation for Data Science HT25, Steepest descent with inexact line search^U

• Notes - Optimisation for Data Science HT25, Proximal methods^U

• Notes - Optimisation for Data Science HT25, Accelerated methods^U

• Notes - Optimisation for Data Science HT25, Nesterov’s accelerated gradient method^U

• Notes - Optimisation for Data Science HT25, Stochastic gradient descent^U

• Notes - Optimisation for Data Science HT25, Stochastic variance reduction methods^U

• Notes - Optimisation for Data Science HT25, Coordinate descent^U

• Notes - Optimisation for Data Science HT25, Misc^U

• redacted^?

Problem Sheets

• redacted^?
• Sheet 1, partial answers
• Sheet 2, partial answers
• Sheet 3, partial answers
• Sheet 4, partial answers

To-Do List

Relevant reading

• “Optimisation for Data Science” course at ETH Zurich
• “Convex Optimisation” at CMU
• Convex Optimisation, Boyd and Vandenberghe
• Numerical Optimisation, Nocedal and Wright
• Deep Learning, Goodfellow
• Optimisation for Machine Learning, Sra, Nowozin and Wright