Skip to content

ECE490

ECE490 (Introduction to Optimization) is a 3 (undergrad) or 4 (graduate) hour course that satisfies the Technical Electives requirement for ECE majors. It is offered in both the fall and spring semesters.

Content Covered

  • Linear Algebra (MATH257) review
  • Eigenvalues/Eigenvectors
  • Gradients/Hessians
  • Detecting global/local minima of vector-valued functions
  • Properties of matrices (semidefiniteness, matrix norms)
  • Convex Sets, Convex Functions, strict/strong convexity
  • Newton's Method descent
  • Gradient Descent, parameter-tuning (Armijo's rule)
  • Projection of vectors to (convex) sets
  • Lipschitz Inequality
  • Constrained Optimization (inequality/equality)
  • Karush-Kuhn-Tucker (KKT) Conditions
  • Lagrangians
  • Duality, Strong Duality
  • Subgradients, Subdifferential

The class begins with an introduction to linear programming and its various applications within engineering and outside, such as in finance and economics. Techniques for solving them are discussed, and algorithms are implemented in homework problems. The class quickly adopts a more rigorous approach and discusses unconstrained optimization of arbitrary functions, along with necessary and sufficient conditions for optimality. Similar analysis is then repeated for constrained optimization and well-known results such as the KKT conditions are derived. The last few weeks are spent on special topics such as compressed sensing, simulated annealing, and so forth.

Prerequisites

ECE220 and MATH257 are listed as the official prerequisites. However, no specific concepts from ECE220 are discussed in the class and it suffices to have adequate programming skills, at the level of implementing an algorithm in C or Matlab. Recently, the programming component of the course has been removed, so ECE220 is hardly necessary for the course, as it is now completely exam/quiz-based. Likewise, with MATH257, the class seldom relies on gory details beyond elementary matrix algebra, using only concepts like row reduction, determinants, and so forth. The only real prerequisite is "mathematical maturity," for which there are many pathways to arrive at.

When to Take It

The prerequisites (or the lack thereof) do not imply you should take the course early on. In fact, it is best to take it whenever you have an accurate picture of your future goals and interests. Optimization in itself is interesting, but it defeats the purpose of the course if you take it without any knowledge of specific applications. For example, you might find a use for optimization in research projects you do outside of class. It is therefore recommended that you take it in your junior or senior year, and perhaps even as a first year graduate student.

Course Structure

As of Spring 2024, the programming component of the course has been removed.

For the 3-credit-hour section, the grade is split into

  • 5% participation
  • 50% quizzes (6 quizzes, 1 quiz is dropped)
  • 45% exams (3 exams)

For the 4-credit-hour section, the grade is split into

  • 4% participation
  • 40% quizzes (6 quizzes, same drop policy)
  • 36% exams (3 exams)
  • 20% final project

Quizzes are usually bi-weekly, but there sometimes will be a period of 1 week or 3 weeks before quizzes, so make sure to be aware of the quiz schedule (posted at the start of the semester). Sometimes an exam will take place between quiz periods, so it's common to have very crunched exam weeks. Homeworks are posted a few days before the quizzes. Homeworks are always posted with solutions, and so they are not graded. They are meant to be used as study tools and practice for the quizzes. The final project is a report on two optimization algorithms.

Instructors

The course has been taught by various professors in different semesters. In Spring 2023, the course was taught by Professor Venugopal Veeravalli. In Fall 2023, the course was taught by Professor Rayadurgam Srikant, who will also teach the course in Fall 2024. In Spring 2024, the course was taught by Professor Bin Hu.

Course Tips

  • Don't underestimate the low workload... the questions are very conceptually difficult even for those skilled in proofs. Studying for quizzes/exams take a decent chunk of time when they come. Sometimes, homeworks and solutions are posted only a few days before the quizzes, giving rather little time to study, so be on top of the homeworks in preparation for the quizzes.
  • Brush up on your ability to do proofs by doing the homeworks and analyzing how the solutions carry out the proofs. This will be useful for the quizzes, as proof questions are very common.
  • Don't overthink questions. You hear this exam tip all the time, but it's especially important here. 20 minutes for the quiz period is not a lot of time, and silly mistakes and/or overthinking questions can really cost you. Usually, Bin Hu tries to award as much partial credit as he can, so make sure you write something plausible for each question. Quizzes and exams are the ONLY portion of the grade (aside from participation and the final project), so mistakes can cost you.
  • Review with Veeravalli's past notes -- they are hard to read, but very useful and cover the majority of the content. Focus mostly on studying the concepts rather than the proofs behind the theorems.

Life After

The class is targeted towards those interested in the theoretical aspects of signal processing, communications, and control systems. ECE310, ECE461, ECE449 and ECE486 are all good courses to take concurrently with ECE490, though not necessarily all of them simultaneously. Mathematics courses such as MATH 444/447 (Real Analysis) and MATH448 (Complex Analysis) should be taken before, during, or after the course. Since many applications reside outside of ECE, economics and finance classes such as ECON465 may broaden your understanding of the applications. ECE491 (Numerical Analysis) is a must for those interested in efficient implementation of many of the methods discussed in the class. This is a theoretical course and does not lead to immediate job placement; however, since it is of fundamental importance, it will equip you with the skills necessary for any career---be it a quant or a power systems engineer.

Infamous Topics

  • Newton's Method and Gradient Descent - this is an iterative method of solving optimization problems for local/global minima. A particular challenging part can be parameter-tuning for steepest descent methods.
  • Lagrange Multipliers - this is a method of solving optimization under equality constraints, which can result in a lot of guessing-and-checking for systems of equations, so make sure you have this optimization strategy locked down.
  • Duality - this is a method of solving optimization under several inequality constraints, which can result in solving for multiple systems of equations.
  • KKT Conditions - another method of solving inequality constraints when equality constraints are toggled between active and inactive.

Resources