Numerical Analysis I

AMSC/CMSC 666, Fall 2021


Course Information

Lecture TuTh 2-3:15pm Room 4122 CSIC Bldg. #406
InstructorProfessor Eitan Tadmor
Contacttel.: x5-0648   e-mail:
Office HoursBy appointment (e-mail: )
Teaching Assistant Anna Sotnikova (e-mail: )
Final Take home final will be handed in classroom
4122 CSIC Bldg. #406 on Thursday Dec. 16 2pm
and will be due, in person, Friday Dec. 17 by 2pm
Grading50% Homework; 50% Final
Q&AInstruction in Fall 2021

 (*) Additional notes on online class format are found here.  

Course Description

  1. Approximation Theory

    1. General overview. Least-squares vs. the uniform norm
      2. • Lecture notes: [ The least-squares problem ]
    2. Orthogonal polynomials
      3. • Lecture notes: [ Orthogonal polynomials ]
    3. Least squares I. Fourier expansions.
      4. • Lecture notes: [ Least Squares as truncated Fourier expansion ]
      • Assignment [ #1 ] ... with [ answers]
    4. Least squares II -- the finite-dimensional case. QR, SVD, PCA (LS rank approximations)
      5. • Lecture notes: [ SVD ] (with additional notes on QR factorization )
      • Additional reading:
      ► Gilbert Strang [on SVD] (and also here)

      ► Lijie Cao [SVD applied to digital image processing]
      • Assignment [ #2 ] with answers [ pdf file]
    5. Gauss quadrature
      6. • Lecture notes: [Gauss quadrature]

  2. Numerical Solution of ODEs: Initial-Value Problems

    1. Preliminaries. Stability of systems of ODEs.
      2. • Lecture notes: [ Stability of Systems of ODEs ]
      • Assignment [ #3 ] with answers [ pdf file]
    2. Examples of basic numerical methods: Euler's method, Leap-Frog, Milne.
      3. • Lecture notes: [ Examples of basic numerical methods ]
      • Additional notes:
      ► [On Duhanel's principle]

      ► Why implicit Euler? [On gradient flows]
      • Assignment [ #4 ] with answers [ pdf file]
    3. Consistency and stability imply convergence
      4. • Assignment [ #5 ] ... with answers [ pdf file]
      • Assignment [ #6 ]
    4. Runge-Kutta methods
      • Local time stepping and error estimates. RK4 and RKF5.
      • Stability and convergence of Runge-Kutta methods
      • Lecture notes: [ Runge-Kutta methods ]
    5. Stiff systems and absolute stability
      6. • Lecture notes: [ Stiff equations and absolute stability ]
      • Assignment [ #7 ] ... with answers [ pdf file]
    6. Strong-Stability Preserving (SSP) methods:
      7. • Additional reading:
      ► SSP methods for linear problems:[coercive problems] and [beyond coercivity];
      and on SSP methods for [nonlinear problems]

  3. Iterative Methods for Solving Systems of Linear Equations

    1. Stationary methods: Jacobi, Gauss-Seidel, SOR, ...
      2. • Lecture notes: [ Jacobi, Gauss-Seidel and SOR iterations]
      • Additional reading:
      ► The original work of [D. Young Thesis (1950)]
      ► On the Fourier approach to the SOR: [Garabedian (1956)] [LeVeque & Trefethen (1988)]
      • Assignment [ #8 ] ... with answers [ pdf file]
    2. Energy functionals and gradient methods

      • Steepest descent. Conjugate gradient.
      • Lecture notes: [ gradient-based methods]
      • Assignment [ #9 ] ... with answers [ pdf file]
    3. Acceleration method.
      4. • Lecture notes: [ Acceleration methods]
      • Additional reading:
      ► Y. Saad [Iterative Methods for Sparse Linear Systems]
      ► Recent review of Krylov-based methods [pdf file]

  4. Numerical Optimization

    1. General overview. Fixed point iterations, low-order and Newton's methods
      • Finding roots of nonlinear equations: scalar and systems. Homotopy
      • Lecture notes: [scalar equations] and [ systems ]
      • Assignment [ #10 ] ... with answers [ pdf file]
    2. Computation of minimizers. Numerical Optimization
      • Gradient flows
      • Line search methods. Trust regions and Wolfe conditions
      • Lecture notes: [Line search, trust regions and Wolfe conditions]
      • Additional reading:
      ► J. Nocedal S. Wright [Line search methods] and [Trust-region methods]

    3. Newton and quasi-Newton: SR1 and BFGS methods
      4. • Lecture notes: [ Gradient, CG and quasi-Newton methods ]
      • Assignment [ #11 ]

  5. Eigen-solvers

    1. The power method
      • The symmetric case: Rayleigh quotient. min-max
      • Acceleartion. Inverse power method. Deflation

    2. Similarity-based methods
      • Rotations and refelctions
      • Jacobi method
      • Reduction to Hessenberg and tri-diagonal forms
      • QR with a shift
      • Additional reading:
      ► P. Deift, T. Nanda & C. Tomei [Toda flows and the symmetric e.v. problem]

    3. Divide and Conquer method
      4. • Additional reading:
      ► J.J.M. Cuppen [Divide and conquer to tridiagonal eigen-problem]
      and [O(N2) method for eigenvectors of Divide and conquer method]

    • Additional reading

    ► Nick Trefethen [ Numerical Analysis]

    References

    GENERAL TEXTBOOKS

    W. Gautschi, NUMERICAL ANALYSIS, Birkhauser, 2012

    K. Atkinson, An INTRODUCTION to NUMERICAL ANALYSIS, Wiley, 1987

    S. Conte & C. deBoor, ELEMENTARY NUMERICAL ANALYSIS, McGraw-Hill
    User friendly; Shows how 'it' works; Proofs, exercises and notes

    G. Dahlquist & A. Bjorck, NUMERICAL METHODS, Prentice-Hall,
    User friendly; Shows how 'it' works; Exercises

    A. Ralston & P. Rabinowitz, FIRST COURSE in NUMERICAL ANALYSIS, 2nd ed., McGraw-hill,
    Detailed; Scholarly written; Comprehensive; Proofs exercises and notes

    J. Stoer & R. Bulrisch, INTRODUCTION TO NUMERICAL ANALYSIS, 2nd ed., Springer
    detailed account on approximation, linear solvers & eigen-solvers, ODE solvers,..

    APPROXIMATION THEORY

    E. W. Cheney, INTRODUCTION TO APPROXIMATION THEORY
    Classical

    P. Davis, INTERPOLATION & APPROXIMATION, Dover
    Very readable

    T. Rivlin, AN INTRODUCTION to the APPROXIMATION of FUNCTIONS
    Classical

    R. DeVore & G. Lorentz, CONSTRUCTIVE APPROXIMATION, Springer
    A detailed account from classical theory to the modern theory; everything; Proofs exercises and notes

    NUMERICAL INTEGRATION

    F. Davis & P. Rabinowitz, NUMERICAL INTEGRATION,
    Everything...

    NUMERICAL SOLUTION Of INITIAL-VALUE PROBLEMS

    E. Hairer, S.P. Norsett and G. Wanner, SOLVING ODEs I: NONSTIFF PROBLEMS, Springer-Verlag, Berlin. 1991, (2nd ed)
    Everything - the modern version

    A. Iserles, A FIRST COURSE in the NUMERICAL ANALYSIS of DEs, Cambridge Texts

    W. Gear, NUMERICAAL INITIAL VALUE PROBLEMS in ODEs, 1971
    The classical reference on theory and applications

    Lambert, COMPUTATIONAL METHODS for ODEs, 1991
    Detailed discussion of ideas and practical implementation

    Shampine and Gordon, COMPUTER SOLUTION of ODES, 1975
    Adams methods and practial implementation of ODE "black box" solvers

    Butcher, NUMERICAL ANALYSIS of ODEs, 1987
    Comprehensive discussion on Runge-Kutta methods

    (mainly) ITERATIVE SOLUTION OF LINEAR SYSTEMS

    A. Householder, THE THEORY OF MATRICES IN NUMERICAL ANALYSIS
    The theoretical part by one of the grand masters; Outdated in some aspects

    G. H. Golub & Van Loan, MATRIX COMPUTATIONS,
    The basic modern reference

    Y. Saad, ITERATIVE METHODS for SPARSE LINEAR SYSTEMS,
    PWS Publishing, 1996. (Available on line at http://www-users.cs.umn.edu/~saad/books.html)

    R. Varga, MATRIX ITERATIVE ANALYSIS,
    Classical reference for the theory of iterations

    James Demmel, APPLIED NUMERICAL LINEAR ALGEBRA,
    SIAM, 1997

    Kelley, C. T., ITERATIVE METHODS for LINEAR and NONLINEAR EQUATIONS,
    SIAM 1995

    NUMERICAL OPTMIZATION

    J. Nocedal, S. Wright, NUMERICAL OPTIMIZATION
    Springer, 1999

    T. Kelly, ITERATIVE METHODS for OPTIMIZATION
    SIAM

    EIGENSOLVERS (mainly)

    B. Parlett, THE SYMMETRIC EIGENVALUE PROBLEM
    Prentice-Hall, 1980

    James Demmel, APPLIED NUMERICAL LINEAR ALGEBRA,
    SIAM, 1997

    J. H. Wilkinson The ALGEBRAIC EIGENVALUE PROBLEM, 1965
    The classical reference

    Eitan Tadmor