Numerical Analysis I

AMSC/CMSC 666, Fall 2022


Course Information

Lecture TuTh 2-3:15pm Phys 1219
Note special place: on Fri 10/14 and 10/21 Math Bldg. Rm. 3110
Note special place: on Fri 11/11                  Math Bldg. Rm. 3110
InstructorProfessor Eitan Tadmor
Contacttel.: x5-0648   e-mail:
Office HoursBy appointment (e-mail: )
Teaching Assistant Michael Rozowski (e-mail: )
Final Dec. 17, 10:30--12:30pm; place TBA
Grading50% Homework; 50% Final
Q&A

 (*) Additional notes on class format will be posted here.  

Course Description

  1. Approximation Theory

    1. Interpolation I. Algebraic interpolants. uniform approximation. Lebeasgue constant.
      2. • Lecture notes:
        [Interpolation: Lagrange and Newton forms]
        [Error bounds for equi-spaced and Chebyshev nodes]
        [The min-max problem and Lebesgue constant]
      • Additional reading:
        ► M.J.D Powell [On Lebesgue constant for Chebyshev interpolants]
        ►[The sin(x) subroutine in MATLAB]
      • Assignment [ #1 ] with [ solutions ]
    2. Interpolation II. Interpolation with equi-spaced modes
      3. • Lecture notes: [ Synthetic calculus with equi-spaced nodes ]
    3. Interpolation III. Interpolation with derivatives. Splines.
      4. • Lecture notes: [ Hermite and spline interpolants ]
      • Additional reading ► [Splines in Industry ] by T. Sauer
      • Assignment [ #2 ] ... with [ solutions ]
    4. Least-squares I. Fourier expansions. Orthogonal polynomials
      5. • Lecture notes: [ The least-squares problem ]
      • Lecture notes. [ Orthogonal polynomials ] and [ Least Squares as truncated Fourier expansion ]
      • Assignment [ #3 ] ... with [ solutions ]
    5. Least squares II -- the finite-dimensional case. QR, SVD, PCA
      6. • Lecture notes: [ Greedy aspects of Fourier expansion ]
      • Assignment [ #4 ] with [ solutions ]

  2. Numerical Solution of ODEs: Initial-Value Problems

    1. Preliminaries. Stability of systems of ODEs.
      2. • Lecture notes: [ Stability of systems of ODEs ]
    2. Examples of basic numerical methods: Euler's method, Leap-Frog, Milne.
      3. • Lecture notes: [ Examples of basic numerical methods ]
      • Additional notes:
      ► [On Duhanel's principle]
      ► Why implicit Euler? [On gradient flows]

    3. Consistency and stability imply convergence. Predictor-corrector methods
      4. • Lecture notes: Stability and Convergence [ Accuracy, stability and convergence ]
      • Assignment [ #5 ] ... with [ solutions ]
      • Lecture notes: Predictor-Corrector methods [ ABM schemes and adaptivity ]
      • Assignment [ #6 ] ... with [ solutions ]
    4. Runge-Kutta methods
      5. • Lecture notes: [ Runge-Kutta methods ]
    5. Stiff systems and absolute stability
      6. • Lecture notes: [ Stiff equations and absolute stability ]
      • Assignment [ #7 ] ... with [ solutions ]
      • Additional reading:
      ► SSP methods for linear problems:[coercive problems] and [beyond coercivity];
      and on SSP methods for [nonlinear problems]

  3. Iterative Methods for Solving Systems of Linear Equations

    1. Stationary methods: Jacobi, Gauss-Seidel, SOR, ...
      2. • Lecture notes: [ Jacobi, Gauss-Seidel and SOR iterations]
      • Additional reading: ► The original work of [D. Young Thesis (1950)]
      • Assignment [ #8 ] ... with [ solutions ]
    2. Energy functionals and gradient methods. Steepest descent. Conjugate gradient
      3. • Lecture notes: [ gradient-based methods]
      • Additional reading: ► Another look at (conjugate) gradient methods made easy [pdf file]
      • Assignment [ #9 ] ... with [ solutions ]
    3. Acceleration methods.
      4. • Lecture notes: [ Acceleration methods]
      • Additional reading: ► Y. Saad [Iterative Methods for Sparse Linear Systems]

  4. Numerical Optimization

    1. General overview. Fixed point iterations. Newton's method. Homotopy.
      • Finding roots of nonlinear equations: scalar and systems. Homotopy
      • Lecture notes: [scalar equations] and [systems ]
      • Assignment [ #10 ] ... with answers [ solutions ]
    2. Computation of minimizers. Gradient flows. Line search methods. Trust regions
      3. • Lecture notes: [Line search, trust regions and Wolfe conditions]
      • Additional reading: ► J. Nocedal S. Wright [Line search methods] and [Trust-region methods]

    3. Newton and quasi-Newton: SR1 and BFGS methods
      4. • Lecture notes:
      • Assignment [ #11 ] ... with [ solutions ]




• Additional reading

► Leonhard Euler [pdf file] (by W. Gautschi)

References

GENERAL TEXTBOOKS

W. Gautschi, NUMERICAL ANALYSIS, Birkhauser, 2012

K. Atkinson, An INTRODUCTION to NUMERICAL ANALYSIS, Wiley, 1987

S. Conte & C. deBoor, ELEMENTARY NUMERICAL ANALYSIS, McGraw-Hill
User friendly; Shows how 'it' works; Proofs, exercises and notes

G. Dahlquist & A. Bjorck, NUMERICAL METHODS, Prentice-Hall,
User friendly; Shows how 'it' works; Exercises

A. Ralston & P. Rabinowitz, FIRST COURSE in NUMERICAL ANALYSIS, 2nd ed., McGraw-hill,
Detailed; Scholarly written; Comprehensive; Proofs exercises and notes

J. Stoer & R. Bulrisch, INTRODUCTION TO NUMERICAL ANALYSIS, 2nd ed., Springer
detailed account on approximation, linear solvers & eigen-solvers, ODE solvers,..

APPROXIMATION THEORY

E. W. Cheney, INTRODUCTION TO APPROXIMATION THEORY
Classical

P. Davis, INTERPOLATION & APPROXIMATION, Dover
Very readable

T. Rivlin, AN INTRODUCTION to the APPROXIMATION of FUNCTIONS
Classical

R. DeVore & G. Lorentz, CONSTRUCTIVE APPROXIMATION, Springer
A detailed account from classical theory to the modern theory; everything; Proofs exercises and notes

NUMERICAL INTEGRATION

F. Davis & P. Rabinowitz, NUMERICAL INTEGRATION,
Everything...

NUMERICAL SOLUTION Of INITIAL-VALUE PROBLEMS

E. Hairer, S.P. Norsett and G. Wanner, SOLVING ODEs I: NONSTIFF PROBLEMS, Springer-Verlag, Berlin. 1991, (2nd ed)
Everything - the modern version

A. Iserles, A FIRST COURSE in the NUMERICAL ANALYSIS of DEs, Cambridge Texts

W. Gear, NUMERICAAL INITIAL VALUE PROBLEMS in ODEs, 1971
The classical reference on theory and applications

Lambert, COMPUTATIONAL METHODS for ODEs, 1991
Detailed discussion of ideas and practical implementation

Shampine and Gordon, COMPUTER SOLUTION of ODES, 1975
Adams methods and practial implementation of ODE "black box" solvers

Butcher, NUMERICAL ANALYSIS of ODEs, 1987
Comprehensive discussion on Runge-Kutta methods

(mainly) ITERATIVE SOLUTION OF LINEAR SYSTEMS

A. Householder, THE THEORY OF MATRICES IN NUMERICAL ANALYSIS
The theoretical part by one of the grand masters; Outdated in some aspects

G. H. Golub & Van Loan, MATRIX COMPUTATIONS,
The basic modern reference

Y. Saad, ITERATIVE METHODS for SPARSE LINEAR SYSTEMS,
PWS Publishing, 1996. (Available on line at http://www-users.cs.umn.edu/~saad/books.html)

R. Varga, MATRIX ITERATIVE ANALYSIS,
Classical reference for the theory of iterations

James Demmel, APPLIED NUMERICAL LINEAR ALGEBRA,
SIAM, 1997

Kelley, C. T., ITERATIVE METHODS for LINEAR and NONLINEAR EQUATIONS,
SIAM 1995

NUMERICAL OPTMIZATION

J. Nocedal, S. Wright, NUMERICAL OPTIMIZATION
Springer, 1999

T. Kelly, ITERATIVE METHODS for OPTIMIZATION
SIAM

EIGENSOLVERS (mainly)

B. Parlett, THE SYMMETRIC EIGENVALUE PROBLEM
Prentice-Hall, 1980

James Demmel, APPLIED NUMERICAL LINEAR ALGEBRA,
SIAM, 1997

J. H. Wilkinson The ALGEBRAIC EIGENVALUE PROBLEM, 1965
The classical reference

Eitan Tadmor