EL-GY 6123 Image and Video Processing (Spring 2018)

Course Description:

This course introduces fundamentals of image and video processing, including color image capture and representation; contrast enhancement; spatial domain filtering; two-dimensional (2D) Fourier transform and frequency domain interpretation of linear convolution; image sampling and resizing; multi-resolution image representation using pyramid and wavelet transforms; feature point detection and global alignment between images based on feature correspondence; geometric transformation, image registration; video motion characterization and estimation; video stabilization and panoramic view generation; image and video segmentation; selected advanced image processing techniques; basic compression techniques and standards (JPEG image compression standard; wavelet transform and JPEG2000 standard; video compression using adaptive spatial and temporal prediction; video coding standards (MPEGx/H26x); Stereo and multi- view image and video processing (depth from disparity, disparity estimation, video synthesis, compression). Students will learn to implement selected algorithms in Python. A term project will be required.

Prerequisites:
Graduate status. EL-GY 6113 and EL-GY 6303 preferred but not required. Undergraduate students must have completed EE-UY 3054 Signals and systems and EE-UY 2233 Probability.

Instructor:
Professor Yao Wang, MTC2 Room 9.122, (646)-997-3469, Email: yaowang at nyu dot edu. Homepage

Teaching Assistants:
Chenge Li (MTC2 Room 9.123, cl2840 at nyu dot edu) and Amirhossein Khalilian-Gourtani (MTC2 Room 9.130B, akg404 at nyu dot edu)

Course Schedule:
Friday 12:25 AM-2:55PM, Room RH215.

Office Hour:
Yao Wang: Wed. 4- 5:30PM and Fri. 10:30-12:00 or appointment by email.
TAs: Chenge Li: Tue. 4:00-5:00 PM and Fri. 3:00-4:00 PM or appointment by email
Amirhossein Khalilian-Gourtani: Mon: 10:00 am – 11:30 am and Thur: 4:30 pm – 6:30 pm or by appointment

Text Book/References:

  1. Richard Szeliski, Computer Vision: Algorithms and Applications. (Available online:”Link“) (Cover most of the material, except sparsity-based image processing and image and video coding)
  2. (Optional) Y. Wang, J. Ostermann, and Y.Q.Zhang, Video Processing and Communications. Prentice Hall, 2002. “Link” (Reference for image and video coding, motion estimation, and stereo)
  3. (Optional) R. C. Gonzalez and R. E. Woods, Digital Image Processing, Prentice Hall, (3rd Edition) 2008. ISBN number 9780131687288. “Link” (Good reference for basic image processing, wavelet transforms and image coding).

Grading Policy:
Exam: 40%, Final Project: 30%, Programming assignments: 20%, Written assignments: 10%.

Homework Policy:
Written HW will be assigned after each lecture and due at the beginning of the following lecture time. Programming assignment will be due as posted. Late submission of written assignment and programming assignment will not be accepted unless under extraordinary circumstances and must be approved in advance by the instructor. Students can work in teams, but you must submit you own solutions. For programming assignment, please include your code (with documentation) and results (plots etc.) and discussions.

Project Guideline: Link

Suggested Project List: Link (Updated 02/06/2018)

Sample Data:
Sample Images
Middelbury Stereo Image Database

Links to Resources (lecture notes and sample exams) in Previous Offerings:

Other Useful Links

  • Basics of Python and Its Application to Image Processing Through OpenCV: Link
* Example codes and images used in the above guide: Link

Tentative Course Schedule

  • Week 1 (1/26): Part 1: Image Formation and Representation: 3D to 2D projection, photometric image formation, trichromatic color representation, video format (SD, HD, UHD, HDR). Lecture note (Updated 1/25/2018). Part 2: Contrast enhancement (concept of histogram, nonlinear mapping, histogram equalization). Lecture note (Updated 1/25/2018)
     
  • Tutorial on python (Time /location to be arranged)
     
  • Computer assignment 1 (Learning Python and histogram equalization) (Due 2/5)
     
  • Week 2 (2/2): Review of 1D Fourier transform and convolution. Concept of spatial frequency. Continuous and Discrete Space 2D Fourier transform. 2D convolution and its interpretation in frequency domain. Implementation of 2D convolution. Separable filters. Frequency response. Linear filtering (2D convolution) for noise removal, image sharpening, and edge detection. Gaussian filters, DOG and LOG filters as image gradient operators. Lecture note (Updated 2/1/2018).
     
  • Computer assignment 2 (2D filtering) (Due 2/8)
     
  • Week 3 (2/9): Image sampling and resizing. Design of interpolation filters. Multiresolution representation: Pyramid and wavelets. Lecture note on ImageSampling (updated 2/8/18), Lecture note on Wavelet Δ (updated 2/8/2018).
     
  • Week 4 (TBA): Sparsity and dictionary based image processing: Image representation using orthonormal transform/dictionary. Sparsity assumption. General formulation of image enhancement as an optimization problem, L0 vs. L1 vs. L2 prior, Basic optimization techniques. Applications in debluring, denoising, inpainting, compressive sensing, superresolution, dictionary learning (PCA and KSVD).
     
  • Programming assignment 3 (Wavelet-based denoising) (Due 2/22)
     
  • Week 5 (2/23): Project Proposal Due
     
  • Week 5 (2/23): Feature detection (SIFT), feature descriptors and matching, and feature based global mapping estimation (Robust least squares and RANSAC). Lecture note.
     
  • Programming assignment 4 (Due 3/8): Stitching a panoramic picture (Feature detection, finding global mapping, warping, combining).
     
  • Week 6 (3/2): Geometric transformation. Image warping. Image morphing. Panoramic view stitching. Video stabilization. Lecture note
     
  • Week 7 (3/9): Image segmentation: region growing, split and merge, Otsu’s method, K-means, GMM clustering. Lecture note.
     
  • Programming assignment 5 (Due 3/22): Image Segmentation.
     
  • 3/12/17-3/16/17: Spring Recess
     
  • Week 8 (3/23): Midterm Project Feedback (Individual meeting)
  • Week 8 (3/23): Dense motion/displacement estimation: optical flow equation, optical flow estimation; block matching, multi-resolution estimation. Deformable registration (medical applications). Lecture note.
     
  • Week 9 (3/30): Moving object detection (background/foreground separation) (Gaussian mixture model, RPCA). Simultaneous estimation of camera motion and moving objects. Object tracking. Video shot segmentation. Lecture note.
     
  • Programming assignment 6 (Due 4/5): Moving object detection.
     
  • Week 10 (4/6): Stereo and multiview video: depth from disparity, disparity estimation, view synthesis. Depth camera (Kinect). 360 video camera and view stitching. Light field imaging. Lecture note.
     
  • Week 11 (4/13): Exam
     
  • Week 12 (4/20): Fundamentals of source coding: characterization of random sources by entropy, binary encoding, scalar quantization, vector quantization. Lecture note.
     
  • Week 13 (4/27): Transform coding: Image representation using unitary transforms (orthogonal bases), Transform coding, JPEG image compression standard, Image representation using wavelet transform; concept of layered coding, JPEG2000 image compression standard. Lecture note.
     
  • Programming assignment 7 (Due 5/10): Video Coding
     
  • Week 14 (5/4): Part 1: Video coding: block-based motion compensated prediction and interpolation, adaptive spatial prediction, block-based hybrid video coding, rate-distortion optimized mode selection, rate control, Group of pictures (GoP) structure, tradeoff between coding efficiency, delay, and complexity. Lecture note. Part 2: Overview of video coding standards (AVC/H.264, HEVC/H.265); Layered video coding: general concept and H.264/SVC. Multiview video compression. Error resilience issues. Adaptive Video Streaming (DASH). Lecture note.
     
  • Week 15 (5/11): Project Presentation.
     
  • 5/18: Project Report and all other material must be uploaded.

Sample Exams:

Sample Images:

  • Sample image for image stitching

Policy on Academic Dishonesty:
The School of Engineering encourages academic excellence in an environment that promotes honesty, integrity, and fairness. Please see the policy on academic dishonesty: Link