ECEE 5653/4653 - Real-Time Digital Media, ESE Program

2015 Term-D Class: Wednesdays, Flipped Classroom using BlueJeans @ 9:00-10:00 PM Colorado Time [8PM Pacific/Arizona], Starting on June 3rd until August 5th, 2015

BlueJeans Meeting Link (Participants), Instructor Login

Please watch Video Lectures and review Lecture Notes before each flipped classroom session Wednesday Nights at 9:00PM Colorado Time.

Office Hours: Available by e-mail Request.
SkypeID: sam.siewert
Google+ID: samuel.siewert
Office: 928-777-6929
Cell: 303-641-3999
TA/Grader: Abhishikta S Pandit, Abhishikta.Pandit@Colorado.EDU, Skype ID:
Office Hours: Saturdays and Sundays 8:00-9:30AM

Remote Lab Note: NATIVE Linux on a PC or Embedded System are BOTH acceptable platforms for this course and there are numerous options for your lab setup. Please review and configure the option that works best for you. Note that the NVIDIA Jetson can also be used for ECEE5623.
Graduate Students with Questions - please e-mail Undergraduates please e-mail Jessica Drelles - Names A-K [] or Beth Curtis - Names L-Z []. For more general info on Be Boulder Anywhere e-mail Beth Morledge Webb [].
Please note that all course materials should be consulted on Desire-2-Learn; If for some reason you can't access D2L, this page is provided as a backup.

Course Description: An introductory course on real-time digital media including digital video and audio. Topics covered include digital media focused real-time threading, synchronization, hardware acceleration and function offload, real-time theory, and debug/analysis methods for real-time systems in Linux; Linux drivers, kernel interfaces, GPU digital video offload acceleration, and Linux real-time application implementation strategies; MPEG2 digital video parser/generators and encoder/decoders, SDTV/HDTV broadband/baseband transport, open source digital video tools, real-time RTP/UDP transport, and transport stream analysis and standards. Students configure and extend Linux applications and kernel drivers for digital video systems in project and lab work.

Lab Description:The course makes use of Linux-based digital media tools and requires the student to install Ubuntu Linux (ideally native) on their own PC or laptop or make use of an embedded Linux system such as the NVIDIA Jetson. If a native install is not possible, Oracle Virtual-Box installation will work, but only with pre-recorded video. For work with live video please note recommended cameras and make sure you have access to native Ubuntu Linux.

Recommended platforms for the class: 1) Native Linux Ubuntu LTS (12.04 or 14.04) download and install on any x86 laptop/desktop, 2) Jetson TK1 from NVIDIA embedded using L4T, 3) Embedded Debian Linux on TI-OMAP Beagle xM or Beagle Bone Black (Debian likely pre-installed), 4) Oracle Virtual Box on Windows or Mac OS-X with Ubuntu LTS installation on VM (note that you can only process pre-recorded MPEG), 5) Use of MobaXterm or Xming/Putty and ECES server with ecee-gpu4 or ecee-gpu5. For Linux, once you have an Ubuntu platform set up, (excluding ECES server), you'll want to install OpenCV as described here and more generally here. If you have a Linux laptop, you can most likely use your built-in webcam (test this with class example code found here) or a simple UVC supported external webcam such as a Logitech C270.

  1. Digital Media Primer (2nd Edition), Yue-Ling Wong, Prentice Hall; 2 edition (February 27, 2012), ISBN-10: 0132893509, ISBN-13: 978-0132893503, Amazon Link, also available in CU bookstore.
  2. Linux Kernel Development (3rd Edition), Robert Love, Addison-Wesley Professional; 3 edition (July 2, 2010), ISBN-10: 0672329468, ISBN-13: 978-0672329463, Linux Kernel Development (3rd Edition), by Robert Love

Important Course Links
Link to Example Media
Link to Example Code
Video Lectures
Papers Read in Class
Course and Grading Policies

Important Course Tools
NVIDIA Jetson Tools: NVIDIA Jetson Page, Jetson TK1 eLinux Page, Class Documents for Jetson. OpenCV
Oracle Virtual-Box
Ubuntu Linux LTS, 14.04 LTS Download
Linux UVC Supported Cameras, e.g. Logitech C200 - cost ranging from $10 up to $50.
Digital Audio and Video Tools for Linux: ALSA Page, Libav, FFMPEG.

Syllabus (See D2L for Definitive Lab Due Dates and Quizzes) - Term-D, June 1 to August 7
Week-1, Session-1: Introduction - Lecture 1 , PDF Download
        - Course goals
        - Course labs
        - Course project
        - Course grading policies
        - Demo: Real-time Media Projects

       Optional Reading: RTECS (Real-Time Embedded Components and Systems) pp. 1-33

Start Lab-0: Using your NATIVE Linux installation, NVIDIA Jetson, or with Virtual Box

Week-1, Session-2: Finish up Introduction and Terminology Overview
        - Home lab demo and Q&A on your home lab setup
        - Amazon EC2 demo (Cloud for Education Part-1 and Cloud for Education Part-2 Pre-print)
        - VB installation demo
        - Study Real-time Media and Linux TERMINOLOGY
          INITIAL QUIZ REVIEW - Terminology Overview, Terminology PDF, Wikipedia (Great Resource for Technical Terminology)
         - Analog vs Digital Media
           - Fundamental Discussion of NTSC OTA Analog Transmission and Analog Cable
           - ATSC OTA, Digital Cable, IPTV and Streaming, On-demand, Download, and Playback

Week-2: Fundamentals of Real-Time Theory for Digital Media Systems - Lecture 2 , PDF Download

       - Computer Architectures for Digital Media
         - Many-Core x32, x64 Systems Programming
         - Cell Broadband Engine and Programming - Offload for Masses Paper
         - GPU and GP-GPU Digital Video Encode/Decode and Programming - CUDA - Compute Unified Device Architecture

       - Fundamentals of Real-Time Scheduling and Services
       - Fundamentals of POSIX Threads and NPTL
       - RTECS discussion

      Linux Skills - code build, load, and debug
        - Building multi-threaded code
        - Setting up "make" build systems
        - Using SVN to manage code (Google Code) and development projects
        - Software engineering practice on Linux platforms
        - Using syslog, dmesg, gdb and DDD debuggers

Lab-0: Lab-0 DUE - You must have home lab done to start Lab-1
      Required Reading: Liu and Layland Paper
       Optional Reading: RTECS (Real-Time Embedded Components and Systems) pp. 50-62

Start Lab-1: Start Lab-1

Week-3: Fundamentals of RT Theory Cont'd - Services and High-Level Design - Lecture 3 , PDF Download

       - Lecture on RM, EDF, LLF, RM Feasibility, and EDF/LLF Feasibility
       Discuss: Liu and Layland Paper

      Required Reading: Seeing Forbidden Colors, Scientific American, pp. 72-77
      Optional Reading: RTECS (Real-Time Embedded Components and Systems) pp. 123-130

      INITIAL QUIZ - On Terminology - On CULearn, Available from 9:00PM TONIGHT through 9:00 PM FRIDAY
      QUIZ on Terminology Overview, Terminology PDF

TERMINOLOGY QUIZ: Please Complete on Desire-2-Learn

Week-4: Digital Video - Encoding, Decoding and Transport - Lecture 4 , PDF Download

       - Digital Video Encoding
       - Color Models and Perception (RGB, HSV, Luminance/Chromanence)
       Discuss: Seeing Forbidden Colors, Scientific American, pp. 72-77
       - Uncompressed Frame Encoding and Image Processing
       - Motion Picture Expert's Group (MPEG) Encoding Basics
       - Elementary, Program, and Transport Stream Encapsulation
       - System Overview of Packet Switched IPTV and Modulated Digital Cable Systems

      Required Reading: IPTV Architecture Paper
      Optional Reading: RTECS (Real-Time Embedded Components and Systems) pp. 113-122

Lab-1: Lab-1 DUE

Start Lab-2: Start Lab-2

Week-5: Digital Video - MPEG2 Transport - Lecture 5 , PDF Download, MPEG Guest Lecture - Howdy Pierce

       - Fundamentals of Audio and Video Encode, Transport and Decode
       - Digital Video Standards
       - 13818-1 Transport discussion
       - 13818-2 Digital Video Encoding (DCT, Macro-blocks, I-frames, B/P frames)
       - RTECS discussion
       Discuss: IPTV Architecture Paper

      Required Reading: ISO 13818-1 pp. x-50
      Required Reading: ISO 13818-2 to understand Elementary Stream format and video start codes, also documented on Source Forge MPEG Headers
      Optional Reading: RTECS (Real-Time Embedded Components and Systems) pp. 263-280
      Required Reading: Rate Monotonic vs. EDF: Judgement Day Paper

Week-6, Session-1: MPEG Deeper Dive into I-frame Encoding - Lecture 6 Part B , PDF Download
Week-6, Session-2: Real-Time Implementation Challenges - Static vs Dynamic Approaches and Pitfalls - Lecture 6 , PDF Download
PART A - Kinect and Mind-controlled Robot Talk/Demo - Noritsuna Imamura, OESF, Mind Controlled Robot Video PART B - Scheduling Revisited and Completion of MPEG Overview (if needed) - Overview of Static Feasibility Tests - Dynamic Priority Feasibility Tests - RM vs EDF - Implementation - Overhead - Feasibility Testing - Methods to Detect and Manage Deadline Over-runs - Jitter and Latency - Deadlock, Unbounded Priority Inversion Discuss: ISO 13818-1 pp. x-50 and ISO 13818-2 Lab-2: Lab-2 DUE Lab-3: Start Lab-3 Week-7: Linux Device Driver Programming and Media Filesystems - Lecture 7, PDF Download - Driver framework - Character devices - Block devices - RAID Mappings for Streaming Media Discuss: Rate Monotonic vs. EDF: Judgement Day Paper Required Reading: A Case for Redundant Arrays of Inexpensive Disks (RAID) Required Reading: Intelligent RAID 6 Theory - Overview and Implementation Optional Reading: NetApp Implementation of Double Parity RAID for an alternative apporoach to Galois P,Q encoding Theory Review for Midterm Exam next week - Lecture MT Review, PDF Download Weekend-7: MIDTERM MIDTERM (2.5 hours) - On Desire2Learn Week-8: SIMD Vector Processing with GPU, GP-GPU, and Instruction Set Extensions for Video - Lecture 8, PDF Download Overview of CUDA Programming for GPU and GP-GPUs - Driver debugging techniques - Example Driver walkthrough Overview of x86 SIMD Instruction Set Extensions and Use - Intel Performance Primitives and Compilers - Speed-up - Using SIMD to Accelerate Image Processing Algorithms Discuss: A Case for Redundant Arrays of Inexpensive Disks (RAID) Discuss: Intelligent RAID 6 Theory - Overview and Implementation Discuss: NetApp Implementation of Double Parity RAID for an alternative apporoach to Galois P,Q encoding Lab-3: Lab-3 DUE Start Lab-4: Start Lab-4 Week-9: Digital Content From Post Production to Delivery - Lecture 9, PDF Download - Digital Rights Management, Conditional Access, Content Protection - Digital Cable vs Switched Digital Video - End of Broadcast TV? ATSC, IPTV, and Everything on Demand - Viral Video - Games on Demand - Personal Content Production, Sharing, Storage - Digital Cinema Initiatives - Digital Content End to End (Shoot, Edit, Distribute, Presentation) - Looking Ahead: Augmented Reality, Immersive Virtual Reality, 3-D Cinema - Grand Challenges - Storage (100's to 1000's of hours/streams to millions) - Bandwidth (Last Mile Problem) - Processing - Power Consumption, Multi-core, App Specific (GPU, FPGA, Cell) - User Interaction (Return path, Peer-to-Peer, Telepresence) - Future Look (Highlights From NAB Conference and GPU Tech) - Nat'l Association of Broadcasters - GPU Technology Conference - Digital Media - Augmented Reality (Computer Vision, Graphics, Video, Interactive) - Immersive Virtual Reality - 3-D Cinema Discuss: Video Fingerprinting and Encryption Principles for Digital Rights Management Discuss: Business Week, March 2010 - "Revenge of the Cable Guys" Extended Lab Background on Digital Media Systems - Lecture 10, PDF Download Required Reading: Video Fingerprinting and Encryption Principles for Digital Rights Management Required Reading: Business Week, March 2010 - "Revenge of the Cable Guys" Lab-4: Lab-4 DUE Start Lab-Extended: Start Lab-Extended Week-10, Session-1: Linux Post Production - Performance Measurement and Acceleration - Lecture 11, PDF Download * ME - ECEE Graduate Design Program for 2012 (Prof. Mark Rentschler) Speed-up Possible with GCC Compiler Optimizations - "-O1", "-O2", "-O3" Speed-up Possible with Multi-threading for Multi-core SMT Processors - NPTL threading for multi-core and SMT (Hyper-threading) Speed-up Possible with Vector Processing Instructions (SSE 4.x) - Generating code with SSE code generation for speed-up of bit processing Speed-up Possible with GPU or GP-GPU Co-Processors - CUDA toolkit (4.1) - Data transfer, kernels, threads - CUDA example Performance Debug Tools on Linux - LTT NG Linux Trace Toolkit - Next Generation - Using Trace tools (Systemtap, ftrace) - Using Profiler tools (oprofile, VTune) - Using VTune for Real-Time Media Optimization Week-10, Session-2: Scaling and Speed-up - Lecture 12, PDF Download Threading Design for IO Latency Hiding and Multi-core Platforms - Ahmdal's Law and Parallel Processing Concurrency Speed-Up Limits - First Method to Hide IO Latency - Second Method to Hide IO Latency - Linux IO Scheduling Policy Week-10, Session-3: Lecture 13 (Final Quiz Review), PDF Download Week-10 FINAL QUIZ:AVAILABLE ON DESIRE-2-LEARN August 7th, EXTENDED LAB WRITE-UP due on Desire2Learn no later than midnight. GRADES COMPLETE - Grades Finalized for all students, including remote CU Engineering Anywhere