Please watch Video Lectures and review Lecture Notes before each flipped classroom session Wednesday Nights at 9:00PM Colorado Time. Office Hours: Available by e-mail Request.
SkypeID: sam.siewert Google+ID: samuel.siewert Office: 928-777-6929 Cell: 303-641-3999 E-mail: email@example.com [E-mail: firstname.lastname@example.org]TA/Grader: Abhishikta S Pandit, Abhishikta.Pandit@Colorado.EDU, Skype ID: email@example.com
Lab Description:The course makes use of Linux-based digital media tools and requires the student to install Ubuntu Linux (ideally native) on their own PC or laptop or make use of an embedded Linux system such as the NVIDIA Jetson. If a native install is not possible, Oracle Virtual-Box installation will work, but only with pre-recorded video. For work with live video please note recommended cameras and make sure you have access to native Ubuntu Linux.
Recommended platforms for the class: 1) Native Linux Ubuntu LTS (12.04 or 14.04) download and install on any x86 laptop/desktop, 2) Jetson TK1 from NVIDIA embedded using L4T, 3) Embedded Debian Linux on TI-OMAP Beagle xM or Beagle Bone Black (Debian likely pre-installed), 4) Oracle Virtual Box on Windows or Mac OS-X with Ubuntu LTS installation on VM (note that you can only process pre-recorded MPEG), 5) Use of MobaXterm or Xming/Putty and ECES server with ecee-gpu4 or ecee-gpu5. For Linux, once you have an Ubuntu platform set up, (excluding ECES server), you'll want to install OpenCV as described here and more generally here. If you have a Linux laptop, you can most likely use your built-in webcam (test this with class example code found here) or a simple UVC supported external webcam such as a Logitech C270.
Week-1, Session-1: Introduction - Lecture 1 , PDF Download - Course goals - Course labs - Course project - Course grading policies - Demo: Real-time Media Projects Optional Reading: RTECS (Real-Time Embedded Components and Systems) pp. 1-33 Start Lab-0: Using your NATIVE Linux installation, NVIDIA Jetson, or with Virtual Box Week-1, Session-2: Finish up Introduction and Terminology Overview - Home lab demo and Q&A on your home lab setup - Amazon EC2 demo (Cloud for Education Part-1 and Cloud for Education Part-2 Pre-print) - VB installation demo - Study Real-time Media and Linux TERMINOLOGY INITIAL QUIZ REVIEW - Terminology Overview, Terminology PDF, Wikipedia (Great Resource for Technical Terminology) - Analog vs Digital Media - Fundamental Discussion of NTSC OTA Analog Transmission and Analog Cable - ATSC OTA, Digital Cable, IPTV and Streaming, On-demand, Download, and Playback Week-2: Fundamentals of Real-Time Theory for Digital Media Systems - Lecture 2 , PDF Download - Computer Architectures for Digital Media - Many-Core x32, x64 Systems Programming - Cell Broadband Engine and Programming - Offload for Masses Paper - GPU and GP-GPU Digital Video Encode/Decode and Programming - CUDA - Compute Unified Device Architecture - Fundamentals of Real-Time Scheduling and Services - Fundamentals of POSIX Threads and NPTL - RTECS discussion Linux Skills - code build, load, and debug - Building multi-threaded code - Setting up "make" build systems - Using SVN to manage code (Google Code) and development projects - Software engineering practice on Linux platforms - Using syslog, dmesg, gdb and DDD debuggers Lab-0: Lab-0 DUE - You must have home lab done to start Lab-1 Required Reading: Liu and Layland Paper Optional Reading: RTECS (Real-Time Embedded Components and Systems) pp. 50-62 Start Lab-1: Start Lab-1 Week-3: Fundamentals of RT Theory Cont'd - Services and High-Level Design - Lecture 3 , PDF Download - Lecture on RM, EDF, LLF, RM Feasibility, and EDF/LLF Feasibility Discuss: Liu and Layland Paper Required Reading: Seeing Forbidden Colors, Scientific American, pp. 72-77 Optional Reading: RTECS (Real-Time Embedded Components and Systems) pp. 123-130 INITIAL QUIZ - On Terminology - On CULearn, Available from 9:00PM TONIGHT through 9:00 PM FRIDAY QUIZ on Terminology Overview, Terminology PDF TERMINOLOGY QUIZ: Please Complete on Desire-2-Learn Week-4: Digital Video - Encoding, Decoding and Transport - Lecture 4 , PDF Download - Digital Video Encoding - Color Models and Perception (RGB, HSV, Luminance/Chromanence) Discuss: Seeing Forbidden Colors, Scientific American, pp. 72-77 - Uncompressed Frame Encoding and Image Processing - Motion Picture Expert's Group (MPEG) Encoding Basics - Elementary, Program, and Transport Stream Encapsulation - System Overview of Packet Switched IPTV and Modulated Digital Cable Systems Required Reading: IPTV Architecture Paper Optional Reading: RTECS (Real-Time Embedded Components and Systems) pp. 113-122 Lab-1: Lab-1 DUE Start Lab-2: Start Lab-2 Week-5: Digital Video - MPEG2 Transport - Lecture 5 , PDF Download, MPEG Guest Lecture - Howdy Pierce - Fundamentals of Audio and Video Encode, Transport and Decode - Digital Video Standards - 13818-1 Transport discussion - 13818-2 Digital Video Encoding (DCT, Macro-blocks, I-frames, B/P frames) - RTECS discussion Discuss: IPTV Architecture Paper Required Reading: ISO 13818-1 pp. x-50 Required Reading: ISO 13818-2 to understand Elementary Stream format and video start codes, also documented on Source Forge MPEG Headers Optional Reading: RTECS (Real-Time Embedded Components and Systems) pp. 263-280 Required Reading: Rate Monotonic vs. EDF: Judgement Day Paper Week-6, Session-1: MPEG Deeper Dive into I-frame Encoding - Lecture 6 Part B , PDF Download
Week-6, Session-2: Real-Time Implementation Challenges - Static vs Dynamic Approaches and Pitfalls - Lecture 6 , PDF Download
PART A - Kinect and Mind-controlled Robot Talk/Demo - Noritsuna Imamura, OESF, Mind Controlled Robot Video PART B - Scheduling Revisited and Completion of MPEG Overview (if needed) - Overview of Static Feasibility Tests - Dynamic Priority Feasibility Tests - RM vs EDF - Implementation - Overhead - Feasibility Testing - Methods to Detect and Manage Deadline Over-runs - Jitter and Latency - Deadlock, Unbounded Priority Inversion Discuss: ISO 13818-1 pp. x-50 and ISO 13818-2 Lab-2: Lab-2 DUE Lab-3: Start Lab-3 Week-7: Linux Device Driver Programming and Media Filesystems - Lecture 7, PDF Download - Driver framework - Character devices - Block devices - RAID Mappings for Streaming Media Discuss: Rate Monotonic vs. EDF: Judgement Day Paper Required Reading: A Case for Redundant Arrays of Inexpensive Disks (RAID) Required Reading: Intelligent RAID 6 Theory - Overview and Implementation Optional Reading: NetApp Implementation of Double Parity RAID for an alternative apporoach to Galois P,Q encoding Theory Review for Midterm Exam next week - Lecture MT Review, PDF Download Weekend-7: MIDTERM MIDTERM (2.5 hours) - On Desire2Learn Week-8: SIMD Vector Processing with GPU, GP-GPU, and Instruction Set Extensions for Video - Lecture 8, PDF Download Overview of CUDA Programming for GPU and GP-GPUs - Driver debugging techniques - Example Driver walkthrough Overview of x86 SIMD Instruction Set Extensions and Use - Intel Performance Primitives and Compilers - Speed-up - Using SIMD to Accelerate Image Processing Algorithms Discuss: A Case for Redundant Arrays of Inexpensive Disks (RAID) Discuss: Intelligent RAID 6 Theory - Overview and Implementation Discuss: NetApp Implementation of Double Parity RAID for an alternative apporoach to Galois P,Q encoding Lab-3: Lab-3 DUE Start Lab-4: Start Lab-4 Week-9: Digital Content From Post Production to Delivery - Lecture 9, PDF Download - Digital Rights Management, Conditional Access, Content Protection - Digital Cable vs Switched Digital Video - End of Broadcast TV? ATSC, IPTV, and Everything on Demand - Viral Video - Games on Demand - Personal Content Production, Sharing, Storage - Digital Cinema Initiatives - Digital Content End to End (Shoot, Edit, Distribute, Presentation) - Looking Ahead: Augmented Reality, Immersive Virtual Reality, 3-D Cinema - Grand Challenges - Storage (100's to 1000's of hours/streams to millions) - Bandwidth (Last Mile Problem) - Processing - Power Consumption, Multi-core, App Specific (GPU, FPGA, Cell) - User Interaction (Return path, Peer-to-Peer, Telepresence) - Future Look (Highlights From NAB Conference and GPU Tech) - Nat'l Association of Broadcasters - GPU Technology Conference - Digital Media - Augmented Reality (Computer Vision, Graphics, Video, Interactive) - Immersive Virtual Reality - 3-D Cinema Discuss: Video Fingerprinting and Encryption Principles for Digital Rights Management Discuss: Business Week, March 2010 - "Revenge of the Cable Guys" Extended Lab Background on Digital Media Systems - Lecture 10, PDF Download Required Reading: Video Fingerprinting and Encryption Principles for Digital Rights Management Required Reading: Business Week, March 2010 - "Revenge of the Cable Guys" Lab-4: Lab-4 DUE Start Lab-Extended: Start Lab-Extended Week-10, Session-1: Linux Post Production - Performance Measurement and Acceleration - Lecture 11, PDF Download * ME - ECEE Graduate Design Program for 2012 (Prof. Mark Rentschler) Speed-up Possible with GCC Compiler Optimizations - "-O1", "-O2", "-O3" Speed-up Possible with Multi-threading for Multi-core SMT Processors - NPTL threading for multi-core and SMT (Hyper-threading) Speed-up Possible with Vector Processing Instructions (SSE 4.x) - Generating code with SSE code generation for speed-up of bit processing Speed-up Possible with GPU or GP-GPU Co-Processors - CUDA toolkit (4.1) - Data transfer, kernels, threads - CUDA example Performance Debug Tools on Linux - LTT NG Linux Trace Toolkit - Next Generation - Using Trace tools (Systemtap, ftrace) - Using Profiler tools (oprofile, VTune) - Using VTune for Real-Time Media Optimization Week-10, Session-2: Scaling and Speed-up - Lecture 12, PDF Download Threading Design for IO Latency Hiding and Multi-core Platforms - Ahmdal's Law and Parallel Processing Concurrency Speed-Up Limits - First Method to Hide IO Latency - Second Method to Hide IO Latency - Linux IO Scheduling Policy Week-10, Session-3: Lecture 13 (Final Quiz Review), PDF Download Week-10 FINAL QUIZ:AVAILABLE ON DESIRE-2-LEARN August 7th, EXTENDED LAB WRITE-UP due on Desire2Learn no later than midnight. GRADES COMPLETE - Grades Finalized for all students, including remote CU Engineering Anywhere