ECEN 5623 - Real-Time Embedded Systems, ESE Program

Class: Wednesdays, Flipped Classroom using Zoom, May 31st until August 16, 2016
Final project demonstration day - August 16, Dr. Siewert available for demos all day on campus

6:00PM Colorado Time on Wednesdays with ZOOM

Please watch Video Lectures and consult Lecture Notes before each flipped class. Please use one of the supported embedded platforms for the course along with supporting tools as outlined here to complete Labs and turn them in on D2L (Desire to Learn).

Video Lectures Office Hours: Available by e-mail request.
SkypeID: sam.siewert
Google+ID: samuel.siewert
Office: 928-777-6929
Cell: 303-641-3999

SA, Grader: TBD, TBD@Colorado.EDU
Office Hours: TBD.
Skype ID: TBD.

Remote Lab Note: Embedded Linux, Linux Foundation Zephyr or FreeRTOS are ALL acceptable platforms for this course and there are numerous options for your lab setup. Please review and purchase and configure the option that works best for you. The ESE Program office DOES HAVE SOME LOANER SYSTEMS you can check out as well - see your TA for help if you prefer this option. Note that the NVIDIA Jetson can also be used for ECEN5043 (Computer and Machine Vision) and note that the DE1-SoC has an FPGA which may be used in other embedded courses. For Zephyr, the Intel Curie/Quark (Arduino 101) is a good option for Zephyr Development using the FlySwatter2 to flash the board and Adafruit FTDI cable for a console.
Graduate Students with Questions - please e-mail Adam Sadoff (Adam.Sadoff@Colorado.EDU) and Kimberley Smith (kimberley.smith@Colorado.EDU) for enrollment questions.
Please note that all course materials should be consulted on Desire-2-Learn; If for some reason you can't access D2L, this page is provided as a backup.

Course Description:In this course, students will design and build a microprocessor-based embedded system application using a real-time operating system or RT POSIX extensions with Embedded Linux. The course focus is on the process as well as fundamentals of integrating microprocessor-based embedded system elements for digital command and control of typical embedded hardware systems. Standard project options include: stereo vision, computer vision, voice-over-IP, computer vision tracking systems, facial recognition and numerous related projects. The student will be introduced to the full embedded system lifecycle process in this course including: analysis, design (using extended finite state machine specification tools), programming, hardware assembly, unit testing, integration and system testing.

Lab Description:The course requires the student to install embedded Linux or an RTOS microkernel (FreeRTOS or Zephyr) on a System-on-Chip processor such as the NVIDIA Tegra K1, the Altera DE1-SoC, or the Intel Curie / Quark (Arduino 101). In addition, students may want to set up an Oracle Virtual-Box Linux installation. For work with live video please note recommended cameras and make sure you have access to native Ubuntu Linux on a Jetson. For other types of sensor interfaces and audio, the DE1-SoC or Curie/Quark is fine. This course must be completed using an embedded system RTOS (FreeRTOS or Zephyr) or embedded Linux only, not a PC running Linux. You will however find Linux as a useful host development system (Windows is ok too).

Recommended platforms for the class:

  1. Jetson TK1 from NVIDIA embedded using L4T,
  2. Altera DE1-SoC which can run Linux or FreeRTOS,
  3. Intel Curie/Quark SoC running Zephyr microkernel Intel Curie/Quark (Arduino 101) is a good option for Zephyr Development using the FlySwatter2 to flash the board and Adafruit FTDI cable for a console.
  4. Embedded Debian Linux on TI-OMAP (Being Phased Out, but OK) Beagle xM or Beagle Bone Black (Debian likely pre-installed),
  5. Host Development and Test on Native Linux Ubuntu LTS (12.04 or 14.04) download and install on any x86 laptop/desktop,
  6. Host Development and Test with Oracle Virtual Box on Windows or Mac OS-X with Ubuntu LTS installation on VM (note that you can only process pre-recorded MPEG), 5) Use of MobaXterm or Xming/Putty and ECES server with ecee-gpu4 or ecee-gpu5. For Linux, once you have an Ubuntu platform set up, (excluding ECES server), you'll want to install OpenCV as described here and more generally here. If you have a Linux laptop, you can most likely use your built-in webcam (test this with class example code found here) or a simple UVC supported external webcam such as a Logitech C270. NOTE THAT YOU MUST COMPLETE YOUR EXTENDED LAB on an EMBEDDED Jetson, DE1-SoC or Beagle Board and that other Linux installations are intended to be used only for practice and support - i.e. you must use an embedded Linux or RTOS system to complete the course.

  1. REQUIRED TEXT: Real-Time Embedded Components and Systems with Linux and RTOS, Sam Siewert and John Pratt, December 2015, 978-1942270041, Mercury Learning, Amazon, E-book
  2. Linux Kernel Development (3rd Edition), Robert Love, Addison-Wesley Professional; 3 edition (July 2, 2010), ISBN-10: 0672329468, ISBN-13: 978-0672329463, Linux Kernel Development (3rd Edition), by Robert Love

Important Course Links (Linux, RTOS=VxWorks, FreeRTOS, Zephyr)
Lab Exercises
Video Lectures
Linux Example Code, RTOS Example Code, FreeRTOS Example Code
Linux Docs, RTOS Docs, FreeRTOS Docs, Linux Top-Errors, RTOS Top-Errors, 5623 Classic
Papers Read in Class

RTECS 2nd Ed Figures and Examples RTECS 1st Ed CDROM

Course and Grading Policies

Important Course Tools
DE1-SoC Tools: DE1-SoC Board Terasic Page, ARM DS-5 Community Edition, Altera FTP Site, Altera DE1-SoC Labs, DE1-SoC Resources, Altera SoC EDS Downloads, Quartus II FPGA Design Tools,Class Documents for DE1-SoC.
NVIDIA Jetson Tools: NVIDIA Jetson Page, Jetson TK1 eLinux Page, Class Documents for Jetson.
TI Beagle Board Tools: Beagle Board Page, Class Documents for Beagle.
Intel Curie/Quark Tools: Curie Quark How-To, Linux Foundation Zephyr for Quark.

Oracle Virtual-Box
Ubuntu Linux LTS, 14.04 LTS Download
Linux UVC Supported Cameras, e.g. Logitech C200, or for Analog Camera Frame Grabber, Hauppauge USB-Live 2 used with any low-cost NTSC CCTV camera, e.g. Wired Color CCTV Security Camera - cost ranging from $10 up to $50.

Syllabus (See D2L for Definitive Lab Due Dates and Quizzes) Term-D - June 1 to August 7
Week-1, Session-1 Class: Introduction - Video-Lecture 1 , PDF Download
        - Course goals [10 weeks, 5 labs, Mid-Term Exam, 2 Quizzes]
          * Beagle xM and ARM DM architecture [ARM v7te ISA, Coretex A8]
          * Ubuntu Linux
          * Linux user space and kernel space programming
          * Hard Real-Time Theory
          * Drivers and Real-Time Video
          * 1/3 Theory, 1/3 Practice, 1/3 Practitioner Work
        - Course project [Final Extended Lab - USB Camera for Time Lapse Capture]
        - Course grading (see policy below)
        - Fall ECEN 5623 Video Files: Project from Previous Semesters (Robotics, Video, Audio)
        - RT Embedded System TERMINOLOGY
          INITIAL QUIZ REVIEW - Terminology Overview

      Linux Host Development System setup (Tips on Beagle xM or NVIDIA Jetson)
        - Booting Ubuntu and other Linux Distros on Beagle xM
        - Native host Linux (Ubuntu) or Oracle Virtual-Box Linux on Windows
        - See Nisheeth Bhat's IS on TimeSys for Hard Real-Time
        - Kaushik's Instructions for loading Ubuntu on Beagle xM
        - Beagle xM
        - "cat /proc/devices"
        - lsmod
        - dmesg
        - "tail -f /var/log/messages"
        - ECEN 5623 Example Linux Code

       Paper: Liu and Layland Rate Monotonic paper

       Read: RTECS (Real-Time Embedded Components and Systems) - Chapter 1

Start Lab-1

Week-1, Session-2 Class: Fundamentals of Real-Time Theory - Video-Lecture 2, PDF Download
        - What's makes distributed, real-time, embedded multitasking systems challenging?
        - Best effort, Hard real-time, Soft real-time
        - Scheduling Class Taxonomy
          - Best Effort scheduling (Round-Robin Timeslice Scheme - Review)
          - Intro to Fixed priority preemptive scheduling
          - Intro to Dynamic priority scheduling
        - Utility Curves

       Read: RTECS - Chapter 2

       Paper: Lehoczky Sha and Ding paper

Week-2, Session-1 Class: Fundamentals of RT Theory Cont'd - Services and High-Level Design - QUIZ on Terminology Overview, Video-Lecture 3 , PDF Download

      INITIAL QUIZ - On Terminology - Complete on D2L

      Real-Time Services
        - What a Service Release Timeline Really Looks Like
        - The CPU, I/O, Memory Resource Space (Characterizing RT Applications)
        - Intro to Timing diagrams (interference)
        - Intro to Hard real-time safe resource utilization bounds

        Case-Study #1: High-Level RT System Design Examples

        EXAMPLE - Space Infrared Telescope Facility Instrument Control (launched on August 25, 2003)
        - The hard real-time requirements and performance
        - What worked, what didn't and why
        - Overview Presentation
        - The Test Image

        EXAMPLE - DATA-CHASER Space Shuttle Hitch-hiker Class Payload Flown on STS-85, Summer 1997

       Paper: Audsley DM paper

INITIAL QUIZ - On Terminology - Complete on D2L

Lab-1 DUE
Start Lab-2

Week-3, Session-1 Class: RM/DM Policy and Feasibility Test Derivation - Video-Lecture 4 , PDF Download

      Rate Monotonic Policy and Feasibility Overview
        - Rate Monotonic Assumptions and Constraints
        - More on Fixed priority preemptive scheduling
        - Overview of Derivation of RM LUB - Hard real-time safe resource utilization bounds
        - More on Dynamic priority scheduling (Earliest Deadline First and Least Laxity First)
        - EDF and LLF Overview

      Introduction to Feasibility Tests
        - Sufficient Tests - RM LUB, DM
        - N&S Tests
          - Scheduling Point
          - Completion Test

      Deadline Monotonic Policy and Feasibility Overview
        - DM Sufficient Feasibility Test
        - DM Sufficient and More Necessary Test

      Beyond RM and DM:
        - Scheduling Point Feasibility Test
        - Completion Test Feasibility Test
        - Sufficient vs. Necessary and Sufficient Feasibility Tests
        - Computational Complexity of Feasibility Tests (O(1), O(n^2), O(n^3))

      PROJECT BACKGROUNDER: COMPUTER VISION, Video-Lecture-Machine-Vision, PDF Download

      Read: RTECS - Chapter 3

Week-4, Session-1 Class: Services, Synchronization and Shared Resources - Video-Lecture 5 , PDF Download

     "Priority inheritance protocols: an approach to real-time synchronization", Sha, L.; Rajkumar, R.; Lehoczky, J.P.; Computers, IEEE Transactionson, Volume: 39 Issue: 9, Sep 1990.

      Paper Available on:Paper's Read in Class Page

      HW and HW+FW Implementations of RT Services
        - ISRs, task canonical structure, and task/service release
        - FPGA/ASIC-based State Machines for Offloading
        - Microprocessor/Microcontroller Offloading
        - DSP Offloading

      SW Implementations of RT Services
        - Main+ISR Executive
        - Cooperative Non-Preemptive Threads
          - Callbacks
          - Continuation Functions
          - Software State-Machines
        - Priority Preemptive Run-to-Completion RTOS
        - Time-sliced Traditional Best Effort OS

        Synchronization and Resource Issues:
          - Problems with Blocking (resources other than CPU, e.g. I/O)
            - Break up into more threads (better scheduling control)
            - Interrupt driven I/O - e.g. Programmable FIFOs
            - Model Blocking Time
          - Priority inversion (general concept)
            - Unbounded priority inversion problem (mutex C.S.)
            - Priority inheritence
            - Priority ceiling

        Other Practical Problems - Solutions:
          1) Estimating C (execution jitter) - use WCET
          2) Period jitter - transform to highest frequency
          3) ISR and context switch overhead - add to WCET
          4) Deadlock/Livelock - HW Watch Dog
          5) Bad code spinning and wedging - Sanity Monitoring

      Case-Study #2: Mars Pathfinder (As presented from multiple perspectives)
        Mike Jones Overview or What Happened to Mars Pathfinder
        Mars Pathfinder -- WRS Story
        Mars Pathfinder -- JPL Story


      Read: RTECS -- Chapter 4 & 5

Lab-2 DUE
Start Lab-3

Week-5, Session-1 Class: Real-Time Service System Integration and IO
      (RT-Linux User-space and Kernel-space Service Threads and Synchronization) - Video-Lecture 6 , PDF Download

      Service Synchronization, Communication, and IO
        - Using Message Queues to Sync Services and for Communication
        - Using Binary Semaphores to Sync Services
        - What about IO during Service Execution?

      Overlapping CPU and IO Cycles - Latency Hiding
        - Initial Release Input - Block DMA and FIFO Sensor Data
        - Intermediate IO
        - Memory-Mapped IO During Service Processing
        - Hiding IO latency with Overlap

      System Design with RT Services
        - Software process: analysis, design, specification, coding,
                           unit testing, integration, system testing,
                           delivery, operations

        - Implementation characteristics: tasks, message passing,
                                          synchronization, ISRs,
                                          memory-mapped I/O, bus
                                          interface, signals, shared
                                          memory, stack/data/heap
                                          mgt., comm. interfaces

        - Specifying the design of real-time software systems
          Data flow and Control flow (function and interfaces)
          Extended finite state machines and signal block diagrams
          MSC (multi-tasking multi-node protocol/comm model)

        - Processes, threads, and tasks, what's the diff in RT-Linux?

      Read: RTECS - Chapter 6

Week-5: EXAM-1 (2.5 hours to complete exam on D2L)

Week-6, Session-1 Class: Scalable Embedded Systems Architectures Video-Lecture 7 , PDF Download

     Intro to PCI & USB Architecture and I/O Architectures
         - Intro to PCI bus architecture and device interfaces
         - PCI Plug and Play Concept
         - USB Plug and Play Concept
         - Embedded System PCI Form Factors and Standards
           CompactPCI, PC/104+ (PCI+ISA), PMC, PCI-X 1.0a/b, PCI-X 2.0, PCI-Express
         Read: Universal Serial Bus, (chapters TBD)

         - Discussion on I/O Trends (High speed differential serial)
           10G Ethernet (10GEA), which became IEEE P802.3ae
           Open Fabric Alliance for 10/20/40/80Gbps Infiniband
           Other High-speed Serial: Firewire, RapidIO, SAS/SATA

         - Discussion of ASIC Trends
           SoC (Yesterday's Board, Today's Chipset, Tomorrow's ASIC)
             Core + I/O (PowerPC 8xx, 82xx)
             Reconfigurable (Virtex II)
             Configurable (Tensilica)
             IP Modules (CPU Cores, Mem Ctlrs., Local Bus)
           Off-loading (Today's SW is tomorrow's HW)
        - The POSIX 1003.1b compliant Embedded system environment (VxWorks)
        - The POSIX Application Programmer's Interface
            - POSIX overview: message queues, RT signals, and semaphores
            - Clocks and timers

Lab-3 DUE
Start Lab-4

Week-7, Session-1 Class: Device Drivers and Characterization of Embedded I/O Video-Lecture 8 , PDF Download

      I/O interfaces
        - Digital
        - Analog (ADC, DAC interfaces)

      Microprocessor interface types (word or block)
        - Register-based control, status, data
        - Higher rate FIFO I/O
        - Block-oriented 1st/3rd party DMA tx/rx between I/O interfaces and memory
        - Bus burst transfers and block transfers
        - system memory map for MMIO devices - DRAM/SDRAM/DDR, BOOTROM, Flash

      External interface types
        - CPU local bus IO/MMIO
            E.g. PCI 2.x, GPIO, DRAM, Flash
        - Point-to-point or switched devices
            E.g. RS-232, RS-422, PCI-Express
        - Network multi-access devices
            E.g. Ethernet

      Device interfaces -- introduction to drivers
        - User space App interface (driver entry point interface to user processes)
        - Device interface (interface to devices)
        - ring buffers
        - blocking/non-blocking
        - ioctl "swiss army knife"
        - ISRs and signals/semaphores
        - scheduled I/O (handle buffering and processing in task)

Week-8, Session-1 Class: Real-Time Performance, Measuring Performance and HW/SW Debugging Techniques Video-Lecture 11 , PDF Download
       Mid-Term Solutions (Flipped Classroom)

       Estimating/Measuring Performance Based on CPU Architecture

       IBM Paper on Performance Monitoring and Tuning

        - Measuring/Controlling CPU Efficiency
          - Trace Ports (e.g. IBM PowerPC 4xx series, Strong Arm)
          - Built-in PMU (Performance Monitoring Units) (e.g. Intel Pentium, Xscale)
          - External Methods
            - Logic Analyzer Memory Traces (Cache Misses, DMA, Un-cached access)
            - Memory Port Markers (Writes to Un-cached Memory)

          - Profiling Code by Function or Block
            - Software in Circuit Methods (e.g. Workbench Code Coverage using SW In-Circuit, Workbench Profiler)
            - Hardware Supported Profiling (e.g. Intel Vtune)
              - Cycle-based profiling
              - Event-based profiling

          - Cache Coherency
              - Harvard I-Cache, D-cache Architecture
              - Cache Invalidate, Flush, Lock, Pre-fetch

        - Measuring/Controlling I/O Efficiency
          - Bus Analyzers - e.g. PCI Event Traces
          - Logic Analyzer with Support Package

        - PCI Bus Performance Tuning
          - Bus Grant Arbiters and Priority Schemes, Minimum Burst Length
          - PCI-Express Performance Features - PCI Express Tutorial
            - Isochronal Virtual Channels
            - Split Transaction

        Paper: Zen of BDM

Week-8, Session-2 Class: Embedded Volatile and Non-volatile Memory Devices Video-Lecture 12 , PDF Download

        Memory Technologies and Hierarchy
        - Harvard Arch. I-Cache, D-Cache (Separate instruction and data cache)
        - L1 On-Chip, L2/L3 Off-Chip Cache
        - SRAM (Fast, but expensive per bit stored)
        - DRAM, SDRAM, DDR (Double Data Rate DRAM), Quad Rate SRAM

        ECC (Error Correcting Circuitry) memory
        - Hardware/Software Design
        - Hamming encoding for EDAC (Error Detection and Correction) - review handout provided
        - ECC and maintenance and handling of SBEs and MBEs
        - Firmware ECC Req'ts:
          1) initialize memory fully,
          2) handle SBE interrupts, MBE check-stop, and
          3) background scrubbing

        Non-Volatile Memory Devices and Systems
        - EPROM, EEPROM, NVRAM, and Flash (NAND, NOR)
        - Flash File Systems - TrueFFS
        - Solid-State-Disk (SSD) - Intel Tech Journal Article, also available on Papers Read in Class
        - Advanced Non-Volatile Memory/Storage Devices (PCM, FRAM, MRAM, Race-Track) - Overview

Lab-4 DUE

Start Extended Lab and preparation of final submission Report and D2L Digital Video Submission

Week-9, Session-1 Class: High Availability vs. High Reliability (Same?) Video-Lecture 13 , PDF Download

        IBM developerWorks Paper on HA and HR

        Definition of High Availability (5 9's)
        - System is ready to provide service 99.999 % of any given service year
        - System can crash, but must reboot with minimal interruption to services and returning to fully service availability quickly

        Definition of High Reliability
        - Formal methods and testing to prove reliability is designed in and built in
        - SW Path coverage, statement coverage, and condition coverage criteria
        - HW Simulation testing with SW drivers
        - HW built-in test (Built-in Logic Analyzers, JTAG, Built-in Performance Monitoring Counters)
        - System Safe Modes

        Recovery Concepts
        - Conservative Safe Mode Manual Recovery (typical of satellite systems)
        - Automatic Recovery (typical of high availability sytems)

Week-10, Session-1, EXAM-2 REVIEW, Extended Lab Q&A and In-Class Work

Week-10: EXAM-2 (2.5 hours to complete exam on D2L)


Grades Finalized per Registrar Calendar