ECEE 5623 - Real-Time Embedded Systems, ESE Program

Class: ECEE 1B32, Wednesdays, Flipped Classroom (ECEE 1B32) using Zoom, May 29th until August 16, 2019 - During Term-D (June 3 to Aug 9).
Final project demonstration day - August 15, Dr. Siewert available for demos all day on campus (8/17 early morning for overflow).

Join @ 4:00PM Colorado Time Wednesdays with ZOOM (Flipped Class)

DEBUG or DEMO of Help Sessions with SA ZOOM

Please watch Video Lectures and consult Lecture Notes before each flipped class. Please use one of the supported embedded platforms for the course along with supporting tools as outlined here to complete Labs and turn them in on Canvas [Canvas].

Instructor: Sam Siewert,, Sam.Siewert@Colorado.EDU
Office Hours
: Available by e-mail request.
SkypeID: sam.siewert
Office (voice): 928-777-6929
Cell (Text preferred): 303-641-3999
Graduate Student enrollment and advising - please e-mail Adam Sadoff (Adam.Sadoff@Colorado.EDU) with questions.
SA, Grader: Mohit Rane, Mohit.Rane@Colorado.EDU
Office Hours: Mondays and Wednesdays 2-4 PM (on campus).
Skype: live:mrane2010
Phone (Text preferred): 720-789-1750
SA, Grader: Shreya Chakraborty,
Office Hours: Tues & Thursday 1:30-3:30 PM.
Skype: live:shreya12ee1037
Phone (Text preferred): 720-436-8127
Course Prerequisites: Strong C programming skills (some C++ helpful), familiarity with operating systems, computer architecture and hardware/software interface and debug skills are required. Advanced skills and knowledge of embedded CPU multi-core, GP-GPU, and/or FPGA are helpful, but not required. Before taking ECEE 5623, students should have a first course in related subjects such as ECEN 2120/2350, ECEN 3100/3350, and ECEN 1030/1310/CSCI 1300 and/or have completed ECEN 5803 and ECEN 5813 prior to taking this course. Real-time theory will be taught using embedded Linux or an equivalent RTOS or IoT kernel and students will be expected to work with camera interfaces and real-time sensor systems to build a verifiable predictable response application.

Course Description: In this course, students will design and build a microprocessor-based embedded system application using a real-time operating system or RT POSIX extensions with Embedded Linux. The course focus is on the process as well as fundamentals of integrating microprocessor-based embedded system elements for digital command and control of typical embedded hardware systems. Standard project options include: stereo vision, computer vision, voice-over-IP, computer vision tracking systems, facial recognition and numerous related projects. The student will be introduced to the full embedded system lifecycle process in this course including: analysis, design (using extended finite state machine specification tools), programming, hardware assembly, unit testing, integration and system testing.

Lab Description: The course requires the student to install embedded Linux or an RTOS microkernel (FreeRTOS or Zephyr) on a System-on-Chip processor such as the Raspberry-Pi 3b+, NVIDIA Tegra K1 or Nano, the Altera DE1-SoC, or the Intel Curie / Quark (Arduino 101). In addition, students may want to set up an Oracle Virtual-Box Linux installation. For work with live video please note recommended cameras and make sure you have access to native JetPack Linux on a Jetson (TK1 or Nano). For other types of sensor interfaces and audio, the DE1-SoC or Curie/Quark is fine. This course must be completed using an embedded system RTOS (FreeRTOS or Zephyr) or embedded Linux only, not a PC running Linux. You will however find Linux as a useful host development system (Windows is ok too).

  1. REQUIRED TEXT: Real-Time Embedded Components and Systems with Linux and RTOS, Sam Siewert and John Pratt, December 2015, 978-1942270041, Mercury Learning, Amazon, E-book, RTECS 2nd Ed Figures and Examples, RTECS 1st Ed CDROM, ERRATA
  2. Linux Kernel Development (3rd Edition), Robert Love, Addison-Wesley Professional; 3 edition (July 2, 2010), ISBN-10: 0672329468, ISBN-13: 978-0672329463, Linux Kernel Development (3rd Edition), by Robert Love

Recommended platforms for the class (in order of ease of use):
  1. Raspberry Pi 3 model B from Amazon Starter Kit described on eLinux,
  2. Buy Jetson Nano and Get Started with Nano, Jetson Nano eLinux, or Jetson TK1 from NVIDIA embedded using L4T.
  3. Altera DE1-SoC which can run Linux or FreeRTOS,
  4. Intel Curie/Quark SoC running Zephyr microkernel Intel Curie/Quark (Arduino 101) is a good option for Zephyr Development using the FlySwatter2 to flash the board and Adafruit FTDI cable for a console.
  5. Embedded Debian Linux on TI-OMAP (Being Phased Out, but OK) Beagle xM or Beagle Bone Black (Debian likely pre-installed),
  6. Host Development and Test on Native Linux Ubuntu LTS (12.04 or 14.04) download and install on any x86 laptop/desktop,
  7. Host Development and Test with Oracle Virtual Box on Windows or Mac OS-X with Ubuntu LTS installation on VM (note that you can only process pre-recorded MPEG), 5) Use of MobaXterm or Xming/Putty and ECES server with ecee-gpu4 or ecee-gpu5. For Linux, once you have an Ubuntu platform set up, (excluding ECES server), you'll want to install OpenCV as described here and more generally here. If you have a Linux laptop, you can most likely use your built-in webcam (test this with class example code found here) or a simple UVC supported external webcam such as a Logitech C270. NOTE THAT YOU MUST COMPLETE YOUR EXTENDED LAB on an EMBEDDED Jetson TK1 or Nano, DE1-SoC or Beagle Board and that other Linux installations are intended to be used only for practice and support - i.e. you must use an embedded Linux or RTOS system to complete the course.
Remote Lab Note: Embedded Linux, Linux Foundation Zephyr or FreeRTOS are ALL acceptable platforms for this course and there are numerous options for your lab setup. Please review and purchase and configure the option that works best for you. The ESE Program office DOES HAVE SOME LOANER SYSTEMS you can check out as well - see your TA for help if you prefer this option. Note that the NVIDIA Jetson TK1 or Nano can also be used for ECEE57663 (EMVIA) and note that the DE1-SoC has an FPGA which may be used in other embedded courses. For Zephyr, the Intel Curie/Quark (Arduino 101) is a good option for Zephyr Development using the FlySwatter2 to flash the board and Adafruit FTDI cable for a console.

Important Course Links (Linux, RTOS=VxWorks, FreeRTOS, Zephyr)
Lab Exercises
Video Lectures
Linux Example Code, Linux Example Code 2, RTOS Example Code, FreeRTOS Example Code
Linux Docs, RTOS Docs, FreeRTOS Docs, Linux Top-Errors, RTOS Top-Errors, 5623 Classic
Papers Read in Class
Course and Grading Policies

Important Course Tools
DE1-SoC Tools: DE1-SoC Board Terasic Page, ARM DS-5 Community Edition, Altera FTP Site, Altera DE1-SoC Labs, DE1-SoC Resources, Altera SoC EDS Downloads, Quartus II FPGA Design Tools,Class Documents for DE1-SoC.
NVIDIA Jetson Tools: NVIDIA Jetson Page, Jetson TK1 eLinux Page, Jetson Nano Page, Class Documents for Jetson.
TI Beagle Board Tools: Beagle Board Page, Class Documents for Beagle.
Intel Curie/Quark Tools: Curie Quark How-To, Linux Foundation Zephyr for Quark.
TIVA FreeRTOS Tools: Class Documents for TIVA FreeRTOS.

Oracle Virtual-Box
Ubuntu Linux LTS, 14.04 LTS Download
Linux UVC Supported Cameras, e.g. Logitech C200, or for Analog Camera Frame Grabber, Hauppauge USB-Live 2 used with any low-cost NTSC CCTV camera, e.g. Wired Color CCTV Security Camera - cost ranging from $10 up to $50.

Syllabus (See Canvas for Definitive Lab Due Dates and Quizzes) Term-D - June to August
Week-1, Session-1 Class: Introduction - Lecture 1 Video , Lecture 1 Notes , Updated Notes
        - Course goals [10 weeks, 5 labs, Mid-Term Exam, 2 Quizzes]
          * Beagle xM and ARM DM architecture [ARM v7te ISA, Coretex A8]
          * Ubuntu Linux
          * Linux user space and kernel space programming
          * Hard Real-Time Theory
          * Drivers and Real-Time Video
          * 1/3 Theory, 1/3 Practice, 1/3 Practitioner Work
        - Course project [Final Extended Lab - USB Camera for Time Lapse Capture]
        - Course grading (see policy below)
        - Fall ECEE 5623 Video Files: Project from Previous Semesters (Robotics, Video, Audio)
        - RT Embedded System TERMINOLOGY
          INITIAL QUIZ REVIEW - Terminology Overview

      Linux Host Development System setup (Tips on Beagle xM or NVIDIA Jetson)
        - Booting Ubuntu and other Linux Distros on Beagle xM
        - Native host Linux (Ubuntu) or Oracle Virtual-Box Linux on Windows
        - See Nisheeth Bhat's IS on TimeSys for Hard Real-Time
        - Kaushik's Instructions for loading Ubuntu on Beagle xM
        - Beagle xM
        - "cat /proc/devices"
        - lsmod
        - dmesg
        - "tail -f /var/log/messages"
        - ECEE 5623 Example Linux Code

       Paper: Liu and Layland Rate Monotonic paper

       Read: RTECS (Real-Time Embedded Components and Systems) - Chapter 1

Start Lab-1

Week-1, Session-2 Class: Fundamentals of Real-Time Theory - Lecture 2 Video , Lecture 2 Notes , Updated Notes
        - What's makes distributed, real-time, embedded multitasking systems challenging?
        - Best effort, Hard real-time, Soft real-time
        - Scheduling Class Taxonomy
          - Best Effort scheduling (Round-Robin Timeslice Scheme - Review)
          - Intro to Fixed priority preemptive scheduling
          - Intro to Dynamic priority scheduling
        - Utility Curves

       Read: RTECS - Chapter 2

       Paper: Lehoczky Sha and Ding paper

Week-2, Session-1 Class: Fundamentals of RT Theory Cont'd - Services and High-Level Design - QUIZ on Terminology Overview, Lecture 3 Video , Lecture 3 Notes , Updated Notes

      INITIAL QUIZ - On Terminology - Complete on Canvas

      Real-Time Services
        - What a Service Release Timeline Really Looks Like
        - The CPU, I/O, Memory Resource Space (Characterizing RT Applications)
        - Intro to Timing diagrams (interference)
        - Intro to Hard real-time safe resource utilization bounds

        Case-Study #1: High-Level RT System Design Examples

        EXAMPLE - Space Infrared Telescope Facility Instrument Control (launched on August 25, 2003)
        - The hard real-time requirements and performance
        - What worked, what didn't and why
        - Overview Presentation
        - The Test Image

        EXAMPLE - DATA-CHASER Space Shuttle Hitch-hiker Class Payload Flown on STS-85, Summer 1997

       Paper: Audsley DM paper

INITIAL QUIZ - On Terminology - Complete on Canvas

Lab-1 DUE
Start Lab-2

Week-3, Session-1 Class: RM/DM Policy and Feasibility Test Derivation - Lecture 4 Video , Lecture 4 Notes , Updated Notes

      Rate Monotonic Policy and Feasibility Overview
        - Rate Monotonic Assumptions and Constraints
        - More on Fixed priority preemptive scheduling
        - Overview of Derivation of RM LUB - Hard real-time safe resource utilization bounds
        - More on Dynamic priority scheduling (Earliest Deadline First and Least Laxity First)
        - EDF and LLF Overview

      Introduction to Feasibility Tests
        - Sufficient Tests - RM LUB, DM
        - N&S Tests
          - Scheduling Point
          - Completion Test

      Deadline Monotonic Policy and Feasibility Overview
        - DM Sufficient Feasibility Test
        - DM Sufficient and More Necessary Test

      Beyond RM and DM:
        - Scheduling Point Feasibility Test
        - Completion Test Feasibility Test
        - Sufficient vs. Necessary and Sufficient Feasibility Tests
        - Computational Complexity of Feasibility Tests (O(1), O(n^2), O(n^3))

      PROJECT BACKGROUNDER: COMPUTER VISION, Video-Lecture-Machine-Vision, PDF Download

      Read: RTECS - Chapter 3

Week-4, Session-1 Class: Services, Synchronization and Shared Resources - Lecture 5 Video , Lecture 5 Notes , Updated Notes

     Paper: PIP - "Priority inheritance protocols: an approach to real-time synchronization", Sha, L.; Rajkumar, R.; Lehoczky, J.P.; Computers, IEEE Transactionson, Volume: 39 Issue: 9, Sep 1990.

      HW and HW+FW Implementations of RT Services
        - ISRs, task canonical structure, and task/service release
        - FPGA/ASIC-based State Machines for Offloading
        - Microprocessor/Microcontroller Offloading
        - DSP Offloading

      SW Implementations of RT Services
        - Main+ISR Executive
        - Cooperative Non-Preemptive Threads
          - Callbacks
          - Continuation Functions
          - Software State-Machines
        - Priority Preemptive Run-to-Completion RTOS
        - Time-sliced Traditional Best Effort OS

        Synchronization and Resource Issues:
          - Problems with Blocking (resources other than CPU, e.g. I/O)
            - Break up into more threads (better scheduling control)
            - Interrupt driven I/O - e.g. Programmable FIFOs
            - Model Blocking Time
          - Priority inversion (general concept)
            - Unbounded priority inversion problem (mutex C.S.)
            - Priority inheritence
            - Priority ceiling

        Other Practical Problems - Solutions:
          1) Estimating C (execution jitter) - use WCET
          2) Period jitter - transform to highest frequency
          3) ISR and context switch overhead - add to WCET
          4) Deadlock/Livelock - HW Watch Dog
          5) Bad code spinning and wedging - Sanity Monitoring

      Case-Study #2: Mars Pathfinder (As presented from multiple perspectives)
        Authoritative Mars Pathfinder Account
        Mike Jones Overview or What Happened to Mars Pathfinder
        Mars Pathfinder -- WRS Story
        Mars Pathfinder -- JPL Story


      Read: RTECS -- Chapter 4, 5 & 6

Lab-2 DUE
Start Lab-3

Week-5, Session-1 Class: Real-Time Service System Integration and IO
      (RT-Linux User-space and Kernel-space Service Threads and Synchronization) - Lecture 6 Video , Lecture 6 Notes , Updated Notes

      Service Synchronization, Communication, and IO
        - Using Message Queues to Sync Services and for Communication
        - Using Binary Semaphores to Sync Services
        - What about IO during Service Execution?

      Overlapping CPU and IO Cycles - Latency Hiding
        - Initial Release Input - Block DMA and FIFO Sensor Data
        - Intermediate IO
        - Memory-Mapped IO During Service Processing
        - Hiding IO latency with Overlap

      System Design with RT Services
        - Software process: analysis, design, specification, coding,
                           unit testing, integration, system testing,
                           delivery, operations

        - Implementation characteristics: tasks, message passing,
                                          synchronization, ISRs,
                                          memory-mapped I/O, bus
                                          interface, signals, shared
                                          memory, stack/data/heap
                                          mgt., comm. interfaces

        - Specifying the design of real-time software systems
          Data flow and Control flow (function and interfaces)
          Extended finite state machines and signal block diagrams
          MSC (multi-tasking multi-node protocol/comm model)

        - Processes, threads, and tasks, what's the diff in RT-Linux?

      Read: RTECS - Chapter 7 and re-read 4.2, 4.3, 4.4 regarding WCET

Lab-3 DUE

Week-6: EXAM-1 on RTECS Chapters 1-7 (2 hours to complete exam on Canvas)

Week-6, Session-1 Class: Scalable Embedded Systems Architectures - Lecture 7 Video , Lecture 7 Notes , Updated Notes

     Intro to PCI & USB Architecture and I/O Architectures
         - Intro to PCI bus architecture and device interfaces
         - PCI Plug and Play Concept
         - USB Plug and Play Concept
         - Embedded System PCI Form Factors and Standards
           CompactPCI, PC/104+ (PCI+ISA), PMC, PCI-X 1.0a/b, PCI-X 2.0, PCI-Express
         Read: Universal Serial Bus, (chapters TBD)

         - Discussion on I/O Trends (High speed differential serial)
           10G Ethernet (10GEA), which became IEEE P802.3ae
           Open Fabric Alliance for 10/20/40/80Gbps Infiniband
           Other High-speed Serial: Firewire, RapidIO, SAS/SATA

         - Discussion of ASIC Trends
           SoC (Yesterday's Board, Today's Chipset, Tomorrow's ASIC)
             Core + I/O (PowerPC 8xx, 82xx)
             Reconfigurable (Virtex II)
             Configurable (Tensilica)
             IP Modules (CPU Cores, Mem Ctlrs., Local Bus)
           Off-loading (Today's SW is tomorrow's HW)
        - The POSIX 1003.1b compliant Embedded system environment (VxWorks)
        - The POSIX Application Programmer's Interface
            - POSIX overview: message queues, RT signals, and semaphores
            - Clocks and timers

      Read: RTECS - Chapter 8

Start Lab-4

Week-7, Session-1 Class: Device Drivers and Characterization of Embedded I/O - Lecture 8 Video , Lecture 8 Notes , Updated Notes

      I/O interfaces
        - Digital
        - Analog (ADC, DAC interfaces)

      Microprocessor interface types (word or block)
        - Register-based control, status, data
        - Higher rate FIFO I/O
        - Block-oriented 1st/3rd party DMA tx/rx between I/O interfaces and memory
        - Bus burst transfers and block transfers
        - system memory map for MMIO devices - DRAM/SDRAM/DDR, BOOTROM, Flash

      External interface types
        - CPU local bus IO/MMIO
            E.g. PCI 2.x, GPIO, DRAM, Flash
        - Point-to-point or switched devices
            E.g. RS-232, RS-422, PCI-Express
        - Network multi-access devices
            E.g. Ethernet

      Device interfaces -- introduction to drivers
        - User space App interface (driver entry point interface to user processes)
        - Device interface (interface to devices)
        - ring buffers
        - blocking/non-blocking
        - ioctl "swiss army knife"
        - ISRs and signals/semaphores
        - scheduled I/O (handle buffering and processing in task)

      Read: RTECS - Chapter 9

Week-8, Session-1 Class: Real-Time Performance, Measuring Performance and HW/SW Debugging Techniques - Lecture 9 Video , Lecture 9 Notes , Updated Notes
       Mid-Term Solutions (Flipped Classroom)

       Estimating/Measuring Performance Based on CPU Architecture

       IBM Paper on Performance Monitoring and Tuning

        - Measuring/Controlling CPU Efficiency
          - Trace Ports (e.g. IBM PowerPC 4xx series, Strong Arm)
          - Built-in PMU (Performance Monitoring Units) (e.g. Intel Pentium, Xscale)
          - External Methods
            - Logic Analyzer Memory Traces (Cache Misses, DMA, Un-cached access)
            - Memory Port Markers (Writes to Un-cached Memory)

          - Profiling Code by Function or Block
            - Linux Software in Circuit Methods (e.g. Linux Ftrace, eLinux Ftrace, Kernel Shark, eLinux Kernel Shark)
            - VxWorks RTOS Software in Circuit Methods (e.g. Workbench Code Coverage using SW In-Circuit, Workbench Profiler)
            - Hardware Supported Profiling (e.g. Intel Vtune)
              - Cycle-based profiling
              - Event-based profiling

          - Cache Coherency
              - Harvard I-Cache, D-cache Architecture
              - Cache Invalidate, Flush, Lock, Pre-fetch

        - Measuring/Controlling I/O Efficiency
          - Bus Analyzers - e.g. PCI Event Traces
          - Logic Analyzer with Support Package

        - PCI Bus Performance Tuning
          - Bus Grant Arbiters and Priority Schemes, Minimum Burst Length
          - PCI-Express Performance Features - PCI Express Tutorial
            - Isochronal Virtual Channels
            - Split Transaction

        Paper: Zen of BDM

Week-8, Session-2 Class: Embedded Volatile and Non-volatile Memory Devices - Lecture 10 Video , Lecture 10 Notes , Updated Notes

        Memory Technologies and Hierarchy
        - Harvard Arch. I-Cache, D-Cache (Separate instruction and data cache)
        - L1 On-Chip, L2/L3 Off-Chip Cache
        - SRAM (Fast, but expensive per bit stored)
        - DRAM, SDRAM, DDR (Double Data Rate DRAM), Quad Rate SRAM

        ECC (Error Correcting Circuitry) memory
        - Hardware/Software Design
        - Hamming encoding for EDAC (Error Detection and Correction) - review handout provided
        - ECC and maintenance and handling of SBEs and MBEs
        - Firmware ECC Req'ts:
          1) initialize memory fully,
          2) handle SBE interrupts, MBE check-stop, and
          3) background scrubbing

        Non-Volatile Memory Devices and Systems
        - EPROM, EEPROM, NVRAM, and Flash (NAND, NOR)
        - Flash File Systems - TrueFFS
        - Solid-State-Disk (SSD) - Intel Tech Journal Article, also available on Papers Read in Class
        - Advanced Non-Volatile Memory/Storage Devices (PCM, FRAM, MRAM, Race-Track) - Overview

      Read: RTECS - Chapter 10

Lab-4 DUE

Start Extended Lab and preparation of final submission Report and Canvas Digital Video Submission

Week-9, Session-1 Class: High Availability vs. High Reliability (Same?) - Lecture 11 Video , Lecture 11 Notes , Updated Notes

        IBM developerWorks Paper on HA and HR

        Definition of High Availability (5 9's)
        - System is ready to provide service 99.999 % of any given service year
        - System can crash, but must reboot with minimal interruption to services and returning to fully service availability quickly

        Definition of High Reliability
        - Formal methods and testing to prove reliability is designed in and built in
        - SW Path coverage, statement coverage, and condition coverage criteria
        - HW Simulation testing with SW drivers
        - HW built-in test (Built-in Logic Analyzers, JTAG, Built-in Performance Monitoring Counters)
        - System Safe Modes

        Recovery Concepts
        - Conservative Safe Mode Manual Recovery (typical of satellite systems)
        - Automatic Recovery (typical of high availability sytems)

      Read: RTECS - Chapter 11, 12, 13, 14

Week-10, Session-1, EXAM-2 REVIEW, Extended Lab Q&A and In-Class Work

Week-10: EXAM-2 - RTECS Chapters 8-14 (2.5 hours to complete exam on Canvas)


Grades Finalized per Registrar Calendar

Please note that all course materials should be consulted on Canvas; If for some reason you can't access Canvas, this page is provided as a backup.

Honesty Policies:

Any suspected violations of the Honor Code will be submitted to our Office of Student Conduct and Conflict Resolution (OSCCR). Students found responsible by our faculty for violating the cheating policy of the Honor Code will earn an automatic F in the course. Further non-academic sanctions may be rendered by the OSCCR. We take these issues seriously and have a responsibility to all students who uphold the Honor Code, and to the highest industry standards for which we are preparing students. If you have any questions whatsoever regarding what collaboration is permissible in the course, consult your instructor directly before proceeding. Sharing of knowledge between students is highly encouraged; however, each student is expected to independently create and implement their own project files. Students may find that they are able to leverage firmware designs from books, magazines, the internet, or their work environments; however, in these cases, students are expected and required to credit the source of the information clearly and completely. Again, by default, you are expected to turn in your own original work and cite any and all portions you did not create. All aspects of the Honor Code apply.