Lab 7: Android Sensor Processing

Introduction

For this lab, we will be leveraging some of the sensor hardware components of the Android ecosystem, as well as for the first time writing our own Java to use them.

 

The task is to create an Android application that detects and decodes the way a user moves the smartphone or tablet through the air. To do this, we will leverage information from the accelerometer API provided by the Android SDK. Here is a great resource to get your started (Google will also come up with a wealth of information on this topic):

 

http://developer.android.com/guide/topics/sensors/sensors_motion.html

 

To use the data from the accelerometer, you need to be working with an actual Android device (smartphone or tablet); the emulator will not work! If both you and your partner do not own an Android device, please come talk to the TAs so that we can work out an alternative plan for your group. We will also need to solve this problem for when it comes time for you to work on your Final Projects.

Application Description


Figure 1. An example solution application for Lab 7, showing the detection results for a Left Movement (left) and a Right Movement (right).

 

á      Functional Description: Your application will in real-time detect one of 3 possible motion gestures, decode which gesture has been detected, and show the detection decision on the screen. A unique picture describing the detected gesture will be used to notify a user which gesture was detected. As an example, please see the Fig. 1 above. Each time a new gesture has been detected, it will update the displayed detection results. During the time in between detections, no change to the displayed detection results will be made.

 

á      Fragments: Please use fragments to change the gesture pictures on the screen when an action is detected.

 

á      Gesture Detection: For this part, you will need to detect motion actions for 3 unique gestures as defined here.

 

1.    Gesture Left or Right: this gesture is done by quickly moving the phone to the left before quickly stopping the phone in place, or, this gesture is done by quickly moving the phone to the right before quickly stopping the phone in place.

2.    Circular Gesture: this gesture is done by quickly moving the phone along a circular path before stopping at the same location where you began the movement.

3.    Custom Gesture: you come up with your own unique gesture to detect!

Sample Java Code

Below is some sample Java code to help get you started.

 

/*

 * Copyright (C) 2012 The Android Open Source Project

 *

 * Licensed under the Apache License, Version 2.0 (the "License");

 * you may not use this file except in compliance with the License.

 * You may obtain a copy of the License at

 *

 *      http://www.apache.org/licenses/LICENSE-2.0

 *

 * Unless required by applicable law or agreed to in writing, software

 * distributed under the License is distributed on an "AS IS" BASIS,

 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

 * See the License for the specific language governing permissions and

 * limitations under the License.

 */

package com.example.android.fragments;

 

import android.app.Activity;

import android.content.Context;

import android.graphics.Color;

import android.hardware.Sensor;

import android.hardware.SensorEvent;

import android.hardware.SensorEventListener;

import android.hardware.SensorManager;

import android.os.Build;

import android.os.Bundle;

import android.support.v4.app.ListFragment;

import android.view.View;

import android.widget.ArrayAdapter;

import android.widget.ListView;

 

public class HeadlinesFragment extends ListFragment implements SensorEventListener {

    private  SensorManager sensorManager;

    OnHeadlineSelectedListener mCallback;

 

 

    // The container Activity must implement this interface so the frag can deliver messages

    public interface OnHeadlineSelectedListener {

        /** Called by HeadlinesFragment when a list item is selected */

        public void onArticleSelected(int position);

    }

 

    @Override

    public void onCreate(Bundle savedInstanceState) {

        super.onCreate(savedInstanceState);

        sensorManager = (SensorManager)getActivity().getSystemService(Context.SENSOR_SERVICE);

        sensorManager.registerListener(this, sensorManager.getDefaultSensor(Sensor.TYPE_LINEAR_ACCELERATION),

               SensorManager.SENSOR_DELAY_NORMAL);

 

        // We need to use a different list item layout for devices older than Honeycomb

        int layout = Build.VERSION.SDK_INT >= Build.VERSION_CODES.HONEYCOMB ?

                android.R.layout.simple_list_item_activated_1 : android.R.layout.simple_list_item_1;

 

        // Create an array adapter for the list view, using the Ipsum headlines array

        setListAdapter(new ArrayAdapter<String>(getActivity(), layout, Ipsum.Headlines));

 

    }

 

    @Override

    public void onStart() {

        super.onStart();

 

        // When in two-pane layout, set the listview to highlight the selected list item

        // (We do this during onStart because at the point the listview is available.)

        if (getFragmentManager().findFragmentById(R.id.article_fragment) != null) {

            getListView().setChoiceMode(ListView.CHOICE_MODE_SINGLE);

        }

    }

 

    @Override

    public void onAttach(Activity activity) {

        super.onAttach(activity);

 

        // This makes sure that the container activity has implemented

        // the callback interface. If not, it throws an exception.

        try {

            mCallback = (OnHeadlineSelectedListener) activity;

        } catch (ClassCastException e) {

            throw new ClassCastException(activity.toString()

                    + " must implement OnHeadlineSelectedListener");

        }

    }

 

    @Override

    public void onListItemClick(ListView l, View v, int position, long id) {

        // Notify the parent activity of selected item

        mCallback.onArticleSelected(position);

 

        // Set the item as checked to be highlighted when in two-pane layout

        getListView().setItemChecked(position, true);

    }

 

    @Override

    public void onAccuracyChanged(Sensor sensor, int accuracy) {

    // TODO Auto-generated method stub

 

    }

 

    @Override

    public void onSensorChanged(SensorEvent event) {

    // TODO Auto-generated method stub

        // check sensor type

        final float aThreshold = 2;

        if(event.sensor.getType()==Sensor.TYPE_LINEAR_ACCELERATION){

 

             // assign directions

             float x=Math.abs(event.values[0]);

             float y=Math.abs(event.values[1]);

             float z=Math.abs(event.values[2]);

             if (x < aThreshold && y < aThreshold && z < aThreshold) {

                     return;

             }

    

             if (x > y && x > z) {

                     mCallback.onArticleSelected(0);

    

                      getListView().setItemChecked(0, true);

    

             } else if (y > x && y > z)  {

                     mCallback.onArticleSelected(1);

    

                     getListView().setItemChecked(1, true);

             }else  {

                     mCallback.onArticleSelected(2);

    

                     getListView().setItemChecked(2, true);

             }

 

        }

 

    }

 

}

Signal Processing Tips

 

1.    You WILL have false positives/negatives! This is okay. Your task is to get the detection accuracy as good as possible, but know and expect that they will occur.

2.    View the live signals first! There are many free apps on Google Play that allow you to view the live sensor data from your phone/tablet. Download and try one or more out in order to get a sense for how to design your gesture recognition code. HereÕs one that we use:

a.    https://play.google.com/store/apps/details?id=com.miian.android.sensors

3.    Thresholding: Thresholding is a simple test for seeing if your signal has changed more than an allowed for amount. ItÕs very useful for determining if a gesture event of any kind has occurred.

4.    Peak detection: Peak detection is a method for identifying positive and negative peaks in your accelerometer data. You will most likely need some form of peak detection in order to determine which gesture has occurred.

5.    Google is your friend! There are tons of resources out there on Google that help you use and understand the signals from the 3-axis accelerometer provided on Android phones/tablets.

a.    For this lab, if you find a bit of useful code online, you may use it, but you cannot get credit for the lab check-off until you fully explain ALL of your code to the TAs.


Grading


For this lab, we thought a bit of competition might be a fun change. Therefore, part of your grade will depend on how good your signal processing algorithms are in the form of how low of a false positive rate you can achieve. Have fun!

 

á      (50 points) A completed lab with all requirements met.

á      (50 points) You will get 5 consecutive tries to detect a Circular Gesture, whereby the TAs will record the detection correctness of each try.

o   Gesture detection with correct recognition: 10 points

o   Gesture detection, but incorrect recognition: 5 points

o   No gesture detection: 0 points