Journal of Multimedia Information System
Korea Multimedia Society
Section

Detecting User Activities with the Accelerometer on Android Smartphones

Xingfeng Wang1, Heecheol Kim2,*
1Information Engineering College, Eastern Liaoning University, Dandong, China / Department of Computer Engineering, INJE University, Gimhae, Korea, wxf6165051@163.com
2Department of Computer Engineering, INJE University, Gimhae, Korea, heeki@inje.ac.kr
*Corresponding Author: Hee-Cheol Kim, Department of Computer Engineering, INJE University, Gimhae, Korea, +82-55-320-3720, heeki@inje.ac.kr.

© Copyright 2015 Korea Multimedia Society. This is an Open-Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/4.0/) which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Received: Jun 08, 2015 ; Revised: Jun 20, 2015 ; Accepted: Jun 29, 2015

Published Online: Jun 30, 2015

Abstract

Mobile devices are becoming increasingly sophisticated and the latest generation of smartphones now incorporates many diverse and powerful sensors. These sensors include acceleration sensor, magnetic field sensor, light sensor, proximity sensor, gyroscope sensor, pressure sensor, rotation vector sensor, gravity sensor and orientation sensor. The availability of these sensors in mass-marketed communication devices creates exciting new opportunities for data mining and data mining applications. In this paper, we describe and evaluate a system that uses phone-based accelerometers to perform activity recognition, a task which involves identifying the physical activity that a user is performing. To implement our system, we collected labeled accelerometer data from 10 users as they performed daily activities such as “phone detached”, “idle”, “walking”, “running”, and “jumping”, and then aggregated this time series data into examples that summarize the user activity 5-minute intervals. We then used the resulting training data to induce a predictive model for activity recognition. This work is significant because the activity recognition model permits us to gain useful knowledge about the habits of millions of users—just by having them carry cell phones in their pockets.

Keywords: Accelerometer; Activity Classification; Data Mining; Smartphone

I. INTRODUCTION

Mobile devices, such as smartphones and music players, have recently begun to incorporate diverse and powerful sensors. These sensors include acceleration sensor, magnetic field sensor, light sensor, proximity sensor, gyroscope sensor, pressure sensor, rotation vector sensor, gravity sensor and orientation sensor. Because of the small size of these “smart” mobile devices, their substantial computing power, their ability to send and receive data, and their nearly ubiquitous use in our society, these devices open up exciting new areas for data mining research and data mining applications. In this paper, we explore the use of one of these sensors, the accelerometer, in order to identify the activity that a user is performing—a task that we refer to as activity recognition.

Android phones, as well as virtually all new smartphones and smart music players, contain tri-axial accelerometers that measure acceleration in all three spatial dimensions. These accelerometers are also capable of detecting the orientation of the device (helped by the fact that they can detect the direction of Earth’s gravity), which can provide useful information for activity recognition. Accelerometers were initially included in these devices to support advanced game play[1] and to enable automatic screen rotation[2] but they clearly have many other applications. In fact, there are many useful applications that can be built if accelerometers can be used to recognize a user’s activity. For example, we can automatically monitor a user’s activity level and generate daily, weekly, and monthly activity reports, which can be automatically emailed to the user. These reports would indicate an overall activity level, which could be used to gauge if the user was getting an adequate amount of exercise and estimate the number of daily calories expended. These reports could be used to encourage healthy practices and might alert some users to how sedentary they or their children actually are. The activity information can also be used to automatically customize the behavior of the mobile phone. For example, music could automatically be selected to match the activity or send calls directly to voicemail when the user is exercising.

In order to address the activity recognition task using supervised learning, we first collected accelerometer data from 10 users as they performed the activities such as “phone detached”, “idle”, “walking”, “running”, and “jumping”. Here, “idle” refers to either “sitting” or “standing”, meaning no exercise. We then aggregated this raw time series accelerometer data into examples (or episodes), where each example is labeled with the activity that occurred while that the was being collected. We then built predictive models for the activity recognition using classification algorithms. The topic of the accelerometer-based activity recognition is not new. Bao & Intille [5] developed an activity recognition system to identify twenty activities using bi-axial accelerometers placed in five locations on the user’s body. Additional studies have similarly focused on how one can use a variety of accelerometer based devices to identify a range of user activities. The other work has focused on the applications that can be built based on the accelerometer-based activity recognition. This work includes identifying a user’s activity level and predicting their energy consumption [8], detecting a fall and the movements of user after the fall [15], and monitoring user activity levels in order to promote health and fitness [16].

Our work differs from most prior work in that we use a commercial mass-marketed device rather than a research-only device, we use a single device conveniently kept in the user’s pocket rather than multiple devices distributed across the body, and we require no additional actions by the user. Our work makes several additional contributions. One contribution is the data that we have collected and continued to collect, which we plan to make public in the future. This data can serve as a resource to other researchers, since we were unable to find such publically available data ourselves. We also demonstrate how raw time series accelerometer data can be transformed into examples that can be used by conventional classification algorithms. We demonstrate that it is possible to perform activity recognition with commonly available (nearly ubiquitous) equipment and yet achieve highly accurate results. Finally, we believe that our work will help bring attention to the opportunities available for mining wireless sensor data and will stimulate additional work in this area.

The remainder of this paper is structured as follows. Related work is described in Section 2, Section 3 describes the process for addressing the activity recognition task, including data collection, data preprocessing, and data transformation. Section 4 describes our experiments and results. Section 5 summarizes our conclusions and discusses the areas for the future research.

II. RELATED WORKS

Activity recognition has recently gained attention as a research topic because of the increasing availability of accelerometers in consumer products, like smartphones, and because of the many potential applications. Some of the earliest work in accelerometer based activity recognition focused on the use of multiple accelerometers placed on several parts of the user’s body. In one of the earliest studies of this topic, Bao & Intille [5] used five biaxial accelerometers worn on the user’s right hip, dominant wrist, non-dominant upper arm, dominant ankle, and non-dominant thigh in order to collect data from 20 users. Using decision tables, instance-based learning, Naive Bayes classifiers, they created models to recognize twenty daily activities. Their results indicated that the accelerometer placed on the thigh was the most powerful for distinguishing between activities. This finding supports our decision to have our test subjects carry the phone in the most convenient location—their pants pocket.

Some studies have also focused on combining multiple types of sensors in addition to accelerometers for activity recognition. Maurer et al. [18] used “eWatch” devices placed on the belt, shirt pocket, trouser pocket, backpack, and neck to recognize the same six activities that we consider in our study. Each “eWatch” is consisted of a biaxial accelerometer and a light sensor. Decision trees, k-Nearest Neighbor, Naïve Bayes, and Bayes Net classifiers with five-fold cross validation were used for learning. Choudhury et. al [19] used a multimodal sensor device consisting of seven different types of sensors (tri-axial accelerometer, microphone, visible light phototransitor, barometer, visible+IR light sensor, humidity/temperature reader, and compass) to recognize activities such as walking, sitting, standing, ascending stairs, descending stairs, elevator moving up and down, and brushing one’s teeth. Cho et. al. [20] used a single tri-axial accelerometer, along with an embedded image sensor worn at the user’s waist, to identify nine activities. Although these multi-sensor approaches do indicate the great potential of mobile sensor data as more types of sensors are being incorporated into devices, our approach shows that only one type of sensor—an accelerometer—is needed to recognize most daily activities. Thus our method offers a straightforward and easily-implementable approach to accomplish this task.

A few studies, like ours, did use an actual commercial mobile device to collect data for activity recognition. Such systems offer an advantage over other accelerometer-based systems because they are unobtrusive and do not require any additional equipment for data collection and accurate recognition. Miluzzo et. al. [21] explored the use of various sensors (such as a microphone, accelerometer, GPS, and camera) available on commercial smartphones for activity recognition and mobile social networking applications. In order to address the activity recognition task, they collected the accelerometer data from ten users to build an activity recognition model for walking, running, sitting, and standing. This model had particular difficulty distinguishing between the states of sitting and of standing activities, a task that our models easily achieve. Yang [22] developed an activity recognition system using the Nokia N95 phone to distinguish among sitting, standing, walking, running, driving, and bicycling. This work also explored the use of an activity recognition model to construct physical activity diaries for the users. Although the study achieved relatively high accuracies of prediction, stair climbing was not considered and the system was trained and tested using data from only four users. Brezmes et. al. [6] also used the Nokia N95 phone to develop a real-time system for recognizing 10 user activities. In their systems, an activity recognition model is trained for each user, meaning that there is no universal model that can be applied to new users, for whom no training data exists. Our models do not have this limitation.

III. THE ACTIVITY RECOGNITION TASK

In this section, we describe the activity recognition task and the process for performing this task. In Section 3.1 we describe our protocol for collecting the raw accelerometer data, in Section 3.2 we describe how we preprocess and transform the raw data into examples, and in Section 3.3 we describe the activities that will be predicted/identified.

3.1. Data Collection

In order to collect data for our supervised learning task, it was necessary to have some users carry an Android-based smart phone while performing designated activities. We enlisted the help of 10 volunteer’s subjects to carry a smartphone while performing a specific set of activities. These subjects carried the Android phone in the front pockets of their pants and were asked to “phone detached”, “idle”, “walking”, “running”, and “jumping” for specific periods of time.

The data collection was controlled by an application we created and executed on the phone. This application, through a simple graphical user interface, permitted us to record the user’s name, start and stop the data collection, and label the activity being performed. The application permitted us to control what sensor data (e.g. accelerometer, gyroscope) was collected and how frequently it was collected. In all cases we collected the accelerometer data every 20 milliseconds, so we had 50 samples per second.

3.2. Feature Generation & Data Transformation

Standard classification algorithms cannot be directly applied to raw time-series accelerometer data. Instead, we first must transform the raw time series data into examples [17]. Note that a sample window of 512 samples was used. 512 samples were chosen based on the recommendation of prior papers, because it was deemed sufficient for the activity recognition and because 512 is a number of 2n, which is ideal for the Fast Fourier Transformation algorithm. In terms of a sliding window, we use the window, the size of which is 100 samples.

The following features were chosen based on our own discussion and the recommendations of previous accelerometer based activity recognition research:

  • Fundamental Frequencies: The fundamental frequencies of the signal in the window.

  • Maximum Amplitude: The maximum value of the signal in the window.

  • Minimum Amplitude: The minimum value of the signal in the window.

  • Intensity: The intensity value of the signal in the window.

  • Fundamental Frequencies Magnitude: The fundamental frequencies magnitude of the signal in the window.

  • Step Num: The step num value of the signal in the window.

  • Position Relation: Placement direction of the smartphone .

How can we get the fundamental frequencies of the signal? These were found from the Fourier Transformation of the signal over the sample window. The final value was the average of the three dominant frequencies of the signal. Figure 1 is the frequency spectrum acquired.

jmis-2-2-207-g1
Fig. 1. Fundamental frequencies
Download Original Figure

The “Step Num” feature requires further explanation. The different persons have a different walking (or running) speed. For example, a woman’s running speed is probably equivalent to man’s fast walking speed. But how can distinguish walking and running? The step number is a way. No matter your speed is fast or slow, walking’s step number always far less than running. Therefore, we can use the step number to distinguish walking and running. Figure 2 is the step number (each peak is each step), the left side is the walking data, and the right side is the running data. You can see the walking’s step number far less than the running step.

jmis-2-2-207-g2
Fig. 2. Diagram for step number.
Download Original Figure

The “Position Relation” feature requires further explanation, too. No matter your smartphone is a phone detached or idle, the both acceleration data has a bit wave. But how can distinguish phone detached and idle? The position relation is a way. If your smartphone is the phone detached, it always lies on the flat ground. But if your telephone is the idle, it is always vertical or oblique nearby your body. Therefore, we can use the smartphone’s orientation to distinguish phone detached and idle. Figure 3 is the position relation (blue curve represents x axis, green curve represents y axis, red curve represents z axis), the left side is the phone detached data, the right side is the idle data. You can see the obvious difference of the orientation between phone detached and idle.

jmis-2-2-207-g3
Fig. 3. Diagram for position relation.
Download Original Figure
3.3 The Activities

In this study, we consider five activities: phone detached, idle, walking, running, and jumping. We selected these activities because they are performed regularly by many people in their daily routines. The activities also involve motions that often occur for the substantial time periods, thus making them easier to recognize. Furthermore, most of these activities involve repetitive motions and we believe that this should also make the activities easier to recognize. When we record data for each of these activities, we record acceleration in three axes.

Figure 4 plots the accelerometer data for a typical user, for seven features of continuous five activities (each activity lasts 30 seconds, entire five activities are 150 seconds in all). It is clear that static activities (phone detached, idle) and dynamic activities (walking, running, jumping), showed on accelerometer maximum amplitude and minimum amplitude, accelerometer intensity, accelerometer fundamental frequencies magnitude and the step number.

jmis-2-2-207-g4
Fig. 4. Acceleration plots for the seven features
Download Original Figure

About distinguishing different dynamic activities, you can see the accelerometer intensity and accelerometer fundamental frequencies magnitude. It is clear to distinguish the walking, running and jumping.

If you see the position relation to distinguish different static activities, it is also clear to distinguish the phone detached and non-detached.

3.4 Classification Algorithm

During the classification stage, we labeled unknown patterns of the data based on the knowledge acquired from known patterns of data. For the implementation of the activity recognition application, we implemented a k-nearest neighbor classifier on the data. The k-nearest neighbor algorithm is a simple algorithm which matched unknown patterns of the data with a known pattern of the data based on the Euclidean distance between the two feature vectors. The known pattern of data with the lowest Euclidean distance from the unknown pattern of the data was determined to be the best match.

When given an unknown tuple, a k-nearest-neighbor classifier searches the pattern space for the k training tuples that are the closest to the unknown tuple. These k training tuples are the k “nearest neighbors” of the unknown tuple. “Closeness” is defined in terms of a distance metric, such as Euclidean distance. The Euclidean distance between two points or tuples, say, X1 = (x11, x12, …, x1n) and X2 = (x21, x22, …, x2n), is

dist ( X 1 , X 2 ) = i = 1 n ( x 1 i x 2 i ) 2
(1)

Typically, we normalize the values of each attribute before using Eq. (1). This helps prevent attributes with initial large ranges (e.g., intensity) from outweighing attributes with initial smaller ranges (e.g., position relation).Min-max normalization, for example, can be used to transform a value v of a numeric attribute A to v′ in the range [0, 1] by computing

v = v m i n A m a x A m i n A
(2)

where minA and maxA are the minimum and maximum values of attribute A.

“How can I determine a good value for k, the number of neighbors?” This can be determined experimentally. Starting with k=1, we use a test set to estimate the error rate of the classifier. This process can be repeated each time by incrementing k to allow for one more neighbor. The k value that gives the minimum error rate may be selected. For our experiments, the best k value equals to 2.

IV. EXPERIMENTS

In this section, we describe our experiments and then present and discuss our results for the activity recognition task.

4.1 Description of Experiments

Our experiments, first, require us to collect the labeled raw accelerometer data and then transform that data into examples. This process was described in Section 3. The resulting examples contain 7 features and cover 10 users. This forms the data set, which is subsequently used for training and testing.

The whole testing process was on the playground. At first, certain person gathered continuous standard data that each activity is performed for 30 seconds, entire five activities take 150 seconds in all. It is used as training data. By another 10 persons, it then gathered continuous test data which demands 1 minute per each activity, and entire five activities require 5 minutes in all. It is used as test data.

Once the raw data was gathered, first, we preprocess the data which includes removing the repeated data, linearizing and smoothing the data. After the data set was prepared, we used k-Nearest-Neighbor Classifiers algorithm to classify the data. Our experiment shows that the k-value set to 2 is the best.

4.2 Results

Figure5 shows the training raw data and the activity classification which is known. These data are classified as one of the standardized five states from “detached” to “jumping”. Here, we set their values from 1 to 5 which are the five activity types. And we let these training data serve as basic data, which are later used when they are compared with test data acquired.

jmis-2-2-207-g5
Fig. 5. Training data and activity classification
Download Original Figure

The summary results for our activity recognition experiments are presented in Table 1. According to training data and k-Nearest-Neighbor Classifiers algorithm, we predict the classification results of test data. After that, we compare with predictive results and realistic results, thereby obtaining the accuracy of the classification method.

Table 1. Accuracies of activity recognition
subject gender Detached Idle Walking Running Jumping
S1 F
S2 F
S3 M
S4 F
S5 M
S6 M
S7 M
S8 M
S9 M
S10 F
Download Excel Table

We describe the table’s contents which is the activity recognition in detail at below:

  • Phone Detached: The user is not currently holding the phone.

  • Idle: The user is holding the phone in an idle state.

  • Walking: The user is walking.

  • Running: The user is running.

  • Jumping: The user is jumping.

‘✓‘ means the activity can be identified accurately, ‘✗‘ means the activity cannot be identified accurately. For example, ‘S1’ can be identified accurately four activities which are “Phone Detached”, “Idle”, “Walking”, and “Jumping”, but it cannot be identified accurately “Running”. When he was running, the experimental results indicate that he was jumping.

Based on the results, we can exactly distinguish the phone detached, idle, walking and running. However, the algorithm cannot always distinguish running and jumping. We describe the results in detail as follows:

  • It cannot distinguish running and jumping in Figure 6.

  • It can distinguish running and jumping in Figure 7.

jmis-2-2-207-g6
Fig. 6. S6’s raw data and activity classification.
Download Original Figure
jmis-2-2-207-g7
Fig. 7. S10’s raw data and activity classification
Download Original Figure

About how to distinguish running and jumping, we plan to implement this correction in future work

V. CONCLUSIONS

We described how a smart phone can be used to perform activity recognition, simply by keeping it in one’s pocket. The method that we used can successfully get exact classified result except for distinguishing between running and jumping.

We plan to improve our activity recognition in several ways. The straightforward improvements involve: 1) Learning to recognize additional activities, such as descending stairs and climbing stairs. 2) Obtaining training data from more users with the expectation that will improve our results. 3) Generating additional and more sophisticated features when aggregating the raw time-series data. 4) Evaluating the impact of carrying the smartphone in different locations, such as on a belt loop.

The work described in this paper is part of a larger effort to mine sensor data from wireless devices. We plan to continue our project, applying the accelerometer data to other tasks besides the activity recognition and collecting and mining other sensor data, especially GPS data. We believe that mobile sensor data provides tremendous opportunities for data mining and we intend to leverage our Android-based the data collection/data mining platform to the fullest extent.

REFERENCES

[1].

Jiawei Han & Micheline Kamber. 2011. In Data Mining Concepts and Techniques, 3rd Edition.

[2].

Jennifer R. Kwapisz, Gary M. Weiss, Samuel A. Moore. 2010. Activity Recognition using Cell Phone Accelerometers. In ACM SIGKDD Explorations Newsletter archive Volume 12 Issue 2, December 2010 Pages, 74-82.

[3].

Qiang Li, John A. Stankovic, Mark Hanson, Adam Barth, John Lach. 2009. Accurate, Fast Fall Detection Using Gyroscopes and Accelerometer-Derived Posture Information. In Wearable and Implantable Body Sensor Networks, 2009. BSN 2009. Sixth International Workshop on, 138-143.

[4].

Sauvik Das, LaToya Green, Beatrice Perez, Michael Murphy. 2010. Detecting User Activities using the Accelerometer on Android Smartphones. In Sauvik Das — Georgia Institute of Technology.

[5].

Bao, L. and Intille, S. 2004. Activity Recognition from User-Annotated Acceleration Data. Lecture Notes Computer Science 3001, 1-17.

[6].

Brezmes, T., Gorricho, J.L., and Cotrina, J. 2009. Activity Recognition from accelerometer data on mobile phones. In IWANN ‘09: Proceedings of the 10th International Work-Conference on Artificial Neural Networks, 796-799.

[7].

Gyorbiro, N., Fabian, A., and Homanyi, G. 2008. An activity recognition system for mobile phones. In Mobile Networks and Applications, 14(1), 82-91.

[8].

Inooka, H., Ohtaki, Y. Hayasaka, H. Suzuki, A., and Nagatomi, R. 2006. Development of advanced portable device for daily physical assessment. In SICE-ICASE, International Joint Conference, 5878-5881.

[9].

Lester, J., Choudhury, T. and Borriello, G. 2006. A practical approach to recognizing physical activities. Lecture Notes in Computer Science: Pervasive Computing, 1–16.

[10].

A. K. Bourke and G. M. Lyons. A threshold-based falldetection algorithm using a bi-axial gyroscope sensor. Medical Engineering and Physics, 30:84–90, Jan 2008.

[11].

A. K. Bourke, J. V. O’Brien, and G. M. Lyons. Evaluation of a threshold-based tri-axial accelerometer fall detection algorithm. Gait and Posture, 26:194–199, 2007.

[12].

J. Chen, K. Kwong, D. Chang, J. Luk, and R. Bajcsy. Wearable sensors for reliable fall detection. In Proceedings of the 27th Annual International Conference of the IEEE EMBS, pages 3551–3554, Shanghai, China, Sept 2005. IEEE.

[13].

R. K. Ganti, P. Jayachandran, T. F. Abdelzaher, and J. A. Stankovic. SATIRE: A software architecture for smart attire. In Proceedings of MobiSys’06, Uppsala, Sweden, June 2006. ACM Press.

[14].

M. Kangas, A. Konttila, I. Winblad, and T. Jämsä. Determination of simple thresholds for accelerometry-based parameters for fall detection. In Proceedings of the 29th Annual International Conference of the IEEE EMBS, pages 1367–1370, Lyon, France, Aug 2007. IEEE.

[15].

Mathie, M., Celler B., Lovell N., and Coster A. 2004. Classification of basic daily movements using a triaxial accelerometer. In Medical & Biological Engineering and Computing, 42.

[16].

Anderson, I., Maitland, J., Sherwood, S., Barkhuus, L., Chalmers, M., Hall, M., Brown, B., and Muller, H. 2007. Shakra: Tracking and sharing daily activity levels with unaugmented mobile phones. In Mobile Networks and Applications. 12(2-3).

[17].

Weiss, G. M., and Hirsh, H. 1998. Learning to predict rare events in event sequences, In Proceedings of the Fourth International Conference on Knowledge Discovery and Data Mining, AAAI Press, Menlo Park, CA, 359-363.

[18].

Maurer, U., Smailagic, A., Siewiorek, D., & Deisher, M. 2006. Activity recognition and monitoring using multiple sensors on different body positions. In IEEE Proceedings on the International Workshop on Wearable and Implantable Sensor Networks, 3(5).

[19].

Choudhury, T., Consolvo, S., Harrison, B., LaMarca, A., LeGrand, L., Rahimi, A., Rea, A., Borriello, G., Hemingway, B., Klasnja, P., Koscher, K., Landay, J., Lester, J., Wyatt, D., and Haehnel, D. 2008. The mobile sensing platform: An embedded activity recognition system. In IEEE Pervasive Computing, 7(2), 32-41.

[20].

Cho, Y., Nam, Y., Choi, Y-J., and Cho, W-D. 2008. Smart-Buckle: human activity recognition using a 3-axis accelerometer and a wearable camera. In HealthNet.

[21].

Miluzzo, E., Lane, N., Fodor, K., Peterson, R., Lu, H., Musolesi, M., Eisenman, S., Zheng, X. and Campbell, A. 2008. Sensing meets mobile social networks: The design, implementation and evaluation of the CenceMe application. In The 6th ACM Conference on Embedded Networked Sensor Systems, 337-350.

[22].

Yang, J. 2009. Toward physical activity diary: Motion recognition using simple acceleration features with mobile phones, In First International Workshop on Interactive Multimedia for Consumer Electronics at ACM Multimedia.