Combining low sampling frequency smartphone sensors and video for a Wearable Mobility Monitoring System [version 2; referees: 1 approved, 1 approved with reservations]

A proof-of-concept Wearable Mobility Monitoring System (WMMS) was developed to identify daily activities and provide environmental context, using integrated BlackBerry Smartphone low sensor and video data. Integrated accelerometer data were used to identify mobility changes-of-state (CoS) in real-time, trigger BlackBerry video capture at each CoS, and save activity outcomes on the Smartphone. System evaluation involved collecting WMMS output and (separate) camcorder video under realistic conditions for five able-bodied subjects. The subjects each performed a consecutive series of mobility tasks; including, walking, sitting, lying, stairs, ramps, elevator, bathroom activities, kitchen activities, dining activities and outdoor walking. Activity, timing and contextual information were obtained from the camcorder for comparison. Sensitivity results for sensor-based CoS identification were 97-100% for standing, sitting, lying and taking an elevator; 67-73% for walking-related CoS (stairs, ramps); 40-93% between walking and small movements (brushing teeth, etc.); and below 27% for daily living activities. False positives occurred in less than 12% of all activities, with less than 5% false positives for half the measures. Better classification results were achieved when using both acceleration features and Smartphone integrated video for all activities except sitting. The evaluation demonstrated that the WMMS algorithm and BlackBerry platform were effective for detecting mobility activities, even with low sampling rate sensors. The combined sensor and video analysis enhanced mobility task identification and contextual information.


Introduction
Understanding how people move within their daily lives is important for healthcare decision-making.Typically, a person's mobility status is self-reported in the clinic, thereby introducing error and increasing the potential for biased information.Functional scales can be administered to help gain an understanding of a person's mobility status [1][2][3][4] .However, these tests do not measure how a person moves when leaving the clinic.A quantitative method of characterizing mobility in the community would provide valid and useful information for healthcare providers and researchers.
Wearable systems have been developed to evaluate mobility in any environment or location.These portable systems are worn on the body and collect mobility information within the home and community.Examples include accelerometer-based systems [5][6][7][8] portable electromyographs (EMG) 9 , and foot pressure devices 10 .Some wearable systems can be used for long-term and low effort monitoring.
Smartphones provide an ideal interface for mobility assessment in the community since they are small, light, easily worn, and easy to use for most consumers.These phones are multitasking, computing platforms that can incorporate accelerometers, GPS, video camera, light sensors, temperature sensors, gyroscopes, and magnetometers 11,12 .
As shown in Table 1, cell phones have been used to recognize multiple activities by analyzing the phone's accelerometer or external sensor data.However, most systems only used the phone as a wireless

Amendments from Version 1
Following the comments from reviewer #2, comments regarding the GPS in the methods section were removed since although the WMMS algorithm includes processing the GPS signal, the GPS signal was not used for this particular proof-of-concept study.
In addition, subsequent to the comments from reviewer #2, a highlevel overview flowchart of the WMMS algorithm was included as Figure 1.

See referee reports
data transfer device, not as a wearable computing platform.These systems used an external computer for analysis of the data.
Wearable video-sensor systems have also been developed to log digital memories.Video, GPS, electrocardiogram, and acceleration were used to record location and context of a person's daily life [16][17][18] .These Lifelogs had several limitations, including information retrieval, synchronization, light quality, low visual resolution, artifacts, and automatic annotation.
A previous project showed that sensors located in a "smart-holster" could be combined with a BlackBerry Smartphone to provide image-assisted mobility assessment 5 .An external accelerometer, temperature sensor, humidity sensor, light sensor, and Bluetooth were integrated into the Smartphone holster.Software was written for the BlackBerry 9000, which had an embedded camera and GPS.A hierarchical decision tree combined with a double threshold algorithm classified signals to recognize changes-of-state (CoS) and activities.The BlackBerry then automatically took a digital picture at each CoS.This preliminary work confirmed that a wearable mobility monitoring system (WMMS) could use acceleration and pictures to identify walking movements, standing, sitting, and lying down, and additionally provide context for these activities.
Subsequent research 19 used the BlackBerry 9550 Smartphone, with an integrated accelerometer, and revised algorithms to detect changes of state (CoS) and classify activities of daily living.The low sampling rate of the BlackBerry 9550 required some of the algorithm for the first WMMS to be revised.Having all sensors and computing power within the Smartphone provides a broadly accessible platform for wearable activity monitoring.A single subject case study resulted in an average sensitivity of 89.7% and the specificity of 99.5% for walking-related activities, sensitivity of 72.2% for stair navigation, and sensitivity of 33.3% for ramp recognition.Ramp results were poorer since accelerations for ramp gait were similar for walking gait.
Since it was demonstrated that new BlackBerry Smartphones can identify CoS in real-time 19 , and Smartphone video capture is available on these devices, a preliminary evaluation of the usefulness of wearable video for improving activity classification and context identification is needed.The present study builds on this prior work and presents a proof-of-concept evaluation of a new BlackBerry-based WMMS that uses internal sensors and cell phone video to identify a range of mobility and daily living activities by using the CoS to trigger the acquisition of short video clips.A proof-of-concept evaluation is a necessary step before a large scale evaluation with people with disabilities.This study is the first to integrate sensor and (non-automated) video analysis for wearable mobility monitoring.

WMMS system
The prototype WMMS was developed for BlackBerry OS 5.0 on the Storm2 9550 Smartphone.The WMMS used the BlackBerry's integrated accelerometer (3-axis, ± 2g), GPS, camera, and SD memory card for mobility data collection.The 3.15 MP camera had a maximum video resolution of 480×352 pixels and a video frame rate of 30 frames per second 20 .During video capture, no accelerometer or GPS data were available.With video control active, the accelerometer sampling rate was approximately 8 Hz 21 .The WMMS was worn in a holster on the right, front pelvis.
The new WMMS software developed in 22 , sampled time, acceleration, and GPS location and then saved the raw data to a 16Gb SD-card.Accelerations were processed to calibrate the axis orientation, using the gravity rotation method 4 .The processed acceleration was input into a feature extraction algorithm 19 for analysis within 1-second data windows.Following feature extraction, the features were analyzed in a decision tree to determine if a CoS had occurred.If a CoS was identified, a three-second video clip was captured (Figure 1).A second decision tree was used to categorize the activity 19 .Time, features, and activity classification were saved to an output file on the SD card.
( 2 1) arctan ( ) In equation (1) to equation (7), m y is the mean Y-acceleration, y i is the individual acceleration value, N is the samples per data window, SR2 is the sum of ranges in current window, SR1 is the sum of ranges in previous window, and finally m z is the mean Z-acceleration.

Change of state/classification
The CoS algorithm described in 19 used pre-set thresholds for feature analysis.Each feature was independently analyzed using single or double-thresholds and scored as true or false.These Boolean values were combined to recognize the activities reported in 19 (static state, taking an elevator, walking-related movements) and also to identify changes of state for small movements during activities of daily living (ADL) (meals, hygienic activities, working in the kitchen).CoS classification ran in real-time to enable accurate Smartphone video recording.

Activity classification
The decision tree for activity classification, described in 19 , used the same features and thresholds as the CoS algorithm to classify eight activities: sitting, standing, lying, riding an elevator, small standmovements, small sit-movements, small lie-movements, and walking.The small movement categories included ADL and movements while sitting and standing.Following data collection, the video clips were manually reviewed by a human operator to help classify activities.Clips were played back on a BlackBerry 9700 phone.Video and audio were available for qualitative assessment of the current activity.
Test procedure A convenience sample of five able-bodied subjects (four males and one female, age: 35.2 ± 8.72 years, height: 174.58 ± 10.75 cm, weight: 66.46 ± 9.7 kg) were recruited from the Ottawa Hospital Rehabilitation Center (TOHRC, Ottawa, Canada) for the proof-of-concept evaluation.A form that included informed consent, information sheet, and media agreement was obtained for each subject.Ethical approval for the study was obtained from the Office of Research Ethics and Integrity, University of Ottawa, File Number: A08-12-01.Subjects who had injuries or gait deficits were excluded.
Data collection took place within the TOHRC (hallways, elevator, one bedroom apartment, stairs, Rehabilitation Technology Laboratory) and outside on a paved pathway.Participants wore the WMMS on their right-front waist, in a regular BlackBerry holster attached to a belt, with the camera pointing forward.No additional instructions were given for WMMS positioning.The participants were asked to follow a predetermined path and perform a series of mobility tasks, including, standing, walking, sitting, riding an elevator, brushing teeth, combing hair, washing hands, drying hands, setting dishes, filling the kettle with water, toasting bread, a simulated meal at a dining table, washing dishes, walking on stairs, lying on a bed, walking on a ramp, and walking outdoors.Each activity in the sequence took approximately 10 to 20 seconds to complete.Each trial had 41 changes of state.Three trials were captured on the Smartphone and on a digital video camcorder.
Activity timing was obtained from the camcorder recording for comparison with the WMMS output.Start and end points of each trial were identified by shaking the Smartphone for 2 seconds.The camcorder also provided contextual information for analysis.
Data were imported from the Smartphone SD card into Microsoft Excel for statistical analysis.The Smartphone video clips were reviewed by two independent evaluators to qualitatively classify activities and assess video quality.The evaluators had access to the WMMS activity classification results during video classification.

Results
The results were analyzed in terms of CoS and activity classification.For CoS identification at the transition point between activities (Table 2), sensitivities for standing, sitting, lying, and taking an elevator were between 97% and 100%.Walking-related CoS, such as stairs and ramps, had 67% to 73% sensitivity.Sensitivity 5 related to CoS between walking and small movements, such as brushing teeth, were between 40% and 93%.The CoS results for daily living activities were poorer (below 27%) since the continuous series of small movements produced similar acceleration features.
Table 3 shows the specificity 5 results for activity classification using the CoS algorithm 19 .These classifications are compared between data windows for CoS identification.The number of false positives was less than 12% for all measures, with half the measures reporting less than 5% false positives.Walking produced the most false positives, identifying a walking-CoS when no CoS occurred, for 324 out of 2700 cases (12%).Large standard deviations were found for some measures due to timing differences (i.e., people who walked slower took more time and had more data points for analysis, leading to more true negatives).
Better activity classification results were achieved when using both acceleration features and video clips for all activities except sitting, as compared to using the accelerometer only (Table 4).Acceleration-only analysis with the activity classification algorithm 19 had a greater sensitivity than acceleration-and-video for sitting because one CoS was missed in one trial, resulting in no associated video data.Using the accelerometer only, none of the ADLs were identified; however, no false positives were reported, resulting in a high specificity.
Since accelerometer data were unavailable during video recording, no acceleration could be recorded for a minimum of 2.9 seconds after a CoS was identified.For example, if a person sits in a chair,  The BlackBerry 9550 included a light sensor, but the sensor output was unavailable with Java API 5.0.Light intensity level could be beneficial for recognizing indoor and outdoor environments.
As shown in Figure 2, the extracted features were used to recognize small movements, identified using a combination of SR and STD_Y.Discrete small movements were easily distinguished; however, a CoS could be missed for a continuous series of small movements stands up, and sits down again within 4 seconds (3 seconds of video capture and 1-second data window), the CoS will not be detected or identified.However, digital video would be available during this period for activity classification, enabling the evaluator to see the additional movements during post-processing.The average accelerometer sampling frequency was 7.88 ± 1.39 Hz.Video analyses became difficult in some circumstances, in particular when video images were dark.that has similar accelerometer features.For example, brushing teeth and then combing hair.
The sum of ranges was more sensitive than the STD_Y for distinguishing mobility states (Figure 3).Further, the SMA-SR curves were smoother than the sum of ranges curves.Smooth SMA-SR curves were better at defining classification thresholds for climbing stairs and walking.
Recording a video was a valuable medium for evaluating mobility activities since audio information was available and multipleimages could be used in the analysis (Table 5).The sense of motion also provided useful information for categorizing the activity and context.
Activity recognition for static and walking states was confirmed by video-clip evaluation.Occasionally, the video showed a movinghand during walking, as the arm swung during locomotion.Since the phone was located on the right waist, the video usually showed more of the right side than the left.For ramp navigation, the Smartphone video was not able to show the ramp, however, the video did allow the evaluator to determine if the person was ascending or descending.Similarly, the video was unable to show the stairs when descending, but the downward movement and sound of the footfalls on the stairs allowed the evaluator to categorize the stair descent activity.
The elevator CoS was usually triggered by a transition from walking to standing, rather than from elevator movement.The activity was classified using the video by seeing the elevator door or the inside of the elevator in the video-clip.The elevator inside was unclear and dark, but the elevator sound was clearly defined.Further, the bathroom, kitchen, and dining room areas were clearly identified from video analysis.
Small movements, such as brushing teeth, combing hair and moving plates could not be identified because these activities were out of the Smartphone video field.Some clips could clearly identify moving a kettle, toasting bread, meal preparation, and washing dishes (Figure 4).Furthermore, drying hands and meal preparation needed continuous video images to identify the activities (i.e., single images would not be useful).Moreover, the video-based activity classification was hampered for people with shorter lower bodies when a tabletop or kitchen counter obstructed the phone.For example, moving a kettle and toasting bread were not visible in the video field for these shorter participants.The Raw-acceleration-file contains the raw accelerations as output by the BlackBerry Smartphone during all trials.The Raw-data-files (subjects 1-2-3-4-5) contain the signal features calculated from the accelerations plus the 'real status' of the subject (what they were doing at that instant) during each sample of each trial.The real status is required to verify the accuracy of the wearable mobility monitoring system (WMMS) prediction.

Discussion
This study demonstrated that BlackBerry accelerometer signal analysis and cell phone video assessment can be combined to identify many mobility activities and the context of these activities.Since a readily available Smartphone was worn in a typical manner, at the waist in a holster, and no external sensors or hardware was required, the WMMS could be easily implemented in the community.
The ability to appropriately identity a CoS is critical for a WMMS that uses video, with accurate and real-time CoS identification needed to trigger the acquisition of video clips at the appropriate time to enable post-processing.The WMMS successfully recognized CoS for both static and walking activities.By taking the initial activity classification from the sensor data and refining the classification decision using the BlackBerry video, superior activity recognition results were obtained.
Static activity (sitting, standing, lying) classification accuracy improved from outcomes in the literature (below 94.6%, 5,13,23 ) to 97.3%.The results would have been 100% except one CoS was missed during a walk-to-sit CoS of one of the trials.More thorough biomechanical analysis is required to identify the movement pattern that created this outlying error.Standing and lying down were recognized with 100% accuracy.
The WMMS produced better sensitivity results for walking than the previous Smartphone-smart-holster system 5 .While various studies in the literature had better walking accuracy (above 96% 7,23 ), these studies only distinguished differences between very distinct mobility states, such as running, jumping, and no-movement.Accuracy of these systems would likely reduce if they were categorizing similar activities, such as running on level ground and running up an incline.
Stair navigation identification improved from 60% in the literature 5,13 to 75% with the new WMMS.However, the result did not reach the 95% target.Acceleration-based categorization remains difficult for stairs since the signal patterns between walking on level ground and stairs are similar, and the stair slope may not greatly change accelerations at the waist for able-bodied people.
CoS identification for ramp walking was enhanced from 43.3% for the previous system 5 to 66% because the new WMMS used a specific ramp judgment within the decision tree.Ramp classification also improved from 16.6% 5 to 43.5%.Accelerometer signals from able-bodied people do not always change when moving from level ground to an incline, however, accelerometer signals from people with mobility disabilities may be sufficiently distinct to enable consistent ramp CoS triggering.During ramp descent, classification errors may occur if a walking CoS is triggered before the person finishes descent and the waist-mounted video does not show the ramp bottom.An altitude sensor may help detect changes in body vertical position during ramp and stair navigation, thereby providing more accurate CoS triggering and sensor-based classification.
Specificity is also important for a WMMS that incorporates video since identifying a CoS when no change actually occurs (i.e. a false positive) results in inappropriate video capture that affects storage capacity, battery life, and increases the workload for video postprocessing.Most prior research did not report specificity or false positive results [13][14][15] .For the two studies that reported false positives, the maximum false positive rate was above 12% for walking-related activities 5,24 .Improved threshold calibration methods that are specific to the individual may help to decrease false positives and false negatives.
The BlackBerry device was able to process the features and algorithms in real-time and save outcome data for each 1-second window.
Most mobility research used cell phones to collect raw data and then analyzed features and algorithms offline [13][14][15]25 . Forthe new WMMS, threshold settings, decision trees, feature extraction and timing were executed on the Smartphone in real time.
The combination of accelerometer and video camera was superior to using the accelerometer data alone to provide mobility information for activities of daily living.Bathroom, kitchen, and dining room activities were recognized by videos in 63% to 94% of the cases.Although the combination of accelerometer and video could not consistently identify small ADL movements, this method had better accuracy than the accelerometer only, which could not categorize any of these ADLs.Further, CoS for small movements could not be clearly identified.A higher accelerometer sampling rate may allow for more complex signal analysis that could improve small movement CoS identification.Additional features, such as signal magnitude area, skewness 5 , energy, correlation 26 , and kurtosis 27 , might help to recognize the small movement CoS, but these features need greater sampling rates and/or greater accelerometer range.
The video was better than still images for refining activity classification and recognizing context, such as flooring type, bathroom/kitchen, and outdoors.From previous research 5 , still images had limitations in dark areas such as elevators, but video clips with audio were helpful for identifying an elevator and other states.Since videos had continuous images to distinguish upward and downward movement, these clips were useful for stairs and ramp navigation.
Although video cameras could potentially recognize small ADL movements, the waist-mounted Smartphone location limited the camera to activities occurring about waist height.
While BlackBerry video proved useful, the current implementation requires human time to review and classify the activities.Manual video assessment could be used for research and specific clinical applications; however, automated video analysis would make the system more efficient and promote widespread use of video data for activity classification.Further research into automated video analysis is required to achieve this goal.

Conclusions
Building on previous work that demonstrated the BlackBerry's ability to identify changes of state in real-time, a new WMMS was presented that uses only the Smartphone's accelerometer and video for unsupervised and ubiquitous mobility analysis for research and healthcare applications.This study was the first to integrate sensor and video analysis for wearable mobility monitoring.This sampling rate for the phone sensors was lower than used in previous studies, demonstrating that a standalone WMMS can be implemented even for older-hardware phones.This standalone WMMS was designed for independent community ambulation and has minimal space requirements and setup time.Furthermore, the context of a person's mobility activities can be identified, including the environment in which mobility takes place.
While monitoring mobility, the new system saves raw sensor data and features.The raw data could be analyzed by other researchers or used in post-processing to improve activity classification.Novel multi-feature algorithms were developed to recognize activities under lower accelerometer sampling rates and range.By combining and weighting sum, range, and covariance statistics, the WMMS was able to recognize standing, sitting, lying, riding an elevator, walking on level ground, ramps, stairs, and ADL (washing hands, drying hands, setting dishes, moving a kettle, toasting bread, preparing a meal, and washing dishes).Static activities, walking on level ground, walking on stairs, walking on a ramp, and riding an elevator had higher sensitivities than previous studies, but the overall CoS identification and activity classification performance could be improved by further research.
Adding additional sensors, increasing the accelerometer sampling rate/sensitivity, and adding new user-specific threshold calibration methods can be considered in future work.The classification of other small movement ADL activities also requires further research to increase sensitivity and specificity.While evaluation with ablebodied participants was warranted for this proof-of-concept study, further evaluation research with various patient populations is required before this WMMS can be accepted for use by people with mobility disabilities.

Consent
Informed written consent to publish the results of this study was obtained from each participant.

F1000Research
detecting a transition and subsequent fall.
It appears that the accelerometer data could distinguish very little of the activities with manual video observations proving key and the only suitable means to distinguish tasks.

: Discussion
In a few instances the authors refer to "sounds" -perhaps this might be a better tool to help distinguish activities in an automated system?I have read this submission.I believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.
No competing interests were disclosed.

Competing Interests:
Author Response 28 Feb 2015 , University of Ottawa, Canada Natalie Baddour Thank you for taking the time to read the manuscript.In response to comments: The abstract and introduction both make reference to the fact that this is a proof-of-concept study.
We have uploaded a new version of the manuscript removing the reference to the GPS in materials and methods.The new version of the manuscript also includes a figure of the high-level overview of the WMMS algorithm.
Although it was requested by reviewer #2, we have not included the figure of the decision tree used in the WMMS algorithm in the new version of the manuscript.Such a figure would not be meaningful without explanation, which would greatly lengthen the manuscript.The details of the algorithm are given in reference #19, the focus of which was the algorithm.The focus of this manuscript was on the evaluation of the algorithm.Correspondingly, the circuit used to evaluate the algorithm was designed to include activities from "real life" such as hair-combing and teeth-brushing.The intent was not for the a waist-worn WMMS system to identify these activities but rather to see how the WMMS algorithm would perform once these everyday (small movement) activities were included.We felt that this would be be a more realistic test of the proof-of-concept WMMS, rather than investigating the performance of the algorithm with only a limited set of activities.
The comment about the use of the inclination angle to distinguish sitting from standing likely causing problems with older or heavier adults is quite astute.This was, in fact, our experience and an algorithm to correct for the larger girth of some waists has been developed and incorporated into subsequent versions of the WMMS.M.  As can be anticipate some activities are easy to classify; others with movement overlap are more difficult.Data are provided which can allow the work to be progressed by other researchers.Whilst the individual elements of the work are not novel (accelelerometry for activity monitoring and video for life logs and reminiscence), the combination and the superior performance of the sensor modalities provide a useful contribution.

Abstract:
This provides good orientation for the reader and cites the headline results.The sensitivity of 27% for daily living activities is low and show the challenge of higher level classification.Overall activity and mobility (changes of state, CoS) may however be more relevant features anyway (in a subsequent referral).The sample number (N=5) is small and is described as a convenience sample, for this initial feasibility study.

Material and methods:
The WMMS system is based on a Blackberry OS5.0.There is no mention as to whether this can be easily implemented on other smart phones (Android and IOS).Accelerometer sampling rate of 8Hz is appropriate but may miss faster transitions.The feature extraction algorithm and decision tress rely on cited literature.These are key components of the activity classification used for CoS detection.The use of thresholds usually makes it difficult to generalize to the wider population (male/female, height, weight etc.) for accelerometry.

Results:
"Better activity classification results were achieved when using both acceleration features and video clips for all activities except sitting, as compared to using the accelerometer only" This is the crux of the paper and the results underpin this.Of course this is a relative measure.It's difficult to determine what would be a clinically useful measure.
The figures are useful and inform the reader as to the difficulty of differential classification between some of these activities.
There is some useful reflection on the use of video.The time required to manually classify the video is a limiting feature.

Conclusions:
The conclusions are realistic and pave the way for further research questions.In particular it may be possible to improve the classification algorithms and the publication of the data sets is advantageous.The Blackberry has a restricted market and the evolution of smartphone hardware and software continues to advance rapidly.Porting of the algorithms to other operating systems and provision source code would enable further refinement of the approach.

Figure 3 .
Figure 3. Top: Sum of ranges (SR) and STD_Y to distinguish walking and static states (BT=brushing teeth, CH=combing hair, WH=washing hands, and DH=drying hands).Bottom: SMA-SR to identify climbing stairs and walking.

Figure 4 .
Figure 4. Images from BlackBerry video for small movements: a) kitchen cabinet and oven, b) bathroom sink, c) dining table and meals, d) moving a kettle, e) toasting bread, f) meal preparation, g) washing dishes, h) towel and towel rack, i) drying hands.
Institute, University of Ulster, Newtownabbey, UK The paper: "Combining low sampling frequency smartphone sensors and video for a Wearable Mobility Monitoring System", examines the feasibility of combining the accelerometry and video, available in a commercial smart phone for classifying activities of living in the home.

Table 1 . Studies that use a tri-axial accelerometer for mobility monitoring.
1 Location = sensor location on the person 2 Freq = data acquisition frequency

Table 2 . CoS sensitivity during the transition between activities. Changes of State Subject 1 Subject 2 Subject 3 Subject 4 Subject 5 Sensitivity
2 FN = average false negatives

Table 4 . Activity classification sensitivity and specificity for accelerometer only and accelerometer with Smartphone video.
1 Acc.= accelerometer 2 SE = sensitivity 3 SP = specificity