Journal of Multimedia Information System
Korea Multimedia Society
Section A

Collocated Wearable Interaction for Audio Book Application on Smartwatch and Hearables

Hyoseok Yoon1,*, Jangmi Son1
1Division of Computer Engineering, Hanshin University, Osan-si, Gyeonggi-do, Korea,,
*Corresponding Author : Hyoseok Yoon, 137 Hanshindae-gil, Osan-si, Gyeonggi-do, 18101 Korea, +82-31-379-0645,

© Copyright 2020 Korea Multimedia Society. This is an Open-Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License ( which permits unrestricted non-commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

Received: Jun 10, 2020; Accepted: Jun 20, 2020

Published Online: Jun 30, 2020


This paper proposes a wearable audio book application using two wearable devices, a smartwatch and a hearables. We review requirements of what could be a killer wearable application and design our application based on these elicited requirements. To distinguish our application, we present 7 scenarios and introduce several wearable interaction modalities. To show feasibility of our approach, we design and implement our proof-of-concept prototype on Android emulator as well as on a commercial smartwatch. We thoroughly address how different interaction modalities are designed and implemented in the Android platform. Lastly, we show latency of the multi-modal and alternative interaction modalities that can be gracefully handled in wearable audio application use cases.

Keywords: Audio book; Collocated interaction; Hearables; Wearables


Wearables market is continuously on the rise, paving the way for wearable devices to become a strong candidate as a future computing platform in the post-smartphone era. Wearables market is promising due to the increasing demand for various wearable applications covering users’ changing needs in medical, infotainment and fitness tracking domains. Especially wearable devices such as smartwatches and hearables are demonstrating commercial and consumer-level success. A market analysis forecasts that wearables technology market will reach $57.49 billion by 2025 with a CAGR (Compound Annual Growth Rate) of 9.9% during 2020 to 2025 [1]. Another forecast reports that wearable device user penetration is 5.9% in 2020 and will remain around 4.8% by 2024 even with the negative impact of COVID-19 [2]. Global smartwatch growth YoY (Year-over-Year) is 20.2% led by global brands such as Apple, Samsung and Garmin [3]. According to the International Data Corporation [4], demand for hearables helped the worldwide wearable markets to surge 94.6% during the third quarter of 2019. According to this source, top 5 wearable units include earwear (48.1%), wrist band (22.7%), smartwatch (20.9%), basic watch (7.1%) and other (1.3%).

Still, there is a skeptical view concerning what could be the ultimate killer application for wearables. Currently, various academic and industrial attempts are exploring stand-alone wearable applications [5] as well as cross-device interactive applications [6] for eXtended Reality (XR) that cover Mixed Reality (MR), Virtual Reality (VR) and Augmented Reality (AR). In the light of COVID-19, wearables are deployed to a new use case of tracking social distancing and employees at work [7].

In this paper, we propose a wearable audio book application based on collocated interaction using a combination of smartwatch and hearables.

Our contributions on this paper are as follows.

  • We describe our rationale behind this application and propose our approach as a promising killer application on wearables.

  • We identify 7 scenarios for wearable audio book applications that justifies use of our application over other platforms (i.e., smartphones or tables).

  • We demonstrate feasibility of our approach by developing a working prototype and demonstrating performance of interaction modalities in terms of interaction latency.


We review related work in emerging technology for innovative wearables and audio-book applications.

2.1. Emerging Technology for Wearables

Recently cross-device interaction with wearables and other smart devices is studied in many areas. Weiss et al. used smartphone and smartwatch to measure biometrics using activities of daily living (ADL) [8]. Bi et al. studied identifying pen-holding gestures with a smartwatch [11]. Turner et al. compared input methods for typing on a smartwatch [12]. Buddhika et al. used smartwatch PPG sensor for gesture interactions [13]. Others are focusing on potential of hearables to augment new capability [15], physiological in-ear sensing [16] and in-ear health monitoring [17]. There are movements to provide gaming experience using smartwatch [9], applying smartwatch design fundamentals [10], and distributing smartwatch components for various scenarios [14].

2.2. Audio Book Applications

There are several subscription-based audio book applications and services to be used in mobile and PC platforms. A partial list include Welaaa (, Storytel (, Millie (, Audible (, Google Play Book and Some of these services provide a wearable application, yet most services operate and target on smartphone applications.


We design and implement a Wearable Audio Book Application (WABA) based on a concept of collocated interaction [20] with a smartwatch with an earwear (i.e., hearables). In this section, we present our rationale for design considerations and analyze requirements for the WABA.

3.1. Design Considerations

We have elicited criteria for what could be the ultimate killer application for wearables. To do so, we have studied and reviewed previous wearable applications such as text-entry applications [18][19] and input modalities for wearables [21][22][23]. Furthermore, we conceived the idea of transforming popular smartphone apps to counterpart wearable apps.

We have observed following characteristics that formed the basis of our design on the WABA.

  • An application that requires highly expressive input is not suitable for wearables. For example, messenger apps need a user to perform text entry on a small touchscreen. This kind of applications require expressive input capability and there exists a better alternative input modality such as voice-based input.

  • An application that requires a user’s intentional and focused attention consistently is burdensome. Unlike smartphones, wearable devices such as smartwatches should be used as “glanceable standalone devices” for completing quick tasks [24].

  • Video-based media applications may work well with head-mounted displays (HMD) for virtual reality (VR) experience. However, wrist-worn wearables are not fit for such use and HMD will also face high competition from portable smartphones and TVs with much spacious screen.

  • We have observed a momentum is building up on medical and fitness applications for wearables. Indeed, this is a promising area where the wearables will excel. This type of applications does not require high expressivity in inputs and work well with glanceable interactions.

  • We have identified a trendy and plausible approach using collocated interaction and cross-device computation offloading [25]. Notably, sales of hearables have surged recently.

To reflect aforementioned observations, we designed the WABA to provide simple expressions in inputs, less attentive (i.e., glanceable), audio-based media use case and exploit the cross-interaction and collocated interaction with a smartwatch with a hearables.

3.2. Requirements

In this section, we describe how previous design considerations are reflected on the WABA in various forms of functional and non-functional requirements.

First, the WABA operates on simple expressions in wearable inputs. We simplified inputs on the WABA that uses traditional touchscreen inputs and multi-function side buttons. An audio book application can be designed to provide a simple set of functions (i.e., read, pause, volume control) that is mapped to the wearable input modality.

Second, the WABA works in less attentive or glanceable fashion. An audio book can be played as a background service that can continuously persist. Once the WABA is launched on the smartwatch, it does not require the wearer’s full attention, unless the user wants to directly control the app.

Third, we focus on audio-based media use case. The WABA provides media control functions (i.e., play, stop, fast-forward) as well as other audio book related functions (i.e, read in 2x speed, skip 15 seconds forward/backward, previous/next chapter). As discussed earlier, the WABA targets reading audio books using simple and glanceable interactions.

Lastly, the WABA benefits from cross-interaction and collocated interaction with a hearables. We envisioned the WABA to be used with a Bluetooth paired hearables to serve a natural audio book application. When a smartwatch is paired with a hearables, input and output modalities are decoupled and distributed to two devices.

3.3. Use Cases and Scenarios of the WABA

To successfully claim the killer application for wearables, there needs to be designated use cases and scenarios that are distinguished from other devices (i.e., smartphones). We identified several compelling use cases and scenarios for the WABA. The WABA is applicable to people commuting (Figure 1) and jogging/running (Figure 2) as well as accompanying them when they work out (Figure 3). In these scenarios, people can concentrate on their main task (i.e., keeping eyes on walking, running and exercising) while keeping their hands free of smartphones and opening their ears for the WABA.

Fig. 1. Scenario 1: Commuters scenario.
Download Original Figure
Fig. 2. Scenario 2: Running and jogging scenario.
Download Original Figure
Fig. 3. Scenario 3: Workout scenario.
Download Original Figure

Additionally, the WABA is useful for employees working from home (WFH) or freelancers with a designated workspace as shown in Figure 4. For example, people can still use their laptops while using the WABA without any interference. Moreover, the WABA could replace radio and in-vehicle infotainment when driving a vehicle or doing housework in the home as shown in Figure 5 and Figure 6 respectively.

Fig. 4. Scenario 4: Work from home (WFH) scenario.
Download Original Figure
Fig. 5. Scenario 5: Driving a car scenario.
Download Original Figure
Fig. 6. Scenario 6: Doing a housework scenario.
Download Original Figure

Figure 7 demonstrates a multi-user scenario. For parents, an audio book for children can be played by the parents’ smartwatch while the audio book is played on the child’s hearables. All these scenarios make wearables (i.e., smartwatch and hearables) a better choice than smartphones or tablets or PCs.

Fig. 7. Scenario 7: Audio book for children scenario.
Download Original Figure


We designed and implemented a proof-of-concept (PoC) of the WABA using Android development platform. Android Studio was used to design the layout of the WABA and we programmatically implemented functions of the WABA using Android SDK APIs. The PoC application was tested on an Android Virtual Device (AVD) as well as on a commercial smartwatch running WearOS.

4.1. Home Screen of the WABA

Figure 8 shows the WABA when it is first launched. It has 3 main touch-based control areas as highlighted in Figure 8. Android ImageView widgets are implemented as touch-based controls. Two arrows highlighted in two red dotted boxes are used to navigate between chapters or between different books. The speaker icon highlighted in the yellow dotted box shows mute on and off toggle control. The green dotted box shows an area designated for audio book controls such as play, pause, stop and play in 2x speed. Two functions not shown in the home screen are skipping 15 seconds forward and backward. These two functions will be triggered by other means of input modalities such as using multi-function side button or using gestures. In the center of the home screen, a book cover for currently playing audio book is displayed.

Fig. 8. Home screen of the WABA.
Download Original Figure
4.2. Wearable Interaction Modalities

The WABA uses four different wearable interactions modalities. First modality is traditional touchscreen-based interaction used on the touchscreen. This modality is useful when the user is attentive to look down on the smartwatch. Second modality is the multi-function side buttons on the smartwatch as shown in Figure 9. These multi-function buttons are programmable and provide tactile feedback when clicked. Third modality is contactless modality such as gesture-based inputs. Lastly, cross-device input modality is supported by receiving touch-input from the hearables to the smartwatch by Bluetooth communication. All these four input modalities summarized in Table 1, have low input expressivity, yet they are expressive enough to operate an audio book application when their multi-modal inputs are enabled.

Fig. 9. Multi-function side buttons on Fossil Q Explorist.
Download Original Figure
Table 1. Wearable input modalities employed by the WABA.
Wearable Input Modality Contact Expressivity
Touchscreen Screen Low
Multi-function Button Side Button Low
Gestures Contactless Low
Cross-device Input Modality Hearables Low
Download Excel Table
4.3. Implementation of the WABA

We implemented the WABA prototype on Android emulator and commercial smartwatch Fossil Q Explorist as shown in Figure 10. We paired the smartwatch with a Bluetooth earphone (QCY QY19-YL) to listen audio books through the connected hearables. To implement audio book controls, we used Android MediaPlayer class to play, pause, stop and speed-play audio books in mp3 format. For audio books, we downloaded free audio books from IEEE-USA audio books ( and details are shown in Table 2.

Fig. 10. Implementation of the WABA on Android emulator and a commercial smartwatch.
Download Original Figure
Table 2. Audio book sources for the WABA prototype.
Audio Book Title Size (MB) Length (minute:seconds)
Why STEM is important 22 23:38
Secrets to Being a World-Changer-Part 2 50 35:24
Women in Engineering – Book 4 80 55:15
Download Excel Table

To capture the user’s touch inputs, we set a ClickListener for each ImageView widgets. To capture multi-function side button events, we implemented KeyEvent.Callback function onKeyDown() to check for click events of two multi-function buttons. On our tested smartwatch Fossil Q Explorist, it has two multi-function buttons. When they are clicked, they generate KeyEvent.KEYCODE_STEM_1 and KeyEvent.KEYCODE_STEM_2 respectively. To use wrist gestures such as flick wrist out and flick wrist in, we also check for KEYCODE_NAVIGATE_NEXT and KEYCODE_NAVIGATE_PREVIOUS in onKeyDown(). Bluetooth hearables also generated KeyEvent when its hardware buttons (i.e., Media Button in Android). KeyEvents accepted in the WABA are summarized in Table 3. Our current implementation does not allow direct controls from the hearables. Since different Bluetooth hearables have various Media Buttons, we need to explore further on generic KeyEvents that can be widely used in the WABA regardless the brands of hearables.

Table 3. KeyEvent experimented in the WABA.
User Input KeyEvent Constant
Multi-function Button 1 KEYCODE_STEM_1
Multi-function Button 2 KEYCODE_STEM_2
Bluetooth Headset KEYCODE_MEDIA_*
Download Excel Table


We measured different wearable interaction latency on the WABA. The four tested wearable interaction modalities were 2 multi-function buttons and 2 gestures. In the experiment, 20 consecutive same inputs (i.e., events) are generated and recorded. The result is shown in Figure 11. We calculated an average for each event to measure the interaction latency. Hardware buttons (i.e., side multi-function buttons) were much faster than gestures. This can be explained by the fact “flick wrist out” and “flick wrist in” gestures are composed of two directional movements (i.e., move wrist outward and inward). If a user’s hands are full then gestures-based interaction could be used. Otherwise, a faster option is to use multi-function buttons when there is a free hand to spare. For the discussed 7 scenarios, gesture-based interaction is more suitable for running/jogging, work out and vehicle-driving scenarios.

Fig. 11. Wearable interaction latency measured in seconds.
Download Original Figure


In this paper, we designed and implemented a wearable audio book application (WABA). As a possible killer application for wearables, we proposed using a smartwatch with a hearables for listening to audio books on wearables. We presented 7 scenarios for the WABA that are more realistic to be used with wearables than smartphones or tablets. We also designed the layout of the WABA and explored several different wearable interaction modalities expanded by collocated wearable devices (i.e., a pair of smartwatch and hearables). Through the implemented WABA prototype, we found that the proposed interaction modalities such as multi-function buttons and gestures demonstrate modest latency. Therefore, our presented WABA is expressive enough as an audio book application while providing alternative and multi-modal wearable interaction modalities.

There are several limitations of our approach that could be further polished in future studies. First, we did not have much programmatic control over the hearables. If a programmable earwear is used, there are many interesting approaches beyond just using the hearables as a mere output device. Second, a usability test on users’ behaviors of the WABA is remained that should help identify important functions and user experiences. We believe the WABA and its multi-modal interaction can be adopted by visually challenged people, but carefully orchestrated user studies are required. Lastly, the WABA uses media playback functions while communicating with a paired Bluetooth device (i.e., hearables). This leads to draining battery unexpectedly fast on wearables. Further studies are needed to investigate these important issues. Nonetheless, our proposed wearable application, WABA has shown promising initial results using collocated wearable devices with multi-modal interaction modalities for wearable audio book applications.


This work was supported by the Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2018R1D1A1B07043983). Photos used in Figure 1 to Figure 7 are all from Pexels with the free license to use (



Wearable Technology Market - Industry Analysis, Market Size, Share, Trends, Application Analysis, Growth And Forecast 2020 – 2025,, 2020.


Digital Market Outlook – Wearables – worldwide,, 2020.


Wearable Devices Market Share,, 2020.


H. Yoon, J. Sung and S. Park, “Re-targeting user interfaces on stand-alone smart wearables,” in Proceedings of 2019 International Conference on Information and Communication Technology Convergence, Jeju Island, South Korea, pp. 1342-1344, Oct. 2019.


H. Yoon and C. Shin, “Cross-device computation coordination for mobile collocated interactions with wearables,” Sensors, vol. 19, no. 4, Article no. 796, Feb. 2019.


E. Waltz, “Back to work: wearables track social distancing and sick employees in the workplace,” IEEE Spectrum, May 2020;


G. M. Weiss, K. Yoneda, and T. Hayajneh, “Smartphone and smartwatch-based biometrics using activities of daily living,” IEEE Access, vol. 7, pp. 133190-133202, Sep. 2019.


M. Williams, J. R. C. Nurse, and S. Creese, “Smartwatch games: encouraging privacy-protective behavior in a longitudinal study,” Computers in Human Behavior, IEEE Access, vol. 99, pp. 38-54, Oct. 2019.


W. Jackson, SmartWatch Design Fundamentals. Berkeley, CA: Apress, 2019.


H. Bi, J. Zhang, and Y. Chen, “SmartGe: identifying pen-holding gesture with smartwatch,” IEEE Access, vol. 8, pp. 28820-28830, Jan. 2020.


C. J. Turner, B. S. Chaparro, and J. He, “Typing on a smartwatch while mobile: a comparison of input methods,” The Journal of the Human Factors and Ergonomics Society, Feb. 2020.


T. Buddhika, H. Zhang, S. W. T. Chan, V. Dissanayake, S. Nanayakkara, and R. Zimmermann, “fSense: unlocking the dimension of force for gestural interactions using smartwatch PPG sensor,” in Proceedings of the 10th Augmented Human International Conference, Article no. 11, Mar. 2019.


R. Khurana, M. Goel, and K. Lyons, “Detachable smartwatch: more than a wearable,” in Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Article no. 50, Jun. 2019.


P. Crum, “Hearables: here come the: technology tucked inside your ears will augment your daily life,” IEEE Spectrum, vol. 56, no. 5, pp. 38-43, Apr. 2019.


V. Goverdovsky, W. v. Rosenberg, T. Nakamura, D. Looney, D. J. Sharp, C. Papavassiliou, M. J. Morrell, and D. P. Mandic, “Hearables: multimodal physiological in-ear sensing,” Scientific Reports, vol. 7, Article no. 6948, Jul. 2017.


A. Martin and J. Voix, “In-ear audio wearable: measurement of heart and breathing rates for health and safety monitoring,” IEEE Transactions on Bio-medical Engineering, vol. 65, no. 6, pp. 1256-1263. Jun. 2017.


A. S. Arif and A. Mazalek, “A survey of text entry techniques for smartwatches,” in Proceedings of HCI 2016: Interaction Platforms and Techniques, pp. 255-267, Jun. 2016.


G. Costagliola, M. De Rosa and V. Fuccella, “Handwriting on smartwatches: an empirical investigation,” IEEE Transactions on Human-Machine Systems, vol. 47, no. 6, pp. 1100-1109, Dec. 2017.


J. Grubert, M. Kranz, and A. Quigley, “A challenges in mobile multi-device ecosystems,” mUX: The Journal of Mobile User Experiences, vol. 5, Article no. 5, Aug. 2016.


H. Yoon, S.-H. Park, K.-T. Lee, J. W. Park, A. K. Dey, and S. Kim, “A case study on iteratively assessing and enhancing wearable user interface prototypes,” Symmetry, vol. 9, no. 7, Article 114, Jul. 2017.


L.-H. Lee and P. Hui, “Interaction methods for smart glasses: a survey,” IEEE Access, vol. 6, pp. 28712-28732, May 2018.


H. Yoon and S.-H. Park, “A non-touchscreen tactile wearable interface as an alternative to touchscreen-based wearable devices,” Sensors, vol. 20, no. 5, Article 1275, Feb. 2020.


Wear OS by Google – Introduction,, 2020.


H. Yoon and C. Shin, “Cross-device computation coordination for mobile collocated interactions with wearables,” Sensors, vol. 19, no. 4, Feb. 2019.


Hyoseok Yoon

jmis-7-2-107-i1 received his B.S. degree in Computer Science from Soongsil University in 2005. He received his M.S. and Ph.D. degrees in Information and Communication (Computer Science and Engineering) from the Gwangju Institute of Science and Technology (GIST), in 2007 and 2012, respectively. He was a researcher at the GIST Culture Technology Institute from 2012 to 2013 and was a research associate at the Korea Advanced Institute of Science and Technology, Culture Technology Research Institute in 2014. He was a senior researcher at Korea Electronics Technology Institute from 2014 to 2019. In September 2019, he joined the Division of Computer Engineering, Hanshin University where he is currently an assistant professor. His research interests include ubiquitous computing (context-awareness, wearable computing) and Human-Computer Interaction (mobile and wearable UI/UX, MR/AR/VR interaction).

Jangmi Son

jmis-7-2-107-i2 is an undergraduate student in the Division of Computer Engineering at Hanshin University. In May 2020, she joined the HCI Lab at Hanshin University as an undergraduate researcher. Her research interests include assistive technology, wearable computing and Human-Computer Interaction.