Analysis of Intermediary Roles and Technical Components for Smartglass-Assisted Interactive Remote Collaboration

The first intermediary role is authoritative supervisor where one participant has significantly greater CIE than other parties. The party with this authoritative CIE acts as a supervisor to other participants. This relationship is common in telementoring scenarios where mentors have greater knowledge and expertise than mentees [4]. The smartglasses of the supervisor displays visual data from other participants that help the supervisor to give concrete instructions or guidance. Non-supervisory participants basically play two functions. They play the role of passive sensors to deliver details captured with their smartglasses to their supervisor on the remote site. Furthermore, they play the role of the designated actuator to physically carry out a task on behalf of the supervisor. This intermediary role is depicted in the left figure of Fig. 2.

The second intermediary role involves participants with approximately the same level of CIE. The smartglasses on these peers display the same visualization and they can zoom in or out visual data on one’s own device. However, the CIE of one device is on equal terms with the other device. Since collaborators or peers have a similar level of knowledge and expertise, their collaboration can create synergy. This intermediary role is depicted in the center figure of Fig. 2.

The third intermediary role involves participants carrying out tasks in unknown or unfamiliar domains. In this role, smartglasses are used as an exploration device that recognizes, interprets, and augments additional visualization on the smartglass. A notable application on this role includes social and gaming applications that do not require prior knowledge or experience from the users. In this role, users are explorative in previously unknown or unexperienced tasks. This intermediary role is depicted in the right figure of Fig. 2.

The aforementioned three intermediary roles have already been examined in previous remote collaboration systems and applications. We have classified tasks and applications found in remote collaboration studies according to the three intermediary roles in Table 1.

Surgery and medical tasks [10], [13], [14], [15] in remote collaboration scenarios include intermediary roles of authoritative supervisor (AS) as well as synergetic collaborator (SC). Since surgical and medical tasks require an expertise, speculative explorer (SE) is not common. In education and training tasks [11], [16], [17], all three intermediary roles are addressed. In AS, we have scenarios involving teachers and students. We can have a group of peers or team-based activities with prior knowledge (i.e., SC) or for completely different or new domain (i.e., SE). Similar to surgery and medica tasks, industrial and physical tasks [3], [18], [19], [20] also typically involve AS and SC roles. Lastly, in social and gaming [21], [22], [23] tasks, all three intermediate roles addressed where there exist various solutions to carry out such tasks.

Due to the always-(on, accessible, connected) characteristics of wearable devices, embedded sensors on these devices are used to monitor and analyze users physically via various physiological sensors and IMU (inertial measurement unit). A notable example includes location-based services where the user’s location is tracked for providing location-based notifications. Personal big data collected from smartglass and smartwatches are important sources for machine learning approaches. Moreover, quantified-self [6] and personal tracking [7] approaches have already been effectively deployed in healthcare, sleep, and food journaling. For example, users’ activities can be tracked with motion sensors and physiological sensors to measure physical fatigue as well as a cognitive load of a task of interest. The lessons and insights from these studies pave the way for preparing successful remote collaboration. Power-consuming personal tracking components for wearable devices should be carefully designed to deal with the limited battery capacity on these compact and portable devices.

By the definition of AS, SC, and SE, participants (including mentors, mentees, and peers) communicate with audio and video in real-time. At least one side (i.e., mentor or mentee) needs to wear smartglasses, but the remaining side can use non-smartglasses computers. To effectively communicate in terms of real-time feedback and communicative instructions, audio-visual teleconference functions are required. It is critical to provide and share users’ views when collaborating, while the actual resolution of the streaming video may differ in various tasks of interest (i.e., medical/surgical, educational, and industrial).

Due to the COVID-19 pandemic, audio-visual teleconference functions are used globally in several platforms such as Zoom, Google MeetTM, Microsoft Teams and implemented with open sources such as Jitsi Meet (https://github.com/jitsi/jitsi-meet), BigBlueButton (https://bigbluebutton.org/), OpenVidu (https://openvidu.io/), Element (https://element.io/), and Wire (https://wire.com/en/) as well as using WebRTC API [3]. Audio-visual components should be designed and implemented considering various display sizes and display resolution.

Participants in remote collaboration need tools and user interfaces for pointing or marking problematic areas that needs the other participants’ feedback. Mentors also need tools and user interfaces to guide and instruct the mentees. To support the needs of mentors, mentees, and collaborators, various interaction styles including touch-based, voice-based, gesture-based, and telestration (i.e., pointing, annotating, drawing) are required.

Wearable devices’ various embedded sensors can be used to improve the user experience and usability of the intermediary roles of AS, SC, and SE. For example, tactile physical feedback generated via vibration can be used to alert users in situations where the users’ attention level is low (i.e., performing some activities), hard-of-hearing users (i.e., physically, and noisy ambient sound) are involved, and multimodality are exploited for emphasis.

Fig. 3 shows our proposed technical components for the smartglass-based remote collaboration in three categories of personal tracking components, teleconference components, and interaction components. Personal tracking components are employed to track the wearer’s physiological, cognitive load, physical fatigue, and activity levels while managing and analyzing personal big data generated from the smartglass. Teleconference components are used to live-stream video and audio to mentors and mentees, respectively. Interaction components allow the user to pick and choose various interaction styles (i.e., touch-based, voice-based, gesture-based, telestration) that suit the user’s task and environment. All three components rely on embedded sensors such as motion sensors [24] and smartglass hardware. For example, on a Google Glass Enterprise Edition 2, activity levels in personal tracking components, video and audio streaming in teleconference components, and touch-based UI, voice-based UI, and gesture-based UI in interaction components can be exploited to implement remote collaboration applications. Similar to personal tracking components, power-consuming and computationally intensive interaction components such as voice-based UIs and gesture-based UIs must be carefully integrated to handle limited battery capacity.

Fig. 3. Technical components for SAIT.

Download Original Figure