Research Projects

Novelty detection in on-body device localization problem (2018.4-)

On-body device position-awareness plays an important role relative to providing smartphone-based services with high levels of usability and quality. Existing on-body device localization methods employ multi-class classification techniques, and the number of positions is fixed during use. However, data for training classifiers must be collected from all possible positions in advance, which requires significant effort. In addition, people do not always use all positions. Instead, people tend to have a few preferred storage positions. Therefore, we incrementally add recognition targets when a new carrying position is detected. In contrast, we propose a framework to discover new positions that the system does not initially support and add them as recognition targets during use, in which a novelty detection method is a core functionality.

Personalizing context recognition model using active learning accelerated by compatibility-based base classifier selection (2017.4-)

In context recognition, the recognition model is often built by supervised machine-learning techniques. The recognition performance is likely to be influenced by individual difference, e.g., rhythm of walking, in the target domain. We investigate a framework that allows mobile device users to be involved with customization process, in which the key concept is active learning and persuasive technologies. We show a case study with smartphone localization problem, in which we evaluated the effectiveness of active-learning supported by compatibility-based base classifier selection.

Personalizing activity recognition model based on compatibility of classifiers (2018.4-)

Personalization related research has been conducted for decades with the objective of investigating personal-level effectiveness in irregular or abnormal cases among many types of domains. However, the activity recognition community generally investigates one-fits-all model, i.e., a single recognition model for all people and suffers from poor performance generalization or a huge amount of training data for high generalization. We propose Compatibility-based classifier personalization (CbCP) as a subject-dependent activity recognition method that uses the classifier with the highest compatibility (similarity) with a particular user.

CrosSI: A Desktop Environment with Connected Horizontal and Vertical Surfaces (2017.4-)

A typical desk workspace includes both horizontal and vertical areas, wherein users can freely move items between the horizontal and vertical surfaces and edit objects; for example, a user can place a paper on the desk, write a memo, and affix it to the wall. However, in workspaces designed to handle digital information, the usable area is limited to specific devices such as displays and tablets, and the user interface is discontinuous in terms of both the software and hardware. Therefore, we propose a multi-surface information display system CrosSI, which includes a horizontal surface and multiple cubic structures with vertical surfaces on the horizontal surface. Using this system, we investigate the ability to visualize and move information on a continuous system of horizontal and vertical surfaces.

Investigating Psychological Effects on Positional Relationship between a person and a person-following robot (2017.4-)

A person-following robot must understand the psy- chological effects of a user to maintain a good relationship with them. In this paper, we define factors that affect the user’s psychological states by factor analysis. Also, we present an overall score of the following effect for various following positions based on user evaluation. The result shows that a higher score is obtained if the robot follows close and is beside the person than that if the robot follows the person from a distance.

Estimating Smartphone Addiction Proneness Scale through the State of Use of Terminal and Applications (2017.4-2019.3)

Overuse of smartphone applications causes addiction to smartphone, which affects the user’s physical and mental health. Interventions such as providing persuasive messages and controlling the use of applications need to assess the level of addiction to smartphone. To measure this addiction level, Smartphone Addiction Proneness scale (SAPS) has been proposed. However, it requires the user to answer 15 questions, which makes it burdensome, unreliable (because it is based on user response rather than their behavior), and slow (the estimation takes time). To overcome these limitations, we propose a technique for automatic recognition of SAPS score based on the actual daily use of the smartphone device. Our technique estimates the SAPS score using a regression model that takes the smartphone’s states of use as explanatory variables (features). We describe the effective features and the regression model.

Visibility-aware label placement (VisLP) in spatial augmented reality (2015.4-2018.3)

Categories of visibility degradation

The notion of (a) building an estimator and (b) running online

Augmented Reality (AR) technologies enhances the physical world with digital information. In graphic-based augmentation, text and graphics, i.e., label, are associated to an object for describing it. Various view management techniques for AR have been investigated. Most of them have been designed for two dimensional see-through displays, and little have been investigated for projector-based AR called Spatial AR. We propose a view management method for spatial AR, VisLP, that finds visible position for a label and a linkage line, while taking into account the characteristics of projecting information on physical world. VisLP employs machine-learning techniques to classify visibility that reflects human’s subjective mental workload in reading information and objective measures of reading correctness.

Visible Position Estimation in Whole Wrist Circumference Devices towards Pose-aware Display (2015.4-2018.3)

Smart watches allow instant access to information; however, the visual notification is not always reachable depending on the forearm posture. Flexible and curved display technologies can enable full-wrist circumference displays that show information at the most visible positions using pose awareness. A prototype device is implemented with 10 LEDs and 10 accelerometers around the wrist. The most visible LED is estimated using a machine learning technique. The main idea is to utilize direct relationship between the raw acceleration signals and the position of the most visible LED, rather than assigning the position by particular classes of activities or forward-kinematic model-based estimation. Also, sensor reduction is attempted by introducing new features.

Impressive Picture Selection for Pleasurable Recall of Past Events (2015.4-2018.3)

Wearable cameras allow us to capture large amount of video or still images in an automatic and implicit manner. However, the only necessary images should be filtered out from the captured data that contains meaningless and/or redundant information. We proposed a method to identify a set of still images by audio and video data, which is intended to let users feel pleasurable when they watch the images later.

Palco: Printable Avatar System for Loose Communication between Distant People (2016.4-2019.3)

We propose a method for providing users with a sense of security and connectedness with others by facilitating loose communication. Loose communication is defined by the presentation of abstract information and passive (one-way) communication. By focusing on the physicality and anthropomorphic characteristics of tangible avatars, we investigated a communication support system, Palco, that displays three types of contextual information with respect to the communication partner—emotional state, activity, and location—in a loose manner. Our approach contrasts with typical SNS interaction methods characterized by tight communication with interactivity and concrete information.

UnicrePaint: Digital Painting through Physical Objects (2014.4-2017.3, 2018.4-2019.9)

Mankind’s capacity for creativity is infinite. In the physical world, people create visual artistic works not only with specific tools, such as paintbrushes, but also with various objects, such as dried flowers pressed on paper. In contrast, digital painting has a number of advantages; however, such painting currently requires a specific tool, such as a stylus, which might diminish the pleasurable experience of creation. In this project, a digital painting system called UnicrePaint is investigated, that utilizes daily objects as tools of expression. UnicrePaint is designed to capture the appearance of contact points on a drawing surface and provide real-time co-located digitized feedback. Contact detection is realized by FTIR-based sensing.

Spot and Set: Controlling Lighting Patterns of LEDs through a Smartphone (2014.4-2017.3)

People enjoy various social events, and illumination plays an important role in creating a good atmosphere during these events. Usually, people illuminate trees and walls by laying cables with a bunch of LEDs. However, they do not have control over the lighting patterns; instead, they just select a preset pattern. In this project, we propose a system to control the lighting patterns of LEDs. In our system, a user’s smartphone works as a controller. The user holds his/her smartphone with a backside light over a light sensor that corresponds to a particular LED. Once one or more LEDs are selected, the user sets the desired lighting pattern. We consider that this approach extends a user’s range of expression at little learning cost.

Mis-choice Detection of Public Transportation Route based on Smartphone and GIS (2014.4-2017.3)

Public transportation systems in large cities are so complex. People who are not familiar with a certain city often make mistakes. Even though a number of route finder or transit support services have already been provided to avoid route mischoice, still people fail and become very anxious about the recovery. In this project, we propose a system to detect a mischoice of a public transportation route before arriving at the next station. The system runs on a smartphone in cooperation with Geographical Information System (GIS) on the network. Unlike other methods that aim at self-contained train localization based on a timetable, our method intends to handle delays and congested train services by an approach similar to map-mataching. The system is expected to support people to recover from a failure as early as possible.

Pedestrian’s Avoidance Behavior Recognition for Opportunistic Road Anomaly Detection (2014.4-2017.3)

Recognizing the avoidance behavior and aggregating the events with locations allows automatic anomaly reports. Automatic road anomaly detection techniques for cars and bikes have already been proposed, which deals with relatively large movement. By contrast, the pedestrian’s avoidance behavior is too slight to adapt the existing methods. In this project, an opportunistic sensing-based system for road anomaly detection is investigated. To detect road anomalies such as crack, pit and puddle, we focus on pedestrian’s avoidance behavior that is characterized by the azimuth changing patterns. Three typical avoidance behaviors are defined.

Understanding where to project information on the desk to support work with paper and pen (2014.4-2015.3)

Paper-based work still remains in daily life, and digital information may help such paper-based work. In this project, we assume an environment, in which a projector presents information on the desk that is used in a transcription task such as filling a form of notification of change of address to post office. We explore the effect of the distance and the direction between the transcription target (printed) and the reference information (projected) on the efficiency and effectiveness. We confirmed that presentation on the upper sides of the transcription target and the area close to the target showed better results (faster task completion and lower error rate), while worse results were observed on the lower right side that was around the dominant hands.

Avoiding texting while walking during crossing roads and railways (2013.4-2016.3)

In late years, a number of people walk carelessly while continuing watching a smartphone screen, which often causes fatal accidents. In this project, a system is investigated to restrain smartphone usage at the time of crossing road and railroad based on the combination of the present and past location of a pedestrian, walking direction and walking speed by means of smartphone-mounted sensors. A geographical database that includes roads and railway information is also combined.

An Ironing Support System with Superimposed Guidance (2012.4-2015.3)

Ironing is one of the troublesome houseworks. A number of people do not like to iron. Based on a questionnaire survey with housewives, we found that the lack of skills and knowledge in beginners were the reason for dislikes. To lower the barrier, a system to support beginners of ironing shirts was investigated. The system recognizes the position of shirts on an ironing board and the user’s operational states. The position of wrinkles are recognized based on texture analysis of Infra-red camera images. Then, supportive information is presented by a video projector in a way that the information is projected on an appropriate position on the working surface.

View Management Techniques for Tabletop Spatial Augmented Reality Systems (2012.4-2015.3)

Augmented reality (AR) by a projector allows easy association of information by using a label with a particular object. When a projector is installed above a workspace and pointed downward, supportive information can be presented; however, a presented label is deformed on a non-planar object. Also, a label might be projected in a region where it is hidden by the object. In this project, a view management technique is investigated that facilitates interpretation by improving the legibility of information. Our proposed method, the Nonoverlapped Gradient Descent (NGD) method, determines the position of a newly added label by avoiding overlapping of surrounding labels and linkage lines. The issue of presenting in a shadow area and a blind area is also addressed by estimating these areas based on the approximation of objects as a simple solid.

Augmented Card Playing (2011.4-2014.3)

This project aims at investing a software toolkit for augmenting a traditional card playing with a projector and a camera to add playfulness and communication. The functionalities were specified based on a user survey session with actual play, such as card recognition, player identification, and visual effect. Cards are recognized by a video camera without any artificial marker with an accuracy of 96%. A player is identified by the same camera from the direction of the hand appearing over a table. The Pelmanism game was augmented on top of the system as a case study to validate the concept of augmentation and the performance of the recognitions.

Pointing and Calling Recognition for Remote Operatives Management (2011.4-2014.3)

This project aims at investigating a new facilitated assessment technique for the pointing and calling enforcement. The pointing and calling is an activity for workers to keep occupational safety and correctness. Our proposed system provides early detection for the false enforcement of the pointing and calling by recognizing the performance of the activity. We implemented a prototype system on an Android smartphone terminal to record occurrence of the performance, the place where it performed, and the performed time. The recognition accuracy showed the feasibility of recognition with a self-contained wearable computer system.

INCA: Interactive Context-aware System for Energy Efficient Living  (2011.1-2012.12)

This work is conducted in cooperation with University of Oulu with the support by Japan Society for the Promotion of Science (JSPS) and Academy of Finland under the Japan-Finland Bilateral Core Program. The aim of this joint-project is to develop an interactive context-aware sensor-based feedback and control system to support energy efficient housing. The system is intended to self-motivate inhabitants to be aware of their energy consumption habits and this way to be able to decrease energy costs. A network of sensors will be installed to environment to perceive human context and the energy consumption of devices. Then, statistical machine learning algorithms will be developed to utilize the sensor data to recognize low-level human related contexts from the inhabitants. Different human computer interaction (HCI) techniques will be studied to build smooth feedback and control system through persuasive interface. For more detail on the project, click [here].

Placement-aware Mobile Phone (v1: 2007.10-2010.3, v2: 2010.4-2013.3, v3:2013.4-2016.3, V4: 2016.4-2019.3)

We are investigating a method to identify the location of a mobile phone on the user’s body, e.g. trousers front pocket, hanging on the neck. The identification algorithm utilizes only one 3-axes accelerometer which is getting popular in commercial mobile phones. The location information is utilized by an application on a mobile phone as a context of a user and the device itself. The application scenarios are 1) smart call/message notification, 2) assurance of sensor placement, and 3) location-aware functionality control. Popular six locations are specified as the targets, against which our identification algorithm works even when the location of a mobile phone changes while the user is in nonstationary motion, e.g., standing. This makes the above applications work seamlessly.

On-body Placement-aware Heatstroke Alert  (2011.4-2015.3)

In Japan, various consumer products for heatstroke risk alert have been available on the market, in which Wet-Bulb Globe Temperature (WBGT) is utilized as a heat stress index. An issue in such a portable environmental measurement device is that the measurement might not be correct if the instrument is not outside, such as in the pocket due to the body temperature and sweat. The over-estimate from the incorrect measurement may lead to a user’s distrust due to frequent warning, while under-estimate might cause a severe damage on the user. To address the issue, we propose a trustworthy and effective heatstroke risk alert device that provides a user with possibility of over (under)-estimate based on a storing position on body as well as calibrated level of the risk. For this purpose, we also developed an external temperature and humidity sensing module that is attachable to an Android 3.0 or higher terminal via USB.

Facilitating Bothersome Task by Affection for a Pet  (2010.5-2012.3)

We propose a persuasive interface that leverages affection for a pet to start a bothersome task. Such task includes labeling dataset for supervised learning, tagging pictures taken by a digital camera, etc. A virtual pet “lives” in a wallpaper of a smartphone, changes its facial and body expressions, talks to a person based on a user’s context, which is designed to allow a person to have affection for it. The virtual pet asks a person to conduct a task, i.e. labeling, when a person finish utilizing an application. If he/she accepts the request, the pet express delights, while showing sad facial expression in case that he/she refuses the request.

Walking Support with a Wearable Projector (2010.7-2013.3)

We propose a walking support system for elderly people who need preventive care and patients with gait disorder, for example, parkinson’s syndrome. The system presents visual information for gait support to a user by a hip-mounted projector. The information is determined based on a user’s walking condition and physical function. Currently, we are investigating a method to reduce vibration that is found in a projection image on the floor while walking. This is because the projected image on the floor is shaking with the walking. Such unstability of projected image not just affects the visibility, but also makes it difficult to present spatial information.

ost4ce: On-site Safety Training for Chemistry Experiments (2009.4-2012.3, 2012.4-2015.3, 2015.4-2018.3)

In Japan, safety training for a chemistry course is basically done at the beginning of a semester as a classroom lecture, using materials such as videos and textbooks. This particular training style may generate gaps between the safety procedures being learned and actual practice. This disparity may impair the effect of safety training and subsequently result in failure of preventing accidents. We propose a tangible learning system that displays a message regarding possible accidents. One of the most important design issues is making students independent of the system. More specifically, if a safety-training system is too suggestive all of the time, it would certainly be helpful for a student during the course of an experiment, but it would deprive him or her of the opportunity to learn to avoid accidents. Keeping this balance between learning safety measures and being safe is an important issue.

Augmented Bike for Safe and Effective Notification (2009.4-2013.3)

We are investigating a method to provide a notification to a bicycle rider in safe and effective ways. A navigation system for a bicycle and a mobile phone can interrupt into a rider no matter how he or she is riding a bike, which would make him or her unsafe. Our idea is to determine an appropriate timing of the presentation based on the state of riding, e.g. speed, balancing, seating. Our method might intentionally delay the notification of an incoming e-mail message to avoid unsafe allocation of cognitive load when the speed of a bike is high, for example. Also, the modality of notification could be determined in the same manner. Currently, we are developing a sensor/actuator-rich bicycle to collect data and find relationship among a state of riding, the effectiveness of notification and the safety of interruption.

ClassifyingBox: An Interface to shake up People during a Menial Job  (2009.4-2011.3)

We are investigating a method to shake up a menial job worker during the job. A menial job can increase the mental tiredness of a worker, which can finally cause a human error and/or decrease the performance of the job. So, our method is intended to “refresh” the mental state by providing information at an appropriate timing that is determined based on the activity and physiological state of a worker. As a case study, we are developing a system for postal sorting workers. A pressure sensor-augmented shelf for postal sorting has been developed, and an image that represents a particular region grows up based on the activity and state of a worker.

pMirror: Ambient Persuasive Mirror with Life-Like Expression (2007.12-2010.2)

The advancement of technologies has led people to a sedentary lifestyle with a lack of physical activity. Consequently, preventing lifestyle-related disease is now a social issue. We investigated an augmented mirror in order to increase the awareness of daily walking. The augmented mirror is a two-way mirror that displays symbols representing the number of steps walked on a daily basis. The mirror was designed with four strategies to reflect daily walking: 1) pleasurable interaction with the system through daily walking, 2) avoiding negative feelings while providing negative feedback, 3) increasing self-reflection of each day’s walking, and 4) facilitating encouragement among a group of participants.

Aha!: Ad-hoc Association of Objects Moving Together (2008.4-2009.10)

Any information that is relevant to an application can be a context. Not only a context obtained from a single entity, but an attribute of a group is also subject to a context, and grouping complements a missing piece of information in each entity. We investigated a method to identify multiple sensor nodes that have similar movement patterns in an ad-hoc manner. The similarity of the movement pattern provides more reliable information than mere coexistence. A standard agglomerative clustering algorithm was applied at each time step to find the group members as a set of clusters within a certain height. The threshold is determined based on simple statistics that are obtained from the previous clustering results. This method allows objects to from a group in an unsupervised manner.

aBook: Presenting Supplemental Information during Reading a Book (2008.4-2009.11)

We investigated an augmented book cover and a bookmark as means for implicit interaction with a book. The augmented items sense their state of use by accelerometers and detect page-flipping events. Accumulation of the events is utilized by an application, the Virtual Illustration System, to estimate the current two-page spread and then to infer a reader’s “context”. In the application, corresponding multimedia contents are identified by referring to a predefined page-contents mapping table. Thus, a user reads the book in an ordinal way, and at the same time, her context is implicitly captured and given to an application as an input. Here, we assumed that the class of books that are read in a sequential manner, e.g. a novel book or a travelogue, is the target for the application.

Person Identification and Tracking by Pressure Sensitive Floor (2007)

In cooperation with Mr. Jaakko Suutala (Univ. of Oulu), the methods and sensor technology used to identify persons from their walking characteristics were investigated. An array of simple binary switch floor sensors were utilized to detect footsteps. Feature analysis and recognition were performed with a fully discriminative Bayesian approach using a Gaussian Process (GP) classifier. We showed the usefulness of our probabilistic approach on a large data set consisting of walking sequences of nine different subjects. In addition, we extracted novel features and analyzed practical issues such as the use of different shoes and walking speeds, which were usually missed in this kind of experiment. Using simple binary sensors and the large nine-person data set, we were able to achieve promising identification results.