Research Projects

UnicrePaint: Digital Painting through Physical Objects (2014.4-2017.3)

Mankind’s capacity for creativity is infinite. In the physical world, people create visual artistic works not only with specific tools, such as paintbrushes, but also with various objects, such as dried flowers pressed on paper. In contrast, digital painting has a number of advantages; however, such painting currently requires a specific tool, such as a stylus, which might diminish the pleasurable experience of creation. In this project, a digital painting system called UnicrePaint is investigated, that utilizes daily objects as tools of expression. UnicrePaint is designed to capture the appearance of contact points on a drawing surface and provide real-time co-located digitized feedback. Contact detection is realized by FTIR-based sensing.

Spot and Set: Controlling Lighting Patterns of LEDs through a Smartphone (2014.4-2017.3)

People enjoy various social events, and illumination plays an important role in creating a good atmosphere during these events. Usually, people illuminate trees and walls by laying cables with a bunch of LEDs. However, they do not have control over the lighting patterns; instead, they just select a preset pattern. In this project, we propose a system to control the lighting patterns of LEDs. In our system, a user’s smartphone works as a controller. The user holds his/her smartphone with a backside light over a light sensor that corresponds to a particular LED. Once one or more LEDs are selected, the user sets the desired lighting pattern. We consider that this approach extends a user’s range of expression at little learning cost.

Mis-choice Detection of Public Transportation Route based on Smartphone and GIS (2014.4-2017.3)

Public transportation systems in large cities are so complex. People who are not familiar with a certain city often make mistakes. Even though a number of route finder or transit support services have already been provided to avoid route mischoice, still people fail and become very anxious about the recovery. In this project, we propose a system to detect a mischoice of a public transportation route before arriving at the next station. The system runs on a smartphone in cooperation with Geographical Information System (GIS) on the network. Unlike other methods that aim at self-contained train localization based on a timetable, our method intends to handle delays and congested train services by an approach similar to map-mataching. The system is expected to support people to recover from a failure as early as possible.

Pedestrian’s Avoidance Behavior Recognition for Opportunistic Road Anomaly Detection (2014.4-2017.3)

Recognizing the avoidance behavior and aggregating the events with locations allows automatic anomaly reports. Automatic road anomaly detection techniques for cars and bikes have already been proposed, which deals with relatively large movement. By contrast, the pedestrian’s avoidance behavior is too slight to adapt the existing methods. In this project, an opportunistic sensing-based system for road anomaly detection is investigated. To detect road anomalies such as crack, pit and puddle, we focus on pedestrian’s avoidance behavior that is characterized by the azimuth changing patterns. Three typical avoidance behaviors are defined.

Avoiding texting while walking during crossing roads and railways (2013.4-2016.3)

In late years, a number of people walk carelessly while continuing watching a smartphone screen, which often causes fatal accidents. In this project, a system is investigated to restrain smartphone usage at the time of crossing road and railroad based on the combination of the present and past location of a pedestrian, walking direction and walking speed by means of smartphone-mounted sensors. A geographical database that includes roads and railway information is also combined.

An Ironing Support System with Superimposed Guidance (2012.4-2015.3)

Ironing is one of the troublesome houseworks. A number of people do not like to iron. Based on a questionnaire survey with housewives, we found that the lack of skills and knowledge in beginners were the reason for dislikes. To lower the barrier, a system to support beginners of ironing shirts was investigated. The system recognizes the position of shirts on an ironing board and the user’s operational states. The position of wrinkles are recognized based on texture analysis of Infra-red camera images. Then, supportive information is presented by a video projector in a way that the information is projected on an appropriate position on the working surface.

View Management Techniques for Tabletop Spatial Augmented Reality Systems (2012.4-)

Augmented reality (AR) by a projector allows easy association of information by using a label with a particular object. When a projector is installed above a workspace and pointed downward, supportive information can be presented; however, a presented label is deformed on a non-planar object. Also, a label might be projected in a region where it is hidden by the object. In this project, a view management technique is investigated that facilitates interpretation by improving the legibility of information. Our proposed method, the Nonoverlapped Gradient Descent (NGD) method, determines the position of a newly added label by avoiding overlapping of surrounding labels and linkage lines. The issue of presenting in a shadow area and a blind area is also addressed by estimating these areas based on the approximation of objects as a simple solid.

Augmented Card Playing (2011.4-2014.3)

This project aims at investing a software toolkit for augmenting a traditional card playing with a projector and a camera to add playfulness and communication. The functionalities were specified based on a user survey session with actual play, such as card recognition, player identification, and visual effect. Cards are recognized by a video camera without any artificial marker with an accuracy of 96%. A player is identified by the same camera from the direction of the hand appearing over a table. The Pelmanism game was augmented on top of the system as a case study to validate the concept of augmentation and the performance of the recognitions.

Pointing and Calling Recognition for Remote Operatives Management (2011.4-2014.3)

This project aims at investigating a new facilitated assessment technique for the pointing and calling enforcement. The pointing and calling is an activity for workers to keep occupational safety and correctness. Our proposed system provides early detection for the false enforcement of the pointing and calling by recognizing the performance of the activity. We implemented a prototype system on an Android smartphone terminal to record occurrence of the performance, the place where it performed, and the performed time. The recognition accuracy showed the feasibility of recognition with a self-contained wearable computer system.

INCA: Interactive Context-aware System for Energy Efficient Living  (2011.1-2012.12)

This work is conducted in cooperation with University of Oulu with the support by Japan Society for the Promotion of Science (JSPS) and Academy of Finland under the Japan-Finland Bilateral Core Program. The aim of this joint-project is to develop an interactive context-aware sensor-based feedback and control system to support energy efficient housing. The system is intended to self-motivate inhabitants to be aware of their energy consumption habits and this way to be able to decrease energy costs. A network of sensors will be installed to environment to perceive human context and the energy consumption of devices. Then, statistical machine learning algorithms will be developed to utilize the sensor data to recognize low-level human related contexts from the inhabitants. Different human computer interaction (HCI) techniques will be studied to build smooth feedback and control system through persuasive interface. For more detail on the project, click [here].

Placement-aware Mobile Phone (v1: 2007.10-2010.3, v2: 2010.4-2013.3, v3:2013.4-2016.3, V4: 2016.4-2019.3)

We are investigating a method to identify the location of a mobile phone on the user’s body, e.g. trousers front pocket, hanging on the neck. The identification algorithm utilizes only one 3-axes accelerometer which is getting popular in commercial mobile phones. The location information is utilized by an application on a mobile phone as a context of a user and the device itself. The application scenarios are 1) smart call/message notification, 2) assurance of sensor placement, and 3) location-aware functionality control. Popular six locations are specified as the targets, against which our identification algorithm works even when the location of a mobile phone changes while the user is in nonstationary motion, e.g., standing. This makes the above applications work seamlessly.

On-body Placement-aware Heatstroke Alert  (2011.4-2015.3)

In Japan, various consumer products for heatstroke risk alert have been available on the market, in which Wet-Bulb Globe Temperature (WBGT) is utilized as a heat stress index. An issue in such a portable environmental measurement device is that the measurement might not be correct if the instrument is not outside, such as in the pocket due to the body temperature and sweat. The over-estimate from the incorrect measurement may lead to a user’s distrust due to frequent warning, while under-estimate might cause a severe damage on the user. To address the issue, we propose a trustworthy and effective heatstroke risk alert device that provides a user with possibility of over (under)-estimate based on a storing position on body as well as calibrated level of the risk. For this purpose, we also developed an external temperature and humidity sensing module that is attachable to an Android 3.0 or higher terminal via USB.

Facilitating Bothersome Task by Affection for a Pet  (2010.5-2012.3)

We propose a persuasive interface that leverages affection for a pet to start a bothersome task. Such task includes labeling dataset for supervised learning, tagging pictures taken by a digital camera, etc. A virtual pet “lives” in a wallpaper of a smartphone, changes its facial and body expressions, talks to a person based on a user’s context, which is designed to allow a person to have affection for it. The virtual pet asks a person to conduct a task, i.e. labeling, when a person finish utilizing an application. If he/she accepts the request, the pet express delights, while showing sad facial expression in case that he/she refuses the request.

Walking Support with a Wearable Projector (2010.7-2013.3)

We propose a walking support system for elderly people who need preventive care and patients with gait disorder, for example, parkinson’s syndrome. The system presents visual information for gait support to a user by a hip-mounted projector. The information is determined based on a user’s walking condition and physical function. Currently, we are investigating a method to reduce vibration that is found in a projection image on the floor while walking. This is because the projected image on the floor is shaking with the walking. Such unstability of projected image not just affects the visibility, but also makes it difficult to present spatial information.

ost4ce: On-site Safety Training for Chemistry Experiments (2009.4-2012.3, 2012.4-2015.3, 2015.4-2018.3)

In Japan, safety training for a chemistry course is basically done at the beginning of a semester as a classroom lecture, using materials such as videos and textbooks. This particular training style may generate gaps between the safety procedures being learned and actual practice. This disparity may impair the effect of safety training and subsequently result in failure of preventing accidents. We propose a tangible learning system that displays a message regarding possible accidents. One of the most important design issues is making students independent of the system. More specifically, if a safety-training system is too suggestive all of the time, it would certainly be helpful for a student during the course of an experiment, but it would deprive him or her of the opportunity to learn to avoid accidents. Keeping this balance between learning safety measures and being safe is an important issue.

Augmented Bike for Safe and Effective Notification (2009.4-2013.3)

We are investigating a method to provide a notification to a bicycle rider in safe and effective ways. A navigation system for a bicycle and a mobile phone can interrupt into a rider no matter how he or she is riding a bike, which would make him or her unsafe. Our idea is to determine an appropriate timing of the presentation based on the state of riding, e.g. speed, balancing, seating. Our method might intentionally delay the notification of an incoming e-mail message to avoid unsafe allocation of cognitive load when the speed of a bike is high, for example. Also, the modality of notification could be determined in the same manner. Currently, we are developing a sensor/actuator-rich bicycle to collect data and find relationship among a state of riding, the effectiveness of notification and the safety of interruption.

ClassifyingBox: An Interface to shake up People during a Menial Job  (2009.4-2011.3)

We are investigating a method to shake up a menial job worker during the job. A menial job can increase the mental tiredness of a worker, which can finally cause a human error and/or decrease the performance of the job. So, our method is intended to “refresh” the mental state by providing information at an appropriate timing that is determined based on the activity and physiological state of a worker. As a case study, we are developing a system for postal sorting workers. A pressure sensor-augmented shelf for postal sorting has been developed, and an image that represents a particular region grows up based on the activity and state of a worker.

pMirror: Ambient Persuasive Mirror with Life-Like Expression (2007.12-2010.2)

The advancement of technologies has led people to a sedentary lifestyle with a lack of physical activity. Consequently, preventing lifestyle-related disease is now a social issue. We investigated an augmented mirror in order to increase the awareness of daily walking. The augmented mirror is a two-way mirror that displays symbols representing the number of steps walked on a daily basis. The mirror was designed with four strategies to reflect daily walking: 1) pleasurable interaction with the system through daily walking, 2) avoiding negative feelings while providing negative feedback, 3) increasing self-reflection of each day’s walking, and 4) facilitating encouragement among a group of participants.

Aha!: Ad-hoc Association of Objects Moving Together (2008.4-2009.10)

Any information that is relevant to an application can be a context. Not only a context obtained from a single entity, but an attribute of a group is also subject to a context, and grouping complements a missing piece of information in each entity. We investigated a method to identify multiple sensor nodes that have similar movement patterns in an ad-hoc manner. The similarity of the movement pattern provides more reliable information than mere coexistence. A standard agglomerative clustering algorithm was applied at each time step to find the group members as a set of clusters within a certain height. The threshold is determined based on simple statistics that are obtained from the previous clustering results. This method allows objects to from a group in an unsupervised manner.

aBook: Presenting Supplemental Information during Reading a Book (2008.4-2009.11)

We investigated an augmented book cover and a bookmark as means for implicit interaction with a book. The augmented items sense their state of use by accelerometers and detect page-flipping events. Accumulation of the events is utilized by an application, the Virtual Illustration System, to estimate the current two-page spread and then to infer a reader’s “context”. In the application, corresponding multimedia contents are identified by referring to a predefined page-contents mapping table. Thus, a user reads the book in an ordinal way, and at the same time, her context is implicitly captured and given to an application as an input. Here, we assumed that the class of books that are read in a sequential manner, e.g. a novel book or a travelogue, is the target for the application.

Person Identification and Tracking by Pressure Sensitive Floor (2007)

In cooperation with Mr. Jaakko Suutala (Univ. of Oulu), the methods and sensor technology used to identify persons from their walking characteristics were investigated. An array of simple binary switch floor sensors were utilized to detect footsteps. Feature analysis and recognition were performed with a fully discriminative Bayesian approach using a Gaussian Process (GP) classifier. We showed the usefulness of our probabilistic approach on a large data set consisting of walking sequences of nine different subjects. In addition, we extracted novel features and analyzed practical issues such as the use of different shoes and walking speeds, which were usually missed in this kind of experiment. Using simple binary sensors and the large nine-person data set, we were able to achieve promising identification results.