When the Pillow controls the light
The smart home offers a wealth of everyday objects that can be used as remote controls. But what is a suitable object, and how do you wire it up? The development of new materials requires scientific ingenuity, as demonstrated by the project “Rich Interactive Materials for Everyday Objects in the Home” (RIME) project.
A small, light green box with a black display and rainbow-coloured cables appears on the video screen of the conference room. In the video, a hand slowly guides the green box as it moves over a sheet of paper. The device leaves a silver-grey trail of neat, straight, wavy and jagged patterns. At first glance, this device, “Print-A-Sketch”, looks like a craft toy. The simple design belies the fact that the printer has the potential to revolutionise everyday life: Because what Narges Pourjafarian, a scientist at Saarland University, prints out here on paper are freely designable sensors and circuits. These silver-grey lines are essential to our future as they form the foundation for controlling smart devices in our private spaces. Today, numerous smart devices can be found in homes and apartments, with even more on the horizon. However, the current challenge is evident: managing the control of these devices efficiently. Often, we are faced with the frustration of handling multiple remote controls for the TV, front door, lighting, and music system, among others, which tend to get misplaced easily. To address this issue, many manufacturers now offer smartphone apps for universal control of smart homes. However, as Susanne Boll, Professor of Computer Science at the University of Oldenburg, points out, this solution is not optimal: “It’s annoying always to have to have your smartphone at hand for control.” The ultimate goal is to establish a more intuitive and seamless control system where everyday utensils can operate lights, heating, and various smart devices, allowing the technology to take a back seat and blend harmoniously into our daily lives.
With scientists from the Universities of Oldenburg, Aachen and Saarbrücken, Boll wants to find to what extent objects from everyday life can be used sensibly to control smart homes. The interaction should be simple and immediate. The operation of switches and sensors should be intuitive, catering to our needs and varying contexts seamlessly. This is the vision of the “Rich Interactive Materials for Everyday Objects in the Home” (RIME) project.
Pourjafarian has developed “Print A Sketch” for this purpose: With her printer, electrical buttons can be attached to materials such as fabric, paper or cork. Using conductive ink and miniature computers such as Arduinos, a couch or kitchen table can be enhanced and find a secondary use as a remote control for the music system.
Pourjafarian, who studied electrical engineering in Iran and computer science in Germany and is now doing her doctorate in computational design, has been working on this for a long time: “Different surfaces require different amounts of ink,” she explains, “and the printer also compensates for the always slightly irregular hand guidance so that the lines are nice and straight.” This method of printing circuits on the spot is known as in-situ fabrication, and it enables researchers to wire even uneven surfaces or large objects.
“Here, research and inventiveness go hand in hand,” explains Jürgen Steimle proudly. Steimle, the professor of computer science at Saarland University, builds a bridge to materials research: An interdisciplinary team is indispensable for a promising future: Thanks to knowledge from different subjects, new surfaces can be used for HCI research. The systematic approach and the exchange of (interim) results in scientific papers make it clear that the invention of new technology is also part of science – even if this is not immediately recognisable to all colleagues in other disciplines, says Steimle: “This is common in the natural sciences. We want to shape the future.”
However, our focus should not solely be on the technology, as emphasised by Susanne Boll: “We also need to know the best places to embed sensors, as well as the most intuitive gestures to control our devices.” Ensuring these gestures are universal is essential for seamless navigation, whether we are at home or visiting someone else’s residence. At the University of Oldenburg, doctoral student Michael Chamunorwa is researching to identify typical human gestures applicable to everyday objects. This endeavour requires considerable inventiveness since we are yet to control devices using household objects such as sofa cushions.
First, he contemplated which objects in each room would make sense to use as remote controls. Among other options, the choice settled on the couch cushion. “Originally, I wanted to equip one with sensors and record the interaction with it.” Photos documented how he spent hours attempting to intricately wire numerous sensors together and integrate them into the fabric. However, the prototype didn’t function as intended, prompting a reevaluation. The sensor-equipped cushion was replaced with an ordinary one. Chamunorwa observed how test subjects interacted intuitively with it, such as when they were asked to switch on the light using the pillow. Through this method, he sought to identify typical gestures that could reveal the optimal placement of sensors on a “remote control cushion”.
But then came the pandemic, disrupting daily activities and resulting in the temporary closure of the University of Oldenburg research laboratory. “I wasn’t able to do a number of studies as planned,” Chamunorwa explains while attaching a smartphone to a camera tripod, directing it towards a couch. But here, too, he has shown creativity by creating a “Pop-up Observation Kit”, which can be mailed to study participants, enabling studies to be done remotely.
It contains instructions that explain how to set up the experiment environment and film themselves as they experiment. “This makes it more difficult to monitor the experiment, but the decentralised survey also has advantages: the subjects have less of the impression of being under observation and interact more naturally.”
It is precisely this natural interaction that Chamunorwa is after. The researcher sits on the couch and takes a pillow in both hands to demonstrate the setup. Small cards contain instructions such as: “Change the radio station.”, “Turn up the music” or “Open the blinds”. “What gestures would you make with your interactive pillow?” It still seems unusual for someone to squeeze a pillow, fold its corners or swipe along the surface, as Chamunorwa does. No wonder because we humans are not used to using everyday objects as remote controls in this manner.
Jan Borchers finds this richness, which can be discovered in the interaction with everyday objects, precious: “The fact that we have so far focused so much on touchscreens is an insult to the complex capabilities of the human hand,” states the professor of media informatics and human-computer interaction at the Rheinisch-Westfälische Technische Hochschule Aachen, demonstratively moving his hands.
When asked how well similar behaviour uniform for many people can be detected, Chamunorwa laughs. “When it comes to pillows, people are a little more creative.” He conducted the same experiment with a black pad at his desk about the size of an A4 paper and equipped with pressure sensors. Here, the gestures are more similar. “When it comes to surfaces like the table, we’re primed by the smartphone.” None of the test subjects came up with the idea of banging their fists on the table, putting their whole hand on them, drawing a circle, or anything like that. You can probably already see the first effects of our daily use of touchscreens, which Borchers feels is not complex enough: Our hands have become accustomed to low-complexity tasks.
Technology should be integrated into our everyday lives as inconspicuously as possible in the smart home. Buttons are not meant to look like conspicuous electronic components. This is where textile surfaces come into play; the Borchers team is trying to determine how such textile buttons can be created. Borchers founded Germany’s first FabLab in Aachen, i.e. a “Fabrication Laboratory” – an open workshop where people can use modern machines and tools such as 3D printers. The interweaving of invention and experimentation is also standard practice in Aachen.
For example, the team experimented with converting the armrest of an ordinary grey couch into an interactive surface by embroidering and embossing buttons and sliders on it with white thread. With such a textile interface, it should be possible to brighten or darken the ambient lighting in the living room. The sliders vary; some are flat, some are raised, and others are recessed. They also differ in shape: some are arcuate, others are w-shaped, while some have lines and stitches that thicken outwards like a classic volume indicator. Nonetheless, they all have one similarity in the form of tickmarks that signify the scale for controlling the volume of a sound system or the brightness level of a lightbulb in a smart home setting.
But which textile-based user interface is suitable for the intuitive and successful operation of smart lighting? At best, even without looking? Where do mistakes happen, and what strategies do people develop? The results of the empirical study are intriguing, explains Oliver Nowak, who is researching textile interfaces with René Schäfer for RIME: “There is no such thing as the best design. When there are orientation points, the control works relatively accurately.” There are also a few differences in the speed of operation.
However, and this surprised the researchers, the test subjects assessed the success of individual controls differently during the test. “Some designs were perceived as better, even if they work equally well in the factual result.” That’s the difference HCI research makes through its focus on people. “We can leave the design and implementation of technology into our everyday lives to the industry. Or we can start exploring such questions,” says Boll, summarising the situation.
The researchers hope that general design guidelines can be developed from the results in the long term. So in the future, anyone who wants to put buttons on devices or everyday objects could build on the scientific recommendations.
Ultimately, RIME is about simplifying everyday life for the benefit of people. “I’m lazy; I wanted fast, simple controls,” Pourjafarian laughs when asked what motivated her to develop her printer. To illustrate this, she presents a yoga mat made of cork. The upper right end of the mat has familiar multimedia control icons printed in silver-grey ink. In this way, the training video or music on the tablet during yoga can be easily controlled from the mat’s surface instead of on the tablet, which may cause disruptions during a workout.
But it’s also about giving users the freedom to design their own environment. Therefore, the blueprint for “Print-A-Sketch” is available to everyone on the GitHub platform. Pourjafarian speaks of self-determination. Boll on the democratisation of manufacturing. The idea behind this is the same: we can help develop the home of our future. Develop it with us!