LinkIO: Tangible Interface as Bridge

2023-2024

Teamwork: Yue Zhu, Zhiyuan Zhou, Zhitao Yu
Advisor: Cun Lin, Rao Yin, China Academy of Art

Exhibition and Award:

LinkIO: Tangible Interface towards Space
Dutch Design Week, as part of ”Fly me to the Moon” exhibition by Tongji University | Oct 2024
Design Intelligence Award (DIA) 2024 | Honorable Mention | October 2024
The Kunpeng Global Design Award (KPA), Digital Design Track | Excellence Award | April 2024
Selected to exhibit at SaloneSatellite Shanghai 2024, Salone del Mobile.Milano Shanghai | April 2024
Kilobot
China College Students Creativity Festival(CCFCS) | Nomination Award | April 2024
Euphony: A Novel Concept in Instrument Design Based on Bodily Interaction
X Journey Xiaomi Future Design Award 2024 | Third Prize | January 2024

Contribution:

Led the project, including conceptualization, experiments, design, implementation, and academic writing.

Tangible interface towards space

Ubiquitous computing has witnessed to develop with TUIs, but not limited to. Live Wire is one of the earliest examples of visualizing digital information in a tangible representation which ideally couples to the identity and physical configuration of representational artifacts and makes the system easier to understand.

Figure 1: Research on discrete TUIs in space

Later examples like HERMITS explore a modular system for table-top, wheeled robots offering a variety of interactive functionality, one the applications of which is to serve as storytelling systems on tables. AeroRigUI uses rigging swarm robots on ceilings for spatial interaction as actuated TUIs and Rovables proposes an interaction space for mobile on-body TUIs to develop application scenarios including on-body sensing, modular display, tactile feedback, and interactive clothing. Those actuated TUIs in everyday space have depicted a scenario of ubiquitous computing where the space has gradually merged into a whole. However the trend, these TUIs are still discrete in space, which is hard to achieve the scenario of natural human-computer interaction.

Could we create a form that unifies these segments of space?

Figure 2: Scenario of LinkIO : TUIs as bridge

Based on the scenario, we extend the flexible surface climbing capacity of Rovables and hope it work as a bridge among these discrete spaces and also between human and space. Meanwhile we simplify the process of hardware and software deployment to lower technical barriers for artists and designers for quick prototyping & testing, also for artistic live performance, making the system more accessible.

This adaptation allows tangible interfaces to move vertically along flexible surfaces. Our design transcends being merely a technological device; it serves as a bridge connecting different spatial facets and the space to its users. The collective behavior of these robots in space, functioning as a tool for real-time data mapping, not only gathers data but also physically represents it, intuitively displaying complex data flows and system statuses through their positioning, movement, and interaction. (This point is further developed for artistic live performance in Enlivening Performance Art: Enhanced Interactivity through Embodied Cognition and Real-Time Physical Visualization on Swarm Tangible Interfaces)

The inclusion of IO ports in the vehicle design retains modularity and expands possibilities for human-computer interaction by allowing the integration of various input and output modules. This design of a tangible interactive interface, through its modularity and flexibility, enhances spatial interaction experiences and paves the way for future applications, such as the envisioned spatial tangible reminder systems and creative aids for designers. It explores the potential of merging spatial physical interaction with AIGC (Artificial Intelligence Generated Content) workflows.

Adaptation and application for space-linking scenario

The ‘rovable’ units have been structurally(a) and circuit-wise adapted(c). We replace the initial customed PCB board with ESP32C3, leveraging its WiFi capability to control motors and wheels, and adapt the circuit design as is shown below(b). Accordingly, the structure design is also modified. And We tested wheels with different 3D structures and found that the tread pattern affects the robot’s gripping force(c). The more tread fold inflection points there are, the stronger the fabric adhesion ability. Then we extended the on-body climbing to spatial climbing and designed an installation for test and demonstration. This enhancement facilitates space-linking scenarios, allowing for like the dynamic integration of these units in multi-dimensional performance environments and utility of working space etc. Such cluster control is essential for executing intricate interactive installations that extend across various planes, including vertical and overhead spaces. However, its efficiency is temporarily limited, and we will optimize the cluster control module in future work. On the other hand, our deployment with ESP32C3 significantly lowers the threshold for learning and development for artists.

Figure 3: Structure and circuit design of LinkIO for space-linking scenario

Introduce LinkIO and Physical Visualization to Live Performance

Joint action performance: Bodies and Minds Moving Together, to achieve a collaboration with inter-coherence.

BOX showcases a collaboration between human and robotic arms to produce an illusionary stage performance, as an attempt for computer to understand human’s intentions and to better achieve the joint action performance. Threading Space” (2023) investigates the alteration of human spatial perception through the control of swarm robots , whereas it’s more like pre-arranged performance rather than improvising one.

Although these works in the realm of human-robot collaborative performance have already made some attempt in pursuit of better Joint action performance, we found them still can not achieve inter-coherence both spatial and temporal in live performance.

Accordingly, we introduced both LinkIO and the approach of Physical Visualization to the realm of Live Performance, providing with natural flow and continuous movement for spatial coherence, and real-time feedback and synchronization for temporal coherence.

Overall system design

Figure 4: Interactive system overview and control artchitecture

Application: Physical visualization and creative visual pattern for Artistic live performance

Our interactive improvising performance framework was tested in a live exhibition, offering a unique opportunity to observe its real-world applicability and gather direct feedback from participants. The demonstration showcased the possible integration of human performers and robots, highlighting the framework’s capability to facilitate an intuitive and collaborative artistic process.

Figure 5: Demonstration on exhibition

In our framework, ArUco markers are employed for the precise positioning and attitude estimation of TUIs using the OpenCV library, a method underscored by its adaptability across varying environmental conditions. These markers are affixed to the TUIs via modular magnetic units, ensuring both ease of application and tracking stability. Complementing this setup, IP cameras are positioned to monitor TUIs, facilitating real-time data assimilation into our remote control system to collect and visualize spatial data and embodied data of TUIs effectively. Our approach integrates digital physicalization and stream diffusion techniques to dynamically convert data into visual formats (Figure 4.b), enhancing audience engagement through visual storytelling. We employ the StreamDif f usion (Figure 4.b4) to translate data into visual information using the method of digital physicalization. This process is enhanced by the residual classifier-free guidance (RCFG) algorithm, facilitating rapid image style transfer based on spatial art data, thus enabling real-time interactive art mapping. To augment this, we incorporate creative algorithms from OpenProcessing to diversify visual effects, expanding artistic expression in performances, broadening our visual repertoire and fostering innovative avenues for spatial interactive art. These methodologies collectively empower our system to produce dynamic, responsive visual content that significantly enhances the depth and engagement of our interactive installations.