Generative Design Project
Digital Media, M.A. programme
Faculty of Communication and Environment
Rhine-Waal University of Applied Sciences
The idea of a project related to sign language is based on personal experience in educating a hearing-impaired child. It was sensed that when signing in public people were interested in observing conversation but kept their distance. Frequently aversion was shown to the unknown gestures and facial expressions. Sign languages might seem strange to outsiders who have little to no experience in this field. In addition, they might not feel competent to approach aurally handicapped people.
Public perception of these linguistic differences should be changed. Therefore, the project emphasizes on the beauty of sign language by visualizing hand motion. It is used as a representative for sign languages. Simple forms and characters replace the fingertips of the hands employing data based on actual gestures in German Sign Language. The signed text is a translation of music lyrics. Thus the visualization is presented according to the music. This forms a contrast and can attract audio or visual oriented visitors. Characters used in the lyrics are chosen in random order. No readable message is meant to be generated.
In contrast to common visualization projects, this is not aiming to gain deeper understanding of data. Still, reducing sign language to the two-dimensional coordinates of the fingertips can reveal information about the performed motion.
This experimental approach aims to trigger positive emotion through the visualization of motion. A novel experience of sign language is designed.
In contrast to common visualization projects, this is not aiming to gain deeper understanding of data. Still, reducing sign language to the two-dimensional coordinates of the fingertips can reveal information about the performed motion.
This experimental approach aims to trigger positive emotion through the visualization of motion. A novel experience of sign language is designed.
Music
The song ‘Leave This Island’ by Maxïmo Park was chosen because of the referred topics of isolation and togetherness. It is sung from the perspective of a person whose partnership is in a difficult phase where a closely related person is seeking distance. Figuratively speaking, their situation can show the relatedness of deaf and hearing people which belong together as being people, but are easily isolated due to their linguistic differences.
The lyrics were subdivided into eight states based on the content of the song. Each of them represents a part of the song’s content and is visualized in a proper form.
The lyrics were subdivided into eight states based on the content of the song. Each of them represents a part of the song’s content and is visualized in a proper form.
Applications
All applications created for this project were written in Processing. The first is used to record the coordinates of the fingertips according to the song. Coordinates are mapped to the canvas size and saved to be used in the interactive exhibits. Meanwhile a video was filmed that was used in the hands-on exhibit.
In this second application, the song is played while the previously recorded coordinates are used in each frame. At the same time new objects of the presently activated style are created. The activation can be evoked by user input or time based in the underlying algorithm that specifies the choreography.
In the third application, the Leap Motion controller provides a continuous data stream. It is read once in each frame to receive the current position of the hands’ fingertips. All other processes are identical to the second application except for the background. Here, it shows a filmed recording of the performance of the German Sign Language Translator.
In the third application, the Leap Motion controller provides a continuous data stream. It is read once in each frame to receive the current position of the hands’ fingertips. All other processes are identical to the second application except for the background. Here, it shows a filmed recording of the performance of the German Sign Language Translator.
Leap Motion controller
The main exhibit was projected on a wall and showed visualization and sound of the song “Leave this Island” by Maxïmo Park. Visitors could choose between 16 styles of visualization. Likewise they could switch between two versions of the visualization, one version with shape styles and a second version with character styles.
The underlying algorithm also changed between modes automatically.
Stills main exhibit
In an additional exhibit the film of the recording was shown in the background. Visitors could use the Leap Motion controller to create their own gestures. For this exhibit music was played on headphones.
Stills hands-on experience
