Carnegie Mellon Students Release Ed Software for Deaf and Hard of Hearing

• By Dian Schaffhauser
• 09/22/15

Carnegie Mellon University students have released two new, free educational software products intended to help students who are deaf or hard of hearing. SignBook is an open source program that allows teachers to create custom video dictionaries of local sign language. Speak Up! is a set of games that help children learn how to vocalize sounds. Both sets of programs come out of the university's TechBridgeWorld, which is dedicated to helping people around the world with technology.

SignBook is a platform for teachers to create a local database of animations depicting a sign. It was specifically launched for teachers at Mathru Center's Free School for Hearing Impaired Children in Bangalore, India. Although ample reference material exists for teaching American Sign Language, developer Erik Pintar noted that there are more than 300 other documented sign languages without such supports.

This fifth-year senior in human-computer interaction and electrical and computer engineering worked with the Mathru teachers to create a program that would allow them to make a custom video dictionary of the local sign language, Kannada, in order to support new teachers and staff in learning the language too.

SignBook includes the ability to capture sign videos and relevant pictures for each entry and categorize entries by topic and is usable by any teacher who wants to create a dictionary for any sign language.

Speak Up!, created by a student team, is a series of games intended to motivate students to make sounds and to give them visual feedback on the volume and pitch of the sounds. The idea is to prevent teachers working with students in grades 1-4 from having to make loud sounds themselves as prompts for their students, which can be hard on vocal cords.

As explained by Amal Nanavati, a sophomore computer science major who worked on the team, a game might cause a picture to appear and move on the screen if the child successfully makes a noise. In one game, the sound will cause a rickshaw to move; to keep it climbing on a hill, the student has to sustain the sound. Another game causes a bird to fly at different altitudes, depending on the pitch of the sound. In a third, a fish swims up or down depending on volume.

"Teachers would use these in their classes to keep students more engaged," said Nanavati.

Speak Up! includes 10 games that can be used by teachers with little or no speech training and that can be adapted to fit a given curriculum or the local culture.

Both programs were created during the summer by Carnegie Mellon students participating in TechBridgeWorld's Student Technology ExPerience (iSTEP). iSTEP allows the students to conduct technology-oriented research abroad to fill unmet needs in the communities where they work.

"By providing teachers with the tools to create their own signs and come up with their own games, these solutions have the potential to be relevant for different communities and their unique needs," said M. Bernardine Dias, associate research professor of robotics and TechBridgeWorld founder and director. "By releasing the software open source, our hope is that other educators will find our tools useful and that the open source community will build on the work we started."