Students in the Carnegie Mellon University chapter of Design for America, which include Indian Americans Sashank Gogula and Mehar Sawhney, have developed a free mobile app that uses visual cues to help children and adults — particularly those with language difficulties — form meaningful sentences that are grammatically correct, the university said in a Sept. 18 report.
The education app, called Sentence Mosaics, uses photos and color-coded parts of speech to prompt users as they construct sentences, it said.
The app is useful for elementary students and for a wide range of children and adults who are on the autism spectrum, or who have developmental disabilities, hearing disorders, stroke-related language deficits, dementia and other problems with language, said Alyse Baker, a certified speech-language pathologist who has honed the system for decades, the report added.
"Alyse has used colored blocks, colored cards and foam tiles attached with Velcro to a foam board as aids, but that's not practical for use outside the classroom," said Sawhney, a DFA@CMU member and a senior majoring in mathematics and physics, in the report. "We worked with her to translate her system into an iPad app that can be taken anywhere."
Baker developed Sentence Mosaics based on longstanding research and practice within the field of speech-language pathology. She has long recognized that transforming the system into an app would make sense, but found the prospect daunting, the university report said.
David Kosbie, an associate teaching professor of computer science at CMU, connected Baker two years ago with DFA@CMU, an interdisciplinary group of students who design and engineer solutions for social good.
A group of DFA members, including Sawhney, computer science majors Elizabeth Board and Gogula, and cognitive science major Elizabeth La, worked with Baker to translate physical aids into digital ones. Jennifer Chou, who graduated last spring with a computer science degree, also was a major contributor to the project, the report said.
Photos are key to the system. Users write sentences that describe actions depicted in photos, preferably photos that include the student or show a scene from their everyday life. Board said the app can import photos from an iPad or iPhone, it added.
The app is designed to be used with a teacher, parent or other "coach," who helps by loading photos into the app and by designating a string of colored boxes that signal to the student which parts of speech — nouns, pronouns, verbs, adverbs, adjectives, etc. — are necessary to describe the photo, the report added.