WeBooks

Cuddling with your child, enjoying a picture book -- one of the joys of parenthood. These moments can build children’s language and story understanding, but not all parents read with children in ways that promote these abilities. Many children enter school lacking the vocabulary and comprehension skills needed for school success.

An adaptive digital tool that builds parents’ ability to support children’s language, matches the complexity of the book to a child’s level, and provides games tailored to the child that reinforce vocabulary and comprehension would be a game-changer for society’s most vulnerable students.

An interdisciplinary team with expertise in language, cognitive science, and artificial intelligence, working with programmers from Adaptive Technology Consulting, LLC, a small company specializing in research-based software product design, has developed such an adaptive tool – an eBook application that can solve a pressing societal problem in a scalable, cost effective manner.

Children from low-income homes and dual language learners lag far behind more affluent and monolingual peers in aspects of language critical to academic success. Parents with fewer resources and less education have fewer conversations with their young children using less-complex language, compared to more educated parents.

Between ¼ and ½ of children in the United States receive limited language input; they process language more slowly and have smaller oral vocabularies as early as 2 years. Dual language learners (children of recent immigrants and displaced persons) frequently lack sufficient exposure to the majority language used in school. All too often, they are unprepared for the high-level reading competencies required for school success.

Children’s kindergarten language competence predicts later language, math, reading, even social abilities — skills students must develop in preparation for 21st century employment. Many children must make extraordinary strides in language learning in the preschool years to be able to succeed in school. Given the early point in development at which the long-term trajectory for language development is established, there is a pressing need to find practical, tech-based strategies to help parents, preschool teachers, and caregivers promote children’s language growth.

We aim to support all parents’ hopes that their children succeed in school, including those with less economic/educational advantages, children with developmental challenges, and parents whose language differs from the one used in school.

Strategies to build a strong language foundation are effective, but many working parents don't have time to attend in-person training. A simple, accessible solution is needed—one that can be used at home.

In pilot work with an earlier eBook, urban and rural parents with fewer socioeconomic advantages appreciated strategies they learned from an eBook character. They realized how much their young children were capable of learning. Parents of all socioeconomic backgrounds wanted a more adaptable eBook.

The platform ultimately will support content tailored for monolingual and dual language learners, and for a range of neuro-atypical groups (individuals on the Autism Spectrum; children with language and hearing impairment). Speech Language Pathologists are eager to use WeBooks to support clients, noting adaptability of the eBook to the child’s language level and the parent strategies. Preschool teachers could also benefit, as teacher-child conversations in preschool classrooms frequently are inadequate to support strong language development.

WeBooks is a prototype eBook application for early childhood learning.  Based in the learning sciences, WeBooks elicit parent-child conversation by adapting the story on the screen to the child’s current language level.

A helpful character in the corner of the page (Barkley the dog) models proven strategies for parents to support their child’s learning in the moment. These prompts adapt across readings: as parents begin to use the strategies (defining difficult words and asking open-ended questions), hints change or become less frequent.

The novel platform leverages multimedia content (text, audio, and animations) and incorporates multi-modal input signals (eye gaze, speech, and task performance) to create a dynamic and adaptive learning experience.   

In pilot work, parents of lower socioeconomic status talked three times as much with their children and used more diverse language reading a WeBook with the helpful character, versus using the plain eBook. By the end of the session, parents took over asking questions and children’s talk increased. After eBook reading for 2 weeks at home, WeBook families talked significantly more than control group families when reading a different story in a traditional paper book.  

Why eBooks?

* Books introduce children to many situations beyond those of daily life and contain more diverse words than adult conversation. Parents across socioeconomic group use more varied vocabulary/more words while reading than in other activities. Electronic books engage children’s interest, and a tablet can contain a whole library.

*Audio narration in eBooks may help parents who are not confident reading aloud. WeBooks story narration and parent supports can be switched between languages (currently English and Spanish). For instance, narration can be in English and parent supports in Spanish.

* While reading, parents and children share looking at pictures that illustrate the meaning of words. WeBook animations and sound effects illustrate difficult vocabulary as the story is narrated. In research, audiovisual enhancements aid learning, particularly for children at risk for school failure.

An interdisciplinary team with a wide range of expertise is applying
recent technical innovations to realize the creation of this platform,
including eye tracking, speech recognition and language understanding,
closed-loop and adaptive systems, and machine learning/artificial
intelligence. 

WeBooks is a universal system that can be applied to any text for children. Underlying our prototype is a process for supporting language, and software that can be used with any content by educators and content developers (e.g., children’s book publishers). 

This solution draws on our team's extensive knowledge of typical and atypical language development and on our collaborators' expertise in dual language learning. Team members include leaders in early childhood literacy and experienced preschool teachers and clinicians. The content and parent strategies in WeBooks are backed by research and knowledge of how young children learn.

Our engineering team brings expertise in developing AI systems to support learning games for youth with autism (e.g., social games, a driving simulator). They understand both machines and young people.

The adaptive WeBook application utilizes a game engine (Unity) to create the interactive, dynamic content, and eye tracking technology (currently Tobii remote eye trackers) to capture visual attention metrics. Future development will focus on eye tracking using the device's front-facing camera. We are currently working to include real-time speech recognition of parent-child talk using cloud-based services (e.g., Microsoft Azure Cognitive Services, Google Cloud, Google’s Dialogflow Natural Language Understanding (NLU) engine with predefined intents). Touch input from companion assessment games gives the system feedback on child uptake of story content and vocabulary. 

During initial testing, parents of low SES expressed being tired after work; they appreciated built-in reminders to ask questions while reading with their children, the simplicity of the character “popping up” and modeling language-learning supports. Speech-Language Pathologists strongly agreed that the eBook would be useful working with children with speech/language impairments. 

Short-term outcomes are giving parents skills for building children’s language. Low-SES parents remarked on how much their children actually understood when questioned. In our research, they adopted the modeled language-support strategies such as asking questions about feelings and connecting between the story and their child’s life.  

An anticipated short-term outcome: WeBook features (animations, definitions, comprehension questions) might also teach vocabulary and support story understanding when children use the book by themselves. This outcome would broaden the usefulness of WeBooks to include a variety of family literacy practices (e.g., children’s solo use of narrated eBooks).  

Desired medium-term changes of behavior and practices: In a higher-SES sample of families who used the eBook for 2 weeks, parents generalized the strategies learned from WeBooks to paper books. A second mid-term outcome (currently being tested) is children learning vocabulary from the WeBook reading interaction with their parents.  

Long-term outcome: Much research shows that over early childhood, parent conversations support children’s language learning. The goal of WeBooks is to help parents develop the habit and skills of rich conversation, leading to children better prepared for school success.  

We are in the testing stage with our prototype WeBook. In the last 3.5 years, we have collected data with and interviewed 173 families using earlier eBook versions with a supportive character; this number includes 73 families who used the eBook at home for 2 weeks. We also interviewed a group of 15 Speech-Language Pathologists who pilot-tested the eBook as part of a workshop. We are in the process of collecting data with an additional 50 families using the current build of the WeBook including eye tracking, and will use automatic voice capture of parent-child talk to tune up the speech recognition feature.  

In one year, we hope to be doing iterative testing funded by an IES grant with 100 children to further develop our knowledge of the eBook features (animations, definitions, comprehension questions) that best support children’s learning of different kinds of vocabulary words (concrete nouns, abstract nouns, verbs, adjectives) from eBooks.  

In five years, we hope to have a series of proof-of-concept adaptive WeBooks developed, demonstrating a commercially viable universal system based on AI that can be applied to any text for children. The developed WeBooks can be marketed to parents of the 20 million preschool children in the U.S., parents who, regardless of SES, grew up as “digital natives” with electronic media and readily adopt new tech innovations. The universal system underlying WeBooks is intended to interest children’s book publishers, educational and textbook publishers, and companies developing children’s tablet applications.

Within the next year, we aim to produce a fully-operational prototype WeBook application that can be immediately used for data-gathering in order to demonstrate proof-of-concept and preliminary efficacy. Ongoing data collection using the current build of the WeBook with 50 families will help further develop the AI components of the prototype. Concurrently, the language team will be finishing a second, partially-completed WeBook story to again include two levels of text difficulty, two languages (English and Spanish), and a suite of animations, sound effects, and parent supports (definitions and questions). 

Within the next 5 years, we aim to (a) continually refine the AI and Machine Learning components of the platform to create more fluid and intelligent interactions, (b) further develop the novel platform to support more sophisticated and engaging content, and (c) model learning using AI techniques to develop an adaptive modality wherein story content is dynamically adjusted to the ability of the reader in order to maximize engagement and learning. The goal of the language team is to specify eBook features to best support specific aspects of children’s language learning. This knowledge will ensure development of a universal system that is not only technologically advanced, but that effectively targets a broad range of words that preschool children must master for school readiness. In this time, we will be working to learn how to market this product to be accessible to those who need it most.

Technical barriers that must be overcome in the next year include (a) implementation and validation of the camera-based eye tracking module; (b) completion of the speech module that will generate insightful metrics regarding communication between the parent and child; and (c) creation of a content-authoring tool to facilitate rapid creation of new story content.   

In the next 5 years, technical barriers include (a) the development of a robust model of learning (e.g. a Hidden Markov Model) that will allow us to intelligently present training material and (b) the extension the platform to present content with greater customizability (e.g., animations).  

Financial barriers are the need to secure grants or other support to fund technical development across the next year and the next five years, as well as the work of the language team.  

Cultural barriers involve the challenge of designing WeBooks to appeal to and best support preschool-aged children and their parents of diverse cultural, socioeconomic, and language backgrounds; of testing the eBooks with a representative variety of the diverse parents who are potential users; and of learning how to market this helpful product to those with limited resources.

Technical  (a) Although less accurate than traditional eye trackers, implementation of the (accessible, affordable) camera-based eye tracking module will utilize the latest advances in webcam-based eye-tracking to adequately capture real-time gaze metrics.

(b) We will leverage existing speech services--e.g., Google Dialogflow; Microsoft Azure Cognitive Services (with which the proposing team has experience)--to capture robust real-time speech metrics.

(c) Development of a content-authoring tool to facilitate rapid creation of new story content presents user-interface design challenges. We will enlist experts in User Experience and design for guidance, integrating our team’s deep background in Human-Computer Interaction.  

Five Years: (a) Our significant experience in Artificial Intelligence will support the development of a robust model of learning (e.g. a Hidden Markov Model).

(b) Robust content support will be informed by expert consultation in User Experience.  

Financial: We will seek corporate/foundation funding through Vanderbilt’s Office of Corporate and Foundations Relations to continue work in coming months. We are applying for 3-years’ grant funding through the Institute of Education Sciences. The Engineering team likely will solicit National Science Foundation funding for challenges advancing knowledge in AI and machine learning. Seeking an SBIR (Small Business Innovative Research) award may be feasible in upcoming years.  

Cultural: We have developed connections with preschools serving diverse families, and are exploring collaboration with experts focused on culturally-meaningful translation between languages, as well as colleagues studying additional target populations (e.g., blind parents co-reading with their children).

Solution team members are faculty and research staff at Vanderbilt University, and employees of Adaptive Technology Consulting, LLC, a small company specializing in research-based software product design.

Vanderbilt supported the development of our solution with an internal Trans-Institutional Programs Award for the last 2 years.

The solution team includes 13 part-time staff: 5 faculty, 1 PhD-level staff research associate/programmer, 2 graduate students, 1 undergraduate honors student, 1 temporary staff programmer, and 3 contract programmers. 

The engineering team (Sarkar and Wade) has significant expertise relevant to the successful completion of the project, including extensive backgrounds in the development of closed-loop adaptive systems, Artificially Intelligent systems, real-time Machine Learning, and creation of platforms for dynamic presentation of complex content (e.g., a job interview training platform for individuals on the autism spectrum). The engineering team also has experience in the translation of university-based innovation to commercial products.   

The language team has many years of collective research experience in early literacy development (Dickinson), atypical language development (Camarata), gamifying of language and literacy assessment (Goodwin), and young children’s use of and learning from media (Troseth). Several of us have turned out to be quite creative in content development. We are supported by graduate student expertise in family interactions and stories (Stuckelman) and bilingual learners (Flores and Oh). Several faculty have also been classroom teachers and clinicians.  

Based on earlier pilot research carried out by Troseth, our team has been meeting regularly for the last 2 years, combining our knowledge to design the WeBook proof-of-concept prototype and then going off to work individually on aspects of the whole before re-convening again. 

Vanderbilt financially supported our work for 2 years. Current resources include access to a videographer, graphic designer, and experts in technology transfer and legal protection of products. Vanderbilt offers training in entrepreneurship for graduate students and some consulting support to faculty.  We have support to develop, submit, and administer grants.

Working with the Fred Rogers Company, we developed an initial prototype eBook by adding an embedded helpful questioner into an eBook they had developed based on Peg + Cat, the PBS Kids TV show.

Work on this eBook with character was funded by a Health Resources Services Administration (HRSA) "Word Gap Challenge" cash award in 2016.

The original Big Dog Problem eBook without questioner (which we are using as our control eBook in research) can be seen here on the PBS Kids website

Continuing research with this eBook is funded by the Spencer Foundation and the kindness of the Fred Rogers Company (now Fred Rogers Productions) in hosting the experimental eBook.

With additional funding, one useful outcome is to finish the non-adaptive (but still quite effective) experimental eBook with questioner to commercial quality for wide release, possibly on the PBS Kids website. In theory, Fred Rogers Productions is open to this outcome. Additionally, in 2018, Lesli Rotenberg (Chief Programming Executive and General Manager, Children's Media and Education at PBS Kids), was thrilled we were using their property for research and excited by the possibility of further development if we acquired the necessary funding.

Pilot data is expected to validate the concept of the Adaptive eBook as an educational tool, provide a clear value proposition to customers, and lead to commercialization of the technology.  Initial steps towards commercialization may be taken by a combination of Vanderbilt internal efforts and/or a faculty start-up company, but is ultimately likely to need a collaboration with a publishing company as a strategic partner for access to copyrighted works and for sales and distribution channels.

The envisioned commercial product is a compilation of three distinct parts:  a hardware platform with proprietary software for collecting and processing the user feedback; the story content including story text, animations, parent supports (questions and definitions), and assorted assessment tools; and the mechanism for adaptation of the story content – the logic that adjusts the text and questions as the reader improves their performance. All will be protected by copyrights, and by patent rights for the adaptive feedback loop.

The initial market will be middle- and higher-income parents interested in supporting their children's early learning. Using a service subsidization model, parents' purchases would also support donating a copy to parents less able to afford the application.  Other initial (and eager) customers would be Speech-Language Pathologists, who often serve children of families with fewer advantages. Lower-income families might also gain access through school systems, public libraries. and non-profit foundations.  We will be able to gain access to this market through contacts and relationships with the Metro Nashville School System.

Continued funding of our work, beyond the next few years of grant funding and donations, would target marketing to initial targets and actual targets.

Our probable revenue model would be service subsidization, in which parents with more income (an initial target population) would "Buy one, give one" in order to provide the WeBook app free or at reduced cost to parents of low income (our actual target population) either directly our through libraries or foundations. Initially, the product would fulfill the desires of more highly educated parents to provide their children with learning apps that might advance their pre-academic skills; this strategy might also appeal to many educated parents' sense of social responsibility. Our pilot research with parents with more financial resources indicated that they also benefited from the model provided by the embedded character, increasing their child-directed conversations and use of challenging questions.

Other early adopters could be Speech Language Pathologists (to use in therapy or to give to parents to use at home). Numerous SLPs have expressed interest in obtaining the commercial product as soon as possible.

Corporations might offer a subscription to this research-based tool as a benefit to Gen X and Y employees, today's parents who trust technology for learning and appreciate employers with a social conscience. School contracts take longer, but would also be a good target.

A major barrier for our team is the need for business and entrepreneurship training, to get our innovation from prototype to fully developed product and into people's hands and homes.

Another barrier is the opportunity to learn from innovators who have successfully connected with and involved the populations who would benefit from their innovation in the design and delivery process. More focus groups and data from our target populations are needed. It would help our development as a team to learn successful strategies how to best approach and work with parents and children of diverse backgrounds.

Connections to technology consultants with relevant expertise (e.g., in User interface design) would be very useful.

Finally, the process of connecting to accelerators and funders seems  mysterious and help would be valued.

Omidyar Network: mentoring for product development, connections to expertise regarding diverse populations, connections to funders

Promise Venture Studio: mentoring, workshops, Early Childhood Demo week, connections to funders

Reach Capital's focus on Early Childhood Education, learning beyond classroom: funding

Bezos Foundation Early Learning focus: connections and funding

If we receive the AI Innovations Prize, we will use the funding to develop a model of user learning within the eBook application. With such a model, we will be able to explore whether an AI-based system can enhance parent and child learning in an eBook reading context, without the need for direct human instruction. In other words, an AI reasoner can be designed to optimally select the content configuration (e.g., challenge level in text and language-learning supports) that will most effectively engage and challenge readers, leading to the best learning.  

For parents, the AI reasoner would provide the right amount of training and support at the right time for the individual to begin using the taught strategies; then the system would adapt as the parent either continued or stopped using, or needed new ideas about, effective supports to engage their children in conversation and language learning. For children reading with or without a parent, text difficulty can become more (or less) complex to provide the right amount of challenge to interest (but not overwhelm) the individual child. For children reading alone, the AI-controlled character can step in to provide the right level of language-learning support, based on prior inputs (focus of gaze, attentiveness, verbal answers, and touch responses in companion assessment games). When fully developed, the WeBooks application will be extremely adaptable to different family situations, child language level, and degree of parent skill and involvement.

If our team is selected to receive funding from the Innospark Ventures Prize, we will enlist personnel to quickly advance the development and deployment of our AI-based learning model. Significant personnel time is needed to realize an intelligent model that determines the optimal content to be presented to the reader based on multi-modal inputs from task performance, visual attention, and speech signals.

Once developed for the eBook, the intelligent model also can be deployed for content with older learners, such as adults receiving job re-training with adaptable text and AV content tailored to support the individual’s areas of challenge, and a helpful agent that modifies support to the user’s level.  Such a tool could make a large contribution both to education leading to school success and to training the 21st century workforce for a changing employment landscape.