Socratic Mind

We are a team of researchers affiliated with Georgia Tech and University of California San Diego. This is a research project that we are working to develop into a for-profit organization.

Socratic Mind - AI-Powered Oral Assessment with Socratic Questioning

Socratic Mind (SM) is a scalable, AI-powered oral assessment platform leveraging Socratic questioning to engage students in deep, thought-provoking dialogues, challenging them to articulate their ideas and reason through problems in ways no traditional automated assessments could.

With Socratic Mind, students are encouraged to explain, justify, and even defend their answer to showcase their understanding. We believe that true comprehension is evidenced by the ability to explain ideas clearly and cogently. 

How does Socratic Mind work?

To begin using Socratic Mind (SM), teachers set up oral assessments through the SM web interface. This process is streamlined by pre-defined question templates that adhere to established frameworks like Webb's Depth of Knowledge and Bloom's Taxonomy. Teachers select a question template, provide a rough idea of their questions in one or two sentences, and an AI question designer will assist in refining these into high-quality assessment questions with corresponding rubrics. 

Once teachers finalize the questions, they share the assessment with their students via a web URL. Students can take the assessment using any internet-accessible web browser. During assessment, students engage with the AI examiner via speech or text to demonstrate material understanding and high-order thinking skills. After the conclusion of each assessment, SM offers students immediate, personalized feedback as well as suggestions on how students can improve. SM will also generate comprehensive reports for teachers, analyzing class performance and suggesting learning plan improvements. 

In the future, we plan to allow students to initiate oral assessments based on their interests or lesson materials for extra practice.

Technologies Behind Socratic Mind

SM has four major components. These components are:

1. AI Question Designer: Generate high-quality oral assessment questions, ensuring broad coverage across various domains.

2. AI Socratic Questioner: Employs Socratic-style questioning to conduct oral assessments, simulating the depth and interactivity of human questioners.

3. AI Performance Evaluator: Analyzes student responses against specific metrics to evaluate performance accurately and objectively.

4. Web Interface: A user-friendly web application for users to create, conduct, and manage oral assessments seamlessly. 

The web interface is built using popular web technology like Nextjs. The AI Question Designer, AI Socratic Questioner, and AI Performance Evaluator are currently powered by commercially available large language models (LLM) like gpt-4-turbo and Claude 3 with sophisticated prompt engineering. We plan to use fine-tuned Llama 3 in the future to reduce cost, improve domain specific performance. The speech-to-text functionality is powered by OpenAI Whisper and Deepgram Nova-2. The text-to-speech (currently in beta) is powered by Deepgram Aura.

Landing page with demo: https://socraticmind.com/

Note: We are aware of regulations like Children's Online Privacy Protection Rule ("COPPA"). We will take necessary steps to adapt our solution to make it COPPA compliant prior to piloting for K-8 students.

Importance of “Meta-Skills” in the Era of AI   

In the rapidly evolving educational landscape, the advent of AI presents a pivotal challenge: determining the essential skills for the next generation as AI increasingly handles routine tasks and knowledge. We contend that future education should focus on cultivating "meta-skills" (OECD Future of Education and Skills 2030; Gardner, 2011). These are foundational, higher-level skills that not only foster the development of other skills but also enhance their applicability and adaptability across various domains. Meta-skills, including problem-solving, self-reflection, logic, creativity, and judgment, are vital for adaptive and self-driven learning in a dynamic environment. 

Current education systems, rooted in traditional models, must evolve to meet this challenge. The crux of this evolution lies in nurturing learners' abilities to navigate complex problems, adapt to new situations, and engage in critical and creative thinking — areas where AI cannot easily match human capability.

Role of Oral Assessment in Cultivating Meta-Skills and Current Gaps in Education 

Oral assessments have been shown to significantly enhance student understanding across a broad range of disciplines, extending beyond more speech-heavy social sciences to include STEM fields like Mathematics, Chemistry, Engineering, and Medicine (e.g., Utami et al., 2021; Qi et al., 2023). 

Benefits of oral assessments, in addition to enhanced levels of understanding, include:

• Increase motivation and engagement (Delson et al., 2022, Qi et al., 2023), belongingness (Reckinger & Reckinger, 2022) and confidence (Sabin et al., 2021)

• Provide targeted, customized feedback and encouragement (Reckinger & Reckinger, 2022, Sabin et al., 2021)

• Enhance academic integrity (Reckinger & Reckinger, 2022, Qi et al., 2023) 

Importantly, recent research supports the benefits of oral assessment and verbalizing one’s thoughts for enhancing meta-skills, which allows students to be a critical and creative thinker in broad domains. They include: 

• Enhance verbal communication and teamwork skills (Gardner & Giordano, 2023)

• Enhance higher-order thinking skills, such as structuring thoughts, identifying logical flaws (Mahendra, 2023; Theobold, 2021; Yanai & Lercher, 2024)

• Enhance creativity by allowing for focused, iterative development of ideas through cooperation (Pilkington, 2019; Yanai & Lercher, 2024)

Despite their benefits, oral assessments face limited adoption due to logistical and time constraints for educators (Gardner & Giordano, 2023; Young, 2023). 

Proposed Impact 

We propose the following impact:

• Deepened Material Comprehension: Through the process of verbalizing their thoughts during oral assessments, students actively engage with the material, leading to a more profound grasp of the subject matter.  

• Enhance Critical Thinking: By challenging students with thought-provoking questions, they develop sharper analytical skills and a deeper understanding of concepts.

• Improve Verbal Communication: Regular interaction with the AI hones students' ability to express complex ideas clearly and concisely.

• Foster Self-Reflection and Judgment: Engaging with open-ended questions encourages students to evaluate their own thinking and learn from self-discovery.

Our team is a blend of distinguished academics and emerging researchers from Georgia Tech and UC San Diego with partnership interests from a K-12 educational organization.

Jui-Tse (Ray) Hung is a Computer Science (CS) Master’s student who has been researching large scale educational technology tools under Prof. Thad Starner that resulted in publication at the Learning @ Scale conference. Jui-Tse also has extensive industry software engineering and AI infrastructure experience from interning at Meta, Amazon, Scale AI that makes him well-suited for the technical development of Socratic Mind. 

Christopher Cui is an incoming CS PhD student at UC San Diego concentrating on Natural Language Processing (NLP), Reinforcement Learning (RL), and Large Language Model. Christopher has several publications in NLP and RL, making him well-suited for the AI research work for Socratic Mind. Additionally, Christopher also has extensive classroom experience as the former head teaching assistant for one of the largest introductory AI courses at Georgia Tech and has also been contributing to large scale educational technology tools under Prof. Thad Starner. 

Jeonghyun (Jonna) Lee is the director of research in education innovation at the Center for 21st Century Universities (C21U). As part of Georgia Tech's new division of Lifetime Learning, she is committed to conducting transformational research to enhance learning experience at all stages of life, including Pre-K-8 learners. Her expertise is students' learning motivation and engagement from behavioral, cognitive, and socio-cognitive perspectives. Based on her background as a learning scientist, Lee is interested in how technology facilitates effective learning and student success. 

Additionally, this project is also advised by 

• Prof. Thad Starner, specializing in human computer interaction

• Prof. Alan Ritter, specializing in natural language processing

• Prof. Prithviraj (Raj) Ammanabrolu, specializing in natural language processing and reinforcement learning

Last but not least, our team has got partnership interest from Georgia Tech Center for Education Integrating Science, Mathematics and Computing (CEISMC), which serves the preK-12 STEM education community in Georgia, U.S. through summer/after-school programs, teacher training, STEM curriculum development and evaluation research. According to CEISMC’s website, “annually, CEISMC programs impact more than 62,000 students and 3,500 teachers in over 99 school districts throughout the state of Georgia.” In particular, we have the opportunity to pilot SM in the CEISMC Horizon Program, a six-week summer learning experience which targets K-12 students from underserved communities. 

With funding and support from MIT Solve and Gates Foundation, we will collaborate with CEISMC (and more partners) to pilot SM for underserved K-8 learners to receive first-hand feedback and iterate our solutions. 

We built a prototype used during a pilot study where we replaced traditional essay assignments with SM’s interactive Socratic questions in a Computer Science course at Georgia Tech, which enrolls approximately 600 students. 77.8% reported that SM provides more educational value than traditional essay-style assignments (with 12.9% remaining neutral).   

We have also done another small pilot with roughly 40 students in a business strategy course at Georgia State University. We created an SM assessment for students to review strategy concepts. 88.2% reported that SM is more educational than traditional multiple choice assessment (with 5.9% remaining neutral).

Georgia

Novelty of Socratic Mind

Socratic Mind (SM) introduces a transformative approach to educational assessments by shifting the role of AI from a passive responder/assistant to a proactive questioner. Unlike most AI tutor products that primarily respond to students’ queries, SM engages students through Socratic questioning, challenging them to think deeply and articulate their reasoning. This method pushes learners to not just receive information but to interact dynamically with content, fostering a deeper level of cognitive engagement.

Our solution places a strong emphasis on the oral component of learning, recognizing that verbal expression is crucial in developing and demonstrating understanding (Chi et al., 1989, King A., 1994). This focus is particularly innovative as it assesses students’ ability to articulate thoughts clearly and critically, a skill often overlooked in conventional assessment methodologies.

Moreover, SM uniquely addresses the assessment of advanced meta-skills by evaluating students' abilities to identify logical fallacies, challenge existing assumptions, and ask the right questions. These capabilities are critical for developing sophisticated thinking skills yet are rarely the focus of existing educational assessments.

Lastly, the requirement for students to explain their thought processes during oral assessment allows SM to capture the nuances of their understanding. This is particularly valuable for identifying where and how errors or misconceptions occur, providing educators with insights that are not typically accessible through standard testing methods.

How Socratic Mind Impacts Education and EdTech

We envision SM to serve as a critical piece of learning engineering infrastructure, enabling the reliable and accurate evaluation of various pedagogical approaches. Just as measurements are essential parts of scientific experiments, SM provides a robust method to assess educational outcomes and improvements. This capability could catalyze a significant shift in how educational effectiveness is measured and enhanced.

Additionally, there is a noted gap in the availability of high-quality, multi-turn dialogue datasets for developing intelligent tutoring systems (Macina et al., 2023). By collecting a rich corpus of oral assessment dialogues, SM contributes a valuable resource to the field. While these datasets are not specifically tutoring dialogues, they closely parallel such interactions and can significantly advance the development of intelligent dialogic tutoring systems.

A Paradigm Shift in Education

Socratic Mind represents a significant leap in educational technology for the era of AI. By supporting Socratic Mind, you're not just supporting an educational tool. You're championing a movement towards an educational paradigm where self-reflection, critical thinking, and creative problem-solving play integral roles at the forefront of education, preparing students to thrive in an ever-evolving world.

AI at Socratic Mind 

Our work encompasses Natural Language Processing (NLP), Speech Recognition and Text-to-Speech. 

Currently, we use OpenAI gpt-4-turbo as well as Anthropic Claude 3 models to power our application. We use OpenAI Whisper and Deepgram Nova-2 to power our Speech Recognition. For Text-to-Speech, we use Deepgram Aura.

With funding, we will start fine-tuning Llama 3 to build models that are more performant yet can operate at lower costs. 

Note: We are aware of regulations like Children's Online Privacy Protection Rule ("COPPA"). We will take necessary steps to adapt our solution to make it COPPA compliant prior to piloting for K-8 students. This may include building our own LLM (e.g. fine-tuned Llama 3) and speech recognition modules to remove dependence on OpenAI, which imposes restrictions on users under the age of 13.

Socratic Mind Web Application

Socratic Mind web application is built using popular technologies. The frontend is built with Typescript, Nextjs, and Tailwind CSS. Postgres database is used for data storage, and Redis is used as in-memory cache.

SM web application can be deployed as a Docker Container to any cloud  platform that supports hosting containers (such as AWS ECS).

Proven Benefits of Oral Assessment

The benefits of oral assessments and verbalizing one’s thoughts through active self-explanation are well-documented in literatures:

• Improve material understanding and content mastery (Qi et al., 2023; Reckinger & Reckinger, 2022)

• Increase motivation and engagement (Delson et al., 2022, Qi et al., 2023), belongingness (Reckinger & Reckinger, 2022) and confidence (Sabin et al., 2021)

• Provide targeted, customized feedback and encouragement (Reckinger & Reckinger, 2022, Sabin et al., 2021)

• Enhance academic integrity (Reckinger & Reckinger, 2022, Qi et al., 2023) 

• Enhance verbal communication and teamwork skills (Gardner & Giordano, 2023)

• Enhance higher-order thinking skills, such as structuring thoughts, identifying logical flaws, engaging in deep reasoning (Mahendra, 2023; Theobold, 2021; Yanai & Lercher, 2024)

• Enhance creativity by allowing for focused, iterative development of ideas through cooperation (Pilkington, 2019; Yanai & Lercher, 2024)

Demonstrated Success of Intelligent Tutoring System (ITS) in K-8 Settings

To some extent, SM resembles an intelligent tutoring system that is specifically designed for oral assessment and enhancing meta-skills.

Researchers have tested the effectiveness of conversational intelligent tutoring systems in K-8 context and shown positive impacts on various domains of learning, including language (Liu et al., 2024), STEM (Ward et al., 2013, Lippert et al., 2020), and history (Mack et al., 2019). 

Promising Results From Early Pilot 

Additionally, our pilot with 600 college students in a Georgia Tech Computer Science course shows that 77.8% out of the 225 students who filled out the survey found Socratic Mind to be more educational than traditional short essay questions (+ 12.9% neutral). And, from another pilot with 40 students in a business strategy course at Georgia State University, 88.2% out of the 17 students who filled out the survey reported that SM is more educational than traditional multiple choice assessment (+ 5.9% remaining neutral).

Although our initial pilots were conducted in higher education settings, the positive outcomes observed offer promising implications for potentially adapting Socratic Mind to K-8 education. We are eager to collaborate with MIT Solve and the Gates Foundation to tailor and extend the advantages of Socratic Mind to meet the unique needs and aspirations of K-8 students.

Preventing Bias and Hallucinations

A primary risk with using Large Language Models (LLMs) in educational settings is their tendency to replicate biases from their training data and to produce incorrect or “hallucinated” information. This can be especially problematic in contexts that require sensitivity to different cultural norms, or where the educator's reliability and trustworthiness are critical.

To mitigate this, we plan to implement a cross-validation system where outputs from the main model are checked by a suite of secondary models, including toxicity classifiers and other LLMs prompted to be especially critical to such misattribution or misinformation. Additionally, instructors will have the capability to intervene and provide sentence-level feedback on both AI and student responses should our safeguards fail. 

Moreover, when we gather the dataset to finetune and benchmark the performance of our AI models, we will collect data from learners of various backgrounds, particularly Black and Latino learners to ensure that our dataset contains diverse training examples and does not bias towards data from a particular demographic group.

Equity

In our commitment to fostering equity at Socratic Mind, we recognize the importance of cultural and linguistic diversity, especially for Black and Latino learners. To this end, we will introduce customizable features that adapt our AI to various cultural contexts and norms. These adaptations will be guided by the learner's demographic background, allowing our AI to modify its communication style to align with culturally relevant teaching methods and linguistic nuances.

Additionally, we will enhance our tool's language capabilities to better support non-English speakers, including ESL students and those from bilingual communities. This multi-language support is not merely a feature—it is a fundamental component of our strategy to bridge language barriers and create a more inclusive learning environment. By doing so, we aim to ensure that all students, regardless of their native language or cultural background, have equitable access to educational opportunities and can fully engage with the content in a manner that resonates with their personal and community identities.

Furthermore, we recognize the potential for our tool to inadvertently favor students from specific demographic backgrounds. We are committed to conducting ongoing research to monitor and address any disparities, ensuring that Socratic Mind is a universally beneficial and equitable educational tool. Such research may include formal studies on how our solution improves the learning outcomes of learners from different demographic backgrounds, and what strategies can be implemented to effectively mitigate the disparities.

Accessibility

We acknowledge that accessibility is also a concern, particularly for students with learning or verbal disabilities who may find oral assessments challenging. To address this, we will offer the option for students to type their responses if they are uncomfortable or unable to engage verbally. Our future plans extend these considerations to students who experience anxiety either when interacting with the AI or in a test-taking environment. We are exploring technologies that can detect emotional fluctuation in voices, enabling human instructors to intervene if the student is experiencing distress. 

Socratic Mind is a research project at Georgia Tech. We have about three full-time students and three full-time staff at Georgia Tech contributing to this solution.

We also have three full-time professors as advisors. 

See the bottom of https://socraticmind.com/ for a list of members contributing to Socratic Mind.

Our solution is already available for pilot testing at small scale if the automated performance evaluation is not required. And we have already piloted our solution with 600 Georgia Tech students twice. 

By end of Q3 this year, we plan to 

• Collect training data and fine-tune Llama 3, enabling us to scale our solutions cost-effectively to a broader audience and reduce our dependence on OpenAI, which imposes restrictions on users under the age of 13.

• Have the automated performance evaluation feature ready.

• Do a few more pilot evaluations at Georgia Tech to keep iterating our solution

Overall, we believe that our solution should be ready for large scale pilot by end of this year. 

Note: We are aware of regulations like Children's Online Privacy Protection Rule ("COPPA"). We will take necessary steps to adapt our solution to make it COPPA compliant prior to piloting for K-8 students.

Research-Driven Impact: Our team is committed to conducting rigorous learning science research. We aim to transparently demonstrate the efficacy and efficiency of Socratic Mind and its capacity to significantly enhance learning outcomes. Our findings will be periodically published, contributing valuable insights to the broader educational community and continually refining our approach based on real-world data.

Technological Innovation and Affordability: Leveraging our extensive background in software engineering and NLP research, we are uniquely positioned to develop Socratic Mind using cutting-edge technologies while maintaining cost-effectiveness. By integrating advancements from NLP and employing Parameter-Efficient Fine-Tuning (PEFT) on open-source LLMs like Llama 3, we anticipate reducing costs significantly. Our goal is to offer this transformative educational solution at approximately $4 per student per month, making it a financially viable option for schools and learners across diverse economic backgrounds.

Expanding Partnerships: To further ensure the reach and impact of our solution, we plan to broaden our network by forming more partnerships with K-8 educational organizations. This expansion will be supported through direct outreach efforts and potential support from MIT Solve and the Gates Foundation, enhancing our ability to meet the needs of underserved communities effectively.

We are particularly excited about the potential to collaborate with MIT Solve and Gates Foundation to overcome several significant barriers:

Funding for Scalable Solutions 

We believe that with the financial backing from Gates Foundation, we can scale our solutions to reach millions of learners, especially to underserved learner communities, thereby democratizing access to quality education.

Strategic Partnership

Connections with K-8 schools and educational organizations are vital for real-world impact. We hope to leverage the extensive network of MIT Solve and the Gates Foundation to forge partnerships that will allow us to deploy our solutions in environments where they can make the most difference.

Cultural and Market Access

We aim to navigate cultural and market challenges that often impede the adoption of new technologies in diverse educational settings. With the guidance from MIT Solve and the Gates Foundation, we can better tailor our solutions to meet the nuanced needs of various cultural contexts and market dynamics.

We are not just looking for support; we are seeking to be part of a community that shares our commitment to reshaping the future of education through innovative solutions that cater to all learners, irrespective of their background.