Capti SBA

Capti SBA is a platform that uses AI to create localized, personalized, engaging, and predictive Scenario-Based Assessments (SBAs) for students in Grades 3-12. Capti SBA is being developed on top of our existing Capti Assess platform that offers a unified framework and UI for rostering, assignments, reports, etc., which will greatly expedite the development and deployment of Capti SBA. 

SBAs are a type of performance assessment that reflect modern learning theory and are well-suited to assess complex reading and writing skills (McCarthy, 2023; Pearson, 2020; Sabatini, 2019a). As reading tests, SBAs present students with a context and purpose for reading a set of thematically related sources within a simulated digital environment where they engage with simulated peers and a teacher to comprehend an issue, solve problems, make decisions, and apply knowledge to a new situation (O'Reilly, 2018). Reading tasks assess a spectrum of skills, ranging from locating key information to viewing the text through multiple perspectives (Braasch, 2018). Therefore, Capti SBA: 1) provides a purpose for reading, 2) promoting coherence among materials, 3) measures test takers’ background knowledge, 4) promotes collaboration, 5) simulates literacy contexts of practice, and 6) promotes interest and motivation (O'Reilly, 2013; O’Reilly, 2014).

Capti SBA propels educational-assessments forward by utilizing digital technologies to measure students’ abilities to critically think while exploring diverse sources, solving real-world problems, and unleash their creativity while working toward specific goals. Reflecting the multifaceted demands of modern readers, Capti SBA challenges readers to master purpose-driven comprehension, navigate digital landscapes, and collaborate effectively. Capti SBA utilizes AI by including: A) customizable assessment form templates with a menu of AI prompts and supporting NLP algorithms to generate content, and B) a Wizard that walks users through each step of creating reading assessments. 

Since there are few commercially available SBAs, and, for the most part, they are not culturally responsive or instructionally aligned to districts’ localized curricula. The need for SBAs, especially those aligned to curricula, would greatly change the ways students are assessed; however, SBAs require a great deal of time and effort to create, and teacher-created SBAs may fall short of the expectations called for by state standards. Unless there was a tool to facilitate SBA creation, guided by learning science principles (Graesser, 2022; NASEM, 2018), reading and discourse processing research (Britt, 2017; Magliano, 2017; Magliano, 2007; McNamara, 2009), and sound, evidence-centered measurement design that takes culture into account (Mislevy, 2018; Mislevy, 2006; Oliveri, 2019). The tool now exists in Capti SBA, a platform that empowers teachers to create localized SBAs for reading that are culturally responsive to the students in any given classroom and aligned to the school’s curriculum and state standards. Teachers use an AI-powered wizard to 1) define grade and standards addressed on the assessment, 2) define the topic of the assessment, and 3) rapidly generate SBAs with sound measurement properties.

The latest National Assessment of Educational Progress (NAEP) revealed that 67% of Grade 4 and 8 students continue to read below a NAEP Proficient level, meaning they likely have partial mastery of the fundamental skills for proficient grade-level reading (Irwin, 2022). Results are worse for students who are Black, Indigenous, and people of color (BIPOC), especially for Black students who scored 28 and 25 points lower on average than white students in Grades 4 and 8, and for students experiencing poverty who scored 28 and 23 points lower on average than students not experiencing poverty in Grades 4 and 8. The disparity is not new. In fact, reading achievement scores have been stagnant for decades with achievement for BIPOC students and those experiencing poverty faring far worse than their white peers or students not experiencing poverty (NCES, 2019). 

Because of reading achievement gaps, scholars have long called for culturally relevant, responsive, or sustaining education that centers on students’ lived experiences to increase achievement (Gay, 2002; Ladson-Billings, 2022; Paris, 2012). This is because BIPOC students often confront Eurocentric curricula that perpetuate bias and devalue other groups (Arday, 2018). However, when BIPOC students encounter culturally relevant texts, including familiar characters and events, their reading comprehension improves (Christ, 2018), which is true for all students as representation matters given that learning and reading are deeply personal (Menon, 2021).  

To become proficient readers, students require instruction that builds on what they know (Fleming, 2016). Starting with what students know is important because readers can transfer their knowledge to less familiar topics only when the topics are near their knowledge base (Kim, 2023). Conversely, when a text is too far from readers’ knowledge base, transfer does not occur. While good instruction builds on what students know, assessments should too. Yet, standardized reading assessments have not accounted for what students know (O’Reilly, 2019). So, reading tests become measures of content knowledge rather than reading skills or comprehension (Kim, 2023). Cognitive and learning science research has illuminated the shortcomings of traditional assessments for measuring the skills required to be literate in the 21st century, (e.g., Goldman, 2016; Partnership for 21st Century Skills, 2009). Literacy today requires engaging in multimodal content through reading, watching, and engaging with material in various formats to solve problems (Britt, 2018). To overcome the instruction-assessment chasm, reading assessments must be localized to align with students’ lived experiences, cultural and linguistic identities, and to the curriculum.

Capti SBA addresses the needs of students who face the biggest barriers to opportunity, specifically BIPOC students and those experiencing poverty by addressing students’ needs through a platform that empowers teachers to: 1) define the grade, standards and topic of the assessment, and 2) rapidly generate SBAs with sound measurement properties using AI. Capti SBA will democratize reading assessment development, resulting in thousands of culturally responsive SBAs. Capti SBA will greatly improve reading comprehension, reduce test anxiety, and make assessments more equitable for BIPOC learners and those experiencing poverty.

Our team is deeply connected to the communities with which we work and partner. In the research and design process of our solutions, we use the Lean Startup and Agile Scrum methodologies to maximize stakeholder participation, and identify issues early in projects. We also use participatory design to engage stakeholders and evaluate our solution’s usability throughout the project. During implementation, we partner with the schools and communities so they can guide us by identifying their needs and aspirations. 

Our Research and Development team includes world-class computer scientists and experts in the fields of Computer Accessibility and Machine Learning. The R&D team is led by Capti’s CEO, Dr. Yevgen Borodin, who holds a doctorate in computer science from Stony Brook University and has 10-year expertise in research on Machine Learning, Computer Accessibility, and Natural Language Processing, and a 20-year expertise in software development. The R&D team also includes Dr. Yury Puzis who is the COO and Chief of Product. Dr. Puzis received his doctorate in Computer Science from Stony Brook. He is an expert in Computer Accessibility, Machine Learning, and Automated Reasoning. For the past 5 years, he has been managing software development and UI/UX design of Capti Assess. Young Ho Seo is the team’s Software Engineer specializing in Natural Language Processing and Machine Learning. He has a BS in Computer Science from Stony Brook University, and 3 years of experience at backend programming at Capti.

Our implementation team works closely with the communities where we live and work. In Buffalo, we collaborate closely with local schools to address the pressing issue of low reading proficiency, which stands at just 27%. Partnering with researchers from the University of Buffalo like Dr. John Strong, whose expertise in reading interventions for adolescent students complements our initiatives, we are able to offer targeted support to schools with predominantly Black student populations. This collaboration ensures that our solutions are not only informed by academic research but also deeply rooted in the local context.

Dr. Margaret Opatz is a former teacher and equity coach in Salt Lake City. She works with various schools and organizations in her community to co-create solutions to improve outcomes. Her community connection runs deep as a former public school teacher. While teaching, she co-created solutions to better educational outcomes for culturally and linguistically diverse students. She co-created Cultural Conversations where community members and teachers came together to learn, Elimu a two-week program for newly-arrived refugee and immigrant K-12 students, and a reading intervention program for students with disabilities at Mana Academy, a culturally-based school. Overall, Dr. Opatz’s commitment to community-based work positions us to co-create solutions with community input and ideas.

Our belief in co-creation is a guiding principle that shapes our work. Through localized, personalized, engaging, and predictive reading assessments, we are empowering communities to chart their own path towards educational equity and social justice. As we look to the future, our vision is clear: amplify voices and drive sustainable change that transcends generations through meaningful partnerships.

We have built the Capti Assess platform for creating and delivering assessments. We have an initial working version of Capti SBA that includes human-created SBAs, as well as NLP algorithms for auto-generating  assessment items. We have designed several SBA templates that will serve as the base for AI-generated SBAs. Now, our work is focused on engineering AI prompts and developing NLP algorithms for generating the content for SBAs.

We have school customers in the following states, in no particular order: NY, UT, OH, MN, MI, WA, CA, AZ, MT, ND, IA, MO, AK, MS, WI, IL, IN, GA, KY, TN, RI, MA, VT. We have customers in more states joining next school year. Our goal is to have some presence in most states by 2025.

Capti SBA has the potential to positively transform the educational assessment field by revolutionizing the way student learning is evaluated.The ability to measure students’ higher-level reading skills is a stark contrast to today’s commercially-available assessments that employ the traditional passage-question paradigm with several drawbacks, including artificial and narrow passages, an over-reliance on basic item formats, weak links to instruction, a lack of diagnostic information, a lack of explicit connection between theoretical models of reading and assessment design, a narrow focus on the product of comprehension rather than the process of how it unfolds over time, and the failure to control for individual differences (O'Reilly, 2013) such as student motivation and background knowledge (Magliano, 2007), causing limited achievement based on decontextualized tests.  Ultimately, reading assessments currently available do not evaluate deeper reading skills, are not curriculum aligned, and, importantly, are not culturally relevant for students or localized to students’ contexts. 

Reading assessments today require innovative approaches that employ the latest digital technologies. For too long, assessments have focused on memorization and regurgitation of facts. Students deserve better, specifically, they deserve to complete assessments that reflect the technological advances of the past half century and they deserve to take part in assessments that relate to the real world. The adoption of SBAs could raise the bar for what is expected of assessments. Ultimately, Capti SBA could fundamentally change the landscape of the educational assessment field by shifting the focus away from standardized testing towards more holistic and authentic forms of evaluation. This change would encourage educators to redesign curriculum and instruction to better align with the skills and competencies valued in the 21st century. Additionally, it would foster innovation in assessment design and technology, leading to the development of more dynamic and adaptive assessment tools. Importantly, by basing assessment practices in our communities, we can create localized assessments that are culturally sustaining and contribute to positive outcomes for BIPOC students and for students experiencing poverty.

Capti SBA utilizes the transformative power of AI through a sophisticated combination of AI prompts and custom NLP algorithms. The technologies work together to generate dynamic content that is tailored to our Capti SBA templates. Capti SBA’s innovation relies on its AI prompts that leverage the use of cutting-edge AI models such as ChatGPT and other advanced Generative AI (GenAI) systems. The AI prompts are intelligently designed to generate the text and images required for Capti SBA templates. Through experimentation and testing, we have confirmed the viability of AI in generating high-quality text and images, ensuring that the content is authentic and relevant to test forms. 

When assessment items require more structure than what is capable with ChatGPT or GenAI, our custom NLP algorithms are used. The NLP algorithms are crafted in a way that provide the necessary structure, coherence, and precision needed for the assessment items, while also harnessing the flexibility and creativity of AI-generated content. Using both AI prompts and custom NLP algorithms allows Capti SBA to provide assessments that are engaging and methodologically sound. 

The integration of AI prompts and custom NLP algorithms allows for educators to create relevant and personalized SBAs for students in Grades 3-12. In doing so, educators are provided with valuable insights into students’ cognitive abilities and problem-solving skills. Students are captivated by the online learning environment that mimics real-world applications. By leveraging the latest innovative AI tools, Capti SBA sets a new standard for reading assessments. Capti SBA paves the way for more personalized, interactive, and impactful learning experiences.

Scenario-based assessments (SBAs) represent a type of performance assessment aligned with the various purposeful literacy activities needed for 21st century skills (Partnership for 21st Century Skills, 2009). SBAs are particularly effective in evaluating the intricate skills and strategies essential for success in K-12 education, including reading, writing, critical thinking, and collaboration. SBA techniques promote a set of innovative, performance-based assessment models including project-based, simulation, or portfolio assessments (Shute, 2010). SBAs are designed to afford students opportunities to apply the knowledge and skills they acquire in their content-area classes by asking students to address or solve broad, purpose-driven problems in real-world settings. A well-designed SBA walks students through a task sequence that models the strategies used by experts, while providing valid and reliable measurement of critical skills. When designed well, SBAs emulate project-based learning (Guo, 2020; Kokotsaki, 2016) of genuine professional practices that require the use of applied literacy skills and enable students to reflect on and learn from their experiences.

Since the 1970s, scenario-based designs have been employed and refined in the cognitive science fields (Carroll, 2003; Niesser, 1976). Over the past 5 decades, researchers have imagined, developed, and evaluated richer literacy-based assessments that are, as (Bennett, 2010) described it, of, for and as learning. For years, SBAs were constrained by technology, but now, technology is heavily used to create simulated performance tasks. In recent years, researchers’ approaches have been to develop SBAs that model complex, authentic applications of literacy practices (Deane, 2019). This enables teachers to use SBAs for formative purposes while maintaining fairness and validity standards (AERA, 2014) necessary to evaluating student success and instructional quality. More recently, SBAs have been widely applied to measure complex reading and writing tasks (Deane, 2021; O'Reilly, 2015; Pearson, 2020), and are appropriate for measuring higher-order literacy skills.

SBAs demonstrate value as evidenced in students’ outcomes. For instance, (Deane, 2019) found that an SBA in writing led to more fluent and efficient writing when compared to students who completed their writing task in isolation. In another study, (Bardach, 2021) discovered that SBAs with feedback and reflection increased student teachers’ cognitive classroom readiness because of more meaningful engagement. Given the positive effects of SBAs, they are currently employed in workforce and military training, e.g., (Johnston, 2022), and their use can expand to more literacy-related practices to better assess and engage readers.

Our approach for ensuring equity and combating bias in our implementation of AI stems from others’ work (e.g., Smith & Rustagi, 2020, Varsha, 2023) who have created processes for mitigating bias and promoting fairness, particularly for marginalized groups such as Black and Latino learners, as well as for students experiencing poverty. Our first step is to recognize the challenge of understanding the nuanced nature of bias and its implications in AI systems. In recognizing the challenge, we focused on partnering with educators and community members who are culturally and linguistically diverse to provide support through various feedback cycles throughout development. Second, as algorithms are created, we employ algorithmic fairness techniques that play a pivotal role in mitigating bias throughout the AI lifecycle. Fairness-aware machine learning algorithms are employed to detect and rectify biases in training data, model architectures, and decision-making processes. Regular audits and evaluations are conducted to assess the fairness and equity of AI systems, with a focus on identifying and rectifying disparities in outcomes. By sharing results on an ongoing basis with our partners, we are able to understand and challenge potential biases as we co-create AI systems. In this way, our partners’ feedback serves as a guiding force in shaping AI solutions that are responsive to the lived experiences of marginalized communities. Third, we are committed to an ongoing journey of learning and adaptation by taking part in academic conversations surrounding the topic of AI ethics, fairness, and equity. We stay informed of the latest research and emerging best practices. We also ensure we stay up-to-date on the latest policy developments as safe-use of AI in K-12 classrooms has been a focus over the past couple of years. Ultimately, our approach to ensuring equity and combating bias in our implementation of AI relies on our partners to provide feedback on the generated content through an ongoing feedback loop, and it relies on our team staying abreast of the latest research and best practices.

Full time employees: 9

Full time contractors: 6

Part time contractors: 7

Capti SBA is being developed on top of our existing Capti Assess platform that offers a unified framework and UI for rostering, assignments, reports, etc., which will greatly expedite the development and deployment of Capti SBA. Capti Assess is already utilized in 150+ schools to assess foundational reading skills with the help of ETS-developed reading diagnostic assessment. 

In our preliminary work, we have developed a Form Builder for manual creation of SBAs, which we have used to define SBA templates for ETS developed scenario-based assessment we are launching in the fall. We can demo the manually crafted SBAs; see screen shots in the video.

Capti SBA will add an AI component to enable teachers to create localized SBAs from their own curriculum and on any topic. We have already developed NLP/AI algorithms for generating simpler skills-based assessment items. We will be piloting the first version of the AI generated SBAs this fall. The 150+ schools already using Capti Assess provide a vast pool of pilot sites. The schools that have a majority of Black and Latino learners will be prioritized for pilots.

Our mission is to build solutions that improve students’ reading. While most solutions cease to exist at the prototype stage, we develop functional products, as evidenced by Capti Assess. We believe that commercialization is the only path to long-term sustainability for school-based products. We have found that schools need professional development, product support, improvements, system integrations — all requiring ongoing effort that is growing at an increasing scale; these services can only be provided for fee. All U.S. schools have a budget to purchase EdTech products.

We have successfully brought to market Capti Assess, which is now the most innovative, comprehensive, and affordable reading assessment (e.g., compared to iReady, STAR Reading, NWEA, DIBELS, etc.).  Educators have distrust for free and cheap products. We often have to fend off questions such as “Why is it so cheap? There must be something wrong with it.” 

As for accessibility, Capti SBA will incorporate screen-reader compatibility, keyboard navigation, and customizable settings for font size and color contrast. The user interface design will accommodate various preferences, allowing for multi-modal interaction (touch, speech, and gestures). Concise communication, inclusive visual design, and consideration for diverse cognitive abilities will be prioritized. Usability testing with users of different abilities, including those with disabilities, will help identify and address potential barriers. Capti SBA will be compatible with various devices, and regularly updated based on user feedback to improve accessibility and inclusivity. We will address all potential user needs, especially BIPOC learners.

We do not have enough resources to accelerate the development of Capti SBA. However, while we were initially interested in the financial assistance, we would appreciate support in accelerating our ability to market, as well as navigate apparent legal challenges (e.g., IP and Privacy) and those that are yet unknown. With support from the Bill & Melinda Gates Foundation, we will be able to accelerate the development of Capti SBA and bring it to market faster, and strengthen out existing Capti Assess product.