Leveraging both LLMs and ML/NLP to Support Deeper Student Learning: Writing & Reading Comprehension

English Proficiency continues to be the lingua franca of global commerce, technology, research, and foundational to economic mobility.  However, scaling English proficiency in America and across the world is challenging especially for the deeper learning expectations of our modern world.  See EF English Proficiency Index – Comparing English skills between countries – EF EPI Archived 18 June 2015 at the Wayback Machine. After great teachers, research confirms that unbiased curriculum-embedded formative assessment drives more student improvement than any other resource. (Hattie 2009; Wiliams 2011)  However, no organization has created a scalable, accurate system providing feedback on deeper learning. Standardized selected response item banks designed by large-scale testing companies are too distant from, and lack contextualized connections to curriculum and instruction, nor to deeper student learning aligned to rigorous national standards. (Shepherd; Atkin) Multiple choice banks and low quality tasks do not respond to deeper learning needs. (Elmore 2010, Santelises 2015) Even putatively IRT-“validated” multiple choice and short constructed responses have little to do with the deep thinking skills contemplated by CCRS, NGSS, and C3 rigorous standards necessary to remain competitive in a global intellectual and economic market. The recent clamor around Science of Reading strategies is only the beginning of a student’s reading comprehension journey. Once students master phonics, phonemic awareness, etc. rigorous standards require developing student‘s extended writing in response to complex text – a deeper learning journey that requires connecting student data to detailed, scaffolded instructional strategies at the core of LDC’s statistically significant-confirmed framework. Even before Common Core, NGSS, and C3 were adopted, teachers working with high-needs students struggled to keep up with the demands to score, grade, and provide effective feedback on extended writing. National standards’ rigorous expectations and COVID have exacerbated this challenge, highlighting inequities in communities with high student-to-teacher ratios. An urban ELA teacher with 130 high needs ninth-grade students asking students to read informational text and write every month faces a student essay volume often too high for teachers to provide students with adequate feedback on even one round of their writing. Too many papers frustrate teacher efforts to collect and compile data to enable real-time grouping and differentiation. By the time students finish argumentative essays, the teacher has missed multiple opportunities to address skill deficits in the absence of usable formative data revealing student learning difficulties in their reading and writing. Likewise, students have also lost multiple opportunities to take ownership of their learning, build on teacher formative feedback, and practice the reading/writing skills essential for 21st-century learning. (Moreover scores alone are not as useful as inline discursive feedback on extended writing in response to complex text which this project would provide). More than ever, rigorous CCRS expectations for student achievement require teachers to provide frequent, direct, substantive feedback on students’ deep thinking work as manifested by frequent rigorous student writing. As Wiliam asserts, formative assessment is the “bridge between teaching and learning – only through some kind of assessment process can we decide whether instruction has had its intended effect.”

Since 2017, LDC has conducted research with SRI International’s Artificial Intelligence Center to prototype a scalable generative AI mechanism to measure and provide real time discursive feedback accurately and validly of deeper student learning.  While SRI historically has almost entirely provided support to Department of Defense and Homeland Security, for this passion project SRI would design an AI technology solution that will “Ensure that all children are learning in good educational environments, particularly those affected by poverty....  Use inclusive design to ensure engagement and better outcomes for learners with disabilities and neurodivergent learners, while benefiting all learners, [and]. Provide the skills that people need to thrive in both their community and a complex world, including social-emotional competencies, problem-solving, and literacy….”  SRI’s current Generative-AI work now combines their proprietary ML/NLP algorithms with ChatGPT. In addition, partner Stanford Center for Assessment Learning and Equity (SCALE) – which created California’s teacher certification test used in 9 additional states as well as SBAC advisors and item writers – provide the analytic student rubric and curriculum rigor analysis rubric designs and validation. LDC’s collaboration with SRI’s AI neuroscientists and SCALE measurement experts would produce a teacher and student facing mechanism that can assess whether assigned performance writing tasks incorporate the standards cognitive demand rigor and then measure the student’s mastery of those standards (CCRB/NGSS/C3) in students’ extended writing in response to these validated writing performance tasks. SCALE already has validated performance task rubrics by grade (K-12) and discipline (Sci/SS/ELA) but the current process requires extended manual calibration of humans to gain inter-rater reliability. This solution would in an instant, create an expertly calibrated, interrater reliable mechanism to provide feedback to individual students, teachers and administrators on extended writing performances – not short open ended answers, not multiple choice, not even five paragraph formulaic essays. The data would then be used to direct teachers to research-backed instructional strategies. LDC recently was awarded two highly competitive USDOE AHC grants to create culturally-responsive K12 Civics writing prompts to add to LDC’s already robust OER K-12 prompt bank. LDC will transform SRI’s backend AI in 3 ways: (1) develop APIs and a teacher interface to enable SRI’s AI to integrate with ChatGPT; (2) use manual student writing LDC currently collects to improve SRI’s AI engine in combination with ChatGPT to 95%+ scoring accuracy (from current 85%) but more importantly use the learning data to integrate discursive “in-line” writing feedback at score deprecations for far greater student learning impact, and (3) make the ongoing, expanding learning dataset available to researchers to define the most efficacious ways student writing develops – i.e. data will demonstrate the measurable patterns of subskill development most effective to students mastering writing on summative state tests. No existing commercial solution scores student work to untrained prompts, much less provide inline discursive feedback at AI-determined score deprecations.

This project targets high need urban and rural districts, schools, and student populations – all the places LDC currently operates in its competitively-awarded federal grants and district contracts.  For example, NYC, LAUSD, rural Kentucky, North Carolina, and other rural districts have all provided letters of support in connection with parallel federal funding requests. Since 2016 in LDC’s USDOE 100+ NYC/LAUSD Investing in Innovation (i3) grant schools, teachers have clamored for an accurate mechanism to support their efforts to instruct rigorous deeper learning performance writing tasks without crushing them with scoring and student feedback burdens.  Schools with the highest need students also often have teachers with the thinnest support or experience and the least exposure to high-quality instructional materials.  In both LDC’s 2022 USDOE Education Innovation and Research grant in 140 urban (NYC/LAUSD) and rural (KY) schools and LDC’s USDOE 150 school culturally responsive curricula and PD grant, school populations exceed 82% free lunch, 80’s BIPOC students, and 20% Special Education.  This solution would address three targeted student and teacher needs: (1) provide instantaneous analytic rubric scoring and discursive coaching student feedback; (2) instantaneous data that assesses the efficacy of teacher-chosen instructional strategies and practices, and (3) assess the quality of curricular lesson plans.  Having an expertly-calibrated, tech-enabled instant feedback mechanism would enable teachers working with high-need students to do what they do best: support their deeper learning based on standards-aligned materials and high quality feedback and guidance.  CCRB standards’ more rigorous expectations and COVID have only exacerbated school challenges and highlighted inequities in communities with high teacher-to-student ratios.  For the past year, LDC has collaborated with National Writing Project sites in ad hoc exploration of ChatGPT’s effectiveness in student feedback. And more recently (Oct 2023), LDC was awarded a small Cambiar Foundation grant to assess the efficacy of using ChatGPT with low income, multilingual, and immigrant parents to support their students’ learning, particularly extended writing in response to complex text.  Over 10 NWP Writing Project sites oversubscribed this year’s Cambiar-funded ChatGPT research.  LDC already always engages teachers, students, and parents from high need, underserved populations in user centered design and prototyping of LDC instructional product development.

LDC’s team and collaborating organizations have intensively designed instructional and technological solutions for high needs students, teachers, and schools for several decades first in urban low-income settings and more recently also low-SES rural American schools.  LDC’s team lead (Chad Vignola) began as a NYCDOE senior central office leader in 1999 charged with transforming several offices to more effectively support high need students including special education and English language learners.  Later Chad was trained at IHI in Cambridge on continuous improvement as well as IDEO user-centered design and rapid prototyping of instructional resources, leveraging technology in New York City and Los Angeles schools among others.  Our instructional design team is highly diverse with several only recently having left urban classrooms to join LDC at creating scalable teacher-friendly tools with specific focus on multilingual and special education students.

Sabrina Alicea (Latina) LDC’s Director of Curriculum Development is a former Chicago public school teacher, curriculum designer, and Harvard graduate who currently drives two USDOE Civic’s design grants awarded to LDC to create high-quality, culturally-relevant lessons, tasks, and resources for students and teachers as well as culturally sustaining and relevant professional development and has always sought to elevate access to educational and creative opportunities for marginalized and underrepresented communities. 

Jabari Sellars, (Black) Curriculum Associate Director, joined LDC after over a decade of middle school and high school classroom teaching experience, most recently in Oakland. Jabari was awarded Harvard’s Intellectual Contribution Award for his work with comics and graphic novels in literacy education and recently was asked to judge the Walter Dean Myers award honoring diverse books by diverse creators.  Additional diverse instructional staff include Andrea Fullington, a former New Orleans teacher and “Black is Brilliant Fellow, Chadae MacAnuff, former NYC teacher (Black) and Lydia Mantis (Egyptian).

Rob Kasher, Chief Program Officer, a former Brooklyn TFA social studies teacher in Brooklyn has served in roles at The Broad Center (TBC) and at Leadership for EducationalEquity (LEE) providing leadership development to equity-minded organizations before returning to LDC to oversee the design, delivery, and innovations.

In addition to internal staff, collaborating organization staff include Dr. John Niekrasz, a computer scientist in the Advanced Analytics group of SRI's Artificial Intelligence Center focusing on the use of automated discourse analysis technologies. Niekrasz holds a PhD. in linguistics from Edinburgh and a Stanford BS in symbolic systems.  SCALE created the California Teacher Certification test (TPAC) used in 9 other states and were SBAC advisors and item writers.  Dr. Ruth Wei SCALE’s VP has led research studies on the efficacy of project-based learning curricula, the quality of large-scale assessment items used in state tests across the United States, and the National Board for Professional Teaching Standards portfolio assessment. Dr. Wei has degrees from Harvard and Teachers College (Columbia), spent six years teaching in New York City prior to obtaining her PhD from Stanford.

We trained and tested the viability of SRI's proprietary ML/NLP over two years recently to accurately assess students deeper learning by measuring and providing feedback to student writing in response to complex text.  However, this was SRI's backend research engine.  Pending external funding, LDC has not designed the user interface for teacher (and later student) access.  Thus, arguably the user interface is only conceptual though LDC has previously built teacher user interfaces through user centered design and rapid prototyping including 140 sprint cycles from 2013 to the present to the build LDC's teacher-facing platform https://CoreTools.LDC.Org/ accessed by over 105,000 unique educator users.  Awards for the CoreTools platform include the Carnegie Foundation for the Advancement of Teaching Spotlight Honor in 2019 as one of the three best organizations in the world for LDC's online user centered design, product prototyping, and analytic infrastructure leading to proven student achievement improvements.  In addition, the USDOE asked LDC to present to all 10+ years of USDOE Innovation grantees followed up by a full day workshop to 100 attendees on user centered design and rapid prototyping.  

Three primary challenges we hope this can 'Solve': (1) technical, (2) financial, and (3) legal.

The technical challenge LDC seeks to solve involves three subproblems (a) ensuring diversely generated data is used to both assess, validate, and improve the performance of ML/NLP/LLM interactions.  If the data to inform and assess the performance of AI reflects existing social, racial, and especialy socioeconomic societal biases, the solution will reinforce if not reify those existing biases further increasing inequity in American or ay other country's use of a solution.  (b) the most effective ways to use LLMs in tandem with SRI's proprietary ML/NLP (e.g. "walled gardens", alternative LLMs to ChatGPT, etc.), and (c) user interface design vs. rapid prototyping on existing reporting platforms (e.g. Learnosity, etc.). (2) Financial:  Connection to funding opportunities outside of Solve of course but also fundraising strategy planning/business model/licensing with for profits, social venture capital investing options as well as consideration of launching a separate for profit entity to LDC, a non profit.  Proof of concept funding vs. beta prototype investment.  And (3) Legal: multiple IP issues including. (a) How to license and use SRI's proprietary ML/NLP (maybe niche education exclusive license for X years), and (b) student data privacy as well as LLM purported "ownership" of all generated data.

Prior to an LLM like ChatGPT, existing commercial scoring of student writing had multiple severe limitations.  Big publisher and researcher auto essay scoring engines (e.g. Pearson, McGraw Hill, ETS) only worked with a defined publisher’s prompt and text, manually calibrated human scorers of ~200 student papers that were then used to calibrate the AI engine.  Change a word in the prompt or the text or have poor or limited teacher scorer accuracy and the engine did not work well.  Worse, the writing assessed typically was limited to 800 words or less and of low Depth of Knowledge: – e.g low cognitive level student writing or formulaic 5 paragraph essays.  They were not usable with “blind prompts” – prompts the engine hadn’t seen before – nor able to assess the deeper student learning expectations of national (or international) standards of learning (e.g. Common Core, Next Generation Science Standards, College, Career, and Civic Life) necessary for participating effectively in the non-routine work of the global future.  Scoring and feedback on extended student writing in response to authentic and complex disciplinary text also has not been available in the marketplace.  Although ChatGPT has created new opportunities, the shortcomings are (1) most emerging solutions driven by for profit companies driven are low quality bot responses to low cognitive level questions of highly uncertain response quality or even inaccuracy and bias.  (2) ChatGPT or other LLMs in and of themselves are not of sufficient quality or accuracy.  Using an LLM as a component rather than standalone, with SRI’s ML/NLP created and tested over decades in high profile contexts (e.g. national research laboratories, Dept of Defense and Homeland Security contracts) to date has proven far more effective.  Creating a teacher facing solution more user friendly than current ChatGPT interfaces.  Finally, over time the user learning engineering data generated will both create self reinforcing improvements in tool performance but equally importantly provide feedback on which instructional pedagogical practices and instructional materials most impactfully improve student deeper writing development and reading comprehension and – potentially – the most effective sequencing of those strategies and content in such student learning development.

A visual of LDC’s Logic Model used across the organization and in all our federal grants targeting high need students and schools is available upon request or link.  Short term immediate impact to building the LDC-SRI-SCALE solution is the generation of both scoring and discursive coaching formative feedback to guide and inform student thinking and learning as manifested in extended writing in response to complex disciplinary text.  Longer term outcomes are improved performance on sophisticated international assessments or less sophisticated U.S. state assessments.  Inputs in LDC’s Logic Model include (1) Rigorous CCRS-Aligned, SCALE-validated Performance Writing Tasks; (2) Bi-monthly ILT virtual professional learning supports; (3-4) Support for teacher weekly PLCs with both synchronous and asynchronous professional learning, and (5) Manual collection and use of SCALE analytic rubric data.  Independent research cited above (Elmore’s Instructional Core, Santelises “Task Predicts Performance”) confirms that student learning can only increase if students are exposed to engaging, high quality instructional materials, supported by effective, skilled teachers.  LDC’s statistically significant (0.37) doubling of student learning in our USDOE i3 Validation grant in 100+ NYC and LAUSD schools (2015-2020) controlled for prior student, teacher, school state test performance, socioeconomic level, ethnicity, English language proficiency, and special education status enacted all three legs of Elmore’s Instructional Core.   (UCLA-CRESST, Wang J.)  But for LDC and SCALE to validate the rigor level and engagement of instructional materials as well as teacher skill development (microcredentials) has required significant time and costly manual calibration.  In addition, also as noted above, research and meta studies confirm that formative assessment feedback is the second most efficient mechanism to improve and accelerate student learning.  However, manual calibration and data collection and display hampers the scalability and, more, the accuracy of the data collected.  A lightning fast and nationally calibrated accurate mechanism to collect and provision this data to teachers and students is especially important for high need schools who continue to have the least resourced, most skill-challenged teachers.

LDC’s collaboration with SRI and SCALE will directly address several UN Sustainable Development Goals.  According to UNESCO, if all students in low-income countries had just basic reading skills (nothing else), an estimated 171 million people could escape extreme poverty. If all adults completed secondary education, we could cut the global poverty rate by more than half. Education has the potential to “End poverty in all its forms everywhere”.  See Education and Economic Growth Eric A. Hanushek, Hoover Institution, Stanford University and Ludger Woessmann, ifo Institute, Ludwig Maximilian University of Munich Eric A. Hanushek, Hoover Institution, Stanford University and Ludger Woessmann, ifo Institute, Ludwig Maximilian University of Munich https://doi-org.ezproxyberklee.flo.org/10.1093/acrefo... Published online: 31 August 2021.  Historically, as new instructional innovations, particularly technological innovations are rolled out in education, they typically increase inequities, not reduce them.  Thus, it is more than likely that as for profits drive the introduction of Generative-AI this pattern will continue.  LDC, on the other hand, as a mission driven nonprofit designing instructional technology and operating in the highest need school districts in the country, will ensure the project will promote “inclusive and equitable quality education and promote lifelong learning opportunities for all, [and] sustained, inclusive and sustainable economic growth, full and productive employment and decent work for all.”  Again, education has long been recognized as the primary engine behind economic mobility.  R. Haskins, Education and Economic Mobility, (Brookings Institution 2016).  LDC’s creation of a tech-enabled and nationally-calibrated formative assessment regime will as noted above (Hattie, Wiliam) be the most successful driver of scalable improvements in student achievement.  And as the UNESCO note above confirms, promoting basic and secondary literacy proficiency has the highest potential to “[r]educe inequality within and among countries.”  LDC will measure its progress and impact on these global goals through three primary data collection strands: (1) increases in student reading and writing comprehension as measured by the AI generated scoring data utilizing the SCALE grade level analytic rubrics (leading indicator); (2) International and/or U.S. state assessment scores (lagging indicator), and (3) increases in the number of educator microcredentials captured in LDC’s CoreTools.LDC.Org platform (leading indicator). In short, LDC will capture improvement in student literacy and achievement as the indirect but causal mechanism leading to progress toward the above-noted UN Sustainable Goals.  Finally, simple usage KPIs, session times, unique user logins, etc. would regularly be reviewed once the prototype is in market tested teacher user form.

LDC will use both traditional innovations (analytic rubrics) and new technologies (Generative AI and SAS software development).  The SCALE analytic student rubric measures the student’s learning progress, signaling to both student and teacher the next step toward proficiency. For example, both the teacher and the student can easily understand the R2 learning progression and next proximal learning steps for the 3rd grade student scoring at 2nd grade level 2 with additional complex text based on the explicit SCALE analytic student rubric.  This will ensure the project builds upon the already established benefit of rubrics in that they are statistically more likely to mitigate bias in grading practices by standardizing scoring criteria, reducing the space for subjective assessment grading. This ultimately makes educators more effective in assessments and feedback built on years of research showing students learn best when they are exposed to relevant content and provided with opportunities to expand their critical literacy skills.  Even before Large Language Models, Machine Learning and Natural Language Processing demonstrated efficiency and accuracy to provide formative data and insights into student learning including reading comprehension and extended writing in response to complex text.  LDC began exploring ML/NLP solutions in 2013, ultimately beginning a research partnership with SRI International’s Artificial Intelligence Center.  From late 2016 to the end of 2019, LDC invested a half million in internal dollars to confirm the viability of using SRI’s proprietary AI to generate accurate and valid data to support student deeper learning goals.  Notwithstanding demonstrating the practical and conceptual benefit to the work, the absence of external funding stalled ongoing efforts.  However, with the release of ChatGPT in 2022, SRI reached out to suggest restarting our work as LLMs had significantly improved the potential for our collaborative work.  But while LLMs like ChatGPT are “out-of-the-box” remarkably capable of providing feedback to students, adapting them to the unique characteristics of student writing, and avoiding biases and errors that come with default models, requires tuning the existing model weights and the implementation of additional model components (e.g., model layers). By supplementing LLMs with additional model elements and training them on student data and validated scores, SRI will significantly improve upon foundation model accuracy, mitigate bias, and in tandem with LDC continuously improve its efficacy and impact on student writing over time.  The last innovation is standard software engineering sprint cycles coupled with parallel instructional resource sprint design cycles.  Throughout our SRI collaboration, LDC has continued to build CoreTools.LDC.Org to provide teacher-friendly online Open Educational Resources including online asynchronous learning microcredentials.  The asynchronous online learning includes a course that enables educators to calibrate their scoring of the SCALE analytic rubric.  However, the quantity of student work to calibrate against is small and does not include the more powerful inline discursive feedback.  CoreTools itself – which we would build upon – is a Ruby on Rails server, and Angular client running on Heroku hardware. We are currently in the process of converting from angular.js to the more modern Angular framework.