LiteraSee by CENTURY Tech

LiteraSee by CENTURY Tech is an automated writing assessment and improvement tool that analyzes learner writing samples using language models, provides detailed diagnostic feedback, and recommends personalized learning to address gaps.

LiteraSee is designed to assess the complex cognitive processes of writing such as creativity, argumentation and style and provide meaningful and personalized feedback. LiteraSee will also provide the exact micro-lesson - or series of micro-lessons - to ensure that the student knows how to improve their writing.

How does LiteraSee work for a student?

1. Student answers a short series of written questions - typed text, transcripts or even scanned handwriting will be acceptable.

2. LiteraSee analyzes responses and provides a level for each writing attribute using a proprietary LLM-based classification model.

3. LiteraSee, using Generative AI, immediately provides written feedback to the student on their writing skills, with a comprehensive breakdown of performance on each of the component skills.

4. LiteraSee, using CENTURY’s existing knowledge tracing recommendation system, provides the student with a list of recommended nuggets to complete to ensure development in areas of weakness and to support progress in areas of strengths

5. When ready, students can complete another writing assessment to ensure improvement.

The assessment and analysis of writing are the foundational steps of LiteraSee - required to understand and diagnose a student’s current ability level. But understanding a student's current ability is just a small part of the puzzle - what comes next is much more important for making progress. LiteraSee provides a seamless link between assessment outcomes and personalized learning content recommendations designed for maximum impact.

Firstly, students will receive detailed feedback to highlight their strengths and areas for growth. Students will know exactly how they are performing with respect to each of the different writing ability attributes, enabling them to learn from the assessment process. And this feedback will be presented to them in an easily digestible format, so they’re not having to guess what is meant.

Then - and this is what truly distinguishes this tool - LiteraSee turns the feedback into a concrete learning recommendation so that students know exactly what they need to do. Building on CENTURY’s existing ML-based recommendation system, LiteraSee will list the next micro-lesson(s) that a student needs to complete in order to improve. These micro-lessons already exist as part of the CENTURY platform used by organizations around the world. We have shown in multiple contexts that CENTURY learning programmes improve outcomes by a whole grade, on average.

For speed, we will pilot the assessment and feedback component as a standalone solution, allowing us to pilot and iterate the analysis and feedback models in the most agile way. Once we reach our required level of confidence in these components of LiteraSee, we will incorporate them into the CENTURY platform and link the assessment output to our highly personalized learning system. We will then host the assessments in our robust assessment player (which is currently used in a variety of contexts, including for high-stakes assessment) to provide all expected assessment functionality out-of-the-box.

Writing proficiency is a vital skill for learning. It is important for critical thinking, such as analysis and inference. Writing and reading comprehension are deeply entwined. Transforming ideas into words also enhances encoding and memorisation more deeply than passive reading per findings from MRI scans. Writing is a valuable tool for all learners and one that should not be discounted.

However, despite clear evidence for its importance, a significant proportion of learners cannot write well. In the US, most 8th graders lack proficiency per NAEP data. Many other countries face similar problems with low achievement rates and declining ability. Early intervention is necessary, with research showing that children who struggle early on “encounter considerable difficulties in following education later on”.

Unfortunately these challenges with writing are likely to disproportionately impact priority learners due to structural disparities within the education system. 

1. Schools in low-income communities typically receive less funding and are lower performing. Additionally, teachers in these areas report being less well equipped to teach writing, compared to higher-income areas.

2. Wealthier families can supplement inadequate school-based education with tutors and extracurricular educational programs. These resources are out of reach for low-income families, who are disproportionately Black and Latino.

3. Priority learners often encounter curricula that do not resonate with their cultural backgrounds. This is particularly pernicious in written assessments, where prompts often presume certain cultural knowledge, further disadvantaging priority learners.

4. Research indicates that implicit racial biases can result in educators having lower expectations for priority learners, which often translates into less constructive feedback.

LiteraSee addresses these challenges in a multi-faceted way.

For educators:

• LiteraSee’s robust writing rubric and CENTURY’s learning content offers a complete curriculum that supports educators' ability to teach writing effectively, regardless of their previous experience or resource availability.

• The tool also equips educators with comprehensive data and feedback about student writing, which can be used to monitor individual and class progress and effectively intervene at the point of need. This data is also accessible at the school, district, and national levels, allowing for real-time monitoring of writing achievement across larger populations and - crucially - for data-informed intervention.

For students:

• LiteraSee is available 24/7 - it brings the teacher into the home at a fraction of the cost of a tutor, serving as an on-demand educational resource that fits the complex schedules of low-income families,

• LiteraSee’s writing rubric will adhere to Culturally Responsive Teaching methods. Writing prompts will be diverse and encourage students to incorporate their cultural heritage into their writing. 

• To mitigate bias we will ensure training data includes many examples of genuinely culturally diverse writing, to ensure that the analysis and feedback can support priority learners, rather than penalize them. Further, during assessments, the model will not know the racial identity of the students who are submitting writing responses, defending against the implicit racism of low expectations or qualitatively different feedback.

By integrating these features, LiteraSee aims to bridge the achievement gap for priority learners, providing them with the tools they need to succeed academically and beyond.

• Have significantly engaged with priority communities

• Have demonstrable success in building solutions for similar demographics

• Are diverse and experienced ourselves

• Incorporate Inclusive design as standard

Engagement with Priority Communities: The design and development of LiteraSee are deeply influenced by the needs and inputs of the communities we serve. We have already extensively consulted with schools with high proportions of priority learners to develop our initial plans. We are committed to a participatory development process and have plans to pilot LiteraSee in priority communities to ensure that we meet the specific needs and aspirations of these communities - we already work with these schools and know that they will hold us to the highest standards here!

Experience with Similar Demographics: In 2022, our team developed Smart-IA for students in Further Education colleges in the UK (read more here). A component of Smart-IA is an automatically scored written assessment which was used by over 80,000 students in its first year. Although Smart-IA's output differs from that of LiteraSee, the target demographics overlap significantly - these students often come from lower-income backgrounds and minority groups. Smart-IA garnered highly positive feedback, particularly due to the way the assessments sits alongside the learning materials and improves motivation and confidence of students. CENTURY is leading the market in the UK. We will bring our experience from this highly successful written assessment to the development of LiteraSee for similar learners in the US.

Diverse and Experienced Internal Team: Our founder, of East African Indian descent, has personal experiences with the impacts of implicit bias in education. This insight is critical in guiding our mission to uplift all students and one that she imbues in all of us. Our wider team includes teachers who have taught in the UK's most deprived schools, such as Michaela school, known for its high achievement rates among priority learners. These individuals will be responsible for defining our assessment rubric, but they also bring a very real and lived understanding of the challenges faced by these communities and their expertise will be imbued throughout the project to ensure an inclusive and effective tool.

Inclusive Design: LiteraSee will be integrated within the CENTURY platform, which has been designed from the outset to be culturally inclusive. This includes small but significant features like offering a range of representative avatars - and we don’t stop with racial diversity: we recently added a set of bald avatars for children undergoing chemotherapy to ensure every student feels represented. We incorporate accessibility throughout the platform, providing readability features for visual impairments, audio transcripts, multiple input methods, screen-reader compatibility, etc. so that CENTURY can be for everyone. Our plan is for LiteraSee to accept submissions of typed answers, transcripts and even scanned handwritten responses.

We have technically planned LiteraSee and conducted initial market and user research. We have yet to start technical development.

LiteraSee will partially build upon three separate tools developed by our team in the past:

• Smart-IA: a NLP classifier providing a writing ability score level across different attributes to essays produced by FE students. 

• Knowledge Tracing: a DL Transformer architecture providing recommendations to students on what content to learn next based on previous learning.

• Teachers Tools: a LLM-based system to generate new alternative questions on a topic to streamline Teachers’ workflow in creating assessments for their classes.

Firstly, on the technical front, LiteraSee will incorporate a cutting-edge system that combines generative AI with established state-of-the-art supervised learning models. The result will be a so-called  "compound AI system", namely an agent designed for highly sophisticated decision-making that surpasses the capabilities of its individual AI components (see e.g. https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/ ). Within the LiteraSee system, supervised learning will establish a robust foundation for this decision-making process, quantifying students' writing skills and identifying their weaknesses and potential areas for enhancement. Building on this foundation, the generative AI will then generate targeted feedback and actionable recommendations.

Secondly, we are being highly ambitious in the scope of our rubric, significantly improving on the currently available tools for assessing writing. There are any number of tools that will analyze aspects of writing such as grammar and structure. There are even tools now that will analyze coherence and feedback on the content. However, these still do not comprehensively assess all the skills of writing. LiteraSee’s writing rubric will encompass the full range of skills required for effective and compelling writing. We are confident that we will be able to meaningfully assess key skills in writing such as style, argumentation, and even creativity. 

Thirdly, LiteraSee builds an immediate bridge between assessment results and learning which is so often lacking in schools. Typically assessments are infrequently administered as they require a significant amount of educators time to mark - especially if the educator also needs to give specific feedback and recommendations. This time lag in results and feedback also lessens the effectiveness of assessments as part of learning. Because feedback on LiteraSee will be immediate and highly personalized - more detailed than most educators have the time to provide when marking - LiteraSee will allow students to transition seamlessly from assessment to learning back to assessment and see progress in real time. Our hypothesis is that this will have an impact on performance and also - excitingly! - on student motivation and engagement. Students will be able to see how progress in the recommended learning material has a direct impact on their assessment outcomes, creating a virtuous loop. These features mean that writing assessment can be a meaningful part of learning to write effectively in a way that hasn’t previously been possible.

LiteraSee will leverage advances on several AI domains, including:

• DL: in particular Transformer architectures (both third-party pre-trained models as well as models trained in-house).

• NLP: using Large Language Models (LLM) at the heart of our generative AI modeling of feedback. 

• Knowledge Representation: creating automatically annotated knowledge graphs to map connections within CENTURY’s knowledge base to offer tailored recommendations.

Below we provide an in-depth overiew of how these components come into play and combine together into LiteraSee. LiteraSee will be structured across three main axes:

• Analysis & Synthesis, 

• Feedback, and 

• Guidance. 

Analysis & Synthesis (A&S) assigns written ability scores to text via a natural language processing (NLP) classifier. Following cutting-edge strategies in deep learning, our training process for this model will be to “fine-tune” a pre-trained large language model (LLM) - such as the RoBERTa architecture used in our Smart-IA project - to tackle writing attribute classification. The fine-tuning dataset will initially consist of 2000+ samples of text written by students and marked by expert educators according to CENTURY’s rubric. This number will grow as CENTURY will collect more samples through LiteraSee usage, yielding improved versions of the A&S model. 

Feedback combines OpenAI's GPT-4 and our A&S model’s output to generate actionable feedback for learners and educators. To achieve this we will:

1. leverage the A&S’ attention mechanism to identify text elements highly correlated with individual attribute marks

2. use GPT-4 to verbalise this correlation into directly interpretable feedback

3. visualize this feedback by highlighting the text element and displaying the feedback to the student.

For step 2 we will design a prompt combining the A&S analysis, CENTURY’s rubric, and general instructions we will develop with our research collaborators. We will further contextualise this prompt with examples from expert educators to tailor the generated feedback to LiteraSee’s purpose.

Guidance combines A&S with CENTURY’s Knowledge Tracing (KT) model. The KT is CENTURY’s core recommendation engine tailoring bespoke learning journeys for each learner on the platform, based on their individual learning needs. The KT processes all the information available about a user (past nuggets attempted, their performance, etc.) to recommend the most appropriate nuggets to attempt next. We will design and train a variant of CENTURY’s KT able to ingest the A&S’ output as further information to recommend nuggets that will boost the student’s writing ability and push them to the next level. KTs can also provide insights into the reasons behind a specific recommendation. We will surface this information to the student via GPT-based feedback to maximise the effectiveness of our recommendations.

After initial piloting of the writing analysis model, we will integrate LiteraSee into the existing CENTURY platform. This will give us the frontend interface out-of-the-box, including functionality to allow students to write their written responses, students and teachers to see the results and an interface for viewing feedback.

As LiteraSee is still at concept-stage, we aren’t yet to be able to prove that this tool works. However, we have multiple reasons for having confidence that it will.

During the initial discovery phase we spoke with customers, analysed customer feedback on our content and looked at usage data. We also scanned the market to ensure that we weren’t reinventing the wheel. This discovery phase validated our hypotheses that

1. Teaching writing is a “black box” to many teachers and they are desperate for a solution which supports them.

2.  There is not an existing solution available to schools in our target markets which is able to analyse the full range of writing skills as well as connect this with learning material.

3. The link between the assessment outcome and the learning recommendation(s) is really exciting for potential users.

As we move into the prototyping phase, users and stakeholders will continue to play an important role.  We conduct usability testing with designs at various stages both with internal teams and target users, and incorporate feedback into further iterations.

We have evidence that the our technical implementation will work from the success of tools that our team developed in the past: 

• SmartIA (described in answer to ‘How are you and your team (if you have one) well-positioned to deliver this solution?’): our team developed a model capable of estimating writing ability scores (reported as a score of 1 to 5) on six separate writing attributes for students in Further Education colleges, aged 16+. The model has high accuracy, differing from the ground truth human’s assessment by less than 0.5 on average. These results give us a high level of confidence that we can build a similarly performant model for grades 3-8 with a new rubric.

• Knowledge Tracing: our KT model is already providing empowering recommendations to students based on their journey on CENTURY’s platform. This system is well-tested throughout the different phases of education, with many examples of CENTURY contributing to improvements in outcomes. Linked here are just a few of the analyses that which show how using CENTURY supports improvements of an entire grade on average:

  • https://www.inspirededu.com/news/inspired-ai-technology-proven-improve-student-performance-entire-grade 

  • https://www.century.tech/news/ufi-voctech-project-using-artificial-intelligence-to-improve-further-education-resits-results/ 

  • https://www.century.tech/our-impact/ 

• Integrating the rich information gathered from a written essay into this KT model will naturally improve the system ability to recommend material to revise or study anew.

Finally, we are confident in the usability of LiteraSee as, eventually, this assessment will be hosted by CENTURY’s existing assessment player which we know is appropriate and effective for our target audience. It is currently used by:

• Students aged 10 and 11 for high-stakes entrance examinations

• Students as young as 8 for exam preparation

• Adolescent and adult learners as part of our Smart-IA product.

LiteraSee is designed to elevate all students to a high level of writing proficiency, irrespective of their background. To achieve this goal, we have implemented several measures to ensure fairness, combat algorithmic bias, and respond to the diverse needs of learners, particularly those from Black, Latino, and low-income backgrounds.

Our approach to mitigating bias begins with the selection of training data. We use authentic writing samples from a culturally diverse student body, avoiding synthetic or unrepresentative data. Each sample will be carefully reviewed to eliminate content that might reinforce stereotypes or discrimination. Additionally, the outputs of LiteraSee will be manually reviewed to identify and correct any discriminatory patterns.

The model itself will not have access to any defining information about a student, including sex, race, nationality, etc., in order to ensure that the model does not inadvertently use this information to inform any outcome or recommendation. The system will likely need to standardize results by age, so this attribute will be available.

In order to ensure equity within the assessment process we will follow Culturally Responsive Teaching practices, which incorporate diverse and inclusive writing prompts that are more likely to resonate with students from all cultural backgrounds. For example, prompts like ‘In your community, what cultural tradition is most at risk of being lost? Explain why it's important to preserve it,’ encourage students to draw on their personal and cultural experiences. Our rubric values a wide range of writing styles and cultural expressions, ensuring that assessments are fair and inclusive.

In order to ensure that we don’t unintentionally cause harm to marginalized communities, LiteraSee's development will be continuously informed by feedback from our target users - priority learners and their educators. We maintain a deep commitment to understanding the educational challenges and systemic barriers these communities face and will continue to engage in direct consultation with priority students, their teachers, and academic experts in this area. This engagement will help to ensure LiteraSee is a tool that meets the real-world needs of these learners.

Further to this, we will regularly review and re-validate the LiteraSee models against human markers to ensure consistency and fairness. These evaluations are crucial for maintaining the integrity of the assessment and we will conduct this using diverse writing samples to ensure broad representation. The evaluations will also always include a review for any language that reinforces stereotypes or discriminatory language and ensure the removal of this language from the training data.

Before full deployment, LiteraSee will be piloted in schools that serve a high proportion of priority learners. This step is vital to ensure that the tool operates as intended without unintended consequences. Our close relationships with these schools enable us to receive honest feedback and make necessary adjustments, ensuring that LiteraSee serves as a beneficial educational tool.

Our idea with LiteraSee is that it will eventually live within the existing CENTURY platform. As such, members of the wider CENTURY team will work on this project as required. Note, that these staff members will not be working on the LiteraSee project full time.

As implementation work has not yet started, we do not have a confirmed project staffing list. However, we expect around 3 parttime and 10 fulltime members of CENTURY staff, plus 2 contractors would be involved in the project. 

We have a comprehensive plan to get LiteraSee to pilot stage and beyond - image attached with indicative workstreams and timelines.

Initially we will pilot the assessment scoring and feedback components only. This will allow us to avoid the lengthy process of integrating the assessment into the CENTURY platform for pilot, allowing us to iterate the assessment and feedback components far more efficiently. Once the assessment and feedback models are operating at a sufficiently accurate level, we will be able to integrate the assessment into the CENTURY platform and hook the output up to our recommendation system.

CENTURY has very significant experience in developing and piloting new features and we are confident in our timeframes. We have been the lucky recipients of multiple grant and project funds and have always delivered against our plans.

In terms of specific plans to bring LiteraSee to pilot: We are already in discussion with Wareham Middle School about piloting LiteraSee. They are already working with us on on the wider CENTURY platform. Wareham’s proportion of Black and Hispanic students on roll is higher than the state average, making them good candidates for understanding the impact of LiteraSee on priority learners (data on number of children in poverty was not available). They also have around 70% of students in grades 5 to 8 either not or only partially meeting expected standards in English overall. Although writing is only one part of the curriculum, we think there is ample opportunity for driving improvements at this school, particularly with their priority learners.

We also have contacts more widely in Massachusetts and in New York states and we hope to find more schools to pilot LiteraSee via these contacts. We would really welcome input from Solve and/or Bill & Melinda Gates Foundation in finding other appropriate schools.

Public Sharing of Learnings

As part of our commitment to educational outcomes, we are happy to share findings from our pilot phase publicly. We are accustomed to doing this in projects. Previously, this has looked like publishing our own blogs, seeking publication in education-specific media, speaking to Governments and speaking at large educational conferences, such as BETT, “the world’s biggest EdTech event”.

Go-to-market Strategy

Currently, LiteraSee is still a concept, and we intend to shape our go-to-market strategy based on feedback and data from our pilot phase. We plan to collaborate closely with schools, districts, and state governments to tailor our approach to meet their specific needs and circumstances in a way that is both scaleable and inclusive. We are considering all options up to and including a “freemium” model to make LiteraSee accessible to a broader audience.

Eventually LiteraSee will be integrated into the existing CENTURY platform, which is already an established paidfor service available to schools, governments, and individuals. Subscription models are designed to be accessible and affordable globally:

• Sliding Scale Pricing: We implement sliding scale pricing for schools, making our services more affordable for institutions with limited budgets or those that prioritize support for underserved students.

• Targeted Support Subscriptions: Some schools purchase accounts specifically for priority learners, enabling targeted interventions that are evidenced, impactful and cost-effective..

• Government Partnerships: Governments can purchase (or part-purchase) licenses that include comprehensive data sharing, allowing them to monitor and direct educational investments effectively. This model supports targeted improvements and ensures that resources are allocated where they are most needed.

Long-term sustainability and availability is our ultimate aim. All our products are built with the aim of being self-sustaining, rather than reliant on public or grant funding. This also allows and ensures that we remain hyper-responsive to market needs.

Market Entry and Legal Expertise: Although we have presence in the US, the complexity of state and federal regulations can pose significant challenges. Mentorship and strategic advice via Solve and the Gates Foundation would be invaluable.

Financial Support for Development: Funding the development of LiteraSee ourselves divides our resources among various ongoing projects. Financial support from this Challenge allows us to prioritize the early developmental stages of LiteraSee, significantly accelerating how quickly we reach piloting. Specifically, this financial support would enable the hiring of external educators to provide initial training data.

Pilot and Network Opportunities: While we have successfully secured a partnership with a school for initial piloting, we want to increase the number. Introductions via the Challenge would enable a richer data set and more robust proof of concept. Furthermore, collaborations facilitated through your networks—for instance, in the development of our educational rubric—would enrich our tool's effectiveness and inclusivity.

Experience as a Solve Winner: As a previous Solver, we have experience of the tremendous value that this recognition and the subsequent network expansion bring. These opportunities not only aid in refining our solutions but also significantly boost our credibility and outreach within the educational sector.