This is the third blog in a series outlining how we approach AI for automarking, and also introducing our products in this area. You may also be interested in this blog, which explains the principles and research that underpin our automarking work, and this one which introduces our AI-marked primary HeadStart Reading and GPS assessments.

We’re starting to see automarked assessments become available to schools, but most available solutions require a school to use an external assessment provider. That’s fine in some circumstances - for example, our automarked reading and GPS assessments for primary are curriculum agnostic - but in many situations, you want an assessment that aligns with your curriculum.

For this reason, we’ve built an AI automarking engine which is flexible enough to mark assessments you create yourself. We’ve been piloting the approach with MATs, and results so far are really encouraging.

How do AI-automarked custom assessments work?

The first step is for the MAT to share their assessment with us, including marking guidance as would be provided to teachers. For now we’re focusing on one to three mark questions, as we find that our marking accuracy is more consistent and reliable on this kind of question.

We then review the assessment and work with the MAT to refine the questions and marking guidance where required. Once that’s sorted, the assessment is set up in the Smartgrade platform, which is a simple process that the MAT themselves can perform. Students take the test in the same way they would with any other online Smartgrade test - and access is via Single Sign On with Microsoft or Google, so there’s no need to learn new passwords.

During our pilots, after we’ve run an assessment we generate a test dataset of 500 randomly selected responses, and we get this marked by one or more expert markers to establish a “ground truth”. We scrutinise any disagreements between the ground truth and the AI mark to help us understand the performance issue and improve our approach going forward. The ground truth also allows us to establish an overall marking accuracy for the test. While there are no hard and fast rules, and it depends on the subject and assessment complexity, broadly we’re aiming for 90%+ accuracy, as in our tests using real teacher-marked assessments, this usually corresponds with “better than teacher” marking levels.

It should be noted that marking is less of an exact science than we sometimes take it to be: questions and marking guidance are often written in a way that leaves ambiguity as to what is considered correct. What’s more, even expert markers change their mind - we’ve had a number of situations where we’ve presented responses back to our experts for reflection, and they’ve decided to adjust their marks!

“Huge timesaver”

Feedback from our pilots so far has been really encouraging. First, pupils like taking tests online - Dale Bassett from United Learning commented that “pupils really liked the experience and found it much easier than writing on paper. It also felt very accessible for pupils with SEND - they seemed more at ease and confident with the format than with a paper test.” The AI marking was also considered by United Learning’s teachers to be a “huge timesaver”, Dale tells us, and teachers found it “easy and quick to moderate marks”.

We were particularly excited that the consensus from our United Learning participants was that our aggregated feedback reports were “the best bit”. Dale commented that “feedback was much more specific, detailed and in greater volume than a classroom teacher would generally identify from this kind of assessment – e.g. the AI picked up really effectively on phrases the students were using incorrectly.”

Next steps

We’ve got some further enhancements we’re now working on, including a clever way of directing teachers’ attention to the questions most likely to require moderation, and additional feedback for students. To be clear, we’ve always considered student feedback to be important, but we already provide neat data-driven student reports, and we don’t want to adjust this approach until we’re confident that AI-driven feedback is a meaningful improvement on what’s already in place.  More generally, our approach is that AI-enhanced assessments isn’t a product that will ever be “finished” - we’ll keep refining and improving things as we go.

We’re ready to work with your MAT!

Now that we’ve had encouraging results from our pilots, we’re able to sign up more MATs to trial and embed AI-automarked assessments. If you’d like to reduce teacher workload while enhancing the insights you can glean from your MAT-wide assessments, click here to share your details and we’ll be in touch to arrange a consultation!

To register your interest and be kept up to date with our AI-automarked products, click the link below and share your details or get in touch for a chat.

Register your interest.

Book a chat