Does using AI assistance affect grader consistency?

Yes, and not in Lectora's favour. In the grader-reliability study, the same papers were graded several times by different student graders, with and without AI assistance. Agreement between graders fell from ICC 0.87 without AI to 0.61 with AI on the early prototype, and more of the grade variation came down to which grader handled the paper. No significant difference in average grade was detected. Familiarity with the tool and how graders used its suggestions are possible explanations the study did not isolate; grader training is a recommendation whose effect still has to be measured.

Do students find AI feedback useful?

It depended on the study design. When students saw instructor, peer and AI feedback side by side, they read the AI feedback least and rated it significantly less useful than either human source. When each student received either human-curated AI feedback or human-only feedback without knowing which, no significant difference in usefulness was detected (p = 0.45). That is not proof of equivalence, and the two designs differ in more than presentation. The findings concern the autumn 2025 prototype and do not measure current feedback quality.

Has Lectora been independently validated for AI grading on mathematics courses?

An early Lectora prototype was independently evaluated in two introductory mathematics courses at the University of Bergen in autumn 2025. The researchers designed, ran and analysed the studies; Lectora provided the tool. The findings cover score agreement, grader consistency and student perceptions. They do not validate all mathematics courses or the current product. See the validation methodology for how to interpret score comparisons.

Can AI grading replace human grading on mathematics courses?

No. In Lectora, a qualified examiner reviews and approves every grade. Pass/fail agreement with the professor was high but not complete, and grader consistency fell with AI assistance on the prototype. The appropriate role is a draft for review, evaluated against the course's own rubric and earlier assessment material. These studies do not establish that reviewing a draft takes less time than assessing from scratch.

Can we collaborate with Lectora on follow-up research instead of evaluating the tool for deployment?

Yes. The reported studies leave questions about student perceptions, outcomes on later assessments, and how educators review AI drafts. A research collaboration could investigate those questions with a design suited to the institution. Request access to start a conversation about a pilot or research collaboration.

How do we set up a pilot to evaluate AI grading for our course?

A course-specific evaluation could use already-graded papers from a recent sitting. Run Lectora without access to the reference scores against the same rubric, then compare individual scores and pass/fail decisions with the course teacher's reference. Examine R² alongside absolute errors and boundary disagreements, and assess agreement across several graders using the tool. If students receive feedback, measure perceived usefulness separately from learning outcomes. Request access to discuss the scope.

Can AI grade handwritten math submissions?

Lectora can use scanned PDFs and images of handwritten working when drafting an assessment against the rubric. Reading errors remain possible, especially in faint scans or ambiguous notation. The educator should compare the draft with the original submission; the UiB results do not establish perfect handwriting recognition.

Does Lectora actually run student calculations to verify them?

The assessment workflow has access to code execution and instructions to use it for quantitative checks, including numerical answers, algebra and matrix operations. That capability does not guarantee that every calculation or line is executed, or that every error is detected. The resulting score and feedback remain drafts for examiner review.

Does Lectora recognize 2(x+1) and 2x+2 as equivalent?

These expressions are algebraically equivalent. Lectora can consider equivalent forms when drafting against a rubric, but the published studies do not establish reliable recognition of every symbolic identity. The rubric’s required form and the expression’s domain still matter, and the examiner reviews the proposed credit.

How does Lectora handle proof by induction?

A rubric can distinguish the base case, inductive hypothesis, inductive step and conclusion. Lectora uses the rubric to draft a score and explanation; the examiner checks the logic and proposed partial credit. Testing representative numerical values is not a proof of the inductive step.

Can Lectora grade step-by-step working when the final answer is wrong?

A rubric can award method marks when a later arithmetic error makes the final answer wrong. Lectora drafts against those criteria, and the examiner checks whether the proposed partial credit is justified. The studies do not establish correct line-by-line error identification for every submission.

Does Lectora work for university-level mathematics?

The independent evidence reported here concerns two introductory courses. It does not establish performance in every university mathematics subject, including advanced analysis or abstract algebra. Other courses require evaluation with their own submissions and rubrics.

INDEPENDENT RESEARCH · UIB MATHEMATICS

Can AI be trusted to grade student mathematics assignments?

In autumn 2025, independent researchers at the Department of Mathematics and the STEM Education Research Centre at the University of Bergen tested AI-assisted assessment with an early Lectora prototype in two introductory mathematics courses. In the milestone-check study, the AI draft agreed with the professor's scores at R² = 0.68, with about 88% agreement on pass or fail. In the grader-reliability study, AI assistance lowered agreement between graders, from ICC 0.87 to 0.61. On feedback, students who could see the source preferred the human; when they could not, no significant difference was detected. These findings support course-specific evaluation, not a general conclusion that AI assessment is ready for deployment at scale. The page below reports them, including where AI assistance performed worse, alongside the published validation summary.

About the UiB research

The research project set out to find evidence-based ways of giving effective feedback in large introductory mathematics courses. In autumn 2025, researchers at the Department of Mathematics and the STEM Education Research Centre at the University of Bergen ran two parallel studies, one in each of two large first-year courses. Lectora was the AI tool the researchers evaluated; we provided an early prototype but had no role in study design, data collection or analysis.

The grader-reliability study used AI as a grading partner on a mandatory assignment. Papers were assigned at random to human-only grading or to human grading with an AI draft, and students did not know which condition they were in. A subset of papers was graded several times by different graders to measure agreement between them, and a follow-up survey asked students how useful they found their feedback.

The milestone-check study used AI as one of three feedback sources on milestone submissions. Each student received feedback from the instructor, from peers and from Lectora side by side, and the AI draft scores were compared with the professor's. A follow-up survey asked which feedback students read and how useful they found each source.

The handwritten mathematics example uses fictional work, authored example feedback and illustrative cohort data. It is not a record of students in either course or a measurement of results from this research.

Did the AI draft agree with the professor's scores?

In the milestone-check study, Lectora's draft agreed with the professor's scores at R² = 0.68 on a 0–6 scale, and the two agreed on pass or fail for about 88% of submissions. The disagreements went both ways and clustered close to the pass mark. On average the draft scored slightly lower than the professor, mostly at the top of the scale. A lower draft score is not inherently safer; it can also disadvantage a student if accepted without review.

R² describes the fitted linear relationship with the professor's scores on this dataset. We have no published human-versus-human R² for mathematics to set beside it, and R² alone does not establish correct individual scores or reduced review workload.

How did AI assistance affect grader consistency?

This is the finding most likely to give institutional buyers pause, and the reason it is reported here. In the grader-reliability study, agreement between graders, measured as ICC, fell from 0.87 for human-only grading to 0.61 when graders used AI assistance. The share of grade variation that came down to which grader handled the paper rose from 3.7% to 26%. No significant shift in average grade was detected, which does not prove that there was none.

Two possible explanations deserve investigation. The graders were using an early prototype with a course rubric that had not been tuned beforehand, so familiarity and rubric interpretation may have mattered. Reviewing AI suggestions may also change how graders arrive at a score. The study does not separate these explanations or establish that onboarding and tuning would recover the lost agreement.

How did students perceive the AI feedback?

Two study designs, two answers. In the milestone-check study, students saw instructor, peer and AI feedback side by side on the same submission. They read the AI feedback least and rated it significantly less useful than either human source, and their comments often described it as unreliable or inconsistent with the instructor.

In the grader-reliability study, each student received only one condition, human-curated AI feedback or human-only feedback, and did not know which. No significant difference in usefulness was detected (p = 0.45). A non-significant difference does not establish equivalence, and the AI condition included human curation, so it is not a test of unreviewed AI feedback.

When students could see the source, they preferred the human; when they could not, no difference was detected. Course, task, presentation and curation all differ between the two studies, so they cannot identify presentation as the cause. A controlled comparison of AI-only, peer-only and instructor-only feedback could help separate these effects.

What state was the prototype in, and what has changed since?

The autumn 2025 studies evaluated an early Lectora prototype, and the results describe that prototype in those courses. They do not establish how the current version performs, and subsequent development is not evidence that the observed problems have been resolved. The grader-consistency finding remains relevant to any institution considering a pilot: an evaluation should examine both the draft and how different graders use it.

How does Lectora read handwritten math and verify the calculations today?

Mathematics submissions can include scanned pages, images and typed working. Lectora's document workflow supplies submission content for assessment against the course rubric. Faint scans and ambiguous symbols can be misread, so the educator should check the original material when reviewing the draft. The research reported here does not measure recognition accuracy by line or establish a calibrated handwriting-confidence display.

For quantitative work, the assessment workflow has access to code execution and instructions to use it for checks such as numerical answers, algebra, calculus and matrix operations. Executed checks can support the assessment, but access to the tool is not a guarantee that every line is executed or that every arithmetic error is caught. The code's inputs and assumptions also need to be correct.

A numerical check and a mathematical proof serve different purposes. Computing an integral or matrix operation can help check a result; testing an induction step at representative values does not prove it for all values. The rubric governs the proposed credit, and the examiner reviews both the reasoning and the draft score.

How does Lectora handle partial credit, symbolic equivalence, and notation correctness?

Partial credit is where rubric design matters. A student who identifies the base case and inductive hypothesis but makes an algebraic error in the inductive step has shown different work from a student who omits the induction entirely. If the rubric assigns separate marks for those elements, Lectora can use them when drafting a score and explanation. The examiner decides whether the proposed method marks and deductions are justified; the published studies do not establish correct partial-credit allocation for every proof.

Equivalent expressions may deserve the same credit when the rubric allows either form. For example, 2(x+1) and 2x+2 are equivalent; e^(ln x) and x require the domain condition x > 0 over the reals. Lectora can consider such expressions in its assessment draft, but this is not a claim of a universal symbolic verifier. Domain assumptions and requirements for simplified or factored form must be checked against the task and rubric.

Notation can be a separate criterion when the rubric defines one. Variable definitions, set-membership symbols, and the distinction between equality, equivalence and implication may then be assessed separately from method. Lectora drafts against those criteria, and the examiner checks the proposed deductions and edits the explanation before it reaches the student.

What should an institution evaluating AI grading tools take from this research?

The UiB research gives institutions reasons to evaluate AI-assisted assessment carefully. Score agreement with the professor held up, grader consistency fell with AI assistance on the early prototype, and the feedback results depended on whether students could see the source. Together, these findings do not establish readiness for deployment at scale or equivalent feedback quality.

A course-specific evaluation can compare drafts with previously assessed submissions under the same rubric, examining R² alongside absolute errors, boundary decisions and agreement between several graders using the tool. Grader calibration is worth evaluating too: the drop in grader agreement gives reason to examine how graders use a draft, but the study did not include a trained comparison arm and cannot show that training closes the gap.

If you want to evaluate Lectora with your course's submissions and rubric, request access to discuss a pilot. The same conversation can cover follow-up research, such as separating feedback-source and presentation effects, measuring outcomes on later assessments or examining examiner review workload. The study design and Lectora's role should be agreed with the institution before work begins.

In the independent milestone-check study, Lectora's draft agreed with the professor's scores at R² = 0.68 on a 0–6 scale, and the two agreed on pass or fail for about 88% of submissions. R² describes the fitted linear relationship, not the percentage of correct scores, and we have not published a human-versus-human R² for mathematics. The disagreements clustered close to the pass mark, which is one reason an examiner reviews every draft.

Request access →