VALIDATION · METHODOLOGY
How is Lectora's score agreement actually measured?
This is the long-form methodology behind the headline figures on the validation summary. It covers what R² actually measures when two people grade the same exam, how the R² = 0.64 baseline between two human graders on written exams at the Faculty of Medicine at the University of Bergen was produced, why Lectora's R² = 0.85 against the course teacher sits where it does, and how targeted manual scoring turns that agreement into less manual scoring while keeping an examiner on every grade. Every figure here is the same one reported on the validation summary: this page explains it rather than adding a separate claim.
What does R² mean in the context of grading?
R² (the coefficient of determination) describes the proportion of variation accounted for by a fitted model. In the linear score comparisons reported here, it describes how closely scores follow a fitted line. It does not measure the percentage of correct grades or the size of the differences between individual scores.
R² = 1.0 means the observations lie on the fitted line; it does not require the two graders to give identical scores. A systematic offset or difference in scale can remain. R² = 0.0 means the fitted model accounts for none of the variation around the mean in this comparison, not that a grader is making random decisions.
The comparison on written exams at the Faculty of Medicine at the University of Bergen reports R² = 0.85 for Lectora against the course teacher and R² = 0.64 between the two human graders. These are results on that dataset, not a universal range for human grading. Evaluating agreement also requires examining score differences and the consequences at the assessment's decision boundaries.
What evidence is there that Lectora is accurate?
Evidence comes from three ongoing academic partnerships. The published anchor is a set of long-form written exams at the Faculty of Medicine at the University of Bergen. Separate studies cover handwritten math at the Department of Mathematics at UiB, written up in the math case study, while NHH Finance is an ongoing pilot on long-form analytical answers. The studies have different tasks, designs and outcome measures.
Two independent human graders had already scored the medicine papers, giving the baseline R² = 0.64. Lectora was run blind on the same papers and rubric without access to either human grader's scores. The comparison with the course teacher's reference scores produced R² = 0.85. Course codes, exam identities and candidate-level data are not published.
The published math case study reports the milestone-check study's score comparison and the grader-reliability study's inter-rater findings, including where AI assistance performed worse. Finance per-cohort results are not published; the pilot summary is shared with the course coordinator. The medicine result does not establish the result for a different course, rubric or assessment format.
Lectora (0–100) ↑
Course teacher (0–100) →
- Lectora vs. course teacher
- R² = 0.85
- Two human graders
- R² = 0.64
Long-form written medical exams, University of Bergen · scored blind
Dots are illustrative.
What's the inter-rater agreement among human graders?
The two independent human graders on the written medical exams at the Faculty of Medicine at the University of Bergen had R² = 0.64. The UiB Medicine case study explains that comparison. It is a measured baseline for these exams, not a general benchmark for all human graders.
In the separate grader-reliability study at the Department of Mathematics at UiB, student graders scoring the same papers had an inter-rater ICC of 0.87; with AI assistance, ICC dropped to 0.61 on the early prototype, and the share of grade variance attributable to the grader rose from 3.7% to 26%. ICC and R² are different statistics and cannot be ranked against each other as if they measured the same thing. Subject, rubric, study design and grader calibration all matter when interpreting these findings.
Differences in rubric interpretation can produce different scores for the same answer. An institution evaluating Lectora should examine those differences on its own assessment material, alongside the aggregate statistics.
Is Lectora more consistent than human graders?
The medicine comparison found a higher R² for Lectora versus the course teacher (0.85) than between the two human graders (0.64). That is stronger linear association in this comparison; R² alone does not establish that individual grades are correct, that review takes less time, or that the result transfers to another course.
The mathematics grader-reliability counterstudy is also part of the evidence: AI assistance reduced inter-rater ICC on the tested prototype. Its companion milestone-check study found score agreement holding at R² = 0.68 against the professor, with about 88% agreement on pass or fail, so the two results have to be read together rather than one at a time. Institutions should assess the draft quality and review workload in their own setting.
In the product workflow, a qualified examiner reviews and approves every grade before publication. The published studies inform that evaluation; they do not replace examiner review or validate every Lectora capability.
What does this mean for workload? Targeted manual scoring at the pass/fail boundary
Targeted manual scoring asks how few submissions an examiner must score in full for a pass/fail decision to stay sound, while every candidate keeps a human assessment. Lectora drafts a score for every submission. Examiners score a small sample without seeing the drafts, and those scores calibrate Lectora's drafts to the marking standard of that exam. Every submission whose pass/fail outcome remains in doubt goes to full manual scoring; examiners review the remaining drafts, and a qualified examiner approves every grade.
HUMAN IN THE LOOP · HOW THE SCORING WORK IS ALLOCATED
- Lectora
Draft every submission
Lectora drafts a score for each answer from the rubric and the examiner guidance.
- Examiners
Calibrate on a human sample
Examiners score a small sample without seeing the drafts, and Lectora's scores are aligned to their marking standard for this exam.
- Examiners
Score the uncertain cases in full
Every submission whose pass/fail outcome is in doubt goes to full manual scoring.
- Examiners
Approve every grade
Examiners review the remaining drafts, and a qualified examiner approves every grade before it is published.
Evaluated retrospectively on earlier written exams at the Faculty of Medicine, University of Bergen, the method needed about 70% fewer manually scored tasks than a full manual pass at one pass/fail boundary. The saving was smaller for small cohorts and for exams where many candidates sat close to the boundary.
These are simulated task counts, not measured minutes or a result for a letter-grade scale, and the calibration does not guarantee correct decisions for a new cohort. The detailed calibration and selection rules are shared with participating institutions rather than published, while the workflow is verified prospectively in pilots. The product's requirement for qualified examiner approval of every grade remains.