Faculty of Medicine
University of Bergen
R² = 0.85
against the course teacher
The published anchor: 12 sittings of the final clinical exam, 889 candidate-exam pairs, scored blind against the same rubric.
Read the case study →VALIDATION
How has Lectora been validated?
Lectora has not been validated in a single one-shot study. It has been built with three academic partners over several years, and every partnership re-runs the same comparison against that course's own teacher's prior grading. The published anchor is the final clinical exam at the Faculty of Medicine at the University of Bergen: twelve sittings, 889 candidate-exam pairs of long-form written exams, about 43,700 item-pair comparisons, run blind. Across them, Lectora's draft agreed with the course teacher at R² = 0.85, a stronger fitted linear relationship than the R² = 0.64 reported between two human graders on the same material. The question every partnership asks is the same one: is the draft close enough to review, or are you starting from scratch?
How closely does Lectora agree with the course teacher?
Across twelve sittings of the final clinical exam at the Faculty of Medicine at the University of Bergen, 889 candidate-exam pairs and roughly 43,700 item-pair comparisons, Lectora's draft agreed with the course teacher at R² = 0.85, reaching 0.86 on the summative exam. The two independent human graders who had already scored the same papers agreed with each other at R² = 0.64. Lectora was run blind, against the same rubric, with no access to either grader's scores.
Lectora (0–100) ↑
Course teacher (0–100) →
- Lectora vs. course teacher
- R² = 0.85
- Two human graders
- R² = 0.64
12 exams · 889 candidate-exam pairs · 43,683 item-pair comparisons
Dots are illustrative.
What did the retrospective workload simulation find?
In a retrospective simulation, the targeted manual scoring method used a stratified calibration set of fourteen candidates and flagged candidates whose 99% prediction intervals crossed one pass/fail threshold. Across the twelve sittings, it would have required manual scores on 12,694 of the 41,733 examiner-task pairs: 69.6% fewer, rounded to 70%, with about 40% to about 84% per exam. This measures simulated task-pair reductions, not minutes saved or performance on a letter-grade scale. The method and the simulator below are not a feature activated at installation; a qualified examiner approves every grade in the product workflow.
RETROSPECTIVE SIMULATION
Exam 01 · 78 candidates · 48 short-answer tasks
4. Manually score the risk block
Switch exam:
010203040506070809101112
Simulated task reduction
73.1%
2 736 of 3 744 examiner-task pairs omitted in the simulation, not measured time saved
Manual scoring
26.9%
Risk block
7
Calibration R²
0.882
Full-cohort R²
0.833
Selected task pairs
1 008
Full-pass task pairs
3 744
Prediction interval
90%95%99%
Where does the evidence come from?
Three academic partnerships, each re-running the same validation against its own course teacher's prior grading, including the parts where AI underperformed.
In the two independent mathematics studies at the Department of Mathematics at the University of Bergen, grading agreement held: R² = 0.68 against the professor (Pearson r = 0.82), mean absolute error 0.64 points on a 0 to 6 scale, and 87.7% pass/fail agreement at the boundary. The feedback result split by study design. In the milestone-check study, where students saw instructor, peer and AI feedback side by side, they read the AI feedback least (57% against 94% for the instructor) and rated it significantly less useful (4.38 against 5.72 on a 7-point scale). In the grader-reliability study, where each student received either human-plus-AI or human-only feedback without knowing which, no significant difference in usefulness was detected (p = 0.45); the case study states that this is not proof of equivalence. When students could see the source they preferred the human; when they could not see it, no difference was detected. The same researchers also found inter-rater reliability falling from ICC 0.874 to 0.608 with the early prototype, which is on the cards below rather than left out.
Department of Mathematics
University of Bergen
R² = 0.68
milestone-check study, measured by UiB researchers
Not our measurement. The same researchers found AI assistance dropped inter-rater agreement from ICC 0.87 to 0.61 on the early prototype, reported here because it is part of the evidence. On feedback, students who could see the source preferred the human.
Read the full findings →Finance
NHH, Norwegian School of Economics
—
Pass/fail screening on long-form analytical hand-ins. Per-cohort figures are not published yet — the per-pilot summary goes to the course coordinator.
Read the pilot scope →Read the full methodology → — what R² measures in grading, how the human baseline was produced, and the targeted-scoring math in full.
~248,500
QUESTIONS & CASES GRADED ACROSS DEPLOYMENTS
…and growing every week.
20+ courses
IN PRODUCTION AND VERIFIED
Across medicine, mathematics, finance and more.
12 sittings · 889 candidates
UIB MEDICINE TARGETED-SCORING PILOT
~43,700 AI-vs-examiner item-pair comparisons via the targeted manual scoring workflow.
Across twelve sittings of the final clinical exam at the Faculty of Medicine at the University of Bergen, Lectora agreed with the course teacher at R² = 0.85 over 889 candidate-exam pairs and roughly 43,700 item-pair comparisons, and 0.86 on the summative exam. It was run blind, against the same rubric. That is score agreement on long-form written exams. For comparison, two independent human graders on the same material agreed at R² = 0.64. The fitted score relationship was stronger for Lectora versus the course teacher in this comparison; R² alone does not measure the size of individual score differences. The UiB Medicine case study carries the per-exam breakdown. The same loop is in pilot at the Department of Mathematics at UiB and at NHH Finance.
R² describes how much variation a fitted model accounts for. In these linear score comparisons, it measures how closely scores follow a fitted line. Even R² = 1.0 can coexist with systematic differences between graders; it is not a percentage of correct grades. The final clinical exam at the Faculty of Medicine at the University of Bergen reported R² = 0.85 for Lectora versus the course teacher and R² = 0.64 between two human graders. Those figures describe this dataset, not a universal human benchmark. The methodology explains the distinction.
On the validated dataset from the final clinical exam at the Faculty of Medicine at the University of Bergen, Lectora's agreement with the course teacher (R² = 0.85) was higher than the agreement between the two human graders the exam was originally scored by (R² = 0.64). This is a comparison of linear score relationships on that one dataset; it does not establish perfect agreement or a general advantage over human graders.
Yes. The published anchor is the final clinical exam at the Faculty of Medicine at the University of Bergen: twelve real six-hour sittings, 889 candidate-exam pairs, roughly 43,700 item-pair comparisons, run blind. Not a synthetic test set, not a small sample, not a friendly subset. The same methodology runs against real cohorts in every new partnership before deployment.
889 candidate-exam pairs, across twelve sittings of the final clinical exam at the Faculty of Medicine at the University of Bergen.
Yes: the published validation is a medical exam, run across twelve sittings of the final clinical exam at the Faculty of Medicine at the University of Bergen on the same calibration loop. See the UiB Medicine pilot case study for the per-exam breakdown. For other clinical exams the same validation methodology applies; we run new validations as institutions onboard with their own rubrics and exam corpora.
Independently, yes. Researchers at the Department of Mathematics and the STEM Education Research Centre at the University of Bergen ran two studies in autumn 2025 on two mathematics courses, testing AI grading with Lectora as the tool. The milestone-check study (n = 1,051 paired AI/professor scores from 356 students on a 0–6 scale) found agreement of R² = 0.68 (Pearson r = 0.82), mean absolute error 0.64 pt, and 87.7% pass/fail agreement at the 3.0/6 boundary. The grader-reliability study (20 papers graded four times by 11 student graders) found that AI assistance dropped inter-rater ICC from 0.874 to 0.608 on the early prototype, with grader-attributable variance rising from 3.7% to 26.3%. Feedback was tested under two designs. In the milestone-check study, students saw instructor, peer and AI feedback side by side: they read the AI feedback least (57% against 94% for the instructor) and rated it significantly less useful (4.38 against 5.72 on a 7-point scale, every pairwise difference significant). In the grader-reliability study, each student received either human-plus-AI or human-only feedback without knowing which, and no significant usefulness difference was detected (p = 0.45), which the case page states is not proof of equivalence. When students could see the source they preferred the human; when they could not see it, no difference was detected. Full findings, including where AI underperformed and the methodological caveats, are in the UiB MatNat case study.
Finance is a separate, ongoing pilot at NHH Finance, Norway's leading business school, where Lectora's workflow has been tested for pass/fail screening on large multi-page hand-ins in financial asset management. The validation pattern is the same as for medicine: a real cohort the course teacher has already graded, scored blind by Lectora, compared item-by-item. Per-cohort R² numbers aren't published yet; the per-pilot summary is shared with the course coordinator.
Three academic partners. The Faculty of Medicine at the University of Bergen: the published validation across twelve sittings of the final clinical exam (R² = 0.85 vs 0.64 between two human graders), written up in the UiB Medicine case study. The Department of Mathematics at UiB: independent UiB MatNat and STEM Education Research Centre studies in autumn 2025, a milestone-check study (R² = 0.68 AI versus professor agreement) and a grader-reliability study (AI assistance dropped inter-rater ICC from 0.874 to 0.608), with the full findings, including where AI underperformed. NHH Finance: long-form analytical hand-ins, pass/fail screening pilot. Each partnership runs its own validation against the course teacher's prior grading before deployment.
Only in a retrospective simulation, and only at one pass/fail boundary. Re-run over the twelve sittings of the final clinical exam at the Faculty of Medicine at the University of Bergen, the targeted manual scoring method would have needed manual scores on 12,694 of the 41,733 examiner-task pairs, 69.6% fewer than the full pass the examiners had actually done. Per exam, the figure ranges from about 40% on sittings where many candidates cluster around the threshold to about 84% on larger sittings with sparse risk. A 14-candidate stratified calibration set plus the small block of boundary-risk candidates the method flags stands in for the full-cohort pass. It is task pairs, not minutes; it defends a pass/fail boundary rather than a letter scale; and it is a published methodology and a simulator on this site, not a feature switched on at install. Smaller cohorts save proportionally less because the 14-candidate calibration is a bigger fraction of them.
Twelve sittings of the final clinical exam at the Faculty of Medicine at the University of Bergen to date: 889 candidate-exam pairs and roughly 43,700 item-pair comparisons across the cohort. The published validation R² of 0.85 vs 0.64 is the aggregate across those twelve sittings, which were also used for the retrospective targeted manual scoring simulation; the strongest single exam reached 0.86. See the UiB Medicine case study for the workflow walkthrough; an anonymized public demo of the per-exam analysis is also available on request.