Faculty of Medicine
University of Bergen
R² = 0.85
against the course teacher
The published anchor: long-form written exams, scored blind against the examiners' own rubric and compared with the scores they had already given.
Read the case study →VALIDATION
How has Lectora been validated?
Lectora has not been validated in a single one-shot study. It has been built with three academic partners over several years, and every partnership re-runs the same comparison against that course's own teacher's prior grading. The published anchor is a set of long-form written exams at the Faculty of Medicine at the University of Bergen, run blind against scores the examiners had already given. On them, Lectora's draft agreed with the course teacher at R² = 0.85, a stronger fitted linear relationship than the R² = 0.64 reported between two human graders on the same material. The question every partnership asks is the same one: is the draft close enough to review, or are you starting from scratch?
How closely does Lectora agree with the course teacher?
On long-form written exams at the Faculty of Medicine at the University of Bergen, Lectora's draft agreed with the course teacher at R² = 0.85. The two independent human graders who had already scored the same papers agreed with each other at R² = 0.64. Lectora was run blind, against the same rubric, with no access to either grader's scores.
Lectora (0–100) ↑
Course teacher (0–100) →
- Lectora vs. course teacher
- R² = 0.85
- Two human graders
- R² = 0.64
Long-form written medical exams, University of Bergen · scored blind
Dots are illustrative.
How does targeted manual scoring keep the examiner in the loop?
Agreement matters because it decides where an examiner's time goes. In targeted manual scoring, examiners first score a small sample without seeing Lectora's drafts, which calibrates the drafts to their marking standard for that exam. Every submission whose pass/fail outcome is in doubt is then scored in full, examiners review the remaining drafts, and a qualified examiner approves every grade.
In a retrospective simulation on earlier exams, this allocation needed about 70% fewer manually scored tasks at one pass/fail boundary. That is a simulated task count, not minutes saved or a result for a letter-grade scale. The detailed calibration and selection rules are shared with participating institutions while the workflow is verified prospectively in pilots.
HUMAN IN THE LOOP · HOW THE SCORING WORK IS ALLOCATED
- Lectora
Draft every submission
Lectora drafts a score for each answer from the rubric and the examiner guidance.
- Examiners
Calibrate on a human sample
Examiners score a small sample without seeing the drafts, and Lectora's scores are aligned to their marking standard for this exam.
- Examiners
Score the uncertain cases in full
Every submission whose pass/fail outcome is in doubt goes to full manual scoring.
- Examiners
Approve every grade
Examiners review the remaining drafts, and a qualified examiner approves every grade before it is published.
Where does the evidence come from?
Three academic partnerships, each re-running the same validation against its own course teacher's prior grading, including the parts where AI underperformed.
In the two independent mathematics studies at the Department of Mathematics at the University of Bergen, score agreement held: R² = 0.68 against the professor and about 88% agreement on pass or fail. The feedback result split by study design. When students saw instructor, peer and AI feedback side by side, they read the AI feedback least and rated it significantly less useful. When each student received either human-plus-AI or human-only feedback without knowing which, no significant difference in usefulness was detected (p = 0.45); the case study states that this is not proof of equivalence. When students could see the source they preferred the human; when they could not see it, no difference was detected. The same researchers also found agreement between graders falling from ICC 0.87 to 0.61 with the early prototype, which is on the cards below rather than left out.
Department of Mathematics
University of Bergen
R² = 0.68
score agreement with the professor, measured by UiB researchers
Not our measurement. The same researchers found AI assistance dropped inter-rater agreement from ICC 0.87 to 0.61 on the early prototype, reported here because it is part of the evidence. On feedback, students who could see the source preferred the human.
Read the full findings →Finance
NHH, Norwegian School of Economics
—
Pass/fail screening on long-form analytical hand-ins. Per-cohort figures are not published yet — the per-pilot summary goes to the course coordinator.
Read the pilot scope →Read the full methodology → — what R² measures in grading, how the human baseline was produced, and how targeted scoring keeps an examiner on every grade.
~248,500
QUESTIONS & CASES GRADED ACROSS DEPLOYMENTS
…and growing every week.
20+ courses
IN PRODUCTION AND VERIFIED
Across medicine, mathematics, finance and more.
~70% fewer
UIB MEDICINE · TARGETED MANUAL SCORING
manually scored tasks in a retrospective simulation at one pass/fail boundary. Every grade still needs examiner approval.
On long-form written exams at the Faculty of Medicine at the University of Bergen, Lectora agreed with the course teacher at R² = 0.85, run blind against the same rubric. That is score agreement on long-form written exams. For comparison, two independent human graders on the same material agreed at R² = 0.64. The fitted score relationship was stronger for Lectora versus the course teacher in this comparison; R² alone does not measure the size of individual score differences. The UiB Medicine case study explains the comparison. The same loop is in pilot at the Department of Mathematics at UiB and at NHH Finance.
R² describes how much variation a fitted model accounts for. In these linear score comparisons, it measures how closely scores follow a fitted line. Even R² = 1.0 can coexist with systematic differences between graders; it is not a percentage of correct grades. The written exams at the Faculty of Medicine at the University of Bergen gave R² = 0.85 for Lectora versus the course teacher and R² = 0.64 between two human graders. Those figures describe this dataset, not a universal human benchmark. The methodology explains the distinction.
On the written medical exams at the Faculty of Medicine at the University of Bergen, Lectora's agreement with the course teacher (R² = 0.85) was higher than the agreement between the two human graders the exams were originally scored by (R² = 0.64). This is a comparison of linear score relationships on that one dataset; it does not establish perfect agreement or a general advantage over human graders.
Yes. The published anchor is a set of real long-form written exams at the Faculty of Medicine at the University of Bergen that examiners had already scored, run blind. Not a synthetic test set and not a hand-picked subset. The same methodology runs against real cohorts in every new partnership before deployment.
Yes: the published validation uses written medical exams at the Faculty of Medicine at the University of Bergen. See the UiB Medicine pilot case study. For other clinical exams the same validation methodology applies; we run new validations as institutions onboard with their own rubrics and exam material.
Independently, yes. Researchers at the Department of Mathematics and the STEM Education Research Centre at the University of Bergen ran two studies in autumn 2025 in two introductory mathematics courses, with an early Lectora prototype as the tool. The milestone-check study found score agreement with the professor of R² = 0.68 and about 88% agreement on pass or fail. The grader-reliability study found that AI assistance lowered agreement between graders, from ICC 0.87 to 0.61. On feedback, students who could see the source preferred the human; when they could not, no significant difference was detected (p = 0.45), which is not proof of equivalence. Full findings, including where AI underperformed, are in the UiB mathematics case study.
Finance is a separate, ongoing pilot at NHH Finance, Norway's leading business school, where Lectora's workflow is being tested for pass/fail screening on large multi-page analytical hand-ins. The validation pattern is the same as for medicine: a real cohort the course teacher has already graded, scored blind by Lectora and compared item by item. Per-cohort figures are not published; the pilot summary is shared with the course coordinator.
Three academic partners. The Faculty of Medicine at the University of Bergen: the published validation on long-form written exams (R² = 0.85 vs 0.64 between two human graders), written up in the UiB Medicine case study. The Department of Mathematics at UiB: independent studies with the STEM Education Research Centre in autumn 2025, a milestone-check study (R² = 0.68 against the professor) and a grader-reliability study (AI assistance lowered inter-rater ICC from 0.87 to 0.61), with the findings, including where AI underperformed. NHH Finance: long-form analytical hand-ins, pass/fail screening pilot. Each partnership runs its own validation against the course teacher's prior grading before deployment.
Only in a retrospective simulation, and only at one pass/fail boundary. Re-run on earlier written exams at the Faculty of Medicine at the University of Bergen, targeted manual scoring needed about 70% fewer manually scored tasks than the full manual pass the examiners had actually done. Examiners first score a small sample without seeing the drafts, every submission with an uncertain pass/fail outcome is scored in full, and a qualified examiner approves every grade. The saving was smaller for small cohorts and for exams with many candidates near the boundary. It is task counts, not minutes, and not a result for a letter-grade scale. The workflow is being verified prospectively with pilot institutions; it is not a feature switched on at installation.