Can AI reliably score open-ended Hindi handwritten answers?

Evidence from 11 tests across MP and Rajasthan

By Rahul Khandelwal Source: Educate Girls internal validation, Jan–Feb 2026. Published with permission of Akhil Agrawal, Educate Girls, April 2026.

---

In a Social Science test given to students in Madhya Pradesh and Rajasthan, the long-answer question produced a striking result. MP students averaged 4% on it. RJ students averaged 44%.

That's a 40-point gap. Same subject, same exam structure, different cohorts — 28 handwritten Hindi scripts evaluated by the same AI system. The gap wasn't found by a teacher reviewing scripts. It came from AI-generated scores.

Which raises the obvious question: can we actually trust those numbers?

That's what I spent the first two months of 2026 trying to answer.

---

AI grading of open-ended Hindi handwritten answers can achieve reliable internal consistency at field conditions. In a study across 11 tests covering approximately 140 Hindi-medium students in MP and Rajasthan, AI evaluation achieved a mean Cronbach's alpha of 0.808 — above the 0.80 "good" threshold. 10 of 11 tests cleared the 0.70 research-use floor. The system produced 2,447 unique feedback variants for 2,297 student-question pairs.

---

The standard critique — and why it's mostly untested

The consensus view on AI grading of Hindi open-ended answers is sceptical, and reasonably so. The argument goes: AI can handle multiple-choice, maybe fill-in-the-blank, but it cannot reliably grade open-ended reasoning questions in Hindi, handwritten, from students in rural government school and NGO contexts.

The problem with that view isn't that it's wrong. It's that it's mostly untested.

The published research on AI evaluation reliability is almost entirely English, almost entirely typed, and almost entirely higher education. Studies from European universities show GPT-class models agreeing with human raters on English essays at rates that are encouraging. What those studies don't address: Hindi handwriting, rural K12, open-ended reasoning in a language with significant regional variation in spelling and phrasing, and field conditions rather than controlled lab settings.

When NGO program directors and EdTech product heads ask me whether they can trust AI evaluation for Hindi-medium programs, I haven't had a clean answer. There wasn't field evidence either way.

The deployment I'm describing here was an attempt to produce that evidence.

---

What I was measuring, and why it matters

Between January and February 2026, an AI evaluation system was deployed across 11 tests run by Educate Girls — an NGO operating girls' education programs in Madhya Pradesh and Rajasthan. The tests covered eight subjects: Hindi, Social Science, Home Science, Entrepreneurship, Indian Heritage, Data Entry, Chitrakala (drawing), and Culture & Heritage. Approximately 140 students participated across both states.

The tests included multiple-choice questions, short-answer questions, and open-ended questions. The open-ended evaluation section is where AI evaluation is genuinely hard. A student might write two sentences or two paragraphs. The language is Hindi, handwritten, with regional spelling conventions and abbreviations that vary between MP and Rajasthan. There's no single "correct" way to express a correct answer.

The question I set out to measure wasn't "does the AI score the same as a teacher?" That's the accuracy question, and it requires a different study design — parallel human graders scoring the same scripts. What I measured was internal consistency: does the AI evaluate in a coherent, non-random way across questions within the same test?

The standard measure for this is Cronbach's alpha.

Cronbach's alpha measures whether the scores a system produces correlate with each other in the way you'd expect if the system were tracking something real. If a student who performs well on one question in a test also tends to perform well on others — accounting for question difficulty — that's a sign the evaluation is coherent. If the scores are random noise, alpha will be low. The accepted threshold for research use is α ≥ 0.70. Above 0.80 is considered "good." Above 0.90 is "excellent" (George & Mallery, 2003; Nunnally, 1978).

---

The reliability result

Across 11 tests, the mean Cronbach's alpha was 0.808.

10 of 11 tests cleared the 0.70 threshold. The full range ran from α = 0.525 (Rajasthan Culture & Heritage) to α = 0.962 (Rajasthan Chitrakala). At the subject level: Hindi tests came in at α = 0.767 (MP) and α = 0.833 (RJ). Social Science at α = 0.817 (MP) and α = 0.747 (RJ).

The one test below threshold — Rajasthan Culture & Heritage at α = 0.525 — had only 5 students. Cronbach's alpha is statistically unstable at that sample size regardless of evaluation quality. Of the tests with N ≥ 18 students, every single one cleared α = 0.80.

That surprised me. Open-ended Hindi handwritten evaluation, at field conditions, producing "good" internal consistency ratings is not the result a reasonable sceptic would predict going in.

---

Reliability is necessary. It isn't sufficient.

Internal consistency tells you the AI is evaluating in a coherent pattern. It doesn't tell you the feedback is useful. A system that consistently assigns zero to every open-ended answer would score well on alpha. A system that consistently writes "wrong, try again" regardless of what the student wrote would also be internally consistent.

So I looked at the feedback itself — not to check whether each judgment was correct, but to understand what kind of reasoning was generating these evaluations.

---

What the feedback actually contains

The system generated 2,447 unique feedback variants for 2,297 student-question pairs. More unique reasoning artefacts than student-question pairs.

That's the first thing worth noting. If the system were template-stamping — drawing from a fixed library of "correct," "partially correct," "see model answer" responses — the output would converge on a small number of repeating strings. Instead, each student received a response shaped to their specific answer.

The feedback was analysed across 10 distinct signal types, using Hindi keyword pattern matching on the actual text. Key findings:

58.2% of feedback variants included forward-looking guidance — telling the student what to do differently next time, not just flagging what went wrong.

47.9% explicitly referenced what the student actually wrote. The system read the response; it didn't generate a generic comment based on the question alone.

39.0% both identified the specific error and provided guidance on correction — what I'd describe as the complete loop: here's what went wrong, and here's what to do about it.

The mean number of distinct signal types per feedback variant was 2.90. More than half — 52.4% — contained three or more distinct signal types in a single response.

To make this concrete, here's one feedback variant from the dataset, translated from Hindi. It's for a Social Science question where a student named the correct author but misspelled the name:

"The student gave the correct answer that 'The Prince' was written by Niccolò Machiavelli, an important literary work of the Renaissance period. However, due to spelling errors in the name and incorrect spacing, full marks cannot be awarded. Full marks are possible with correct spelling. Please attend to correct spelling of names."

That response does five distinct things: recognises the student's conceptual understanding as correct, identifies the specific error (spelling and spacing), quotes the student's mistake type, explains the scoring logic, and gives actionable guidance — in one feedback string, in Hindi, for a handwritten script.

The industry uses the word "grading" for this. What it looks like when you build the infrastructure properly is closer to teaching.

---

What the cross-cohort variation reveals

Back to the 4% and 44%.

The Social Science test in MP included one long-answer question worth 5 marks. Of 21 students, 18 scored zero. The three who scored anything managed 20%, 20%, and 40% respectively. Average: 3.8%.

In Rajasthan, the equivalent question — an explanatory question worth 6 marks — produced scores of 100%, 63%, 72%, 75%, 0%, 0%, 0% across 7 students. Average: 44.3%.

The MP cohort appears to have largely not attempted or not engaged with the open-ended format. The Rajasthan cohort shows genuine engagement with variable quality. That's a real difference between two groups of students in the same program — and the AI evaluation surfaced it from 28 handwritten scripts.

The Hindi tests showed a different pattern: MP students averaged 49%, RJ students 36% — a 12.9-point gap in the opposite direction. MP students outperformed RJ students on Hindi across 38 scripts. This directional finding is broadly consistent with state-level learning outcome patterns documented in ASER surveys for MP and Rajasthan, though the mechanisms behind state-level differences are far more complex than any single deployment can resolve.

I'm not drawing causal conclusions from this. The samples are small. There may be curriculum differences, teacher variation, or enrollment selection effects I can't see from the data. What I am saying is that the AI-generated scores track real variation between groups in a coherent pattern — subject-dependent, directionally consistent with what a teacher familiar with these programs might expect.

That is, in itself, a form of validity signal.

---

What this study does NOT establish

I want to be direct about the limits of this evidence, because I've seen too many AI evaluation claims overstate what their data proves.

Cronbach's alpha measures internal consistency. It does not measure accuracy.

This study does not establish that the AI's score on any individual student's answer matches what a trained human teacher would give. Proving that requires a controlled inter-rater agreement study — parallel evaluation of the same scripts by both AI and trained human raters, with agreement measured via ICC (intraclass correlation) or Cohen's kappa.

I have that data for a different deployment. In a parallel-grading study conducted at government schools in Andhra Pradesh, AI evaluation achieved an ICC of 0.93 (CI: 0.805–0.976) against human raters — cited in the IndiaAI Impact Summit 2026 Compendium (MeitY). But that was a different corpus, different language context, and different evaluation design.

For this Hindi-medium corpus — Educate Girls, MP and Rajasthan, January–February 2026 — the evidence shows: the AI evaluates consistently, it produces multi-dimensional feedback grounded in what students actually wrote, and its scores track real between-cohort variation. It does not show, from this data alone, that each individual score matches a trained Hindi teacher's judgment.

That is the next study. It requires bringing in trained raters, having them grade a sample of the same scripts, and comparing. That work has not yet been done for this corpus.

The distinction matters depending on what you're doing with the scores. For population-level program analysis — which subjects need intervention, how cohorts compare, where learning gaps sit — α = 0.808 is sufficient to act on. For individual student-level decisions (promotion, remediation, reporting to parents), you need the accuracy data before you act.

---

Three things to take from this if you're building at scale

Measure alpha before you ship. Cronbach's alpha is a cheap metric to compute once you have the scoring data. For any AI evaluation deployment, run it per test before using scores for any purpose. Flag anything below 0.70 for review. It costs nothing except the calculation; it surfaces whether your system is behaving coherently or generating noise.

Build for feedback, not just scores. The 58% forward-looking guidance rate and the 2.90 signals per variant aren't accidental — they come from a system designed to produce pedagogical feedback, not score labels. If your AI evaluation deployment returns a number with no reasoning, you're discarding the most useful signal the evaluation produces.

Plan the accuracy study before you need it. Reliability is the necessary first step. Accuracy — ICC against human raters — is the necessary second. NGOs and government programs will eventually ask whether scores can be trusted at the individual student level. The answer should come from data, not vendor assurance. Plan the parallel-grading validation before a funder or partner demands it.

---

The full methodology for this study — including the feedback quality analysis and the raw reliability data — is documented. If you're running Hindi-medium evaluation programs and want to compare notes on the architecture, or talk through what evaluation infrastructure looks like at production scale, reach us here.

---

Frequently Asked Questions

Q: What does Cronbach's alpha measure in AI grading? Cronbach's alpha measures internal consistency — whether the AI assigns scores in a coherent, non-random way across questions within the same test. A score of α ≥ 0.70 is the standard threshold for research use; α ≥ 0.80 is considered "good." It does not measure whether the AI's scores match what a human teacher would give. That question requires a separate inter-rater agreement study with parallel human graders.

Q: Is α=0.808 good enough to trust AI grades for program decisions? For population-level program analysis — comparing cohort averages, identifying low-performing subjects, directing remediation resources — yes. α = 0.808 provides sufficient internal consistency to act on aggregate patterns. For individual student-level decisions such as promotion, certification, or parent reporting, you need additional validation: a controlled study measuring agreement between AI scores and trained human raters via ICC or Cohen's kappa.

Q: Can AI evaluation handle handwritten Hindi scripts reliably? Based on this field study: yes, at sufficient sample sizes. 10 of 11 tests achieved α ≥ 0.70, with 8 of 11 clearing α = 0.80. The one exception (α = 0.525, Rajasthan Culture & Heritage) was a 5-student cohort — alpha is statistically unstable at that sample size regardless of the evaluation system. OCR accuracy on Hindi handwritten text was validated separately at 90%+.

Q: What's the difference between reliability and accuracy in AI grading? Reliability (Cronbach's alpha) means the AI evaluates in an internally consistent pattern. Accuracy means the AI's scores match human expert judgment. A system can be reliable but inaccurate — consistent errors — or accurate on average but unreliable across questions. Both need to be measured. This study measured reliability. A separate study conducted at government schools in Andhra Pradesh (ICC = 0.93, cited in the IndiaAI Impact Summit 2026 Compendium, MeitY) measured accuracy with parallel human graders — a different corpus, different methodology.

Q: How is this different from AI evaluation of English essays? Most published research on AI grading covers English, typed responses, and higher education contexts. This study covers Hindi, handwritten, open-ended questions in K12 and NGO settings at genuine field conditions — a largely unmeasured problem space. The compounding challenges are OCR variation on handwritten scripts, regional Hindi spelling conventions across MP and Rajasthan, open-ended reasoning questions without a single correct answer form, and evaluation at NGO program scale without controlled test conditions.