|
Getting your Trinity Audio player ready...
|
Ask a school what its AI tools have achieved and the answer is usually a usage number: logins, prompts, minutes. Usage is not impact. As AI spending grows and the novelty fades, boards and ministries are beginning to ask a harder question: did it improve learning, reduce workload or save money? The shift in 2026 from AI hype to measurable outcomes means school leaders need a way to measure AI impact in schools that goes beyond vendor dashboards.
Why Vendor Dashboards Mislead
Vendors report what their product can count, and what flatters it. Time on platform, number of questions answered and “engagement” all rise when a tool is used more, whether or not students learn more. Some metrics even move in the wrong direction: a tutor that answers more questions may be doing more of the students’ thinking.
Four Dimensions of Impact
- Learning: Did attainment or skill improve compared with a fair baseline or comparison group?
- Workload: Did teacher time on specific tasks actually fall, and was it reinvested in teaching?
- Equity: Did gaps between student groups narrow, widen or stay the same?
- Cost: What is the full cost per student, including licences, devices, training and support time?
A Practical Evaluation Method
- Define the problem first. “Improve Grade 8 algebra results” or “reduce marking time in English” is measurable. “Use AI” is not.
- Set a baseline. Record current results and time use before rollout.
- Use comparison groups. Pilot in some classes while similar classes continue as usual.
- Measure unassisted performance. Assess students without the tool to see what has been learned, not just what the tool can do.
- Review at a fixed point. One term or one year, with a pre-agreed decision: expand, adjust or stop.
The Equity Dimension
Averages hide gaps. A tool can raise average scores while helping strong students a lot and weaker students not at all. Always break results down by prior attainment, language background and additional needs. If the gap widens, the tool is not working as an equity measure, whatever the average says.
The Hardest Part: Stopping
Schools rarely stop using technology that does not work, because contracts, sunk costs and staff habits create inertia. Building a sunset clause into every AI pilot, a written condition under which the tool will be discontinued, is the most powerful accountability tool a leader has.
Frequently Asked Questions
How do you measure the impact of AI in education?
Define a specific problem, set a baseline, compare pilot groups with similar non-pilot groups, and measure learning, workload, equity and full cost.
Are usage statistics a good measure of AI success?
No. Usage shows activity, not learning. It should be combined with outcome measures assessed without the tool.
How long should an AI pilot run in a school?
Typically one term to one academic year, with a pre-agreed review and decision point.




