Measurement
The question is validity
I am a psychologist by training. My question about a system is whether it measures what it claims to and whether the inference drawn from it is warranted — not how many posts it can replace.
Working with institutions
Advisory
What an AI consultant in higher education can usefully be — and what I am not.
“AI consultant” has acquired an unhelpful default meaning: someone who arrives with a deck, asserts that generative AI will save an organisation a great deal of money, and proposes to demonstrate this by removing staff. That is not this.
My work is narrower and, I think, more useful. I work on the responsible use of artificial intelligence in higher education — particularly in assessment, academic integrity and institutional governance. It rests on three things that are hard to acquire separately: training as a measurement scientist, three decades inside universities, and having done the sector’s finances myself.
Institutional
I know what a Senate is
Three decades in London universities. I know how an exam board actually runs, what a condition of registration requires, and how long a policy change genuinely takes to land.
Financial
I know the money is not there
Having read 150 universities’ accounts line by line, I do not propose things institutions cannot afford. Advice that ignores the sector’s finances is advice for a sector that does not exist.
Areas of work
Assessment design
Whether an assessment still evidences what it claims to.
- Auditing assessment portfolios against what current models can actually do, rather than against what a vendor says they can do.
- Redesigning tasks so that the thing being measured survives — and so the redesign survives an exam board, an external examiner and a professional body.
- Distinguishing the assessments that are genuinely compromised from the much larger number that are merely assumed to be.
Academic integrity
Processes that stay fair when the evidence is probabilistic.
- Testing detection and similarity claims against the evidence that actually supports them, including base rates and differential false-positive rates.
- Writing misconduct policy and panel guidance that can withstand appeal, an Office of the Independent Adjudicator complaint, or a legal challenge.
- Training for panels and academic staff on what an AI-generated-text finding does and does not entitle them to conclude.
Institutional governance
Reading an AI proposal against the institution that has to run it.
- Assessing proposals and procurements against the institution’s own regulations, data protection position, budget and regulatory obligations.
- Briefings for executive teams, academic boards and governing bodies in language that neither oversells nor dismisses.
- Saying plainly which parts of a proposal will not work here, and why — before the contract, not after.
Evaluation and evidence
Establishing whether a deployed system is doing what was claimed.
- Designing evaluations for AI tools already in use, with pre-specified measures rather than post-hoc satisfaction surveys.
- Fairness and validity review of automated or AI-assisted scoring, screening and selection.
- Reporting written to be read by people who will be asked to defend the decision later.
Formats
Most of this arrives as one of four things: a written review of a specific proposal, policy or assessment portfolio; a briefing or workshop for a committee, department or executive team; a longer piece of evaluation work on a system already in use; or a conference or in-house talk. Teaching and media work sit alongside it.
What I will tell you
Including when the answer is that the tool is fine and the policy is the problem, that the assessment was never valid and generative AI has merely made that visible, or that the proposed system would work but the institution cannot currently support it. Those answers are not always what a project sponsor wants, which is rather the point of commissioning someone external.
Enquiries. Email justin@obrien.vision with a short description of the problem and the decision it attaches to. If it is not something I can help with, I will say so and, where I can, suggest who might.