From ideas to evidence

Madeline Newman, Executive Director, AI Forum New Zealand

I gave a talk recently at GOVIS 2026 for their ‘Delivering with AI’ workstream. The brief challenged me to gather some of the success stories that demonstrate what implementation looks like in real life — and to look for the common learnings.

The stories start small and build from there, with successes and a cautionary tale. The important thing is that they all point to these 5 lessons:

1. Start with a problem to solve

2. Measure before and after

3. Never underestimate the power of good, accessible data

4. Watch for unexpected benefits and consequences

5. Look for where AI can augment people, rather than replace them

A couple of questions from the audience afterwards have stuck with me, and they’re worth answering here before we get into the dispatches themselves.

QUESTION FROM THE AUDIENCE: For a public sector team about to start an AI project, what’s 1 thing you’d tell them to check before they even begin?

Check that you’re solving a real problem — not reaching for AI because it’s the technology everyone is talking about. Every story in this piece started with a genuine, specific piece of friction: records that needed classifying, a worksite that needed additional safety, a queue of customers that needed answering. If you can’t define the problem, it’s difficult to design a workable solution.

QUESTION FROM THE AUDIENCE: Of the 4 case studies, was there a common thread that separated those that delivered value from those that didn’t?

Measurement. The case studies that stood up to scrutiny — Dunedin, the Reserve Bank, Hamilton City, and even Klarna after it changed course — all measured their position before they began and kept measuring after launch. In my experience, being able to measure change implementation for benefits realisation is essential.

Dispatch 1 — Small Wins

Let’s start with Silicon Valley, no wait — Dunedin.

Dunedin — a $332 business case

Deep in every City Council’s building services team sit the Building Warrant of Fitness records. Classification of records like these are good examples of robot work that uses up skilled people’s time.

Dunedin City Council built an AI classification system for those records for $332 — less than the cost many would spend on catering for a meeting about AI.

Importantly, they measured before and after, discovering they were able to quantify savings of 160 administrative hours annually, and achieved a 93% improvement in accuracy.

Without measuring the impact this might have gone unnoticed. Instead, it was picked up by Mike Manson, ALGIM’s Chief Executive, who wrote it up: “This wasn’t a seven figure transformation programme, but it didn’t need to be. It was a focused application of LLMs to a real compliance problem.”

KEY FACT: $332 investment saved 160 administrative hours a year and lifted accuracy by 93%.

Fonterra’s “Arnie” — the same lesson in the private sector

In the private sector the small wins look identical.

Fonterra built a bot called Arnie to handle inventory reclassification checks. Work that used to take 2 hours was completed in 10 to 20 minutes, without any errors. Calculated benefits were more than $180,000 a year in working capital costs.

Fonterra’s own advice from that journey is: “Don’t chase AI magic. Chase friction.”

KEY FACT: A 2-hour manual check now takes 10–20 minutes — and saves Fonterra $180,000+ a year.

In other words, start with the business problem and develop a solution — which may or may not involve AI.

Christchurch City Council — preventable losses

Our third small win was developed in response to the 2017 Port Hills fire in Christchurch. 1,600 hectares burned, 9 homes were destroyed, and it took 66 days to put it out.

In response, Christchurch built a network of 15 solar-powered sensor towers across the city. Using AI, they read air composition, smoke, wind speed and direction, plus visual and thermal imaging, watching continuously for the earliest signs of a fire starting.

When something is detected, the system automatically routes real-time situational intelligence straight to council staff and to Fire and Emergency New Zealand.

The result is 15 connected smart sensors helping to prevent loss of life, damage to property and harm to the environment. Last year, the project won the SOCITM International Innovation Award in the UK.

KEY FACT: 15 sensors installed after the 2017 Port Hills fire, now actively watch for the earliest signs of fire.

This dispatch’s lesson

Small wins teach us to start where the friction is and measure what changes — and sometimes the whole business case can cost less than the meeting to approve it.

Dispatch 2 — Wins at Scale

But what happens when we implement this at scale?

Reserve Bank of New Zealand — managing the systems that underpin the financial reputation and stability of Aotearoa

ESAS and NZClear settle New Zealand’s banking and financial markets — transacting billions of dollars every day. Both are key to international confidence in, and the reputation of, New Zealand’s financial markets. If they go down, the world notices.

The Reserve Bank partnered with Datacom on a long-term embedded team, to automate and modernise the infrastructure that underpins these systems.

The result was reliability with 100% availability of both systems over a full year. Added to that was improved agility, meaning that infrastructure changes that used to take hours can now take as little as 30 minutes, while a full site swap (moving the whole operation to backup infrastructure) can now be completed in under an hour. RBNZ’s satisfaction score was 9.2 out of 10.

KEY FACT Reliability: 100% availability for over 12 months, on 2 of the country’s most critical financial systems.

When confidence and reputation matter, don’t move fast and break things — move carefully and don’t break anything at all.

The UK’s 20,000-person experiment

The UK Government Digital Service gave Microsoft Copilot to 20,000 civil servants and evaluated the result.

The average saving was 26 minutes per day, or about 2 weeks per person per year. More than a third of people saved over half an hour daily, and 82 percent said they didn’t want to go back to working without it.

The Department for Work and Pensions ran a more rigorous trial over 6 months with a comparison group.

They saved 19 minutes per day, with the biggest gains predictably in searching for information and drafting emails. 2-thirds of users reported feeling more fulfilled at work as AI took care of the drudgery, and the people enjoyed doing more of the job they joined for.

Implementation gave them improved efficiency and improved employee satisfaction.

KEY FACT: 26 minutes a day — about 2 working weeks a year, per person — saved by 20,000 UK civil servants using Copilot.

Health, at clinical scale

Ambient scribes — software that listens to a consultation and drafts the clinical notes – are a great example of AI augmentation. The clinician isn’t replaced, they’re freed up to do the actual job they trained for.

Health New Zealand is now rolling out the Heidi AI Scribe to 16 of our 38 emergency departments, aiming for every hospital by the end of this year.

And in primary care, a homegrown alternative is already at work: Tuhi, built by Awa Digital — a Māori-owned and built health tech company. Around 450 clinicians, largely independent primary care providers, are already using it.

But what impact can we expect when it’s measured?

The Permanente Medical Group in the US deployed ambient scribes across more than 2.5 million patient encounters in a year. It saved an estimated 15,791 hours of physician time, with reduced clinician burnout and better patient communication, because the doctor is looking at the patient not at their keyboard.

Interestingly, while recent studies indicate modest time savings, AI scribes have been found to reduce burnout and improve wellbeing in a field currently under intense pressure.

KEY FACT: 15,791 hours of physician time saved in a year — time spent looking at patients, not keyboards.

Government trial in New Zealand

Over 300 Copilot licences were deployed across 8 business groups within a large New Zealand government agency. Microsoft reported a 287 percent return on investment, calculated against participants’ hourly rates.

The study also found real benefits for neurodiverse staff and for colleagues working in English as a second language, demonstrating both efficiency and inclusion gains, from the same tool.

KEY FACT: 287% ROI — plus measurable gains for neurodiverse staff and colleagues working in a second language.

The dispatch’s lesson

These larger wins show us the pilots weren’t a fluke.

When you test with thousands of people instead of dozens, with comparison groups instead of enthusiasm, the returns are still there — and when you implement change that responds to a real business problem, unexpected benefits often emerge that far outweigh the original business case.

Dispatch 3 — Into the Physical World

What happens when we bring AI out of the office and into the physical world?

Hamilton — better decisions, not just faster ones

Hamilton City Council manages almost every road in a fast-growing city. To decide where to build, where to maintain, where to change speed limits and how to zone, they need to know how vehicles actually behave.

The traditional way of counting traffic cost upwards of 70,000 dollars a year. This research provided 4 week-long snapshots at up to 150 locations — roughly 600 data points, which they extrapolated to make decisions that shape an entire city.

Through the Smart Hamilton programme, the council partnered with Aware Group — a Waikato company, now part of HSO — on computer vision that classifies and counts vehicles in real time. That work began back in 2018 and is in production today.

Hamilton didn’t use AI to do the old job faster. They used it to replace a periodic sample with a continuous picture — a productivity gain measured in better decisions for Hamilton’s future, because they now have access to millions of real-time data points collected as the city moves.

The same partnership produced pedestrian and cycling counters that drew orders from 2 other NZ cities — a good example of 1 council’s pilot becoming a sector capability.

KEY FACT: From 600 data points a year to millions — Hamilton traded a periodic sample for a continuous picture of how its city moves.

New Zealand Rugby

The national game can’t escape the nation’s gaze. The Black Ferns win 82% of their matches and the All Blacks win 76%. The competition is intense and data insights are incredibly important.

NZ Rugby was sitting on more data than they could use — over a 1,000 data points per player, per game. The problem was it was scattered across fragmented systems, meaning insights were taking days to surface.

Working with AWS, they consolidated it all onto 1 platform, meaning high-performance staff can now query performance data in plain language instead of waiting on an analyst. More than 150 high-performance teams now get insights in near-real-time, not days — fast enough to inform in-game adjustments, not just next week’s training plan.

And best of all, now we all get to see the results!

KEY FACT: 1,000 data points per player, per game — for teams where the margin between winning and losing is razor thin.

Tupua Horo Nuku — AI watching out for the crew

In Lower Hutt, the Tupua Horo Nuku project is building a 4.4-kilometre seawall and shared path along Marine Drive.

On that confined coastal worksite there is live traffic on 1 side and open water on the other. AI-enabled cameras developed by Kiwi company RUSH Digital watch the site in real time.

The system can recognise a fall the moment it happens, or a pedestrian stepping into a restricted zone. A worker without their PPE, or a vehicle moving too fast through the temporary traffic management.

This is another great example of an AI implementation that augments existing staff. AI is being used to enhance human judgement, giving the team on the ground real-time information that a human spotter simply couldn’t provide across a 4-kilometre corridor — helping safety staff provide better protection between a hazard and a harm.

They also engaged early with the Office of the Privacy Commissioner. Downer is very clear about what it is doing: it identifies roles and hazards, not individuals. It is not collecting people’s personal information.

The measures are about everyone getting home safely.

KEY FACT: A second set of eyes, not a replacement for 1 — AI cameras watching a 4.4km worksite for falls, PPE breaches and restricted-zone entries.

Dispatch 4 — A Cautionary Tale of a Big Win

But what happens when things go a little too well?

Klarna — the win that overshot

In early 2024, the Swedish fintech Klarna became the poster child for AI customer service. Their assistant handled 2.3 million conversations in its first month — the work of 700 full-time agents.

Resolution times fell from 11 minutes to under 2.

The projected profit improvement of $40 million US dollars grew to nearly $60 million by late 2025 as the assistant took over work equivalent to over 850 agents.

Then, in May 2025, Klarna began hiring human agents back — the CEO announced that they had gone too far.

While Klarna were basking in the glory of being the AI poster child, they kept measuring and analysing their results, and what they found was that while their AI assistant was brilliant at the routine 80 percent, it was not good at the 5 to 10 percent of cases that were complex: disputed refunds, incorrect charges, hardship payment plans, fraud cases. These are situations where the customer arrives already frustrated or under stress, with little emotional bandwidth. A poorly answered query can quickly escalate and is better answered by an empathetic human ear.

KEY FACT: 700 agents’ worth of AI-handled work grew to 853 — and in May 2025, the CEO still said: “We went too far.”

The dispatch’s lesson

Klarna kept measuring after launch, even when they had runaway success. They also had the confidence and ethos to change course the moment the evidence told them to.

The lesson from Klarna is to keep measuring even after implementation, and watch for unexpected results — both benefits and complications — and be prepared to pivot even when the world is watching.

The 5 Key Points

Across these 4 dispatches, the throughline holds whether the story is a Council team, a Crown agency, or an international brand — and, fittingly for this crowd, most of the strongest examples were homegrown.

A note of caution here: the best results we are seeing currently are from deeper, bespoke builds that include agents — but the underlying message behind our 5 key points of learning remains the same.

5 key points of learning:

1. Start with a real problem to solve — sometimes the answer is AI.

2. Measuring matters — before you start, so you know what success looks like; and after you launch, keep counting honestly, including the bruises. Because we’re all still learning.

3. Never underestimate the power of good, accessible data. Whether you are Hamilton City Council or the All Blacks.

4. Keep your eyes open for unexpected benefits and complications — and be prepared to change your own measure of success. Sometimes the AI hands you a better scorecard than the 1 you started with.

5. Some of the most successful AI implementations are the ones where AI has been used to augment people — helping them make better decisions, serve customers better, have more time to build empathy in a conversation, and improving inclusion by removing accessibility barriers.

Most of all, use these tools well.

Don’t just build things better — build better things.