Prototyping an AI feature

A team with an idea for an AI feature can try it out before spending much on it. A prototype is a quick, rough version built to try the idea out. This page draws on the GOV.UK Service Manual and on one of the government's own AI experiments.

What a prototype is for

The Service Manual describes prototypes as a way to explore, share and test designs before committing to build anything. Teams can make several quickly and throw away the ones that do not test well. A prototype can be a pen and paper sketch or code that works like a real website. Sketches help colleagues discuss basic ideas. Code prototypes are better for research with users, because they feel more real.

Test the riskiest assumption first

The Service Manual's guide to the alpha phase asks teams to find their riskiest assumptions and test those first. A team does the minimum it needs to test them, often just the hardest part of the journey. Alphas tend to last between 6 and 8 weeks. By the end, the team decides which ideas are worth taking forward.

In one of the government's own AI experiments, GOV.UK Chat, the big open question was the quality of the answers.

What GOV.UK Chat learned

The Government Digital Service built GOV.UK Chat to answer questions using content from GOV.UK. The team ran it as a series of experiments, each lasting a couple of weeks. Content designers and other experts checked the answers for accuracy and completeness. Hundreds of users tried it in a controlled setting.

Nearly 70% of users found its answers useful. The system also occasionally gave hallucinated answers. Its answers were accurate 80% of the time, and the team concluded it needed to improve before going on the site. The team found its manual checking would not scale, and planned a bank of quality-checked questions to test against.

Keep real data out of early versions

The government's AI Playbook tells its staff never to enter official information into public AI tools unless it is already public. It also warns against assuming that default settings are secure when experimenting. The GOV.UK Chat team carried out a data protection impact assessment and removed personal data from the pages the model could see. Security experts also tried to break the system to find its weak points.

Expect to throw the code away

The Service Manual says prototype code does not need to meet production standards. It does not have to be secure or handle heavy traffic. For that reason, teams should not copy prototype code straight into a live service. A prototype published on the web needs a password, so that members of the public do not mistake it for a real service.

Useful links