Certification & Testing | Item Writing Automation

Automating Certification Exam Writing at More Than 2x Human Speed

2.5x

Faster than AWS's fastest human item writer

97%

Field pass yield (goal: 80%)

100%

Knowledge domain coverage (goal: 50%)

Prompts per exam item

The Challenge

Certification programs need a steady pipeline of new exam items, each grounded in the right knowledge domain and reference material. Written by hand, that work happens one item at a time, by subject matter experts (SMEs) who are already stretched thin. At AWS, that meant an SME drafting, revising, and proofreading each item alone, catching subtle issues like an answer option that gives itself away just by being longer or more precisely worded than the others. Scaling that process to cover a whole certification exam, every knowledge domain, not just the easy ones, meant even more manual, parallel effort.

The Solution

In January 2026, AWS piloted AWStin, a Betty-powered assistant built into its exam-item development workspace, on its Welding Fundamentals certification exam. SMEs select a knowledge domain pulled straight from AWS’s own test specification, point AWStin at the references that matter most, and set how many draft items they want back, then refine each one with plain-language follow-up prompts. Instead of bogging down subject matter experts with creating items from scratch, AWS has empowered them to focus on item review and other exam development steps that make better use of their talents.

Key Elements of Success

Faster than the fastest human writer. AWStin drafted review-ready items at 12.5 per hour, more than double the 5 per hour AWS’s fastest human item writer could produce on their own.
Minimal back-and-forth. SMEs needed an average of just one follow-up prompt per item, often to fix option length, a common giveaway sign, addressing a top frustration for AWS’s volunteer item writers.
Strong first pass yield. 97% of items met AWS’s own item-writing rules and industry standards on the first pass, well above the pilot’s 80% goal.
Full knowledge domain coverage. 100% of knowledge domains were covered, not just the easy ones, a first step toward generating complete, ready-to-use exam forms.

The Results

AWS set clear goals before its January 2026 pilot. AWStin beat every one of them.

Metric	Goal	Actual
Productivity (items drafted per hour)	10 (2x fastest human writer, 5/hr)	12.5
User involvement (prompts per item)	5 or fewer	~2
First pass yield	80%	97%
Knowledge domain coverage	50%	100%

A workspace built for real item writing

Structured inputs, not blank prompts. Domain fields pulled straight from AWS’s test specification, reference fields pointed to the standards that mattered most, an item-quantity field let SMEs generate a few drafts and pick the best, and a special-instructions field let SMEs add real-world context, like a specific industry or shop environment, to a knowledge domain.
A quality issue, caught and fixed. The pilot surfaced one persistent issue: an answer option that was longer or more precisely worded than the others, a common giveaway. That was behind most follow-up prompts, and AWS has already begun resolving it.