Phishing remains one of the leading causes of cybersecurity breaches, succeeding by exploiting how people make decisions under pressure rather than by defeating technical defences. Awareness training has been shown to reduce susceptibility, but evaluation in this field focuses overwhelmingly on outcome accuracy and tells us little about which behavioural changes explain any improvement.

This project addresses that gap through Something’s Phishy, a web-based cybersecurity learning game in which players judge messages as safe or suspicious across three character-based difficulty tiers. The system is supported by a behavioural telemetry pipeline that captures dwell time, inspect usage, and link-click behaviour, alongside a contextual feedback system that combines a structured prompt for Google’s Gemini API with a hardcoded blueprint fallback. Message content is controlled through the blueprints, so that ground truth, cues, and difficulty remain researcher-defined while the prose is generated by the language model. A three-strand evaluation combined scripted playthroughs of three player archetypes for technical validation, a retrospective review of all 48 live LLM-generated feedback responses against four design criteria, and a usability study with seven peer participants.

The peer findings show why an accuracy-only evaluation would have been incomplete. Inspect rates rose steadily across the three tiers (38%, 50%, 62%) and link-clicking on suspicious messages was eliminated entirely by the final tier. The retrospective LLM review found 100% label accuracy, cue alignment, and behavioural responsiveness with zero hallucinations across the live responses reviewed. The findings support the use of behavioural telemetry as a research instrument in serious game evaluation, alongside conventional accuracy and self-report measures. They also identify the specific behavioural changes behind the improvement seen within the session.