This thesis explores how large scale cyber exercises work in the 21st century, going in-depth on Exercise Cyber Shield, the Department of Defense’s largest unclassified cyber defense exercise run by the Army National Guard. It highlights why these cyber exercises are so relevant, going over several large scale cyber attacks that have occurred in the past year and the impact they caused. This research aims to illuminate the intricacies around cyber exercise assessment involving manual vs automated scoring systems; this is brought back to work on creating an automated scoring engine for Exercise Cyber Shield. This thesis provides an inside look behind the scenes of the operations of the largest unclassified cyber defense exercise in the United States, including conversations with the Exercise Officer-In-Charge of Cyber Shield as well as a cyber exercise expert working on assessment of Exercise Cyber Shield, and the research also includes information from past final reports for Cyber Shield. Issues that these large scale cyber exercises have faced over the years are brought to light, and attempts at solutions are discussed.