How many people would get a 3/3 on the following questions without even watching a training video?

1)   Do you need a password?

Yes
No

2)   Should you give your password to a stranger?

Yes
No

3)   True or False: All passwords should be displayed in the open

True
False

What if 100 people were asked the following question on the local news, how many do you think would honestly say yes?

Have you ever had racist, sexist or ageist thoughts?

Maybe 1%? What about if another 100 were asked under complete and utter anonymity?

Think the number would jump up?

Each of these examples demonstrates a valuable point, ASKING QUESTIONS IS HARD. It’s not as easy as just slapping a question mark at the end of a sentence and calling it a day. There are millions –and I’m not exaggerating- of factors to keep in mind when making a test, making a survey, conducting an interview, taking a poll, or anything similar. Since some form of content retention is needed after training, lets focus on quizzes in this blog.

Challenges of Making a Quiz

What’s got 2 thumbs and took an ENTIRE graduate level course/and part of a graduate degree to just learn how to write a good series of questions?

Thanks right, this girl. But rather than put you through that, or more importantly rather than put me through that, I am going to focus on the top challenges of making an effective quiz.

(If you want to know more about any of the other forms of questions/surveys/polls/etc. feel free to ask)

1-    Being too easy-The goal of a quiz is to evaluate an individual’s comprehension, or even mastery of the topic at hand. Sometimes we even use this in annual training as a criterion for taking the next lesson. Because of this, making the quiz questions too easy is not only useless but also damages the overall training efforts. The previous phishing ‘quiz’ is a perfect example of questions that are too easy. Each question is a no brainer, ‘no duh’ question that does not require any learning. Therefore, users can just skip to the quizzes and be finished with your 25 video annual training in 10 minutes.

Yeah, lots of learning there.

Not only does this not evaluate their comprehension of the topic, but also renders the rest of your training efforts, and the information in it, completely useless. You have just made the one time a year that they have to pay a little attention into a wash. The quiz sucks, and now you need to find another way to get them new information so that your enterprise is not made vulnerable with attacks like the Nigerian Phishing Scam.

2-    Being too hard- Just like making a quiz too easy is counter productive, the same is true when the quiz is too hard. When aquiz is impossible to pass users will first spend loads of time trying to complete your training –not great when you are paying them to do so. Once learned helplessness settles in users will start to give up rendering your training message useless.

3-    Getting actionable results- Even though quizzes are made to evaluate a users performance, they also tell the trainers/teachers/managers something as well. If evaluated correctly you can see where there is large levels of misunderstanding, or needed improvement. For example, if you notice that 75% of the users got a 20% or less on their first attempt at a quiz on cloud computing, that tells you that supplemental efforts need to be made to close that gap. Make a newsletter. Start that security awareness campaign sooner rather than later. Regardless, structure your quiz so that you, and your enterprise, can evaluate the user knowledge and adapt accordingly.

Welcome to Fantasyland where the budget is limitless and the users pay attention to everything you say!

In Fantasyland you have amazing annual training that lays a solid foundation of information for your users. You have created testing that accurately and effectively measures user understanding of the training without being too hard or too easy. You have created additional content (e.g., posters, viral videos, newsletters, lunch and learns) that calls back to the concepts taught in training and changes user behavior. You have done it all.

So how do you implement this amazing content?

All-at-Once?

Imagine that every year your user comes to a room that is plastered with your amazing posters. They sit down at a computer and watch training videos on topics like ‘secure cloud computing.’ This is followed by a quiz, followed again by a wonderfully crafted newsletter you created on how to ensure that all data in the cloud is safe. It all ends with showing them a funny viral video involving cats, Megan Fox, or David Hasselhoff. Since we know they fully attended to all that information –remember this is Fantasyland- how long do you think their behavior will be affected by the training?

1 week? 1 month? 1 year?

Considering that most annual awareness training programs contain at least 20 topics -all needing a video, quiz, poster, and additional content- I’d give it 2 weeks. Maybe 6 weeks for the topics that really resonated with them (e.g., Protecting your family on Facebook). That’s right, not even 2 months after presenting all this content most of it will be gone until next year pointing out an important part of any security awareness architecture.

Immediate v. Delayed Stimulation

In the previous example, all of the content was set up as immediate stimulation. The user was presented with all information at once and did not see it again until a year later. While this does get all of the information across, it does NOT produce consistent behavior change across the entire year. To do this you have to use a mixture of immediate and delayed stimulation. By combining the two techniques you are able to lay a solid foundation of awareness that is consistently recalled by the user throughout the year. If done correctly, you can even manipulate what is recalled based on what is presenting the most vulnerability within your organization at the time.

When to Implement Different Types of Content

Annual Training- This type of content can include everything from basic videos on passwords that everyone has to watch, to more specific role-based training that targets the information to fit the tasks of the user (e.g., Data classification for all users with a clearance). Annual training is where the foundation of information is established and is essentially ‘ground zero.’ Considering the density of the information, as well as the time required by the user, annual training should only occur once a year. Some companies choose to spread it over the year, and that is fine. The main point is that there is little to no value of using annual training in a delayed stimulation capacity.

Content Testing- After seeing a video the user has this large body of information and it needs to be stored (see previous blogs on the process of memory storage). One way to facilitate retention is through immediate testing. This requires the user to recall the information that they just learned through the training video, use it to answer questions, and re-store it thereby strengthening the memory. Without this, the message is not strengthened and the literacy foundation is much weaker. Because of it’s placement immediately after the video, content testing is most effective as immediate stimulation.

Posters and Additional Content- Something probably painfully obvious as wrong in the previous example was the fact that the only exposure the user was getting to the posters and newsletters was immediate and in conjunction with training. I have never seen a client use posters and other additional content in an immediate stimulation fashion because it does no good. Each are intended to call the user back to the information in training, facilitate recollection, and encourage more secure behavior across the entire year. Showing everything all at once is like placing all your cards on the table. You have nothing left.

While timing of your content requires more finesse and thought, classifying each part as either an immediate or delayed stimulation tool is vital in figuring out exactly where everything goes.

Percent Retained =

Information acquired

 

 *100

Information presented