Assessment

Bloom’s Taxonomy was developed and introduced in the 1956 by educational psychologist Benjamin Bloom. It was the first time educators were introduced to a formal classification of educational objectives and to the theory of mastery learning. The image you see here is the original taxonomy.

This is the revised taxonomy for the cognitive domain developed by Anderson & Kraftwohl in 2001.

We usually focus on the first three rungs of the taxonomy:

• Remember
• Understand
• Apply

The New Bloom’s Taxonomy, as some call it, is a way to define the type of knowledge or skill that is to be learned.  Ideally, we want students to be learning at the higher levels of the taxonomy. These include: application, analysis, evaluation, and creation.

A key concept in developing objectives is to most closely align them with the kinds of things our future physicians will be doing in practice.

The table below will give you some insight into how to craft a learning objective using key action verbs.

Important to Keep in Mind

Consider including SMART when writing objectives.

Specific

What do you want students to be able to do?

Measurable

How will you know if it’s accomplished?

Achievable

How can students achieve it? Is it realistic?

Relevant

How is it applicable?

Time-bound

By when should students be able to do it?

Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom’s Taxonomy of educational objectives: Complete edition, New York: Longman.

Writing excellent NBME-style questions is important so our students have experience answering these types of questions before they take their step exams. The NBME has a nice tutorial that you can take that guides you through the process: https://www2.nbme.org/nbme/IWTutorial/intro/index.html. In addition, check out the attached resources (at the bottom) such as the “gold book” to learn more about writing NBME-style questions.

Mindful Language for Question Development

Something else we need to think about is using mindful language when writing or editing questions. The NBOME has a short guide on appropriate language. As it says in the guide, language is evolving quickly! This means we might need to review our questions more often to ensure we are using appropriate language. Sometimes patient characteristics (e.g., age, sex, gender identity, disability, socioeconomic status, native language, country of origin, etc.) are mentioned in case vignettes. A best practice is to measure meaningful and plausible testing points without the influence of assumptions, bias, or stereotypes. Consider the following (from the NBME writing guide) regarding including patient characteristics:

• Is it clinically relevant and / or can it aid in the quality of distractors?
• Is it necessary to better understand the context in which the patient is being seen; in other words, would the assessment item be unreasonably difficult if the characteristics were excluded?
• Do the characteristics add to the overall representativeness of the referenced patient population?
• Consider inclusion of characteristics as long as they do not contain negative stereotypes or potential for bias.

Exam Performance

When we review exam performance, we look at the discrimination and the difficulty of how questions performed on the exam and over time.

• Discrimination – indicates how well a question differentiates between students who know the subject matter those who do not. A question is a good discriminator when students who answer the question correctly also do well on the test. Values can range from -1.0 to 1.0 and are calculated using the Pearson Correlation Coefficient. A discrimination value of less than 0.1 or that is negative indicates that the question might need review. Discrimination values cannot be calculated when the question’s difficulty score is 100% or when all students receive the same score on a question.
• Difficulty – the percentage of students who answered the question correctly. Difficulty values can range from 0% to 100%, with a high percentage indicating that the question was easy. Questions in the Easy (greater than 80%) or Hard (less than 30%) categories might need review.

Check out this text from Rachel Yudkowsky, MD, Yoon Soo Park, PhD, and Steven Downing, MD on assessment in higher ed medical institutions:

Yudkowsky, R., Park, Y-S., & Downing, S. M. (2020). Assessment in health professions education. (2nd ed.). New York: Routledge.

See the links below for some information on writing NBME-style questions.

• NBME Question Style Writing Guide (Request Page)
• Infographic: 5 Steps for Multiple-Choice Question Writing
• Writing MCQ Checklist from the Office of Educational Affairs