Assessment
Please navigate to this password protected page for more detailed information about creating new questions or editing existing questions in BenWare. You will need to log in with your NETID and password.
What are your assessment goals? Heading link
What are your assessment goals?
What are your assessment goals? Heading link
Assessment for Learning (AfL)
- Inform decisions about current teaching
- Opportunities for interaction and feedback
Assessment as Learning (AaL)
- Evidence-based learning strategies
- Self-regulation and metacognition
Assessment of Learning (AoL)
- Data to confirm student outcomes
- Opportunities for students to demonstrate proficiency
Best Practices for Assessment Heading link
Open each of the sections below to learn more about writing NBME-style questions, writing effective goals and learning objectives, and more.
Writing LO's and Goals
Bloom’s Taxonomy was developed and introduced in the 1956 by educational psychologist Benjamin Bloom. It was the first time educators were introduced to a formal classification of educational objectives and to the theory of mastery learning. The image you see here is the original taxonomy.
LO's and Goals Part II
This is the revised taxonomy for the cognitive domain developed by Anderson & Kraftwohl in 2001.
We usually focus on the first three rungs of the taxonomy:
- Remember
- Understand
- Apply
The New Bloom’s Taxonomy, as some call it, is a way to define the type of knowledge or skill that is to be learned. Ideally, we want students to be learning at the higher levels of the taxonomy. These include: application, analysis, evaluation, and creation.
A key concept in developing objectives is to most closely align them with the kinds of things our future physicians will be doing in practice.
The table below will give you some insight into how to craft a learning objective using key action verbs.
Important to Keep in Mind
Consider including SMART when writing objectives.
Specific
What do you want students to be able to do?
Measurable
How will you know if it’s accomplished?
Achievable
How can students achieve it? Is it realistic?
Relevant
How is it applicable?
Time-bound
By when should students be able to do it?
LO's and Goals Part III
| Level | Verbs | Examples for Usage |
|---|---|---|
| Remember | define, describe, identify, label, list, match, name, outline, recall, recognize, reproduce, select, state |
|
| Understand | categorize, clarify, classify, convert, compare / contrast, defend, distinguish, estimate, explain, extrapolate, generalize, illustrate, infer, interpolate, interpret, paraphrase, translate |
|
| Apply | apply, change, compute, construct, demonstrate, execute, implement, manipulate, operate, predict, prepare, produce, show, solve |
|
| Analyze | analyze, break down, compare, contrast, deconstruct, diagram, differentiate, discriminate, distinguish, illustrate, integrate, map, organize, parse, structure |
|
| Evaluate | appraise, compare, conclude, contrast, critique, defend, evaluate, explain, interpret, justify, support, test |
|
| Create | categorize, combine, compile, compose, create, devise, design, explain, generate, hypothesize, modify, organize, plan, produce, rearrange, reconstruct, relate, reorganize, revise, rewrite, summarize, tell, write |
|
Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom’s Taxonomy of educational objectives: Complete edition, New York: Longman.
Writing NBME Style MCQs
Writing excellent NBME-style questions is important so our students have experience answering these types of questions before they take their step exams. The NBME has a nice tutorial that you can take that guides you through the process: https://www2.nbme.org/nbme/IWTutorial/intro/index.html. In addition, check out the attached resources (at the bottom) such as the “gold book” to learn more about writing NBME-style questions.
Mindful Language for Question Development
Something else we need to think about is using mindful language when writing or editing questions. The NBOME has a short guide on appropriate language. As it says in the guide, language is evolving quickly! This means we might need to review our questions more often to ensure we are using appropriate language. Sometimes patient characteristics (e.g., age, sex, gender identity, disability, socioeconomic status, native language, country of origin, etc.) are mentioned in case vignettes. A best practice is to measure meaningful and plausible testing points without the influence of assumptions, bias, or stereotypes. Consider the following (from the NBME writing guide) regarding including patient characteristics:
- Is it clinically relevant and / or can it aid in the quality of distractors?
- Is it necessary to better understand the context in which the patient is being seen; in other words, would the assessment item be unreasonably difficult if the characteristics were excluded?
- Do the characteristics add to the overall representativeness of the referenced patient population?
- Consider inclusion of characteristics as long as they do not contain negative stereotypes or potential for bias.
Exam Performance
When we review exam performance, we look at the discrimination and the difficulty of how questions performed on the exam and over time.
- Discrimination – indicates how well a question differentiates between students who know the subject matter those who do not. A question is a good discriminator when students who answer the question correctly also do well on the test. Values can range from -1.0 to 1.0 and are calculated using the Pearson Correlation Coefficient. A discrimination value of less than 0.1 or that is negative indicates that the question might need review. Discrimination values cannot be calculated when the question’s difficulty score is 100% or when all students receive the same score on a question.
- Difficulty – the percentage of students who answered the question correctly. Difficulty values can range from 0% to 100%, with a high percentage indicating that the question was easy. Questions in the Easy (greater than 80%) or Hard (less than 30%) categories might need review.
Check out this text from Rachel Yudkowsky, MD, Yoon Soo Park, PhD, and Steven Downing, MD on assessment in higher ed medical institutions:
Yudkowsky, R., Park, Y-S., & Downing, S. M. (2020). Assessment in health professions education. (2nd ed.). New York: Routledge.
See the links below for some information on writing NBME-style questions.