Please navigate to this password protected page for more detailed information about creating new questions or editing existing questions in BenWare. You will need to log in with your NETID and password.
What are your assessment goals? Heading link
What are your assessment goals?
What are your assessment goals? Heading link
Assessment for Learning (AfL)
- Inform decisions about current teaching
- Opportunities for interaction and feedback
Assessment as Learning (AaL)
- Evidence-based learning strategies
- Self-regulation and metacognition
Assessment of Learning (AoL)
- Data to confirm student outcomes
- Opportunities for students to demonstrate proficiency
Best Practices for Assessment Heading link
Open each of the sections below to learn more about writing NBME-style questions, writing effective goals and learning objectives, and more.
Writing LO's and Goals
Bloom’s Taxonomy was developed and introduced in the 1956 by educational psychologist Benjamin Bloom. It was the first time educators were introduced to a formal classification of educational objectives and to the theory of mastery learning. The image you see here is the original taxonomy.
LO's and Goals Part II
This is the revised taxonomy for the cognitive domain developed by Anderson & Kraftwohl in 2001.
We usually focus on the first three rungs of the taxonomy:
The New Bloom’s Taxonomy, as some call it, is a way to define the type of knowledge or skill that is to be learned. Ideally, we want students to be learning at the higher levels of the taxonomy. These include: application, analysis, evaluation, and creation.
A key concept in developing objectives is to most closely align them with the kinds of things our future physicians will be doing in practice.
The table below will give you some insight into how to craft a learning objective using key action verbs.
Important to Keep in Mind
Consider including SMART when writing objectives.
What do you want students to be able to do?
How will you know if it’s accomplished?
How can students achieve it? Is it realistic?
How is it applicable?
By when should students be able to do it?
LO's and Goals Part III
Level Verbs Examples for Usage Remember define, describe, identify, label, list, match, name, outline, recall, recognize, reproduce, select, state
- Outline the steps of a procedure,
- List the organs,
- Recognize a disease,
- Name an instrument.
Understand categorize, clarify, classify, convert, compare / contrast, defend, distinguish, estimate, explain, extrapolate, generalize, illustrate, infer,
interpolate, interpret, paraphrase, translate
- Summarize medical condition of a patient,
- Interpret an MRI,
- Explain contributing factors.
Apply apply, change, compute, construct, demonstrate, execute, implement, manipulate, operate, predict, prepare, produce, show, solve
- Predict probability that patient will present diabetes,
- Operate an ultrasound machine,
- Relate the various factors in a differential diagnosis
- Perform the steps of a catheter procedure.
Analyze analyze, break down, compare, contrast, deconstruct, diagram, differentiate, discriminate, distinguish, illustrate, integrate, map, organize, parse, structure
- Diagram the key components of the patient's health ecosystem,
- Differentiate between three similar skin rashes,
- Diagram the brain stem,
- Break down the functions of the key parts of the community health infrastructure.
Evaluate appraise, compare, conclude, contrast, critique, defend, evaluate, explain, interpret, justify, support, test
- Assess the severity of a condition,
- Critique the clinical process and conclusions presented in a vignette,
- Interpret the various data and make a diagnosis,
- Critique the top three diagnoses listed below.
Create categorize, combine, compile, compose, create, devise, design, explain, generate, hypothesize, modify, organize, plan, produce, rearrange, reconstruct, relate, reorganize, revise, rewrite, summarize, tell, write
- Design a process for effectively screening for Alzheimer's,
- Generate the key factors leading to Alzheimers and explain the most effective treatments,
- Reconstruct a broken bone,
- Organize a patient's history from disjointed data and verbal accounts,
- Generate a # part differential diagnosis and a # part treatment plan.
Anderson, L. W., & Krathwohl, D. R. (Eds.). (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom’s Taxonomy of educational objectives: Complete edition, New York: Longman.
Writing NBME Style MCQs
Writing excellent NBME-style questions is important so our students have experience answering these types of questions before they take their step exams. The NBME has a nice tutorial that you can take that guides you through the process: https://www2.nbme.org/nbme/IWTutorial/intro/index.html. In addition, check out the attached resources (at the bottom) such as the “gold book” to learn more about writing NBME-style questions.
Mindful Language for Question Development
Something else we need to think about is using mindful language when writing or editing questions. The NBOME has a short guide on appropriate language. As it says in the guide, language is evolving quickly! This means we might need to review our questions more often to ensure we are using appropriate language.
When we review exam performance, we look at the discrimination and the difficulty of how questions performed on the exam and over time.
- Discrimination – indicates how well a question differentiates between students who know the subject matter those who do not. A question is a good discriminator when students who answer the question correctly also do well on the test. Values can range from -1.0 to 1.0 and are calculated using the Pearson Correlation Coefficient. A discrimination value of less than 0.1 or that is negative indicates that the question might need review. Discrimination values cannot be calculated when the question’s difficulty score is 100% or when all students receive the same score on a question.
- Difficulty – the percentage of students who answered the question correctly. Difficulty values can range from 0% to 100%, with a high percentage indicating that the question was easy. Questions in the Easy (greater than 80%) or Hard (less than 30%) categories might need review.
Check out this text from Rachel Yudkowsky, MD, Yoon Soo Park, PhD, and Steven Downing, MD on assessment in higher ed medical institutions:
Yudkowsky, R., Park, Y-S., & Downing, S. M. (2020). Assessment in health professions education. (2nd ed.). New York: Routledge.
See the links below for some information on writing NBME-style questions.