Source Video (Third-Party Educational Media)
This lesson incorporates a publicly available video hosted on YouTube and created by Stephen Joy. The video is referenced for educational and instructional purposes in support of free, open-access EMS education. Full credit and copyright remain with the original creator.
Video:
Basic Item Analysis
Creator: Stephen Joy
Platform: YouTube
License: Standard YouTube License
Item Analysis in EMS Education
Overview
Item analysis is a systematic evaluation of assessment questions (items) to determine their quality, fairness, and contribution to the overall reliability of an exam. It ensures that each question validly measures the knowledge and judgment essential for safe clinical practice.
Purpose of Item Analysis
- Evaluate question performance: Assess difficulty, discrimination, and reliability.
- Identify flawed items: Detect ambiguous or misleading questions.
- Improve assessments: Refine future versions of exams for better accuracy.
- Ensure fairness: Maintain consistency across cohorts and demographic groups.
Key Metrics in Item Analysis
| Metric | Definition | Ideal Range |
|---|---|---|
| Difficulty Index (p-value) | Proportion of students who answered correctly. | 0.3 – 0.8 (Moderate) |
| Discrimination Index (D-value) | How well the item differentiates high and low performers. | ≥ 0.3 (Good) |
| Point-Biserial Correlation (rpbis) | Correlation between item score and total test score. | > 0.2 (Acceptable) |
| Distractor Efficiency | Performance of incorrect answer options. | Chosen by ≥ 5% of students |
| Reliability Coefficient (KR-20) | Overall test internal consistency. | ≥ 0.7 acceptable; 0.8 preferred |
1. Difficulty Index (p-value)
The difficulty index represents the percentage of students who answered an item correctly. A p-value of 0.70 means 70% of students were correct. p-values below 0.3 indicate excessive difficulty, while those above 0.8 may be too simple or contain obvious clues.
2. Discrimination Index (D-value)
This index identifies whether top performers consistently get the item right compared to lower performers. A D-value below 0.20 suggests the item does not effectively separate different levels of competence and should be revised.
3. Point-Biserial Correlation (rpbis)
This measures how well an item fits with the rest of the test. Negative values are a critical warning indicating a likely key error or a seriously flawed question where top students were misled.
✓ Distractor Efficiency
A good distractor should be selected by at least 5% of students. Distractors chosen by 0% are non-functional and should be replaced to improve the item's effectiveness.
Continuous Quality Improvement (CQI)
Regular item analysis supports programmatic self-studies and accreditation reports. It transforms exam data into actionable insight, ensuring written exams measure clinical judgment and field readiness.
Checklist for EMS Educators
| Question | Check |
|---|---|
| Are difficulties distributed across levels (easy, moderate, hard)? | [ ] |
| Do items discriminate between strong and weak learners? | [ ] |
| Are distractors functional and free of bias? | [ ] |
| Is overall test reliability ≥ 0.7? | [ ] |
References
- Allen, M. J., & Yen, W. M. (2002). Introduction to Measurement Theory. Waveland Press.
- NHTSA. (2021). National EMS Education Standards.
- Biggs, J. (1996). Enhancing teaching through constructive alignment. Higher Education, 32(3), 347–364.