Posted by:
Mike Pollitt, eAssessment Specialist at risr/
Since its inception in the 1970s, the Objective Structured Clinical Examination (or OSCE) has been widely considered the gold standard for the assessment of clinical competence across medical and allied healthcare education programmes. However, OSCEs are notoriously difficult to administer. They require a large resource outlay, complex logistical management and, not to mention, traditionally about half a forest’s worth of paper.
risr/assess changes that by bringing the entire ecosystem into a single digital application.
Drawing on our decades of experience working directly with healthcare education organisations, we designed a solution that simplifies the OSCE design and delivery process for candidates, examiners, and exam managers alike. Users benefit from a full suite of tools to support the entire lifecycle of the assessment process; from authorship, blueprinting, and test design to multimodal exam delivery, built-in standard setting, psychometric analysis and candidate feedback.
Let’s delve further into how the solution transforms each stage of the examination process.
Question banking and station design
At the heart of risr/ assess is a secure, permission-segregated and version-controlled item bank, and this is where authors work and collaborate to produce station materials. A powerful, user-configurable metadata system allows items to be tagged and categorised with virtually any taxonomy to ensure robust content organisation and alignment. Whether your priority is to map to the GMC’s MLA content map, the USMLE content outline or a proprietary tagging structure, risr/assess supports you with multi-level hierarchies and conditional tagging approaches.
The OSCE item type itself is completely user-configurable, offering full support for checklist, domain-based, or hybrid approaches to mark sheet design. Mark sheets can accept both selectable values and open-ended text input (for examiner comments), and specific elements can be made mandatory where required.
Instructions for examiners, candidates, or simulated patients can be uploaded or built directly into the item. Furthermore, a secure resource repository handles additional digital artefacts – like images, ECGs, and even high-resolution DICOM images – that need to be attached to the station.
As you deliver your OSCEs using risr/ assess, a comprehensive usage log is maintained against each station alongside aggregated metrics, providing a longitudinal view of how your stations perform over time.
OSCE circuit preparation
There are two broad parts to the exam preparation process: compiling the clinical content and organising the logistics for the day. Both are managed directly within risr/assess.
Using templatable blueprints, test authors can ensure appropriate, consistent sampling of the item bank across different editions of an exam. Once selection parameters are specified, the system can even choose stations on your behalf (including the automatic exclusion of items used in recent administrations). The person compiling the exam can then review and refine as appropriate.
Whether your physical exam circuits are straightforward or highly convoluted, risr/assess supports both simple and complex formats. Exams can be set to run over several days with multiple sessions per day, with full support for parallel circuits. As candidate and examiner records are added into the exam, risr/assess automatically schedules each individual person interaction, including automatic provision for rest stations where necessary.
Exam Delivery
When it comes to the delivery of the exam, the solution offers two distinct deployment options: face-to-face and remote.
Face-to-face
Traditional face-to-face delivery is achieved using iPads, where station materials and mark sheets are presented to examiners in a side-by-side, split-screen view on the device. Examiners can then record their observations for each candidate interaction using the exact mark sheet designed during the authorship stage. Candidates are presented to examiners in the order in which they are expected to appear in the exam to keep the day moving like clockwork. But for additional peace of mind you can also generate candidate QR codes to further ensure that the correct scores are attributed to the correct individual. The iPad also features full offline resilience for those times when the internet connection might be unstable.
Remote
More recently, we have introduced a fully remote OSCE delivery option which affords all the benefits and efficiencies of the iPad-based exam, but without the people involved ever needing to be in the same physical location as one another. Candidates, examiners and role players join an integrated remote video meeting, meaning they can participate from anywhere in the world. The station materials are displayed directly in the examination interface with options for the examiner to release materials to candidates on demand and even to synchronise screens. The system will manage automated progression through the meetings, ensuring everyone is in the right place at the right time to complete the circuit, thus combining the ease of a video call with built-in, robust examination workflows.
There are huge benefits to collecting the scoring data in this way. Validated mark sheet fields mean no more missing data. There are no scanning errors like you might find in optical mark reading applications. Also the reduction in administrative time and effort collecting and scanning or transcribing all of that data makes life easier for everyone involved and with zero risk of human error.
For exam administrators, real-time dashboard views track mark sheet submission instantaneously during the exam, eliminating the need to run frantically between stations to check that paper forms have been completed properly.
Result processing and psychometrics
Once all of that scoring data has been collected, risr/ assess automates its processing into examination outcomes. Common standard-setting processes and algorithms are all built into the system with full support for borderline regression and borderline group methodologies (including automatically calculated regression plots) alongside criterion-referenced methods such as Angoff.
In addition to the baseline exam-level cut score, advanced passing criteria can also be added to the calculation. A conjunctive criterion (i.e. a minimum number of stations) can be set, as well as introducing dimensional passing criteria (e.g. candidates must achieve n% in station elements tagged to a specific category). Optionally, a standard error can be applied to the generated cut score where it’s needed, and there is full support for sequential examination strategies using this methodology.
The post-processing functionality is supported by a full suite of psychometric tools to ensure that the exam is performing as expected. Test and item-level reporting includes Classical Test Theory, Item Response Theory and Rasch Modelling techniques. Hawk/dove analysis is conducted using our examiner analysis report, and individualised and targeted feedback can be provided to your examiners as necessary.
Whilst we’re on the subject of feedback, the loop would, of course, not be complete without a mechanism to provide results and feedback to candidates. The highly configurable Candidate Feedback Report can provide both numerical feedback and verbatim examiner comments directly back to the candidate. Users have the option to configure the report to show or hide cohort averages, question-level feedback and even aggregated dimensional feedback (e.g. you scored n% in communication skills across all stations). As well being able to distribute the reports to candidates by email directly from the system, results and feedback can also be returned to the risr/ advance module for reflection and continuous development.
Hopefully this gives you some idea of the benefits and efficiencies that can be realised using the risr/assess solution to digitise your OSCEs. But don’t just take our word for it. Hear from some of our friends and colleagues at the organisations we work with, such as:
- The Dental School at the University of Bristol who reported a 116-hour reduction in administrative processing time.
- The University of Melbourne’s Veterinary School who radically improved and streamlined student feedback processes.
- Or the Midwifery school at the University of East Anglia who received incredibly positive feedback about the examiner and candidate experience.
So if you feel like you are ready to modernise your clinical assessments, talk to us today to find out about how risr/ can help your organisation deliver robust, defensible, and highly efficient digital OSCEs.
