Policy and Procedure for the Recording of Stations at the Fellowship Clinical Examination (OSCE)

TA689 V4

Document Review

Timeframe for review: Every two (2) years, or earlier if required

Document authorisation: Council of Education

Document implementation: Examinations Committee

Document maintenance: Executive Director, Education

Revision History

Version	Date	Pages revised / Brief Explanation of Revision
v1	Feb-2019	Approved by COE
v2	Nov-2021	Remote marking provisions and other updates
v3	Jan-2024	Remove references to specific examination venues, add content re support person.
v4	Mar-2024	Amend content re support person

Copyright

1. Purpose

The purpose of this document is to outline policy and procedure(s) relating to the recording of stations at the conduct of the Fellowship Clinical Examination (OSCE; the examination) of the Australasian College for Emergency Medicine (ACEM; the College) and the permitted use of those recordings.

Recording is primarily undertaken for College internal quality assurance and continuous improvement of processes relating to the OSCE, but may be used for other purposes in special circumstances.

This document has been prepared as a guide to all stakeholders involved in the conduct of the examination, including candidates and their supporters, Directors of Emergency Medicine Training (DEMTs), examiners, examination writers, standard setters, members of the Board, the Council of Education (COE), the Examinations Committee (ExC), examinations working groups and other key personnel, including ACEM staff.

For the avoidance of doubt, this document applies solely to the recording of stations at the conduct of the OSCE.

2. Definitions

ACEM; the College

means the Australasian College for Emergency Medicine.

Candidate

means any trainee or SIMG who enrolls in and attempts the Fellowship Clinical Examination (OSCE).

Member

means a person admitted as a member of the College pursuant to the provisions of the ACEM Constitution and associated regulations, and, for the purposes of this policy, also includes any other person serving on any College entity or as a College representative.

OSCE; the examination

means the Fellowship Clinical Examination (OSCE).

Recording

means the recording of both video and associated audio components of an examination in real time in digital format.

Specialist International Medical Graduate (SIMG)

means an individual enrolled in the ACEM SIMG Pathway undertaking College requirements for the purpose of attaining eligibility for election to Fellowship of the College.

Trainee

means an individual enrolled in and undertaking the FACEM Training Program.

3. Practical aspects

3.1 The process of recording

From March 2019 , all OSCE stations will be recorded utilising the recording facilities that are available at the examination venue(s). All stations will be recorded at the time they are conducted, and the resultant recordings will be held in a secure format by the College, operated by and accessible only to authorised staff and officers of the College.

Although the facilities of OSCE venue(s) will be utilised for the initial recording of stations, recordings will be transferred to the College at the conclusion of each sitting of the examination and retained in accordance with the provisions of the ACEM Privacy Policy. No copy of any recording of any examination will be held at the OSCE venue(s), any other facility or service provider, or at any other location other than the ACEM office(s) following the conduct of any examination.

3.2 Legal and privacy matters

ACEM appreciates the importance of ensuring that all material supplied by members and others who have dealings with the College is treated in a manner that respects the rights of all individuals (including examiners and candidates) and which meets the requirements of all relevant privacy and related legislation. The College’s Privacy Policy may be accessed through the College website.

Accordingly, appropriate consent for the recording of all stations will be obtained from all participants involved in the conduct of examination stations (candidates, examiners, confederates, standardised patients and others, as identified).

Video footage that is not used within twelve (12) months of recording, will be either destroyed/deleted from College files, or retained for designated training or examiner calibration purposes.

Access to recordings of stations will not be available to trainees other than as expressly stated in this policy or as required through relevant legislation.

3.3 Examination processes

Every station of the OSCE is assessed by two (2) examiners marking independently. On occasion, there may be three (3) examiners present as part of the training of new examiners. Examiners are members of the ACEM Court of Examiners and are appointed by COE following consideration of applications received by ExC, following a call to all College Fellows inviting applications. Examiners undergo initial and ongoing training as part of the requirements of their role and, on each day of the examination, attend a briefing in relation to the OSCE, and participate in station workshopping and calibration of the stations they examine.

Examiner performance at the OSCE is monitored by Peer Support Examiners (PSEs) and examiners are further audited and calibrated across all OSCEs they examine. PSEs are examiners of more than five (5) years’ experience who have been appointed to this senior role within the Court of Examiners by COE on the recommendation of ExC, to provide leadership, training, and support to members of the Court of Examiners.

The ACEM Procedure for the recruitment, training and appraisal of OSCE non-examiner personnel outlines procedures undertaken for the recruitment, training and appraisal of non-examiner personnel at the OSCE to ensure the maintenance of high standards of consistency across the stations and the candidate cohort. These non-examiner personnel include simulated patients, as well as FACEM confederates, nurses, technicians, and medical students who may be used for roles where specific medical knowledge is required.

4. Use of recordings

The recording of each OSCE by the College allows for the review of the conduct of individual stations under specific circumstances. This is primarily for the purposes of internal quality assurance and continuous improvement, and review of candidates with a borderline result. Where necessary, recordings may also be used to facilitate remote marking by examiners. Also, in some special circumstances, approved in accordance with this policy, a recording may be used where a candidate is of the view that an error has occurred in the conduct of a station(s) that has resulted in them being disadvantaged on that station(s).

The recordings are not available to be used as a vehicle for reconsideration of a candidate’s marks in an examination where, for example, a candidate is of the view that they should have passed that sitting of the examination and are seeking to have their examination ‘remarked’. This is not available under the College’s Reconsideration, Review and Appeals Policy (COR355), and this document should be consulted prior to any submission in relation to this issue.

In summary, recordings of OSCE stations may only be utilised under the circumstances outlined below.

4.1 Resource for training purposes

Recorded OSCE stations have the potential to provide relevant and valuable source material for the calibration, education and training of examiners and other station personnel. Any use for this purpose will be approved by the College CEO (or their nominee) or in accordance with guidelines approved for this purpose.

4.2 Quality assurance and continuous improvement

Recorded OSCE stations have the potential to enable the College to improve future examinations through the review of stations for the purposes of designing new stations or revising stations for use in future examinations. Recorded stations are also of use in enabling the College to audit the performance of examiners or any other individuals involved in the conduct of examinations. The College regards these activities as being in the best interests of all involved in the examination and will not, as a matter of course, seek further consent from individuals for the purposes of the use of station recordings for audit and/or quality improvement processes. Any use for this purpose will be approved by the College CEO (or their nominee) or in accordance with the guidelines approved for this purpose.

4.3 Review of results of candidates with a borderline result

As part of the College’s processes, and prior to the finalisation and release of examination results, where a candidate’s score for an examination was such that they:

Achieved a score that is below, but within 1 SEM, of the examination passing score; and
Have at least one station in which the mean of the differences between the domain scores for examiner pairs for the station(s) concerned, (i.e., the mean disparity of examiners’ scores across all the domains for that station) is greater than or equal to 1.

A review of the recording of the relevant station(s) will be undertaken by a senior examiner nominated by ExC who, where possible, is familiar with the station, however, did not examine the candidate in question.

The reviewing examiner will:

View the recording in consideration of the requirements of the station, and
Remark the station domains.

Note: Reviewing examiners will not have knowledge of the scores given by the original examiners.

When the marking for a station has been completed by the reviewing examiner, the candidate’s score for that station will be recalculated using the mean of the two (2) highest scores (of the two (2) original examiner scores and the reviewing examiner’s score) for each domain.

The candidate’s total OSCE score and examination result will be recalculated using the revised station score/s.

Reviewing examiners will not be required to give a Global Rating for the purposes of Standard Setting, as examiners’ global ratings are reviewed for all candidates prior to the calculation of the examination raw cut score, and where a disparity is found between examiners’ global ratings for a station, these are adjusted to ensure no candidate is disadvantaged by such disparity. The procedure for the review and adjustment is described in the ACEM Procedure for Determining Minimally Competent Criteria for FEx OSCE 2019.

In order to mitigate against possible bias regarding stations being reviewed as part of this process, examiners may be asked to review stations for the purposes of routine quality assurance and/or calibration, in addition to the purpose of review of candidate scores. To that end, examiners will not be made aware of the purpose of the review, for the stations to which they have been assigned.

As part of College quality improvement activities, records will be kept in relation to the number of such reviews undertaken and their outcome, and this information will be reported routinely to ExC and COE.

4.4 Review for candidate feedback after a third unsuccessful attempt

ACEM Regulation B, and Regulation G, permit candidates a maximum of four (4) attempts at the OSCE.

For the purpose of assisting trainees who have attempted and been unsuccessful at the OSCE on three (3) occasions, candidates may apply to view the recorded performance of their most recent attempt strictly in accordance with the provisions of this policy.

Candidates may apply to view their performance at stations under the following conditions:

Applications may be submitted only after the publication of candidate results, candidate feedback letters and the examination report.
A fee aligned to cost recovery will be payable at the time of application.
Candidates will be permitted to view the recorded stations at a venue, date and time as determined by the College.
When viewing the recorded stations, candidates must be accompanied by their DEMT, Supervisor or another FACEM support person (see section 5.) approved by the ExC Chair or their nominated representative.
A College staff member will be in attendance when the recorded stations are viewed.
Duly signed and completed confidentiality agreements from both the trainee and their support person must accompany the application.
Such viewing may not be used as the basis for any challenge to the result or outcome of the stations viewed.

4.5 For review in the event of a candidate complaint about the conduct of a station

Where an adverse incident is alleged to have occurred during the conduct of a particular OSCE station, such that a candidate believes they suffered a disadvantage, that candidate may submit a request within three (3) business days of the relevant examination as per the provisions of the ACEM Exceptional Circumstances and Special Consideration Policy (TA79), for the conduct of that station to be reviewed.

Applications must give a full account of the circumstances leading to the application and the basis on which the application is made.
The application will be reviewed by the ExC Chair or nominated ExC member(s), and, if accepted, they shall authorise a review of the recording of that station. Applications will only be accepted if it is alleged that there were adverse or unusual conduct or circumstances occurring at that station which affected the outcome, and not simply a challenge of the marking or assessment.
Where an application is accepted, an independent review of the recording of the station will be conducted by two (2) examiners nominated by the ExC Chair. Where feasible, the reviewing examiners will be examiners who are familiar with the station, either as a reviewer or examiner, but who did not examine that candidate in that station.
After reviewing the station, the reviewers will make a recommendation as to whether the ground(s) on which the application is made are valid and make a recommendation as to whether the application is to be upheld or dismissed. The Chair of ExC or nominated ExC member(s) will determine whether the recommendation is approved.
If the application is upheld, the station will be removed from the calculation of the candidate’s OSCE result when examination results are being compiled. If the application is not upheld, no further challenge to the result or outcome may be made. The College’s Reconsideration, Review and Appeals Policy will not apply to this decision.

4.6 Reconsideration, review and appeals

Where a specific allegation is made in relation to a matter(s) occurring at a station, which may have adversely affected the result or outcome of that station, and where that allegation is accepted for progress according to the College’s Reconsideration, Review and Appeals Policy:

ACEM, acting through the College CEO, may, in its discretion, provide a transcript derived from the recording of that station(s) for the purposes of any reconsideration, review or appeal relevant to that allegation.
The person conducting (or Chair of the committee conducting) the reconsideration, review or appeal may request that the College provide a transcript derived from the recording of the station, or in exceptional circumstances, may request a viewing of the recording for the purposes of the reconsideration, review or appeal by that person (or committee, as the case may be).

In each case above, where the transcript or recording has been provided to an individual (or committee as the case may be) to assist in the consideration of a reconsideration, review or appeal, the transcript may be provided to, or the viewing may be permitted for, the candidate/applicant. Where a viewing of a recording is permitted, it will be conducted under the following conditions:

Candidates/applicants will be permitted to view the recorded station in question, at a venue, and at a date and time as determined by the College.
When viewing the recorded station, candidates/applicants may be accompanied by a support person (see section 5.) or representative assisting in their application for reconsideration, review or appeal.
A College staff member will be in attendance when the recorded station is viewed.
Duly signed and completed confidentiality agreements from both the candidate/applicant and their support person must be submitted prior to the viewing.

Records will be kept in relation to the number of applications received under this section of the policy, the nature of the application and the outcome of the application. This information will be reported routinely to ExC and COE, and a record retained by the Office of the CEO for compilation and reporting as part of College processes.

4.7 Remote marking

Where circumstances exist that preclude onsite marking for one or more examiners in a station, recordings of the station(s) may be used by appointed examiners to complete the marking of the station. This may occur concurrently or after the examination.

5. Support person

The College is committed to providing training programs, including associated assessments, that include appropriate support structures that are fair, consistent, transparent and robust. ACEM acknowledges the importance of the safety and wellbeing of trainees and SIMGs, and individuals whose role it is to support trainees and SIMGs. The following provides guidelines for the involvement and management of a support person before, during or after the review of an OSCE station(s) for the purposes described in section 4 of this policy.

5.1 Circumstances for requiring a support person

The College recognises the benefit the presence of a support person may afford candidates when reviewing OSCE recordings and supports their appropriate use as a measure that seeks to increase the learning that results from the review session.

Circumstances in which a candidate may consider that they require the presence of a support person may include, but are not limited to, the following:

It is anticipated that negative feedback will be given in a review session.
The candidate finds receiving feedback stressful.
The OSCE was traumatic or distressing for the candidate.
The candidate feels more comfortable with another person being present at the review session.

5.2 Role of support person

The support person is in attendance to support the candidate, rather than to advocate for them or speak on their behalf.

Support persons may:

Provide emotional support for the candidate.
Facilitate communication, as appropriate, including supporting the candidate to seek clarification about the process, questions asked, or responses given.
Take notes throughout the review session.
Provide advice to the candidate.
Request a break, either to speak to the candidate or to allow the candidate to compose themselves when they become distressed or emotional.
Be available to debrief after the review session.

Support persons may not:

Advocate or speak for the candidate.
Disrupt or interfere with proceedings.
Take notes of examination station content

Support may need to continue after the review session. Wellbeing resources for trainees and SIMGs can be found on the ACEM website at: https://acem.org.au/Content-Sources/Members/My-Wellbeing

5.3 Suitable support person

The following general guidelines should be considered by candidates when choosing a support person:

The support person must be over 18 years of age.
The support person should not be someone who holds a College role more senior to that of the reviewing examiners, or examiners involved in the stations being viewed.
The support person should demonstrate commitment to the feedback process and engage with the process in a constructive manner.

The support person may, but need not necessarily, be a candidate’s mentor.

5.4 Conduct of review session at which a support person is present

Prior to the review session, the candidate must:

Advise the College that they wish to have a support person present at the review session.
Provide information in relation to the support person’s name, contact details and relationship to the candidate.

The ExC Chair may request an alternative support person if the chosen person is deemed to be unsuitable.

The support person may participate in person, via electronic means (i.e. telephone or videoconference).

The reviewing examiners shall communicate the role of the support person at the start of the review session to all present. If the reviewing examiners find the support person is becoming overly involved in the review/feedback process, the reviewing examiners may suspend the session until the role of the support person is clarified or an alternative support person is nominated. In the case of the latter, the session may need to be re-scheduled.

6. Associated documents

ACEM Constitution
ACEM Regulations
Exceptional Circumstances and Special Consideration Policy (TA79)
Privacy Policy
Procedure for the recruitment, training and appraisal of OSCE non-examiner personnel
Reconsideration, Review and Appeals Policy (COR355)