E-ISSN:2455-5436
P-ISSN:2456-9518
RNI:MPENG/2017/70870

Research Article

OSLER

Surgical Review: International Journal of Surgery, Trauma and Orthopedics

2020 Volume 6 Number 3 May-June
Publisherwww.medresearch.in

Comparative study of conventional assessment and OSLER conducted at the model clinical examination in a teaching hospital

G Prabhu S.1, Abraham G.2*, Malavika Nair L.3
DOI: https://doi.org/10.17511/ijoso.2020.i03.03

1 Satish G Prabhu, Professor and Unit Chief, Department of General Surgery, MOSC Medical College, Kochi, Kerala, India.

2* George Abraham, Associate Professor, Department of General Surgery, MOSC Medical College, Kochi, Kerala, India.

3 Lakshmi Malavika Nair, Department of General Surgery, MOSC Medical College, Kochi, Kerala, India.

Background: Assessment of competency of the long case at the end of the clinical posting is quite often subjective and the assessment need not always be foolproof. In this method objectivity, validity and reliabilityare lacking. The major drawback of this system is that the student is not observed by the examiners. To avoid these criticisms the Osler has been developed. It is a 10 item analytical record of the traditional long case to improve the objectivity, validity, and reliability of the traditional method. All the candidates were assessed by theexaminer for the same ten items. Though not perfect, Osler is envisaged as a solution for the ongoing assessment challenge. This study was undertaken as part of the thesis for Advanced Course in Medical Education-2018. Objectives: Comparison betweenthe conventional method and the OSLER at long case clinical examination. Materials and Methods: This is an educational interventional study conducted by the department of general surgery, MOSCMedical CollegeKolenchery on students of the 8th semester. The students were assessed by two examiners based on a commonly agreed clinical examination format. Data on scores by both groups were analyzed statistically. Result: The marks showed a correlation between the two methods ofassessment. However, the grade awarded by the examiners showed lesser correlation. However, a multicentric study by different examiners may be required to arrive at the final conclusion of the superiority of OSLER over the traditional method. Conclusion: The present study did not find a statistically significant difference between the marks awarded to the student by the conventional method and OSLER method.

Keywords: OSLER, Conventional method, Clinical examination

Corresponding Author How to Cite this Article To Browse
George Abraham, Associate Professor, Department of General Surgery, MOSC Medical College, Kochi, Kerala, Indiaa.
Email: 		Prabhu SG, Abraham G, Nair LM. Comparative study of conventional assessment and OSLER conducted at the model clinical examination in a teaching hospital. Surgical Review Int J Surg Trauma Orthoped. 2020;6(3):154-158.
Available From
https://surgical.medresearch.in/index.php/ijoso/article/view/178

Manuscript Received Review Round 1 Review Round 2 Review Round 3 Accepted
15-05-2020 25-05-2020 01-06-2020 06-06-2020
Conflict of Interest Funding Ethical Approval Plagiarism X-checker Note
No Nil Yes 13%

© 2020 by Satish G Prabhu, George Abraham, Lakshmi Malavika Nair and Published by Siddharth Health Research and Social Welfare Society. This is an Open Access article licensed under a Creative Commons Attribution 4.0 International License https://creativecommons.org/licenses/by/4.0/ unported [CC BY 4.0].

Surgical Review: International Journal of Surgery, Trauma and Orthopedics 2020;6(3)154
Prabhu S.G. et al: Comparative study of conventional assessment and OSLER

Introduction

Assessment of clinical competence especially long case assessment is quite often subjective and is a cause of concern to both the student and the examiner.In the traditional long case examination, the student spends about forty-five minutes with the patient, and often the examiner will not be present during this time. The student is then assessed by two examiners over a period of twenty minutes [1]. The time limit may increase or decrease depending upon the complexities of the case or due to various subjective factors like lack of time,the number of students awaiting their turn,etc which paves room for criticism. Practical examinations play a very important role in the certification of candidates before they can practice medicine.OSLER is a more structured and more objective method of assessment [1]. The criticisms are centered mostly on lack of objectivity, validity, and reliability.In the present scenario,the current study cannot do away with a major assessment tool like a long case examination. It is in this context OSLER is introduced as a novel method of long case assessment at model clinical examination. Objective Structured Long Examination Record is a ten-item analytical record of the traditional long case which attempts to improve the objectivity, validity, and reliability of existing practice.All candidates are assessedover 20minutes by the examiners on the same 10 items.The items in the record include pace and clarity of case presentation,communication skills, systematic approach,and establishment of case facts [3]. During these activities, the candidate‘s affective domain is also assessed. The Items evaluated in physical examination includes a systematic approach, examination technique, and establishment of the correct physical findings. The remaining 3 items include the construction of appropriate investigations in a logical sequence, appropriate management, and final clinical acumen. The later item helps to assess the candidates’ ability to identify and solve problems. The initial assessment is essentially criterion-referenced through P+, P, P- a system which is followed by a selection of an appropriate mark, each of which has its own written descriptive profile. Greater emphasis is now being placed on communication skills in medical schools [1]. Increasingly, the importance of identifying problem-solving ability is being recognized [2]. In a study conducted by Bhalerao in 2017, a questionnaire was devised for the evaluation of the OSLER by students and the faculty

[3]. Quality of performance testing was judged by the performer as per the questionnaire devised. The questionnaires were revised based on the feedback from the faculty. It was found that 75% of the students and 100% of the teachers agreed that the conduct of examination was fair,while 75% of students and 72% of teachers agreed that a wide area of knowledge was covered. 88% of the students and 86% of teachers agreed that the technique compensated when there were areas of weakness, and 80% of teachers agreed that it was well structured. Though not perfect the OSLER is envisaged as a solution to ongoing long case assessment challenges.

Materials and Methods

Study Setting:Department of General Surgery, MOSC Medical College Kolencherry, Kochi, Kerala, India

Aim:To compare the conventional method and the OSLER at the end posting clinical long case examination in the eighth-semester students

Study Population: Eighth-semester MBBS students from a medical college in Kerala

Sample Size: 100 students from the eighth semester

Sampling Method: Students were selected on the basis of their roll numbers sequentially from Roll no: 1 to 100.

Study Design: Educational interventional study

Study Period: 3 months (January 2019- March 2019)

Inclusion Criteria:Regular batch students of the eighth semester who have scored more than 50% at the session exams

Exclusion Criteria: Eighth-semester supplementary batch students and differently-abled students were excluded

Ethical Consideration: - The project was approved by the IEC with reference IEC/MOSCC/355/2019

Study Design: This was a non-randomized study carried in the department of GeneralSurgery in a tertiary rural medical college, after taking approval from the IEC, IRB and informed consent from the students of the eighth semester. 100 students were examined during the study. Two examiners assessed the candidates based on a commonly agreed clinical


Surgical Review: International Journal of Surgery, Trauma and Orthopedics 2020;6(3)155
Prabhu S.G. et al: Comparative study of conventional assessment and OSLER

examination format. All students appearing in the long case were given 60 minutes to take history and perform a physical examination of their respective patients. The recording of the difficulty of the case before the start of the examination was done by the examiners. Two examiners scored the performance of each student at the same time, using the OSLER scale and traditional method respectively. The ten-item in OSLER scale includes 4 items on history taking, 3 items on physical examination and one item each on the formation of appropriate investigation in a logical sequence, appropriate management, and clinical acumen. There were two groups of examiners. One of the examiners assessed the student by the conventional method and the other examiner by the OSLER format. Both the examiners independently rated the candidates and awarded marks as well as grades on the OSLER scale and traditional method respectively. The scores and the grades awarded by both the examiners were recorded and used only used for the purpose of this study. All candidates were assessed on the same 10 item scale in 20 minutes duration. The examiner asked each student to take some part of history and perform a specific part of the examination under direct observation, to assess technique and communication skills.

Statistical analysis: The data was entered into excel sheet Data and were analyzed statistically on SPSS 23. Inter-rater reliability was calculated by using the Intraclass Correlation Coefficient (ICC) as well as by Kappa and the paired t-test was also done.

Results

A total of 100 students took a long case examination. All of them were rated by both the examiners. Mean scores of traditional method examiner and Osler method examiner are given in Figure 1 whereas scores awarded by traditional examiners and OSLER examiners are shown in Figure-2.

surgery_178_01.jpg

Fig-1: Mean marks.

surgery_178_02.jpg

Fig-2: Scores awarded by traditional examiners and OSLER examiners.

Inter-rater reliability of the OSLER scale, using the Intraclass Correlation Coefficient (ICC) was found to be low (0.796) (p=.000). Inter-rater agreement as analyzed by Kappa statistic was found to be low (p=.497). Paired samples t-test was also carried out to analyses the difference between mean scores of traditional and OSLER examiners. The difference was found to be not significant.

Discussion

The OSLER has now been used for more than 10 years during which important data has emerged. It has long been recognized that awarding marks is unreliable [4]. Detailed information available following OSLER assessment has found serious defects in basic clinical skills. This has been noted in the post-undergraduate and postgraduate study [5]. Similar findings have been reported by Maguire and Rutler, 1976 [6], Wiener and Nathanson, 1976 [7],Chan Yan et al 1988 [8]. However, the present studyfound that the marks awarded by both the method tallied well although the grades did not show the same degree of correlation. In a recent Lancet commentary, it was stated “ OSLER seems to be a powerful tool for providing feedback and therefore has great potential to increase clinical competence [9]. Objectivity is enhanced by prior agreement of what is to be assessed. The three variables in along case are the patient, candidate, and the examiner. Ideally, the only variable should be the candidate. The OSLER addresses the problem to acertain extent. However such standardization will not be practical in the foreseeable future. This may be one of the reasons the current study did not find the results tallying with the other studies. Criticism about the validity of the traditional method of examination has been well recorded by highlighting the construct and content validity by increasing the number and history taking to be measured. Validity could be improved by OSLER (Van Thiel et al) [9]. Gleeson’s OSLER requires only partial observation of performance and is thus more feasible which was also practiced by our traditional


Surgical Review: International Journal of Surgery, Trauma and Orthopedics 2020;6(3)156
Prabhu S.G. et al: Comparative study of conventional assessment and OSLER

examiner also [10]. Case specificity is another matter of concern in a long case examination. Some students may get an ‘easier’ case than others. The OSLER provides for consideration of this factor [11]. The examiner assesses the difficulty level of the case beforehand. The difficulty level is determined by the number of problems that the case presents. If one problem needs to be resolved it represents a standard case, more than three problems would be very difficult Wilkinson et al have reported that case selection has minimal impact on reliability [12]. In the present study, patients were not standardized but the difficulty level was determined by examiners beforehand. In the current study possibly because both the methods of assessment were done by the examiners sitting together the validity of both the methods ofassessment appeared the same. However, it was found that results obtained by both the method were more or less the same because there was only one solution to a clinical problem.for ex swelling in front of the neck which moves with deglutition is understood to be thyroid swelling in both the method of assessment and naturally the candidate stands to score the same grade and marks in either method of assessment.

Limitations

AS the study conducted over a short period with small sample size,the findings cannot be taken as unequivocal supremacy of one system of examination over others. Hence this study needs to be validated by conducting the study in a larger sample size in multiple departments.

Conclusion

In the present study,the traditional method of long case examination and OSLER was not statistically significant to prove the superiority of OSLER over the traditional method of examination. However interdepartmental variation in the results obtained by OSLER and traditional case examination is likely as the clinical parameterfor assessment in general surgery is fixed and less subjective, unlike other non-surgical specialties. Comparative study between both the methods of assessment require single large size and involvement of other specialties to come to a definite conclusion.

What does this study addto the existing knowledge?

At present, the long case clinical examination is

highly subjective which affects the assessment of the student and this study was aimed at comparing the results of osler and the traditional method of assessment and prove the superiority of osler overthe traditional method of examination

Author’s contribution

Dr. Satish G. Prabhu: Principal investigator Dr. George Abraham: Co-investigator, manuscript preparation Dr. Lakshmi Malavika Nair: Junior resident in general surgery who coordinated with the Students.

Reference

  1. Gleeson F. AMEE Medical Education Guide No; 9 Assessment of clinical competence using the Objective Structured Long Examination Record (OSLER). Med Teach. 1997;19(1)7-14.
  2. Barrows HS, Feltovich PJ. The clinical reasoning process. Med Edu. 1987;21(2)86-91.
  3. Bhaleroo P. Perception of medical teachers and students regarding objective structured long examination record as assessment tool-A prospective study. Int J Res Med Sci. 2017;5(4)1679-1682.
  4. Wilson G, Lever R, Harden RM, Robertson JI, Macritchie J. Examination of clinical examiners. The Lancet. 1969;293(7584)37-40.
  5. Gleeson F. Defects in postgraduate clinical skills as revealed by the objective structured long examination record (OSLER). Ir Med J. 1992;85(1)11-14.
  6. Maguire GP, Rutter DR. History-taking for medical students. The Lancet. 1976;308(7985)556-558.
  7. Wiener S. Physical Examination- Frequently Observed Errors. JAMA. 1976;236(7)852-855.

Surgical Review: International Journal of Surgery, Trauma and Orthopedics 2020;6(3)157
Prabhu S.G. et al: Comparative study of conventional assessment and OSLER
  1. Chan-Yan C, Gillies JH, Ruedy J, Montaner JS, Marshall SA. Clinical skills of medical residents- a review of physical examination. CMAJ Can Med Assoc J, J Assoc Medicale Can. 1988;139(7)629-632.
  2. Thiel J, Kraan HF, Vleuten CPM. Reliability and feasibility of measuring medical interviewing skills- the revised Maastricht History-Taking and Advice Checklist. Med Educ. 1991;25(3)224-229.
  1. Nithyanandam S, Joseph M, Vasu U. Can conventional long case examination be improved?. Indian J Ophthalmol. 2012;60(4)333.
  2. Sood R. Long case examination-Can it be improved. J Indian Acad Clin Med. 2001;2(4)252-255.
  3. Wilkinson TJ, Campbell PJ, Judd SJ. Reliability of the long case. Med Edu. 2008;42(9)887-893.

Surgical Review: International Journal of Surgery, Trauma and Orthopedics 2020;6(3)158