Saeed Sadiq Hamid  ( Department of Medicine, Aga Khan University, Karachi, Pakistan. )
Tabassum Zehra  ( Department of Educational Development, Aga Khan University, Karachi, Pakistan )
Muhammad Tariq  ( Department of Educational Development and Medicine, Aga Khan University, Karachi, Pakistan. )
Azam Saeed Afzal  ( Department of Educational Development and Medicine, Aga Khan University, Karachi, Pakistan. )
Hashir Majid  ( Department of Medicine, Aga Khan University, Karachi, Pakistan. )
Erfan Hussain  ( Department of Medicine, Aga Khan University, Karachi, Pakistan. )

Objective: To assess the skill level of residents regarding central venous catheterisation insertion, and to assess the reliability of scores in a simulated situation.

Methods: The quasi-experimental study with pre- and post-test design was conducted from February to June 2013 at the Aga Khan University, Karachi, and comprised four workshops attended by residents. The workshops were video-recorded for feedback and self-assessment. At the end of the workshops, knowledge and procedural skills were assessed using a self-generated 38-item, task-specific instrument after ensuring its content validity. Data was analysed using SPSS 19.

Results: There were 40 residents in the sample. The self-generated instrument was reliable with Cronbach's alpha value 0.83 and inter-rater coefficient 0.79. There was significant improvement in the skills level post-intervention compared to the baselines mean values (p=0.001). The subjects were satisfied with the workshops, as indicated by a mean score of 8.83±1.367.

Conclusions: The workshops appeared to improve the central venous catheterisation insertion skills of the residents.

Keywords: Direct observation of procedural skills, Workplace-based assessment, Postgraduate medical education, Resident, Central venous catheterisation. (JPMA 72: 620; 2022)

 

 

DOI: https://doi.org/10.47391/JPMA.970

 

Introduction

 

Workplace-based assessment (WPA) has been used to increase the clinical confidence of medical residents by increasing the level of realism of the assessment and providing a mechanism for formative feedback.¹

Direct observation of procedural skills (DOPS) of residents by the faculty remains a vital component of assessment across specialties. Assessment through observation provides ongoing data on resident performance with actual patients.¹ Similarly, maintenance of professional competence is a lifelong process, and is motivated by a number of factors, including curiosity, self-identified gaps in knowledge, and the desire to provide the very best care to patients.²

DOPS focusses on observing and evaluating the procedural skills of postgraduate residents in a workplace setting. Central venous catheter (CVC) insertion is one of the commonly performed bedside procedures³ by Internal Medicine (IM), Nephrology, Critical Care, Pulmonology, Cardiology, Anaesthesia and Emergency Medicine residents who are part of the Aga Khan University (AKU) postgraduate medical education residency programmes. Literature reports a 15% rate of complications associated with the procedure that include arterial puncture, blood steam infection and pneumothorax.³ However, attention towards the insertion technique and appropriate training with increased procedural experience have shown to reduce errors.³

The importance of appropriate CVC insertion training is recognised by professional societies, accreditation bodies and hospitals with residency training programmes across the globe with competence requirements and standards in place. The Accreditation Council for Graduate Medical Education (ACGME) has also stipulated that IM residents must show basic proficiency in CVC insertion.⁴ The American Board of Internal Medicine stipulates that residents should demonstrate competence and safe performance of CVC insertion by means of assessments performed during residency training.⁵

Continuous, comprehensive and elaborative assessment and feedback are integral components of competency-based residency education with emphasis on WPA.⁶ However, there are many challenges to overcome for successful, sustainable assessment and feedback strategies. This system of education ultimately requires a demonstration that the resident is truly competent to progress in training or at least to the next phase of a professional career.⁷ The competency demonstration and assessment require development of a valid and reliable assessment tool using direct observation coupled with feedback. Self-assessment by residents and intensive faculty development are also essential.⁷ Simulation-based mastery learning (SBML) is used to teach clinical skills. However, in Anaesthesia, the utmost value of such practice is expected to be with the most intricate and invasive procedures, like airway management, epidural puncture for spinal anaesthesia, CVC and lumbar epidural.⁸

The use of innovative assessment tools to assess procedural competence of residents in the clinical environment is challenging, time-consuming and labour-intensive. Studies report successful teaching of CVC insertion on partial-task simulator designed mannequins. This also simplifies gathering validity-based evidence, which otherwise would require training of multiple experts.⁹ However, there is a dearth of studies to report assessment of the procedure through direct observation of a resident's performance on a mannequin and its effectiveness.

Residents at AKU Department of Medicine are inducted in six Medicine and Allied specialties, including IM, Gastroenterology, Neurology, Cardiology, Pulmonology and Nephrology. Currently there are no structured formative or summative system assessment methods being used to gauge the procedural skills of residents in any specialty. There is also an absence of set standards of performance over which to assess the residents' procedural skills progression in training. Each sub-specialty has its own system where residents are evaluated either on a monthly or quarterly basis on a generic rating form.

Sufficient evidence is thought to be essential in making a judgment about the proficiency of the learner in any procedural skill. Competence requires more than just technical skills, like humanistic and team competencies. Therefore, it was thought to introduce a modified version of DOPS in order to make a judgment of the humanistic, technical and team competencies.⁹ Despite the importance of safe practice and assessments during the training of residents, there is a dearth of literature on specific measurement tools to assess the procedure. There is a need for a locally developed tool that could test procedural skills in an objective manner.^10,11 The current study was planned to do that among residents performing CVC insertion in a simulated situation.

 

Subjects and Methods

 

The quasi-experimental study with pre- and post-test design was conducted from February to June 2013 at the AKU Department of Medicine, Karachi. Kirkpatrick's training evaluation model¹² was used as the theoretical framework. The reaction and learning levels of Kirkpatrick's model were evaluated related to the teaching and assessment of safe and accurate insertion of CVC as a pilot project in order to establish DOPS methodology. Ethical approval was obtained from the institutional ethics review committee. A project team of six content experts through an iterative process worked to develop a 38-item, task-specific DOPS instrument relevant to local and contextual needs based on literature and already in-use checklist for central line insertion at AKU. Focus group discussions (FGDs) were held with content experts from Pulmonary Care, IM, Nephrology and Critical Care who reviewed and modified the form based on evidence-based best practices with an amalgam of the principles of objective structured technical skills (OSATS) based on their extensive experience of performing and supervising the procedure. The instrument included components of technical skills, communication skills and feedback. The instrument was piloted on a group of residents for any possible revisions. The modified DOPS (mDOPS) instrument was further revised on the basis of the findings of the pilot study.

A faculty refresher workshop was conducted to build the capacity of the faculty involved in performing the procedure to assess the residents' performances on a mannequin through mDOPS.

Four workshops were conducted to assess the competence of residents in CVC insertion on a mannequin. The sample size estimation was based on expert judgment of the faculty members with the specific purpose of correct use of CVC insertion technique. Residents from IM, Nephrology, Pulmonology, Cardiology, Critical Care and Emergency Medicine were included using non-probability purposive sampling technique. Each workshop had 10 residents from level 2 to 5 after furnishing informed consent.

The residents went through a training process consisting of video demonstration, didactic lectures, faculty demonstrations on a mannequin followed by hands-on training process on mannequins and direct observation and assessment by expert faculty on mannequin with a questionnaire-based post-test. Knowledge and procedural skills were assessed while being video-recorded during the workshop for feedback and self-assessment. Residents' acceptability and feasibility of the process was evaluated via a questionnaire. In case a resident failed to perform adequately on a mannequin, s/he was required to go through the training again on the mannequin. Since this was a formative process and meant for learners' improvement in performance based on feedback, there was no effect on residents' evaluations.

Data was analysed using SPSS 19. Descriptive analysis was done for mDOPS data. Mean ± standard deviation was used for quantitative variables, while frequencies and percentages were used for qualitative variables. The mean change scores from baseline to post-intervention was compared using paired t-test. P<0.05 was considered significant.

Descriptive data analysis was done regarding the feasibility and satisfaction questionnaire. The generalisability theory was used to model the reliability of scores with different number of assessors.

 

Results

 

There were 40 residents in the sample. The mDOPS instrument was reliable with Cronbach's alpha value 0.83 and inter-rater coefficient 0.79. There was significant improvement in the skills level post-intervention compared to the baselines mean values; from 10.1±4.2 to 12.6±3.1 (p=0.001).

The subjects were satisfied with the workshops, as indicated by a mean score of 8.83±1.367 out of a maximum possible 10 (Figure-1, Table).

 

 

 

 

Residents agreed to commit over the next three years to improving and enhancing their CVC insertion skills through WPA (Figure-2).

 

 

 

Discussion

 

Professional practice requires effective clinical decision-making and patient safety measures. However, assessing these elements is difficult.¹³ Literature reports that a structured tool in assessing the observed residents' skills improves accuracy of observation.¹⁴ DOPS is now being used in a number of programmes across the globe to assess the junior residents' performance.¹⁵ In some training programmes it has been reported to replace the existing logbooks for the assessment of procedural skills.^16,17

Our findings showed that mDOPS improved the performance of the residents in a simulated situation. The residents were satisfied with the assessment of the procedure through direct observation and reported it to be effective in meeting patient safety, and in enhancing their confidence, and showed willingness to participate in the next phase of real-life setting and its implementation as an assessment tool. Furthermore, the generalisability coefficient¹⁸ shows that the differences detected between residents; knowledge component and skills demonstration component on the assessment tool were real differences. We were able to report the first two levels of Kirkpatrick's model of resident reaction and learning to provide the educational effectiveness of the training programme which is the internal criteria and most commonly assessed. Learning criteria are measures of the learning outcomes, assessed by 'pre' and 'post' knowledge tests, and skill demonstration in the training context. However, the results and behaviour criteria, which are external and focus on changes that occur after the training programme¹² will be assessed later by immediate post-training measures of performance in real-life patient encounters (Figure-3).

 

 

Literature reports conceptually relatively modest relationship between Kirkpatrick's learning criteria and behavioural criteria^19,20 which may be due to less opportunities provided for the learnt skills to be demonstrated in real-life context or ineffective observation. Therefore, enough opportunities need to be provided to the residents in order to collect and interpret behavioural data related to direct observation in order to report the behaviour and results criteria of Kirkpatrick's model of training evaluation.

The current study has limitations as residents from specialties, like Anaesthesia and General Surgery, where CVC insertion is carried out were not included.

After having assessed and certified residents' core knowledge and procedural skill in a simulated environment, competency on CVC Insertion will be assessed in the next phase of this ongoing study. There is a need to implement the simulation-based training for CVC insertion across specialties and include other speciality residents for credentialing the residents for safe practice in CVC insertion. Furthermore, the practice may be used for other residency programmes within the country to reduce complications related to CVC insertion.

 

Conclusion

 

Acquisition of skills requires both day-to-day learning opportunities and supervision of opportunities. Development of a reliable and valid instrument for assessing workplace-based performance of procedural skills is crucial. Adopting strengths of DOPS and OSATS was beneficial in assessing CVC insertion. The evidence-based mDOPS provided a standard for assessing the procedural skill of central line insertion in simulated and real-life situation which can be used across specialties. Further, mDOPS instruments could be developed for other procedures performed by IM residents.

 

Limitation: Submission was after 7 years due to unavoidable circumstances.

Disclaimer: The text has been presented as a project on Research Day at the AKU Department of Medicine.

Conflict of Interest: None.

Source of Funding: None.

 

References

 

1.       Wilkinson JR, Crossley JG, Wragg A, Mills P, Cowan G, Wade W. Implementing workplace-based assessment across the medical specialties in the United Kingdom. Med Educ 2008;42:364-73. doi: 10.1111/j.1365-2923.2008.03010.x.

2.       Ringsted C, Pallisgaard J, Østergaard D, Scherpbier A. The effect of in-training assessment on clinical confidence in postgraduate education. Med Educ 2004;38:1261-9. doi: 10.1111/j.1365-2929.2004.02018.x

3.       Huang GC, Newman LR, Schwartzstein RM, Clardy PF, Feller-Kopman D, Irish JT, et al. Procedural competence in internal medicine residents: validity of a central venous catheter insertion assessment instrument. Acad Med 2009;84:1127-34. doi: 10.1097/ACM.0b013e3181acf491.

4.       Goroll AH, Sirio C, Duffy FD, LeBlond RF, Alguire P, Blackwell TA, et al. A new model for accreditation of residency programs in internal medicine. Ann Intern Med 2004;140:902-9. doi: 10.7326/0003-4819-140-11-200406010-00012.

5.       Ramsey PG, Carline JD, Inui TS, Larson EB, LoGerfo JP, Wenrich MD. Predictive validity of certification by the American Board of Internal Medicine. Ann Intern Med 1989;110:719-26. doi: 10.7326/0003-4819-110-9-719.

6.       Holmboe ES, Sherbino J, Long DM, Swing SR, Frank JR. The role of assessment in competency-based medical education. Med Teach 2010;32:676-82. doi: 10.3109/0142159X.2010.500704.

7.       Iobst WF, Sherbino J, Cate OT, Richardson DL, Dath D, Swing SR, et al. Competency-based medical education in postgraduate medical education. Med Teach 2010;32:651-6. doi: 10.3109/0142159X.2010.500709.

8.       Fettes P. Skills Training for Novice Anaesthetists – Why Simulation Based Mastery Learning. [Online] 2020 [Cited 2021 August 04]. Available from URL: https://das.uk.com/files/2020/page/Why%20Mastery%20Learning%20final.pdf.

9.       Fleming GM, Mink RB, Hornik C, Emke AR, Green ML, Mason K, et al. Developing a Tool to Assess Placement of Central Venous Catheters in Pediatrics Patients. J Grad Med Educ 2016;8:346-52. doi: 10.4300/JGME-D-15-00365.1

10.     Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA 2009;302:1316-26. doi: 10.1001/jama.2009.1365

11.     Lindsell D. Changes in postgraduate medical education and training in clinical radiology. Biomed Imaging Interv J 2008;4:e19. doi: 10.2349/biij.4.1.e19.9.

12.     Praslova L. Adaptation of Kirkpatrick’s four level model of training criteria to assessment of learning outcomes and program evaluation in higher education. Educ Assess Eval Account 2010;22:215-25.

13.     Neufeld VR, Norman GR. Assessing clinical competence (Springer Series on Medical Education). New York, USA: Springer Pub Co; 1985.

14.     Noel GL, Herbers JE Jr, Caplow MP, Cooper GS, Pangaro LN, Harvey J. How well do internal medicine faculty members evaluate the clinical skills of residents? Ann Intern Med 1992;117:757-65. doi: 10.7326/0003-4819-117-9-757

15.     Shah SH, Pouladi A, Bahram Rezaei M, Farhadifar F, Khatibi R. Evaluation of the Effects of Direct Observation of Procedural Skills (DOPS) on Clinical Externship Students’Learning Level in Obstetrics Ward of Kurdistan University of Medical Sciences. J Med Educ 2009;13:29-33.

16.     Sidhu RS, Grober ED, Musselman LJ, Reznick RK. Assessing competency in surgery: where to begin? Surgery 2004;135:6-20. doi: 10.1016/s0039-6060(03)00154-5.

17.     Carr S. The Foundation Programme assessment tools: an opportunity to enhance feedback to trainees? Postgrad Med J 2006;82:576-9. doi: 10.1136/pgmj.2005.042366.

18.     Crossley J, Davies H, Humphris G, Jolly B. Generalisability: a key to unlock professional assessment. Med Educ 2002;36:972-8. doi: 10.1046/j.1365-2923.2002.01320.x.

19.     Alliger GM, Tannenbaum SI, Bennett Jr W, Traver H, Shotland A. A meta‐analysis of the relations among training criteria. Pers Psychol 1997;50:341-58.

20.     Arthur W Jr, Bennett W Jr, Edens PS, Bell ST. Effectiveness of training in organizations: a meta-analysis of design and evaluation features. J Appl Psychol 2003;88:234-45. doi: 10.1037/0021-9010.88.2.234