Rukhsana W. Zuberi  ( Department of Community Health Sciences, The Aga Than University Medical Centre, Karachi. )
Naeem A. Jafarey  ( Department of Medical Education, College of Physicians and Surgeons of Pakistan, Karachi. )
Asma Fozia Qureshi  ( Department of Community Health Sciences, The Aga Khan University Medical Centre, Karachi. )
Fazal Elahi  ( Department of Medical Education, College of Physicians and Surgeons of Pakistan, Karachi. )

The Diploma in Family Medicine (DFM) Examination Is a new certification offered by the College of Physicians and Surgeons of Pakistan, and its Department of Medical Education designed a scientific examination. First, the Expert Advisory Committee for Family Medicine was formed, relevant training objectives Were determined, a training programme to achieve the objectives was designed and a valid syllabus was chosen. Then the examination was designed, where the candidates must pass the objective theory papers before taking the clinical examination. The clinical examination consisted of an Objective Structured Clinical Examination (OSCE) and traditional case presentations. The candidates had to pass each of the components, and attain an overall aggregate of 60%. In the first six examinations, 752 candidates sat for the theory examinations, 332(44.14%) were eligible for the clinical examination, and 170 (23%) passed. If 60% marks obtained in case presentations is taken as the gold standard which is the current CPSP policy and compared to OSCE marks, then 75% marks in OSCE had a sensitivity of 67% and a specificity of 79% (JPMA 43:217,1993).

People tend to follow traditional patterns. The patterns may remain unchanged for decades without anyone knowing the basis on which they were designed. With increased awareness of recent developments in educational (pedagogic) technology, particularly the development of rational basis for designing evaluation, some changes have been implemented in the postgraduate examinations of the College of Physicians and Surgeons of Pakistan (CPSP). These changes, how­ever, have been piecemeal and patchy. To upgrade the knowledge and skills of family physicians in order to meet the health care needs of Pakistan, a Diploma in Family Medicine was approved by the CPSP; in June, 1988. With the initiation of a new programme in Family Medicine the Department of Medi­cal Education of the College had the opportunity to devise an entirely new examination de novo. The ex­perience of designing this examination and the results of the first six examinations conducted along these lines are described in this article. The first step was formation of an expert advisory committee. The committee set competency based objec­tives, and determined an appropriate training programme and syllabus to meet the objectives. An objective theory and clinical examination was designed according to tables of specifications and using the Objective Structured Clinical Examination¹ (OSCE). OSCE was introduced for the first time in a certifying examination in Pakistan. The case presentation/discur­sive method of clinical examination was also part of evaluation, in order to have a traditional “gold standard” against which OSCE could be measured. Candidates were only eligible to take the clinical examination after passing the theory examinations with approximately 60 % marks.
The Expert Advisory Committee
The chairperson of the expert advisory committee on family medicine was a community paediatrician. The members included faculty from family medicine, three family practitioners of Pakistan, a professor of community medicine, a CPSP councilor and educational advisors from the Department of Medical Education of the CPSP. This was analogous to formation of test committees in National qualifying examinations in the United States of America².
Training Objectives
The training objectives were based on the am­bulatory health care needs and the workload of family physicians of Pakistan. The overall objectives stated that the family physicians should be able to:
(a) undertake initial management of all medical and surgical emergencies upto transportation to a hospital;
(b) manage commonly occurring diseases in the community which can be treated at home;
(c) refer early and appropriately those cases which cannot be managed by the primary care doctor, recogniz­ing limitations of primary care;
(d) prevent locally endemic diseases and promote health.
Training Programme
A very specific need based training programme for Pakistan was derived which would enable the family physicians to acquire the requisite knowledge and skills. On completion of the house job, a one year’s special supervised training was designed. Ideally, the years specialized training should be in family practice or other ambulatory care units. Due to a lack of trained super­visors/trainers in the field of family medicine, it was recommended that to begin with, ten months of this one year training should be in teaching hospitals and the remaining two months should be an elective period. As nearly 50% of our population is less than 16 years of age, and as nearly 60% of the patients presenting to the general practitioners are children, special emphasis was placed on maternal and child health. The elective could be in any of the traditional hospital disciplines and subspeciahies of medicine and surgery, or in a duly approved and supervised rural health centre or family practice clinic. As the formal training programme had only recently been devised, it had not been implemented. The training sites and trainers had not been identified. Therefore, a grandfather clause was recommended that graduates of five or more years of experience of family practice would be eligible to take the examination without training.
Syllabus
The syllabus was based on the problems commonly seen in family practice and includes presenting symptoms/problems as well as diseases rather than the traditional pathophysiological classifications seen in a text book (Table 1).

A short course, covering the main topics of the syllabus, is repeated annually, and includes a component of hands-on clinical training to practice and refine specific psychomotor skills critical to family prac­tice.
Criteria for Selection of Examiners
A criteria for selection of examiners was developed based on the patients to be examined. There were three groups of two examiners each, six examiners in all.
Group I: A community pediatrician anti a general practitioner for the paediatric case;
Group II: A general physician and a general surgeon for the adult male patient;
Group III: An Obstetrician/Gynecologist and a general practitioner to examine the candidates on the adult female patient.
As there was no postgraduate certification in Family Medicine before the Diploma Examination was designed, criteria for selection of General Practitioner Examiners were chosen and agreed upon by the Expert Advisory Committee (Table II).


Evaluation
Keeping in view most of the suggestions by Tinkel­man³, a table of specifications (Table III)

was determined for the examination. Common problems like respiratory and gastrointestinal problems, problems of com­municable diseases, pregnancy and birth were weighted more heavily than other less frequently occurring problems.
The theory examination was designed as a com­prehensive multidisciplinary examination with the cluster variety of true or false type of questions. In each of the six theory examinations, there were 750 true and false items, grouped as 150 questions with five items under each heading or stem. The questions were constructed by specialists and generalists working in the field, who had participated in the basic ‘Workshops on Education Planning and Evaluation” held by the Department of Education and National Teacher Training Centre, at CPSP. A large number of questions were elaborately formulated to be interpretive rather than recall⁴. They were reviewed to ensure that the questions meet the objectives, were appropriate for the level of the examina­tion, that the distractors were plausible and there was no ambiguity in the wording or statements in accordance with the recommefidations of Gronlund⁵. These questions were randomly allocated to two papers, each paper of two and a half hours duration. Only those candidates who achieved the previously estab­lished criteria of 60% or more in the theory examination were eligible to take the clinical examination. When the Diploma in Family Medicine Examination was started, it was the first time in Pakistan that a professional examination in medicine was entirely based on objective questions. The clinical examination was then conducted in two parts. The first part consisted of an Objective Structured Clinical Examinations (OSCE), which carried 50% of the clinical examination marks, and the second part consisted of the traditional clinical case presenta­tions to examiners, which carried the remaining 50% marks. Regular pre-examination meetings of the examiners were held to review OSCE units and stations, and to discuss strategy of the clinical case presentations includ­ing the allocation of marks to history taking technique, physical examination and management appropriate to the level of a minor diploma in family medicine. In 1990, OSCE was used in a professional examina­tion in medicine for the first time in Pakistan. Prior to the first examination, one of the examiners was designated as coordinator for OSCE and was trained to design and conduct such an examination. There was apprehension that there would be some resistance to it, as it was a new evaluation technique, to which the candidates had not been exposed before. The candidates were given a special briefing by the coordinator preceding each OSCE examination. The design and conduct of the examination process was fully explained to and discussed with the candidates. The candidates invited to take the traditional clinical case presentation examination should have obtained at least 50% marks in OSCE. The clinical case-presentation “traditional” examinations consisted of three patients: a child, an adult female patient (who may or may not be pregnant, or who may or may not have gynaecological problems), and an adult male patient. This was again a departure from the tradition of having cases categorized by specialty such as Surgery, Gynaecol­ogy, etc., and was based on the fact that women, men and children are all representative of a general prac­titioner/family physician’s patient population. Twenty minutes were allotted for the clinical encounter with each patient, followed by fifteen minutes for presentation of the case to, and discussion with a set of examiners. Each set of examiners marked the can­didates separately according to the strategy defined beforehand. Then the marks of all three set of examiners was totaled.

In the first six DFM examinations, 752 candidates took the True or False multiple choice theory examina­tion. 332 (44.14%) candidates passed the theory ex­amination and were eligible to take the clinical examina­tion (Table IV) but 4 were absent.

The OSCE was conducted at each of the two examination centers (Karachi and Lahore), for all candidates on the first day of each of the six clinical examinations. The traditional clinical examinations and discussions on the cases, were conducted on subsequent days. 170 (23.0%) candidates passed the DFM Examina­tion in the first six examinations, securing a minimum of 50% marks in OSCE and the traditional clinical examina­tion separately, with an overall aggregate of 60% pass marks. No-one failed in the OSCE as the pass percentage was kept at 50% with no negative marking at the question and response stations. Results were announced at the end of each day. Within a week of the examination, a written feedback was sent by the CPSP to each unsuccessful candidate, stating how well or badly the candidate performed in each component of the examination. To assess die relevance of OSCE against the traditional format of case presentations, we took 60% pass marks in case presentations as the goal standard and compared it to different cut-offs⁶ for OSCE. A four fold comparison (Tables V, VI)


showed that with no negative marking and a pass percentage of 75%, OSCE has: a positive predictive value of 55% (the proportion of the candidates correctly predicted as passed or “ruled in” was 55%). The low positive predictive value denotes that the prevalence of people getting> 60% marks in the traditional clinical examination was much less than 50%, a negative predictive value of 86% (the proportion correctly predicted as failures or “ruled out”), a sensitivity of 67% (the index to detect the can­didates who would pass), a specificity of 79% (the index to detect the ones who would fail), an overall accuracy of 75.6%.

These examinations were the first step in im­plementing rigorous standards for objective assessment of primary health care providers. As recommended by Bloom⁷, first of all, the training objectives were deter­mined, as the objectives are not only the goals toward which the curriculum is shaped and toward which instruction is guided, but they are also the goals that provide the detailed specifications for the construction and use of evaluative techniques. The examination content was designed according to the objectives and the common clinical problems that present to family physicians. In designing the training programme, we reviewed Tylers ideas⁸, and special care was taken to include sites offering wide varieties of learning experiences, which would enable the candidate to deal with the kind of “content” implied by the objectives. In weighting the examination, the available morbidity and mortality data in Pakistan^9,10 was of primary importance. Efforts were made to make the examination as objective as possible. An Objective Structured Clinical Examination, with 5-10 minutes OSCE stations, lends itself very well to the assessment of clinical competence of family physicians and general practitioners, as the average patient-doctor contact time is 5-10 minutes per visit¹¹ or less. One must be careful to give enough background information about the patients while structuring the stations to make it a valid examination, because in real life situations the family physicians have prior knowledge regarding the family and past histories of their patients.

We need to use a wider variety of questions, for example, modified assay questions and one out of five best choice, in the theory papers in order to assess the width and the depth of knowledge of the candidates. An appropriately structured OSCE is a relevant technique to assess the clinical competence of family physicians. The pass marks of OSCE have been arbitrarily defined! as 50% in our examination. Based on our findings, OSCE alone is used for the pass/fail criteria, the minimum pass marks should be around 75%. Our goal is to take further steps in ensuring high quality examinations in the discipline of family medicine, appropriate to the needs of Pakistan, through extensive supervised training and greater objectivity in evaluation.

1. Harden, R. and Gleeson, F.A. Asaeasment of clinical competence using an Objective Structured clinical Examination (OScE). Med. Educ., 1978;13:41-54.
2. Hubbard, J.P. TestCommittees and theirTaska. In: MeaauringMedical Education.The Tests and the Experience of the Board of Medical Examincrs, 2nd ad, Philadelphia, Lea and Febiger, 1978; pp. 19-25.
3. Ttnkelman, SN. Planning the Objective Teal. Thorndike, R.L. ed. In: Educational Measurement 2nd ed. Washington, D.C., American Council on Education, 1971;pp.46­-80.
4. Gronlund, N.E. Measuring complex Achievement. The Interpretive Exerciae. In: Measurement and Evaluation in Teaching, Sixth ed, New York, MacMillan Publishing Company, 1990; pp. 192-210.
5. Gronlund, N.E. Conauucting Objective Test Items: Multiple Choice Forms. In: Measurement and Evaluation in Teaching, Sixth ed, New York. MacMillan Publishing Company, 1990; pp.166-91.
6. Sacket,D.L, Haynes, R.B., Guyatt, G.H. etal. Theint.erpretation of Diagnostic Data. In: Clinical Epidemiology: A basicScience forClinical Medicine, Second ed. Boston, Little Brown and Co., 1991; pp. 69-152.
7. Ed Bloom, E.S. Educational Objectives and Curriculum Development. In:Taxonomyof Educational Objectives. Book 1, Cognitive Domain, London, Longman Group Ltd., 1978; pp.25-43.
8. Tyler, LW. How can learning Experiences be Selected which are likely to be useful in attainingthese Objectives? In: BssicPrinciples of Curriculum and Instruction, Chicago, the Universityof Chicago Press, 1949; pp.63-82.
9. Karim, M.S. Disease Patterns, Health Services Utilization and cost of treatment in Pakistan. J. Pat Med. Assoc., 1993;43:159- 164.
10. Ed. Rukanuddin, AR. and Farooqui, M.N.I. Mortality. In: The State of Population of Pakistan, 1987. National Inatituteof Population Studies, 1987; pp. 67-89.
11. Rashid Aly, Forman, W., Jagger, C. et at Consultations in General Practice: a comparison of Patient’s and Doctor’s Satisfaction. Br. Med. J., 1988;299:1015-16.