Muhammad Amin Chinoy  ( Indus Hospital )
Tashfeen Ahmad  ( Aga Khan University, Karachi. )
Muhammad Tayyab  ( Aga Khan University, Karachi. )
Soulat Raza  ( Aga Khan University, Karachi. )

Objective: The recent emphasis on using "evidence based medicine" for decision-making in patient care issues has prompted many publishers to mention the level of evidence of articles in their journals. The "quality" of a journal may thus be reflected by the proportion of articles with high levels of evidence, apart from other criteria. We aimed to determine the level of evidence of articles in indexed Pakistani medical journals.
Methods: Two journals were selected: Journal of Pakistan Medical Association (JPMA) and Journal of College of Physician and Surgeons, Pakistan (JCPSP). Based on the information in the abstracts, all articles from 2003 and 2006 were categorized according to guidelines of Center for Evidence Based Medicine, Oxford, UK.
Results: 882 items/articles were reviewed. Of these, 270 (31%) were scientific articles within which 51% belonged to the "Therapeutic" and 25% to the "Prognostic" type. Only 27% had a high level of evidence (1 and 2) while a majority of 55% had level 4 evidence. Although there was a higher proportion of scientific research articles in JCPSP than JPMA (36% vs. 25%), no major difference in the levels of evidence was noted between the two journals, nor between 2003 and 2006. Moreover, the results were quite comparable to similar international studies.
Conclusion: The level of evidence in articles in our leading local journals compares favourably with international literature. We recommend that levels of evidence be stated with abstracts in local journals not only to help the clinicians in making decisions on the best available evidence, but also to elevate the "quality" of these journals (JPMA 59:5; 2009).

The recent trend in clinical research and patient care, towards evidence based medicine has resulted in efforts to estimate the strength of evidence described in scientific research articles published in medical journals. A leading international journal of orthopaedics, the Journal of Bone and Joint Surgery - American Volume (JBJS-A) has started stating the so-called "Level of Evidence" with the abstract of each study.¹ Moreover, the Board of directors of the American Orthopaedic Association has approved the use of levels of evidence for proper classification of orthopaedic studies.² According to the guidelines from the Center for Evidence Based Medicine (CEBM) in Oxford, UK,³ most studies can be categorized into four main types i.e. Therapeutic, Prognostic, Diagnostic or Economic Analysis and Decision making. Each type is subdivided into five levels of evidence, 1 being the highest, referring to e.g. well-designed randomized control trials or systematic reviews thereof in the "Therapeutic" type, and 5 being the lowest referring to expert opinions. Hence, the 'quality' of a journal may be considered to be reflected by the proportion of articles belonging to the high levels of evidence. In the medical community in Pakistan there is a general concern about the quality and standard of articles published in local journals,⁴ which may be reflected by the level of evidence in their articles. Moreover, most Pakistani journals are not easily accessible internationally because they are not indexed in the National Library of Medicine's bibliographic database MEDLINE®, which is publicly accessible through PUBMED®. We therefore aimed to determine the level of evidence of articles published in leading indexed Pakistani medical journals and compare with a leading international journal.

The four authors, i.e. a consultant orthopedic surgeon without any formal research training, a research-trained orthopedic surgeon at another institution, a fresh medical graduate and a final year medical student, familiarized themselves with the categorization system of CEBM according to the instructions to authors from JBJS-A(5) independently as well as in a group setting; no formal training was undertaken in this regard. A search was made in PUBMED® to identify indexed Pakistani medical journals; four were found - Journal of Ayub Medical College, Journal of College of Physicians and Surgeons, Pakistan (JCPSP), Journal of Pakistan Medical Association (JPMA) and Pakistan Journal of Pharmaceutical Sciences. Of these, the first three were relevant to clinical sciences, and for the current study the leading two journals, i.e. JCPSP, and JPMA were selected. All issues of these journals were evaluated retrospectively for the year 2003. The abstracts for all items published were downloaded from PUBMED and reviewed. All editorials, surveys, reviews, descriptive items, case reports and letters to Editors were categorized as "non-scientific items", while abstracts of the remaining, scientific (research) articles were analyzed. This study was then prospectively done for the same two journals for the year 2006 to look for any changes and trends in the type of articles and their levels of evidence over the three-year period.
Data validation: This was performed in three ways. Firstly, intra-observer reliability was determined from categorizations done multiple times by each observer, blinded to their own previous ratings. Secondly, inter-observer reliability was determined by analysis of ratings from the four observers, blinded to each other's ratings. Finally, in order to confirm that our categorization of articles matched that done by an international journal, we categorized articles published in JBJS-A from January 2003 to June 2003, and compared with the categorization by the journal itself, for each article. These analyses enabled us to determine the differences between raters with different clinical and research experience, and also to validate our categorization system with that used by international journals.
Data was analyzed on SPSS version 14.0. Non-parametric tests were used for statistical comparison. P-value <0.01 was considered significant.

The total number of items published in both journals in 2003 and 2006 was 882 (Table 1), but the [(T1)], [(T2)] majority (69%) were not clinical scientific articles, comprising of descriptive studies or surveys, case reports and series, editorials, review articles, letters to the editors or animal studies. As many as 35% of all items were case reports. As for scientific articles (n=270), the majority (51%), were of therapeutic type, while the level of evidence in majority (55%) of the articles was level 4. One hundred and seventy-four articles were published in JCPSP (84 in 2003, 90 in 2006), while 96 were published in JPMA (50 in 2003, 46 in 2006) (Table 2). In JCPSP, 36% of items were scientific research articles compared to 25% in JPMA (P<0.01). However, both journals had a similar distribution of levels of evidence in their articles; about 55% of all articles in both journals had level 4 evidence. When comparing the distribution of articles between 2003 and 2006, no clear difference was noted in the type of articles nor in the levels of evidence in either  [(T3)] of the two journals.
When comparing the Pakistani journals with JBJS-A (Table 3), a higher proportion of diagnostic studies and a lower proportion of therapeutic studies was seen in Pakistani journals (P<0.01 for both).

Data validation:

         Validation of data by rating of JBJS articles by our raters showed disagreements of type or level in some articles. However, the overall agreement on type was 87% and level was 77% (Table 4). The agreement with JBJS [(T4)] ratings was slightly better for the senior authors than the junior authors, but the difference was not statistically significant. In the ratings of Pakistani articles, the intra-observer agreement was 88% on type and 71% on level, and inter-observer agreement was 78% for both type and level. The results were similar to those reported in an article on interobserver reliability in level of evidence assignments published in JBJS.⁶

 

 

The primary aim of our study was to assess the type and level of evidence in articles published in two leading and indexed medical journals of Pakistan i.e. JCPSP and JPMA. We found that our articles compare favourably with the type and level of evidence in articles published in the international journal, JBJS-A. We adopted the same method of rating the levels of evidence for our journals as was described in JBJS-A in 2003.⁵ The reason for choosing JBJS-A was that it is a leading orthopaedic journal, and orthopaedics is the leading speciality in which journals have started mentioning the level of evidence in their abstracts (e.g. Arthroscopy, Clinical Orthoapaedics and Related Research, Journal of Bone and Joint Surgery -American volume, Journal of Pediatric Orthopaedics), while there are few such journals in other specialities (e.g. British Journal of Urology, Obstetrics and Gynecology).
Quality of a medical journal can be measured in different ways. Although there is certainly no "One" universally agreed upon criteria to do so, Eugene Garfield's Impact factor^7,8 probably enjoys the status of the most widely accepted criterion for measuring the quality of a scientific journal despite all its limitations and objections.⁹ "Prestige" of a journal is another unscientific criterion; the vague and qualitative term presumably reflecting the popularity of a particular journal.⁴ With time, other factors like Web Impact factor, Euro factor and the concept of how clinically important the question of a study is i.e. "POEM" were introduced.⁹ One, amongst the many limitations of impact factor, is its inability to show how a particular journal or an article improves the decision making in patient care or clinical practice. This is mainly because it is just a number representing the number of bibliographic citations that a journal receives.⁸ Quantifying the level of evidence for clinical research is expected to cater to the concern of improving clinical decision making. It provides the proper context in which the study should be interpreted; a higher level of evidence in a study plays an important role in changing clinical practices for the better. Hence yet, another way of measuring the quality of a journal is by the level of evidence in its articles.
Evidence based medicine is the conscientious, explicit and judicious use of current best evidence in making a decision about the care of an individual patient.¹⁰ Understanding evidence based medicine needs the basic knowledge of research methodology. Practicing evidence based medicine not only requires clinical expertise but also expertise in retrieving, interpreting and applying the results of scientific studies and in communicating the risks and benefits of the different courses of action in patient care. It categorizes different types of clinical evidence and ranks them according to the strength of their freedom from various biases that beset medical research. Systematic review or meta-analysis of Randomized Controlled Trials (RCTs) are the source of strongest evidence. In contrast, patient testimonials, case reports, and even the traditional expert opinion have little value as research evidence.^11,12 Unfortunately local journals are not popular enough amongst the researchers in Pakistan. Among the 5,000+ journals which the Institute for Scientific Information (ISI) of Philadelphia (now Thomson Scientific) includes in their Journal Citation Report (JCR),¹³ there are only three Pakistani journals with reported impact factors, and none of them relates to health.¹⁴ Apart from the impact factor, JCR contains the immediacy index and cited half life for each journal; the information can be used to compare and rank journals internationally. Having no impact factor reported for, the Pakistani medical journals do not receive enough good quality research manuscript submissions from local or foreign researchers. The researchers aim to publish their work in journals with high impact factors because the latter are often used as surrogate criteria for grant allocations, promotions in academic career and faculty output evaluations.
The result of our study showed a pleasantly surprising similarity to the result of those published in JBJS-A¹⁵ (Table 3) reflecting that the evidence contributed by these two Pakistani journals to evidence based clinical practice is similar to that produced by highly reputed international journals around the scientific world. Notably, in a Japanese study¹⁶ the proportion of articles with high levels of evidence (clinical trials and cohort studies) was 20.5%, in comparison to 50% in the American journals which they compared with. The proportion of articles belonging to level 1 and 2 in our journals was 27% while that in international orthopaedic journals ranges from 20% to 48%, as reported by Obremskey et al.¹⁵ Majority of our scientific articles were therapeutic interventions with level of evidence 4, in line with the international journals. Economic studies, which have a very high impact on national health decisions, were very small in number in both journals. Both these facts urge the need for studies which produce a higher level of evidence for patient care and also studies which look at the economic aspect of a disease or a research question.
For researchers, raising the level of evidence of their study may entail addition of a control (comparison) group and performing the study prospectively. This would raise the level of evidence of e.g. a level 4 study (a retrospective case series) to level 3 or 2. However the ultimate goal should always be to produce level 1 evidence. The national health sector needs to plan investing in randomized control trials, to ensure that funds are directed to areas of clinical need and relevant research training while ensuring that proper economic and cost effects have been looked upon through well conducted studies.
Our inter-observer agreement was quite similar to that reported by Bhandari et al. in their analysis of JBJS articles,⁶ supporting the validity of our data. We further validated our data by rating the articles from the issues of JBJS-A (2003 Jan to June) by blinding our reviewers to the level of evidence given by the journal itself and then compared it with the ratings done by JBJS-A. The comparison was once again favourable and in accordance with that stated by Bhandari et al. The other strengths of our study were: We reviewed a large number of articles (270) published in two journals i.e. JCPSP and JPMA during two years; between the two local journals, there was no significant difference in the type of scientific articles nor their level of evidence. In addition, we tried to look at changing trends, by rating the articles both retrospectively from 2003 and prospectively in 2006 issues; the trends did not seem to have changed significantly over time, reflected by the comparison of data from 2003 and 2006.
Our study, indeed, had some limitations. Due to the large number of items that we reviewed, it was not possible to read the whole articles; we relied on the information in the abstracts. Although the information in the abstracts was generally enough for a fair assessment of level of evidence, it is quite possible that closer examination in the article's text may have altered some of the results. We could not sub-classify the articles within each level of evidence and thus the reliability of each level could not be examined in depth by our study. We may have benefited from more research-trained reviewers, to further reduce the variation. We noted that the articles in local journals differed from the international articles in terms of clarity of language and objectives of the study. Moreover, study designs were generally not described in explicit detail and several articles had a mixed study design. In the latter case, for data analysis we chose the higher level of evidence to select the type for that article.
In conclusion, the current trends favouring evidence based medicine urges that we ensure raising the standard of our local journals in trying to produce the best possible evidence for clinical practice. Our researchers should not underestimate the local journals. They might not be widely popular, reflected by their lack of impact factor, but are producing levels of evidence comparable to international journals. We recommend that local researchers consider the evidence level guidelines when designing studies, and direct more of their manuscripts to local journals. This is expected to improve the "quality" of local journals and also break the vicious cycle of reluctance of researchers to send their work to the local journals because of their low ranking. It's about time we started stating the level of evidence in our studies and strive to produce the best possible evidence according to international norms. In future level of evidence might well be 'the criteria' for evaluating journals. We believe, that with the results presented, we have shown the way forwards towards improving the quality and image of Pakistani journals.

1. Wright JG, Swiontkowski MF, Heckman JD. Introducing levels of evidence to the journal. J Bone Joint Surg Am 2003; 85:1-3
2. Nowicki SA. Board approves SOPs, patient safety projects, position statements. Am Acad Orthop Surgeons Bull 2005; 53:45-8.
3. Centre for Evidence-Based Medicine. Levels of evidence [online] [cited 2007 December 29] Available from : URL: http://www.cebm.net/index.aspx?o=1025.
4. Shiwani MH. Quality of a medical journal: where do we stand. J Pak Med Assoc 2006; 56: 570-2
5. Journal of Bone and Joint Surgery - American Volume. Instructions to authors. Levels of evidence for primary research question [online] 2008 [cited 2007 December 29] Available from : URL:  http://www2.ejbjs.org/misc/instrux.dtl#levels
6. Bhandari M, Swiontkowski MF, Einhorn TA, Tornetta P 3rd, Schemitsch EH, Leece P, et al. Interobserver agreement in the application of levels of evidence to scientific papers in the American volume of The Journal of Bone and Joint Surgery. J Bone Joint Surg Am 2004; 86:1717-20.
7. GARFIELD E. Citation indexes for science; a new dimension in documentation through association of ideas. Science 1955; 122: 108-11.
8. Garfield E. The history and meaning of the journal impact factor. JAMA 2006; 295: 90-3.
9. Sohail S. Impact factor -- uses, abuses, obstacles and alternatives. J Coll Physicians Surg Pak 2007; 17: 311-2
10. Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn't. BMJ 1996; 312 :71-2.
11. Zaidi Z, Hashim J, Iqbal M, Quadri KM. Paving the way for evidence-based medicine in Pakistan. J Pak Med Assoc 2007; 57:556-60
12. Petrie A. Statistics in orthopaedic papers. J Bone Joint Surg Br 2006; 88:1121-36.
13. Thomson Scientific. ISI web of knowledge. [online] 2008 [cited 2007 December 29] Available from : URL: http://scientific.thomson.com/isi/
14. List of HEC recognized Journals upto Sept 30th 2006. [online] 2008 [cited 2007 December 29] Available from : URL: http://www.hec.gov.pk/new/AcademicAffairs/JCR/rec_journals.htm
15. Obremskey WT, Pappas N, Attallah-Wasif E, Tornetta P 3rd, Bhandari M. Level of evidence in orthopaedic journals. J Bone Joint Surg Am 2005; 87:2632-8.
16. Fukui T, Rahman M, Sekimoto M, Hira K, Maeda K, Morimoto T, et al. Study design, statistical method, and level of evidence in Japanese and American clinical journals. J Epidemiol 2002; 12:266-70.