Epidemiological characteristics and methodological quality of meta-analyses on diabetes mellitus treatment: a systematic review

in European Journal of Endocrinology

Objective

Well-conducted meta-analyses (MAs) are considered as one of the best sources of clinical evidence for treatment decision. MA with methodological flaws may introduce bias and mislead evidence users. The aim of this study is to investigate the characteristics and methodological quality of MAs on diabetes mellitus (DM) treatments.

Design

Systematic review.

Methods

Cochrane Database of Systematic Review and Database of Abstract of Reviews of Effects were searched for relevant MAs. Assessing methodological quality of systematic reviews (AMSTAR) tool was used to evaluate the methodological quality of included MAs. Logistic regression analysis was used to identify association between characteristics of MA and AMSTAR results.

Results

A total of 252 MAs including 4999 primary studies and 13,577,025 patients were included. Over half of the MAs (65.1%) only included type 2 DM patients and 160 MAs (63.5%) focused on pharmacological treatments. About 89.7% MAs performed comprehensive literature search and 89.3% provided characteristics of included studies. Included MAs generally had poor performance on the remaining AMSTAR items, especially in assessing publication bias (39.3%), providing lists of studies (19.0%) and declaring source of support comprehensively (7.5%). Only 62.7% MAs mentioned about harm of interventions. MAs with corresponding author from Asia performed less well in providing MA protocol than those from Europe.

Conclusions

Methodological quality of MA on DM treatments was unsatisfactory. There is considerable room for improvement, especially in assessing publication bias, providing lists of studies and declaring source of support comprehensively. Also, there is an urgent need for MA authors to report treatment harm comprehensively.

Abstract

Objective

Well-conducted meta-analyses (MAs) are considered as one of the best sources of clinical evidence for treatment decision. MA with methodological flaws may introduce bias and mislead evidence users. The aim of this study is to investigate the characteristics and methodological quality of MAs on diabetes mellitus (DM) treatments.

Design

Systematic review.

Methods

Cochrane Database of Systematic Review and Database of Abstract of Reviews of Effects were searched for relevant MAs. Assessing methodological quality of systematic reviews (AMSTAR) tool was used to evaluate the methodological quality of included MAs. Logistic regression analysis was used to identify association between characteristics of MA and AMSTAR results.

Results

A total of 252 MAs including 4999 primary studies and 13,577,025 patients were included. Over half of the MAs (65.1%) only included type 2 DM patients and 160 MAs (63.5%) focused on pharmacological treatments. About 89.7% MAs performed comprehensive literature search and 89.3% provided characteristics of included studies. Included MAs generally had poor performance on the remaining AMSTAR items, especially in assessing publication bias (39.3%), providing lists of studies (19.0%) and declaring source of support comprehensively (7.5%). Only 62.7% MAs mentioned about harm of interventions. MAs with corresponding author from Asia performed less well in providing MA protocol than those from Europe.

Conclusions

Methodological quality of MA on DM treatments was unsatisfactory. There is considerable room for improvement, especially in assessing publication bias, providing lists of studies and declaring source of support comprehensively. Also, there is an urgent need for MA authors to report treatment harm comprehensively.

Keywords:

Introduction

Diabetes mellitus (DM) refers to a complex metabolic disorder caused by defects in insulin function (1). It is one of the most common illnesses that increase the global disease burden. There were 387 million DM patients worldwide in 2014 and will increase to 592 million by 2035. In 2014, DM caused 4.9 million of deaths and contributed to 11% of total health expenditure (at least USD 612 billions) among adults (2). DM is also a significant disease burden in developing countries. For example, the overall prevalence of DM among adult population in China was 11.6% (3).

With a heavy disease burden worldwide, effective therapies for DM are urgently needed. Systematic review (SR) is an efficient strategy to summarize clinical evidence on treatments for a given disease (4). In an SR, meta-analysis (MA) is a statistical method that allows quantitative combination of data from different studies, which provides a more precise estimation of treatment effect (4). However, a biased MA may mislead evidence users, and even do more harm than good (5). Thus, it is crucial to inform evidence users such as clinicians and researchers on methodological quality while they are using evidence from a MA, so as to make sure that the evidence used for guiding clinical practice or future research is trustworthy.

Methodological quality of MAs on DM has not been comprehensively assessed. This systematic review was conducted to (i) investigate the bibliographical characteristics of MAs on the management of DM, (ii) assess the methodological quality of MAs on DM treatments and (iii) explore the association between bibliographical characteristics and methodological quality of MAs on DM treatments.

Methods

Eligibility criteria

We included publications satisfying the following criteria: (i) study design was SR with at least one MAs and (ii) the MA should focus on treatment effect of DM interventions. Network meta-analyses were excluded.

Literature search

Literature search was conducted in May 2016. The Cochrane Database of Systematic Review and Database of Abstract of Reviews of Effects were searched for potential MAs. Our literature search through these two databases covered both Cochrane reviews and non-Cochrane reviews (6). No restriction on language or year of publication was applied during the literature search. ‘Diabet*’ was used as the search keyword.

Literature selection and data extraction

All the retrieved citations were screened and assessed for eligibility. Full texts of potentially eligible publications were retrieved for further assessment. The most updated version was chosen if a MA had been updated. Data on bibliographic characteristics were extracted from each included MA.

Methodological quality assessment

The methodological quality of included MAs was evaluated by the validated Assessing the Methodological Quality of Systematic Reviews (AMSTAR) tool (7). It includes 11 methodological items, which are rated as ‘yes’, ‘no’, ‘can’t answer’ or ‘not applicable’ during the critical appraisal. AMSTAR was chosen as it is proven to be a valid and reliable tool for assessing the methodological rigor of Mas (8, 9).

Literature selection, data extraction and methodological quality evaluation were conducted by two researchers independently. Discrepancies were discussed and resolved by discussion and consensus between the two researchers. If disagreements cannot be resolved, a third assessor was referred to and his judgment was used as final decision.

Data analysis

Data on bibliographic characteristics and methodological quality were analyzed descriptively. Median and range were used for continuous variables. Frequencies were calculated for categorical variables. Methodological quality of included MAs was separately reported among Cochrane and non-Cochrane reviews and different DM types. Respectively, multivariable logistic regression analysis and multinominal logistic regression were used to examine the association between bibliographic characteristics and AMSTAR item ratings among item 1, items 5–11 (rating as yes or no) and items 2–4 (rating as yes, no or cannot answer). Rating on each AMSTAR item was used as a dependent variable in the regression analyses. Independent variables included being an update of a previous MA, whether the MA is published in the Cochrane Library, year of publication, impact factor of the journal where the MA was published, location of the corresponding author, number of authors, type of treatment and study designs included in the MA. Hosmer and Lemeshow (HL) test was used for checking model fit in multivariable logistic regressions. The model fit was considered as adequate when the P value was greater than 0.1. The likelihood ratio test was used to check the model fit in multinominal logistic regressions, with P < 0.05 considered as adequate model fit. Respectively, adjusted odds ratio (AOR) and adjusted relative risk ratio with their 95% confidence interval (CI) were calculated from multivariate logistic regressions and multinominal logistic regressions. All statistical analyses were conducted using IBM Statistical Package for Social Sciences (SPSS) 21.0. A P value <0.05 indicated statistical significance (except for HL test), and all the analyses were two sided.

Results

Results on literature search and basic characteristics of included MAs

Electronic databases search yielded 1586 citations, among which 252 MAs were included in this study (Fig. 1). A total of 4999 primary studies and 13,577,025 patients were included in the 252 MAs, which were published between 1995 and 2015, with half of them published after 2011. More than half of the included MAs (65.1%) included type 2 diabetes mellitus (T2DM) patients only. Of the 252 MAs, 160 (63.5%) focused on pharmacological treatments and the remaining 92 (36.5%) were on non-pharmacological interventions. Only 62.7% MAs mentioned treatment harm: 75.6% MAs on pharmacological treatments and 40.2% MAs on non-pharmacological treatments mentioned treatment harms.

Figure 1
Figure 1

Sampling of meta-analyses on diabetes mellitus treatments: flow chart.

Citation: European Journal of Endocrinology 175, 5; 10.1530/EJE-16-0172

Cochrane reviews accounted for 14.3% of the included MAs. Only 50 (19.8%) MAs were updates of previous MAs. The proportion of updated MAs was much higher among Cochrane MAs (69.4%) than that of non-Cochrane MAs (10.6%). Nearly half of the MAs were conducted by teams led by corresponding authors based in Europe (44.4%), followed by North America (24.6%) and Asia (21.0%). Similar regional distribution was also observed in terms of funding locations. European countries funded the largest number of MAs (20.3%), followed by North American countries (10.7%) and Asian countries (6.7%). The included MAs searched a median of four international databases but only 15 MAs (6.0%) searched non-English databases. Majority of the MAs (86.9%) included randomized controlled trials (RCTs). Details on bibliographical characteristics of the included MAs are presented in Table 1.

Table 1

Bibliographical characteristics of 252 included meta-analyses on diabetes mellitus treatments.

Bibliographical characteristicsResults
Cochrane MA36 (14.3%)
Non-Cochrane MA216 (85.7%)
An update of a previous MA50 (19.8%)
 An update of a previous MA among Cochrane MA25 (69.4%)
 An update of a previous MA among non-Cochrane MA25 (11.6%)
Median year of publication (range)2011 (1995 – 2015)
Median impact factor of the journal for which the MA was published (range)3.4 (0.0 – 39.1)
Median number of review authors (range)5 (1 – 21)
Location of the corresponding author
 Europe112 (44.4%)
 North America63 (24.6%)
 Asia53 (21.0%)
 Other continents24 (9.5%)
Type of diabetes mellitus
 Type 2 diabetes mellitus164 (65.1%)
 Type 1 diabetes mellitus26 (10.3%)
 Multiple types of diabetes included48 (19.0%)
 Gestational diabetes mellitus14 (5.6%)
Type of treatment
 Pharmacological160 (63.5%)
 Non-pharmacological92 (36.5%)
Total number of included primary studies4999
Median number of included studies in each MA (range)14 (2 – 166)
Total number of participants included in primary studies13,577,025
Number of MAs that reported harm of the intervention158 (62.7%)
 MAs reported harm of pharmacological treatments121 (75.6%)
 MAs reported harm of non-pharmacological treatments37 (40.2%)
Funding location of the MA
 Europe51 (20.3%)
 North America27 (10.7%)
 Asia17 (6.7%)
 Other continents6 (2.4%)
 Not reported151 (59.9%)
Number of MAs that searched international databases252 (100. 0%)
Median number of international databases searched (range)4 (1 – 24)
Number of MA searched non-English databases15 (6.0%)
Reporting of coverage year of search
 Yes, reported both starting and ending years131 (52.0%)
 Partially, only reported starting or ending years94 (37.3%)
 Not mentioned27 (10.7%)
Search terms reported for one or more electronic databases
 Topics/free text/keywords/MeSH143 (56.7%)
 Full Boolean85 (33.7%)
 No search terms24 (9.6%)
Eligibility of study design
 RCT only219 (86.9%)
 RCT, Quasi-RCT and observational studies30 (11.9%)
 Observational study only3 (1.2%)
Eligibility criteria based on language of publication
 English and languages other than English128 (50.8%)
 Included English publications only70 (27.8%)
 Language criteria not reported54 (21.4%)
Number of MAs that included a PRISMA-like flow diagram197 (78.2%)
MA, meta-analysis; PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-analyses; RCT, randomized controlled trial.

Methodological quality

Most MAs (89.7%) performed comprehensive literature search and provided descriptive characteristics of the included studies (89.3%). Around one third (30.6%) of the MAs combined findings inappropriately (i.e., MAs combined the results of primary studies without exploring the potential sources of heterogeneity, even when it is found). Only around half (54.0%) of the MAs performed duplicate literature selection and data extraction. The included MAs generally had poor performance on the remaining seven AMSTAR items. Less than half of the MAs (45.2%) assessed and documented scientific quality of included studies, and only 34.5% MAs considered that when formulating conclusions. The included MAs were also performed poorly in providing protocol (33.3%), in the inclusion of gray literature (39.3%) and in assessing publication bias (39.3%). Furthermore, lists of both included studies and excluded studies (19.0%) were not often provided. Comprehensive declaration on sources of support was only observed in 7.5% of the MAs (8.8% in MAs on pharmacological treatments and 5.4% in MAs on non-pharmacological treatments). Cochrane MAs generally had good performance in all aspects except for assessing the likelihood of publication bias (36.1%) and declaring sources of supports (25.0) (Table 2). However, MAs on type 1 or 2 diabetes generally showed better performance on AMSTAR than those on gestational DM or multiple types of DM (Table 3).

Table 2

Methodological quality of the included 252 meta-analyses on diabetes mellitus treatments with comparison between Cochrane and non-Cochrane meta-analyses.

Yes (%)No (%)Cannot answer (%)
Individual AMSTAR itemsTotal (252)Cochrane MA (36)Non-Cochrane MA (216)Total (252)Cochrane MA (36)Non-Cochrane MA (216)Total (252)Cochrane MA (36)Non-Cochrane MA (216)
Item 184 (33.3)36 (100.0)48 (22.2)168 (66.7)0 (0.0)168 (77.8)NANANA
Item 2136 (54.0)30 (83.3)106 (49.1)101 (40.0)6 (16.7)95 (44.0)15 (6.0)0 (0.0)15 (6.9)
Item 3226 (89.7)36 (100.0)190 (88.0)20 (7.9)0 (0.0)20 (9.2)6 (2.4)0 (0.0)6 (2.8)
Item 499 (39.3)25 (69.4)74 (34.3)31 (12.3)11 (30.6)20 (9.3)122 (48.4)0 (0.0)122 (56.4)
Item 548 (19.0)36 (100.0)12 (5.6)204 (81.0)0 (0.0)204 (94.4)NANANA
Item 6225 (89.3)36 (100.0)189 (87.5)27 (10.7)0 (0.0)27 (12.5)NANANA
Item 7114 (45.2)36 (100.0)78 (36.1)138 (54.8)0 (0.0)138 (63.9)NANANA
Item 887 (34.5)34 (94.4)53 (24.5)165 (65.5)2 (5.6)163 (75.5)NANANA
Item 9175 (69.4)24 (66.7)151 (69.9)77 (30.6)12 (33.3)65 (30.1)NANANA
Item 1099 (39.3)13 (36.1)86 (39.8)153 (60.7)23 (63.9)130 (61.2)NANANA
Item 1119 (7.5)9 (25.0)10 (4.6)233 (92.5)27 (75.0)206 (95.4)NANANA
AMSTAR, Assessing the Methodological Quality of Systematic Reviews; MA, meta-analysis; NA, not applicable. Individual AMSTAR items: 1. Was the protocol provided? 2. Was there duplicate study selection and data extraction? 3. Was a comprehensive literature search performed? 4. Was the status of publication used as an inclusion criterion (i.e., gray literature would be included, if located)? 5. Was a list of studies (included and excluded) provided? 6. Were the characteristics of the included studies provided? 7. Was the scientific quality of the included studies assessed and documented? 8. Was the scientific quality of the included studies used appropriately in formulating conclusions? 9. Were the methods used to combine the findings of studies appropriate? 10. Was the likelihood of publication bias assessed? 11. Were the sources of support for both the systematic review and the included primary studies reported?
Table 3

Methodological quality of meta-analyses on diabetes mellitus treatments stratified by diabetes mellitus types.

Yes (%)No (%)Cannot answer (%)
Individual AMSTAR itemsType 1 (26)Type 2 (164)Gestational (14)Multiple types (48)Type 1 (26)Type 2 (164)Gestational (14)Multiple types (48)Type 1 (26)Type 2 (164)Gestational (14)Multiple types (48)
Item 111 (42.3)55 (33.5)4 (28.6)14 (29.2)15 (57.7)109 (66.5)10 (71.4)34 (70.8)NANANANA
Item 220 (76.9)90 (54.9)6 (42.9)20 (41.7)6 (23.1)66 (40.2)3 (21.4)26 (54.2)0 (0.0)8 (4.9)5 (35.7)2 (4.2)
Item 325 (96.2)143 (87.2)13 (92.9)45 (93.7)1 (3.8)17 (10.4)0 (0.0)2 (4.2)0 (0.0)4 (2.4)1 (7.1)1 (2.1)
Item 410 (38.5)67 (40.9)5 (35.7)17 (35.4)1 (3.8)23 (14.0)0 (0.0)7 (14.6)15 (57.7)74 (45.1)9 (64.3)24 (50.0)
Item 59 (34.6)25 (15.2)2 (14.3)12 (25.0)17 (65.4)139 (84.8)12 (85.6)36 (75.0)NANANANA
Item 624 (92.4)151 (92.1)8 (57.1)42 (87.5)2 (7.6)13 (7.9)6 (42.9)6 (12.5)NANANANA
Item 712 (46.2)76 (46.3)7 (50.0)19 (39.6)14 (53.8)88 (53.7)7 (50.0)29 (60.4)NANANANA
Item 810 (38.5)58 (35.4)4 (28.6)15 (31.2)16 (61.5)106 (64.6)10 (6.1)33 (68.8)NANANANA
Item 921 (80.8)113 (68.9)8 (4.6)33 (68.8)5 (19.2)51 (31.1)6 (71.4)15 (31.2)NANANANA
Item 1011 (42.3)65 (39.6)6 (42.9)17 (35.4)15 (57.7)99 (60.4)8 (57.1)31 (64.6)NANANANA
Item 110 (0.0)15 (9.1)0 (0.0)4 (8.3)26 (100.0)149 (90.9)14 (100.0)44 (91.7)NANANANA
AMSTAR, Assessing the Methodological Quality of Systematic Reviews; MA, meta-analysis; NA, not applicable. Individual AMSTAR items: 1. Was the protocol provided? 2. Was there duplicate study selection and data extraction? 3. Was a comprehensive literature search performed? 4. Was the status of publication used as an inclusion criterion (i.e., gray literature would be included, if located)? 5. Was a list of studies (included and excluded) provided? 6. Were the characteristics of the included studies provided? 7. Was the scientific quality of the included studies assessed and documented? 8. Was the scientific quality of the included studies used appropriately in formulating conclusions? 9. Were the methods used to combine the findings of studies appropriate? 10. Was the likelihood of publication bias assessed? 11. Were the sources of support for both the systematic review and the included primary studies reported?

Association between bibliographical characteristics and methodological quality of non-Cochrane MAs

Our logistic regressions demonstrated four factors that were associated with several AMSTAR items (Table 4). Those published more recently had better performance in (i) assessing and documenting scientific quality of primary studies (AOR = 1.29, 95% CI = 1.13 – 1.47), (ii) considering scientific quality when drawing conclusion (AOR = 1.25, 95% CI = 1.09 – 1.43), (iii) conducting meta-analysis appropriately (AOR = 1.13, 95% CI = 1.03 – 1.23), (iv) assessing the likelihood of publication bias (AOR = 1.15, 95% CI = 1.04 – 1.27) and (v) stating conflicts of interest comprehensively (AOR = 1.46, 95% CI = 1.04 – 2.03). Only MAs that included RCTs were more likely to provide characteristics of included studies and (AOR = 2.74, 95% CI = 1.02 – 7.42) and perform critical appraisal of included studies (AOR = 2.86, 95% CI = 1.13 – 7.24). When compared with MAs with corresponding authors from Asia, those with authors from Europe were more likely to provide SR protocols (AOR = 5.96, 95% CI = 1.85 – 19.23). MAs with more review authors also performed better in providing MA protocols (AOR: 1.19, 95% CI: 1.04 – 1.37). Cochrane MAs only showed statistically significant better performance in considering scientific quality during conclusion (AOR = 59.55, 95% CI = 11.39 – 311.26) and in declaring conflicts of interest comprehensively (AOR = 17.40, 95% CI = 2.71 – 111.49). Good model fit was found in all the multivariate logistic regression analyses as the P values of all HL tests were >0.1. However, poor model fits were found among all the multinominal logistic regressions for AMSTAR items 2, 3 and 4; as a result, regression results were not reported on these three items.

Table 4

Association between publication characteristics and methodological quality of meta-analyses on diabetes mellitus treatments: logistic regression analyses.

AMSTAR item (dependent variable)#PredictorsAdjusted odds ratio (95% CI)P values
 1. Was an ‘a priori’ design (protocol) provided?Corresponding author from Europe*5.96 (1.85–19.23)0.003
More authors1.19 (1.04–1.37)0.012
 6. Were the characteristics of the included studies provided?Corresponding author from other countries*§0.13 (0.02–0.87)0.035
Only included RCT2.74 (1.02–7.42)0.047
 7. Was the scientific quality of the included studies assessed and documented?Published more recently1.29 (1.13–1.47)<0.001
Only included RCT2.86 (1.13–7.24)0.027
 8. Was the scientific quality of the included studies used appropriately in formulating conclusions?Being a Cochrane MA59.55 (11.39–311.26)<0.001
Published more recently1.25 (1.09–1.43)0.002
 9. Were the methods used to combine the findings of studies appropriate?Published more recently1.13 (1.03–1.23)0.006
10. Was the likelihood of publication bias assessed?Published more recently1.15 (1.04–1.27)0.005
11. Were the sources of support for both the systematic review and the included primary studies reported?Being a Cochrane MA17.40 (2.71–111.49)0.003
Published more recently1.46 (1.04–2.03)0.027
AMSTAR, Assessing the Methodological Quality of Systematic Reviews; CI, confidence interval; MA, meta-analysis; RCT, randomized controlled trial; #the P values of all Hosmer and Lemeshow tests were >0.1, indicating good model fit for all logistic regression analyses; §Corresponding authors from countries other than Asia, Europe and North America; *Corresponding authors from Asian countries were used as references.

Discussion

A large number of MAs on the treatment of DM are published, guiding the practice of day-to-day DM care. We identified 252 MAs with 4999 primary studies and more than 13 million participants that assessed the effects of DM interventions. One-seventh of them were Cochrane reviews and half of them were published after 2011. The majority of these MAs focused on pharmacological treatments and T2DM patients. Our results suggested that methodological quality of the MAs included in this study were generally poor, with less than 50% of them satisfying seven out of 11 AMSTAR items. This is consistent with the result of a study that focused on strategies for improving quality of care among DM patients (10). That study only found that 60% of the 125 included SRs only met four of the 11 AMSTAR methodological criteria. When compared with MAs in other clinical areas, MAs in DM performed no better than those focused on pharmacist health intervention (11), Chinese herbal medicine interventions (12) as well as treatment for chronic obstructive pulmonary disease (13). Our regression analyses showed that MAs published more recently were significantly related to better performance. Although these indicate methodological progress in the field, there are several key areas that systematic reviewers, evidence users and editors should pay special attention to.

First, it has been demonstrated that MAs based solely on published data are likely to be bias as primary studies with positive results are more likely to be published regardless of their rigor (14). In our study, only 37.3% of MAs attempted to search for gray literature and assessed publication bias respectively. These results indicate that special attention should be paid to both prevention and evaluation of publication bias for future MAs on DM treatments.

Secondly, it is well known that scientific quality of primary studies has a substantial effect on treatment effect estimation (15). Critical appraisal of included primary study is hence considered as a crucial step for gauging the trustworthiness of MA (16). However, less than half of the included MAs have assessed scientific quality of included primary studies or have considered that when formulating conclusion. Improvements on these methodological areas are essential in ensuring trustworthiness of MA results.

Thirdly, current studies have suggested that MAs with financial support are less transparent and are less likely to appraise included studies. These MAs usually draw more favorable conclusions for the financially supported intervention (17, 18). Unfortunately, it is difficult for readers to detect such sponsorship as majority of authors did not state sources of support for both the MA and for each of the included primary studies (10, 12, 13, 19). Only 7.5% of included MAs reported this information comprehensively. It is noteworthy that 63.5% of included MAs investigated the effects of pharmacological interventions. Bias might be introduced if pharmaceutical industries are involved in these MAs. Readers should also note that 24.4% MAs on pharmacological treatment did not report harm for the intervention, which is a crucial factor for treatment decision-making.

From our results, it seems that Asian research teams neither published as many MAs as European and North American teams: only 21.0% of these MAs had the corresponding authors from Asia. Although MAs conducted in Asia were on par with those conducted elsewhere in most of the AMSTAR items, results from regression analyses showed that MAs with the corresponding authors from Asian countries had worse performance in providing protocol and characteristics of included studies. Asian countries are facing increasing burden of DM and more than 60% of the global DM population is based in Asia (20). Policymakers in Asia may consider scaling up research capacity in this area, as it is likely that participations from Asian teams would facilitate the searching of regionally relevant evidence for DM management.

Strengths and weaknesses of this study

First, this is the first study that comprehensively assessed the methodological quality of MA on DM treatments, covering both Cochrane and non-Cochrane MAs. Secondly, a validated tool, AMSTAR (8, 9), was used for critical appraisal of MA in a reliable manner. Thirdly, instead of reporting a total overall score on methodological quality, we reported detailed performance for each AMSTAR item so as to inform specific areas that require improvements in future MAs. One limitation of this study is that methodological quality could only be assessed based on information provided by the MA authors. For example, nearly half (48.4%) of included MAs did not provide information on whether gray literature was searched. This reflects poor adherence of the PRISMA reporting guideline for MA (21). Another limitation is that AMSTAR did not cover the assessment on the appropriateness of subgroup analysis and sensitivity analysis, of which both are key aspects in the appraisal of MA (22).

Our results indicate that methodological quality of MAs is variable, and the common perception that MA would provide best evidence is not always true. Clinicians and other evidence users are suggested to assess and consider the methodological quality of MAs while making decision based on evidence generated from them. Future MAs on DM should provide a protocol of the review and comprehensive information on lists of studies and gray literature search. Reporting of conflicts of interests, assessment of publication bias and scientific quality of the included studies as well as consideration of scientific quality when drawing conclusion should also be improved. MA authors are also suggested to report harm of the treatment comprehensively.

Declaration of interest

The authors declare that there is no conflict of interest that could be perceived as prejudicing the impartiality of the research reported.

Funding

This research did not receive any specific grant from any funding agency in the public, commercial or not-for-profit sector.

Author contribution statement

V C and S W involved in the study concept and design. Y F Y and V L performed the literature search. Y F Y, Y F and R H conducted literature selection, data extraction and methodological quality assessments. X Y W, V L, B Y and K T performed the analysis and interpretation of data. V C, C W and X Y W involved in drafting of the manuscript. V C, S W, C W, K T and B Y performed critical revision of the manuscript for important intellectual content.

References

  • 1

    World Health Organization. About diabetes. In Diabetes Programme2015.

  • 2

    International Diabetes Federation. IDF Diabetes Atlas 6th edn. Brussels, Belgium: IDF Executive Office2014.

  • 3

    XuYWangLHeJBiYLiMWangTWangLJiangYDaiMLuJPrevalence and control of diabetes in chinese adults. JAMA2013310948959. (doi:10.1001/jama.2013.168118)

    • Search Google Scholar
    • Export Citation
  • 4

    MulrowCDCookDJDavidoffF.Systematic reviews: critical links in the great chain of evidence. Annals of Internal Medicine1997126389391. (doi:10.7326/0003-4819-126-5-199703010-00008)

    • Search Google Scholar
    • Export Citation
  • 5

    BeroLAJadadAR.How consumers and policymakers can use systematic reviews for decision making. Annals of Internal Medicine19971273742. (doi:10.7326/0003-4819-127-1-199707010-00007)

    • Search Google Scholar
    • Export Citation
  • 6

    Cochrane Community (beta). Database of Abstracts of Reviews of Effects (DARE). 2015.

  • 7

    SheaBGrimshawJWellsGBoersMAnderssonNHamelCPorterATugwellPMoherDBouterL.Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology2007710. (doi:10.1186/1471-2288-7-10)

    • Search Google Scholar
    • Export Citation
  • 8

    SheaBJBouterLMPetersonJBoersMAnderssonNOrtizZRamsayTBaiAShuklaVKGrimshawJM.External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS ONE20072e1350. (doi:10.1371/journal.pone.0001350)

    • Search Google Scholar
    • Export Citation
  • 9

    SheaBHamelCWellsGABouterLKristjanssonEGrimshawJHenryDBoersM.AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology20096210131020. (doi:10.1016/j.jclinepi.2008.10.009)

    • Search Google Scholar
    • Export Citation
  • 10

    WorswickJWayneSCBennettRFianderMMayhewAWeirMSullivanKGrimshawJ.Improving quality of care for persons with diabetes: an overview of systematic reviews - what does the evidence tell us?Systematic Reviews2013226. (doi:10.1186/2046-4053-2-26)

    • Search Google Scholar
    • Export Citation
  • 11

    MelchiorsACCorrerCJVensonRPontaroloR.An analysis of quality of systematic reviews on pharmacist health interventions. International Journal of Clinical Pharmacology2012343242. (doi:10.1007/s11096-011-9592-0)

    • Search Google Scholar
    • Export Citation
  • 12

    ChungVCHHoRSTWuXFungDHYLaiXWuJCWWongSYS.Are meta-analyses of Chinese herbal medicine trials trustworthy and clinically applicable? A cross-sectional study. Journal of Ethnopharmacology20151624754. (doi:10.1016/j.jep.2014.12.028)

    • Search Google Scholar
    • Export Citation
  • 13

    HoRSWuXYuanJLiuSLaiXWongSYChungVC.Methodological quality of meta-analyses on treatments for chronic obstructive pulmonary disease: a cross-sectional study using the AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool. NPJ Prim Care Respiratory Medicine20152514102. (doi:10.1038/npjpcrm.2014.102)

    • Search Google Scholar
    • Export Citation
  • 14

    EasterbrookPJBerlinJAGopalanRMatthewsDR.Publication bias in clinical research. Lancet1991337867872. (doi:10.1016/0140-6736(91)90201-Y)

    • Search Google Scholar
    • Export Citation
  • 15

    SavovicJJonesHEAltmanDGHarrisRJJuniPPildalJAls-NielsenBBalkEMGluudCGluudLLInfluence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Annals of Internal Medicine2012157429438. (doi:10.7326/0003-4819-157-6-201209180-00537)

    • Search Google Scholar
    • Export Citation
  • 16

    HigginsJGreenS.Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration2011 Available from www.cochrane-handbook.org.

    • Search Google Scholar
    • Export Citation
  • 17

    YankVRennieDBeroLA.Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study. BMJ200733512021205. (doi:10.1136/bmj.39376.447211.BE)

    • Search Google Scholar
    • Export Citation
  • 18

    JørgensenAWHildenJGøtzschePC.Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ2006333782. (doi:10.1136/bmj.38973.444699.0B)

    • Search Google Scholar
    • Export Citation
  • 19

    AguiarPMBritoGDCorrerCJLyra JuniorDPStorpirtisS.Exploring the quality of systematic reviews on pharmacist interventions in patients with diabetes: an overview. Annals of Pharmacotherapy201448887896. (doi:10.1177/1060028014529411)

    • Search Google Scholar
    • Export Citation
  • 20

    RamachandranASnehalathaCShettyASNandithaA.Trends in prevalence of diabetes in Asian countries. World Journal of Diabetes20123110117. (doi:10.4239/wjd.v3.i6.110)

    • Search Google Scholar
    • Export Citation
  • 21

    MoherDLiberatiATetzlaffJAltmanDG.Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ2009339b2535. (doi:10.1136/bmj.b2535)

    • Search Google Scholar
    • Export Citation
  • 22

    FlemingPSKoletsiDSeehraJPandisN.Systematic reviews published in higher impact clinical journals were of higher quality. Journal of Clinical Epidemiology201467754759. (doi:10.1016/j.jclinepi.2014.01.002)

    • Search Google Scholar
    • Export Citation

If the inline PDF is not rendering correctly, you can download the PDF file here.

 

     European Society of Endocrinology

All Time Past Year Past 30 Days
Abstract Views 0 0 0
Full Text Views 460 284 15
PDF Downloads 223 162 11
  • View in gallery

    Sampling of meta-analyses on diabetes mellitus treatments: flow chart.

  • 1

    World Health Organization. About diabetes. In Diabetes Programme2015.

  • 2

    International Diabetes Federation. IDF Diabetes Atlas 6th edn. Brussels, Belgium: IDF Executive Office2014.

  • 3

    XuYWangLHeJBiYLiMWangTWangLJiangYDaiMLuJPrevalence and control of diabetes in chinese adults. JAMA2013310948959. (doi:10.1001/jama.2013.168118)

    • Search Google Scholar
    • Export Citation
  • 4

    MulrowCDCookDJDavidoffF.Systematic reviews: critical links in the great chain of evidence. Annals of Internal Medicine1997126389391. (doi:10.7326/0003-4819-126-5-199703010-00008)

    • Search Google Scholar
    • Export Citation
  • 5

    BeroLAJadadAR.How consumers and policymakers can use systematic reviews for decision making. Annals of Internal Medicine19971273742. (doi:10.7326/0003-4819-127-1-199707010-00007)

    • Search Google Scholar
    • Export Citation
  • 6

    Cochrane Community (beta). Database of Abstracts of Reviews of Effects (DARE). 2015.

  • 7

    SheaBGrimshawJWellsGBoersMAnderssonNHamelCPorterATugwellPMoherDBouterL.Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Medical Research Methodology2007710. (doi:10.1186/1471-2288-7-10)

    • Search Google Scholar
    • Export Citation
  • 8

    SheaBJBouterLMPetersonJBoersMAnderssonNOrtizZRamsayTBaiAShuklaVKGrimshawJM.External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS ONE20072e1350. (doi:10.1371/journal.pone.0001350)

    • Search Google Scholar
    • Export Citation
  • 9

    SheaBHamelCWellsGABouterLKristjanssonEGrimshawJHenryDBoersM.AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. Journal of Clinical Epidemiology20096210131020. (doi:10.1016/j.jclinepi.2008.10.009)

    • Search Google Scholar
    • Export Citation
  • 10

    WorswickJWayneSCBennettRFianderMMayhewAWeirMSullivanKGrimshawJ.Improving quality of care for persons with diabetes: an overview of systematic reviews - what does the evidence tell us?Systematic Reviews2013226. (doi:10.1186/2046-4053-2-26)

    • Search Google Scholar
    • Export Citation
  • 11

    MelchiorsACCorrerCJVensonRPontaroloR.An analysis of quality of systematic reviews on pharmacist health interventions. International Journal of Clinical Pharmacology2012343242. (doi:10.1007/s11096-011-9592-0)

    • Search Google Scholar
    • Export Citation
  • 12

    ChungVCHHoRSTWuXFungDHYLaiXWuJCWWongSYS.Are meta-analyses of Chinese herbal medicine trials trustworthy and clinically applicable? A cross-sectional study. Journal of Ethnopharmacology20151624754. (doi:10.1016/j.jep.2014.12.028)

    • Search Google Scholar
    • Export Citation
  • 13

    HoRSWuXYuanJLiuSLaiXWongSYChungVC.Methodological quality of meta-analyses on treatments for chronic obstructive pulmonary disease: a cross-sectional study using the AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool. NPJ Prim Care Respiratory Medicine20152514102. (doi:10.1038/npjpcrm.2014.102)

    • Search Google Scholar
    • Export Citation
  • 14

    EasterbrookPJBerlinJAGopalanRMatthewsDR.Publication bias in clinical research. Lancet1991337867872. (doi:10.1016/0140-6736(91)90201-Y)

    • Search Google Scholar
    • Export Citation
  • 15

    SavovicJJonesHEAltmanDGHarrisRJJuniPPildalJAls-NielsenBBalkEMGluudCGluudLLInfluence of reported study design characteristics on intervention effect estimates from randomized, controlled trials. Annals of Internal Medicine2012157429438. (doi:10.7326/0003-4819-157-6-201209180-00537)

    • Search Google Scholar
    • Export Citation
  • 16

    HigginsJGreenS.Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration2011 Available from www.cochrane-handbook.org.

    • Search Google Scholar
    • Export Citation
  • 17

    YankVRennieDBeroLA.Financial ties and concordance between results and conclusions in meta-analyses: retrospective cohort study. BMJ200733512021205. (doi:10.1136/bmj.39376.447211.BE)

    • Search Google Scholar
    • Export Citation
  • 18

    JørgensenAWHildenJGøtzschePC.Cochrane reviews compared with industry supported meta-analyses and other meta-analyses of the same drugs: systematic review. BMJ2006333782. (doi:10.1136/bmj.38973.444699.0B)

    • Search Google Scholar
    • Export Citation
  • 19

    AguiarPMBritoGDCorrerCJLyra JuniorDPStorpirtisS.Exploring the quality of systematic reviews on pharmacist interventions in patients with diabetes: an overview. Annals of Pharmacotherapy201448887896. (doi:10.1177/1060028014529411)

    • Search Google Scholar
    • Export Citation
  • 20

    RamachandranASnehalathaCShettyASNandithaA.Trends in prevalence of diabetes in Asian countries. World Journal of Diabetes20123110117. (doi:10.4239/wjd.v3.i6.110)

    • Search Google Scholar
    • Export Citation
  • 21

    MoherDLiberatiATetzlaffJAltmanDG.Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. BMJ2009339b2535. (doi:10.1136/bmj.b2535)

    • Search Google Scholar
    • Export Citation
  • 22

    FlemingPSKoletsiDSeehraJPandisN.Systematic reviews published in higher impact clinical journals were of higher quality. Journal of Clinical Epidemiology201467754759. (doi:10.1016/j.jclinepi.2014.01.002)

    • Search Google Scholar
    • Export Citation