Research Guides: Systematic Reviews: How-To: 6. Assess for Quality and Bias

Why Assess for Bias and Quality?

Conducting risk of bias assessment, also called critical appraisal or sometimes quality assessment, is a defining feature of the systematic review process that elevates the methodological rigor and transparency of reported results. To do so, you will need a tool that fits the study designs included in your systematic review and establish a method to conduct the assessment in a dual-blind manner.

IOM Standard: 3.6 Critically appraise each study
3.6.1 Systematically assess the risk of bias, using predefined criteria
3.6.2 Assess the relevance of the study’s populations, interventions, and outcome measures
3.6.3 Assess the fidelity of the implementation of interventions
Cochrane Handbook of Systematic Reviews of Interventions - Chapter 8
Chapter 8: Assessing risk of bias in included studies
Avoiding Bias in Selecting Studies (AHRQ)
This is a chapter from "Methods Guide for Effectiveness and Comparative Effectiveness Reviews."
Assessing the Risk of Bias of Individual Studies in Systematic Reviews of Health Care Interventions
From the AHRQ - "Methods Guide for Effectiveness and Comparative Effectiveness Reviews"

Tools for Assessing Methodological Rigor and Risk of Bias

Tools for assessing the quality of studies - Quality Assessment and Risk of Bias Tool Repository
A searchable spreadsheet to help find an appropriate quality assessment or risk of bias tool for the study types in your systematic review.
Download the file for full functionality.

A few common appraisal tools:

AMSTAR-2
For critically appraising ONLY systematic reviews of randomized controlled clinical trials
CASP Checklists
Multiple study types
JBI Critical Appraisal Tools
Multiple study types
Center for Evidence Based Medicine (CEBM) Critical Appraisal Tools
Multiple study types
Cochrane Risk of Bias Tool
Randomized controlled trials
RoB 2.0
Randomized controlled trials
ROBINS-I
Non-randomized (observational) studies or cohorts of Interventions
ROBINS-E
Non-randomized (observational) studies or cohorts of Exposures other than interventions, including environmental and occupational exposures
Newcastle-Ottawa Scale (NOS)
Case control or cohort studies
QUADAS-2
Diagnostic accuracy
AXIS
Cross-sectional/prevalence studies
Mixed Methods Appraisal Tool (MMAT)
Mixed-methods SRs with a variety of study types including both qualitative & quantitative,
IHE Case Series Studies Critical Appraisal Checklist
Case series

Levels of Evidence

Levels of Evidence and Grades of Recommendations
Oxford-Centre for Evidence Based Medicine
Strength of recommendation taxonomy (SORT): a patient-centered approach to grading evidence in the medical literature
Am Fam Physician. 2004 Feb 1;69(3):548-56. PubMed PMID: 14971837

GRADE and Confidence Assessment
The recent PRISMA 2020 reporting guidelines now request an assessment of the certainty of the evidence. This can be tied into risk of bias through some tools, such as GRADE.

What is GRADE?
"GRADE (Grading of Recommendations, Assessment, Development and Evaluations) is a transparent framework for developing and presenting summaries of evidence and provides a systematic approach for making clinical practice recommendations.[1-3] It is the most widely adopted tool for grading the quality of evidence and for making recommendations with over 100 organizations worldwide officially endorsing GRADE." - BMJ Clinical Evidence (2015)

While GRADE does consider risk of bias, it considers the body of evidence at the outcome level, not the study level. Thus, a risk of bias tool is still needed when conducting a systematic review.
GRADE Working Group
The Grading of Recommendations Assessment, Development and Evaluation (short GRADE) Working Group began in the year 2000 as an informal collaboration of people with an interest in addressing the shortcomings of present grading systems in health care. The working group has developed a common, sensible and transparent approach to grading quality of evidence and strength of recommendations. Many international organizations have provided input into the development of the approach and have started using it.
GRADE Handbook
The GRADE handbook describes the process of rating the quality of the best available evidence and developing health care recommendations following the approach proposed by the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) Working Group

Confidence in the effect estimate

The 4 levels of Certainty in effect estimates / Quality of evidence

High: We are very confident in the evidence supporting the recommendation.
- Further research is very unlikely to change the estimates of effect.
Moderate: We are moderately confident in the evidence supporting the recommendation.
- Further research could have an important impact,which may change the estimates of effect.
Low: We have only low confidence in the evidence supporting the recommendation.
- Further research is very likely to have an important impact, which is likely to change the estimate of effect.
Very low: Any estimate of effect is very uncertain.

How to rate the Quality of Evidence (your confidence in the effect estimate)