Critical Analysis of Reliability and Validity in Literature Reviews
Author
Chetwynd, E.J.
Introduction
Literature reviews can take many forms depending on the field of specialty and the specific purpose of the review. The evidence base for lactation integrates research that cuts across multiple specialties (Dodgson, 2019) but the most common literature reviews accepted in the Journal of Human Lactation include scoping reviews, systematic reviews, and meta-analyses. Scoping reviews map out the literature in a particular topic area or answer a question about a particular concept or characteristic of the literature about a particular topic. They are broad, detailed, often focused on emerging evidence, and can be used to determine whether a more rigorous systematic review would be useful (Munn et al., 2018). To this end, a scoping review can draw from various sources of evidence, including expert opinion and policy documents, sometimes referred to as “grey literature” (Tricco, et al., 2018). A systematic review has a different purpose to a scoping review. According to the Cochrane Library (www. cochranelibrary.com), under the the section heading “What is a Systemic Review?” the following is stated: it will “identify, appraise and synthesize all the empirical evidence that meets pre-specified eligibility criteria to answer a specific research question” https://www.cochranelibrary.com/about/ about-cochrane-reviews.). Meta-analysis takes the process of systematic review one step further by pooling the data collected and presenting aggregated summary results (Ahn & Kang, 2018). Each type of analysis or review requires a critical analysis of the methodologies used in the reviewed articles (Dodgson, 2021). In a scoping review, the results of the critical analysis are integrated and reported descriptively since they are designed to broadly encapsulate all of the research in a topic area and identify the current state of the science, rather than including only research that meets specific established quality guidelines (Munn et al., 2018). Systematic reviews and meta-analyses use critical analysis differently. In these types of reviews and analyses, the quality of research methods and study instruments becomes an inclusion criterion for deciding which studies to include for analysis so that their authors can ensure rigor in their processes (Page et al., 2021). Reliability and validity are research specific terms that may be applied throughout scientific literature to assess many elements of research methods, designs, and outcomes; however, here we are focusing specifically on their use for assessing measurement in quantitative research methodology. Specifically, we will be examining how they are used within literature review analyses to describe the nature of the instruments applied to measure study variables. Within this framework, reliability refers to the reproducibility of the study results, should the same measurement instruments be applied in different situations (Revelle & Condon, 2019). Validity tests the interpretation of study instruments and refers to whether they measure what they have been reported to be measuring, as supported by evidence and theory in the topic area of investigation (Clark & Watson, 2019). Reliability and validity can exist separately in a study; however, robust studies are both reliable and valid (Urbina & Monks, 2021). In order to establish a benchmark for determining the quality and rigor across all methodologies and reporting guidelines (Dodgson, 2019), the Journal of Human Lactation requires that the authors of any type of literature review include two summary tables. The first table illustrates the study design broadly, asking for study aims, a description of the sample, and the research design for each of the reviewed articles. The second required table is focused on measurement. It guides authors to list each study’s variables, the instruments used to measure each variable, and the reliability and validity of each of these study instrument (https://journals .sagepub.com/author-instructions/jhl#LiteratureReview; Simera et. al., 2010). The techniques used to describe the measurement reliability and validity are sometimes described explicitly using either statistical testing or other recognized forms of testing (Duckett, 2021). However, there are times when the methods for evaluating the measurement used have not been explicitly stated. This situation requires the authors of the review to have a clear understanding of reliability and validity in measurement to extrapolate the methods researchers may have used. Lactation is a topic area that incorporates many fields of specialty; therefore, this article will not be an exhaustive exploration of all types of tests for measurement of reliability and validity. The aim, instead, is to provide readers with enough information to feel confident about finding and assessing implicit types of measurement reliability and validity within published manuscripts. Additionally, readers will be better able to evaluate the usefulness of reviews and the instruments included in those reviews. To that end, this article will: (1) describe types of reliability and validity used in measurement; (2) demonstrate how realiability and validity might be implemented; and (3) discuss how to critically review reliability and validity in literature reviews.
Citation
Chetwynd EM, Wasser HM, Poole C. Breastfeeding Support Interventions by International Board Certified Lactation Consultants: A Systemic Review and Meta-Analysis. J Hum Lact. 2019 Aug;35(3):424-440. doi: 10.1177/0890334419851482. Epub 2019 Jun 17. PMID: 31206317.
Publication Link