Evaluating studies: First weigh all the evidence

All of us, including myself, are guilty of skimming journals and reading the abstract to get to the bottom line. We erroneously assume that there is no need to read through the details of the study and that the review process will eliminate any flaws in a piece of research. This assumption is flawed too. Why? One study evaluating human medical journals found that 76 percent of the studies in a well-recognized journal were flawed.

Most veterinarians have not been trained to be critical of published manuscripts. Too often we take the information from a source at face value. This approach hinders our effort to elevate the status of veterinary medicine and the quality of medicine we practice.

There is a current emphasis in the veterinary field to practice evidence-based medicine (EBM). The concept of EBM is simplistic, although the practice is not always easy.

EBM is the practice of clinical medicine based on evidence that has been proven. Sources of this proven evidence include journals, abstracts, textbooks, conferences and other types of publications.

Unfortunately, the evidence in veterinary medicine comes from poorer sources compared to that in human medicine. In EBM, the quality of medicine being practiced is directly related to the accuracy of the information provided to the clinician.

Veterinarians need to be able to critically evaluate these sources to ensure that the conclusions are supported by the evidence.

We have several different sources of information. Primary sources would include publications that contain original research, which is the focus of this article. Secondary sources of information would include literature reviews, professional magazines and lectures. Tertiary sources of information include textbooks and clinic manuals.

The accuracy of the information from these sources is only as good as the primary sources. In both secondary and tertiary sources, the data can be biased, based on the opinion of the author(s).

It is important to make the distinction between manuscripts that are published in peer-reviewed journals and those that are not. Articles in peer-reviewed journals are considered superior. In a peer-reviewed journal, an editorial board made up of specialists in the field reviews and edits the articles submitted and then decides which ones are acceptable for publication.

Keep in mind that even articles published in peer-reviewed journals can be flawed. It is possible that an article is reviewed by a committee that does not have expertise in the field of the paper and relies on other factors to determine the worth of the paper, including the reputation of the authors and institutions or the estimated importance or relevance of a particular article rather than the actual science.

Most journal articles are broken into several sections, including an abstract, introduction, materials and methods, discussion, conclusions and references.

The abstract provides a summary of the work and includes the study design, analysis, results and conclusion. The introduction should review what is known on the topic and provide the rationale for the study. It should also outline the hypothesis and objectives.

The materials and methods section should clearly lay out the design of the study, including what type of study it is, inclusion and exclusion criteria, how a control group was selected, what treatment protocol(s) were used, what variables are being evaluated, how outcomes were measured and how the data will be analyzed. In short, it should allow another investigator to duplicate the study.

The results section should present the data and the results of statistical analysis. There should be no attempt to interpret the data. In theory, enough data should be present to allow the reader to determine the accuracy of conclusions.

In the discussion and conclusions, the authors should state whether the hypothesis was proven and the significance of it. The authors should discuss the strengths and limitations of the study and suggest future applications or studies based on their data.

The final parts are the references and acknowledgements.

Retrospective studies

Table 1 (p. 7S) lists the different types of studies. Studies can be classified either as retrospective or prospective.

Table 1: Types of studies

In a retrospective study, all of the events of interest have already occurred, and the data is obtained from the medical records and from recall. Somtimes, retrospective studies are referred to as "chart reviews." Advantages to retrospective studies include lower cost, faster completion and use of records that already exist. However, there are significant disadvantages, including incomplete or inaccurate medical records, biased results due to lack of randomization or blinding, confounding variables and difficulty establishing cause and effect (Table 2).

Table 2: Retrospective studies

The best use of a retrospective study is to generate hypothesis that can be tested in a prospective fashion. Unfortunately, the majority of studies in veterinary medicine are retrospective.

Bias can be a problem in both retro-spective and prospective studies, but is more problematic in retrospective studies (Table 3).

Table 3: Types of bias

Retrospective studies often suffer from recall bias, selection bias and observer bias. In retrospective studies, the information in the medical records was entered without any knowledge of a future retrospective study. This means that the medical records may be missing important data that the study conclusions will be based on.

For example, if a study is looking at tumor size done prior to the treatment of interest, the size of the tumor may not be in the record or the measurements may be estimated. Although the clinician can be asked if they can remember the tumor size, it is not likely to be an accurate measurement due to the length of time elapsed. (This is an example of recall bias.)

Retrospective studies are not randomized, so they often contain selection bias. Inadvertent selection bias occurs when the treatment is chosen based on prognostic factors, cost, treatment scheduling, etc. This results in groups that differ from each other in significant ways.

For example, patients may have received treatment only if they had certain negative prognostic factors, so it is not accurate to compare them to patients that did not receive treatment.

If there is not a contemporary control group, then historical controls may be substituted. Historical controls can introduce bias. For example, if there is an improvement in early detection, the historical controls may have had more advanced disease compared to the study population.

Prospective studies

Prospective studies are preferred over retrospective studies because the results are less likely to be influenced by bias and confounding factors.

In a prospective study, the data collection and the events of interest do not occur until after the patients are enrolled on the study, allowing for more consistent recording of data. Randomized prospective studies are preferred to avoid any selection bias on the part of the researcher or owner.

The goal of randomization is to create groups that are identical with the exception of the treatment being studied. If a study is blinded, it means that the owner does not know which group their pet is in. In some cases, a placebo is used if there is a group that does not receive treatment. In a double-blinded study, neither the owner nor researcher knows who has been assigned to which group. Blinding eliminates the potential for observer or experimenter's bias.

It has been said that it is possible to prove anything depending on how the statistics are manipulated. Without a background in statistics, it can be difficult to determine if the appropriate statistical analysis is being used in the study. There are a few basic considerations.

The significance of a finding is the likelihood that the finding is due to the treatment rather than by chance. If a finding is considered significant with a p value of .05, this means that the researcher is 95 percent confident that the treatment is responsible for the results.

The terms "the results are approaching significance" and "there is a trend toward significance" are misleading. Results should be considered to be either significant or not significant.

The power of the study is its ability to demonstrate an effect if one really exists and is determined by the sample size, variability and magnitude of the effect.

Studies that have small numbers of subjects are less likely to detect a significant difference unless the magnitude of the effect is large. Studies that have large numbers of subjects may detect a difference that is not clinically relevant.

One of the biggest issues with veterinary studies is small sample size. The majority of studies have small numbers of subjects, limiting the accuracy of findings.

Confounding and correlation

Confounding is the distortion of the effect of one factor due to the presence of another.

It often is possible to identify and correct for confounding factors with the appropriate statistical analysis.

Correlation of two variables is a measurement of how likely these variables are related. Keep in mind that correlation should not be taken to be cause and effect. It is entirely possible for two variables to be related by coincidence.

Although it is more time-consuming, it is important to read through the entire article so that we can judge the validity of the results for ourselves.

Suggested Reading

Only by doing this can we decide if the findings in a study warrant a change in the way we practice veterinary medicine.

Dr. Cronin earned her DVM degree from Cornell University in 1990. She completed an internship at the Animal Medical Center in New York and a medical oncology residency at North Carolina State University. She is a diplomate of the American College of Veterinary Internal Medicine in the specialty of oncology. After completing her residency, she was lecturer at the University of Pennsylvania Veterinary Teaching Hospital and a medical oncologist at Angell Memorial Animal Hospital in Boston. In 2001, she co-founded the New England Veterinary Oncology Group in Waltham, Mass.