All research is not created equal. While I try and champion research any chance I can, I am also aware that what some purport to be research is anything but. Often times, it is pretty easy to spot poor or misleading research. A good method to validate a study is to review the resources the authors referrenced when creating the document. If the study is based on other sound data, there is a good chance that it is valid. If however you find only limited referrences to other relevant material, there is a good chance the research undertaken for the study was poor.
Quality of research is very important. First, it is important to ensure that the justification you’re usinng for your position is worth the weight you’re expecting it to carry. Secondly, the public can put a lot of weight behind good research. I do believe though that the public has a good enough radar to spot biased opinions and regaining credibility once it is lost can be difficult.
To emphasize this point, let’s look at two reports:
- In 2009, the National League of Cities (NLC) commissioned a report to review the available research pertaining to cancer rates in firefighters. The intent of the report was to determine whether or not sufficient research existed to prove a causal relationship between firefighters and increased cancer risk. Ultimately, the 178-page report ‘determined’ that a sufficient body of evidence did not exist to prove a causal link ‘yet.’ This decision was reached even though the report itself states that there is evidence that supports an increased risk in firefighters. This report is presented as ‘research,’ something the public or elected officials could assume to be objective and unbiased. However, the research for this report was funded by the NLC, a group that has a vested interest in lowering its liability to cover cancer ridden firefighters. The goal of this study was to encourage a moratorium on state presumption laws (laws that presume that if a firefighter gets cancer, it was caused by occupational hazard) and possibly repeal them altogether.
- Just this year the National Institute of Standards and Technology (NIST) issued a report outlining the most effective staffing coningent for residential structure fires. In its report, the NIST found that a complement of four firefighters performed simulated fireground tasks significantly quicker than two and three person crews. This report will no doubt be used by the entirety of the fire service as a call to increase minimum staffing levels to safer levels.
Both of these reports look like research. The first study is an attack on the presumptive laws currently in place to provide support for firefighters who experience fire-related cancers led by a group standing to benefit greatly from a presumptive law moratorium. The fire service should most definitely combat such studies with truly objective research found in scientific periodicals such as the Journal of the American Medical Association.
However, if the fire service would attempt to attack the validity of the NLC report, would we simultaneously undermine the credibility of the firefighter staffing study? While the study was performed by the NIST, an agency we would presume to be objective, the public is smart enough to follow the money. Before you even get into the Table of Contents page, the public sees disclaimers noting funding for the study came from the Assistance to Firefighters Grant program. Jumping at the public’s attention on the next page are the logos of both the IAFC and the IAFF.
How and why should we expect the public to treat our report any different than we would treat the NLC report. While the study may never have gotten performed without the AFG funding and while the outcome of the study may be perfectly valid, it might carry more weight if its outcome was not so beneficial to its sponsors.