While I’ve been focusing on outcomes from prevention/wellness programs. I took an opportunity to search the web and get a feel for some of the data being posted. As discussed in column 5, one should be careful when looking at wellness data depending upon how it is reported and whether or not it has been subject to any scrutiny such as third party review or published in a peer reviewed journal.
Here are some examples of what I found and things that should lead the purchaser to ask more questions:
One wellness company has data on their website that purports to show reductions in the percent of people who were high risk for various conditions.
Information like this should lead to some questions, what are the total number of participants (were there 10 people or 10,000), were there any statistical significance measures, how was the data gathered (self report, validated data, etc.) who did the study, was it published?
or on another site a comment is made like this:
“projected to generate five percent savings on health & productivity costs”
With no additional information explaining what this is based upon, was the projected savings from claims data with absentee information, did they project presenteeism, if so how, what was the projected health care cost trend for the company, was the projectd savings 5% percent against trend, 5% against total costs as measured today, 5% against total cost plus projected trend…?
or from another company:
Employees with 5+ risk factors: In the category with the greatest number of risk factors and the potential for the highest medical costs, the number of employees decreased by 3.3 percent over the previous year.
Again how many were in the sample, was there any statistically significance to the 3.3% reduction, who did the study….?
the contracting firm learned that over 12 months, each participant in the program spent $395 less than non-participants
What were the total number of participants, might there have been selection bias (those that enrolled were healthier to begin with), was there an outlier or two in the non participating group that skewed the costs, who did the study, and on and on..
Again as discussed in Column 5, be wary and inquisitive:
- Do they provide any statistics such as confidence intervals?
- Is this data from a study they did or someone else did?
- Has the data been published in a reputable journal?
Often times as the data I showed above demonstrates, there are no accompanying statistics to show whether the data is statistically significant, no supporting information to determine how large a study, who was in the the groups studied, how the data was gathered, the quality of the data, or mention of authors to determine if others were involved in the study to add some outside credibility to it, etc. While each of these examples may in fact be good and relevant information documenting program successes, without the provision of additional information one should be wary.
The “gold” standard by which the data should be judged is whether or not it was published in a peer reviewed journal. In the past few years, there have certainly been examples of problems with peer reviewed journals, it is still the best standard of data reporting. Peer reviewed journals demonstrate a certain rigor to the study and an anonymous review by people with expertise in the field who look for credibility, statistical significance and other issues. The journal reviewers can reject the study if they believe it does not meet standards and have clear measures and relevance. Also when looking at a peer reviewed study, read it closely as the authors should discuss some of the confounding issues or other reasons why the results may have been as documented, but the cause could have been something other than the interventions from the program.
So in closing, remember this the next time a company puts some of their data in front of you; since most of us are not statisticians or actuaries, any study presented without being subject to a peer review process leaves some room to be desired and requires additional questions.