14 November 2007

Sensationalizing Suicide II

UPDATE (5 Dec 2007): Kudos to CBS!

CBS updated their methodology webpage concerning their report on veteran suicides. I found the updated page via a post on their blog, Primary Sources.
...
Anyway, bottom line is this is an awesome response by CBS! Kudos!


Doing the Math

Danger Room links to a CBS story on an allegedly disproportional number of suicides by veterans. Supporting an anecdotal piece on veterans of the current war, CBS pulls up statistics showing that veterans committed suicide at twice the rate of the average population.

Shock! Horror!

No.

UPDATE: "A CBS News Investigation Uncovers A Suicide Rate for Veterans Twice That of Other Americans" -- Or Does It?
Yet here's something odd about the data: For the overall rates to correspond to the male and female rates, the veteran pool would have to be 62% to 64% female, and the nonveteran pool would have to be about 66% female. Check it out, for instance, with the lower bounds on the 2005 data: 11.1 x 0.63 (female) + 31.5 x 0.37 (male) = a bit under 18.7 (overall).
Note that the data collected by CBS is not posted. The statistical analysis on the data conducted by Rathbun and CBS is not posted. Rathbun has not spoken or written about the data he received from CBS, the methodology he employed, or the analysis he provided to CBS. The data and analysis are not peer reviewed. And so on ....

There is no explanation from CBS why even the most basic information about the study isn't provided, such as sample size and representation by cohort. Without this, there's no way to "check" the information.

We can, however, do some simple "back of the envelope" calculations to see how skeptical we should be. These calculations should not be deemed to prove or disprove the CBS investigation, since that can only be done with their data set.

For example, let's take CBS's results for 2004 and calculate the percentages of male and female in the non-veteran and veteran cohorts to come up with the overall rates CBS provides. CBS reports that veterans had a 17.5-21.8 per 100,000 suicide rate in 2004 and a rate of 9.4 per 100,000 for non-veterans. Calculating for the non-veteran population first:
18.3(%male non-veteran) + 4.8(100% - %male non-veteran) = 9.4
This yields a makeup of 34% male non-veterans in the CBS data set and 66% female non-veteran. Why would female non-veterans be represented at twice the rate of male non-veterans? Why wouldn't CBS tell us this or at least explain it?

We can also check the veteran population. Calculating for both the lower and higher rates:
30.6(%male veteran) + 10.0(100% - %male veteran) = 17.5
38.3(%male veteran) + 12.5(100% - %male veteran) = 21.8
Both equations yield a 36% male veteran population and 64% female veteran population. Again, why? Why is the female population so high after Rathbun "adjusted the rates of suicide for age, gender and any potential error in the gathering of the raw data by the states" and, "The overall rates are adjusted for age and gender in both the veteran and non-veteran populations. The male and female rates are age adjusted."

The CDC reports a 10.9 per 100,000 suicide rate in 2004 for the general population, 18.0 per 100,000 for males and 4.5 per 100,000 for females. Let's compare the percentage of male and female in the CDC overall age adjusted numbers with the CBS percentages:
18.0(%male) + 4.5(100% - %male) = 10.9
This yields a 47% male population and 53% female population. Makes you wonder why the CBS populations are so skewed, doesn't it?

Another example, let's take the numbers CBS provides for veteran and non-veteran suicides in 2004 and compare them with the 2004 numbers from the CDC. We should be able to calculate the percentages of each group in the CBS data set that would provide a match with the CDC numbers.

Assuming that the CBS veteran plus non-veteran population sample would be similar to the CDC (total) general population, we can calculate the percentages for the veteran and non-veteran cohorts to produce a 10.9 overall suicide rate. Calculating for both the higher and lower veteran rate gives us a range for the percentages:
9.4(%non-veteran) + 21.8(100% - %non-veteran) = 10.9
9.4(%non-veteran) + 17.5(100% - %non-veteran) = 10.9
This yields percentages of 81.5-87.9% for non-veterans and 12.1-18.5% for veterans, which I consider a little high, but within the range of believable. The census estimate for the 2004 general population is approximately 293.64 million and the Veterans Administration estimates 24.8 million in the 2004 veteran population. This means veterans account for approximately 8.5% of the general population. According to CBS, these are also the numbers they used:
Veteran population numbers were obtained by CBS News from the Department of Veterans Affairs and general population numbers came from the U.S. Census Bureau. The veteran population was then subtracted from the general population in order to get the "non-veteran" population.
Repeating this process for male rates, CBS reports that male veterans had a 30.6 to 38.3 per 100,000 suicide rate in 2004 and 18.3 per 100,000 for non-veterans. The CDC reports a general population rate of 18.0 per 100,000 for males in 2004.
18.3(%non-veteran) + 38.3(100% - %non-veteran) = 18.0
18.3(%non-veteran) + 30.6(100% - %non-veteran) = 18.0
It should be obvious that since both the male non-veteran rate and male veteran rate reported by CBS are higher than the CDC rate for males in the general population, there is no way to make the CBS statistics comparable with the CDC data.

The same is true for the female rates since both the veteran and non-veteran rates are higher than the 4.5 per 100,000 rate reported by the CDC.

Finally, let's look at a statistic that CBS chose to highlight in their story: "The suicide rate for [the 20-24 age group] non-veterans is 8.3 per 100,000, while the rate for veterans was found to be between 22.9 and 31.9 per 100,000."

The 2004 census estimate for the 20-24 age group is approximately 21 million (20,957,254). The 2004 Veterans Administration estimate for the same age group is 316,000 (315,917 reported to the nearest 1,000). This means veterans make up 1.5% and non-veterans make up 98.5% of the 20-24 age group population. That should immediately raise some "sample size" flags.

Let's compare that to the percentages in the CBS data that would match the CDC data for the 20-24 age group suicide rate of 12.5 per 100,000:
8.3(%non-veteran) + 22.9(100% - %non-veteran) = 12.5
8.3(%non-veteran) + 31.9(100% - %non-veteran) = 12.5
This yields a 71.2-82.2% non-veteran population and a 17.8-28.8% veteran population for the 20-24 age group.

Remember, CBS doesn't provide the overall 20-24 age group suicide rate in their methodology and according to the numbers they got from the Department of Veterans Affairs and the U.S. Census Bureau, veterans make up 1.5% of the 20-24 age group.
  • How does CBS explain a result 10 to 20 times the 1.5% figure if we use the CDC suicide rate as a substitute for their missing data?

  • Why would the veteran population in this age group be more than two to three times the overall veteran population of 8.5% in the general population?

  • Why is the 20-24 veteran population a higher percentage than even the 12.1-19.5% veteran population we calculated using CBS and CDC data for all age groups above?
This is where I put on my skeptical Missouri hat and tell CBS, "Show me."

Remember, this quick back of the envelope math is only provided to stress the importance for CBS (and Dr. Rathbun) to provide the actual data and not to prove or disprove what they did (without access to their data). Below are additional links for media reporting on suicide, suicide statistics and recent studies.

Related:
Dart Center: Covering Suicide

• In your story, avoid speculating about the motivation for the suicide. Likewise, avoid quoting the speculation of others. Often, there is not a simple "reason" for a suicide. If you include one in your report, you may cause someone else to feel unwarranted responsibility for the suicide.

• Be careful not to romanticize the suicide with lurid or gratuitous detail.

Reporting on Suicide: Recommendations for the Media

Research finds an increase in suicide by readers or viewers when:

  • The number of stories about individual suicides increases3,4
  • A particular death is reported at length or in many stories3,5
  • The story of an individual death by suicide is placed on the front page or at the beginning of a broadcast 3,4
  • The headlines about specific suicide deaths are dramatic3 (A recent example: "Boy, 10, Kills Himself Over Poor Grades")
Interviewing Surviving Relatives and Friends: Concerns:
  • Dramatizing the impact of suicide through descriptions and pictures of grieving relatives, teachers or classmates or community expressions of grief may encourage potential victims to see suicide as a way of getting attention or as a form of retaliation against others.
Language

Referring to a "rise" in suicide rates is usually more accurate than calling such a rise an "epidemic," which implies a more dramatic and sudden increase than what we generally find in suicide rates.

Research has shown that the use in headlines of the word suicide or referring to the cause of death as self-inflicted increases the likelihood of contagion.3

Suicide and the media (Additional resources)

CBS: Suicide Epidemic Among Veterans
One age group stood out. Veterans aged 20 through 24, those who have served during the war on terror. They had the highest suicide rate among all veterans, estimated between two and four times higher than civilians the same age. (The suicide rate for non-veterans is 8.3 per 100,000, while the rate for veterans was found to be between 22.9 and 31.9 per 100,000.)
CBS's Veteran Suicide: Methodology
Results for 2004
Overall Rates
Veterans: 17.5 to 21.8 per 100,000
Non-Veterans: 9.4 per 100,000

Male Rates
Veterans: 30.6 to 38.3 per 100,000
Non-Veterans: 18.3 per 100,000

Female Rates
Veterans: 10.0 to 12.5 per 100,000
Non-Veterans: 4.8 per 100,000
Data Analysis
You wouldn't buy a car or a house without asking some questions about it first. So don't go buying into someone else's data without asking questions, either.
NCHS: Self-inflicted Injury/Suicide
All suicides (2003)
  • Number of deaths: 31,484
  • Deaths per 100,000 population: 10.8
Death rates for suicide, by sex, race, Hispanic origin, and age: United States, selected years 1950–2004
Deaths per 100,000 resident population

Year: 2004
All ages, age-adjusted4 . . . . . . . 10.9
All ages, crude . . . . . . . . . . . . . . 11.0
20–24 years. . . . . . . . . . . . . . . . 12.5
25–44 years . . . . . . . . . . . . . . . 13.9
25–34 years. . . . . . . . . . . . . . . . 12.7
35–44 years. . . . . . . . . . . . . . . . 15.0
45–64 years . . . . . . . . . . . . . . . 15.4
45–54 years. . . . . . . . . . . . . . . . 16.6
55–64 years. . . . . . . . . . . . . . . . 13.8
65 years and over . . . . . . . . . . . 14.3
65–74 years. . . . . . . . . . . . . . 12.3
75–84 years. . . . . . . . . . . . . . 16.3
85 years and over . . . . . . . . . 16.4


Male
All ages, age-adjusted4 . . . . . . . 18.0
All ages, crude . . . . . . . . . . . . . . 17.7
20–24 years. . . . . . . . . . . . . . . . 20.8
25–44 years . . . . . . . . . . . . . . . 21.7
25–34 years. . . . . . . . . . . . . . . . 20.4
35–44 years. . . . . . . . . . . . . . . . 23.0
45–64 years . . . . . . . . . . . . . . . 23.7
45–54 years. . . . . . . . . . . . . . . . 24.8
55–64 years. . . . . . . . . . . . . . . . 22.1
65 years and over . . . . . . . . . . . 29.0
65–74 years. . . . . . . . . . . . . . 22.6
75–84 years. . . . . . . . . . . . . . 34.8
85 years and over . . . . . . . . . 45.0


Female
All ages, age-adjusted4 . . . . . . . 4.5
All ages, crude . . . . . . . . . . . . . . 4.6
20–24 years. . . . . . . . . . . . . . . . 3.6
25–44 years . . . . . . . . . . . . . . . 6.0
25–34 years. . . . . . . . . . . . . . . . 4.7
35–44 years. . . . . . . . . . . . . . . . 7.1
45–64 years . . . . . . . . . . . . . . . 7.6
45–54 years. . . . . . . . . . . . . . . . 8.6
55–64 years. . . . . . . . . . . . . . . . 6.1
65 years and over . . . . . . . . . . . 3.8
65–74 years. . . . . . . . . . . . . . 3.8
75–84 years. . . . . . . . . . . . . . 3.9
85 years and over . . . . . . . . . 3.6


White male 5
All ages, age-adjusted 4 . . . . . . . 19.6
All ages, crude . . . . . . . . . . . . . . . 19.6
15–24 years . . . . . . . . . . . . . . . . . 17.9
25–44 years . . . . . . . . . . . . . . . . . 23.8
45–64 years . . . . . . . . . . . . . . . . . 26.1
65 years and over . . . . . . . . . . . . 31.2
65–74 years. . . . . . . . . . . . . . . . . 24.2
75–84 years. . . . . . . . . . . . . . . . . 37.1
85 years and over . . . . . . . . . . . . 48.4


4 Age-adjusted rates are calculated using the year 2000 standard population. Prior to 2003, age-adjusted rates were calculated using standard million proportions based on rounded population numbers. Starting with 2003 data, unrounded population numbers are used to calculate age-adjusted rates. See Appendix II, Age adjustment.
5 The race groups, white, black, Asian or Pacific Islander, and American Indian or Alaska Native, include persons of Hispanic and non-Hispanic origin. Persons of Hispanic origin may be of any race. Death rates for the American Indian or Alaska Native and Asian or Pacific Islander populations are known to be underestimated. See Appendix II, Race, for a discussion of sources of bias in death rates by race and Hispanic origin.
US Census Bureau: National Sex and Age

Veterans Administration: VetPop2004

Suicide by profession: lots of confusion, inconclusive data
In the end, say some researchers, occupation may not be much of a factor in suicide. Psychologists have long documented that among the top predictors for suicide are diagnosable mental disorder, co-morbid substance use, loss of social support and availability and access to a firearm.
Testimony of Mark S. Kaplan, DrPH
My colleagues and I found that veterans made up 16% of the sample and comprised 31% of the suicides (according to the Oregon Violent Death Reporting System, of 543 suicide decedents in Oregon in 2005, 153, or 28 percent, were veterans). Our findings showed that over time veterans were twice as likely (Relative Risk = 2.13, p < .05) to die of suicide compared to male nonveterans in the general population.
Veterans, depression & suicide: New VA/U-M study yields key findings for all veterans’ care
Results. Of 807694 veterans meeting study criteria, 1683 (0.21%) committed suicide during follow-up. Increased suicide risks were observed among male, younger, and non-Hispanic White patients. Veterans without service-connected disabilities, with inpatient psychiatric hospitalizations in the year prior to their qualifying depression diagnosis, with comorbid substance use, and living in the southern or western United States were also at higher risk. Posttraumatic stress disorder (PTSD) with comorbid depression was associated with lower suicide rates, and younger depressed veterans with PTSD had a higher suicide rate than did older depressed veterans with PTSD. [link added]
UPDATE: Astounding Arrogance at CBS

Previous: Sensationalizing Suicide

No comments: