Saturday, January 23, 2016

What All the Numbers in the Harassment Study in Ethan Siegel's Post at Forbes Probably Don't Really Mean Afterall

[Update: this criticism has had the useful (if still unacknowledged) effect of correcting a widely published error in the preliminary presentation of the CSWA survey. The 57% percent figure that is cited in this post has now been corrected in the AAS presentation slides. The correct value is 32%.]

In "What All The Harassment Stories In Astronomy Really Mean", Ethan Siegel points to a CWSA study in which "57% [of respondents] reported being personally, verbally harassed (and 9% physically harassed) because of their gender." Slides from a presentation of the results at this year's AAS conference can be downloaded at the Southwest Research Institute's website.

With the slides on hand, Siegel's statement provides an excellent insight into the way the problem of sexual harassment is being constructed. I would have said "exaggerated" but that is both needlessly polemical and, ironically, a sort of understatement. We seem to be witnessing the construction of another science writing factoid here.

On physical "harassment" (please see my previous post to understand the scare quotes), Siegel seems to be referring to slide 9 in the presentation. [Update: astonishingly, the slides havebeen changed without any official corrigendum. Some slides have been changed, others have been removed. This one now appears as slide 6. I've written about this issue here. I hesitate to call it fraud, but it does amount to altering a dated document.]

Here we learn that 27/426 report being "rarely" physically harassed [based on gender], and 8/426 report that it happens "sometimes". The number for "often" is not given, but 9% of 426 is 38.34. If we subtract 35 (27 + 8) from that we get 3.34. Since it has to be a whole number it must be 3 or 4. Which means that less than one percent of respondents report being physically harassed often at work.

I would have reported this as good news. Keep in mind that we don't even know exactly what "physical harassment" involves here, but presumably sneaking up on a woman and snapping her bra would count. If the study is taken to be representative* then we can conclude that even that sort of thing hardly ever happens in astronomy.

The same goes for verbal "harassment", which is covered on slide 8. (The 57% factoid was also interesting enough for Miriam Kramer to tweet it.)

Here it's 81/426 (19%) that say it happens "rarely" and 48/426 (11%) that say it happens "sometimes". And here, again, we have to work out for ourselves how many respondents experience it "often". But now it gets a bit puzzling (HT @ticobas****). To get 57% we're missing 27% or 115 respondents. But that section of the bar doesn't suggest that the "often" group is bigger than the other two. So it's either an error in the graph or a typo in the total percentage. If it's an error in the graph, it would be the only result in which more people experienced something "often" than experienced the same thing "rarely", so I'm going to discount that possibility and suppose that the graph is right and the percentage is wrong.**

We can work out the true number of people who experienced verbal "harassment" often, by comparing the length of each section of the bar. The "often" section is about 1/14 of the whole bar. This means that the 129 responses that are already graphed account for 13/14 of the total, meaning that the length of the "often" section represents about 10 respondents. That's about 2.3%. So this result is probably closer to 3233%*** having been "verbally harassed" at all, and in fact over 97% percent experiencing it never, rarely or only sometimes.

If you want to take the "sometimes" more seriously than I do, that's okay. But then you should report that 87% of respondents report experiencing verbal harassment rarely or not at all. Certainly, it is misleading (and probably outright false) to simply say that "57% [of respondents] reported being personally, verbally harassed," which, again, is what Ethan Siegel would have us believe.

*I have my doubts about how representative the study is of the population of astronomers. I suspect there is some self-selection bias in the sample, which makes even 1% a high estimate of the real situation. In Miriam Kramer's piece at Mashable about the study, Christina Richey confirms that the sample was self-selected and, as Kramer puts it, "the scope of the data collected is limited," but nonetheless insists that "it is still representative of a problem in the field." I'm not entirely sure what she means by that. I have emailed Richey for information about the methodology of the study and will of course blog about it when I hear from her.

**I should be upfront here and say that quite a bit hinges on this assumption. After all, it would have been a sufficiently shocking result if wholly 27% of respondents did in fact often experience verbal harassment. Siegel could have reported that number with equal rhetorical effect.

***A rounding error. Thanks, @ticobas, for catching it.

****The original post credited @shubclimate with noticing this. But it was @ticobas who was the first to see it.

No comments: