Letter to the editor: Misleading statistics in Homewood schools proficiency story
In The Homewood Star story titled “Test results are in: Homewood test scores show growth at elementary, middle schools; high school test scores decline,” published online Aug. 31, 2023, numerous data are presented concerning the proficiency scores of Homewood schools. The way the data are presented in the story is misleading and doesn’t account for how statistics should be used to evaluate a given outcome.
The way statistics are used in science, words like “decline” and “rise” are used to assess trends. As any high school student should know, in order to make comments about a trend, one must use at least three data points. The story does provide three years of data. However, the emphasis is on the year-to-year change in proficiency, not on the three-year trend.
Presenting the data this way allows the author to make comments such as those in the title of the article, “high school test scores decline.” In order to make valid claims of this type, a linear fit of the data should be performed and the slope of that line determined.
For example, the story emphasizes a 6% decrease in Homewood High School math scores from 2022. The trend observed across the three-year period shows a minor decrease in proficiency, only 1% per year on average. This data does support the claim that HHS math scores are on a slight decline, but when the trends for all three subjects are considered, overall, Homewood High School has seen a net increase in proficiency of 0.5% across all subjects per year since 2021. A better way to discuss the changes in proficiency since 2021 would be to tabulate the data. In this table, we can see Shades Cahaba improved proficiency the most of Homewood schools since 2021, and Homewood High School remained stagnant.
But even these data should be taken with a grain of salt. Spring 2021 represents the tail end of the first full COVID-19 school year. So, the change since 2021 may actually represent a re-acclimation of our city schools and students to in-person education. Taken in this light, it would be highly beneficial to have the data for 2018, 2019 and 2020, to truly develop a sense of how our schools performed before, during and after the pandemic.
Do these scores represent a recovery or a net gain in proficiency? We can’t tell without other data. The data provided also contains no statistical error information. So, when we look at a change of 1%, 5%, 10%, are those changes inside or outside the error of scores? We don’t know. For all we know, Homewood High School math scores are completely fine, or even on the rise, if the 1% change is smaller than the error of the data set.
My next issue was on the scale of the y-axes of the graphs provided, especially that of Homewood High School. When readers observe a steep slope in either direction on a graph, a subconscious response to the steepness of that line begins to set in. Between all schools, the lowest proficiency in any subject was 48% (a tie in math between Hall-Kent and Shades Cahaba back in 2021). The highest proficiency was 86% (English, Edgewood Elementary in 2023).
To avoid misleading the readers and manipulating their subconscious responses to the data, all data should be plotted on graphs with the same axis that contains the maximum and minimum of all the data sets, such as from 45% to 90%. Doing so creates the graph provided.
We as a community desire proficiency in subjects like math, science and English. Shouldn’t we hold ourselves and our beloved paper to the same standards? Presenting scientific data in an ethical and clear way is of utmost importance when building trust between the reporter of the science and their audience.
Raising alarms based on poorly understood or misrepresented statistics is akin to shouting “Fire!” in a theater. It’s wrong. I love you, Homewood Star. Do better.
Paul Wiget
Scientific writer, BioCryst Pharmaceuticals