US Economics Analyst
Data Quality Is Still a Problem: Seasonal Distortions and Falling Response Rates (Walker)
The market’s sensitivity to individual data releases has increased sharply over the last couple of years, as those releases will ultimately decide Fed policy. However, two data quality issues have made assessing the implications of data releases for the outlook in real time more difficult.
First, distortions to seasonal adjustment factors caused by level shifts and much larger than usual swings in data early in the pandemic can bias our perception of the state of the economy. These distortions are most apparent in continuing jobless claims—which have predictably risen over the last couple of months because of seasonal adjustment issues—and Census retail sales—which will likely underperform “true” consumption activity during the holiday season—but likely affect a broader set of economic indicators.
We assess whether revisions to pre-pandemic seasonal factors for 22 indicators have introduced new seasonal distortions. Our analysis suggests that, on average, residual seasonality boosts seasonally-adjusted economic indicators by 0.2 standard deviations between March and May—the period when economic activity declined most sharply in 2020—and weighs them down during the rest of the year. On average, the effects are modestly smaller than when we conducted the same analysis last year and should continue to fade in future years.
Second, falling response rates to government statistical surveys can increase the uncertainty around the reported state of the economy. For example, the response rate to the Job Openings and Labor Turnover Survey (JOLTS) has plummeted almost 30pp since the start of the pandemic to just over 30%. The resulting smaller sample size has likely boosted its median monthly standard error—a measure of the expected discrepancy between a sample estimate and the true value of a population—by more than 90% relative to 2002-2013 (equivalent to a 90% confidence interval of roughly 700k).
Lower response rates have also likely contributed to larger-than-usual data revisions, as incremental responses can have a greater influence on updated estimates. For example, JOLTS job openings have been revised by an average of 180k in the second print over the last couple of years, double what was typical four years ago and more than triple what was typical six years ago.
Understanding these data quality problems introduced by the pandemic has been vital to interpreting the economic data over the last few years. In particular, it has enabled investors to see through several misleading signals sent by fluctuations in the jobless claims and official job openings data. We have addressed these challenges by calculating our own seasonal adjustment factors when necessary and supplementing official data plagued by low response rates with alternative data based on much larger sample sizes.
Data Quality Is Still a Problem: Seasonal Distortions and Falling Response Rates
The market’s focus on and sensitivity to individual data releases has increased sharply over the last couple of years, as those releases will ultimately determine Fed policy. Unfortunately, the increased scrutiny comes at a time when data quality issues have made assessing the implications of data releases for the outlook in real time more difficult. In this week’s Analyst, we discuss two data quality issues. First, we explore the extent to which seasonal distortions could be biasing our perception of the state of the economy. Second, we assess the impact of falling response rates to government statistical surveys, which can increase the uncertainty around a reported state of the economy.
Residual Seasonality: Seasonal Echoes of the Pandemic
Almost all US economic data are presented in a seasonally adjusted format. At its most basic level, seasonal adjustment involves comparing unadjusted values in a given month with the average value in other months; months that typically see weaker activity (e.g. housing starts in the winter) are adjusted up, while months that typically see strong activity (e.g. starts in the summer) are adjusted down—with the overall impact roughly netting out over the full year.
While seasonal adjustment allows economists to look through typical seasonal patterns and extract the underlying signal from economic indicators, as we wrote last year, the large swings in economic data during the pandemic have introduced distortions to several series, for two main reasons.
First, if a series is adjusted “multiplicatively”—that is, if seasonal swings are assumed to happen in proportion to the series’ level—a rapid level shift in that series will result in a proportional swing in the seasonal factors. But if the magnitude of seasonal changes remains similar to what it was before the shift, the seasonal factors will over-correct the series’ typical fluctuations, leading it to look higher in months when it is typically lower and vice-versa.
We find that this issue continues to distort seasonally-adjusted nominal retail sales ex. autos, which are multiplicatively seasonally adjusted. Retail sales ex. autos have increased to 35% above Feb. 2020 levels as consumer spending has seen a persistent shift away from services and towards goods. The series’ multiplicative seasonal factors have not meaningfully changed since 2019, meaning that the amplitude of the implied additive seasonal factors has grown substantially as a result of the higher year-round level of spending. For example, the December seasonal hurdle increased by $6bn year-over-year in 2021 and by $2bn in 2022.
It seems unlikely that the typical seasonal increases in spending around the holidays should rise proportionately with the increased year-round level of spending: the fact that consumers buy 35% more goods today does not necessarily imply that they will also buy 35% more at Christmas relative to how much they bought during the rest of the year. If, instead, consumers increased their demand for goods year-round but the typical dollar-increase that happens in the holidays remained unchanged (e.g. a consumer decides to buy a computer to work from home but does not buy more computers for her relatives at Christmas), the “true” seasonal swing would thereby be additive to the level of retail sales, and using a multiplicative seasonal factor would distort the series.
Even almost four years since the start of the pandemic, it is still hard to gauge which of these descriptions—additive or multiplicative—applies to retail sales. The increase in retail sales driven by higher prices—~13pp of the 35% increase—probably should be multiplicatively adjusted while it's less obvious for the remaining 22pp. But if the seasonal “truth” lies somewhere in between, a purely multiplicative approach would result in a meaningful deceleration in retail sales following the strength in the fall, other things equal. And indeed, as shown in Exhibit 1, seasonally-adjusted retail sales declined sharply last November and December, when the seasonal hurdle became sharply more punitive, before reversing the decline in January and February, when the seasonal hurdle swung back in the other direction. Seasonal fluctuations are often very large relative to seasonally-adjusted changes, and as a result, estimates of seasonally-adjusted changes can be very sensitive to any imperfections in the seasonal adjustment factors.
Exhibit 1: Retail Sales Growth Disappointed Last Holiday Season, as Spending Did Not Grow Proportionally with the Elevated Level of Retail Sales
Source: Department of Commerce, Goldman Sachs Global Investment Research
A second issue that can distort seasonal factors is the large swings in many economic indicators at the height of the pandemic. While the pandemic was the shortest recession on record, it was also the deepest in post-war history, with most economic indicators hitting their lows by the start of 2020Q2. Unless special measures are taken, seasonal adjustment programs react to this plunge in activity by generating more of a “boost” for data in surrounding years during those early spring months; adjustment factors for the months of the rapid plunge (those surrounding March 2020) will “help” more and those in other months will become more negative. However, if the underlying seasonal patterns have not actually changed, seasonally-adjusted data will then look stronger in the early part of the year and weaker thereafter.
The left panel of Exhibit 2 shows seasonally-adjusted continuing jobless claims that were reported as of the end of 2019. The right panel shows the seasonally-adjusted series as of today. After the BLS recalculated the old seasonal factors with pandemic era data, a distinct seasonal pattern emerged in the 2018 and 2019 seasonally-adjusted series. In those years, the revised continuing claims series shows a steeper decline from March through September followed by a rebound in September through March.
Exhibit 2: Seasonal Patterns Emerged Across Seasonally-Adjusted Continuing Claims Following the Recalculation of Seasonal Factors With Pandemic Era Data
Source: Department of Labor, Goldman Sachs Global Investment Research
These revisions could in principle simply be the result of properly evolving seasonal factors. “True” seasonal patterns often change over time; when they do, seasonal factors should change with them. For example, there has been a gradual shift in not-seasonally-adjusted retail sales from December to October and November over many years as consumers started holiday shopping earlier. The evolution in the seasonal factors for retail sales has taken that shift into account, as it ought to, and the resulting change in seasonal adjustment was therefore not the result of residual seasonality.
However, naturally-evolving seasonal changes are unlikely to explain the predictable seasonal patterns introduced by the revisions shown in Exhibit 2. As evidence of residual seasonality, Exhibit 3 compares the revisions to pre-pandemic seasonally-adjusted continuing claims[1] to the latest vintages of seasonally-adjusted continuing claims in 2023. The high degree of correlation shown (0.45 on a w/w basis, 0.80 on a 4w change basis) indicates that the changes to seasonal factors that arose from the pandemic are likely continuing to have undue influence on this year’s seasonally-adjusted data. Exhibit 4 shows that if taken at face value, this exercise suggests that essentially all of the 182k increase in continuing claims over the last couple of months is attributable to residual seasonality.
Exhibit 3: Revisions to Pre-Pandemic Data Arising From the Recalculation of Seasonal Factors Have Predicted Changes in Seasonally-Adjusted Continuing Claims This Year
Source: Department of Labor, Goldman Sachs Global Investment Research
Exhibit 4: Seasonal Distortions Can Alter Our Perception of the State of the Economy: Continuing Claims Have Been Flat Over the Last Couple of Months if Properly Adjusted
Source: Department of Labor, Goldman Sachs Global Investment Research
We can apply this same analysis—using revisions to pre-pandemic seasonal factors as a measure of residual seasonality—to a broader suite of economic indicators. Exhibit 5 shows the results of this exercise, expressed relative to the standard deviations of each indicator in 2019 (e.g. residual seasonality boosts seasonally-adjusted continuing claims by 0.7 standard deviations on average in January; note that we flip the sign for claims so that lower levels, which are a positive signal about growth, show up as positive residual seasonality) and ordered so that indicators with greater average absolute residual seasonality are at the top of the table.
Jobless claims appear at the top of the list, which is unsurprising for two reasons. First, the increase in claims at the outset of the pandemic was particularly outsized and thus the impact on the estimation of seasonal factors should be similarly outsized. Second, government statisticians sometimes remove obvious outliers from seasonal calculations, leading some series to be less likely to be distorted by pandemic-related residual seasonality. These kinds of adjustments were made earlier this year to jobless claims, meaningfully reducing the seasonal distortions in initial claims—which topped the list last year—but inadvertently amplifying them in continuing claims—which received special adjustments to a similar set of weeks despite evolving differently in 2020 and 2021. Seasonal distortions are also much more apparent in continuing claims than initial claims today because the series’ underlying volatility has fallen to below pre-pandemic levels while initial claims’ remains ~60% above.
Consistent with adjustment factors evolving to “help” more in the months that experienced the most rapid plunges at the start of the pandemic, our analysis suggests that residual seasonality is most supportive of measured activity in March-May (+0.2 standard deviations on average) and more punitive in the rest of the year, particularly in October-December (-0.2 standard deviations on average).
On average, these estimated effects are modestly smaller than when we conducted the same analysis last year and should continue to diminish with time (Exhibit 6). However, at the same time, the underlying data has become less volatile (the avg. standard deviation is 23% above 2019 levels in 2023 vs. 55% in 2022), suggesting that seasonal distortions could explain a greater share of the data’s variation going forward.
Exhibit 5: On Average, We Estimate That Residual Seasonality Boosts Seasonally-Adjusted Economic Indicators by 0.2 Standard Deviations in March-May and Lowers Them During the Rest of the Year
Source: Goldman Sachs Global Investment Research
Exhibit 6: Our Estimates of the Impact of Residual Seasonality on Seasonally-Adjusted Economic Indicators Have Declined in Magnitude Relative to Last Year
Source: Goldman Sachs Global Investment Research
Falling Response Rates: Would You Answer the Call?
Response rates to government statistical surveys have fallen sharply during the pandemic, accelerating a downward trend seen over the past decade (Exhibit 7). We see two main channels through which a lower response rate can impact economic data. The first is through nonresponse bias (i.e. a declining response rate can bias medium-term trends if the probability of nonresponse is correlated with respondents’ answers). Earlier this year, we showed that nonresponse bias may have been modestly boosting the reported level of job openings, and a few studies from Fed and government economists have suggested that select series have been impacted by nonresponse.[2] However, we do not have reason to think that the broader set of data has been biased by nonresponse.
Exhibit 7: The Pandemic Accelerated the Secular Decline in Response Rates to Economic Surveys
Source: Department of Labor, Department of Commerce, Goldman Sachs Global Investment Research
Second, a lower response rate reduces the sample size, which can increase month-to-month volatility (see Appendix) and widen the confidence interval around point estimates. For this point, we focus on the Job Openings and Labor Turnover Survey (JOLTS). The decline in the JOLTS response rate has been particularly severe, falling from roughly 70% a decade ago to just under 60% in 2019 and just over 30% by the end of last year. Exhibit 8 shows that, as a result, the median monthly standard error—a measure of the expected discrepancy between a sample estimate and the true value of a population—for the JOLTS job openings rate has likely increased by over 90% relative to 2002-2013.[3] An increase of this magnitude translates to a 90% confidence interval of roughly 700k around the latest job openings reading.
Exhibit 8: A Lower Response Rate Has Likely Pushed the Median Monthly Standard Error for the Job Openings Rate More Than 90% Higher Relative to 2002-2013
Source: Department of Labor, Goldman Sachs Global Investment Research
A smaller sample size can also lead to larger revisions in previously reported data, as incremental responses have a greater influence on updated estimates. Exhibit 9 shows that JOLTS job openings have been revised by an average of 180k in their second reading over the last couple of years, double what was typical four years ago and more than triple what was typical six years ago.
Exhibit 9: The Magnitude of Revisions to JOLTS Has Risen Meaningfully Over the Last Few Years
Source: Department of Labor, Goldman Sachs Global Investment Research
Understanding these data quality problems introduced by the pandemic has been vital to interpreting the economic data over the last few years. In particular, it has enabled investors to see through several misleading signals sent by fluctuations in the jobless claims and official job openings data. We have addressed these challenges by calculating our own seasonal adjustment factors when necessary and supplementing official data with alternative data. The decline in response rates to government surveys has increased the relative value of administrative and alternative data (e.g. jobless claims and web-based measures of job openings) that are immune from declining response rates and often calculated from much larger samples. However, as we have highlighted for many years, those series often have their own issues and are often best thought of as complements and cross-checks to the official government statistics.
Ronnie Walker
Economic Data Remains More Volatile vs. Pre-Pandemic—Even After Controlling for Changes in Growth—Potentially in Part Because of Lower Response Rates to Economic Surveys
Source: Goldman Sachs Global Investment Research
The US Economic and Financial Outlook
Source: Goldman Sachs Global Investment Research
- 1 ^ Resulting from changes to the seasonal factors (as opposed to the underlying NSA series).
- 2 ^ For example, see Kristin Butcher, Lucas Cain, Camilo García-Jimeno, and Ryan Perry, “Immigration and the Labor Market in the Post-Pandemic Recovery,” Chicago Fed Working Paper, 2023 or Jonathan Rothbaum and Adam Bee, “Coronavirus Infects Surveys, Too: Survey Nonresponse Bias and the Coronavirus Pandemic,” Census Working Papers, 2021.
- 3 ^ We would note that a given decline in the response rate for JOLTS will have a larger impact on its estimated confidence interval than for the CES and NFP because of JOLTS’s relatively smaller sample size (the CES survey is 30-40x larger).
Investors should consider this report as only a single factor in making their investment decision. For Reg AC certification and other important disclosures, see the Disclosure Appendix, or go to www.gs.com/research/hedge.html.