Study claiming SF flavor ban increased youth smoking dissolves; it is not based on any data collected after enforcing the ban

The research letter, “A Difference-in-Differences Analysis of Youth Smoking and a Ban on Sales of Flavored Tobacco Products in San Francisco, California” Abigail Friedman published in JAMA Pediatrics in May 2021 concluded that “San Francisco’s ban on flavored tobacco product sales was associated with increased smoking among minor high school students relative to other school districts.” This study has been widely used in testimony opposing comprehensive flavor bans, most recently in Connecticut (at 10:49:35) and Colorado (at 6:19:20). 

The fundamental problem with the JAMA Pediatrics study is that there is no “after” data in the comparison.  So, the who differences-in-differences approach falls apart.  This study provides no information about the effect of enforcing the San Francisco flavor ban on high school cigarette smoking and should not be part of future discussions of similar laws.

This study uses a statistical technique called differences in differences that compares changes in outcomes (in this case cigarette smoking) before and after an intervention (in this case enforcement of San Francisco’s comprehensive ban on the sale of flavored tobacco products) in places with (San Francisco) and without (other “control” cities without such laws).  Essentially what the method does is tests whether the before-and-after changes in outcomes for the intervention and control places were statistically different (the “difference in the differences”). 

Dr. Friedman used publicly available data collected by the CDC known as the Youth Risk Behavior Surveillance System (YRBSS) data, which is collected every 2 years in schools, including in 2019.  Friedman assumed, not unreasonably, that all the data in the 2019 sample were collected in 2019.

It turns out that was not the case.  In particular, the San Francisco data were collected in November and December 2018, as documented in this email from the San Francisco school district:

It is not unusually a problem that the data for large national surveys such as YRBSS are collected over a period of several months.  When events are stable, these differences in collection time usually make little difference in the results of subsequent analyses.  But, in this case it makes a huge difference because the data were collected before the law was effectively implemented.

While the formal date in which the law took legal effect was in July (after a referendum by RJ Reynolds and other tobacco interests was defeated in the June election), the San Francisco Department of Public Health explicitly told retailers that they would not enforce the law until January 1, 2019.  As the SFDPH explained in our detailed case study of the implementation of the law,

The delayed compliance was due to the fact the law did not allow for a period of time for retailers to exhaust their inventory and for the SFDPH Enforcement Branch to educate themselves and the retailers on what products are actually flavored. We believe the Department’s presence and early outreach efforts resulted in the compliance of the law. This lesson has informed newer legislation relating to the subsequent San Francisco law prohibiting the sale of e-cigarettes that had not received an FDA market order effective January 1, 2020, which included a 6-month grace period to give the Department enough time to amend rules and regulations and educate retailers on how to comply.

As a result, only 17% of tobacco retailers that were visited in December 2018 has stopped selling flavored tobacco by that time.   (Thanks to effective outreach by the SFDPH, compliance jumped to 77% in January and settled down around 80% for the year; see figure below from our paper.)

Dr. Friedman recognized that the ordinance was not being enforced in fall 2018 in her response to the three letters published criticizing her paper.  She wrote: “retailer compliance jumped from 17% in December 2018 to 77% in January 2019 when the ban went into effect.”

So, in November and December 2018 – when the YRBSS data were being collected in San Francisco schools — retailers understood that there were no restrictions on their sales of flavored tobacco products.

This important detail was first reported by Jessica Liu, Lester Hartman, Andy Tan, and Jonathan  Winickoff in their paper, “Youth tobacco use before and after flavoured tobacco sales restrictions in Oakland, California and San Francisco, California,” published in Tobacco Control last week.  Given this limitation in the data, there is no way to draw any conclusions about the effect of enforcing the San Francisco law on smoking or other tobacco use based on existing YRBSS data.

You can see Jessica Liu present this result at the Connecticut hearing (at 1:30:16; I also testified at 7:00:30).

One could argue that the difference-in-difference analysis in the JAMA Pediatrics paper supports the conclusion that youth smoking increased in anticipation of the law taking effect.  There are examples of smokers modifying their behavior in advance of changes in the law, most notably stocking up on tobacco products before tax increases.  But,  such transient behavior doesn’t tell you anything about the long-term effects of tobacco tax increases, which, unlike the transient pre-implementation behavior, lower consumption.

If there was any such anticipatory purchasing in anticipation of the San Francisco flavor ban going into effect, one would think it would be buying flavored e-cigarettes and other flavored tobacco products while they were still available, not switching to (less desirable) cigarettes.

Direct purchasing data also contradicts against the hypothesis that people increased purchases of tobacco products in fall 2018: sales of all flavored tobacco products – as well as all tobacco products – were dropping in San Francisco, stabilizing at very low levels in January 2019.  This is illustrated by checkout bar code scanner data published by Gammon et al.  The figure below reproduces part of their Figure 1, which deals with flavored tobacco product sales.  Figure 2, not shown below, shows similar drops for all tobacco sales.

Another problem with the idea that youth who could not buy flavored e-cigarettes switched to cigarettes is that it ignores the fact that tobacco flavored e-cigarettes remained on the market after the San Francisco law went into effect and menthol cigarettes were banned.  It is hard to believe that because kids couldn’t get flavored e-cigarettes they switched in large numbers to unflavored cigarettes rather than tobacco flavored e-cigarettes.

Finally, Yang and colleagues also published a small survey of 247 18-34 year old tobacco uses in San Francisco who were paid to answer questionnaires on Amazon Turk.  (This is a much weaker methodology than national random samples such as YRBSS.)  They found statistically significant 17% (95% CI: 8.0%-27.5%) drops in total tobacco use among 18-24 year olds 7.6% (95% CI: 3.7% to 11.4%) drops in 25-34 year olds.  There were no significant changes in reported consumption of smoking products (including cigarettes and cigars).  Because of the small non-random sample this is not a high-confidence study.

While this discussion of alternative possible explanations for Friedman’s finding (which, of course, include random chance because her entire conclusion is based on the divergence of a single point in time (“2019,” actually collected in 2018) from the patterns established in the earlier waves of YRBSS) is of academic interest, it does not change the key fact that the JAMA Pediatrics paper cannot be used to assess the impact of enforcing a ban on the sale of flavored tobacco products because none of the data were collected from youth while a flavor ban was actually being enforced.

This study should not be relied on by any decision makers considering future flavor bans.

Note:  I wrote two earlier blog posts that mention the Friedman study, which should be read in the context of this new information about when the data were actually collected:

• High school student use of cigarettes and e-cigarettes fell after the San Francisco flavor ban took effect
• What we can and cannot say about the immediate effect of SF’s flavored tobacco product ban