What we can and cannot say about the immediate effect of SF’s flavored tobacco product ban

NOTE: This blog post was written before it came to light that the San Francisco data in the Friedman study was all collected before the flavor ban was being enforced. Without data after enforcement started, Friedman’s paper falls apart. Please read the recent blog post detailed why this is the case.

On May 24, 2021 Abigail Friedman published “A Difference-in-Differences Analysis of Youth Smoking and a Ban on Sales of Flavored Tobacco Products in San Francisco, California” in JAMA Pediatrics that concluded that “San Francisco’s ban on flavored tobacco product sales was associated with increased smoking among minor high school students relative to other school districts.” This study attracted a lot of attention from people on both sides of the debate over laws banning the sale of flavored tobacco products.

Friedman correctly applied standard statistical methods and found that, as she wrote, cigarette smoking increased in San Francisco compared to what one would expect based on comparing smoking in San Francisco with other school districts based on one point in time immediately after the favor ban law took effect.

My major concern about the paper is the speculation she made after presenting the data-based conclusion quoted above. She wrote: “While the policy applied to all tobacco products, its outcome was likely greater for youths who vaped than those who smoked due to higher rates of flavored tobacco use among those who vaped. This raises concerns that reducing access to flavored electronic nicotine delivery systems may motivate youths who would otherwise vape to substitute smoking.” The problem with this statement is that Friedman did not provide an analysis of vaping in her paper.

In the paper justified not analyzing the e-cigarette data, arguing that “Recent vaping was not considered because of likely confounding. California legalized recreational marijuana use the same year San Francisco’s flavor ban went into effect; in addition, the YRBSS’s [CDC Youth Risk Behavior Surveillance System, her data source] vaping questions did not distinguish vaping nicotine vs marijuana.”

In fact, YRBSS has very clear questions on both vaping and cannabis use. The 2019 survey first asks about cigarette smoking, with the introduction “The next 4 questions ask about cigarette smoking” (questions 30-33). Next, it asks about vaping with the introduction “The next 3 questions ask about electronic vapor products, such as JUUL, Vuse, MarkTen, and blu. Electronic vapor products include e-cigarettes, vapes, vape pens, e-cigars, e-hookahs, hookah pens, and mods” (questions 34-36). Finally, it introduces questions about other tobacco products with “The next 3 questions ask about other tobacco products” (questions 37-39). While some youth vape (and smoke and eat) cannabis, it is clear from the context that all the questions deal with tobacco products.

In addition, the YRBSS later asks 3 questions about cannabis introduced with “The next 3 questions ask about marijuana use. Marijuana also is called pot, weed, or cannabis” (questions 45-47). It is true that these questions only ask about how much they used cannabis in general rather than specific products. but these questions could be cross-linked with the tobacco questions to quantify overlap.

It may be that marijuana legalization which, as Friedman notes, occurred at the same time as the San Francisco flavor ban confounded the results, but it would probably affect all the products not just e-cigarettes. In addition, most who use cannabis smoke it and there is tremendous crossover between tobacco and cannabis use, particularly in youth.

The bigger problem (which she pointed out to me in an email discussion we had about the paper), is that the vaping questions have only been on YRBSS since 2015. The YRBSS is conducted every two years, so there are only 2 data points (2015 and 2017) before the law took effect in 2019. This is not enough data points to do the statistical difference-in-differences analysis she used in the paper for smoking.

As I noted above, Friedman’s conclusion about San Francisco is based on a single data point. This is because that the law went into effect on January 1, 2019 with active enforcement starting April 1, 2019, around the time the 2019 YRBSS data were collected in the “spring semester.” The fact that the SF health department did active outreach to retailers in late 2018 and early 2019 led to high retailer compliance even before formal enforcement started on April 1, 2019 somewhat mitigates this problem, but the fact is that the law was only in force for a relatively short time before the data were collected.

Another concern with her interpretation that the flavor ban was driving youth from e-cigarettes to cigarettes is that the law banned all flavored tobacco products — including menthol cigarettes that are widely used by youth — not just flavored e-cigarettes. It also did not restrict the sale of unflavored cigarettes or e-cigarettes.

In addition, data on sales collected using checkout bar code scanners showed that the sale of all flavored tobacco products dropped in San Francisco by 96% after the law went into effect. (There were not comparable changes in two California comparison cities, San Jose and San Diego.)

She also justified her conclusion that the flavor ban was leading to increase smoking because youth were substituting cigarettes for e-cigarettes by citing a paper that concluded than applying minimum sales age restrictions to e-cigarettes led to more cigarette smoking: “Indeed, analyses of how minimum legal sales ages for electronic nicotine delivery systems are associated with youth smoking also suggest such substitution.” While the study she cites did reach that conclusion, a study we published with colleagues from the CDC that was based on individual level data drew a different conclusion: ” After adjusting for covariates, state e-cigarette MLSA [minimum legal sales age] laws did not affect youth cigarette smoking. Unadjusted for e-cigarette and other tobacco use, these laws were associated with lower cigarette smoking.”

In sum, the Friedman paper’s statistical method to analyze smoking is appropriate and the observation that smoking increased after the flavor ban is certainly a provocative finding even though it is based on a single point in time. It points to the need for studies based on a longer time period (i.e., more post-policy data points), ideally that explicitly accounted for the effect of marijuana legalization. The conclusion that youth were substituting cigarettes for e-cigarettes, however, is not based on any actual evidence.

Finally, quite separate from a discussion of the methods and interpretation in Friedman’s paper, one has to consider the fact that the cigarette companies’ are continuing to spend tens of millions of dollars on lobbyists and unsuccessful lawsuits fighting flavor bans. In addition, just in California they spent $12 million in San Francisco unsuccessfully opposing its flavored tobacco products ban and $21 million so far to force a referendum on the California law modeled on San Francisco’s law. The tobacco companies have access to more and better data on sales and use of their products than any of us academics have. If flavor bans substantially increased cigarette sales the multinational cigarette companies would be supporting them, not wasting their money fighting them.

That’s the strongest evidence that these laws are good for public health.