The AGA’s Financial Literacy Study Is A Flawed Statistical Mess

The American Gaming Association (AGA) commissioned a study that reported gamblers, on the whole, have higher financial literacy than non-gamblers.

This makes intuitive sense to me.

It’s also probably, well, not quite right — at least not for the right reasons.

Listen, I don’t take joy out of ripping studies apart, and I certainly don’t take joy from ripping apart studies commissioned by trade associations that are only going to release findings that show their world in a positive light, but …

OK, fine, I lied, I do take joy, so let’s start with the two basics: 1) Nowhere in this new paper does it list the exact questions asked, it just points to numerous academic papers that created their own ways of measurement that the study’s authors used, and 2) It doesn’t tell me if the study using findings from 3,149 adults was weighted, instead telling me it “represented a diverse sample of gamblers, prediction market users, and non-gamblers across varying age, race, and household income demographics.”

Those two items are enough to raise one eyebrow, if not both, with the second part even more important. This, I know from experience.

I worked for a survey firm for years, writing up reports on questions, and had full access to the backend. Believe me, if I didn’t weight the survey correctly (race, age, sex, etc.), the results could — and very often would — be skewed, sometimes to the point of presenting the exact opposite conclusions.

So yes, weighting is important, and not knowing if this was weighted — it does not appear to be so — pretty much automatically makes this study fugazi from the jump.

But wait, there’s more.

Literally literacy

The headline takeaway: Gamblers have higher financial literacy than non-gamblers.

To be clear, the paper shows this to be true. On a scale of 0-5, gamblers scored 3.93, non-gamblers 3.72. This number comes from a financial literacy test from a 2011 academic paper. So fine, OK, let’s take it at face value.

Awesome, right? We’re so smart!

Except this is a potentially obvious case of correlation not equaling causation, because three pages earlier in the paper, the authors note there is a “significant difference” in income levels between gamblers and non-gamblers.

Furthermore, they found the more educated someone is, the higher their financial literacy. Says so right in the paper.

Those two findings have precedence: Here’s a Cambridge University Press study that shows online sports bettors have higher income levels and more education than non-online sports bettors, and here’s this Yale study that shows casino gamblers … also have higher income levels and more education than non-casino gamblers.

So uh … do gamblers have a higher financial literacy rate, or do people with more income and higher education levels — two segments of the population with higher financial literacy rates — also happen to gamble more?

A cleaner takeaway, therefore, is people with more money and more education — two huge markers of higher financial literacy — also like to gamble.

Big difference.

Segmented

There are five segments in the study: non-gamblers, casino gamblers, iGamblers, sports bettors, and prediction market users. The respondents were asked to identify which bucket they most identified with.

Translation: This is ridiculous.

I’m a casino gambler, an iGambler, a sports bettor, and a prediction market user. I don’t “identify” as any one of them, but if forced to choose the one I spend most time with, it would be “iGambler.” But this is because there’s no DFS bucket, except maybe DFS is “sports bettor”?

Clearly some issues here, which makes, say, comparing a "prediction market user” and a “sports bettor” problematic, as there’s a ton of crossover between the two.

The study points that out, noting 65% of its "prediction market users” traded sports.

But it didn’t stop the AGA — which is at war with the prediction market sector — from using these numbers to point out how prediction market users are dummies.

This was actually the second takeaway in the press release, and the ground it stands on is even shakier than the first takeaway.

From the presser: “Sports bettors show the greatest mathematical confidence and objective capability, meanwhile ‘prediction market’ users expressed nearly as much confidence in their mathematical skills but performed more in-line with non-gamblers when those skills were objectively tested.”

Big takeaway there, eh? Well, the numbers show sports bettors scored 2.8% better on the objective side and 1.9% higher on the confidence side. Furthermore, the p values on the confidence level amounted to .053, which is, by a whisker, considered statistically significant. If it drops by .004 points, it ain’t.

And again, let’s remember: We are comparing self-selected groups here. I say I identify as a sports bettor, you say you identify as a prediction market user, we both probably use both.

The study itself admits as much, though it’s buried in the bottom of the conclusion section, with the authors writing, “Despite the current study’s use of self-reported and cross-sectional data …”

My takeaway? Trash. (That’s self-reported.)

Listen: I’m not exactly saying this study is all garbage, but this — and so many other gambling studies hitting the news cycle lately — aren’t exactly anywhere close to not-garbage.

I’ve written about this plenty of late, and I’ve avoided the obvious kicker, but I can’t anymore. Sorry. It has to be done. Mark Twain, take us home.

“Lies, damned lies, and statistics."

Thank you, Mark.