🤥 Faked Up #4

Google's AI looksmaxxing, Telegram fact checks, and the Scammers Who's Who

What’s up, y’all?

Faked Up #4 is brought to you by glue-free pizza and 100% innocent fork-bending magicians. The newsletter is a ~7-minute read and contains 61 links.

Top Stories

[AI Overmews] Last week’s launch of AI-generated summaries on the US-English version of Google Search did not go great. The feature told people to put glue on pizza, eat rocks, and stare at the sun. It claimed JFK graduated the University of Wisconsin 6 times, most recently in 1993.

More worryingly, AI Overviews misidentified poisonous mushrooms, botched the remedies to a rattlesnake bite and claimed Barack Obama is Muslim.

Some of this is down to the way queries were formulated. Like an artificial puppy, AI Overviews eagerly returned stuff that matched bizarro queries even if the source was Reddit maestro fucksmith or satirical newspaper The Onion.

This is not a new problem. Seven years ago I wrote about “featured snippets,” Search’s longstanding summarization experience, struggling with a query about the first Black president of the United States.

AI struggled with a similar query last week, despite being reportedly bigger than fire.

To assess AI Overviews beyond one-off gaffes, though, I thought I’d explore multiple queries on a specific user journey.

I looked for a topic seemingly well-attuned to the triggering logic of AI Overviews, which appeared to rely heavily on forum content and prioritize “how-to” responses. The topic also had to be one where online content is of mixed quality and getting the wrong result could lead to somewhat harmful consequences.

I settled on a teenage boy looking for information about “looksmaxxing,” a catch-all buzzword for beauty practices. These range from basic hygiene to aggressive methods that verge into body dysmorphia like under-eating, steroids and surgery.

One of the most popular elements of looksmaxxing is “mewing,” which is not something a cat does, but a tongue posture that allegedly makes your jawline more masculine. The method is named after a controversial father-son duo of British orthodontists. It is not widely endorsed; in a memorable turn of phrase, the American Association of Orthodontists (AAO) says that “scientific evidence supporting mewing’s jawline-sculpting claims is as thin as dental floss.”

So, how did AI Overviews handle a mewing-curious teen1? The result for [looksmaxxing techniques] was a mixed bag, highlighting a questionable app that rates your masculinity based on a selfie while noting that mewing is “not scientifically supported but is said to improve jaw structure.”

[best mewing techniques for 16 year old boy] went straight into how-to mode and recommended “20-30 minutes” of mewing a day.

[benefits of mewing] was a bit better, though I don’t love the both sides approach that opens with “some people” endorsing a range of benefits and only after a scroll flags what experts have said.

Something similar occurs for [does mewing work at 13], though the warning here was attributed and clear.

AI Overviews also went deep on product recommendations for [best mewing products].

Finally, I was able to trigger a few responses for bone smashing, which is generally assumed to be a troll rather than something looksmaxxers actually do.

I shared all these looksmaxxing results with David Scales, a physician, sociologist and assistant professor at Weill Cornell Medicine. First, he noted that this is a lightly researched topic which doesn't lend itself to empirical study. Mewing is likely low risk, but also low certainty.

The bigger question, according to Scales, is "how should AI provide guidance on behaviors in the setting of scientific uncertainty?" In an "evidence-free zone" like mewing, specialty societies like the AAO typically avoid making recommendations. The AI Overviews, instead, "seem to be both-sidesing it."

And this, I think, is the crux of the matter.

Many AI Overviews bugs were embarrassing but fixable. The plan for a fucksmith-free product appears to be to turn up the dial on source quality and show the feature less.

The deeper issue is how sources are combined in an intelligible whole. Rather than “purée results,” Google’s AI summaries should help users understand the relative trustworthiness of the sources behind a topic.

That is Google’s killer app. That’s what its awesome Search quality teams are set up to deliver. Everything else is just looksmaxxing.

[Telegram’s state fact-checking] Updates in Telegram’s source code suggest that the messaging platform will allow governments to appoint “agencies that verify information” (presumably like India’s PIB FactCheck). The resulting “fact checks” would be appended to messages. There’s nothing to worry about here because Telegram is completely free from state-backed disinformation.

[Brazilian fake news] The Brazilian Congress failed to overturn a Bolsonaro-era veto on a law criminalizing fake news about the electoral process. Aos Fatos reports that an active online campaign to cast the law as an instrument of censorship preceded the vote.

[Stop the spread] The OSoMe crew at Indiana University dropped a new paper seeking to define and identify misinformation superspreaders on Twitter. The researchers first isolated almost half a million accounts that shared content from sources on the Iffy+ list. Then, they identified the most influential based on their number of retweets and a repurposed h-index, finding that these are far better predictors of influence than an account’s bot score. They conclude that “just 10 superspreaders (0.003% of accounts) were responsible for originating over 34% of the low-credibility content” between March and October 2020.

[Clamping honesty to the max] At an AI happy hour I attended last week, everyone was buzzing about this research paper by the captivating title “Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet.” I think what this means is Anthropic claims to have isolated certain features behind the reasoning style of their LLM that “respond to and behaviorally cause abstract behaviors.” Wired has more on the paper, but the bit that caught my eye is one of the behaviors is deception.

Anthropic says that tinkering with features can change model behavior at a deeper level than most safety guardrails. For example, increasing the weight of the “honesty” feature stopped Claude from lying about forgetting a word.

Inversely, increasing the secrecy and discreetness feature drives the chatbot to reason about how to “find a way to cover my tracks without revealing the truth.”

[Will DEFIANCE defy Congressional stasis?] Politico reports that a draft bill in the US Senate to criminalize deepfake pornography is getting push back from tech lobby group NetChoice. The issue allegedly comes down to whether the creation and possession of nonconsensual AI sexual content should be penalized as well as its distribution. FWIW, my take is if legislators can’t even ban nudifiers (see FU#1), I’m not sure what will ever make it through Congress.

[Magician walks free] The political consultant and voice provider behind the vote-suppressing AI-generated Joe Biden robocall got an $8 million fine. The fork-bending magician who created the audio claimed he had no idea it would be used for evil and is off the hook. Still, American political consultants told the NYT that generative AI is kind of a nothingburger for now. So I turn to you, dear readers. Will deepfakes be a key element of the 2024 disinfo scenario, or nah? Vote!

Loading...

[Tragedy of the tools] This article on Nieman Lab very concretely lays out the challenges Indian fact-checkers face trying to assess whether something is a deepfake. Typically, these publishers lack the technical know-how to build their own detectors, which means they have to rely on third-party tools. Those provided by the private sector can be expensive and its makers wary of assessing political content. Detectors provided for free by academia can be hard to access. Both types can be inaccurate.

That feels like useful context for this Rest of World analysis that found that most tip lines managed by Indian fact-checkers failed to respond to their requests to verify 15 AI-generated videos and photos that the website submitted as a test.2

[Scammers Who’s Who] Interesting graphic from the US Federal Trade Commission. Based on 2023 Consumer Sentinel Network data, Best Buy/Geek Squad, Amazon and PayPal were the businesses most frequently impersonated by scammers. Microsoft, Publishers Clearing House and Amazon topped the list in terms of the overall dollar amounts scammers were able to extract by impersonating them.

Headlines

1. The disinfluencers: How over 150 anonymous 'Irish' accounts are swamping X with extreme views (The Journal)
2. TikTok says it removed an influence campaign originating in China (WaPo) with TikTok Moves to Limit Russian and Chinese Media’s Reach in Big Election Year (NYT)
3. Study Finds That 52 Percent of ChatGPT Answers to Programming Questions Are Wrong (Futurism)
4. How a Chinese Influence Operation Is Targeting Pro-Trump Communities Online (Lawfare)
5. Trump falsely claims Biden authorized FBI assassination attempt (AFP)
6. Anti-fake news or anti-free speech? The debate over Punjab’s new defamation law (Dawn)
7. Perfil no TikTok tira proveito da tragédia no RS e viraliza com dublagens falsas de celebridades (Aos Fatos)
8. Does provenance build trust? (BBC News Labs)
9. RFK Jr. Is Even Crazier Than You Might Think (Mother Jones) with Inside the fringe worldview of RFK Jr.’s VP pick (NBC News)
10. Pro-Kremlin Propaganda about Neo-Nazis in Ukraine Targets South Africans Online (Institute for Strategic Dialogue)
11. AI disclosure required in campaign ads, FCC chair says (Reuters)
12. Online conspiracies proliferate following the attack on Slovakian PM Robert Fico (DFRLab) with Russia’s Misinformation Machine Targets Slovakian Assassination Attempt (Bloomberg) and Vysvetľujeme špekulácie, ktoré naznačujú, že atentát na premiéra bol zinscenovaný alebo sa stal inak, než hovoria médiá a polícia (Demagog SK)

Before you go

Katie Notopoulos took one for the team and made the glue pizza, I can’t.

This one is worse than glue pizza. Do not try at home, kids.

I wonder what this will be used for!

Grok is good, actually.

1 All of the searches in this section were conducted on Thursday May 23 using Google via the Safari browser on an iPhone. Not all them could be reproduced this morning before I send the newsletter — generally, AI Overviews appears to trigger less — but I was also able to get some new ones like [benefits of mewing on face] and [benefits of mewing on jawline]

2 After I sent the newsletter, Rest of World retracted this story, claiming it “did not meet our editorial standards for accuracy and journalistic integrity.” The fact-checkers who run the tiplines claim several of the queries submitted by RoW fell foul to spam guidelines. I’m keeping my summary of the story striked out in the text for transparency.