🤥 Faked Up #13

Perplexity AI is stumped by hoaxes from outside the US, Apple and Play allow a faceswapping porn app that targets celebrities, and misinformation adds a communal twist to violent events in Bangladesh

This newsletter is a ~7 minute read and includes 66 links.

TOP STORIES

PERPLEXED AI

Perplexity AI is reportedly growing in popularity, so I’m monitoring the AI-powered search engine’s responses to known misinformation. It generally does well on US falsehoods like the second shooter conspiracy theory:

But the tool was not as capable in a few tests I did with misinformation that spread outside America. For instance, Brazilian fact-checker Aos Fatos has debunked the theory that Simone Biles-defeating gymnast Rebeca Andrade made an “L” sign in support of President Lula at the Olympics. Yet Perplexity claims the opposite is true:

Aos Fatos executive director Tai Nalon told me she couldn't replicate my result in Portuguese but noticed that on politically-charged queries the search engine cited as sources "outlets with clear ideological biases [that] have published misinformation."

Equally worryingly, the query [an Algerian boxer changed sex olympics?] in Portuguese resulted in Perplexity mischaracterizing gold medalist Imane Khelif as trans.

The search engine also returned, in Greek, a debunked hoax about boxer Gianmarco Cardillo “feeling like a woman” and challenging Khelif to a fight.

These three examples are not a systemic review. But they are a hint that, in a tale as old as tech, Perplexity AI may not be as reliable outside of the English-US context.

👋Hello! Faked Up #12 led Google to block several ads for AI nudifiers. If you’d like to make this type of change possible and keep up with the misinfo beat, consider upgrading to a paid subscription to access the full newsletter! (Anyone can reach out to get 6 months free, no questions asked.) If you’re already a paying subscriber: Thank you and read on 🙏🤘

HIJACKING THE NARRATIVE

Misinformation is adding a "false communal twist” to the political limbo in Bangladesh. For example, AFP reports that a viral post “falsely claimed that over 500 Hindus had been killed, hundreds of Hindu women raped and dozens of temples burned to the ground.” Events of political violence such as a mayor being pelted by rocks or students being beaten up have also been falsely recast as attacks on the country’s Hindu minority, according to the IFCN-certified Bangladeshi fact-checker FactWatch. BBC Bangla and The Quint note that misleading accounts have often been spread by and for users in neighboring India.

THE ELECTION MISINFO ISSUE

Public Opinion Quarterly dropped a special issue on election misinformation. I was particularly interested in two papers analyzing how consumption of and trust in news media affects belief in misinformation, which the good folks at RQ1 helpfully summarized. Here’s how they present the main takeaways from the two papers:

1. “All things equal, the more exposure that people had to news from ‘legacy news brands,’ the less likely they were to believe in electoral misinformation … The researchers also looked at the role of digital platforms generally, and found no obvious effects on belief in electoral misinformation.”
2. “Trust in traditional news media reduced people’s misperceptions about election integrity while trust in social media increased their misperceptions. Importantly, though: ‘These effects depend on the level of media freedom: in countries with low press freedom, the traditional media effect was significantly smaller, and for social media, the effect is even reversed.’”

IT’S CROWD-CHECKING SEASON AGAIN

Men have been exaggerating their crowd sizes for at least as long as I’ve been on the misinformation beat. But Donald Trump is a particular fan of the practice, and he was back at it last week claiming his biggest crowd drew the “same number of people, if not more” as MLK did during the March on Washington for Jobs and Freedom. PolitiFact has a good fact check on this that pushes back on the claim while recognizing it’s not particularly clear which speech Trump is referring to.

The very 2024 spin to the crowd size argument is, of course, generative AI. On Sunday, Trump claimed that a crowd of many thousands at the Harris-Walz rally near Detroit was “A.I.’d” and that the supporters “DIDN’T EXIST!” This is nonsense, but as the always perceptive Michael Caulfield notes, it should also be interpreted as “copium” from a candidate struggling to readjust to the reality of a newly competitive race. That is different from misinformation meant to convince people.

While there’s only so little you can do to combat copium, we can get smarter about countermeasures to these lies. For one, we should favor credible first-hand evidence over sloppy AI detectors that lead to paragraphs like this one (h/t Jacob Silver):

Secondly, while I have been skeptical about watermarking as an antidote to AI misinformation, this incident certainly makes a compelling case for truth-based campaigns to start documenting their events with a technical standard like C2PA. That way, they can immediately release credentialed photographs and give reporters covering the controversy definitive evidence for a rebuttal.

“EMMA STONE THIS TIME!”

Last week’s Faked Up led Google to take down several Search ads for AI undresser apps (thanks also to 404 Media’s coverage).

This week, I’m hoping Google and Apple will review the FaceHub app available on the App Store and Play Store. Much like other faceswapping tools, the app presents as playful on these better-moderated platforms, but markets itself openly as a generator of non consensual pornographic imagery on the lawless confines of Telegram.

GIFs shared in the Facehub Telegram channel show porn clips overlaid with a menu of faces to insert in the video, including those of Emma Stone and Kim Kardashian. Other messages encourage users to “swap the face of any sex videos to your friend, your friend’s mother, your step-sister, or your teacher” before linking to the apps.

I am pretty certain this behavior violates Apple and Google policies but I haven’t heard back from either company since reaching out on Monday morning.

A MUSK OF MISINFORMATION

According to the Center for Countering Digital Hate, Elon Musk has posted 50 misleading claims about the US election on X this year, none of which were corrected by a Community Note. On Thursday, Musk also quote tweeted (and then deleted) a literal fake news headline about British detainment camps for far-right protesters.

On Monday, the X CEO welcomed Donald Trump for a conversation where the Republican candidate “made a number of questionable and false claims”. The Trump discussion was delayed because of technical glitches that Musk (likely falsely) claimed were an organized attack on his platform. This gave crypto scammers on YouTube a window to run a deepfake livestream that reached at least 200,000 viewers.

DON’T TRUST LLMS WITH YOUR SUMMARIES

Researchers at UMass Amherst and Mendel AI got 15 clinicians to assess the summarization skills of GPT-4o and Llama-3. The raters found hallucinations in almost all of the medical note summaries. Inaccuracies included replacing the term “edema” with “leg swelling” (the latter is a subset of the former).

REAL-TIME DEEPFAKE WATCH

The github repository Deep-Live-Cam has gone viral as people both marvel at it and fret about the self-evident risks of a tool that enables impersonation in live video. Tutorials I watched on YouTube seemed to imply audio sync was still challenging, but the video quality alone makes for a worrying proof of concept.

My go-to deepfake expert Henry Ajder is also worried; here’s his take:

Just a few years ago, these tools were awkward to use and the outputs were typically of a low fidelity. Today, the landscape looks very different:

1️⃣ Occlusion, lighting consistency, and 3d face profiling are impressive in Deep Live Cam compared to other live faceswapping models, although the results aren't perfect- note the glasses 'popping' in and out.

2️⃣ Unlike some older models, Deep Live Cam requires just one image of a target to create a high quality live faceswap. Data aggregation, alignment, and model training were previously significant barriers to access.

[…]

Crucially, this project is open-source and will likely be incorporated into no code platforms for launching models, making it even easier to deploy for non-technical users.

Headlines

1. These rally attendees do not exist (Conspirador Norteño)
2. ELEIÇÕES 2024: Falta da regulação de IA dificulta decisões da Justiça sobre deepfakes (Desinformante)
3. TikTok push alerts include fake news and weeks-old tsunami warning (FT)
4. Children to be taught how to spot extremist content and fake news online (The Guardian)
5. The Fake Indian Cricket League Created for Real Online Betting (Bloomberg)
6. Scientists are falling victim to deepfake AI video scams — here’s how to fight back (Nature)
7. The ad companies making money off of obituary spam (The Verge)
8. British businesswoman accused of being the first person to share false information about the Southport stabbing suspect wrongly being an asylum seeker says she is 'mortified’ (Daily Mail) with Explainer: Is spreading misinformation punishable under the Online Safety Act? (A.M. News) and The real story of the news website accused of fuelling riots (BBC News)
9. Polish billionaire plans to sue Meta over fake advertisements (Reuters)
10. Wyoming reporter uncovers competitor using AI-generated quotes (Poynter)
11. El Pizzagate de la mujer de Zelenski y otros bulos transfronterizos que usan a mujeres y niños para deslegitimar a Ucrania (Maldita)
12. Foreign TikTok Networks Are Pushing Political Lies to Americans (WSJ)
13. AI deepfakes of Prince William and Keir Starmer used to sell scam (The Times)
14. Jobhunters flood recruiters with AI-generated CVs (FT)
15. Accuracy and Political Bias of News Source Credibility Ratingsby Large Language Models (Kai-Cheng Yang and Filippo Menczer)
16. Russian Deep Fake: “Obama” Admits Dems Planned Trump Shooting (NewsGuard)
17. Beijing-based 'Green Cicada' AI network uncovered on social media, fears of US election disruption (ABC News)

Faked Map

You probably know Gerardus Mercator from the popular projection (you may even be a hater, if you watched enough West Wing).

But did you know that the same fellow who gave us Greenland the size of Africa was also convinced by a dubious letter that the North Pole was marked by a big black cliff? The rock, possibly magnetic, was surrounded by four chunks of Arctic.