🤥 Faked Up #20

Researchers say CAPTCHAs are doomed, Thousands of US schools are likely affected by deepfake nudes, and AI slop pivots to Hurricane Helene

This newsletter is a ~5 minute read and includes 39 links.

THIS WEEK IN FAKES

The FCC imposed a $6M fine on the consultant behind the Biden deepfake robocall. South Korea’s parliament passed a bill to criminalize watching deepfake porn. U.S. Senator Cory Booker blocked the Take It Down Act. Stanford invited an Epoch Times editor to moderate a panel on the origins of COVID. Shocker: X still has a bot problem. An Indiana senator may have broken state law with a deceptively altered ad. In Brazil’s local elections, most (disclosed) use cases of AI in political ads involved synthetic jingles. Kamala Harris did not kill a rhino.

TOP STORIES

GOT TO CAPTCHA ‘EM ALL

In a preprint, researchers at ETH Zurich claim that advanced YOLO Models can solve 100% of reCAPTCHAv2 bot-filtering tests (I swear those were all real words). The paper shows that VPN use, mouse movement and user history all affect the likelihood of detection. The authors conclude that “we are now officially in the age beyond captchas.”

On the one hand, great! I won’t miss these capricious and deranged puzzles. But reCAPTCHAv2 is one the internet’s main defenses against automated bots. So this is probably not the best thing to happen just as generative AI unloads hordes of imitation humans in our online spaces.

WORSE THAN IT SEEMS

I have been counting the number of students mentioned in media reports of deepfake nudes in school communities to try and quantify the problem. I had no doubt that my global tally of 530 was a wild underestimate, but now I know just quite how wild.

In a new report, the Center for Democracy & Technology surveyed American public school students in grades 6-12 (roughly speaking, ages 11 to 18). 15% of these students said that they know of a deepfake depicting individuals associated with their school being shared in the past school year.

With 15M students in U.S. public high schools, this suggests the number of deepfake nudes in school settings around the country may be as high as 225,000. Even if some cases were double counted because multiple pupils from the same school took the survey1 and even if half of the students answered falsely, we’d still be looking at tens of thousands of cases. And this would assume that every case only affected one victim (unlikely) and that there were no cases in private schools (impossible).

You should read the whole report, but two other things stood out to me. First, I was surprised to see most students report that non consensual intimate imagery (authentic and deepfaked) is shared primarily via social media, not messaging apps.

Second, ~60% of teachers and students report that their school has not communicated their procedures for addressing deepfake nudes.

THE META TO TELEGRAM PIPELINE

A few weeks ago, some researchers suggested that searching for links to Telegram might be an easy proxy for Meta to detect policy-violating ads.

That advice would have helped in the case of “aitooltool,” which has reached tens of thousands of Instagram users2 with at least 18 ads for an AI undressing bot on Telegram. One of its ads is just a tutorial for the tool:

Others are only slightly less direct, inviting users to the Telegram page for a “different and surprising photo of this girl.”

While I was at it, I checked in on Crushmate. It appears to be trying to circumvent enforcement by creating yet another bogus advertiser profile and using images that are clearly of “AI girls.” But the site it points to — dreamgirl[.]one — is still clearly aimed at nudifying photos of real women. I reached out to Meta just before publication for comment and will update this newsletter online when I hear back.

CLEANING UP WIKIPEDIA

Fascinating review of what AI-generated images are making it into Wikipedia (h/t J. Nathan Matias). This is enough of an issue that there is a dedicated cleanup project on the English-language version of the encyclopaedia. It also has obvious downstream consequences for places like Google, which has over the past years inadvertently featured AI images in visual representations of famous artists and historical events.

Images removed include this trio of scientifically-inaccurate representations:

DEEPFAKE BAITING

US Senate Foreign Relations Committee Chairman Ben Cardin was targeted by a caller posing as Dmytro Kuleba, who was until recently the foreign minister of Ukraine.

Kuleba’s appearance on the likely deepfaked live video call was reportedly “consistent in appearance and sound to past encounters.” The senator became skeptical not because of anything in the appearance of his interlocutor but by the tenor of his questions. The faux Kuleba asked “Do you support long-range missiles into Russian territory? I need to know your answer” and similar politically-charged questions which appeared to set up Cardin for a potentially controversial leak.

HURRICANE HELENE’S AI SLOP

Futurism found that a marine-themed Facebook page that serves as the digital storefront for an Etsy seller of digital prints pivoted to posting AI-generated Hurricane Helene garbage as the natural disaster was hammering the southern United States.

Here’s some comments on one of the pictures representing a fake man saving a fake dog from a fake flood:

• “Wow. Glad to see the 15ft men of the mountains are putting their 6 fingered hands to use rescuing similarly non-existent dogs”
• “BLESS THIS HERO AWESOME JOB 🥰🥰❤️❤️”
• “Thank you for rescuing this furbaby. I always remember that "dog" is "God" and that is why we love them so.”

I honestly can’t tell if that last one is for real or not.

CONDITIONED CONTROL

In an open letter to Science, a group of researchers take issue with a seminal 2023 study on the effect of Facebook’s algorithmic feed on polarization and misinformation. The original study had found that replacing it with a reverse chronological feed actually increased the prevalence of “untrustworthy information” from 2.6% to 4.4%. The letter-writers warn that during part of the period studied, however, Facebook’s algorithmic feed was subject to 63 “break-glass” changes to protect the 2020 election. The complainants’ theory is that this affects the validity of the findings because in that overlapping timeframe, the overall reach of untrustworthy news on Facebook decreased by 24%.

Science deemed the letter noteworthy enough to attach this editorial note, but is not adding a correction because “the authors made clear in the original paper that these kinds of changes could have affected the interpretation.”

NOTED

1. Social media misinformation putting women off contraceptive pill – NHS chief (The Independent)
2. OIG audit calls for more clarity from CISA, DHS on disinformation mission (Cyberscoop)
3. Fake Sites, Fake Authors: Iranian Network Pushes AI-Generated U.S Commentaries Designed to Polarize (NewsGuard)
4. Measuring receptivity to misinformation at scale on a social media platform (PNAS Nexus)
5. China’s Plan to Make AI Watermarks Happen (Wired)
6. TikTok Videos Spread Misinformation to New Migrant Community in New York City (The Markup)
7. Social Media Bot Policies: Evaluating Passive and Active Enforcement (arXiv)
8. The Lies Russia Tells Itself (Foreign Affairs)
9. All the major tech bills California’s governor signed and vetoed (WaPo)
10. "I don't trust them": Exploring Perceptions of Fact-checking Entities for Flagging Online Misinformation (Collaboratory against Hate)
11. Is This The Real Life? Is This Just Fakery? (Ctrl-Alt-Speech)

1 I reached out to CDT who confirmed that while this is technically possible, the survey used a nationwide panel so the chances are low.

2 That’s based on the data from the ads that ran in the EU, where Meta provides additional transparency.