🤥 Faked Up #22

Hurricane season misinfo is worse than usual, TikTok served me 59 conspiracy videos in a session of 100, and a fake Justin Bieber clip does the rounds again.

This newsletter is a ~6 minute read and includes 53 links.

HOW AM I DOING?!

👋 Hello! I have been sending Faked Up for almost 6 months and I’d like to get a sense of how it’s landing.

There’s only so much I can glean by (obsessively) reviewing open and click rates, so I’m asking you to answer five quick multiple choice questions (plus an open-ended one if you’re feeling chatty).

Thank you! 🙏 🙏 🙏 🙏

Please take my survey it's going to be really fast I mean it THANK YOU!!

HEADLINES

WhatsApp wants to add a “Search the web” button to fight misinformation. South Korea is one signature away from criminalizing the creation and consumption of synthetic non consensual intimate imagery. Millions are using Telegram’s nudify bots. Italian Prime Minister Giorgia Meloni testified in a case against the creator of a deepfake porn featuring her likeness. Meta thinks mundane fakes of the Northern Lights are a good use case of its AI image editor; anti-Muslim propaganda is a more likely use case. Reality Defender is pitching a real-time deepfake video detector. OpenAI terminated naughty state actor accounts.

TOP STORIES

STORMS OF MISINFORMATION

Hurricanes spawn misinformation. They are frightful events with dramatic images that are hard to independently verify. Communities engage in rumor-laden collective sensemaking. There’s a reason Highway Shark is a misinfo meme.

Even so, this Atlantic hurricane season feels different.

Conspiracy theorists claimed Helene and Milton were geo-engineered to grab minerals or eliminate Republican voters.

The same folks who reject the scientific consensus that says human actions on aggregate cause climate change are trying to mainstream the idea that individual humans can literally create and direct extreme weather events.

They may be breaking through: NewsGuard claims that in the two weeks to Oct. 9, “174,000 articles and social media posts mentioned the hurricanes alongside terms such as ‘geoengineered,’ ‘manipulated,’ and ‘weather weapons.’”. Even if we generously assume that half of those are rebuttals to the conspiracies, that’s a 15,000% increase on the conversation about this topic in the two weeks prior.

True believers are taking a victory lap. In a recent YouTube video, Dane Wigington claimed “the awakening to the ongoing climate engineering operations taking place over our heads is now accelerating by the day.” Incidentally, Google is still hosting “The Dimming,” Wigington’s debunked documentary, and running ads for it.

Somehow these man-made targeted hurricanes were also not destructive enough. Here’s Wigington again: “it seems that the weather-makers decided to change the weather script. Was Milton’s impact anywhere near the level that Matrix media sensationalized all week long?” (“Milton overhyped” was the second suggestion I got on TikTok when searching for the Hurricane earlier this week.)

As Kate Starbird put it on Lawfare: “Just like the heat of Gulf of Mexico has supercharged the storms, the dynamics of our information platforms are supercharging the rumoring and conspiracy-theorizing that’s happening around these events.”

These dynamics now include a functionally useless Twitter.

During Hurricane Irma, I evacuated St Pete and monitored the platform for updates to share with my friends still in Florida. At the time, it was a great place for a moderately savvy user to track live events; now, there is no signal among the noise. As John Herrman writes, the service Twitter provided in the mid-2010s “was one of those small golden eras you didn’t realize you were in.”

IRL OR IT DIDN’T HAPPEN

Here’s two more good reasons you should now operate with a zero trust model when interacting with most online users.

On Reddit, user lotrfan2004 shared a live deepfake video of himself as a completely different person. He claimed this was “extremely easy” to make with Runway and ElevenLabs. (Here’s another recent-ish and believable example.)

Meanwhile on Threads, the generally AI-optimistic Wharton professor Ethan Mollick warned that “You really, really should not trust audio clips anymore” based on a deepfake of his own voice generated from a mere 10-second recording.

OUR HALLUCINATIONS ARE ON YOU

LinkedIn is updating its user agreement to include this banger of a paragraph offloading the responsibility for any inaccurate content created by its own generative AI features to the end user that posts it:

Generative AI Features: By using the Services, you may interact with features we offer that automate content generation for you. The content that is generated might be inaccurate, incomplete, delayed, misleading or not suitable for your purposes. Please review and edit such content before sharing with others. Like all content you share on our Services, you are responsible for ensuring it complies with our Professional Community Policies, including not sharing misleading information.

I made this infographic to explain the minutiae of what’s going on here:

FLAT EARTH PRIMING

For the latest installment of “Alexios wrecks his algorithmic feeds,“ over the past few weeks I have been casually searching for Flat Earth videos on TikTok. My goal was to get a sense of how easily the platform could be turned into a conspiracy conveyor belt. Turns out: Very easily!

On Monday, I took note of the first 100 videos on my For You Page over a single session. Fully 24 were Flat Earth videos. Another 35 videos were about other conspiracy theories. That means that with minor priming, TikTok fed me 59 videos unhinged from reality out of 100. And I’m not alone watching these videos, which had a cumulative 36.3 million views at the time of writing.

Let’s sample some of the material I got to enjoy:

Flat Earther videos included claims that discrepant satellite imagery of the Strait of Hormuz prove NASA doesn’t have pictures of the Earth from space, that astronauts actually operate underwater, and that Nixon’s phone call to the Moon was a hoax.

Several other conspiracy theories focused on the Tartarian Empire, but there was also QAnon content and a video claiming (I think seriously?) that Pete Davidson served as Joe Biden’s body double.

I repeated this exercise on Tuesday with a smaller sample and results were similar. Six out of the first 15 videos I saw promoted Flat Earth-related theories. An additional one was about an alleged alien landing in D.C.

‘FULL OF MEN IN THE SAME SORT OF MINDSET’

Australian high school teacher Hannah Grundy discovered that the person who'd posted dozens of deepfaked non consensual intimate images of her on a online messaging board was a close friend of hers. The man, Andrew Hayler, had been on holidays with Grundy before she discovered he was behind the thread “The Destruction of Hannah.”

The posts included Grundy’s full name, social media handles, and the name of the suburb she lived in. Grundy wasn’t the only target; in total, Australian police found folders with photos of dozens of women Hayler knew.

The case played out over the past two years, but it was only in June that Hayler was finally sentenced.

In court, he apologized for his actions saying “I was living in this weird messed-up fantasy, not thinking about the consequences of my actions … I think I believed that no-one would ever really see it.” More worryingly, he said the website he posted the pictures on was "full of men in the same sort of mindset."

DOWN BUT STILL DISINFORMING

X paid up and is now allowed in Brazil again. But according to Aos Fatos, even while the platform was blocked in the country, posts spreading a false document alleging São Paulo mayoral candidate Guilherme Boulos was hospitalized received more than 2.6 million views. One of the posts was from a now-elected councilor in the southern city of Curitiba who posted in violation of court orders, presumably through a VPN.

UNBELIEBABLE

“Lost myself at a Diddy Party / Didn’t know that’s how it’d go / I was it in for a new Ferrari / But it cost me way more than my soul / Wasn’t worth all the fortune and fame.”

These are the supposed lyrics of a leaked Justin Bieber song about his encounters with Sean “Diddy” Combs, who is in jail on charges of racketeering and sex trafficking.

As the relationship between Combs and a young Bieber gets revisited, videos of what sounds like Bieber singing the song went viral again. Two clips posted in the past few weeks have alone collected 26 million views on TikTok (h/t to my friend Brian who told me about this).

Several audio detection tools have found the song to be likely AI-generated, including Pindrop, Ai-Spy and Resemble. Bieber has not commented on the video just as he has generally avoid speaking of Combs more generally.

Probably the most convincing piece of evidence for me is that a version of this song has been circulating since April on the YouTube account of a gamer with a side interest in Diddy and Andrew Tate.

In that video, which is labeled “AI” by the creator, a line that in more recent videos has been either cut off or transcribed as “All the girls never walking the same” is rendered as “All that girth, never walking the same." While the recent version is perhaps passably ambiguous for a pop song, the earlier one is improbably lewd. Given that the automated captions on the April video also return “girls,” my assumption is that the original fake has transmogrified into this somewhat more presentable one through laziness rather than deceit.

(AI-generated songs have generated their fair share of controversy: In March of last year, a scammer made thousands of dollars selling fake “leaked” Frank Ocean songs; in September another one was charged in a $10M unlawful royalties scheme involving AI-generated songs and bot accounts.)

NOTED

1. Who should fight the spread of fake news? (arXiv)
2. Anti-vax conspiracies spread about a young TikTok star's death. She previously went viral for spreading anti-vax conspiracies. (disruptionist)
3. Unverified “Grooming” Accusation Against Walz Promoted by Russian Disinformation Operative (NewsGuard)
4. Some of the Web’s Sketchiest Sites Share an Address in Iceland (NYT)
5. What are digital arrests, the newest deepfake tool used by cybercriminals? (Al Jazeera)
6. 7 complaints of deepfake porn targeting female soldiers filed: civic group (The Korea Times)
7. This threat hunter chases U.S. foes exploiting AI to sway the election (WaPo)
8. Deepfakes and falsehoods are legal in political advertising. Not everyone is on board with fixing it (ABC News)
9. A firehose of antisemitic disinformation from China is pointing at two Republican legislators (WaPo)
10. AI Sludge Local News Site Hoodline Falsely Accuses San Mateo DA Of Murder (TechDirt)