27 Nov 2024 8 min read

44 of the top 100 Bluesky accounts have a duplicate

🤥 Faked Up #28: BlueSky has a doppelgänger problem, World War III narrative leans on a fake cover of The Economist, and Instagram is flooded with cozycore AI slop

Hey look, Ma! Faked Up’s findings are cited in this letter by 26 members of the U.S. Congress urging platforms to do more about non consensual deepfake nudes.

This newsletter is a ~6 minute read and includes 57 links.

HEADLINES

Australia’s government gave up on its proposed bill against online misinformation. The UK Parliament launched a new inquiry on “social media, misinformation and harmful algorithms” and seems to believe Elon Musk will show up. An analysis of health-related sponsored content on Brazilian media outlets found that the vast majority of these ads were misleading. The New York Post’s front page on Saturday equated misinformation research with censorship. UNESCO thinks creators should get fact-checking training. A high school student in Florida who shared deepfake nudes of at least 30 of her classmates was charged of a third-degree felony; her ex-boyfriend, who created the images, walked free.

TOP STORIES

BLUESKY’S DUPLICATION DILEMMA

In last week’s newsletter, I speculated that impersonation may be one of BlueSky’s most immediate challenges on the disinformation front.

I can now ground that take in a little bit more data. Over the past week, I manually searched for any accounts impersonating the top 500 BlueSky users 1 by follower count.

Given that it’s not always possible to ascertain that the bigger accounts themselves are legitimate — e.g. Ryan Reynolds and Arnold Schwarzenegger are in the top 500 with accounts that barely posted — I will refer to the less-followed copies of popular accounts as “doppelgängers” rather than impersonators.

Of the 305 accounts representing a named individual, at least 74 have at least one doppelgänger (paid subscribers can access the data here 2). Because many have more than one double, I found 140 doppelgängers in total.

Among the top 100 most followed named individuals, fully 44% have at least one duplicate. Most are cheap knock-offs of the bigger account, down to the same bio and profile picture. Only 16% of the duplicates that I reviewed had an “impersonation” label.

These are on average small accounts: the median doppelgänger has 90 followers. But consider that at the time of writing, having more than 85 followers puts you in the 95th percentile of most followed users on BlueSky.

For now, this seems like it’s cheap engagement bait. More popular accounts are more targeted, regardless of whether they are better known IRL. Most of the named individuals who got one of the 10 most reposted skeets on each of the days between Nov 21 and Nov 25 have at least one duplicate.

But sloppy verification is an early signal of broader deception and catnip for organized disinformation actors. Kamala Harris, who isn’t even on BlueSky, at one point had 20 impersonator accounts.

Anecdotally, BlueSky appears to realize the importance of this issue. Over the past week, semi-viral posts by impostor accounts for Vermont Senator Bernie Sanders and Trump universe gajillionaire Bill Ackman were relatively swiftly deleted.

BlueSky COO Rose Wang has promoted the feature that allows users to claim a handle tied to a domain they own as a verification mechanism.

But it is a mechanism that is relatively simple for impersonators to circumvent. Media analyst Kat Abughazaleh, who was in my sample of BlueSky’s top 500 users, posts on BlueSky with the handle @katmabu.bsky.social. She confirmed to me that she does not own the handle @katmabu.com, which belongs to an impostor who went to the trouble of creating a custom domain in the hope of impersonating her.

This is not a sustainable setup for a platform that hopes to overcome Threads and X.

BlueSky CEO Jay Graber discussed the issue in a live stream on Monday, suggesting the platform might eventually ship its own verification feature while also allowing other services to provide their own.

In the interim, perhaps we’ll have to follow Sky News presenter Sophy Ridge’s lead and verify via a selfie-with-a-unique-handwritten-note.

MANUFACTURING EVIDENCE

On the eve of legislative elections in the Indian state of Maharashtra, the X account for the governing Bharatiya Janata Party posted four recordings purporting to prove that leaders of its rival parties were using ill-gotten bitcoin to fund their campaign. Fact-checkers at BOOM claim that at least three of these audio clips were AI-generated.

Despite several tools and experts consulted by the collaborative Misinformation Combat Alliance's Deepfakes Analysis Unit agreeing that the notes were likely synthetic, the BJP’s post remains live on X, with no visible Community Note.

WHAT’S UP WITH WORLD WAR III?

A fake cover of The Economist purporting to dramatically warn about an upcoming nuclear showdown between Russia and the United States was widely shared last week across several social networks, including by pro-Kremlin Telegram accounts like Golos Mordora (h/t NewsGuard).

Through a quick reverse image search, I found at least 38 different tweets sharing the fake cover. The tweets, which had more than 200,000 cumulative views, were all accompanied by commentary condemning the Biden administration’s choice to provide Ukraine with long-range missiles that can be used to hit targets inside Russia.

This narrative is not limited to accounts sharing fake Economist covers: last Friday, podcaster Joe Rogan made the same argument. Two days later, Ukrainian former boxer Wladimir Klitschko, who is also the twin brother of Kyiv’s mayor, called Rogan out for “repeating Russian propaganda.”

MODERN DAY HEARSTS

Speaking about geopolitical hot spots and questionable sources… AFP uncovered “a coordinated network of dozens of Facebook and YouTube channels that direct users to a bogus news website” that runs sensational and misleading articles claiming the Philippines and China are on the brink of war. (There are dozens of fact checks about the maritime dispute between the two countries.)

The network “appears to use artificial intelligence (AI) to rapidly churn out unfounded claims for advertising revenue” and has a combined following of more than 10 million people. AFP wasn’t able to attribute the network to state actors, but did track down a Thailand-based manager of some of the pages, who said his team does "not have a deep understanding of these issues" but focuses “on the potential virality that the posts can attract” in order to earn money through ad placements.

‘HOW DO I SEND YOU $7,000?’

Aspiring immigrants into Canada have been targeted by a deepfake video of Max Chaudhary, a Toronto-based immigration lawyer. Chaudhary found out because someone called his office to arrange the payment of $7,000 that they had discussed with his impersonator via WhatsApp.

COZYCORE AI SLOP

Big thanks to my wife for flagging another niche of Instagram overcome with AI slop. Accounts with millions of followers are mass-posting AI-generated reels of cozycore interior design. In what I’ll admit is an eye-catching touch, they tend to animate the televisions in the image with short snippets from the classic cartoon Tom and Jerry.

I found ten related accounts3 that are likely run by the same person(s). Collectively, they have more than 33 million followers. Many evoke Los Angeles in their name but then add random pairings of locations like #chicago with #texas or #hawaii with #arizona to their posts. None of their reels are currently labeled as AI-generated.

Several of these pages make money by linking out to the e-commerce websites thestorche[.]com and thevoxar[.]com. These websites also appear to rely heavily on generative AI. Their banner has been promising a sale that ends on midnight for the past three days now and their top products include a deranged Santa-themed bubble blower. You can also purchase PleasurePeak Oral Drops that come endorsed by a “Dr Michael Dawson” whose photo is taken from a stock image site.

NIST’S GROWTH MINDSET

The US National Institute of Standards and Technology released an updated version of its report on reducing risks posed by synthetic content. The most helpful elements are those that NIST politely calls “opportunities for further development.” These are some that had me nodding:

On detection, NIST says we need to improve performance “on synthetic content that was post-processed or corrupted by noise, transmission, compression, or reformatting”
On labeling, NIST argues “more extensive and thorough sociotechnical research and evaluations” are needed “to understand how people interact with digital content transparency information across demographics, technical systems, and environments. This will help to inform not just labeling, but also the design of technical interventions that feed into the labels.”
On AI-generated non consensual intimate imagery, NIST notes that “conducting red-teaming to determine whether a model can generate AIG-CSAM and AIG-NCII may result in the generation of CSAM, which may cause legal barriers and complexities for providers, depending on applicable laws“ and argues that new tools and approaches should be developed and assessed.

NOTED

Documenting the Assault on Disinformation and Hate Speech Research (Tech Policy Press)
Dear Senators Klobuchar & Lujan: Now Do You See Why Letting HHS Censor “Misinformation” Is A Terrible Idea? (Techdirt)
Election Deniers Got What They Wanted. They’re Still Going (Wired)
RFK Jr. Said His Organization Funded Viral Covid Conspiracy Film ‘Plandemic’ (Rolling Stone)
ChatGPT was tasked with designing a website. The result was as manipulative as you’d expect (Fast Company)
How Hank Green Makes the Truth Go Viral. Plus, the Escape Fantasies of the Uber Rich. (On the Media)
Seeing Through a GLASSBRIDGE: Understanding the Digital Marketing Ecosystem Spreading Pro-PRC Influence Operations (Google Threat Intelligence Group)
Accuracy prompts protect professional content moderators from the illusory truth effect (PNAS Nexus)
Yes, That Viral LinkedIn Post You Read Was Probably AI-Generated (Wired)

To add my two cents to the “BlueSky echo chamber” discourse, I will note that the top 500 accounts were an eclectic combination of Brazilian society, manga and pet accounts and very online US academics. That said, the slant of US accounts with a political angle in the top 500 was predominantly left-leaning.

I will provide access to the sheet to anyone whose name/email matches a paid subscriber account. If I can’t verify that, I’ll ask you to confirm your identity in response to your request for access.

They are: lanatureshub, lanaturesmood, lanatureshorts, lanatureviews, lanaturescorner, thenatureshub, yourlifesnature, yournaturelifes, cozyzenai and naturesms