🤥 Faked Up #30

YouTube takes down bogus billionaire bracelet ads, Britons pour milk down the drain, and Romanian NGOs call for more data from platforms

This newsletter is an ~8 minute read and includes 77 links. Consider sharing this newsletter wherever you post about stuff.

HEADLINES

The Pentagon is paying Hive AI $2.4 million over two years for its deepfake detectors. Lula’s health conditions and surgery are inspiring conspiracy theories. A spam network was pushing plagiarized pro-Democratic content on BlueSky. International human rights groups condemned the criminal case against Indian fact-checker Mohammed Zubair. OpenAI says its deceptive election content classifier scored 98% on recall and 89% on precision in evaluations for its new video generator Sora. A threat intelligence company claims that Eleven Labs synthetic voices were ‘very likely’ used in a Russian info op. The U.S. Senate unanimously approved criminalizing the publication of synthetic NCII and mandating that platforms remove it within 48 hours of notice; the bill now moves to the House.

TOP STORIES

BILLIONAIRES’ BOGUS BRACELET

YouTube and Facebook ran1 hundreds of ads for a “feng shui bracelet” that’s supposed to make you instantly rich. There are multiple variations of the video, but they follow roughly the same script.

Speaking to the camera, an unidentified man or woman claims to have been struggling with money until a mysterious good-hearted stranger gave them the bracelet, suddenly reversing their fortunes. Promotions, lottery tickets, house purchases! Everything is supposedly possible with this bracelet.

In a recent variation, the bracelet was sloppily photoshopped on the wrists of Adam Sandler, Jennifer Lopez, Kim Kardashian, Lady Gaga, Taylor Swift, Tom Cruise, and Will Smith.2 They are all part of a cabal that’s apparently been hiding the secret of their success…on their wrists?

I couldn’t find an exact equivalent on TikTok, though there are loads of ads for the bracelet making less bombastic claims (and a few that have been taken down for policy violations).

I emailed Google for comment on Tuesday night and did get a swift response from spokesperson Nate Funkhouser. He told me that “we expressly prohibit ads that falsely claim celebrity endorsements in order to scam people” and that Google had taken down the ads and a related advertiser. The company enforced on tens of millions of scammy and fraudulent ads in 2023.

I emailed Meta for comment on Wednesday morning. By that evening, the company had removed the relevant ads and blocked the advertiser.

If you want to get it on the scheme, the miraculous talisman can still be acquired for a mere $39. And it comes endorsed by these 100% human reviewers!

ROMANIA’S RUINED ROUND

Romania’s constitutional court annulled the first round of the presidential election that saw far-right candidate Călin Georgescu win 22.9% of the votes. According to fact-checking site Factual, Georgescu has said that COVID-19 isn’t real, the moon landings and the Twin Tower attacks were staged, and bottled water isn’t H₂O.

The election was canceled in part due to documents declassified by the country’s intelligence services suggesting Russian interference in the run-up to the vote.

Much of the international coverage of the alleged interference has referred to a reported 25,000 pro-Georgescu TikTok accounts created two weeks before the vote.

But it’s more than just TikTok. As early as June, Context had uncovered a network of Facebook pages masquerading as news sites promoting right-wing politicians. The publication claims this network spent 46,000 euros in ads supporting Georgescu, who reported spending no money on campaign expenses. Meta told Context it is reviewing the network but found no evidence of inauthentic behavior nor any violation of its policies.

Building on these findings, the nonprofit Check First claims “the network has spent at least 139,858 EUR and up to 224,083 EUR on the campaign since August 2024, reaching a total audience of 148 million, according to data from the Meta Ad Library.” Some of these ads may have violated rules on the electoral silence period.

Check First believes this network is affiliated to a company called DGI Multimedia Design SRL, which has previously collaborated with the right-wing party Alliance for the Union of Romanians. DGI spent 27,850 euros on Google ads, receiving over 30 million views.

Finally, there’s the role of Telegram. According to the media startup Public Record, “dozens of websites coordinated from Russia and Telegram channels supported Călin Georgescu's candidacy.” The sites include Russian state-aligned media like Tsargrad and the Romanian version of Pravda, part of an internationally-facing pro-Kremlin network dissected by France’s Viginum.

I am certainly not an expert of Romanian constitutional law, so I don’t know whether any of this is enough to warrant an extraordinary measure like annulling the vote. What is certain is there will be unprecedented scrutiny on the role of online electioneering and the more data platforms make available, the better.

As a coalition of Romanian NGOs wrote in an open letter to the EU Commission:

At the moment, we lack sufficient information to assess whether or not other relevant online platforms in Romania (Facebook, Instagram, Telegram, YouTube) have made real efforts to facilitate a fair electoral process, and what the result of these efforts was.

Among other things, the open letter calls for the “correct implementation” of the Digital Services Act in Romania. Expect this to be a major test for the transparency requirements embedded in the act.

Thanks to Elena Calistru for helping me find useful sources for this item.

BO-SCARE

Angry Britons are spilling their milk and calling for a boycott of dairy producers who use Bovaer, a supplement meant to reduce methane in cow burps and farts. Their concern flows from a misunderstanding of the findings around the safety of Bovaer, a brand name for 3-Nitrooxypropanol or “3-NOP”.

The controversy erupted last week following news that multinational dairy company Arla was trialing the feed additive in 30 British farms. The scandal was fanned by Reform MPs, tabloid news and TV channels. It has since expanded to Australia, where the supplement is used by cattle farmers to produce “carbon-neutral” beef sold at the supermarket Coles.

The British Food Safety Agency wrote in an advisory last Thursday that “there are no safety concerns when Bovaer is used at the approved dose.” (The European Food Safety Authority has also deemed it safe.)

It is probably too late to change perceptions: A search for [Bovaer] on TikTok returns overwhelmingly conspiratorial takes, including that it’s just “another way of f***ing over the farmers” by putting people off animal produce or that it’s actually a depopulation scheme aimed at reducing the sperm count of those who drink it. (Similar results emerge on Instagram, X, and YouTube).

CONSPIRACY KASH

Donald Trump’s nominee for FBI director, Kash Patel, co-hosted a show on The Epoch Times from 2021 to 2023. NBC News reporter Brandy Zadrozny watched all 79 episodes and concluded that it “spun detailed but unfounded claims of conspiracies involving government officials, law enforcement agencies, the media and tech companies, among others, all aiming to rig elections, silence conservative voices and undermine Trump’s presidency and re-election.” Patel has also promoted supplements that purport to be an “mRNA vaccine detoxification system.”

And then there’s the trilogy of children’s books about alleged election fraud that Patel has co-authored (I recommend their semi-serious review by Benjamin Wittes on Lawfare). The second one, “The Plot Against the King 2000 Mules,“ relies on the widely debunked documentary whose central claim was recently retracted by its creator Dinesh D’Souza. The US president-elect has said “let’s put this amazing book in every school in America!”

REVISITING THE 1%

I heard from several of you about last week’s item on quantifying AI-generated election misinformation, following Meta’s claim that the figure is “less than 1%” of the total. The company wouldn’t give me additional background on how it calculated the statistic, so here are some additional estimates to add to the mix.

• Logically Facts, which is working on an end of year report about this topic, told me that 23 out of the 1,695 fact checks they published this year concerned AI misinformation. That comes out to about 1.4%.
• From my review of Full Fact’s coverage of the UK election, only 2 out of 74 individual claims fact-checked were about content that was potentially AI-generated (two more were edited with less modern techniques like basic Photoshop).3 That’s 2.7%.

The European Digital Media Observatory tracks the monthly output of 30-odd EU-based fact-checkers. Its most recent brief estimates that AI content featured in 86 out of the 1,812 fact-checking articles published in October 2024. Since EDMO has started tracking in March 2023, the share has fluctuated from a minimum of 2% to a maximum of 5%.

These figures are on smaller samples than those Meta will have access to and don’t dramatically contradict its findings. It looks likely that the share of falsehoods explicitly leveraging deepfakes that were fact-checked this year was comfortably below 10%.

I would habitually end this item here, but this year we’ve seen plenty of exaggerated predictions and smug dismissals about AI misinformation in the election sphere — and I don’t want to join either side of the argument.

First, there were indeed prominent deepfakes targeting elections around the world including in Brazil, India, the US and the UK. Worse, as I wrote about in FU#23 and FU#24, fact-checkers are sometimes unable to say with certitude that an audio or video is synthetic because of mixed responses from the detectors they can access.

In addition, as LeadStories’ Maarten Schenk points out, it is less likely that fact-checkers will make a big deal about a viral fake that was possibly written with ChatGPT, choosing instead to debunk the underlying claim and move on.

So here’s my boring both-sides take: 2024 saw AI-generated content introduce new vectors of disinformation in elections around the world. AI-generated fake evidence was a small part of the overall universe of election falsehoods, though AI tools are boosting the productivity of threat actors. In terms of overall prevalence, synthetic content was probably most impactful on the harm vectors of scams and harassment.

‘COOL AND VERY FEASIBLE’

A British court heard testimony last week that former Wirecard COO and suspected Russian asset Jan Marsalek suggested creating a deepfake porn video featuring the son of Kazakhstan’s president. This was part of a wider set of false flag operations targeting the Central Asian country that Marsalek allegedly designed. Russian secret services would then tip off Kazakh authorities with a view to ingratiating themselves to a regime that appears to be moving away from Moscow. It’s not clear whether the video was ever created and disseminated, but Marsalek’s handler deemed it a “cool and very feasible” idea.

THE KREMLIN’S FACT-CHECKING COALITION

I’m a little late to this, but apparently Russian state-controlled outlets TASS and Dialog recently launched a “Global Fact-Checking Network” as an alternative to the “highly political” International Fact-Checking Network (of which I was the founding director).

TASS has been suspended by the European Association of News Agencies for “not being able to provide unbiased news.” Dialog is the organization behind the Doppelganger influence operation that has been spoofing Western media outlets to spout pro-Kremlin talking points. But no matter! At an event attended by fact-checking luminary Steven Seagal, the two organizations signed a memorandum to create the GFCN, which will be deposited with the Russian Ministry of Justice.

The new organization even has a code of principles, which reads quite a bit like the IFCN’s, committing signatories to being transparent about sourcing and to corrections. It’s unclear who will verify signatories’ compliance to the code.

Russian state media has been trying to co-opt fact-checking for its propaganda since at least 2017, with not much obvious success. I don’t expect a wide range of publishers to queue up for verification under the GFCN, but I’ll be keeping an eye out.

WORSE THAN REAL NUDES

A survey of British teenagers by the industry-funded nonprofit Internet Matters found that at least 13% of them had either created a deepfake nude, knew someone who had, or had encountered this type of content online.

The most interesting finding of the report to me was that 55% of the teens thought being targeted by a deepfake nude was worse than by a real nude. One girl, aged 16, had this to say in a open-ended response: “If a nude image was sent of me currently that I consented to filming even though it's sad/unfortunate I would know that (it) was my choice that led to that image being shared. However, with a deepfake I didn't choose for that image to be created and its not realistic to me.”

NOTED

1. In the Central African Republic, a former propagandist lifts the veil on the inner workings of Russian disinformation (Forbidden Stories)
2. Podcast: Is the Misinformation Crisis Overblown? (Undark)
3. YouTuber reveals how CCP buys ‘influencers’ (Taipei Times)
4. Who’s Behind One of the Major Accounts Promoting Climate Denialism on X? (Mother Jones)
5. Deepfakes weaponised to target Pakistan's women leaders (AFP)
6. Tricked by a Fake Viral Food Product? You’ve Just Been Snackfished (Wired)
7. Influencers selling fake cures for polycystic ovary syndrome (BBC)
8. Companies in Mexico embrace AI to resurrect the dead (Rest of World)

1 The links are now broken because the platforms removed the ads in question, but I’m keeping them because the urls contain unique identifiers of the ad/vertiser.

2 I reverse-searched each of these images to make sure. Here are the original pictures of all the celebs in question in case you don’t believe me and are tempted by the bracelet: Adam Sandler, Jennifer Lopez, Kim Kardashian, Lady Gaga, Taylor Swift, Tom Cruise and Will Smith

3 Excludes manifestos. Here’s my tally.