Google reimagines deceptive photo editing

🤥 Faked Up #15: The Pixel 9's Reimagine feature is a disinfo delight, Facebook's AI single parent cops drive a content farm and Telegram's channels for undresser apps reach hundreds of thousands

This newsletter is a ~7 minute read and includes 40 links. I❤️ to hear from readers: please leave comments or reach out via email.

THIS WEEK IN FAKES

Pakistan dropped charges against the man who misidentified the Southport stabber. The company that distributed the Biden deepfake robocall agreed to a $1M fine. McAfee released an audio deepfake detector. Australia criminalized non consensual deepfake porn. Zuck got played (context | analysis). A deadbeat dad playing dead got his dues. Megalopolis retracted a trailer full of fake reviews.

TOP STORIES

Corrections: I noticed after publishing last week’s newsletter that “AI info” labels appear on some of the images in my sample when viewing Instagram from the iOS app rather than via the web. I apologize for the inaccuracy. More humorously, I left a “not” out of a sentence, flipping my position on kink-shaming. As always, my corrections policy is here.

GOOGLE REIMAGINES SCAMS

Google (and others) have been releasing AI editing tools for the past year. But the “Reimagine” feature released with the new Pixel 9 phone amps up the risk factor.

I haven’t gotten my hands on the phone yet but, as demoed, the functionality works like this: You click on a part of the photo, write a description of what you would like added and the phone adds it in. In its communication material, Google suggests using Reimagine to add a sunset to a mountain scene or a volcano to a picture of a child.

The Verge did a great job showcasing how the tool can be used to distort reality in more unsavory ways. Others also messed around with it and shared their results: You can add a tank on a side street, reptiles to a meal and a syringe next to an enemy. You can change the weather, or add a UFO.

It is not difficult to imagine how people might use this tool to manufacture evidence. I was reassured in talking to Cornell’s Ben Sobel that the legal system is probably ready for this:

Witnesses have always been able to lie or mislead, and the sorts of penalties that deter lies in spoken testimony will be available to deter fabricated images. I think these technologies will reaffirm the importance of chain of custody, but that’s always been important — the same principle that allows us to trust that a knife offered into evidence was actually the same knife found at the crime scene is the principle that will allow us to trust that surveillance footage being offered into evidence is a contemporaneous record that hasn't been tampered with.

But even if these images don’t upend the legal system, they can be used for scams and misinformation.

Vandalize a house with hateful graffiti and stoke communal hatred. Add a stack of ballot boxes in a photo of a ditch and seed doubt in the election result. Show an explosion near a poll site and depress voter turnout. Deepfake an accident to a parked car and extract money from its owner to get it to the auto shop.

What is more, some of the standard tools used to verify images will be rendered useless by Reimagine. Fact-checkers often use weather records, street signs and landmarks that can be verified with satellite imagery or Google Earth to confirm that an image is from where it claims to be. These verification tools will be increasingly ineffective against the blended unreality that Google’s Pixel phone offers up.

Additional guardrails are needed. The AI label in the metadata should be made visible and hard to remove; it should travel with the photo as it gets shared by email, messaging app and social network. Google should use geo-fencing to avoid the use of Reimagine around sensitive locations like police stations, places of worship, schools and polling places.

Some will dismiss this as alarmist, pointing to reactions to the launch of Photoshop. But Reimagine makes faking much much simpler. The only compelling argument left for not worrying may be that few people actually buy Pixel phones — but Samsung is doing the same thing.

SINGLE PARENT COPS OF AI

American Patriots is a Facebook page that has amassed more than 200 thousand likes since its launch in June. Several of its posts are just pictures of AI-generated single parents in uniform holding a child they kept despite being divorced / widowed / otherwise abandoned. The photos get thousands of likes.

Despite negative reviews and several comments flagging pictures as AI-generated, many users engage with the content at face value. Take this photo of “Lara” and her mother. Patriotic commenters appear unfazed by the disastrous rendition of the American flag and the use of what appears to be Aurebesh on the little girl’s name tag.

The tone used by American Patriots evokes America’s Last Line of Defense, the conservative-baiting troll page run by Christopher Blair. But the goal here appears to be profit, rather than satire.1

Beyond birthday posts for artificial humans, the page also posts clickbaity articles like “After the Olympics Simone Biles shares some unexpected news with the American Public;” “Elon Musk with an unexpected statement that the public will surely appreciate. What he said…” or “Laura Ingraham: A Look into the Personal Life of Fox News’ Prominent Conservative Host.”

These articles are hosted on the website serieaenglish[.]com, which ceased publishing content about Serie A (Italy’s premier football league) six months ago. It has since been taken over by AI-generated dreck and is *dripping* in ads, including several served through Google. According to SimilarWeb, 80% of its July traffic came from Facebook.

Thanks to a reader for sending me the American Patriots page. If you see weird things in your feed — email me!

TELEGRAM IS FOR DEEPFAKE NUDES

Telegram founder Pavel Durov was arrested on Sunday in Paris over allegations that his app is being used, among other things, for the distribution of child sexual abuse imagery.

The press release from the French prosecutor in charge of the case does not mention AI nudifiers that have been used to target dozens of teenagers. But I’d be surprised if this doesn’t come up at all in the investigation, since Telegram is used by many of these services for customer service, marketing and direct sale.

Before Durov’s arrest, Hankyoreh reported that a single South Korean Telegram channel dedicated to deepfake nudes has more than 220,000 users. It is by no means the only one: I track 11 other channels with a combined 880,000 subscribers among them. On Sunday, one of them took a break from sending messages about nudifying to post #FREEDUROV.

MUSK v MORAES

ICYMI, X closed its Brazil office on August 17 following a dispute with the country’s Supreme Court judge Alexandre Moraes. Lupa published an overview of the accounts that Moraes had requested be blocked or identified. This included accounts that circulated electoral disinformation.

DEM PINK SLIME

Columbia’s Tow Center for Digital Journalism found that The Morning Mirror — a news-presenting website that is arguing in court for a press exemption to avoid making financial disclosures to campaign regulators — is almost certainly a front for the Democratic political action committee Forward Majority.

The website itself is pretty basic but appears to exist primarily as a landing page for almost $90,000 in ads it has bought on Meta to promote individual Democratic candidates. As it happens, 29 out of 31 of them are running in districts that Forward Majority has identified as targets for 2024. The website has also used multiple photos whose file name includes the string “FwdMaj.”

COMMUNITY NOTES: IMPACT UNCLEAR

Not going to lie, this preprint on X’s Community Notes is pretty dense, even for a lover of the subject matter. But best I can tell from my read, the study concludes that evidence that the program reduced engagement with misinformation is hard to come by. One thing that stands out — which is a critique common to much ex post fact-checking — is that the notes may arrive too late to have a sizable impact. The typical delay between a tweet being posted and a note being appended was 2.23 days. Even the fastest note in the sample of 30,267 tweets analyzed in the study took longer to be displayed than the typical half-life2 of a tweet.

NOTED

1. Iran Hack Illuminates Long-Standing Trends—and Raises New Challenges (Lawfare)
2. A Fox News host's debunked election conspiracy appears to have prompted a state investigation (Texas Tribune)
3. Dieses Bild von knienden Polizisten in Großbritannien ist KI-generiert (Correctiv)
4. This system can sort real pictures from AI fakes — why aren’t platforms using it? (The Verge)
5. Welcome to Fakesville: Inside an AI nightmare that tore apart a school (The Information)
6. Facts alone fall short in correcting science misinformation (Phys.org)
7. Israel Is Buying Google Ads to Discredit the UN’s Top Gaza Aid Agency (Wired)
8. Her SNU classmates tormented her with deepfake porn — now she fights for all victims (Hankyoreh)
9. Can GPT-4 Models Detect Misleading Visualizations? (arXiv)

FAKED MAP

We’re sticking with the classics this week.

This is the Hunt-Lenox Globe (ca. 1508), now at the New York Public Library.

The tiny globe is famous for its “hic sunt dracones” annotation of South East Asia, which could have been a reference to the komodo dragons of Indonesia rather than a fantastical assumption of what creatures resided in faraway lands.

TBH I'd give the cartographer a harder time for the sloppy rendition of parts of Europe — what exactly is going on in Great Britain? — for which s/he would have likely had access to high quality maps.

1 In an article I saw just before publication, Forbes claims American Patriots is one of 67 pages that violated Meta’s inauthentic behavior policy and was taken down. The page continues to be up for me and I also can’t corroborate that the page has only been up for “little over a week” since its posts date back to at least June.

2 A tweet’s half-life is the time it takes to get half of its overall impressions, which is estimated at around 79.5 minutes. The fastest note in the sample took 80.2 minutes. The note at the 25th percentile of the distribution took 0.569 days.