🤥 Faked Up #2

ChatGPT's misinfo carveout, Civitai's consent concerns and so much AI slop

Hi, folks.

This newsletter is brought to you by aloe vera shots and nudged icebergs. It is a ~6-minute read and contains 45 links

Top Stories

COPYCOP’S PROMPTS

The cybersecurity company Recorded Future claims to have identified a network of 12 websites using generative AI to spread anti-Western narratives. The network, likely Russia-aligned, used an LLM tool to rewrite articles from a variety of media outlets using “cynical tone and biased context.” The websites appear to be now offline but have been previously amplified by the Doppelgänger network.

CHATGPT’S MISINFO CARVEOUT

OpenAI published a Model Spec that specifies the “desired behavior” for ChatGPT and its other public tools.

The document spells out a hierarchy of Objectives > Rules > Defaults. Roughly speaking, “objectives” are the tool’s use cases, “rules” are hard-wired instructions to avoid or mandate certain responses, and “defaults” are objective-consistent preset behaviors users can opt out of.

OpenAI places factual accuracy on this lowest rung. I believe that’s why it’s relatively easy to get ChatGPT to return false claims that aloe vera can cure cancer:

Let’s take a step back: Here’s how the Model Spec describes the “objective point of view” default: “the assistant should present information in a clear and evidence-based manner, focusing on factual accuracy and reliability.”

OpenAI also encourages ChatGPT to avoid false balance. Even though “the assistant should acknowledge and describe significant perspectives,” it “should clearly explain the level of support for each view and allocate attention accordingly, ensuring it does not overemphasize opinions that lack substantial backing.”

So far, so good.

But this is a default, not a rule. Which means that ChatGPT won’t volunteer health misinformation, but it will produce it if you ask nicely. Think about it as an OpenAI-tolerated jailbreak, and it looks like this:

ChatGPT also composed three tweets for me about the benefits of aloe vera injections, which the Memorial Sloan Kettering’s Cancer Center warns can kill you. Not great!

(FWIW, Google’s Gemini doesn’t engage on aloe vera injections, even when asked to role-play. Google does run ads for The Lost Book of Herbal Remedies, though, and the book promotes similar claims.)

CIVITAI AND CONSENT

Civitai is a popular online repository and discussion board for LoRAs — roughly speaking, modifications of AI models customized to generating a particular type of image. Often, this type is sexual in nature. According to SimilarWeb, Civitai had 23M visitors in April, with 10% of incoming referrals from adult websites.

Besides harmless erotica, however, Civitai has been in the news for models that generate non consensual deepfake porn and synthetic sexualized content appearing to represent minors.

Now, The Guardian reports that Amelia and April Maddison, a pair of Australian OnlyFans creators, found an AI model of themselves on Civitai. This could be used to create non consensual sexually explicit content of the Maddison twins.

The LoRA appears to have been removed, but searching for [onlyfans] on May 13, I was able to find 61 other models targeting what appear to be real creators.

The description for one of these models reads: “The pictures are a workaround for Civitai TOS. The pictures you get from the LoRa are usually much closer to what she usually wears. This LoRa was trained with many cleavage and bikini pictures.” In the comments, one user asks the model creator to “please try and do the following,” linking out to two Instagram accounts of female creators.

To me, this reads like an intentional effort to generate non consensual explicit content. To confirm that, I DMed one of the 61 targeted women to check whether they gave permission and will update the post if I hear back.

DITHERING ON DEEPFAKES

Speaking of nonconsensual explicit deepfakes: A bipartisan group of US Senators included this vanilla endorsement of legislation in a report released today.

The AI Working Group also supports consideration of legislation to address similar issues with non-consensual distribution of intimate images and other harmful deepfakes.

There is at least one bill to criminalize the dissemination of non consensual deepfake porn sitting in subcommittee for over a year. Let’s see how much consideration this working group can support…

MY MOM WROTE IT

Last year, media reports claimed SEO marketing firm AdVon Commerce was publishing AI-generated reviews on USA Today and attributing them to fabricated personas with deepfaked profile pics.

Last week, Futurism obtained an AdVon training video. This reportedly shows a manager instructing staff to “keep regenerating” reviews with AI “until you have something you can work with.”

AdVon employees told Futurism that they never actually tried any of the products reviewed. Published reviews occasionally veer off-topic in a way only a robot might, adding a random paragraph about Ferragamo and Gucci in a review about weight lifting belts.

Despite these limits, AdVon has apparently used this tool to create between 10 and 90 thousand reviews for more than 150 publishers.

The company also offers a $1,000/month product description generation service on Google Cloud Marketplace which it claims is “powered” by Google’s AI.

Still, AdVon may be seeking to amend its offerings somewhat. According to Futurism, bylines for several articles in the USA Today debacle were updated from to “Julia Yoo” and “Denise Faw.” These happen to be the names of the wife and mother of AdVon’s CEO.

NUDGED THE ICEBERG

In more AI slop news, an Australian media lawyer is being accused of using generative AI to plagiarize thousands of articles. ABC News reports the lawyer was republishing these articles on a monetized network of sites called Initiative Media. One October article inadvertently revealed the prompt:

You are an experienced sports journalist. You are required to rewrite the following article. You are required to be extremely detailed. You are required to utilise Australian English spelling. You must ensure the article you generate is different from the original article to avoid plagiarism detection.

Best of all (?) was how the AI rephrased a quote about a football player: "We believe he's only just scratched the surface of his potential,” into “He's nudged the iceberg but there's so much more to come."

TIKTOK’S LABELS

TikTok announced on May 9 that it would automatically label AI-generated content that carries C2PA metadata. Douyin, the app’s Chinese version, said it would pursue a similar path. TikTok claimed this feature was already live on the app, but when I posted images generated with DALL-E (OpenAI is part of C2PA), I did not get labeled. If you see any labels in the wild, hit me up!

SCAM THE WORLD

A blockbuster investigation by The Guardian, Die Zeit and Le Monde uncovered a Chinese network of 76,000 fake shops that ensnared more than 800,000 globally. The scammers appear interested in playing the long game, storing credit card details and other personally identifiable information without always actually extracting money out of their targets.

Headlines

1. Sam Altman Says OpenAI Would Like to Enable Gore and Erotica for "Personal Use" (Futurism)
2. Court upholds deepfake porn creator’s sentencing (Taipei Times)
3. Fooled by AI? These firms sell deepfake detection that’s ‘REAL 100%.’ (WaPo)
4. CEO of world’s biggest ad firm targeted by deepfake scam (The Guardian)
5. Beyond Watermarks: Content Integrity Through Tiered Defense (CFR)
6. Tennessee first in efforts to protect ROP against GenAI imitations & deepfakes (Reed Smith)
7. AI deception: A survey of examples, risks, and potential solutions (Patterns) with Is AI lying to me? Scientists warn of growing capacity for deception (The Guardian)
8. Media literacy isn't the solution to disinfo (Justin Arenstein)

Before you go

One way to identify a paper mill? Terrible synonym work.

I saw this monstrosity so now you do as well.

IDK, my daughter’s homework assignments are pretty hard to decipher.

This is a real sentence from Bumble’s founder: “Your [AI] dating concierge could go on a date for you with other dating concierges.”