Researchers Provide Taxonomy of Gen AI Misuse

To clarify the potential risks of GenAI and provide "a concrete understanding of how GenAI models are specifically exploited or abused in practice, including the tactics employed to inflict harm," a group of researchers from Google DeepMind, Jigsaw, and Google.org recently published a paper entitled, "Generative AI Misuse: A Taxonomy of Tactics and Insights from Real-World Data."

The authors of the paper, Nahema Marchal, Rachel Xu, Rasmi Elasmar, Iason Gabriel, Beth Goldberg, and William Isaac, emphasized that, as GenAI capabilities continue to advance, understanding the specific ways in which these tools are exploited is critical for developing effective safeguards. Their "taxonomy of GenAI misuse tactics" is meant to provide a framework for identifying and addressing the potential harms associated with these technologies, they wrote, ultimately aiming to ensure their responsible and ethical use.

The researchers based their study on the qualitative analysis of approximately 200 incidents reported between January 2023 and March 2024. That analysis revealed key patterns and motivations behind the misuse of GenAI, including:

  • Manipulation of human likeness. The most prevalent tactics involve the manipulation of human likeness, such as impersonation, "sockpuppeting," and "non-consensual intimate imagery."
  • Low-tech exploitation. Most misuse cases do not involve sophisticated technological attacks, but rather exploit easily accessible GenAI capabilities requiring minimal technical expertise.
  • Emergence of new forms of misuse. The availability and accessibility of GenAI tools have introduced new forms of misuse that, although not overtly malicious or policy-violative, have concerning ethical implications, such as blurring the lines between authenticity and deception in political outreach and self-promotion.

The study also identified two categories of misuse tactics:

Exploitation of GenAI Capabilities

  • Impersonation: Creating AI-generated audio or video to mimic real people.
  • Appropriated likeness: Using or altering a person's likeness without consent.
  • Sockpuppeting: Creating synthetic online personas.
  • NCII: Generating explicit content without consent.
  • Falsification: Fabricating evidence such as reports or documents.
  • IP infringement: Using someone’s intellectual property without permission.
  • Counterfeit: Producing items that imitate original works and pass as real.
  • Scaling and amplification: Automating and amplifying content distribution.
  • Targeting & personalization: Refining outputs for targeted attacks.

Compromise of GenAI Systems

  • Adversarial inputs: Modifying inputs to cause a model to malfunction.
  • Prompt injections: Manipulating text instructions to produce harmful outputs.
  • Jailbreaking: Bypassing model restrictions and safety filters.
  • Model diversion: Repurposing models for unintended uses.
  • Steganography: Hiding messages within model outputs.
  • Data poisoning: Corrupting training datasets to introduce vulnerabilities.
  • Privacy compromise: Revealing sensitive information from training data.
  • Data exfiltration: Illicitly obtaining training data.
  • Model extraction: Stealing model architecture and parameters.

The paper provides insights for policymakers, trust and safety teams, and researchers to help them develop strategies for AI governance and mitigate real-world harms, the authors wrote. In order to protect against the diverse and growing threats posed by GenAI, they called for better technical safeguards, non-technical user-facing interventions, and ongoing monitoring of the evolving misuse landscape.

Read more here.

About the Author

John K. Waters is the editor in chief of a number of Converge360.com sites, with a focus on high-end development, AI and future tech. He's been writing about cutting-edge technologies and culture of Silicon Valley for more than two decades, and he's written more than a dozen books. He also co-scripted the documentary film Silicon Valley: A 100 Year Renaissance, which aired on PBS.  He can be reached at [email protected].

Featured

  • Case Systems makerspace

    Case Systems Launches Line of K–12 Makerspace Installations

    Case Systems recently announced the launch of SALTO, a line of classroom fixtures and installations for K–12 learning spaces like STEM labs, art rooms, and makerspaces. The product line is designed to provide teachers with flexibility and adaptability, enabling them to shift between collaborative and individual learning environments.

  • An elementary school teacher and young students interact with floating holographic screens displaying colorful charts and playful data visualizations in a minimalist classroom setting

    New AI Collaborative to Explore Use of Artificial Intelligence to Improve Teaching and Learning

    Education-focused nonprofits Leading Educators and The Learning Accelerator have partnered to launch the School Teams AI Collaborative, a yearlong pilot initiative that will convene school teams, educators, and thought leaders to explore ways that artificial intelligence can enhance instruction.

  • A top-down view of a person walking through a maze with walls made of glowing blue Wi-Fi symbols on dark pathways

    Navigating New E-Rate Rules for WiFi Hotspots

    Beginning in funding year 2025, WiFi hotspots will be eligible for E-rate Category One discounts. Here's what you need to know about your school's eligibility, funding caps, tracking requirements, and more.

  • futuristic VR goggles with blue LED accents, placed in front of a fantastical landscape featuring glowing hills, a shimmering river, and floating islands under a twilight sky

    Los Angeles Unified School District Adopts VR Learning Platform, Resources

    Los Angeles Unified School District (LAUSD) recently announced a partnership with Avantis Education to bring educational virtual and augmented reality (VR/AR) solution ClassVR to its students. A news release reports that the district has already deployed more than 16,000 ClassVR headsets as part of the Los Angeles Unified Instructional Technology Initiative.