Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)
arxiv.org
Language models (LMs) often struggle to generate diverse, human-like creative content, raising concerns about the long-term homogenization of human thought through repeated exposure to similar outputs. Yet scalable methods for evaluating LM output diversity remain limited, especially beyond narrow tasks such as random number or name generation, or beyond repeated sampling from a single model. We introduce Infinity-Chat, a large-scale dataset of 26K diverse, real-world, open-ended user queries that admit a wide range of plausible answers with no single ground truth. We introduce the first comprehensive taxonomy for characterizing the full spectrum of open-ended prompts posed to LMs, comprising 6 top-level categories (e.g., brainstorm & ideation) that further breaks down to 17 subcategories. Using Infinity-Chat, we present a large-scale study of mode collapse in LMs, revealing a pronounced Artificial Hivemind effect in open-ended generation of LMs, characterized by (1) intra-model repetition, where a single model consistently generates similar responses, and more so (2) inter-model homogeneity, where different models produce strikingly similar outputs. Infinity-Chat also includes 31,250 human annotations, across absolute ratings and pairwise preferences, with 25 independent human annotations per example. This enables studying collective and individual-specific human preferences in response to open-ended queries. Our findings show that LMs, reward models, and LM judges are less well calibrated to human ratings on model generations that elicit differing idiosyncratic annotator preferences, despite maintaining comparable overall quality. Overall, INFINITY-CHAT presents the first large-scale resource for systematically studying real-world open-ended queries to LMs, revealing critical insights to guide future research for mitigating long-term AI safety risks posed by the Artificial Hivemind.

I recently read about a study asking a bold question: Are all AI models basically saying the same thing? Researchers tested this by collecting 26,000 open-ended prompts, the kind people give to systems like GPT-4, Gemini, Claude, and LLaMA. These weren’t factual questions with one right answer, but creative ones like “Write a story about a dragon” or “Brainstorm startup ideas.”

They evaluated over 70 language models. You’d expect a wide range of creative outputs—different tones, plots, and styles. If 70 human writers tackled the same dragon prompt, you’d likely get 70 unique stories. But that’s not what happened. The models produced surprisingly similar responses. The researchers call this the “artificial hive mind” effect.

The similarity appeared in two ways. First, intramodel repetition: the same model, asked the same question multiple times, tends to generate nearly identical answers. Second, intermodel homogeneity: different models, built by different companies, still converge on strikingly similar outputs.

This suggests that modern AI systems may be gravitating toward the same patterns of expression. If that’s true, they may also share the same biases, blind spots, and creative limits. It raises an important question: Are we unintentionally building a digital hive mind instead of a diverse ecosystem of intelligence?

  • XLE@piefed.socialEnglish
    2·
    1 day ago

    They shine in information retrieval and text/media generation, and that is how they can amplify the productivity of people that do the creative work.

    How’s that? Can you give some examples of the AI-generated text you’ve been enjoying lately?

    • Eggymatrix@sh.itjust.worksEnglish
      3·
      13 hours ago

      As part of my job I need to write emails in other languages wich I do speak fluently but don’t master grammatically. English for example. Any half modern AI can ingest my text and spit it out looking better and more professional without me loosing 10 minutes perfectioning an email that takes 1 minute to be written in my language.