The way people seek information has undergone a significant transformation. Instead of "googling" it, many now turn to chatbots like ChatGPT, Claude, or Gemini to get direct answers to their questions. This shift from browsing to being told has disintermediated institutions that once sat between a question and its answer, such as newspapers, review sites, and search engines. A new industry has emerged, with marketers now focusing on "AI visibility" or "generative engine optimisation" (GEO) to ensure their content is mentioned first by AI models.
The AI models doing the answering are not created equal. A South African-built tool, LLMEKNOW, has been tracking how different AI models respond to various queries. In a study of South African banking queries, the results showed that different models drew on live web sources to varying degrees. For instance, Claude relied on live web sources in every response, while DeepSeek never searched the web, instead answering from its frozen training data. This highlights the importance of understanding which model is providing the information and how it was obtained.
The differences in AI models' responses become crucial for modern marketers and organisations. The way AI models present information can significantly impact what people think and believe. LLMEKNOW's study found that the sentiment of AI responses to the same question about Nando's varied wildly depending on the market, with an average AI-sentiment score of 50.8/100 in Australia and 70.5/100 in Malaysia. This demonstrates that a brand's reputation can appear to change depending on the national conversation the AI model is catering to.
The implications of AI models as the new arbiters of truth extend beyond commerce. A study by LLMEKNOW found that when AI models were given personas to react to, their responses changed significantly. For instance, when advising a low-income user, Investec's share of recommendations fell by 87%, while TymeBank's share rose 79% for that segment. This shows that AI models are picking up on existing market positioning and reproducing it as disinterested financial advice.
The same mechanism that skews bank recommendations by income also influences which version of a contested political story a chatbot tells. LLMEKNOW's study on a polarising South African subject found that different AI models drew on different sources to produce their responses. ChatGPT relied on official channels, while Claude leaned on advocacy sites. This highlights the need for transparency and accountability in AI-driven information dissemination.
The issue is not limited to South Africa; a similar pattern has been observed globally. The Financial Times analysed conversations with ChatGPT, Gemini, Grok, and DeepSeek about policy and social issues, finding that all four models nudged people away from their most extreme starting positions. However, the study also showed that the models already sat to the left of the general population before any nudging occurred, raising concerns about the neutrality of the information being presented.
As AI models increasingly become the primary source of information, it is essential to audit what each model "knows" and make that invisible layer visible. This requires a concerted effort to ensure that the information being presented is accurate, unbiased, and transparent.
Key points
- The rise of AI models as the new arbiters of truth has significant implications for how we seek and consume information.
- Different AI models provide varying responses to the same query, highlighting the need for transparency and accountability.
- The lack of visibility into AI models' information sources and biases raises concerns about the credibility and trustworthiness of the information being presented.
