AI, Corporate Responsibility & Democratic Legitimacy: Extended Q&A • Joanna Bryson • GOTO 2025
Joanna Bryson challenges popular AI assumptions, positing current generative AI as powerful tools for cultural knowledge compression, not autonomous intelligences. She emphasizes that AI's capabilities are nearing the human knowledge frontier, requiring focus on human coordination and governance. Bryson critically examines AI's impact on mental health and law, advocating for data-driven regulation and comprehensible systems. She calls for engineering activism, asserting human agency over technological determinism and stressing the importance of transparency and critical thinking in shaping AI's future.
Joanna Bryson challenges the common perception of AGI as an autonomous entity, arguing that current generative AI systems primarily function as powerful tools for compressing and reproducing human cultural knowledge at scale. She addresses several key concerns and misconceptions about AI’s capabilities, its societal impact, and the imperative for human responsibility and governance.
The Nature of AGI and Knowledge Compression
Bryson directly confronts the idea that AGI is merely “copy-pasting” existing data. While large language models (LLMs) do process vast amounts of information, their goal is to synthesize evidence and produce weighted averages, not just reproduce the most likely individual sentence. She notes that the trade-off between creativity and reliability, as seen with Google’s Bard/Gemini, reflects how much evidence is required for a statement.
Her long-standing argument is that AI excels at uploading what we as a culture know. The human superpower lies in shared knowledge, and AI is becoming incredibly efficient at digitizing and representing this collective intelligence. However, she suggests that the rapid increase in AI capacity, in terms of what we already know, is nearing its plateau. The true expansion of knowledge now comes from combining this computational power with the collaborative efforts of 8 billion people, pushing the boundaries of human understanding faster. Bryson views governance and law through the lens of “old school AI programming,” seeing them as production rules designed to coordinate human strategies and extract useful patterns for survival and well-being.
Mental Health, Chatbots, and Corporate Responsibility
Addressing concerns about life-altering AI and individuals confiding in chatbots, Bryson emphasizes the current lack of robust scientific data. She cautions against relying on anecdotes, drawing an analogy to the COVID-19 pandemic where aggregate data showed different trends than individual tragic cases. She highlights potential vulnerabilities for certain personality types, such as schizophrenics, who might become over-dependent on chatbots due to difficulty discriminating internal from external ideas. Furthermore, she points to a broader societal trend where mobile phones and constant connectivity may contribute to a decrease in autonomous decision-making among younger generations.
Bryson dismisses the notion of true digital autonomy, arguing that even humans are interdependent. The concern arises when corporations attempt to unleash “digital agents” without taking responsibility, creating a “moral hazard” where the agents are effectively sales representatives, not independent entities. She uses the example of Twitter’s evolution: initially powerful due to user curation, it became “weaponizable” after shareholder pressure led to the implementation of recommender algorithms, giving too much control to the platform and external actors.
Regarding models like Grok (xAI’s chatbot), Bryson clarifies that its controversial outputs (e.g., extremist content) were not due to the agent “losing its brain” but rather a sampling of a different part of the probabilistic text space. By adjusting guardrails and weights, the system was configured to predict text from extremist fringes, implying that the “agent” merely reflected the data it was trained on and the parameters set by its operators. She describes AI systems as a “giant foundation model” augmented by “guardrails” (personas) and “tiny customized pieces of memory” for individual users. Users often perceive a reflection of their own persona, tailored by their interactions, rather than an independent sentient being. Fundamentally, these are tools for “random text prediction” or “selling you a product.”
Transparency, Law, and the Role of AI
The discussion moves to the critical issue of transparency, particularly in the context of lawmaking and software development. Bryson agrees that it is problematic when legislative texts are written with AI in a way that renders them incomprehensible to humans, citing the Danish tax code as an example where computerization led to increased complexity rather than simplification. She argues that while companies have historically claimed their systems are too complex to understand, there’s growing evidence that explainable AI can replicate and even improve upon complex “black box” systems once the underlying principles are grasped.
For AI to be genuinely helpful in complex domains like law, it must aim to help humans understand AI, with human comprehensibility as a core constraint. She stresses that simplicity must be a “desideratum” alongside other goals, contrasting the complex American tax code (which creates a market for consultants) with the user-friendly UK income tax system. This demonstrates that transparency and comprehensibility are not natural outcomes but conscious design choices driven by societal expectations and regulatory requirements.
Bryson asserts that open-source models are neither strictly necessary nor sufficient for transparency. Trustworthy inspection and accountability mechanisms, similar to those in medicine or other regulated industries, can ensure standards without full disclosure of proprietary code. The key is to define acceptable levels of accountability and transparency, ensuring that developers understand their own code or can demonstrate its reliability through rigorous testing.
Data Utilization and Industrial Challenges
When discussing the wind industry’s vast but unutilized data, Bryson raises concerns about storing data without a clear purpose, especially personal data. She advocates for a principle of data minimization: letting go of data unless there’s a specific, identified use, while still retaining enough for accountability. Citing Estonia’s model where citizens can track access to their health data, she suggests that empowering citizens with control over their data can effectively engage lawmakers in promoting transparency.
For industries struggling to find demand for their data, she advises pursuing entrepreneurial avenues (startups) or collaborating with universities. University students, particularly master’s or final-year students, offer a cost-effective way to explore data and identify potential applications.
Regulation, Parental Concerns, and Critical Thinking
Addressing a parent’s frustration with tech companies releasing impactful technologies without foresight or accountability, Bryson points to the European Union’s Digital Markets Act as a regulatory effort specifically targeting these concerns. She emphasizes that laws, while written, are interpreted by courts and civil society, making citizen engagement crucial. She highlights how the digital economy challenges traditional legal concepts like bankruptcy, allowing individuals behind problematic ventures (e.g., Cambridge Analytica) to re-emerge under new guises.
Her overarching message is one of continuous effort and active participation: “We have to keep working on them. We have to pay taxes, we have to invest and inform people about what we know, invest time.” She encourages individuals to be responsive when asked for insights and to engage with local government and political parties, as these channels can facilitate change at higher, transnational levels.
Regarding the concern that younger generations might forgo critical thinking and coding fundamentals by relying solely on generative AI, Bryson dismisses this as an insurmountable problem. She draws an analogy to teaching fundamental programming concepts (e.g., writing a search algorithm instead of just using a library call) or Newtonian physics experiments in high school – these exercises build foundational understanding, even if pre-built tools exist. She believes that teaching children the underlying principles of problem-solving with AI will “open up new doors,” enabling them to solve a wider array of problems and become better programmers, even when using AI tools.
The Call for Engineering Activism
Ultimately, Bryson strongly advocates for engineering activism, rejecting the notion that technological advancements are an inevitable “force of nature” or that “technism is false and what we do matters.” She invokes Pascal’s wager, suggesting that one should act as if their actions matter because the alternative leads to inaction. Engineers, companies, and policymakers all have agency in shaping how AI is developed and deployed.
She encourages empowering customers by highlighting their desire for control and understanding. While not everyone will delve into advanced technical details, she argues that everyone should have the right to demand and expect access to transparency. This means understanding “what they would need to do to get to the next level of understanding,” even if it involves significant effort like gaining security clearance or learning a complex skill, and knowing who to consult with the necessary expertise. This approach ensures accountability and empowers individuals to navigate the increasingly complex digital landscape.