As always, it’s been a while since my last weeknotes - and a bit of a whirlwind since I returned from a fortnight off over Easter inter-railing with family around Europe. I’d only just got back, when I was off on the rails again to Geneva to run a roundtable exploring next steps for our Citizens Track on AI Governance project. That got a really positive reception, and so I’ve been somewhat sprinting since to both follow-up on momentum, and make sure we hit some key milestones back in Geneva for the first UN Global Dialogue on AI Governance and the AI for Good Summit in early July.
We’re at an interesting point of having lots of energy coming into the project, with many partners contributing and more wanting to contribute, but activity perhaps getting ahead of both governance and sustainable funding - so trying to keep my eyes both on delivering an evidence-based case for centring public voice in AI in July and the kind of long-term inclusive field-building that we started with PAIRS.
Reading and reflecting
I’ve got quite a lot of writing to do in the coming month, working on the Citizens Track white paper. As I’ve been spending the last few months on more organisational tasks and not as much analytical writing - and my note-taking practice has rather collapsed, I need to get back into training, so I thought I would get back into the right frame by both making more time for reading, and spending the rest of these ‘weeknotes’ reflecting on what I’ve read.
Mysticism, black boxes and data
Olivia Guest, Nancy Abigail Nuñez Hernández and Mark Blokpoel’s new pre-print on Understanding Artificial Neural Networks: Mysterianism about Known Mechanism is Mysticism challenges the commonly heard claims that ‘no-one really understanding how AI works, not even the people building it’. Guest et. al. convincingly deconstruct these claims, showing that we can (and do) have mechanistic understanding of many systems even when their behaviours are unpredictable (they use the analogy of the double pendulum), pointing to the idea that an inability to predict all outcomes, or audit all internal states of a system without running it does not equate to a lack of understanding, nor justify an abdication of responsibility to offer appropriate explanations.
The paper also offers a useful treatment of ‘black boxing’ (analysing a system based on inputs and outputs (I/O)), which provides useful additional grounding for the approach we often take in participatory processes of avoiding discussion of the internals of AI systems, and focussing primarily on I/O. This links to one of the critical closing conclusions of the piece: the fundamental element of AI that is practically unknowable is not the mechanisms of computation, but “the dataset and the world” (p 15). In full, the authors state:
“…we can and should admit that slapping a statistical test, whether a single logistic regression or a billion onto a dataset will never bring understanding of the dataset or the world. Only science can do that. Laborious, slow and steady, deep and thoughtful scientific theorising…” (Guest, Olivia, Nancy Abigail Nuñez Hernández, and Mark Blokpoel. ‘Understanding Artificial Neural Networks: Mysterianism about Known Mechanism Is Mysticism’. Preprint, Zenodo, 7 May 2026. https://doi.org/10.5281/zenodo.20071869.)
For me, this feeds into current reflections about the importance of bringing data back into view in discussions of AI, and makes me wonder about the kinds of participatory exercises we might design that help people get to a specific understanding of the fact that one critical explanation of the behaviour of an AI system in relation to a particular domain or task comes from looking at the training data it might have had access to, and exploring how representative or comprehensive such data is, and what unknowns scientific research or data collection efforts are still seeking to address.
AI Agents and Democracy
In A blueprint for using AI to strengthen democracy Andrew Sorota and Josh Hendler present a useful diagnosis of some of the threats to a public sphere that might be brought by an era of AI agents, exploring how delegation of political debate to individual AI agents could lead to a “collection of private worlds, each internally coherent but collectively inhospitable to the kind of shared deliberation that democracy requires.” While this points to real risks, I want to pick up on two particularly problematic assumptions in the article.
(1) Firstly, it assumes universal adoption of AI (“A public sphere in which everyone has a personalized agent”) and argues forward from this assumption. Prior experience tells us technology adoption will be unequal, beset by digital divides. The situation we must be addressing is not one of perfect technology adoption, nor a prior perfect democracy only disrupted by technology - but one of an imperfect and unequal existing democracy, confronted by technologies that, even if better aligned with democratic values, will not reach all hands equally.
(2) Secondly, I question the assumption that ‘more and better AI’ is necessarily the solution. Rather than additional layers of technological mediation, what of dis-intermediating political dialogue? Local and face-to-face democratic conversations don’t face the kind of identity verification challenges that the authors highlight digitally-mediated discourse will have to contend with - and they have a much higher chance of supporting the social, solidarity-building and connection-building functions of democracy - not just the informational.
While I agree with the authors that “What is needed is a new generation of democratic infrastructure”, the omission of social from the specification of this as a ‘technological and institutional’ project is telling.
Unpacking the Civic AI Imaginary
Earlier this year M. R. Sauter posted a pre-print exploring ideas on AI as an Excuse in Participatory Systems: Identifying and Describing the Civic AI Imaginary and its Impact on Representative and Participatory Governance. I’ve been meaning to read it for a while, but Sorota and Hendler’s piece discussed above brought it to the top of my papers pile.
Sauter surveys some of the assumptions and values baked into a ‘Civic AI Imaginary’ such as:
- 5.1. The value of digital equivalency: a convincing performance of intelligence via natural language is the same as an individual intelligent human for most purposes…
- 5.2. The imperial epistemology of data: those things that are important for governance can be rendered into computationally legible data…
- 5.3. Generative and predictive systems are truth-telling systems… [that] … can reveal the core hidden truths of individuals preferences, desires, and behaviors…
- 5.4. Democratic governance is a product or procedure that can occur absent the governed
Each of these are deeply problematic, and yet, as Sauter notes, they lie behind many efforts to digitise democratic discourse. I particularly enjoyed the critique of the idea of ‘data doubles’: the creation of AI agents to stand in for human subjects:
Though often called data doubles, they are more precisely described as data homunculi, as they are fundamentally misshapen depictions, warped toward the perceptive limitations of the technology itself. [Sauter 2017]
The article goes on to outline how democratic governance is much richer than simple the aggregation of preferences, or production of decisions at scale - but instead involves relational processes of working out how to live in community with others. The piece powerfully concludes that:
The civic AI imaginary permits, encourages, and excuses the use of computational systems as replacements for people in participatory and representative governance structures. It is fundamentally corrosive to the ethos of address-ability and response-ability that is fundamental to any functioning democratic project. However, the success of the civic AI imaginary in the space of participatory and representational systems is reflective of how the US democratic project has already eroded. The pressures of efficiency, technocratic quantification, and optimization already present in actually operating governance structures mesh with the values of efficiency, industry deference, and imperial data epistemologies that compose the civic AI imaginary. Actually existing AI systems and the civic AI imaginary are not specially suited to deployment in participatory and representative systems. Rather, they function alongside existing structures of power to provide intellectual and ideological cover for the continued erasure of the governed from structures of governance.
In work on the Citizens Track paper, we’re going to need to grapple with questions of if, and how far, we turn to, or allow, uses of digital platforms and AI systems in supporting particular aspects of distributed and transnational deliberation. Keeping in mind the problems of a ‘Civic AI Imaginary’, and making sure any use is justified in terms of a richer democratic imaginary, will be vital.