Inside the messy business of building public-interest AI
What we’re learning as Suwali meets different languages, channels and organizations it was built to serve.
What we’re learning as Suwali meets different languages, channels and organizations it was built to serve.
We always knew that evaluating AI answers would be one of the messier parts of building Suwali. Was the answer accurate? Did it stick to the right sources? Did it sound right? There is no single test for a good AI answer. So before we launched Suwali, we tested it a ton.
To do this, we brought together journalists, environmental activists, and public health practitioners from our CSO partners, alongside our own program managers, who hail from Brazil, India, and Lebanon. This cross-continental team of testers – that included folks who speak Arabic, French, Hindi, Portuguese and Spanish – posed questions to Suwali on everything from pollution laws and reproductive health to AI safety and ongoing conflicts.
We started with simple evaluations for things like accuracy, sourcing, and determining whether the sources cited actually supported the material that the system retrieved.
Then, we entered more complex territory: Does the answer sound natural? Does its tone match the character of the organization? A newsroom may need something precise and restrained, while a health organization may need more empathy. And sometimes the most important question is also the least technical: Which answer simply works better?
For that, there is no metric that replaces human judgment. It is here that we’ve relied most on our team of testers from around the world.
Even since our launch back in April, this work has continued. Evaluation at Meedan is not a final checkpoint before launch. It is an ongoing process of testing, comparing, finding weaknesses and testing again across languages, topics, and contexts.
In a new piece for our blog, Meedan Research Scientist Terry Zhang takes readers inside that process, including what we measure, where the numbers stop helping, and why evaluating AI ultimately means evaluating much more than the model itself.
Read Terry Zhang: Don’t trust the model, trust the process
With elections coming up in just two weeks, Brazilian politics has been rocked by a massive banking scandal in which two justices of Brazil’s Supreme Electoral Tribunal are embroiled. In recent years, the court has served as an institutional bulwark against election-related fraud and disinformation, and thus been a frequent target of Brazil’s far right. But with the court facing new scrutiny amid the scandal, public trust in Brazil’s electoral process is almost certain to falter.
For our local partners, the circumstances demand that they redouble their commitments to supporting civil society and independent media in the lead-up to election day. We’re proud to be supporting two coalitions that are using our tools to help distribute accurate election information and dispel false rumors and claims. Through the “Confirma seu Poder” campaign, Pacto pela Democracy is using Suwali to answer voters’ questions about the electoral process. If you’re on WhatsApp, go ahead and test their bot for yourself (em português). And the VigIA Project is using Check to help produce fact-checks and deepfake analyses in partnership with Lupa, FAAP, and Recode.
We’ve also got our eyes on midterm elections in the U.S., coming up this November. At a convening of U.S. newsrooms this summer, we wrestled with the question of how we could help our partners reach their audiences better, and arrived at a remarkably simple answer: Text messaging.
Suwali already works over WhatsApp and Telegram, chat channels that make sense for many of the organizations and communities we work with. But these apps are less common in the U.S. With this knowledge at hand, we got to work. Now, Suwali enables newsrooms to invite readers to text a number, ask questions, and get trusted answers in return. They can also receive alerts and updates without downloading yet another app.
This change helps our partners better reach their communities. But it also reflects a key principle at the heart of Suwali’s design: flexibility. We know that organizations’ priorities change, as do the tendencies of the third-party tech platforms that we rely on – think of how changes at companies like Meta can drastically affect how newsrooms approach distribution.
If there’s anything meaningful we can take from tech CEOs’ most recent pressing of the panic button, it’s that we don’t know exactly what to expect from the Big Tech, and that it’s not good for any smaller tech platform – or its users – to rely too heavily on any single third-party provider. Adding SMS (and soon, hopefully, other options) is one way in which we’re building resilience in the face of a less-than-certain future for digital communication.
Read: Bringing Suwali to SMS Ahead of the 2026 Midterms
On October 6 at 10am EDT/4pm CEST, we’re inviting our broader community to join us for a virtual roundtable discussion of tech-facilitated gender-based violence and solutions to this problem that combine community knowledge and computational design. RSVP and learn more here.
October 6-9 ᐧ Online
The Mental Health in Journalism Summit, organized by The Self Investigation, is a global gathering for journalists, newsroom leaders and allies working towards a healthier future for journalism. Register here.
October 14-16 ᐧ Beirut, Lebanon
Bread&Net will convene civil society leaders, researchers, journalists, technologists, and policymakers from across North Africa and Western Asia to discuss the present and future of the digital sphere. Meedanis Haramoun Hamieh and Zahraa Dirani will be on site and eager to connect with potential partners.
October 28-30 ᐧ Barcelona, Spain
Mozilla Festival, aka MozFest, is three days of bold conversations and hands-on building of open source technologies for a better future. Meedan Executive Director Dima Saber will speak at a session on community-owned digital futures, and both Scott Hale and Haramoun Hamieh will be on site too. Come find us and say hello!
Will Apple’s ‘Reference Image’ Feature Help Defend Against AI Manipulation?
We need to be able to use this ground-truth proof in a broader world of reality-determination, and whimsical, communicative and malicious AI edits that spread across messaging, social and search. Accessibility, interoperability, transparent governance and a recognition of the realities of a world where absence of technical proof can and will be leveraged against the most vulnerable accounts are critical next steps to build on this key advance.
(Sam Gregory, Tech Policy Press)
AI and Civil Society: Threats and Obstacles to Deployment and Advocacy
Despite the limited resources, civic space actors demonstrate that an alternative approach exists in contrast to the extractive nature of the AI models proposed and implemented by Big Tech and then used by governments. However, for civic space actors to leverage AI’s inclusive, non-extractive, and beneficial aspects for all, and to prevent its harmful applications, the sector needs resources and capacity building.
(Afef Abrougui, CIVICUS)
