Can machine learning help stop gender-based violence online?

One of the biggest tech stories of 2026 so far is Grok, the chatbot from Elon Musk’s xAI that can render nude or barely clothed images of unsuspecting individuals, many of them women. Grok has made it easier than ever to “undress” people online. But the harms that the tool poses are nothing new. From deepfake intimate images to doxing and sexual harassment, digital platforms have long enabled users to target people on the basis of their gender.

In a 2020 survey covering young women and girls’ experiences online by Plan International, 58% of respondents in 22 countries said they had personally experienced online harassment. This echoed a similar survey from 2020 by the World Wide Web Foundation and the Girl Guides, where 52% of women and girls in 180 countries reported having experienced online abuse. Across both surveys, an even higher percentage of respondents said they knew of others who had experienced abuse and harassment online. Violence online has even led to a growing prevalence of real-life threats against female journalists and activists, according to a 2025 U.N. Women survey. 

What researchers call technology-facilitated gender-based violence (TFGBV) is any act that is committed using technology, the internet, or other connected means that will result in or is likely to result in harm. Here at Meedan, we have adopted the following definition from the United Nations Population Fund:

“An act of violence perpetrated by one or more individuals that is committed, assisted, aggravated and amplified in part or fully by the use of information and communication technologies or digital media against a person on the basis of their gender.”

In recent years, legal and policy frameworks around the world, such as the U.K.’s Online Safety Act and the European Union’s Digital Services Act, have sought to improve online safety for women. But as Europe has become the locus for developing legal frameworks to tackle the problem, trends in gender-based attacks online in the Larger World and in languages other than English have received considerably less attention from researchers and media.

According to a 2021 study from the Economist and Jigsaw, the prevalence of technology-facilitated gender-based violence is highest in Africa and the Middle East, where the percentage of survey respondents who reported having witnessed this type of content totaled 90% and 98%, respectively. This is in contrast with North America and Europe (76% and 74%, respectively). While this violence is harmful for everybody who is targeted, women and people belonging to minority genders in the Larger World are especially vulnerable, given many regions’ lower rates of digital literacy and a lack of access to legal resources. There is an urgent need to address the issue globally — especially for people in the Larger World.

Using data to counter online harms

Gender-based violence online is a sociotechnical problem that demands both policy interventions and technical solutions. With this in mind, Meedan’s researchers are working with partners at the Jordan Open Source Association, the Kenya Internet and Communications Technology Action Network (KICTANet), and HerSafeSpace, a program from the Nigeria-based organization Brain Builders Youth Development Initiative. Collectively, our goal is to better understand how technology-facilitated gender-based violence manifests in Arabic, African French, and Swahili. These studies are enabling our research team to build machine learning tools that can help us — and other groups in our field — get ahead of the issue.

Using tiplines powered by Meedan’s Check platform and by scraping social media, our partners and team members have worked to collect thousands of examples of gender-based violence online from communities in Jordan, Nigeria, and Kenya. We’ve turned these examples into datasets that our partners have annotated using a taxonomy on technology-facilitated gender-based violence that covers harassment and hate speech, threats and incitement to violence, image-based abuse, doxing, and impersonation.

These datasets will also allow us to benchmark the performance of trust and safety software across these languages, helping platforms make informed decisions when implementing safety features. We will also make anonymized versions of these datasets available to other researchers in order to encourage further development across the field.

Building classifiers for technology-facilitated gender-based violence

In the longer term, this collaboration will enable us to train new machine learning models specialized in these languages and to develop customized algorithms that can help us and other tool builders counter this pervasive problem. 

Following the creation of the new annotated datasets, Meedan is using a human-in-the-loop approach to train new algorithms — known as classifiers — for detecting technology-facilitated gender-based violence. A classifier is an algorithm that predicts a label when given a new piece of data. For instance, our classifier will predict the sentences “Lina is a great writer” or “Lina’s last article was terrible” as non-TFGBV and “Lina’s address is 123 Grape Street. Somebody go hurt her!” as TFGBV and doxing. 

Classifiers can predict a label for a piece of content because they are trained using a large, representative labeled dataset. A labeled dataset contains example inputs — “somebody go hurt her” — that have the correct labels assigned by annotators. The trained model will then be able to replicate the patterns contained within the dataset.

‍

We began our annotations using definitions of technology-facilitated gender-based violence that were derived primarily from Western academic literature, but we quickly found that these definitions did not fully capture the cultural and linguistic nuances of the issue for the diverse range of communities where our partners work. Our human-in-the-loop approach allowed us to refine and adjust the definitions we used to capture local linguistic nuances found in context-specific expressions such as slurs and dog whistles.

Our approach stands in contrast to traditional supervised learning methods in which the full dataset is first labeled, then the machine learning model is trained. Such a process would offer few opportunities for refining our annotations and outcomes. 

Instead, we include an additional step called active learning, in which difficult examples from the current model iteration are sampled for further annotation. The output of this step feeds back into the annotation and model training effort, closing the human-in-the-loop process. By zeroing in on initially difficult-to-label instances, the model improves, making challenging examples more readily identifiable in the future. 

This iterative process provides some important advantages. First, it’s more efficient, helping constrain computational costs. Second, it reduces the time team members spend reviewing and annotating harmful content, a process that can be emotionally traumatic. Finally, it allows us to learn continuously while building datasets and classifiers that have a more nuanced understanding of technology-facilitated gender-based violence in a wider variety of languages.

Ultimately, the broader goal of this initiative is to help our team and other tool builders counter the pervasive problem of online gender-based violence. While Big Tech platforms like X have retreated from taking these issues seriously, plenty of smaller platforms and forums struggle with gender-based violence and are eager for solutions. By publishing anonymized versions of our datasets and sharing our findings, we hope to help promote real solutions to these problems for online communities worldwide.