Ideas: Community building, machine learning, and the future of AI

In a conversation celebrating the 20th anniversary of the Women in Machine Learning (WiML) workshop, co-founders Jenn Wortman Vaughan and Hanna Wallach trace their parallel and intersecting careers, the genesis of the WiML community, and their evolution into leading researchers in the field of responsible AI.

Intersecting Paths to Microsoft Research

Both researchers began their PhDs with an interest in the intersection of people and AI systems, albeit from different perspectives.

• Jenn Wortman Vaughan, at the University of Pennsylvania, initially focused on machine learning theory and algorithmic economics. Her early work involved “mathematically ideal people,” allowing for neat, precise models without the “mess of the real world.”
• Hanna Wallach, at the University of Cambridge and later as a visiting student at Penn, worked on Bayesian latent variable models for text analysis. She was drawn to the social aspects of text as a medium of human communication.

Despite finding success in academia—Jenn at UCLA and Hanna at UMass as a founding member of its Computational Social Science Institute—both found the environments to be male-dominated and isolating at times. They eventually transitioned to Microsoft Research (MSR) in New York City, drawn by the lab’s unique focus on machine learning, algorithmic economics, and computational social science, which perfectly aligned with their research interests.

The Genesis and Evolution of WiML

The idea for WiML was born at NeurIPS in 2005. At the time, NeurIPS was a much smaller conference of about 600 people. Sharing a hotel room with four other women, they were struck by how rare it was to have that many women from the field in one place. They made a list of all the women in machine learning they could think of, which amounted to about 10 names.

This led to a proposal for a session at the Grace Hopper Celebration, which was ultimately rejected. Undeterred, they organized their own co-located event, a full-day workshop that brought together around 100 women. The initial challenges were significant:

• Gaining Buy-in: Many in the community didn’t understand the need for such a workshop, with some arguing that women’s experiences were no different from men’s.
• Securing Funding: The first workshop was funded by the University of Pennsylvania, but a sustainable model was needed, which later came through an NSF grant and corporate donations.
• Finding the Right Venue: After two years at Grace Hopper, the workshop was moved to NeurIPS to better align with its research focus. This also allowed travel funding to enable attendees to participate in the main conference, and holding it the day before NeurIPS helped attendees build a network of familiar faces.

Over 20 years, WiML grew from a one-off event into a robust non-profit organization with a formal board. Its success has inspired the creation of numerous other affinity groups, such as Black in AI, LatinX in AI, and Queer in AI.

A Research Pivot to Responsible AI

A shared frustration with the irresponsible claims being made about AI’s capabilities sparked their first major research collaboration. A statement at a panel that AI would soon “take all of the uncertainty out of our decision-making” led them to discuss the importance of communicating uncertainty and explaining assumptions. This evolved into two key areas of research that highlighted a major gap between academic theory and industry practice.

1. Interpretability: They challenged the prevailing claims about model interpretability, which were often made without defining who a model was interpretable to or for what purpose. Through controlled experiments with real people, they found that researchers’ intuitions about what makes a model interpretable are often wrong. They proposed thinking of interpretability as a latent property influenced by various factors, from the number of features to the user interface.

2. AI Fairness in Practice: In a project with Ken Holstein, they interviewed industry practitioners to understand their challenges with AI fairness. The findings revealed a significant disconnect from the academic literature, which was heavily focused on quantitative fairness metrics for predictive models.

   • Practitioners worked with a much wider range of applications beyond prediction, including unsupervised and language-based models.
   • Existing fairness metrics were often not applicable to complex, real-world scenarios due to various data challenges, such as the difficulty in collecting sensitive attributes. This work underscored the necessity of qualitative, HCI-informed research to understand and address the real-world messes of responsible AI.

Current Research Frontiers

Their work continues to push the boundaries of responsible AI, particularly in the context of generative models.

• Hanna Wallach is focused on bringing scientific rigor to the evaluation of generative AI. She argues that evaluating abstract concepts like “harmful content” or “memorization” is fundamentally a social science measurement challenge. Drawing on her background in computational social science, she advocates for adopting frameworks used in fields like political science and psychometrics to measure these “squishy,” contested concepts, thereby improving the validity of claims about AI capabilities.

• Jenn Wortman Vaughan is exploring the intersection of AI and critical thought. Her research aims to design AI systems that support appropriate reliance, preserve human agency, and encourage critical engagement. She sees a future where AI enriches human capabilities, but warns of risks like overreliance, atrophied skills, diminished innovation, and even a “loss of joy” when we delegate fulfilling tasks to AI. Her work focuses on developing principles for both system builders and users to mitigate these risks and measure long-term effects on human critical thinking.

Advice for Researchers

Reflecting on their 20-year journey, they offered advice to their younger selves:

• Be Brave: Speak up, even as an introvert, and share your work, even if you feel it’s not perfect.
• Find Fulfilling Problems: Work on the problems you are passionate about and want to see solved—the ones you enjoy both working on day-to-day and talking about afterward.
• Embrace Collaboration and Community: The best research comes from bringing together diverse perspectives. The social side of research, including friendships and networking, is just as valuable as the technical side, providing a crucial support network.

• Tags:
• Generative AI
• Feature
• Interpretability
• Responsible AI
• Community Building
• WiML
• AI Fairness
• Hanna Wallach
• Jenn Wortman Vaughan