Katherine (Katie) Keith is currently an assistant professor of computer science at Williams College. Her research interests lie at the intersection of natural language processing, computational social sciences and causal inference. During 2021-2022, she was a postdoctoral young researcher with the Semantic Scholar team at the Allen Institute for Artificial Intelligence. She graduated with a PhD from the College of Information and Computer Sciences at the University of Massachusetts Amherst, where she was advised by Brendan O’Connor. She co-organized the First Workshop on Causal Inference and NLP, co-organized the 2022 NLP+CSS Workshop at EMNLP, hosted the Diaries of Social Data Research podcast, co-organized the NLP+ CSS 201 Online Tutorial Series, and recipient of a Bloomberg Data Science PhD fellowship.
Where do you see the most exciting research/debates taking place in your field?
I think the most exciting debate that has lasted for decades is the neuro-symbolic debate about artificial intelligence (AI). This refers to the debate about the most effective approach for AI to understand and interact with the world: via neural networks (deep learning) or symbolic reasoning. Neural approaches (deep learning) learn directly from data and can pick up patterns in data that contain many anomalies and edge cases, such as language. On the other hand, symbolic reasoning, or the classic AI approach, involves predefined rules and logic for problem solving, which is advantageous for tasks that require explicit reasoning and understanding but can be brittle in the presence of anomalies. My prediction is that we will see a mix of the two in the future.
How has the way you understand the world changed over time, and what (or who) has led to the major shifts in your thinking?
I entered high school believing that science was a “pure,” objective, neutral process for uncovering universal truths. Now my perspective has shifted to seeing science as much more of a social process. I have seen how individuals’ values, biases, and social dynamics can and do influence scientific findings and their interpretation.
How has the rapid growth of quantitative analysis methods and techniques affected the way we measure human behavior? Are algorithms playing an increasingly important role in controlling human behavior?
I have seen that this rapid growth has increased the need for scalable computational techniques. This has the advantage of allowing us to have greater statistical power in our analyses, but it also has the disadvantage that anomalies and bugs in the datasets themselves are not as easy to identify. I don’t believe algorithms control human behavior, but I do believe we are extremely susceptible to them. I think we all have a responsibility to educate ourselves and others about its effects.
Why is transparency of large-scale algorithm audits important? Looking at YouTube’s demonetization algorithms, what are the problems in determining the connection between the subject of a video and the characteristics of its demonetization?
Transparency is important because, as we say in our article: “In recent years, changes in monetization policies and the introduction of algorithmic systems for making monetization decisions have been a source of controversy and tension between the content creators and the platform. There have been numerous allegations suggesting that the underlying monetization algorithms give preferential treatment to larger channels and effectively censor minority voices by demonetizing their content.” I think this is a question for policymakers (and one I don’t have an easy answer to) about navigating the tradeoffs between robust competitiveness and protecting consumers and content creators.
As for YouTube, it is not difficult to quantify the connection between the subject of a video and the characteristics of its demonetization, but it is difficult to pinpoint the mechanisms that cause this. As we say in our article: “While we find examples of word types occurring with high demonetization rates, the fact that there is no single keyword that results in a 100% demonetization rate suggests that demonetization decisions are not made solely based on the presence of specific words in titles.” This suggests that there is no automatic keyword-based trigger in the algorithm; we had initially assumed that this might be present.
In your article on Causal inference in NLP you emphasized that causality is becoming increasingly important in NLP. What are the use cases of advances in causality research in NLP?
In that article we highlighted two directions of causality and NLP. One was causality which helped traditional NLP tasks, but the other direction is what I’m most excited about: NLP helps causal inference. We need to combine NLP methods with causal inference and this integration is not 100% easy. I recently wrote a paper estimating the effects of peer review policies and we had to adjust for text as a confounding variable. I see many other areas like this where combining NLP and causal inference could be useful in understanding causal relationships.
You were a crucial member of the team that created an advanced free online tutorial series focused on teaching advanced NLP methods to the computational social sciences. What was the overarching ambition of this project?
Ian Stewart and I realized that this kind of translational work – from NLP to the social science community – was really undervalued in our field. Somehow ‘novelty’ is valued over ‘dissemination of knowledge’. This was frustrating for us and we both wanted to do our best to change this trend. I think new major language models, like ChatGPT, will only make this translation work even more important.
How do you expect current and future AI tools to impact international relations and politics in general? Will these instruments ultimately be positive or negative?
My PhD focused on social measurement. I think AI tools – especially large language models – will simply be another tool in the toolbox of computational social scientists studying international relations and politics, but certainly not a panacea. A large part of my research agenda highlights the moments when the ‘off-the-shelf’ tools fail and need to be improved for valid downstream conclusions. I have no prediction about the overall benefit or harm of these tools. My hope, of course, is that these technologies will lead to public benefit, but I believe this depends on the values of the people who use these technologies and not on the technologies themselves.
What is the most important advice you can give to young scholars in the field of international politics?
Be ready and willing to adapt. We are in an era of rapid technological advancement and the methods used today may not be the methods used in a few years.
Read more about e-international relations