by Eric Moran (Director of Social Science Journals, SAGE Publishing)
Eric Moran, SAGE Publishing’s Director of Social Science Journals, interviews Dr. Arthur Lupia about what gave rise to the current call for better sharing of research data and methods, how the research community is responding, how librarians can become more involved, and how the result will be better for all.
Dr. Lupia is the Hal R. Varian Collegiate Professor of Political Science at the University of Michigan, Chair of the National Academy of Science’s Roundtable of the Application of Social and Behavioral Science Research, an original author of the Data Access & Research Transparency Statement of the American Political Science Association, and Chair of the Board at the Center for Open Science.
Eric Moran: In your opinion, what issues do researchers face that call for increased openness and transparency?
Arthur Lupia: We live in a remarkable era where changes in technology and society have led to the proliferation of all kinds of new information sources. In many ways that’s a great thing. But it’s leading to some questions about knowledge-generating institutions.
People look at universities, libraries, and other research institutions. They notice that we spend a lot of money and that we want to be influential. Some ask, “Why do we need you when we have all this information for free on our phones?” and “Why would we defer to you out of all of the others with opinions telling us what to do?”
As researchers, the temptation is to say that “we are careful in design and measurement” and that alone is why people should listen to us. But increasingly that is not an answer to which people are receptive. Many people want to see why and when they should believe us.
It’s for this reason that I think an increased emphasis on transparency and openness is critical. It provides a basis for people to believe the conclusions that we produce. One thing science can do is say, “This conclusion is true independent of what your political ideology, religious beliefs, or cultural attachments may be. If you follow this procedure, you would get this answer.” Openness and transparency are a way to demonstrate this principle — which is one of the great powers of research in the modern era. The public value of science in this age depends on a bigger and more comprehensive commitment to openness and transparency.
EM: What are the oppositions to this within the academy, if any?
AL: There’s a lot of opposition — from post-modern critics who don’t believe in inter-subjective knowledge to milder forms of skepticism.
From within science, here’s the easiest critique: greater openness is hard to do and the rewards are limited. Researchers notice that we get rewards for making flashy claims, like the ones you read about in the news. Those are the ones that the journal editors want to publish. These are the claims that correlate with higher impact factors. There are real incentives in play. But the efforts related to introspection, and the extra effort it takes to be transparent, don’t have similar rewards and haven’t been required of researchers traditionally.
Another set of critiques is that people from different research traditions don’t agree on how transparency would apply to them. For example, there are people who do ethnographic research who are concerned about privacy because they work with some pretty vulnerable populations throughout the world. The idea of a universal one-size-fits-all transparency requirement to share all of the data we collect would be a problem.
EM: Let’s dive in a little bit into some of the efforts of the research community to address these concerns. For example, you’ve worked on, Data Access & Research Transparency (DA-RT). How did it come to be and what is it?
AL: DA-RT started when President of the American Political Science Association (APSA) Council asked me to convene a group to think about transparency. This is tricky because political science is a discipline that’s unified by a context, not a methodology. We have postmodern philosophers, big data quantitative researchers and everybody in between. So, our goal was to think about transparency for all methods.
At first it was difficult because we didn’t speak the same methodological language. Pretty quickly we discovered an intersection: there was a set of common values we held about what we thought our research could do and how it could be valuable to the public. Transparency was critical for all methods. APSA encouraged us to see where we could take this idea.
After a few key conversations with different groups, my colleague Colin Elman of Syracuse University and I went on a five-year listening tour. We went to anybody who wanted to talk about the issue and made a presentation, he from a more qualitative perspective, and me from a quantitative one, with the goal of figuring out whether there was a set of core values that people might want to pursue to achieve transparency.
At the same time, APSA asked us to lead a revision of its ethics guide in a way that would support data sharing and procedural transparency. The point of the revision was to change status quo assumptions about sharing data and research materials. In the revision, either data would be shared or researchers would provide a compelling reason why they could not share it — such as needs to protect subject confidentiality.
To our surprise, the ethical standards we presented were approved by APSA unanimously and eventually turned into a joint statement (dartstatement.org) that would provide structure and policy for some of political science’s top journals.
However, the statement was not met with immediate acceptance by all. Some critics did not like it. At one point there was even a petition to delay its implementation. Eventually, the controversy led to more journals signing on to adopt the standards and great conversations about how to advance transparency in other ways.
EM: What has been the impact of DA-RT so far and where do you see it going next?
AL: The biggest impact of DA-RT is that it provided a template for the Transparency and Openness Promotion (TOP) Guidelines from the Center for Open Science (COS). In my opinion, the TOP guidelines are more comprehensive and more flexible than what we did with DA-RT and as of right now, 2,900 journals have signed onto them from many disciplines. So just in terms of numbers I think this is the biggest impact of DA-RT.
The other aggregate effect is that studies are showing that more data from more articles are now available. It’s nowhere near perfect because most journals don’t have the ability to enforce it and journals are not going from zero to 10 on the scale of transparency but zero to one or two. But in the aggregate, the more outlets we have moving in the direction of transparency, the more we’re building a culture where transparency is rewarded, measured, and valued. We turn the status quo from “wouldn’t it be nice if…” to actually making data available. That’s the culture we’re trying to move towards and I think every journal that makes a step in that direction gets us closer to greater openness and transparency.
With DA-RT, our endgame was really, could we create moments? Could we create circumstances where people would discuss and move towards greater transparency? We thought originally about building archives or something along those lines, but we realized other people could do that better. So we’ve been more of a “matchmaker.” We direct people to each other and to resources. Now, the DA-RT leaders are doing new things individually, so in some ways DA-RT has evolved through the actions of different people. I’m now an official part of COS, serving as its chair and Colin Elman and Diana Kapieszewski of Georgetown run this amazing summer institution on qualitative research. The things we learned from DA-RT are infused in all of this.
I’m not sure how many more DA-RT-inspired initiatives we’re going to see, but DA-RT has changed the conversation in political science. Now it’s on everybody’s lips and I think that’s all you can ask for. And other organizations like COS and the Association for Psychological Science are working along the same lines by encouraging preregistration, where researchers register the design of their studies before conducting them in an online repository — another great effort that promotes openness and transparency.
My view with transparency is it’s never going to be one-size-fits-all. You’re always going to have this trade-off between how much extra work it is and what’s the value to your stakeholders. My goal is to make it easier for them and the work I’m doing with a number of organizations tries to create the infrastructure to help people who want to be more transparent.
EM: In your opinion, what role do librarians play in these issues?
AL: I think that most researchers don’t understand the critical role that librarians play in distributing research, making it accessible and so forth. I think far too many researchers think of librarians as an afterthought in this process instead of an essential part. Librarians are on the front line of important conversations about the value of different types of information. Every day, they are faced with shifting pressures about the types of information for which various constituencies want to pay — which affects libraries’ abilities to collect, archive and distribute information. A lot of the pressures that the scientific field is facing as a whole in terms of people thinking, “I can get this information on my phone so I don’t need to pay you for it,” or, “An interest group is telling me what’s real so I don’t need science,” librarians feel in a way that most researchers don’t.
Part of the importance of transparency is building a more general narrative about the value of research and scientific information, which librarians can and do play a significant role in. Together, we need to share that the information we provide is reliable and valid because of the scientific method’s properties — a critical task in a competitive marketplace for information and as people navigate between real facts and fake news.
Librarians are at the front lines of these conversations. Researchers can support them by sharing with them our efforts to increase sharing and openness and its effects on the reliability of the research that follows.
EM: Any advice on how librarians can get that message out and ultimately engage more in these efforts?
AL: I’m not sophisticated in the frontiers of library science, nor am I sophisticated in the current best practices in archiving but to me, it seems there’s no fine line anymore between data archivists and librarians. Archiving’s a huge thing right now and goes beyond quantitative data to include the qualitative documentation of what evidence is. Again, folks who think that transparency is some sort of “quantitative takeover,” need to know it’s so not that! Librarians can help our cause by learning how to accurately and effectively convey what a data set is and what it isn’t — what a piece of evidence is and what it isn’t — and the value of quantitative and qualitative approaches.
Ultimately, there are many opportunities for people to better curate, more effectively distribute, and more accurately describe the kinds of research products that we’re putting out. If we’re not fundamentally committed to doing so, then people really should just go to Google for everything. But if researchers, librarians, universities, and publishers find opportunities to communicate that our commitment to transparency means that we provide information that has a set of qualities that can be relied on — then the result can improve knowledge and quality of life for people all over the world. That’s the basis of our service to the world and why a commitment to transparency is so important.