% Template for PLoS
% Version 2.0 July 2014
%
% To compile to pdf, run:
% latex plos.template
% bibtex plos.template
% latex plos.template
% latex plos.template
% dvipdf plos.template
%
% % % % % % % % % % % % % % % % % % % % % %
%
% -- IMPORTANT NOTE
%
% Be advised that this is merely a template
% designed to facilitate accurate translation of manuscript content
% into our production files.
%
% This template contains extensive comments intended
% to minimize problems and delays during our production
% process. Please follow the template
% whenever possible.
%
% % % % % % % % % % % % % % % % % % % % % % %
%
% Once your paper is accepted for publication and enters production,
% PLEASE REMOVE ALL TRACKED CHANGES in this file and leave only
% the final text of your manuscript.
%
% DO NOT ADD EXTRA PACKAGES TO THIS TEMPLATE unless absolutely necessary.
% Packages included in this template are intentionally
% limited and basic in order to reduce the possibility
% of issues during our production process.
%
% % % % % % % % % % % % % % % % % % % % % % %
%
% -- FIGURES AND TABLES
%
% DO NOT INCLUDE GRAPHICS IN YOUR MANUSCRIPT
% - Figures should be uploaded separately from your manuscript file.
% - Figures generated using LaTeX should be extracted and removed from the PDF before submission.
% - Figures containing multiple panels/subfigures must be combined into one image file before submission.
% See http://www.plosone.org/static/figureGuidelines for PLOS figure guidelines.
%
% Tables should be cell-based and may not contain:
% - tabs/spacing/line breaks within cells to alter layout
% - vertically-merged cells (no tabular environments within tabular environments, do not use \multirow)
% - colors, shading, or graphic objects
% See http://www.plosone.org/static/figureGuidelines#tables for table guidelines.
%
% For sideways tables, use the {rotating} package and use \begin{sidewaystable} instead of \begin{table} in the appropriate section. PLOS guidelines do not accomodate sideways figures.
%
% % % % % % % % % % % % % % % % % % % % % % % %
%
% -- EQUATIONS, MATH SYMBOLS, SUBSCRIPTS, AND SUPERSCRIPTS
%
% IMPORTANT
% Below are a few tips to help format your equations and other special characters according to our specifications. For more tips to help reduce the possibility of formatting errors during conversion, please see our LaTeX guidelines at http://www.plosone.org/static/latexGuidelines
%
% Please be sure to include all portions of an equation in the math environment, and for any superscripts or subscripts also include the base number/text. For example, use $mathrm{mm}^2$ instead of mm$^2$ (do not use \textsuperscript command).
%
% DO NOT USE the \rm command to render mathmode characters in roman font, instead use $\mathrm{}$
% For bolding characters in mathmode, please use $\mathbf{}$
%
% Please add line breaks to long equations when possible in order to fit our 2-column layout.
%
% For inline equations, please do not include punctuation within the math environment unless this is part of the equation.
%
% For spaces within the math environment please use the \; or \: commands, even within \text{} (do not use smaller spacing as this does not convert well).
%
%
% % % % % % % % % % % % % % % % % % % % % % % %
\documentclass[10pt]{article}
% amsmath package, useful for mathematical formulas
\usepackage{amsmath}
% amssymb package, useful for mathematical symbols
\usepackage{amssymb}
% cite package, to clean up citations in the main text. Do not remove.
\usepackage{cite}
\usepackage{hyperref}
% line numbers
\usepackage{lineno}
% ligatures disabled
\usepackage{microtype}
\DisableLigatures[f]{encoding = *, family = * }
% for proper rendering of Pawel Szczesny's surname with ogonek
\usepackage[T1]{fontenc}
% rotating package for sideways tables
%\usepackage{rotating}
% If you wish to include algorithms, please use one of the packages below. Also, please see the algorithm section of our LaTeX guidelines (http://www.plosone.org/static/latexGuidelines) for important information about required formatting.
%\usepackage{algorithmic}
%\usepackage{algorithmicx}
% Use doublespacing - comment out for single spacing
%\usepackage{setspace}
%\doublespacing
% Text layout
\topmargin 0.0cm
\oddsidemargin 0.5cm
\evensidemargin 0.5cm
\textwidth 16cm
\textheight 21cm
% Bold the 'Figure #' in the caption and separate it with a period
% Captions will be left justified
\usepackage[labelfont=bf,labelsep=period,justification=raggedright]{caption}
% Use the PLoS provided BiBTeX style
\bibliographystyle{plos2009}
% Remove brackets from numbering in List of References
\makeatletter
\renewcommand{\@biblabel}[1]{\quad#1.}
\makeatother
% Leave date blank
\date{}
\pagestyle{myheadings}
%% Include all macros below. Please limit the use of macros.
%% END MACROS SECTION
\begin{document}
% Title must be 150 characters or less
\begin{flushleft}
{\Large
\textbf{The transformative nature of transparency in research funding}
}
% Insert Author names, affiliations and corresponding author email.
\\
Daniel Mietchen$^{1,\ast}$
\\
\bf{1} Museum f{\"u}r Naturkunde Berlin~-- Leibniz-Institut f{\"u}r Evolutions- und Biodiversit{\"a}tsforschung, Invalidenstr. 43, 10115 Berlin, Germany
\\
$\ast$ E-mail: daniel.mietchen@mfn-berlin.de
\end{flushleft}
\section*{Abstract}
A central element of research funding are grant proposals that researchers send in to potential funders for review and in the hope for approval. A survey of policies at major research funders found that there is room for more transparency in the process, which would strengthen the case for the efficiency of public spending on research. On that basis, debate was invited on which transparency measures should be implemented and how, with some concrete suggestions at hand. The present article adds to this discussion by providing further context from the literature, along with considerations on the effect size of the proposed measures. It then explores the option of opening key components of the process, makes the case for pilot projects in this area and sketches out the potential that such measures might have to transform the research landscape in those corners where they get implemented.
\section*{Transparency in research funding? Discuss.}
In this issue, Gurwitz et al.\ \cite{gurwitz2014grant} (subsequently GMK) report on a survey of transparency at major funders of biomedical research. They looked at what information funders make public about assessment procedures and funded proposals, as detailed in their Table~1.
On that basis, GMK then discuss the merits of adding transparency to the proposal review process in one of two ways: in what they refer to as the ``incremental'' approach, individual components of the process would be made more open. Focusing on components that ``should not do any harm on the evaluation procedure at all'', they suggest that funders publish
\begin{enumerate}
\item (once funding decisions have been announced for a call) a list of the members of that call's review panels, along with a cumulative list of external reviewers
\item statements within the application about the expected impact of the proposed research
\item the final reports of funded projects
\end{enumerate}
The second approach~-- termed the ``radical'' one~-- is about opening up the review process as a whole, rather than selected parts thereof. The authors provide some practical considerations as to why that might be useful:
\begin{itemize}
\item it addresses reviewer fatigue
\item published peer reviews can be helpful to readers
\item it would promote rather than inhibit collaboration between researchers
\item it would allow more public participation in research
\end{itemize}
They suggest the radical approach may be ``quite transformative'' (or ``sweeping'') in terms of both scholarly communication and public participation in research. They caution that the current research system and associated evaluation procedures are not set up for such radical changes, and then conclude by inviting debate on ``which transparency measures to put in place, and how''.
In the following, I would like to follow this invitation by entertaining some thoughts on an open research funding system, its transformative nature and how we might be able to pave the way to get there.
\section*{Where is the evidence?}
The most comprehensive meta analysis of grant peer review to date is a 2007 paper by Demicheli and Di Pietrantonj \cite{demicheli2007peer} (subsequently D\&D). GMK quote one of its key statements: \emph{ ``there is little empirical evidence on the effects of grant giving peer review''}.
This is worth reading again, with an appreciation of how central the peer review of grant proposals has been~-- and continues to be~-- to our research funding system.
D\&D entitled their article ``Peer review for improving the quality of grant applications'', which is interesting from a transparency perspective. Peer review can only improve the quality of grant applications if there is some form of feedback between the proposal and the review process, either within one round of submissions, across rounds or across funders. Such feedback would certainly have a much wider reach (some would say ``impact'') if it were public.
It is thus no surprise that one of the main conclusions by D\&D is that ``[a]ttempts to improve the efficiency and the transparency of the process and actions encouraging innovative ideas should be implemented and evaluated''~-- the other is that ``[e]xperimental studies assessing the effects of grant giving peer review on [$\ldots$] funded research are urgently needed.'' \cite{demicheli2007peer}
Similar thoughts have been expressed in \cite{azoulay2012research}: ``[i]t is time to turn the scientific method on ourselves [$\ldots$] by subjecting proposed reforms to a prospective, randomized controlled experiment.''
This idea makes intuitive sense, but it still does not tell us where to start with transparency.
\section*{Effect size}
Let's assume that some funding agencies would like to open up their procedures and opt to implement the proposed incremental changes outlined above. How long would it take before any potential effects could be measured?
If, on the other hand, they were to implement some more profound (or ``radical'') changes, shouldn't that result in larger effects that could be more readily observed and would allow to inform relevant policy work earlier? Perhaps this works best under some controlled conditions~-- what might these look like?
GMK identified ``at least three'' parameters suitable for incremental change. In light of the complexity of the research funding system, it is not difficult to come up with further candidates by decomposing the funding process into its components. For instance, \cite{demicheli2007peer} have looked at different ways of processing submissions, of getting internal and external reviews, of making decisions based on those reviews, and of providing feedback to submitters. Other changes worth considering could include more transparency about the way funding calls or eligibility criteria are being designed, or to make public (and ideally machine readable) the data management plans now increasingly required for new proposals. Much of this could perhaps be engineered in a way that obeys the ``no harm'' condition put forth by GMK but the magnitude of effects expected from any such incremental changes is not obvious.
What if final reports were public, as GMK suggest? Would they be useful without the context of the original proposals?
\section*{Opening up research proposals}
Now consider publishing the proposals themselves, assuming for the sake of argument that legal issues~-- e.g.\ as to who has the right to publish them~-- are no showstopper. GMK labeled this in their Box~2 as one of two ``radical transparency measures'' deemed ``premature to recommend their open publication by default'', adding that they ``would welcome small-scale experimentation in this area''. The publication of the proposals could happen at various points in time, e.g.
\begin{enumerate}
\item some years after a project has ended
\item along with the final reports for a project
\item at the beginning of a project
\item at the point of announcing funding decisions
\item upon submission to the funder
\item during the drafting phase
\end{enumerate}
Knowing that many researchers have little attention to spare outside of their usual workflows, few of them would be expected to systematically go through past proposals, on the basis of which funds have already been spent (scenarios 1 and 2) or at least allocated (scenarios 3 and 4).
The situation is different if funds have not been allocated yet: scenario 5 may be useful for researchers to compare proposals and to gauge the likelihood of their own ones being funded. Furthermore, it might provide a basis for reaching out to potential collaborators for other proposals they are working on (or for the next stage, in case of multi-stage procedures). Going down the list, the potential for collaboration increases and is greatest in scenario 6 \cite{agrawal2008restructuring,levine2014open}. It can be enhanced further by providing a public version history and allowing comments, edits and forking or other kinds of reuse \cite{slaughter2014continuous}. According to what GMK have framed as the ``conventional wisdom'' that sharing research before formal publication ``would conflict with researchers' interests'', scenario 6 would seem to be the least compatible with the ``no harm'' approach. On the other hand, it is hard to imagine how scenarios 1 and 2 might cause any harm at all for successful proposals. Even for unsuccessful ones from the same call, the effects should be minimal after so much time has gone by. Either way, there is very little data on any of these scenarios, so ``small-scale experimentation in this area'' would indeed seem like a good option to find out more.
The concept of discussing proposals in public is not entirely new: participatory budgeting \cite{peixoto2009beyond} and grantmaking \cite{baiocchi2007could} have been explored in a number of societal contexts, whereas large-scale research infrastructures like the Large Hadron Collider \cite{smith2012large} have long been planned in public, and at the lower end of the budget scale, recent years have seen a first wave of crowdfunding initiatives for research \cite{wheat2013raising}, with proposals being public by default. At the typical scale of projects funded by the agencies surveyed by GMK, the default is certainly not to publish proposals, though a few individuals and groups do it nonetheless \cite{zeilberger2011opinion117,white2011list}. While budgets are interesting, they depend on a number of non-scientific parameters, so making them public is less essential than publishing the core scientific idea along with a detailed plan on how to put it into practice.
\section*{Catalysts for change}
Once a good number of proposals were open, lots of other changes towards openness would follow across the entire research system.
A natural next step would be to make the peer reviews public \cite{mietchen2011peer}~-- there is little benefit in keeping them secret if the corresponding proposals are public. GMK disagree: publishing ``[d]etailed (external) reviews'' is the second of the two ``radical transparency measures'' that they identified in their Box~2. They state "open access to individual grant review reports may damage reviewers and discourage honest review", but this seems to assume that reviewer identities were always required to be published along with the reviews, for which I see no necessity. In fact, Copernicus journals have been running public peer review for a decade, and reviewers can still choose to remain anonymous \cite{poschl2009interactive}. Why shouldn't this work for grant reviews too? Conversely, mixed reviewing models may speed up the process~-- why not let proposal authors solicit signed peer reviews from uninvolved authorities in their field and publishing them along with the proposal, while inviting the broader community to comment? The small remaining potential for dishonest review can be balanced by classical independent reviews with the option to remain anonymous.
Having proposals and reviews out in the open would allow anyone to consult them when writing or reviewing proposals themselves, and thus help with establishing, maintaining and teaching quality standards \cite{bourne2006ten,nicholson2012research}. Other consequences of open proposals could be that rejected proposals could more easily be built upon \cite{tatsioni2010sources} and that data shown in a proposal might become more likely to actually end up in public databases~-- and earlier~-- or that research might be performed more openly, given that the basic ideas are public already. Researchers with a public track record that goes beyond formal publications can eventually be evaluated more on ``what they did'', rather than ``where they published'', which may mean less proposal writing and more time for research \cite{ioannidis2011more}.
Data miners could develop tools that highlight assertions in the proposal, link them to the published literature (which would increasingly include proposals) and ping a paper, dataset, proposal or review when it is cited. Journalists, museums or other science communicators could begin to interact with research projects before these even start and embed themselves and their audiences into the research process much more than now, thereby facilitating new approaches to public engagement with science. Similarly, fellow researchers~-- or their automated tools~-- might engage with proposals or the teams behind them in new ways \cite{patil2009revolution,celi2014crowdsourcing,franzoni2014crowd}, as examplified by the Polymath project \cite{gowers2009massively} or the \emph{E. coli} O104:H4 Genome Analysis Crowd-Sourcing Consortium \cite{rohde2011open}, both of which solved complex problems much faster than usual in their domain, due to efficient collaboration.
\section*{Conclusions}
The article by GMK provides a snapshot of transparency-related practices across major funders of biomedical research and a timely stimulation of debate around this important topic. What I find most illustrative is what none of the surveyed funders publish:
\begin{enumerate}
\item assessment summaries
\item proposals or reviews thereof
\item information about pending or rejected proposals
\end{enumerate}
GMK recommend publication in case 1 but not for the other two cases, although they join \cite{demicheli2007peer} in inviting experimentation around case 2. I think experimentation has to be encouraged along all three lines and well beyond, since ``[i]t would be a fortuitous coincidence if the systems that served us so well in the twentieth century were equally adapted to twenty-first-century needs'' \cite{azoulay2012research}. If there are legal barriers to making funding mechanisms more transparent, these have to be addressed in a timely fashion.
According to GMK, funders ``must embrace transparency more actively.'' To me, this includes research funding mechanisms in general~-- from individual grants and their peer review up to entire calls and programs~-- as well as assessing their efficiencies \cite{stephan2012research}. In particular, ``developing countries could leapfrog ahead by adopting from the start science grant systems that encourage innovation'' \cite{gordon2009cost}, and more systemic transparency may help overcome inequalities in terms of age \cite{kaiser2014call} or other aspects of diversity \cite{fortin2013big}.
It would seem promising to start with publishing successful proposals from the past, along with their reviews, assessment summaries and final reports. Over time, the embargo period could be reduced, and hopefully, it will eventually vanish. Until then, researchers should be encouraged to share their proposals, reports and associated reviews early on, and the public to explore these new opportunities for engaging with research.
% Do NOT remove this, even if you are not including acknowledgments.
\section*{Acknowledgments}
This article has benefited from multiple discussions~-- online or in person~-- on open approaches to research funding, particularly with Pawe{\l} Szcz{\k{e}}sny, Mike Linksvayer, Paul Gardner, Peter Murray-Rust, Phil Bourne, Lyubomir Penev and Jan Velterop.
%\bibliography{main}
\begin{thebibliography}{10}
\providecommand{\url}[1]{\texttt{#1}}
\providecommand{\urlprefix}{URL }
\expandafter\ifx\csname urlstyle\endcsname\relax
\providecommand{\doi}[1]{doi:\discretionary{}{}{}#1}\else
\providecommand{\doi}{doi:\discretionary{}{}{}\begingroup
\urlstyle{rm}\Url}\fi
\providecommand{\bibAnnoteFile}[1]{%
\IfFileExists{#1}{\begin{quotation}\noindent\textsc{Key:} #1\\
\textsc{Annotation:}\ \input{#1}\end{quotation}}{}}
\providecommand{\bibAnnote}[2]{%
\begin{quotation}\noindent\textsc{Key:} #1\\
\textsc{Annotation:}\ #2\end{quotation}}
\providecommand{\eprint}[2][]{\url{#2}}
\bibitem{gurwitz2014grant}
Gurwitz D, Milanesi E, Koenig T (2014) {Grant Application Review: The Case of
Transparency}.
\newblock PLOS Biology 12: in press.
\bibAnnoteFile{gurwitz2014grant}
\bibitem{demicheli2007peer}
Demicheli V, Di~Pietrantonj C (2007) {Peer review for improving the quality of
grant applications}.
\newblock Cochrane Database Syst Rev 2.
\bibAnnoteFile{demicheli2007peer}
\bibitem{azoulay2012research}
Azoulay P (2012) {Research efficiency: Turn the scientific method on
ourselves}.
\newblock Nature 484: 31--32.
\bibAnnoteFile{azoulay2012research}
\bibitem{agrawal2008restructuring}
Agrawal A, Goldfarb A (2008) {Restructuring Research: Communication Costs and
the Democratization of University Innovation}.
\newblock The American Economic Review 98: 1578--1590.
\bibAnnoteFile{agrawal2008restructuring}
\bibitem{levine2014open}
Levine SS, Prietula MJ (2014) {Open Collaboration for Innovation: Principles
and Performance}.
\newblock Organization Science December 25: 1414--1433.
\bibAnnoteFile{levine2014open}
\bibitem{slaughter2014continuous}
Slaughter AE, Gaston DR, Peterson J, Permann CJ, Andrs D, et~al. (2014)
{Continuous Integration for Concurrent MOOSE Framework and Application
Development on GitHub}.
\newblock FigShare 10.6084/m9.figshare.1112585.
\bibAnnoteFile{slaughter2014continuous}
\bibitem{peixoto2009beyond}
Peixoto T (2009) {Beyond Theory: e-Participatory Budgeting and its Promises for
eParticipation}.
\newblock European Journal of ePractice 7: 55--63.
\bibAnnoteFile{peixoto2009beyond}
\bibitem{baiocchi2007could}
Baiocchi G, Lerner J (2007) {Could Participatory Budgeting Work in the United
States?}
\newblock The Good Society 16: 8--13.
\bibAnnoteFile{baiocchi2007could}
\bibitem{smith2012large}
Smith CL (2012) {The Large Hadron Collider: lessons learned and summary}.
\newblock Philosophical Transactions of the Royal Society A: Mathematical,
Physical and Engineering Sciences 370: 995--1004.
\bibAnnoteFile{smith2012large}
\bibitem{wheat2013raising}
Wheat RE, Wang Y, Byrnes JE, Ranganathan J (2013) {Raising money for scientific
research through crowdfunding}.
\newblock Trends in ecology \& evolution 28: 71--72.
\bibAnnoteFile{wheat2013raising}
\bibitem{zeilberger2011opinion117}
Zeilberger D (2011) {Appendix to Doron Zeilberger's Opinion 117: Links to
posted Grant Proposals}.
\newblock Opinions of Doron Zeilberger
\href{http://www.math.rutgers.edu/~zeilberg/Opinion117Appendix.html}{http://www.math.rutgers.edu/$\sim$zeilberg/Opinion117Appendix.html}
(archived at
\href{http://www.webcitation.org/6TpTmJZUC}{http://www.webcitation.org/6TpTmJZUC}).
\bibAnnoteFile{zeilberger2011opinion117}
\bibitem{white2011list}
White E (2012) {A list of publicly available grant proposals in the biological
sciences}.
\newblock Jabberwocky Ecology
\href{http://jabberwocky.weecology.org/2012/08/10/a-list-of-publicly-available-grant-proposals-in-the-biological-sciences/}{http://jabberwocky.weecology.org/2012/08/10/a-list-of-publicly-available-grant-proposals-in-the-biological-sciences/}
(archived at
\href{http://www.webcitation.org/6TpTl9Z84}{http://www.webcitation.org/6TpTl9Z84}).
\bibAnnoteFile{white2011list}
\bibitem{mietchen2011peer}
Mietchen D (2011) {Peer reviews: make them public}.
\newblock Nature 473: 452--452.
\bibAnnoteFile{mietchen2011peer}
\bibitem{poschl2009interactive}
P{\"o}schl U (2009) {Interactive Open Access Peer Review: The Atmospheric
Chemistry and Physics Model}.
\newblock Against the Grain 21: 11.
\bibAnnoteFile{poschl2009interactive}
\bibitem{bourne2006ten}
Bourne PE, Chalupa LM (2006) {Ten simple rules for getting grants}.
\newblock PLoS Computational Biology 2: e12.
\bibAnnoteFile{bourne2006ten}
\bibitem{nicholson2012research}
Nicholson JM, Ioannidis JP (2012) {Research grants: conform and be funded}.
\newblock Nature 492: 34--36.
\bibAnnoteFile{nicholson2012research}
\bibitem{tatsioni2010sources}
Tatsioni A, Vavva E, Ioannidis JP (2010) {Sources of funding for Nobel
Prize-winning work: public or private?}
\newblock The FASEB Journal 24: 1335--1339.
\bibAnnoteFile{tatsioni2010sources}
\bibitem{ioannidis2011more}
Ioannidis JP (2011) {More time for research: fund people not projects}.
\newblock Nature 477: 529--531.
\bibAnnoteFile{ioannidis2011more}
\bibitem{patil2009revolution}
Patil C, Siegel V (2009) {This revolution will be digitized: online tools for
radical collaboration}.
\newblock Disease models \& mechanisms 2: 201--205.
\bibAnnoteFile{patil2009revolution}
\bibitem{celi2014crowdsourcing}
Celi LA, Ippolito A, Montgomery RA, Moses C, Stone DJ (2014) {Crowdsourcing
Knowledge Discovery and Innovations in Medicine}.
\newblock Journal of medical Internet research 16: e216.
\bibAnnoteFile{celi2014crowdsourcing}
\bibitem{franzoni2014crowd}
Franzoni C, Sauermann H (2014) {Crowd science: The organization of scientific
research in open collaborative projects}.
\newblock Research Policy 43: 1--20.
\bibAnnoteFile{franzoni2014crowd}
\bibitem{gowers2009massively}
Gowers T, Nielsen M (2009) {Massively collaborative mathematics}.
\newblock Nature 461: 879--881.
\bibAnnoteFile{gowers2009massively}
\bibitem{rohde2011open}
Rohde H, Qin J, Cui Y, Li D, Loman NJ, et~al. (2011) {Open-source genomic
analysis of Shiga-toxin--producing {\emph E. coli} O104: H4}.
\newblock New England Journal of Medicine 365: 718--724.
\bibAnnoteFile{rohde2011open}
\bibitem{stephan2012research}
Stephan P (2012) {Research efficiency: Perverse incentives}.
\newblock Nature 484: 29--31.
\bibAnnoteFile{stephan2012research}
\bibitem{gordon2009cost}
Gordon R, Poulin BJ (2009) {Cost of the NSERC Science Grant Peer Review System
Exceeds the Cost of Giving Every Qualified Researcher a Baseline Grant}.
\newblock Accountability in Research 16: 13-40.
\bibAnnoteFile{gordon2009cost}
\bibitem{kaiser2014call}
Kaiser J (2014) {A call for NIH youth movement}.
\newblock Science 346: 150-151.
\bibAnnoteFile{kaiser2014call}
\bibitem{fortin2013big}
Fortin JM, Currie DJ (2013) {Big science vs. little science: how scientific
impact scales with funding}.
\newblock PLOS ONE 8: e65263.
\bibAnnoteFile{fortin2013big}
\end{thebibliography}
\end{document}
