iCog research grants selection process

iCog had some legacy funds to disburse in the 2019/20 academic year. In keeping with iCog’s remit to support and encourage interdisciplinary research, the iCog steering committee decided that the best use for these funds would be to award grants for the participant costs of empirical research by junior researchers doing interdisciplinary work in cognitive science.

Projects which were particularly likely to fall through the gaps of the funding remit of other funding bodies due to their interdisciplinary nature would be given priority.

33 applications were received: 9 from the UK, 8 from the US, 4 from Germany, 3 from Switzerland, 2 from Finland, and 1 each from Australia, Austria, Belgium, France, Italy, the Netherlands, and Turkey.

The selection process

Usually all that an applicant for research funding is told is that his or her application was or (more often – in this case about ten times more often) was not successful, while the process leading to that result remains opaque. With this post, we wish to bring some transparency to the selection process we used. By explaining in some detail here how we proceeded and why, we hope to show that the process was fair and systematic.

No selection process is perfect. But we also hope that our way of proceeding with this selection will find imitators across the academy, in particular in its strict commitment to blind reviewing, as detailed in the next section. Preparing this can be a little labour-intensive. But, in the interest of fairness to all applicants, blinding applications as needed was a labour worth doing.

Of the total of 33 applications received, one application was ineligible as its purpose fell outside the remit of the grants, i.e. participant costs in empirical studies. The 32 eligible applications were blind reviewed, initially by two academic reviewers (drawn from a grants review subcommittee of five) with cross-disciplinary expertise but from two different ‘home’ disciplines, who assessed and scored their quality, interdisciplinarity, and feasibility. 

From this process four research proposals emerged with very high aggregate scores from both reviewers. These were automatically shortlisted. 

A further five applications had sufficiently high scores from at least one reviewer to be considered further; these proposals were then each reviewed by a third reviewer. Two of these five proposals thereby reached an aggregate score comparable to that of the four proposals already shortlisted, leading to a final shortlist of six. (The length of the shortlist was dictated by the need to keep it manageable for the grants subcommittee to review all shortlisted proposals.)

Common reasons why proposals were rejected at this stage were that other proposals were qualitatively superior, insufficient detail on the method or feasibility of the proposed studies, or – in some cases – lack of interdisciplinarity.

The six shortlisted research proposals, along with their applicants’ CVs and the two or three initial reviewer’s reports and scores, were then submitted to the full grants subcommittee for final deliberation. This was done by a combination of discussion and, where this proved inconclusive, votes. 

Taking into account available funds and the amount of funding requested by applicants, the subcommittee’s decision was to award full funding to two research proposals and partial funding to a third. The three shortlisted applicants who were ultimately unsuccessful were offered feedback on their applications; all of them took up that offer.

Blind review

The entire review process up to and including the final decisions on funding was ‘blind’. In the stages up to shortlisting, reviewers were not shown the names or affiliations of the applicants. For the shortlisted applications, affiliations were then visible insofar as they appeared on applicants’ CVs, but applicants’ names were still redacted from these. Why and how did we do that?

The rationale for blind reviewing is simple: it is to avoid reviewers having implicit biases on the grounds of an applicant’s gender, presumed background, or location. Personal names often reveal a person’s gender and ethnic origin, so these were withheld from reviewers throughout the process. 

The decision also to withhold applicants’ institutional affiliations during the first stages of the review process, up to shortlisting, was likewise to prevent reviewers from being influenced by favourable or unfavourable biases they might have about an applicant’s institution or department or its location. (For the shortlist, it would not have been practicable to continue withholding affiliations, since applicants’ CVs were also being considered at that stage.)

As to the method of ensuring blind reviewing, this too was relatively straightforward. The application form was designed in such a way that applicants’ research proposals could easily be pulled out of the data submitted without the accompanying personal or institutional details (and just as easily matched again at the end of the process, by means of a unique ID allocated to each application). 

We had asked for the research proposals submitted to be ‘suitable for blind reviewing’. Many applicants were commendably assiduous in complying with this request, though some were not – we ought perhaps to have been more explicit by e.g. putting a note on the application form requesting ‘No names or affiliations in the text of the proposal, please’. We had likewise asked for uploaded CVs to contain the applicant’s name only once, to speed up the blinding process.

To make sure that the material passed to reviewers was indeed ‘suitable for blind reviewing’, it was necessary to redact any mention of applicants’ names or gendered personal pronouns and institutional affiliations from research proposals and, in the CVs of shortlisted applicants, redact the one occurrence of their name from the document. We were able to do this because the iCog committee had designated an administrator for this purpose, as well as for organising the review process generally, without himself being a member of the review subcommittee.