The myth of the “expert panel”


(This post is based on an excerpt from Win More Grants eBook.)

A common phrase to be heard in the procedures that evaluate, and decide upon, research grant applications, is the “expert panel”.  Many of us wish to hold to the idea that the so-called “expert panel” should just be trusted to get it right – to make a wise choices about which grants should be funded, and which should not. They are, after all, experts.

Unfortunately, this is entirely misleading.

That phrase, “expert panel”, is often used to describe the “jury” that will judge your grant application. But the word “expert” can mean many things, and in this case it is used entirely misleadingly. What your funding agency is actually saying is that the panel is composed of members who are knowledgable experts in their field, not that they are experts at making judgements about grant applications. Unsurprisingly, therefore, we tend to conflate the two, and think they are the same. They are not the same. To become an expert at choosing the best grants, you would need feedback on your decision making. Without feedback on a task you simply cannot become an expert in that task. You cannot become an expert dart thrower without knowing what score you got each time. You cannot become an expert decision maker without knowing the consequences of decisions.

To become an expert grant selector would require situations whereby you could grade a set of grant applications, have them all funded and then later receive feedback on how well they all did, and whether your predicted rankings were accurate or not. To the best of my knowledge, this never happens.

Given that most grant applications are of excellent quality anyway, it is not surprising that the grants that are funded are successful. This positively reinforces the funding agency’s perception that the experts are doing their job well, and positively reinforces the panel members perception that they are experts at choosing grants. So, the important point to note is that when people refer to an “expert panel” they are referring to expertise in the subject, not expertise in grading grants. That is a pretty important distinction.

Unfortunately, nobody seems to make that distinction.   But remember it, next time you apply of a grant, or next time you are on a panel.