Everything You Always Wanted to Know About PCs, But Were Afraid to Ask – Communications of the ACM

Okay, PCs in the title could be Political Correctness or Personal Computers or even Peace Corps. But it is not. It stands for Program Committees. As researchers, in academia or industry, we often are asked to serve on Program Committees of conferences in our fields of expertise. Serving on PCs signals one is a good citizen of our global technical village and has its own altruistic rewards. Beyond that, it has substantive and far-reaching impact on our professional careers — through building connections, getting our names out there, and learning the art and the science of getting our work published at the prestigious conferences. Here I share a few lessons I have learned serving on PCs, chairing PCs, and being part of leadership of professional organizations that run conferences (IEEE, in my case). These lessons are meant to help us get the most out of the substantial time and effort we put into serving on PCs.

1. How many PCs is enough?

This is, of course, a very personal decision. For me, and for many sane colleagues of mine who run active and fair-sized research programs, we would say yes to 4-6 PCs in a year, for systems or security conferences, and 5-8 for AI/ML conferences. The AI/ML conferences tend to have much fewer number of submissions assigned to each reviewer. A good balance is needed in choosing the number of PCs one commits to. We need to say no to some requests to serve on PCs so that we can really do justice to the ones we commit to. Reviewing is time-consuming, since you want to do it well. On the other hand, too many declines means you are not playing your part in keeping your technical community vibrant. Further, that sends off the wrong vibe that you are not a good citizen of your technical community and are more of an extractor than contributor — submitting lots of papers, but not stepping up to review submissions.

2. Are review submission deadlines just made up?

Not at all. They are real deadlines, of course with some slack built in. The slack is usually 2-5 days, meaning if you slip by that, you do not seriously derail the subsequent steps of the review process. This means when you are on the PC, you better stick to your end of the bargain and submit your reviews, if not before the deadline, then at least within the slack period.

I know one matter that infuriates PC chairs is if a member batches up all her reviews and submits them all together, rather than uploading when each review is done. We all may have our good reasons for batching; we want to calibrate across all our reviews, we want a certain minimum or maximum number of submissions in our pile that we want to recommend for acceptance. But this approach throws a wrench in the review process. The Chairs cannot plan for missing reviews, such as by asking others to review the ones that you are likely to blow off. Or in cases of asking for tie-breaking reviews when the existing ones are deadlocked. So do stick to your review submission deadlines. We all know how to live by strict deadlines — for our own paper and proposal submissions. Just bring that same mindset to the reviewing process.

3. Panglossian or Downbeat: Where is the balance?

I learned this new word “Panglossian” recently and finally this is my chance to use it! It means excessively optimistic and is not a complementary term. It is based on the fictional Dr. Pangloss from Voltaire’s satirical novel, Candide.

Anyway, coming to the question at hand, of course each of us falls somewhere in the spectrum from the super grouch to the super optimistic. Some of us are firmly stuck to one point in this spectrum, while some calibrate this based on the quality of the submissions. All of us are influenced by other things going on in our lives when we review; reviewers after all are humans, only too human some would say. Knowing all this to be a pragmatic fact of life, the only exhortation I can make is to go in with as open a mind as possible when starting to review a submission and calibrate your bar for acceptance with the quality of the conference — the historical acceptance rate for the conference serves as an imperfect and yet valuable metric for quality.

So, get the most out of the substantial time and effort we put into serving on PCs.“Do not expect a submission to be perfect for you to champion its acceptance.” This has been said almost universally in instructions by PC chairs, so much so that it has become akin to a mantra, but this is an often-ignored mantra. I would like to reinforce that message here. Your submissions may come out without any rough edges, but those of mere mortals always do. And such submissions also deserve to see the light of day.

4. Can I hide behind my anonymity?

One only has to look at the corrosive nature of comments sections on Internet forums to be convinced about the ill-effects of anonymity in posting something. As reviewers, our identities are always hidden from the authors (for very good reason) and to the world at large. Sometime, we tend to lose our commonsense notion of civility or rationality due to the seeming Potteresque cloak of invisibility that this anonymity bestows upon us. But beware, this cloak of invisibility is merely an illusion. Your fellow PC members get to see when one is being uncivil or unjust in one’s reviews. The PC Chairs and sometimes the Steering Committee members get to see your behavior as PC members. And the sum total of the PC members, aggregated over a few conferences, comprises your technical community. So do what you do when you are not anonymous — you are civil in your arguments, backing them up with evidence, and you are not a flame thrower just to enjoy the light show that will ensue. The “Golden Rule” is as true here as in anything else.

5. To ask researchers in my group to help with reviewing or not?

If you are a purist, you will not like my view here. My view is that it is a useful exercise to ask junior researchers in your group — senior Ph.D. students, post-doctoral scholars — to help you with reviewing. This serves the self-serving purpose that it reduces your own reviewing load, and then it serves the broad-minded purpose of teaching these junior researchers an essential skill. This is a good approach in my view, but with some important caveats. First, you should discuss the review with the junior researcher and have confidence in presenting the opinion on the submission in front of the PC. Second, you should review parts of the submission, even large parts, if you find the opinion of the researcher you delegated the review to is suspect. Finally, if the submission is contentious (i.e., there are good arguments both for and against it), then you should dive in and do a full-blown review yourself because you may end up being the tie-breaker.

In Summary

To sum, publications at our top conferences act as the paramount indicator of research excellence. The Program Committees of these conferences act as the sole gatekeepers and Program Committees are composed of us, not some omniscient entities. So there are important ground rules, often learned only through experience, that one needs to follow to make the process fulfilling for us and fair and productive for our technical communities. In this post, I have shared my opinionated view of five such ground rules. I would love to hear if you have contrarian views on them.

This post was originally published on Distant Whispers.

Saurabh Bagchi is a professor of Electrical and Computer Engineering and Computer Science at Purdue University, where he leads a university-wide center on resilience called CRISP. His research interests are in distributed systems and dependable computing, while he and his group have the most fun making and breaking large-scale usable software systems for the greater good.