Stack Overflow (SO) is a pivotal online community where developers gather to learn and share knowledge in a question-and-answer format on a vast array of programming topics.
For their contributions, SO users receive reputation points. This fact raised a question in the minds of five Canadian researchers: Are SO reputation points an accurate measure of a user’s expertise?
To answer it, they launched a study that examined the reputation-related activities of more than 93,000 high-reputation SO users. The answer they found is unequivocally stated in the title of their resulting research paper, “Is Reputation on Stack Overflow Always a Good Indicator for Users’ Expertise? No!”
The Study and Its Findings
When SO users earn reputation points, it strengthens their profiles, which can—as the study points out—bring job opportunities as (according to other research) reputation is sometimes used as a proxy for expertise and skill. A heavy burden, perhaps, but assuming that a reputation is based on knowledge, it is not a terribly problematic one.
For example, as the authors point out, if an SO user provides insightful answers to many questions on a particular topic, it’s reasonable to consider them as possessing some expertise on that topic.
However, relying solely on accumulated SO reputation points to measure expertise can be misleading. The study found, for example, that the knowledge that such points are based on can sometimes recline comfortably on the head of a pin. Or rather, on a single answer to a single question. This was in fact the case in 12% of the 93,000+ users studied.
Further, the study investigated how those high-reputation users earned their points and found that nearly 14% earned more than 50% of their points by asking questions. Putting aside the fading cultural values of curiosity and humility to dig deeper, the study highlights that how other users vote on (that is, up-vote or down-vote) a question or an answer is a key factor in how many points a contribution adds or subtracts from the contributor’s reputation tally.
One question, for example, received nearly 28,000 upvotes, translating to upwards of 280,000 reputation points. While this case is a massive outlier in terms of up-vote activity, it does beg the question of whether the asker has expertise in the topic or is merely someone skilled at pinpointing a vexing problem that would soon haunt every developer on Earth.
Where to Find the Findings (and More)
In addition to describing the SO reputation points system, the paper details the study’s methodology and its results, including several graphs and tables.
The researchers also discuss the implications of their findings and recommend a new measure—similar to academia’s h-value—that SO might consider adopting to better evaluate user expertise.
Download “Is Reputation on Stack Overflow Always a Good Indicator for Users’ Expertise? No!”
Download the Full Study
