What is the value of the research we do? The typical way we have to evaluate the impact of research is to count citations, and possibly weigh them in some way, in Economics and any other sciences (except maybe where patents are relevant). But this only evaluates how the research output is viewed within a narrowly defined scientific community. The contribution to social welfare is an entirely different beast to evaluate.
Robert Hofmeister tries to give research some value. The approach is to consider the scientific process through cohorts, where each wave provides fundamental research as well as end-applications based on previous fundamental research. A particular research results thus can have a return over many generations. It is an interesting way to properly attribute the intellectual source of a new product or process, but the exercise is of little value if it is not possible to quantify the social value of the end-application. Indeed, Hofmeister goes back to using citations in Economics for a data application, which is equivalent to evaluate research only within the scientific community. In terms of the stated goal of the paper, we are back to square one. In terms of getting a better measure of citation impact, this is an interesting application of an old idea. And the resulting rankings of journals and articles look very much like those that are already available.
Showing posts with label citations. Show all posts
Showing posts with label citations. Show all posts
Friday, June 10, 2011
Monday, January 24, 2011
How not to distribute research funds
Citation counts are often used to proxy for the quality of an article, researcher or journal. They are not a perfect measure, everybody agrees on that, but they have proven to be a useful starting point for evaluation. Sometimes they are taken very seriously, too seriously, for the distribution of funds and pay. But at least this is done within a field, as it is obvious that citing conventions and in particular frequencies differ from field to field.
Javier Ruiz-Castillo goes further in trying to infer how budget priorities should allocated across research fields by using citations counts. Of course, for this one first needs to have a good understanding of how citations are distributed. Roughly, citations are distributed following power laws with fields and subfields. This means that few articles garner a lot of citations, while many go empty (especially in Business, Management and Political Science). And if I understand the paper right, one can apply readily a multiplier to compare the citation frequencies across fields. And these multipliers then make it possible to compare researchers or research units across fields within, say, a country, as long as one assumes that an adjusted citation is equally worth citing. For example, is political science worth the same support as biomedical engineering after using these multipliers, to take two randoms fields? And the "size" of the field is important as well. Here the author makes an attempt at some definitions of size which I frankly did not understand.
That said, I wonder why I forced myself in reading this paper. First is it indigestible because it is poorly written and uses very bad analogies. Second, because trying to compare fields and use citations for the allocation of funds or prizes across then is impossible because you have no identification: in statistical speak, the fixed effects capture all the variance. You can only compare how well a field does in a country compared to the rest of the world, but this cannot measure how important the field is. You need more information than just citations.
Javier Ruiz-Castillo goes further in trying to infer how budget priorities should allocated across research fields by using citations counts. Of course, for this one first needs to have a good understanding of how citations are distributed. Roughly, citations are distributed following power laws with fields and subfields. This means that few articles garner a lot of citations, while many go empty (especially in Business, Management and Political Science). And if I understand the paper right, one can apply readily a multiplier to compare the citation frequencies across fields. And these multipliers then make it possible to compare researchers or research units across fields within, say, a country, as long as one assumes that an adjusted citation is equally worth citing. For example, is political science worth the same support as biomedical engineering after using these multipliers, to take two randoms fields? And the "size" of the field is important as well. Here the author makes an attempt at some definitions of size which I frankly did not understand.
That said, I wonder why I forced myself in reading this paper. First is it indigestible because it is poorly written and uses very bad analogies. Second, because trying to compare fields and use citations for the allocation of funds or prizes across then is impossible because you have no identification: in statistical speak, the fixed effects capture all the variance. You can only compare how well a field does in a country compared to the rest of the world, but this cannot measure how important the field is. You need more information than just citations.
Thursday, January 13, 2011
Journal editors are poor selectors of best papers
Journal editors are thought to be exceptional scholars who are capable of identifying the best papers and in particular those that will have the largest impact, typically measured by citation counts. But editors have considerable help: first peer reviewers make reports that should be informative, second authors to some extend self-select in the submission process, they would not send a paper where it has no chance of getting published. But it is difficult to evaluate whether an editor is doing a good job. However, when editors choose to put as lead paper the one they consider the best in the issue, one can in retrospect check whether these papers are cited the most. As I reported earlier, editors turn out not to be particularly good.
What about those papers that get the "best article of the year" award? Tom Coupé does a similar exercise and finds that they are more cited than the median article, and slightly more than the runner-up articles, but they are rarely the most cited in a year. You just cannot trust editors, even though they are even supported by a committee for such awards.
What about those papers that get the "best article of the year" award? Tom Coupé does a similar exercise and finds that they are more cited than the median article, and slightly more than the runner-up articles, but they are rarely the most cited in a year. You just cannot trust editors, even though they are even supported by a committee for such awards.
Friday, August 13, 2010
What makes a great journal in Economics?
If you are editor of an Economics journal that seeks to improve and come across a paper entitled "What Makes a Great Journal Great in Economics?", you read it expecting some great recommendations, for example on how to attract great papers, generate good readership that will cite the articles and how to get established authors interested in submitting.
I supposed this is what Chia-Lin Chang, Michael McAleer, and Les Oxley were out to do, and what a disappointment it is. They take Thomson Reuters ISI Web of Science citation data, torture it in all sorts of innovative ways and come to the conclusion that for the top 40 journals in Economics, looking at the 2-year impact factor gives a distorted picture. First, how does this answer the question in the title? Second, it is already well known that these impact factors are flawed and only appreciated by publishers. Indeed, they encourage self-citations by journals, whose editors strategically require from authors to cite other articles. The two year window is ridiculously short for Economics. And the choice of journals that are indexed is difficult to follow.
I supposed this is what Chia-Lin Chang, Michael McAleer, and Les Oxley were out to do, and what a disappointment it is. They take Thomson Reuters ISI Web of Science citation data, torture it in all sorts of innovative ways and come to the conclusion that for the top 40 journals in Economics, looking at the 2-year impact factor gives a distorted picture. First, how does this answer the question in the title? Second, it is already well known that these impact factors are flawed and only appreciated by publishers. Indeed, they encourage self-citations by journals, whose editors strategically require from authors to cite other articles. The two year window is ridiculously short for Economics. And the choice of journals that are indexed is difficult to follow.
Monday, March 23, 2009
Cheating with publication metrics on SSRN
With the advent of online bibliographic databases like RePEc and SSRN, with the increasing importance of readership statistics in the evaluation of a researcher, and with said evaluation becoming more and more competitive, it is obvious that some researchers will try to find ways to manipulate these statistics. There is plenty of anecdotal evidence that such manipulations are taking place successfully at SSRN, with self downloads and in particular teachers telling their classes to download their works. Some even use social forums like Fark or Facebook to ask total strangers to increase their counts. And it is common place to send emails to friends and family encouraging downloads. But that is only anecdotal evidence. Benjamin Edelman and Ian Larkin now provide better evidence that this is taking place. Their paper is on SSRN, so go and increase their download count...
Their strategy is to identify when in the career of a researcher increased downloads would matter and test whether they indeed increase at that time. Where most of the cheating is going on is with SSRN's "top ten" lists. There are many of those, and thus it is relatively easy to get on them, with a little help, especially when one is already close to the 10th position. Edelman and Larkin also find that one or two years before a career move, downloads "mysteriously" tend to increase. However, they find no evidence for approaching tenure decisions. But I think this underestimates the problem, as this study identifies fraud from the historical SSRN logs and how SSRN subsequently identified suspicious downloads according to its new rules. Those rules are very imperfect and thus obviously do not detect successful manipulation which anecdotal evidence shows is happening.
Fortunately in Economics, people do not rely on SSRN statistics, maybe because of this perception that they can be fraudulent (but SSRN claims to have cleaned up their act). RePEc statistics are followed much more closely, none the least because it is perceived as the "good guy" (whereas SSRN is profit seeking), because it has been open about its statistics from the start, because many safeguards have been put in place to prevent fraud from being successful, as explained on the RePEc blog post, and because downloads are not the only relevant metric. And in particular, RePEc does not insist on every page about its download statistics. And the fact that SSRN statistics are updated in real time is just too tempting (refresh a SSRN page to see what I mean).
Their strategy is to identify when in the career of a researcher increased downloads would matter and test whether they indeed increase at that time. Where most of the cheating is going on is with SSRN's "top ten" lists. There are many of those, and thus it is relatively easy to get on them, with a little help, especially when one is already close to the 10th position. Edelman and Larkin also find that one or two years before a career move, downloads "mysteriously" tend to increase. However, they find no evidence for approaching tenure decisions. But I think this underestimates the problem, as this study identifies fraud from the historical SSRN logs and how SSRN subsequently identified suspicious downloads according to its new rules. Those rules are very imperfect and thus obviously do not detect successful manipulation which anecdotal evidence shows is happening.
Fortunately in Economics, people do not rely on SSRN statistics, maybe because of this perception that they can be fraudulent (but SSRN claims to have cleaned up their act). RePEc statistics are followed much more closely, none the least because it is perceived as the "good guy" (whereas SSRN is profit seeking), because it has been open about its statistics from the start, because many safeguards have been put in place to prevent fraud from being successful, as explained on the RePEc blog post, and because downloads are not the only relevant metric. And in particular, RePEc does not insist on every page about its download statistics. And the fact that SSRN statistics are updated in real time is just too tempting (refresh a SSRN page to see what I mean).
Tuesday, March 17, 2009
Catch-22 for publish-and-perish
When it comes to publication, economists set incredibly high standards. Compared to other fields, rejection rates are higher, editors and especially referees have a much higher impact, papers go through multiple revisions within the same submission (and may still be rejected), papers are expected to be self-contained pieces, and everything needs to be perfect, including grammar. All this leads to the well-lamented huge publication delays in Economics. Yet, publication in the top journals is still a requirement for tenure in any department that wishes to garner any respect, and it is just not possible for top journals to publish all the research fit for tenure.
Bruno Frey calls this the "Publication Impossibility Theorem System" (PITS): there just not enough slots in those journals, especially once one considers that 80% of the articles in the top five journals are written by faculty based in the US. Frey claims that this makes it impossible for non-Americans. Whether there is a US bias remains to be established, but even for Americans, the bottleneck is clearly present.
That said, this brings up more generally the question of publication expectations for tenure and promotion, and the journal culture in Economics. Over the last two decades, a large number of journals have been created, partly because commercial publishers saw opportunities, but mostly because the top, society driven journals failed to realize that the publishing needs of the profession have increased. It is only very recently that the American Economic association has realized that it missed the boat and created its four new American Economic Journals. The econometric Society as well announced the creation of two "field journals", Empirical Economics and the preexisting Theoretical Economics.
This should provide only little relief, as these add-on journals are unlikely to be considered top-journals. In fact they seem to turn into junk mail lying around everywhere in department mail rooms. What is really needed is to write shorter papers, straight to the point with less emphasis on the referees. More papers will then be published faster, and let citations then do the measurement of quality. The latter are a very imperfect measure nowadays, because the excessive lengths of papers allows to cite each and everyone. A shorter paper will have to be more selective and restrict itself on those that it really builds upon.
Bruno Frey calls this the "Publication Impossibility Theorem System" (PITS): there just not enough slots in those journals, especially once one considers that 80% of the articles in the top five journals are written by faculty based in the US. Frey claims that this makes it impossible for non-Americans. Whether there is a US bias remains to be established, but even for Americans, the bottleneck is clearly present.
That said, this brings up more generally the question of publication expectations for tenure and promotion, and the journal culture in Economics. Over the last two decades, a large number of journals have been created, partly because commercial publishers saw opportunities, but mostly because the top, society driven journals failed to realize that the publishing needs of the profession have increased. It is only very recently that the American Economic association has realized that it missed the boat and created its four new American Economic Journals. The econometric Society as well announced the creation of two "field journals", Empirical Economics and the preexisting Theoretical Economics.
This should provide only little relief, as these add-on journals are unlikely to be considered top-journals. In fact they seem to turn into junk mail lying around everywhere in department mail rooms. What is really needed is to write shorter papers, straight to the point with less emphasis on the referees. More papers will then be published faster, and let citations then do the measurement of quality. The latter are a very imperfect measure nowadays, because the excessive lengths of papers allows to cite each and everyone. A shorter paper will have to be more selective and restrict itself on those that it really builds upon.
Monday, March 9, 2009
Lead papers are not particularly better
Much is made about lead papers in journals. Some editors like to put what they think is the best article of a review in front. Is there any truth that this signals quality? Tom Coupé, Victor Ginsburgh and Abdul Noury use a natural experiment to test this idea: The order of articles in European Economic Review was alphabetical by author in some issues from 1975 to 1995. One can accept that this is a random ordering on the quality dimension.
The results are sobering. It turns out papers appearing first do have a citation advantage. This means that at least part of the citation advantage of lead papers is due to their position, not their quality. And knowing how the alphabet ranking of your name matters, it appears latter authors were doubly screwed in the European Economic Review.
The results are sobering. It turns out papers appearing first do have a citation advantage. This means that at least part of the citation advantage of lead papers is due to their position, not their quality. And knowing how the alphabet ranking of your name matters, it appears latter authors were doubly screwed in the European Economic Review.
Subscribe to:
Posts (Atom)
