Distinguishing citation quality for journal impact assessment

Abstract

Introduction The research community has long and often been fervently keen on debating the topic of journal impact. Well, just what is the impact of a journal? Today, the Science Citation Index (SCI) recognizes over 7,000 journals. The sheer number of available journals renders it pivotal for researchers to accurately gauge a journal's impact when submitting their papers, as it has become commonplace that researchers regard publishing their work in established journals to have significant influence on peer recognition. For journals in Management Information System (MIS), such research studies have continuously been published since the 1990s. Nine of them have been summarized by Carol Saunders, whereby seven were based on respondent perceptions by surveying experts, and two were based on the citation quantity to indicate the journal impact. It is generally accepted that citation analysis is purported to be a more objective method than the expert survey. The main reason is citation analysis uses objective measurements, which are based on the viewpoint that the influence of a journal and its articles is determined by their usefulness to other journals and articles, and where their usage can be reflected by citations that they have received. However, using citation quantity only is also considered to have bias to a certain degree, due to a widely held notion that citation quantity does not represent citation quality. Regarding the impact in the MIS discipline for example, a citation by a paper published in a prestigious MIS journal should far outweigh a citation by a paper published in an unremarkable MIS journal or in an external journal outside the MIS field. Such intuition suggests that the citation quality can be divided into the following two aspects: • Citation Relevance (CR): indicating how relevant the journal giving the citation is to the discipline we are interested in; • Citation Importance (CI): indicating how important the journal giving the citation is in the discipline we are interested in. However, these concerns about citation quality have not been properly addressed in citation analysis literature. To address our concerns about citation quality for assessing journal impact, we propose a method, which first clusters "pure" MIS journals to identify relevant citations, and then score the impact for each journal, according to its citations that are received from pure MIS journals and weighted by citation importance. Although our method is only applied to MIS journals, it is general enough to evaluate the impact of journals in other disciplines.