Study Methodology

Study Methodology

Journal Selection

Many studies have ranked IS journals in terms of citations or other measures (Hardgrave & Walstrom, 1997; Mylonopoulos & Theoharakis, 2001; Whitman, Hendrickson, & Townsend, 1999). By using these studies as a guideline and limiting the current study to strictly research journals (e.g., Communications of the ACM would not qualify) and by further limiting the study to specifically IS journals (i.e., Decision Sciences and Management Science would not qualify), we selected three journals to use in our sample: MIS Quarterly, Information Systems Research, and Journal of Management Information Systems. These three final selections were the top-ranked IS research journals in each of the above-mentioned studies. Other researchers (e.g., Boudreau et al., 2001) have used similar guidelines when selecting journals from which to sample. We also selected a relatively large time period to determine if researcher habits had changed over time. Therefore, we examined all articles in the three selected journals for the years 1996–2000.

Classification of Articles

Each article was grouped according to article type, as shown in Table 1. To initially classify an article, one of the authors would read the article and classify accordingly. Then the other author would also read the article and classify accordingly. Any articles for which there existed a disagreement among researchers, later discussions were able to resolve it so that the researchers agreed on the classification of all articles. In studies using more than one methodology, the predominant methodology was used to classify each article—that is, a case study that used a survey as a small part of the analysis would be classified as a case study, not a survey.

Table 1: Article Classification.

Type of Study	Journal (1996–2000)
	�	Information Systems Research	MIS Quarterly	Journal of Management Information Systems	TOTALS
Survey/Questionnaire	31[1] 23.5% [2]	31 25.0%	54 31.40%	116 27.1%
Theory/System Development [3]	41 31.1%	29 23.4%	57 33.1%	127 29.7%
Case Study	11 8.3%	29 23.4%	19 11.1%	59 13.8%
Experiment	11 8.3%	5 4.0%	27 15.7%	43 10.1%
Mathematical Modeling [4]	16 12.1%	4 3.2%	9 5.2%	29 6.8%
Qualitative Analysis	3 2.3%	12 9.7%	2 1.2%	17 4.0%
Interviews	1 0.8%	5 4.0%	2 1.2%	8 1.9%
Event Study/Objective Data	4 3.0%	3 2.4%	0 0.00%	7 1.6%
Action Research	0 0.00%	1 0.8%	1 0.6%	2 0.5%
Delphi Method	0 0.00%	2 1.6%	0 0.00%	2 0.5%
Field Study	13 9.9%	2 1.6%	1 0.6%	16 3.7%
Electronic Brainstorming	1 0.8%	1 0.8%	0 0.00%	2 0.5%
TOTALS	132 100.0%	124 100.0%	172 100.0%	428 100.0%
[1]Number of articles by journal; [2]Percent of total articles by journal; [3]Includes research note/analysis/response; also includes secondary data analysis; [4]Includes simulation, algorithms, economic modeling, etc.

Of 428 total research articles published in the five-year period, 116 (27.1%) were classified as using a survey approach with self-reported data as the predominant method. Another 127 (29.7%) were classified as theory/system development, including research note/analysis/response, as well as secondary data analysis. No other category accounted for more than 13.8% (case study) of the articles. Clearly, with over one-fourth of the published research in the top journals categorized as using a survey methodology, this approach appears important to the IS community. Therefore, common method variance, a potential confounding factor in survey research, should be of interest to the IS community at large. The next section describes how IS researchers have handled common method variance over the time period specified.