Technology

Vol. 11, No. 1
ISSN: 1546-2676

Kappa Omicron Nu FORUM,
Vol. 11, No. 1. 
ISSN: 1546-2676. Editor: Dorothy I. Mitstifer. Official publication of Kappa Omicron Nu National Honor Society. Member, Association of College Honor Societies. Copyright © 1999. Kappa Omicron Nu FORUM is a refereed, semi-annual publication serving the profession of family and consumer sciences. The opinions expressed by the authors are their own and do not necessarily reflect the policies of the society. Further information: Kappa Omicron Nu, PO Box 798, Okemos, MI 48805-0798. Telephone: (727) 940-2658 ext. 2003

Interested in submitting an article to KON FORUM? Papers are now being accepted for review.

Conducting Research on the Internet: Potential, Concerns, and Reflections

Steven M. Harris and Charette A. Dersch

Dr. Harris is an Assistant Professor and Ms. Dersch is a doctoral student in Marriage and Family Therapy, Department of Human Development and Family Studies, College of Human Sciences, Texas Tech University.

---

The Internet has the potential to change the way we conduct scholarly research. Eliminating geographic and prejudicial barriers are among some of the Internet’s claims to fame. Despite its potential, it is difficult to give the Internet an overwhelming endorsement when so many concerns still loom large regarding Internet-based research, specifically data collection. It is possible that as the social sciences transition from a modernist to a postmodernist approach to research that the Internet opens up potential research modalities that have previously been unconsidered. This article reports on the authors’ experience of collecting data via the Internet. Specific concerns and potential regarding confidentiality, anonymity, data security, and methodology are addressed.

Few would deny that the Internet and the World Wide Web (WWW or Web) have the potential to revolutionize the way people conduct themselves in business, education, and personal relationships. The world has become smaller for those who use e-mail on a daily basis. In television commercials the Internet is heralded as a medium immune to the prejudices which so easily beset humans who cannot get past judging others on the basis of age, gender, or race. Distance Learning Technologies (DTLs) have changed the face of correspondence education. A student in Helsinki logs onto a personal computer and takes a course in Vancouver, Canada while being enrolled in the British Open University system. Clearly, geographic and prejudicial barriers are being broken in education as well as personal relationships due to the medium of the Internet.

We are quickly understanding just how prolific the Internet can be. One such area where the Internet can make a contribution is in research and data collection. This paper briefly outlines some of the ways in which research has been conducted via the Internet, the authors’ own experiences with data collection on the Internet, and the problems and potential of such research.

Research on the Internet

Most people use the Internet to do research. Probably, the reason given most often for browsing the Web is to find some piece of information (Graphics Visualization, & Usability Center of Georgia Tech, 1997); this is research. However, for purposes of the present article we discuss how the Internet has been used in connection with scholarly research. A browsing of the Web and a review of published material indicates three primary categories of research involving Internet technology: review research, marketing data collection, and participant recruitment projects.

Review Research

This category includes research projects that access existing information on the Internet. Much like a literature review, this research typically involves collecting information about particular topics found on the Internet. For example, one group of researchers reviewed the content of Internet sites regarding spirituality and transpersonal psychology (Lukoff, Lu, Turner, & Gackenbach, 1995). These authors offer suggestions for Internet based review research as well as the findings of their search. Additionally, their article highlights the ease of conducting a review via the Internet with a popular Web browser, Netscape. They report the results when using the word “transpersonal” on two different Web search engines. One search engine indicated 69 sites with the word “transpersonal” in it while the other reported 134 sites. In addition to reporting the findings of the content, these types of reviews usually include the Internet addresses or URLs (Uniform Resource Locators) of sites that house the reviewed material. There are currently only a few published Internet review studies in peer reviewed journals. The authors expect that these types of studies will grow in number and popularity as people express more interest in trying to find “just the right piece” of information from the Internet.

Marketing Data Collection

Internet technology is quickly emerging as a powerful tool for finding and targeting potential customers. Using Internet technology, companies can glean information from users that can aid in market research. Unknown to many Web users, is the fact that information about a user or computer terminal can be routinely collected by “Webmasters” (males) or “Webmistresses” (female) (owners or creators of specific Web sites). A “cookie” or a numeric and text based file housed within a user’s Web browser communicates information to a server or a particular site’s operating system. In other words, your personal computer provides the hosting Web site with information about you. With this information the organization or corporation can more accurately tailor its Web site to specific users. It is important to note that a cookie can only be generated after a user has provided a Web site’s sponsoring organization with specific information such as name, address, e-mail address, preferences for certain products or consumer behaviors, etc. Typically, anytime a user sends information to a Web site it is possible for information to be stored in the cookie file on that user’s hard drive.

Additionally, in this information age, it is wise to know what information you provide by simply accessing the Internet. Most Web sites routinely collect less specific, non-personal information about the people who visit their sites. For example the IP (Internet protocol) address of a computer terminal as well as time and date of login to a specific site are usually recorded. The IP address is a series of numbers that are probably not too helpful in and of themselves, however most computer technicians know how to record an IP address and count the number of times a specific IP address has logged onto their system. Software does exist that can track an IP address and find the location of the computer terminal used to log onto a site.

Other identifying information can be collected such as the name of the domain from which a site was accessed. Domain name (see Table 1) may be another piece of helpful information when targeting a specific population of computer users. Corporations that conduct research about particular consumer habits or consumer interest trends are relying more and more on Internet technologies to provide an edge in market research.

Table 1: Domain Naming Systems

Participant Recruitment Projects

This type of research uses the Internet as a medium to attract and recruit participants as well as to collect the data for the project. At the time this article was written there were few published papers on the experiences of researchers collecting data via the Internet. One article (Smith & Leigh, 1997) discusses many of the advantages of using the Internet for traditional social science research. For example, one advantage is that it is nearly as accessible as other forms of communication such as radio and television, yet is capable of accommodating several forms of communication in a single medium. For example, unlike radio or television, it can accommodate text, audio material, visual material, video, and live interaction. It also allows one to access geographically remote material and people. This might make it much more practical to conduct cross-cultural research and access material that would otherwise be unavailable. Another article gives a positive review of conducting research on the Internet and its potential for research (Welch & Krantz, 1996). In these types of research projects participants can be solicited via multiple Internet technologies including, e-mail/listserves, electronic bulletin boards, and registration of key words with Internet search engines. Additionally, traditional methods of participant recruitment, such as advertisements in newspapers, as well as trade, academic, or professional journals, can be helpful in enlisting subjects.

These sites typically house a questionnaire or another form of survey instrument that participants use to submit their responses. The participant fills out the questionnaire, usually consisting of a series of check boxes or pull-down menus, and clicks a “submit” button to send the responses to a mainframe or server computer. This type of research is not much different from any other research involving a questionnaire. Researchers must adhere to specific guidelines and protocols to ensure reliable and valid results. Despite the similarities between research conducted via traditional (e.g., pen and paper) means and that conducted on the Internet, there are some potentially drastic differences. Using the Internet to attract and reach participants may be an attractive proposition for many; however, the Internet is not without its problems. The following section highlights the authors’ experience of recruiting participants and collecting data via the Internet.

Authors’ Experience

The authors recently used the Internet for data collection and participant recruitment. The focus of our study was on how mental health professionals view client/therapist attraction. Although departmental support and academic interest for this project were quite high, a thorough analysis of the collected data and the costs involved will help us decide if the information we gleaned from this effort will be as rich as we had originally hoped. Every research project must answer questions about confidentiality, participant demographics, methodology, and appropriate uses of the collected data. However, conducting research on the Internet can complicate the usual concerns and even pose new ones that researchers must address.

Confidentiality and Anonymity

Research projects that solicit human participation sponsored by an institution of higher education must pass a human subjects, or institutional review board. These boards assemble and pour over potential researcher designs to ensure that participants’ confidentiality, among other things, will be respected and protected. Participants’ rights to confidentiality and the security of the collected information become very important when conducting research on the Internet. It is easy to make claims of confidentiality on the Internet but may be more difficult to deliver (Bier, Sherblom, & Gallo, 1996). Unlike traditional instruments where raw data can be locked in file cabinets, the Internet’s locking devices may be less secure. Passwords and encryption codes may be broken. Therefore, it would be wise for anyone interested in housing a questionnaire on the Web to implement a method for rapid and secure data transfer.

Perhaps the most prevalent method of data retrieval used is to have the raw data submitted as an e-mail posting. Here, the participant fills out the questionnaire and after submitting the answers, the document is sent to the researcher’s e-mail address. The researcher then transfers the data from a raw format into a database. One of the major limitations of this type of data transfer is that participants’ e-mail addresses are also submitted with the raw data. Therefore, attrition may occur if potential participants wish to remain anonymous and refuse to send their data simply because their e-mail address may divulge their identity.

Another method of data collection allows for the raw data to be downloaded from the Web site and archived in a database housed on a server. The information stays as a file on the server until the researcher is ready to use it. This information is easily converted into an acceptable format for data analysis (e.g., SPSS, Excel, etc.). Furthermore, with appropriate programming, it is possible for the researcher to have access to the data via the Web. This access helps the researcher monitor data collection without having to download the file from the server. One benefit of this method, as opposed to the e-mail method, is that the researcher can view all the data at once, in raw form, as the data accumulates, instead of sifting through multiple e-mail postings. However, it should be remembered that as long as the information is accessible through the Internet it could be accessible to anyone who knows the URL or passwords. To ensure confidentiality and security of collected data, a daily downloading protocol should be enforced and become part of the project. This way, non-research personnel have fewer opportunities to access confidential data. Another obvious way to enhance confidentiality is not to have access to the raw data via the Web.

In addition to confidentiality as a major issue, anonymity can be a double edged sword. The anonymity afforded by the Internet may be a wonderful asset for data collection. Participants may feel more free to answer truthfully without fear of consequences. Conversely, it could be argued that the Internet offers an unrealistic form of anonymity almost to the point where taking on a new persona has not only become commonplace for frequent Web users (Smith & Leigh, 1997) but is viewed as a right the user should be able to exercise at his or her discretion. Recent research indicates that 40% of Web users have provided false information at some time. Additionally, the same study indicated that 14.59% of users indicated falsifying information over 25% of the time (Graphics, Visualization, and Usability Center - Georgia Tech, 1997). This phenomenon of taking on a new identity or misrepresenting one’s self is prevalent enough on the Internet that it might skew any demographic information collected. It would be difficult, therefore, to describe with any confidence or accuracy the sample submitting the survey. In defense of the Internet, it should be remembered that lying about or misrepresenting one’s self on traditional instruments is not totally uncommon either.

Another issue related to anonymity and confidentiality is how the researcher targets a specific population to study. With the Internet, as opposed to traditional survey research, where the researcher targets a specific audience and mails the surveys directly to them, anyone could access a questionnaire or experiment. There are few if any reasonable ways of limiting the number of people who can log onto a given research site. One way would be to set up the collection site with a password that is distributed to perspective participants by the researcher. Each participant would have to supply the password in order to complete or send a completed survey. This prevents “surfers,” who accidentally access the research, from submitting bogus questionnaires.

Another option is to have questions in the questionnaire designed to check the type of people filling out the survey. For example, in our project we wanted to target mental health professionals. We included a series of questions whose answers needed to be compatible for the survey to be accepted for inclusion in the final data set. We asked for disciplinary identification and gave forced choice options (i.e., psychology, social work, psychiatry, etc.). In addition to the forced choice question we asked an open ended question regarding the participants’ primary organizational affiliation (initials only). Furthermore we asked another open-ended question about the participants’ theoretical orientation. Therefore, data from a participant who might have responded to the three questions by saying s/he was (1) a psychologist who was affiliated with the (2) APA and used (3) psychodynamic theory was accepted for inclusion in the final data set. Conversely, the data from a participant who responded that s/he was (1) a psychiatrist who was affiliated with the (2) NRA and used (3) quantum physics theory was excluded. Although the questions we used reflect a professional mental health bias, most populations of interest would have specific and unique characteristics that when coupled and assessed together, would adequately screen out participants who were not appropriate for the study.

Recruitment & Demographic Potential

When data are collected via the Web the number of people who can participate is limited to those who have access to computers with Internet capability. So, who is on the Web? 1997 statistics from United States based Internet and print publications reveal that anywhere from 30 million (Graphic, Visualization, & Usability Center - Georgia Tech, 1997) to 47 million people had access to the Internet (CyberAtlas, September 1997). Additionally, a Business Week/Harris poll of United States Web users indicated that 45% of those on the Web are over 40 years old (as cited in CyberAtlas, September 1997). The average age of the typical user is 35.2 years with only 31.3% of all users being female (Graphic, Visualization, & Usability Center - Georgia Tech, 1997). IntelliQuest (1999) estimates that 80 million people, 38% of the U.S. population over 16, are online. This research firm projects 100 million by 2000. The Yankee Group (1999) concluded that one quarter of U.S. households are online, and by 2000 that percentage will rise to one third. Similar statistics are reported by Nua Ltd. (1999).

It is important to realize that housing a project on the Web may be a barrier to participation for those who have few economic resources. The Business Week/Harris poll indicated in 1997 that 82% of Web users had incomes of at least $25,000 dollars a year. IntelliQuest (1999) reports that income levels and education levels have been coming down. But the concern still stands that whole, large populations may never have a chance to respond. Furthermore, a certain “type of person” is selected who may not necessarily represent the population to which the findings of the study could be generalized (Swoboda, Muhlberger, Weitkunat, & Schneeweiss, 1997). The specific type of person who will fills out a survey on the Internet is at least one who is not computer or technology phobic (Hewson, Laurent, & Vogel, 1996). One of the biggest potential advantages to Internet solicitation of participants is that the researcher may tap into a more broad national and even international audience of potential participants. Pen and paper instruments are typically only mailed to a very specific, limited audience.

For our project we posted the same call for participation to listserves that cater to mental health professionals. We found these listserve groups by conducting a simple search on the Internet (Keywords: Mental Health Listserve). These lists were operated by individuals who ranged from being very receptive to less enthusiastic about our research. We made sure to explain the nature of the project as well as the fact that it had already passed a Human Subjects Review Board. Some of the lists required us to be members of the list before we could post anything.

To attract an even larger population we advertised in professional trade journals and newsletters targeted toward mental health professionals. These classified-type advertisements are costly, and we discovered that the responses to the printed advertisements did not make a noticeable increase in the overall sample size (completed surveys). However, the print media advertisements may have contributed to an increased “hit” rate (number of people who actually log onto the site) at our Web site. In the end, we concluded that soliciting mental health professionals via trade journals created a greater expense than it was worth in terms of generating participants.

The “Thin” Client

In “computerese” anyone who logs onto a site is referred to as a client. Theoretically, there are all types of clients. The “thin” client, however, is a user who has the bare minimum of computer resources with which to interface the Internet. Obviously, every researcher must decide to whom the research is targeted. When collecting data via the Internet, deciding who can actually interface with your site is another important issue. For example, if a project requires participants to be exposed to any audio, video, or still life photographic stimuli, there is a chance that eligible and willing participants will be excluded simply because they do not have the latest technology on their Internet compatible computers. For this reason every researcher must decide well before the project is up and running how comfortable they are with defining the “thinness” of the clients or targeted participants. Our project serves as an example.

The portrayal of most ethical dilemmas in therapy involves the use of text based stimulus scenarios. However, we know that in many instances video offers a more realistic portrayal of what may actually happen in a clinical situation. Therefore, we saw the inclusion of a video stimulus as an important part to this project. We knew however, that many people would not be able to access the video clip simply because they were not technically outfitted to do so. We were aware that attrition would occur because of this but really had no idea as to the extent. In a sense, we determined that the thin client or the user/participant with the lowest technological capabilities would have to be able to view and hear the video clip in order to fully participate in the research. We defined the thin client by the medium we used to depict the stimulus. If we had used a text based scenario we would have defined the thin client much more broadly, or given access to a greater number of people.

Defining the thin client for our project as someone who needed the technology to download and view video had a tremendous impact on participant recruitment. Soon after going on-line and having the project running for three months we noticed a curious trend in our participation rates. To track participation we had three different counters installed into the Web site. The first counter tracked how many people logged onto our site. The second counted how many people actually went from the introduction page to the questionnaire. The third counter tracked how many people actually submitted the completed questionnaire. After three months of being on-line we counted 310 people who logged onto our data collection site. Of these, only 120 decided that they would either be interested in completing the survey, or had the technology requisite to participate. Of these, only 41 persons actually completed and submitted the questionnaire. Depending on how we define participation rates we had either a 13% or 34% response rate. Optimistically, we like to argue that because only 120 persons actually chose to go to the questionnaire page that our response rate should be 34%. However, we cannot ignore the possibility that several interested people went to the home page with the intent of completing the instrument but decided not to participate for a number of reasons, therefore, a participation rate of 13% might be more realistic.

Methodology

Once consideration has been given to questions of demographics and confidentiality, one must consider how the methodology of a particular study would be enhanced or compromised by conducting a study via the Internet. Methodology becomes a particular concern where high degrees of control are required. In traditional experimental or quasi-experimental designs where the participant is exposed to a stimulus, the researcher has total control over length of exposure, quality of exposure (i.e., photographic quality, touch, etc.) location of exposure (does it matter if one views the stimulus at work, home, library, etc.?), number of exposures, and time of exposure in reference to other parts of the experiment. On the Internet the researcher loses control of these variables.

In our project the participants were required to download and view a seven second video stimulus. It was our hope that the participants would only watch the clip once. However, we had no control over how many times the clip was viewed. Furthermore, we had no way of knowing that the quality of the video clip was the same from one participant’s computer to the next. If our study depended on exactness of stimulus delivery, it would be very difficult to ensure it via the Internet.

Discussion

Critics to Internet research could argue that we have simply taken an old research paradigm and applied it to a new medium without thinking about the methodological and practical implications. To this, we counter that the entire field of Social Sciences seems to be caught in a transition (Burman, 1996; Gergen & Thatchenkery, 1996; Levin, 1991). In the past, research paradigms called for tight control of an experiment as the only valid and reliable method of arriving at scientific knowledge (Campbell & Stanley, 1966). Recently, however, there seems to be a trend away from the use of quasi-experimental designs in social science research as more qualitative, ethnographic, and less rigidly structured projects are used to answer social questions (Burman, 1996; Jacobson, Mulick, & Schwartz, 1996; Reid, Robinson, & Bunsen, 1995). This may reflect the transition from a modernist approach to scientific inquiry to a more postmodern paradigm (Gergen & Thatchenkery, 1996).

The technology of the Internet may help social scientists bridge the gap between the modern and postmodern worlds. If the Internet represents a new communication modality, it may be instrumental in broadening our concept of research as well as the way research is conducted. So, although it may be difficult to accommodate the modernist paradigm of experimental research in many cases, the Internet may be able to accommodate a variety of postmodern research pursuits. We may be in the process of discovering this potential. For example, new programming languages are being written specifically for computerized online social science research (Pallier, Dupoux, & Jeannin, 1997). Additionally, the field of education is currently developing new teaching paradigms to accommodate teaching via the Internet (Songer, 1996). It is highly probable that the Internet is so new that we have not yet established the most efficient or effective way to conduct research with it, nor have we adequately assessed the potential that it offers researchers. If we are in the midst of a technology and paradigm shift, we encourage researchers to realize that just because we have a new medium with which to conduct research does not mean we must adhere to a new paradigm, nor does it mean we discard the old one. Somehow, for a while, the two must co-exist until we become more clear on how to use this medium appropriately for research purposes.

Conclusion

One might think that any type of survey research, regardless of topic, would be well suited for delivery via the Internet. In fact, a general rule might be that if you can collect self-report data via pen and paper instruments you could just as easily do so over the Internet, and it may even cost less. This assertion remains to be validated. In contrast, it probably can be said, with some confidence, that research projects involving a high degree of stimulus control are not suited for delivery via the Internet. Too much variability currently exists between users, computers, and access to the Internet to ensure a tightly controlled design to yield valid and reliable results.

Our experience with data collection via the Web has, at least, been an education. We know that by putting our instrument on the Web, we have accessed persons in other countries that we never would have been able to reach with a standard method of distribution. Conversely, we know that we have excluded a number of people simply by housing the survey on the Internet. This may not necessarily be a bad thing. It may simply be something we need to acknowledge when discussing the results of our study. Using the Internet to recruit research participants is therefore, a double edge sword. What can be gained from reaching a global population may come at the expense of excluding the large number of people who do not have access to the Internet. Self-report surveys may be best suited for adaptation to Internet data collection procedures. In the future, if we were to recruit participants via the Internet for a quasi-experimental project and were interested in accumulating a large sample in a short period of time, we would be wise to use a text based stimulus scenario. Incorporating a video segment, as we did, or even an audio file as a stimulus, places far too many restraints on potential participants. This may change as more and more people gain access to the Internet with higher powered and more sophisticated computers.

We encourage others to consider housing their surveys on the Internet. The concerns, potential, and experiences we mention in this article come from our desire to understand the usefulness of the Internet, our naiveté at conducting this type of research, and our willingness to challenge the status quo. More participation is needed to make the Internet a viable research tool. We recommend further cross-modal research projects that include data collected from both print and virtual media so that threats to and concerns of reliability and validity can be addressed and reconciled. Finally, we encourage others to publish scholarly articles, not simply on the findings of research conducted on the Internet, but equally important, on the process of conducting this type of research.

Acknowledgment

The authors acknowledge the support of the College of Human Sciences at Texas Tech University and the Faculty Development Grant (no. 0096-44-0535) that supported this project.

References