base
100 Research
Research Design and Research Strategies
In Agar's statement above, we get the impression that a credible argument should be systematic and based on a process that informs us about how researchers came to know what they know.
It is the articulation of this "process by which we came to know it" that reflects the elements of research design.
For Stinchcombe (1987:23), the observations produced by how a study was designed are fundamental to the proper assessment of empirical evidence: "We always want to reject evidence if it can be explained by the design of the research or by a large number of small, unorganized causes.
The value of empirical evidence can only be properly evaluated by understanding the details of how the research was conducted.
According to Pelto and Pelto (1978:291): "Research design involves combining the essentials of investigation into an effective problem-solving sequence.
Thus the plan of research is a statement that concentrates on the components that must be present in order for the objectives of the study to be realized.
" This statement illustrates at least two important elements of research design.
First, research design involves an a priori plan or strategy for all phases of the research (such as data collection and analysis) including, for some researchers, the production of the final product (like an ethnography).
By definition, a plan cannot deal with the unanticipated or unknown realities of research, such as tragedies or acts of nature that disrupt fieldwork.
A good understanding of the research problem and the research site allows us to plan for some contingencies, but there is no research design crystal ball.
Still, while luck plays a role in research, planning for such luck is not within the realm of research design (Kirk and Miller 1986).
I use "defensible" in addition to "valid," which I normally use, to make readers aware that I am broadening the traditional application of research design to include the variety of research strategies found in anthropology today.
Nevertheless, a well-articulated project design helps "to promote the effective conduct of research," whether one starts from a positivist or humanist perspective (Ellen 1984:158).
On a practical level, good research design is essential in the competition for research grants and contracts.
There is much variation in what funding agencies and foundations expect regarding research design.
One agency may require a detailed description of the proposed project paying attention to the research design logic of science (for example, validity, reliability, hypotheses, etc.
others may require a description of the research problem and site but require less detail about the methods of data collection and analysis.
A distinction needs to be made between what's sometimes called the laundry-list component of research and research design.
Evidence for the power of research design is all around us.
The invention of the simple control/treatment design of clinical trials allowed researchers in this century to evaluate competing therapies and to select the ones that worked best.
The issue is still under debate, but this series of studies illustrates how the elements of research design help muster evidence in light of competing beliefs and philosophies.
The logic of the research design contributed to the production of credible results.
Some early exceptions include Brim and Spain's (1974) book on hypothesis-testing designs, Pelto and Pelto's (1978) book on research methodology in cultural anthropology, and Naroll and Cohen's (1973) A Handbook of Method in Cultural Anthropology, which has several chapters that address issues in research design (LeVine 1973;
If research design gets relatively little attention from anthropologists, other social scientists have written volumes about it.
What should we make of this apparent dearth of specific treatments of research design in cultural anthropology?
I don't think we should make too much of it because the important elements of research design-reliability, informant accuracy, validity, objectivity, and operationalization of theoretical concepts-have been present in the writings of cultural anthropologists even before Boas.
Boas, Malinowski, and Research Design in the Scientific Tradition
The constraints of field research may lead one to stray from the idealized prescriptions of a research design, but Mead was attempting to exert her authority without necessarily following the research procedures advocated by Boas and others.
First, it demonstrates the differences between the stated scientific objectives of ethnographic work as advocated by Boas and the actual practice of ethnographic research.
Rightly or wrongly, the preeminence of contextualization has been a consistent issue in ethnographic research and has often clouded issues in research design.
The idea that quantification detracts from context and meaning in the ethnographic endeavor-evident even in the time of Boas-and a failure to understand that systematic methods-whether quantitative or qualitative-help minimize the subjectivity of the investigator have impeded the development of well-delineated research strategies in anthropology.
Boas's final sentence in his response to Mead illustrates that even at this early stage the issue of the subjectivity of ethnographic research was of concern.
Mead's position on these various elements of research design provided fuel for the continuing discussions about the validity of her original findings (Brim and Spain 1974;
Thus, while early British and U.S. anthropologists advocated the scientific method in ethnographic research, there is little evidence that they considered appropriate design issues when they actually did the research.
This belief supports the current lack of formal training in methods and research design in anthropology.
Agar (1980) and Bernard (1994) relate stories about Kroeber's recommendations regarding the teaching and conduct of ethnographic research.
Even in the late 1960s, when concern for methodological rigor was probably at its peak in anthropology, many treatments of research methods and design in the literature played down the need for more systematic methods and design detail, particularly with respect to hypothesis-testing approaches (LeVine 1973).
And since I have indicated here that research in culture involves a great deal of unique personal experience for the anthropologist, I have taken the position that it is probably unlikely there can be a rigorous, systematic, and formal presentation of methods in the study of culture like those of the natural sciences and that there are overriding concerns among many sociologists, psychologists, and economists.
LeVine (1973) and others (Johnson 1990) make the point that the nature of fieldwork, in terms of its requisite huge investments in time and geographical focus, has often limited the attractiveness of more formal research designs because of its commitment to studying specific problems in a specific way.
The realities of fieldwork often dictate the need to change the problem focus or, finding that the proposed hypotheses are inappropriate to the cultural setting under study, the need to somehow salvage the research.
Laboratory and survey researchers have some flexibility to change the problem focus and study populations in light of emerging problems, but field workers are limited in their ability to do so.
Thus, the idea of researchers "putting all their eggs in one basket" may have limited the a priori formulation of problems in fieldwork (LeVine 1973:184).
Contemporary Design Issues in Cultural Anthropology There is an ongoing debate in cultural anthropology concerning science and its role in contemporary research.
Suffice to say that traditionally, research design and its logic have been associated with science and an underlying belief in objectivity and explanation.
To say that the research design logic of science has been replaced by something that is recognizable as the research design logic of, say, postmodernism would, I think, be misleading.
It is not that interpretive approaches lack some form of research plan;
A more appropriate term that would encompass the diversity currently found in cultural anthropology might be "research strategy.
Figure I is a taxonomic characterization of the different types of research strategies found in contemporary cultural anthropology.
Many examples of research in anthropology fall within the two extremes of the continuum.
The light line connecting the two categories indicates their complementarity and interrelatedness in that a design may include both within an overall research design framework.
These approaches are by no means mutually exclusive in approaching a research problem (see section on Research Design in Systematic Research, below).
Its practitioners are ultimately interested in research findings that approximate an external truth.
As a result, the assessment of any theory involves research designs more heavily concerned with the means-the research process, rather than simply the way the study was written or argued-since the validity of study results depends on the scientific soundness of the research design.
For any given research problem, it is the purpose of research design to ward off as many threats to validity as possible.
In this line of thinking, the researcher is a field-worker-as-writer.
Types of anthropological research strategies and their features.
This is similar to the grounded theory ideas of Glaser and Strauss (1967), where exploratory descriptive research leads to the development of more meaningful theory and measures.
Exploratory research can be the primary focus of a given design or just one of many components.
Explanatory: Explanatory approaches generally involve testing elements of theory that may already have been proposed in the literature or that have been informed by exploratory research.
Research designs in this mode are determined a priori and their primary purpose is to eliminate threats to validity, where validity is concerned with whether things are what they appear to be or are the best approximation to the truth (Cook and Campbell 1979).
Interpretive strategies, on the other hand, differ from systematic approaches in that they question a researcher's ability to maintain objectivity, particularly in the ethnographic context where the ethnographer is often the instrument of measurement.
Thus, the rather simple characterization of research strategies found in Figure 1 attempts to recognize the variation inherent in the range of work found in contemporary anthropology by placing "interpretive anthropology" adjacent to "exploratory/descriptive" (see, for example, the work of Zabusky 1995).
Further, the purpose of research strategies under these interpretive paradigms is more focused on the production of a believable or plausible account or story rather than a single depiction of the truth, since it is thought that there are a multitude of plausible accounts rather than just a single true story.
Thus, there is less focus on the means of research, such as methods of data collection and analysis as found in the systematic strategies, and more on the ends of research-the ethnographic or literary product.
If we talk of an interpretive method, particularly with regard to postmodernism, it more than likely involves both the researcher's immersion into the cultural context of the actor(s) and some means, usually literary, for conveying the understanding gained from such an immersion.
As stated, many interpretive studies are closer in character to exploratory and descriptive research in the systematic mode than to some of the more extreme postmodern studies.
She contrasts her study with research on cooperation by "experimental" psychologists, emphasizing the cultural and social orientation of her work and the importance of considering context (social, cultural, political, etc.
In a short methodology section, she discusses the challenge of conducting participant observation research in this rather complex, geographically dispersed, cross-cultural setting.
Thus, there is little discussion of research design and methods of data collection as might be found in work in the systematic tradition.
In the following pages, I focus primarily on research designs in systematic research.
For further discussion of research strategies in the interpretive mode, see Fernandez and Herzfeld (this volume).
Research Design in Systematic Research:
In some social science disciplines, like psychology, the design of research is driven by features of the analysis.
In sociology, multiple regression models, structural equation models, and path analytic models (all related analytical techniques) have influenced the design of survey research.
Whereas the analytical techniques most often used in psychology, sociology, and economics often led to rather standard designs, in anthropology the eclectic nature of ethnography leaves the design of research more open ended.
The good news is that ethnographic research is amenable to a wide range of research designs, including the use of multiple designs within a single ethnographic context.
The bad news is that the open-ended character of ethnography contributes to a less well-focused discussion of research design issues in ethnographic approaches.
Thus, ethnography should involve multiple methods, both qualitative and quantitative, and may involve applying more than one research design.
Currently, the qualitative analysis of text and discourse is no longer: restricted to either interpretive or exploratory approaches, but can also be used in hypothesis testing and explanatory research.
As with research in community ecology, ethnographic research can be purely exploratory or descriptive involving a research process focused on producing better theory-or purely explanatory, although this is usually not the case.
Rather, the most common model has exploratory research informing and complementing explanatory research.
As we will see in the examples to come, exploratory research is often an essential component of the explanatory research process.
Exploratory research may contribute to the production of reliable and valid measures, provide information essential for constructing comparison groups, facilitate construction of structured questions or questionnaires, or provide information necessary for producing a sound probability or nonprobability sample.
The figure shows that the overall research process is more than just a matter of study design.
Design, however, is the foundation of good research.
Figure 2 shows that the research process involves a simultaneous concern for the development of empirical statements from theory (for example, hypotheses), the operationalization of theoretical concepts (for example, meaningful and reliable measures), design (for example, groups to be studied), data collection (for example, qualitative versus quantitative), and data analysis (for example, multiple regression and text analysis).
Thus, research design is more than just methods of data collection and analysis.
It involves constructing a logical plan that links all the elements of research together so as to produce the most valid assessment possible of some theory, given some set of realistic constraints (for example, cost, scope, geographical setting, etc.
The purpose of research design is to ward off as many threats to validity as possible and to help one eliminate competing hypotheses.
Without such attention to good design and methodological detail, researchers leave themselves open to one of the worst criticisms of all-of being "not even wrong" (Orans 1996).
(1982) in constructing a typology of research designs.
" Anthropologists involved in development and evaluation research are most likely to use this design.
Howevert, with careful attention to design and ethnographic context, quasi-experimental and natural experimental designs can be applied to anthropological field settings, particularly in evaluation research and development research.
Most research designs in the explanatory mode, like true experimental designs, are comparative (for example, control versus treatment).
(1997) used a cross-sequential design, which involved cross-sectional research on a cohort of children carried out over time.
Discussing "common sense knowing" in evaluation research, Campbell (1988) gives an important critique of ethnography.
Potential errors and bias creep in at various steps in the research process.
In research design, forewarned is forearmed.
Internal validity is concerned with the approximation to the truth within the research setting.
External validity is concerned with the approximation to the truth as expanded to other settings-that is, with the generalizability of research findings.
Each of these threats may hamper a researcher's ability to assess the contribution of a hypothesized effect to any changes observed.
Similarly, threats to external validity, such as problems stemming from biased samples or research in atypical or unique settings, can hamper the generalizability of one's findings.
Many probability and nonprobability sampling designs are available for any given research problem.
The selection of any of these designs or the development of some hybrid design depends on the overall design of the research itself.
Thus, testing for a treatment effect across a wide range of classes in the set of all possible classes (including both extremes and the modal class) in the population allows the researcher to say something about how the effect holds in a variety of settings.
While this might not be generalized to the population as a whole, it does inform the researcher if an effect holds across wide ranging classes within the population.
In some cases, a researcher may not be interested in generalizing to a population but may just want to know whether two subgroups obtained from a snowball sample differ with respect to some variable of interest.
How samples are chosen is an important element of any research design.
These new approaches seem particularly well suited for the imperfect world of ethnographic research, where the rather restrictive assumptions of parametric analysis are often difficult to meet.
Mead Versus Freeman: Research Design as Mediator
But, in common with Brim and Spain, Orans found itemized problems with Mead's research design.
For Brim and Spain, and for Orans, Mead's research design limited her ability to draw the conclusions she did.
More attention to issues of research design and methods would have improved her chances to make valid claims and possibly limited later criticism of her work.
In ethnographic research, no matter the mix of methods, the design of the study should allow for an ethnographers' hypotheses or hunches to be rejected as well as confirmed (Campbell 1975).
Research Design in Anthropological Practice:
Systematic Research Strategies
This example also shows how readily multiple tests can be incorporated into a research design within a field setting.
In all research, but particularly in field experiments like the one described above, there should be a concern for ethics and the well-being of experimental participants.
Young and Garro's (1982) investigation of treatment choice in two Mexican communities is an example of a static-group comparison where the presence or absence of the treatment is based on selection criteria not directly under the control of the researchers.
One of the primary purposes of the research design was the elimination of competing hypotheses-the hallmark of good research design-and the testing of the primary hypothesis is an example of descriptive inference, as opposed to statistical inference.
Descriptive inference is an approach highly suited for much anthropological research.
An important issue in this area of research concerns the factors influencing the use of Western treatments among non-Western populations.
The research design included the comparison of two Mexican communities that were similar in terms of cultural traditions and economies but varied in terms of access to Western medical services.
Later, the researchers collected triads data and what they call term-frame data on informants' perceived similarity of illnesses.
The authors' careful attention to research design and analytical issues contributed to the production of impressive evidence that casts doubt on the validity of the "conceptual incompatibility" hypothesis.
This distinction is important, particularly with regard to anthropological research, in that hypothesis-testing research can be done without narrowly restricting it to analytical methods using statistical inference.
In lieu of equalization through randomization, Young and Garro, through extensive ethnographic background research, produced groups that, although nonequivalent in the quasi-experimental sense, shared similarities with regard to a number of important characteristics.
This is analogous to treatment and control groups without the random assignment of subjects to experimental units and where the treatment is implied rather than researcher directed (that is, natural differences in experience with fish).
But the in-depth ethnographic background research, the particular structure of the hypothesis, and the overwhelming reliability of informant responses make for more confidence in the possible validity of the study's conclusions.
Exploratory Research and the Development of Cultural Models
Often the primary objective of research design is more a matter of discovery and exploration than the testing of hypotheses.
An example of research in this mode is Naomi Quinn's (1996) development of Americans' cultural models concerning marriage.
There is a body of literature that views the interaction of culture with the individual as so deeply unique and personal as to not be researchable in terms of cultural universals, coherence, or even sharing.
Although the model appeared to be widely shared among informants from Quinn's sample and data collected from other studies on marriage, research still has to be designed to test this model across settings and researchers.
However, further research in the explanatory mode is now warranted.
As seen in Figure 2, exploratory and descriptive research are often essential components of an overall explanatory research design.
In most earlier research on injection drug users (IDUs) and HIV risk, the primary risk factor was viewed in terms of direct needle sharing.
This is a good example of the application of exploratory research in the production of better measures of potentially important explanatory variables.
The researchers were careful to make the control group as comparable to the asustados groups as possible.
Based on earlier ethnographic research, social stress, an important component for understanding an individual's inability to perform social roles, was operationalized using the Social Stress Gauge developed by one of the researchers.
This study is important because of the authors' candor about the potential threats to validity they encountered in conducting the research.
But the researchers' awareness of the problems, combined with the strength of their multiple case-control study design, increases our confidence in their conclusions.
For some researchers, the shift from Catholicism to Protestantism helps account for economic change, as Protestantism is more compatible with capitalist ideology and the accumulation of wealth.
Her research design illustrates the danger in relying on a single method without attention to sampling.
This review of research design and strategies in cultural anthropology only scratches the surface of the research designs, hybrid designs, and combinations of designs possible within an ethnographic context.
As anthropologists, we should take full advantage of both our current understanding of research design and these new developments to produce a "powerful mode of argumentation.
Survey Research Methods, 2d ed.
Research Methods in Anthropology: Qualitative and Quantitative Approaches, 2d ed.
Research Design in Anthropology: Paradigms and Pragmatics in the Testing of Hypotheses.
Qualitative Knowing in Action Research.
Entering the Field of Qualitative Research.
In Handbook of Qualitative Research.
In Ethnographic Research: A Guide to General Conduct.
The Discovery of Grounded Theory: Strategies for Qualitative Research.
Qualitative Research Methods Series, Vol.
Rethinking Critical Theory and Qualitative Research.
In Handbook of Qualitative Research.
Reliability and Validity in Quantitative Research.
Qualitative Research Methods Series, Vol.
Epidemiologic Research: Principles and Quantitative Methods.
Research Design in Anthropological Field Work.
Anthropological Research: The Structure of Inquiry, 2d ed.
Real World Research: A Resource for Social Scientists and PractitionerResearchers.
In Handbook of Qualitative Research.
An Example of Research Design: Experimental Design in the Study of Culture Change.
In Ethnographic Research: A Guide to General Conduct.
Qualitative Research Methods Series, Vol.
100 study
For Stinchcombe (1987:23), the observations produced by how a study was designed are fundamental to the proper assessment of empirical evidence: "We always want to reject evidence if it can be explained by the design of the research or by a large number of small, unorganized causes.
Thus the plan of research is a statement that concentrates on the components that must be present in order for the objectives of the study to be realized.
Design, on the other hand, involves the methodological and analytical details that contribute to the credibility, validity, believability, or plausibility of any study.
Fishers petitioned the federal government to fund a study of pinger effectiveness.
The study used the classic control/treatment design in which catch rates for a set of nets with pingers were compared to catch rates for set of nets without pingers.
Some conservationist groups claimed the study was biased in that the treatment nets were placed in areas known not to have large numbers of porpoises.
So another study was conducted placing experimental treatment and control nets in the same proximity.
Lobbying efforts by fishers yielded more funds for a larger, more comprehensive study involving more than 10,000 fishing nets.
The issue is still under debate, but this series of studies illustrates how the elements of research design help muster evidence in light of competing beliefs and philosophies.
In each successive study, investigators tried to control for as many extraneous variables as possible so that the hypothesized effect could be assessed (that is, the effectiveness of pingers compared to not using pingers).
Boas and most of his students advocated a natural science logic in the collection of ethnographic materials and a true concern for the collection of reliable data that could lead to the production of valid theory.
A good example of this tension between the stated early concerns for the methods of science and the actual use of such methods in ethnography comes from correspondence between Boas and his student Margaret Mead during her first fieldwork in Samoa.
She wrote of her doubts about the comparability of cases and about her ability, or even the need, to do a quantitative comparison of the similarity of attitudes among the adolescent girls in her study.
I am very decidedly of the opinion that a statistical treatment of such intricate behavior as the one that you are studying, will not have very much meaning and that the characterization of a selected number of cases must necessarily be the material with which you operate.
In Britain the claims that anthropology not only studied a distinctive body of data but also that it possessed a sophisticated methodology to collect these data, was an important factor in the establishment of anthropology as a discipline.
It is a strange paradox in the development of field methods that the scientific study of other cultures has been built upon such a foundation.
In the stories, one concerning Wagley's teaching of a field methods course and one concerning a graduate student at Berkeley asking for advice before going to the field, Kroeber's response was a terse, one liner that reflected the attitude of the times.
I believe that only someone wholly involved and fully immersed in fieldwork can really communicate the essence of cultural anthropology to students or general readers.
And since I have indicated here that research in culture involves a great deal of unique personal experience for the anthropologist, I have taken the position that it is probably unlikely there can be a rigorous, systematic, and formal presentation of methods in the study of culture like those of the natural sciences and that there are overriding concerns among many sociologists, psychologists, and economists.
I find this stance comfortable, for it is my conviction that so long as prime theoretical concerns in the study of culture are an attempt to record and understand the native's view of his culture and the objective and historical realities of culture, then methods for field study will have to reflect the end purpose of making a whole account of a part of the human experience.
LeVine (1973) and others (Johnson 1990) make the point that the nature of fieldwork, in terms of its requisite huge investments in time and geographical focus, has often limited the attractiveness of more formal research designs because of its commitment to studying specific problems in a specific way.
The realities of fieldwork often dictate the need to change the problem focus or, finding that the proposed hypotheses are inappropriate to the cultural setting under study, the need to somehow salvage the research.
Laboratory and survey researchers have some flexibility to change the problem focus and study populations in light of emerging problems, but field workers are limited in their ability to do so.
The figure distinguishes between strategies within the realm of interpretive studies and those using systematic strategies that have more of the elements of science.
As a result, the assessment of any theory involves research designs more heavily concerned with the means-the research process, rather than simply the way the study was written or argued-since the validity of study results depends on the scientific soundness of the research design.
There is a fundamental belief that the intersubjective, everyday meanings and how they are produced, maintained, and changed in any given context often defy objective study and explanation.
As stated, many interpretive studies are closer in character to exploratory and descriptive research in the systematic mode than to some of the more extreme postmodern studies.
A good example of this is Zabusky's (1995) ethnographic study of cooperation in European space science that she admits "took the form of mutual exploration rather than unidirectional examination" (p. 46).
She contrasts her study with research on cooperation by "experimental" psychologists, emphasizing the cultural and social orientation of her work and the importance of considering context (social, cultural, political, etc.
She also discusses the rationales for selecting the site and the group she studied, problems of working in a linguistically and technically diverse social milieu, the use of semistructured and unstructured interviews, and the effect of her role as ethnographer on informant relations and data quality.
Community ecologists, for example, similarly distinguish between exploratory or descriptive studies that seek to describe and determine patterns in ecological data and those studies that specifically seek to predict or test hypotheses.
The figure shows that the overall research process is more than just a matter of study design.
No amount of sophisticated statistics, computer intensive text analysis, or elegant writing can salvage a poorly designed study.
redoing a year-long ethnographic field study because of such errors is quite another.
Figure 2 shows that the research process involves a simultaneous concern for the development of empirical statements from theory (for example, hypotheses), the operationalization of theoretical concepts (for example, meaningful and reliable measures), design (for example, groups to be studied), data collection (for example, qualitative versus quantitative), and data analysis (for example, multiple regression and text analysis).
Theoretical knowledge is derived either from earlier studies or from exploratory work.
A recognition of limitations doesn't invalidate a study's results.
In other words, a lack of design and methodological detail makes it next to impossible to fairly and adequately assess the validity of any study's conclusions such that "rightness" or "wrongness" may not even be debatable.
Included are experiments, quasi-experiments, observational study designs, and what I refer to as natural experiments.
Random allocation produces equivalent comparison groups, and artificial manipulation of independent variables (also known as explanatory variables or study factors), with all other variables or factors controlled for, allows for the most valid assessment of the causal relationship between the independent and dependent variables or response variables.
Observational studies involve neither random assignment of members to comparison groups nor the manipulation by the observer of independent variables.
This distinction between experimental and observational approaches is similar to one in ecological field studies.
Mensurative designs, then, are observational and characteristic of the types of comparative designs found in field studies in anthropology.
Cook and Campbell (1979) make a similar distinction but refer to these kinds of natural experiments as "passive-observational studies.
(1982) refer to as observational studies.
Table 1 describes examples from observational and quasi-experimental study designs discussed by Kleinbaum et al.
For example, in their study of preschool children, Johnson et al.
The anthropologists have never studied a school system before.
They have been hired after (or just as) the experimental program has got under way, and are inevitably studying a mixture of the old and the new under conditions in which it is easy to make the mistake of attributing to the program results which would have been there anyway.
It would help in this if the anthropologists were to spend half of their time studying another school that was similar, except for the new experimental program.
It would also help if the anthropologists were to study the school for a year or two prior to the program evaluation.
(This would be hard to schedule, but we might regard the current school ethnographies as pre-studies for new innovations still to come.
All knowing is comparative, however phenomenally absolute it appears, and an anthropologist is usually in a very poor position for valid comparison, as their own student experience and their secondhand knowledge of schools involve such different perspectives as to be of little comparative use.
In one way or another, various study designs, in combination with other considerations such as the operationalization of theoretical constructs and sampling, are better or worse at dealing with each.
Here, I stress the importance of thinking through how validity threats have influenced and will influence observations or data (for a more in-depth discussion of how these types of validity can impact study conclusions, see Cook and Campbell 1979).
For example, the pretest/posttest nonequivalent groups design controls for some internal threats to validity, but it's problematic with respect to controlling for changes due to how groups members were selected (selection maturation), changes due to how individuals were tested (instrumentation), changes due to the selection of individuals with extreme pretest measures leading to regression toward the mean (regression), and changes due to local events not a part of the study (history).
Attention and concern with all the potential sources of error, whether stemming from how the study was designed, how the data were collected (for example, face-to-face interviews or mail-out surveys), or how the data were analyzed (for example, statistical conclusion validity), will help lead to the production of solid evidence.
The logic behind this model can be extended beyond the quasi-experimental case to observational studies.
The criticisms and counter-criticisms are difficult to assess, given the time between Mead's and Freeman's studies, the differences in locations of their work, and the differences in their ideological positions (Ember 1985).
In addition, she had a tendency to understate the population and overstate the proportion of girls in her study.
In ethnographic research, no matter the mix of methods, the design of the study should allow for an ethnographers' hypotheses or hunches to be rejected as well as confirmed (Campbell 1975).
Several examples are reviews of studies that incorporate comparative designs of various types in which nonequivalent groups are constructed in order to control for as many extraneous factors as possible, and the manipulation of independent variables is a function of how comparative groups were chosen.
Unlike studies where informed consent is obtained prior to participation, in experiments like Melbin's, individuals often participate without knowing about it.
Termed the "conceptual-incompatibility" hypothesis, a number of studies have suggested that such a congruence was the primary determinant of treatment choice among Third World peoples.
On the basis of the data from the triads study and the term-frame interviews, we see little reason to reject the "null hypothesis" of no significant differences between the responses of the two groups of informants.
Despite the authors' claims of finding no "significant difference," there was no real way, at least when the study was conducted, to assess the extent to which any differences were significant in the sense of statistical probability.
There are, of course, threats to validity in this study.
Boster and Johnson (1989) explored this issue in an ethnobiological study of fish.
These students were the control group.
Some of the criticisms of the Young and Garro study apply to this example as well.
But the in-depth ethnographic background research, the particular structure of the hypothesis, and the overwhelming reliability of informant responses make for more confidence in the possible validity of the study's conclusions.
Although the model appeared to be widely shared among informants from Quinn's sample and data collected from other studies on marriage, research still has to be designed to test this model across settings and researchers.
Issues of validity in this case are not as overriding as they would be in a purely explanatory study.
Thus, most large epidemiological studies of IDUs focused mainly on direct sharing behaviors in attempts to understand seroconversion rates and other risk factors.
In a subsequent study, Koester et al.
A major component of the study was the comparison of IDUs who engaged in both direct sharing and indirect sharing with IDUs who engaged in indirect sharing only and those who several subpopulations of injection drug users (see Figure 6).
A major component of the study was the comparison of IDUs who engaged in both direct sharing and indirect sharing with IDUs who engaged in indirect sharing only and those who neither shared directly nor indirectly.
Case-Control Study Design:
Here we look at an example of a study design used to investigate the extent to which disease is molded by culture.
(1985) report on a study of a folk illness known as susto, found in many cultural groups throughout North and South America.
The ultimate aim of the study was to show the relationship between various social forces and susto susceptibility.
The proposed design could have been conducted in a single community, but the authors felt that the generalizability of the results would be enhanced with a multiple case-control study design.
This study is important because of the authors' candor about the potential threats to validity they encountered in conducting the research.
But the researchers' awareness of the problems, combined with the strength of their multiple case-control study design, increases our confidence in their conclusions.
This is an excellent example of a study design that incorporates within-study replication or multiple tests of a theory.
Her study design incorporates quantitative and qualitative methods in the overall ethnographic enterprise.
The results of my study, of course, must be interpreted within the constraints of the data collection methods.
This study is an example of multimethod ethnography in which there was a combination of exploratory and explanatory approaches-that is, qualitative data and tests of models with data collected using a cross-sectional design.
The study shows how the use of multiple methods fosters triangulation that contributes to the production of valid conclusions (see Figure 8).
"Degrees of Freedom" in the Case Study.
Comparative Political Studies 8(2):178-213.
Anthropological Contributions to the Study of Social Networks: A Review.
The Process of Drug Injection: Applying Ethnography to the Study of HIV Risk Among IDU's.
An Example of Research Design: Experimental Design in the Study of Culture Change.
Shamanism, Colonialism, and the Wild Man: A Study in Terra and Healing.
Field Methods in the Study of Culture.
In the series Studies in Anthropological Method, George Spindler and Louise Spindler, eds.
99 anthropology
I use "defensible" in addition to "valid," which I normally use, to make readers aware that I am broadening the traditional application of research design to include the variety of research strategies found in anthropology today.
Although the power of experimental design is evident, concern for its application in anthropology-particularly cultural anthropology-has been limited.
Some early exceptions include Brim and Spain's (1974) book on hypothesis-testing designs, Pelto and Pelto's (1978) book on research methodology in cultural anthropology, and Naroll and Cohen's (1973) A Handbook of Method in Cultural Anthropology, which has several chapters that address issues in research design (LeVine 1973;
Bernard (1994) has elaborated in more detail on issues of design, but his treatment is necessarily limited, given his task of describing the range of methods available to anthropologists.
If research design gets relatively little attention from anthropologists, other social scientists have written volumes about it.
What should we make of this apparent dearth of specific treatments of research design in cultural anthropology?
I don't think we should make too much of it because the important elements of research design-reliability, informant accuracy, validity, objectivity, and operationalization of theoretical concepts-have been present in the writings of cultural anthropologists even before Boas.
The idea that quantification detracts from context and meaning in the ethnographic endeavor-evident even in the time of Boas-and a failure to understand that systematic methods-whether quantitative or qualitative-help minimize the subjectivity of the investigator have impeded the development of well-delineated research strategies in anthropology.
Thus, while early British and U.S. anthropologists advocated the scientific method in ethnographic research, there is little evidence that they considered appropriate design issues when they actually did the research.
In Britain the claims that anthropology not only studied a distinctive body of data but also that it possessed a sophisticated methodology to collect these data, was an important factor in the establishment of anthropology as a discipline.
This was less necessary in America where, by the late nineteenth century, anthropology was already established in universities, museums and government agencies.
But in spite of claims to scientific methodology, particularly in the British tradition, there are surprisingly few details about actual methods anthropologists used in the field, beyond a few first principles and illustrative anecdotes.
There was a wide belief among British anthropologists that fieldwork could not be taught to new recruits, but could only be experienced by individuals in the field.
This belief supports the current lack of formal training in methods and research design in anthropology.
Even in the late 1960s, when concern for methodological rigor was probably at its peak in anthropology, many treatments of research methods and design in the literature played down the need for more systematic methods and design detail, particularly with respect to hypothesis-testing approaches (LeVine 1973).
I believe that only someone wholly involved and fully immersed in fieldwork can really communicate the essence of cultural anthropology to students or general readers.
And since I have indicated here that research in culture involves a great deal of unique personal experience for the anthropologist, I have taken the position that it is probably unlikely there can be a rigorous, systematic, and formal presentation of methods in the study of culture like those of the natural sciences and that there are overriding concerns among many sociologists, psychologists, and economists.
Contemporary Design Issues in Cultural Anthropology There is an ongoing debate in cultural anthropology concerning science and its role in contemporary research.
The historical tension between interpretive and scientific approaches in anthropology has given way to an outright rejection by some anthropologists of science and its logic of design.
A more appropriate term that would encompass the diversity currently found in cultural anthropology might be "research strategy.
Figure I is a taxonomic characterization of the different types of research strategies found in contemporary cultural anthropology.
Many examples of research in anthropology fall within the two extremes of the continuum.
Types of anthropological research strategies and their features.
Phenomenology, hermeneutics, symbolic anthropology, interpretive anthropology, interpretive interactionism, deconstructionism, postmodernism, and constructivism, to name a few, question, in one way or another, some or all of the ontology, epistemology, and methodology of systematic approaches.
Although some of the older interpretive strategies that emerged from the scientific tradition in the social sciences, such as early interpretive anthropology, still adhered to some logical empiricist methodology and maintained a degree of belief in ethnographic authority, more recent approaches, such as postmodernism and constructivism, are more radical in their sweeping rejection of scientific method and design logic (see Schwandt 1994).
In contrasting Geertz and early interpretive anthropology with some of the later postmodern turns of such ethnographic writers as James Clifford, Rabinow (1986) observes:
Geertz (like the other anthropologists) is still directing his efforts to reinvent an anthropological science with the help of textual mediations.
The other for Clifford is the anthropological representation of the other.
Thus, the rather simple characterization of research strategies found in Figure 1 attempts to recognize the variation inherent in the range of work found in contemporary anthropology by placing "interpretive anthropology" adjacent to "exploratory/descriptive" (see, for example, the work of Zabusky 1995).
Discussions about this debate can be found in Seidman (1994), on the one hand, and Faia (1993), on the other, and, more specifically for anthropology, by Kuznar (1997).
In many ways, this blurs the distinction between what is anthropological and what is literary.
In contrast to Zabusky, there is a body of interpretive work in anthropology that is more extreme in its rejection of systematic design issues.
She rejects the "anthropological `j austerity" of her original work in favor of an "intersubjective understanding" that captures the "flavor" of her ethnographic encounter with the Yanomami.
As Ramos sees it, "I found myself making forays into the self-conscious meanderings of reflexive anthropology in order to shift the axis of analysis from the skeleton-like dissertation to the flesh and blood of ethnography" (p. 6).
These examples offer only a brief glimpse of the range of possible strategies in use by interpretivists in anthropology.
Ethnography, referred to as the anthropological method by William Foote Whyte (1984), has influenced the nature of design in anthropology, but in profoundly different ways.
Whereas the analytical techniques most often used in psychology, sociology, and economics often led to rather standard designs, in anthropology the eclectic nature of ethnography leaves the design of research more open ended.
Designs of this type, however, are often impossible in anthropological fieldwork.
Mensurative designs, then, are observational and characteristic of the types of comparative designs found in field studies in anthropology.
" Anthropologists involved in development and evaluation research are most likely to use this design.
True experiments are, of course, rare in anthropology (but see Harris et al.
Howevert, with careful attention to design and ethnographic context, quasi-experimental and natural experimental designs can be applied to anthropological field settings, particularly in evaluation research and development research.
The most common designs used traditionally by anthropologists have been observational in nature.
Due to their predominance in anthropology, the examples that follow are comparative observational designs.
In anthropological fieldwork, these designs and others can be used in tandem to test or explore components of a theory (such as combinations of time series and repeated measures designs particularly applicable to long-term fieldwork).
His idea is that "to know is to compare" is fundamental to explanatory work in anthropology:
The anthropologists have never studied a school system before.
It would help in this if the anthropologists were to spend half of their time studying another school that was similar, except for the new experimental program.
It would also help if the anthropologists were to study the school for a year or two prior to the program evaluation.
All knowing is comparative, however phenomenally absolute it appears, and an anthropologist is usually in a very poor position for valid comparison, as their own student experience and their secondhand knowledge of schools involve such different perspectives as to be of little comparative use.
Derek Freeman's (1983) criticism of Margaret Mead's work and her findings in Samoa has led to reactions from anthropologists who come from different epistemological traditions.
Research Design in Anthropological Practice:
One of the central concerns of medical anthropologists has been to better understand the relationship between health-related behaviors and native perceptions about illness.
Descriptive inference is an approach highly suited for much anthropological research.
This distinction is important, particularly with regard to anthropological research, in that hypothesis-testing research can be done without narrowly restricting it to analytical methods using statistical inference.
Using a questionnaire to gain background information, 15 college undergraduates who had the least amount of recreational fishing experience were selected from two introductory anthropology classes.
This review of research design and strategies in cultural anthropology only scratches the surface of the research designs, hybrid designs, and combinations of designs possible within an ethnographic context.
As anthropologists, we should take full advantage of both our current understanding of research design and these new developments to produce a "powerful mode of argumentation.
" It is mostly through attention to these concerns that anthropology and anthropologists will have the opportunity to, as Agar says, "move the world.
Research Methods in Anthropology: Qualitative and Quantitative Approaches, 2d ed.
American Anthropologist 22(4):311-321.
American Anthropologist 91(4): 866-889.
Research Design in Anthropology: Paradigms and Pragmatics in the Testing of Hypotheses.
American Anthropologist 87(4):906-910.
Margaret Mead and Samoa: The Making and Unmaking of an Anthropological Myth.
Journal of Quantitative Anthropology I-2(6):49-74.
Anthropological Contributions to the Study of Social Networks: A Review.
Reclaiming a Scientific Anthropology.
Research Design in Anthropological Field Work.
In A Handbook of Methods in Cultural Anthropology.
A Handbook of Method in Cultural Anthropology.
Anthropological Research: The Structure of Inquiry, 2d ed.
Scientific Anthropology at the National Science Foundation.
In Anthropology Between Science and the Humanities.
Representations are Social Facts: Modernity and Post-Modernity in Anthropology.
In A Handbook of Methods in Cultural Anthropology.
American Anthropologist 98(3):555-567.
In A Handbook of Method in Cultural Anthropology.
In the series Studies in Anthropological Method, George Spindler and Louise Spindler, eds.
99 behavior
I am very decidedly of the opinion that a statistical treatment of such intricate behavior as the one that you are studying, will not have very much meaning and that the characterization of a selected number of cases must necessarily be the material with which you operate.
Statistical work will require the tearing out of its natural setting, some particular aspects of behavior which, without that setting may have no meaning whatever. A complete elimination of the subjective use of the investigator is of course quite impossible in a matter of this kind but undoubtedly you will try to overcome this so far as that is all possible.
Implicit in this proposition is the overall theoretical notion that culture is the major factor contributing to human behavior.
Similar to the observation by Brim and Spain, Orans points out that Mead made no comparison of sexual behavior between girls living in a native pastor's household and girls living with their own family.
Among other things, they provide escape and opportunity, tolerate a wider range of behaviors, consist of isolated settlements, have fewer status distinctions, involve novel hardships, have decentralized authority, involve lawlessness and peril, have a reputation for helpfulness and sociability, lag in the development of policies to exploit and regulate, and involve a variety of interest group conflicts.
The idea was to see if there were a difference in key-returning behaviors among the different times.
Contrary to expectations, night-timers were not more amiable than day-timers in their key-returning behaviors;
One of the central concerns of medical anthropologists has been to better understand the relationship between health-related behaviors and native perceptions about illness.
They had to establish differences in treatment choice behavior in the two communities before they could assess any hypotheses concerning differences in beliefs.
1996) offer excellent examples of the role of participant observation in more clearly defining the set of HIV risk behaviors surrounding injection drug use.
Thus, most large epidemiological studies of IDUs focused mainly on direct sharing behaviors in attempts to understand seroconversion rates and other risk factors.
Based on participant observation among IDUs, Koester (1996) identified nine other behaviors that were outside the realm of the direct sharing of a single syringe by two or more IDUs.
Termed "indirect sharing," these nine behaviors can promote the transmission of HIV among IDUs who, although not sharing needles directly, often share water for mixing of drugs or for rinsing syringes, share drug-mixing containers (cookers and spoons), share cottons for filtering, and share the actual drug solution itself.
These findings are undeniably important for larger epidemiological work that examines elements of IDUs' behaviors and such things as producing valid models of seroconversion.
(1996) used these additional distinctions in sharing to look at the prevalence of injection-related HIV risk behaviors among several subpopulations of injection drug users (see Figure 6).
Statistical tests of group differences provided a greater understanding of the risk factors associated with the different types of behavior.
The Prevalence of Additional InjectionRelation HIV Risk Behaviors Among Injection Drug Users.
99 comparison
She wrote of her doubts about the comparability of cases and about her ability, or even the need, to do a quantitative comparison of the similarity of attitudes among the adolescent girls in her study.
She had concerns-and I believe she thought her mentor, Boas, would feel similarly-as to whether a valid comparison of this type could be made given the selection process for her sample of girls.
Analysis-of-variance models and multigroup comparisons (factorial designs) may dictate the whos, whats, and wheres of a given project.
Exploratory research may contribute to the production of reliable and valid measures, provide information essential for constructing comparison groups, facilitate construction of structured questions or questionnaires, or provide information necessary for producing a sound probability or nonprobability sample.
the effects of selection (that is, comparison groups differ because of the way they were selected and not due to the treatment);
Random allocation produces equivalent comparison groups, and artificial manipulation of independent variables (also known as explanatory variables or study factors), with all other variables or factors controlled for, allows for the most valid assessment of the causal relationship between the independent and dependent variables or response variables.
Observational studies involve neither random assignment of members to comparison groups nor the manipulation by the observer of independent variables.
These are basically true experiments involving random assignment, multiple comparisons (for example, treatment versus control), and the manipulation of independent variables.
He refers to the second as mensurative experiments, which involve simply the measurement of variables in space and time and among a number of comparison groups, without random allocation and the manipulation of experimental factors.
While random assignment aids in controlling for confounding variables by producing homogeneous comparative groups, random sampling of units produces comparison groups that are representative of such groups.
Thus, comparison groups may be chosen on the basis of different levels of exposure to some naturally occurring or human-induced phenomena (for example, natural disaster, war, or the building of a dam).
All knowing is comparative, however phenomenally absolute it appears, and an anthropologist is usually in a very poor position for valid comparison, as their own student experience and their secondhand knowledge of schools involve such different perspectives as to be of little comparative use.
The nature of the groups or characteristics to be compared-in terms of such things as the size of the comparison groups in the overall population, the frequency of characteristics of interest in the population, the availability of a sampling frame, the ability to identify members of the population (for example, hidden or clandestine populations)-all influence the choice of a sample design.
Mead used what can be referred to as a static group comparison design with a conjectural treatment group.
The comparison group, Samoan adolescent girls, was compared to a conjectural treatment group, American adolescent girls, to test the proposition that exposure to Western civilization increases adolescent trauma.
In lieu of the between-culture comparisons, Mead could have made a within-case comparison that would have suffered less from problems with possible sources of error.
She could have chosen comparison groups that were as similar as possible in order to rule out the effects of unmeasured variables as much as possible.
There was a lack of comparisons between various sources of data that were crucial to Mead's argument.
She did not, however, make any systematic comparisons among the different units.
Similar to the observation by Brim and Spain, Orans points out that Mead made no comparison of sexual behavior between girls living in a native pastor's household and girls living with their own family.
Young and Garro's (1982) investigation of treatment choice in two Mexican communities is an example of a static-group comparison where the presence or absence of the treatment is based on selection criteria not directly under the control of the researchers.
The research design included the comparison of two Mexican communities that were similar in terms of cultural traditions and economies but varied in terms of access to Western medical services.
Because respondents weren't randomly assigned into comparison groups, it's difficult to know the influences of confounding variables on physician utilization and beliefs about illness.
It is unrealistic to suppose that Young and Garro could have randomly assigned community members to the different comparison groups in order to control for confounding variables and then subject their informants to the treatments of interest.
Boster and Johnson used a static group comparison design to compare several groups of expert fishermen with a group of novice fishermen.
In the comparison, both culture and language were held constant while experience with fish was varied.
A major component of the study was the comparison of IDUs who engaged in both direct sharing and indirect sharing with IDUs who engaged in indirect sharing only and those who several subpopulations of injection drug users (see Figure 6).
A major component of the study was the comparison of IDUs who engaged in both direct sharing and indirect sharing with IDUs who engaged in indirect sharing only and those who neither shared directly nor indirectly.
This design allowed for a variety of comparisons, including comparisons by controls and asustados, by gender, and by matched pairs both within and between cultural groups (see Figure 7).
Comparison of Models
Her selection of variables allowed a comparison of different levels (for example, Catholic versus Protestant) across the four variables.
Form or Function: A Comparison of Expert and Novice Judgments of Similarity Among Fish.
99 discussion
Ironically, this concern for context and meaning over methodological rigor, particularly for those in search of theoretical foundations (that is, the Boasian idea of data leading to the construction of theory), would ultimately hinder the comparability of data from different ethnographic sources (see Moran [1995] for a recent discussion of this issue and see Ember and Ember, this volume).
Mead's position on these various elements of research design provided fuel for the continuing discussions about the validity of her original findings (Brim and Spain 1974;
A discussion of the basic arguments as related to epistemology, objectivity, reality, authority, and the like are beyond the scope of this chapter (see Schweizer in this volume).
Discussions about this debate can be found in Seidman (1994), on the one hand, and Faia (1993), on the other, and, more specifically for anthropology, by Kuznar (1997).
Although Ramos discusses informant interviewing and various sources of data, her introduction is largely devoted to discussions of her reliance on her own memory in writing the ethnography and the shift in the narrative between synchrony and diachrony.
Thus, there is little discussion of research design and methods of data collection as might be found in work in the systematic tradition.
Thus, traditional methods sections are replaced by discussions on how to read the work or on the particular methods used in writing the ethnography itself (see, for example, Panourgia's discussion on the use of the parerga).
For further discussion of research strategies in the interpretive mode, see Fernandez and Herzfeld (this volume).
The bad news is that the open-ended character of ethnography contributes to a less well-focused discussion of research design issues in ethnographic approaches.
(Stinchcombe [1987] provides an excellent discussion of how empirical statements are derived from theory.
Table 1 describes examples from observational and quasi-experimental study designs discussed by Kleinbaum et al.
Discussing "common sense knowing" in evaluation research, Campbell (1988) gives an important critique of ethnography.
Here, I stress the importance of thinking through how validity threats have influenced and will influence observations or data (for a more in-depth discussion of how these types of validity can impact study conclusions, see Cook and Campbell 1979).
Tables 2 and 3 give examples of threats to internal and external validity as discussed in Cook and Campbell (1979) for quasi-experimental designs.
(1982) offer a similar discussion of the strengths and weaknesses of observational designs in terms of controlling for threats to both internal and external validity.
There is a vast literature on sampling theory and random sampling procedures, including discussions of sample sizes (see, for example, Bernard [1994] for a summary and Babbie [1990] for detailed discussion of sampling issues).
The next section shows how concern for the elimination of potential errors and bias through design and attention to methodological detail applies to discussions about the findings of Margaret Mead and Derek Freeman in Samoa.
The following examples illustrate some of the issues discussed so far.
This isn't perfect, but a greater in-depth exploratory understanding and an explicit discussion of design can enhance our chances for the production of valid explanations.
Indeed, I interacted with several individuals who had life histories that were inconsistent with my general characterization and who were the basis for suggesting the competing models discussed above.
99 drug
1996) offer excellent examples of the role of participant observation in more clearly defining the set of HIV risk behaviors surrounding injection drug use.
In most earlier research on injection drug users (IDUs) and HIV risk, the primary risk factor was viewed in terms of direct needle sharing.
Termed "indirect sharing," these nine behaviors can promote the transmission of HIV among IDUs who, although not sharing needles directly, often share water for mixing of drugs or for rinsing syringes, share drug-mixing containers (cookers and spoons), share cottons for filtering, and share the actual drug solution itself.
(1996) used these additional distinctions in sharing to look at the prevalence of injection-related HIV risk behaviors among several subpopulations of injection drug users (see Figure 6).
A major component of the study was the comparison of IDUs who engaged in both direct sharing and indirect sharing with IDUs who engaged in indirect sharing only and those who several subpopulations of injection drug users (see Figure 6).
The Process of Drug Injection: Applying Ethnography to the Study of HIV Risk Among IDU's.
In AIDS, Drugs and Prevention: Perspectives on Individual and Community Action.
The Prevalence of Additional InjectionRelation HIV Risk Behaviors Among Injection Drug Users.
99 ethnographic
First, research design involves an a priori plan or strategy for all phases of the research (such as data collection and analysis) including, for some researchers, the production of the final product (like an ethnography).
Interpretive, hermeneutic, and postmodern approaches make little explicit reference to ethnographic design issues, but well-written examples from ethnography may provide "moral evidence" to deal with current social problems, moving people (including politicians) in ways that numerical facts can't (Seidman 1994:134).
In this chapter I concentrate on elements of design related to the production of valid results or a believable ethnographic account.
Boas and most of his students advocated a natural science logic in the collection of ethnographic materials and a true concern for the collection of reliable data that could lead to the production of valid theory.
His earliest contributions, however, were more a demonstration of the value of ethnographic writing-his "unusual literary sense" (Lowie 1937:231)--rather than of methodological details of proper ethnographic fieldwork (Ellen 1984).
A good example of this tension between the stated early concerns for the methods of science and the actual use of such methods in ethnography comes from correspondence between Boas and his student Margaret Mead during her first fieldwork in Samoa.
First, it demonstrates the differences between the stated scientific objectives of ethnographic work as advocated by Boas and the actual practice of ethnographic research.
Ironically, this concern for context and meaning over methodological rigor, particularly for those in search of theoretical foundations (that is, the Boasian idea of data leading to the construction of theory), would ultimately hinder the comparability of data from different ethnographic sources (see Moran [1995] for a recent discussion of this issue and see Ember and Ember, this volume).
Rightly or wrongly, the preeminence of contextualization has been a consistent issue in ethnographic research and has often clouded issues in research design.
The idea that quantification detracts from context and meaning in the ethnographic endeavor-evident even in the time of Boas-and a failure to understand that systematic methods-whether quantitative or qualitative-help minimize the subjectivity of the investigator have impeded the development of well-delineated research strategies in anthropology.
Boas's final sentence in his response to Mead illustrates that even at this early stage the issue of the subjectivity of ethnographic research was of concern.
Thus, while early British and U.S. anthropologists advocated the scientific method in ethnographic research, there is little evidence that they considered appropriate design issues when they actually did the research.
There is much anecdotal evidence for a belief, across the British and U.S. traditions, in a trial-by-fire method of training for ethnographers.
Agar (1980) and Bernard (1994) relate stories about Kroeber's recommendations regarding the teaching and conduct of ethnographic research.
Further, the huge investment in time and resources limited another important goal of science, that of replication, since an ethnographer couldn't realistically be expected to replicate someone else's work.
Interpretive strategies, on the other hand, differ from systematic approaches in that they question a researcher's ability to maintain objectivity, particularly in the ethnographic context where the ethnographer is often the instrument of measurement.
Although some of the older interpretive strategies that emerged from the scientific tradition in the social sciences, such as early interpretive anthropology, still adhered to some logical empiricist methodology and maintained a degree of belief in ethnographic authority, more recent approaches, such as postmodernism and constructivism, are more radical in their sweeping rejection of scientific method and design logic (see Schwandt 1994).
In contrasting Geertz and early interpretive anthropology with some of the later postmodern turns of such ethnographic writers as James Clifford, Rabinow (1986) observes:
Thus, there is less focus on the means of research, such as methods of data collection and analysis as found in the systematic strategies, and more on the ends of research-the ethnographic or literary product.
For scholars like Geertz, analysis of ethnography has less to do with the methods of observation and description than the inscriptions and writings concerning the meaning of human action.
More extreme forays into experimental ethnography have blurred this distinction even further, and there is more of a focus on writing strategies that include such approaches as montages, evocative representations, polyvocal texts, and even ethnographic fictions (Denzin and Lincoln 1994).
A good example of this is Zabusky's (1995) ethnographic study of cooperation in European space science that she admits "took the form of mutual exploration rather than unidirectional examination" (p. 46).
Following in the "thick description" tradition of Geertz, Zabusky clearly believes in some kind of ethnographic authority.
She also discusses the rationales for selecting the site and the group she studied, problems of working in a linguistically and technically diverse social milieu, the use of semistructured and unstructured interviews, and the effect of her role as ethnographer on informant relations and data quality.
Although Zabusky doesn't talk specifically about design or about concerns for potential threats to validity, there is implicit concern for such issues throughout the ethnography.
Ramos (1995), for example, has recently published an ethnography based on a rewrite of her 1972 ;
: dissertation, with additional ethnographic insights.
She rejects the "anthropological `j austerity" of her original work in favor of an "intersubjective understanding" that captures the "flavor" of her ethnographic encounter with the Yanomami.
As Ramos sees it, "I found myself making forays into the self-conscious meanderings of reflexive anthropology in order to shift the axis of analysis from the skeleton-like dissertation to the flesh and blood of ethnography" (p. 6).
Along with this shift came the freedom not to be concerned with issues of bias and validity or with the need for working systematically, thus allowing for a less restrictive ethnographic narrative.
Although Ramos discusses informant interviewing and various sources of data, her introduction is largely devoted to discussions of her reliance on her own memory in writing the ethnography and the shift in the narrative between synchrony and diachrony.
Instead, Ramos emphasizes the emergent and reflexive nature of data and the literary strategies used in producing the ethnographic product.
The idea of a montage as an organizing principle was also central to Taussig's (1987) historical and ethnographic account of shamanism, colonialism, and terror in South America.
For others, interpretive work is concerned more with the strategies and methods of ethnographic presentation and with the reflexive character of the ethnographic enterprise.
Thus, traditional methods sections are replaced by discussions on how to read the work or on the particular methods used in writing the ethnography itself (see, for example, Panourgia's discussion on the use of the parerga).
Ethnography, referred to as the anthropological method by William Foote Whyte (1984), has influenced the nature of design in anthropology, but in profoundly different ways.
Whereas the analytical techniques most often used in psychology, sociology, and economics often led to rather standard designs, in anthropology the eclectic nature of ethnography leaves the design of research more open ended.
There are generally no ethnographic "analytical techniques" driving the design, although ethnography has been variously associated with a number of qualitative methods.
The good news is that ethnographic research is amenable to a wide range of research designs, including the use of multiple designs within a single ethnographic context.
The bad news is that the open-ended character of ethnography contributes to a less well-focused discussion of research design issues in ethnographic approaches.
Part of the confusion stems from a lack of consensus on what ethnography really is (Johnson 1990).
Although this process might be equated to a method, it's better to think of ethnography as a strategy in which a variety of methods can be used in the quest for knowledge (Pelto and Pelto 1978).
Thus, ethnography should involve multiple methods, both qualitative and quantitative, and may involve applying more than one research design.
Figure 2 illustrates the relationship between exploratory and explanatory approaches within the ethnographic context.
As with research in community ecology, ethnographic research can be purely exploratory or descriptive involving a research process focused on producing better theory-or purely explanatory, although this is usually not the case.
redoing a year-long ethnographic field study because of such errors is quite another.
Howevert, with careful attention to design and ethnographic context, quasi-experimental and natural experimental designs can be applied to anthropological field settings, particularly in evaluation research and development research.
When one is interested in explanation, the importance of comparative thinking in ethnographic work cannot be overemphasized.
Discussing "common sense knowing" in evaluation research, Campbell (1988) gives an important critique of ethnography.
(This would be hard to schedule, but we might regard the current school ethnographies as pre-studies for new innovations still to come.
Nonprobability sampling methods have come to be associated with qualitative approaches or for the selection of ethnographic informants, particularly key informants or consultants (Werner and Schoepfle 1987;
These new approaches seem particularly well suited for the imperfect world of ethnographic research, where the rather restrictive assumptions of parametric analysis are often difficult to meet.
In ethnographic research, no matter the mix of methods, the design of the study should allow for an ethnographers' hypotheses or hunches to be rejected as well as confirmed (Campbell 1975).
Nevertheless, field experiments can be quite informative and, if carefully constructed, can provide formal tests of hypotheses derived from and complementary with ethnography.
In lieu of equalization through randomization, Young and Garro, through extensive ethnographic background research, produced groups that, although nonequivalent in the quasi-experimental sense, shared similarities with regard to a number of important characteristics.
Further, beliefs about the use and functional characteristics of the fish obtained from extensive ethnographic interviews were turned into a sentence-frame completion task described by Weller and Romney (1988).
But the in-depth ethnographic background research, the particular structure of the hypothesis, and the overwhelming reliability of informant responses make for more confidence in the possible validity of the study's conclusions.
Based on earlier ethnographic research, social stress, an important component for understanding an individual's inability to perform social roles, was operationalized using the Social Stress Gauge developed by one of the researchers.
Multimethod Ethnography and the
Her study design incorporates quantitative and qualitative methods in the overall ethnographic enterprise.
This study is an example of multimethod ethnography in which there was a combination of exploratory and explanatory approaches-that is, qualitative data and tests of models with data collected using a cross-sectional design.
This review of research design and strategies in cultural anthropology only scratches the surface of the research designs, hybrid designs, and combinations of designs possible within an ethnographic context.
The newer forays into experimental and other ethnographic forms of presentation are more reflexive in character and more concerned with believable and moving representations rather than the production of valid accounts or conclusions.
The strength of the ethnographic approach is its ability to incorporate a wide range of methods, strategies, and designs within a single enterprise, all combining in ways to improve the chances for credible results.
The Professional Stranger: An Informal Introduction to Ethnography.
In Ethnographic Research: A Guide to General Conduct.
Evidence and Science in Ethnography: Reflections on the Mead-Freeman Controversy.
Selecting Ethnographic Informants.
The Process of Drug Injection: Applying Ethnography to the Study of HIV Risk Among IDU's.
In Ethnographic Research: A Guide to General Conduct.
Tales of the Field: On Writing Ethnography.
Launching Europe: An Ethnography of European Cooperation in Space Science.
99 experience
In the first experiment, the control net caught 10 porpoises while the treatment net caught none.
There was a wide belief among British anthropologists that fieldwork could not be taught to new recruits, but could only be experienced by individuals in the field.
In the American tradition texts provided what was regarded as an objective body of data, whereas the British tradition was more a matter of subjective experience.
And since I have indicated here that research in culture involves a great deal of unique personal experience for the anthropologist, I have taken the position that it is probably unlikely there can be a rigorous, systematic, and formal presentation of methods in the study of culture like those of the natural sciences and that there are overriding concerns among many sociologists, psychologists, and economists.
I find this stance comfortable, for it is my conviction that so long as prime theoretical concerns in the study of culture are an attempt to record and understand the native's view of his culture and the objective and historical realities of culture, then methods for field study will have to reflect the end purpose of making a whole account of a part of the human experience.
Hurlbert (1984) emphasizes this in a classic paper on the design of field experiments in ecology.
On the other hand, the only complete remedy for design or execution errors is repetition of the experiment" (p. 189).
Redoing an experiment because of fundamental design errors is one matter;
True experiments involve random assignment and afford the best chances for controlling for things like: the effects of extraneous factors (that is, unmeasured variables that might affect the dependent variable);
These and other sources of error are all potential rival hypotheses and randomized experiments are best at eliminating the threats of rival explanations.
Included are experiments, quasi-experiments, observational study designs, and what I refer to as natural experiments.
Experiments involve the random allocation of subjects to groups and afford the most control over distorting effects from extraneous factors.
What separates quasi-experiments from true experiments is the lack of random assignment of group members.
Nonrandom assignment lays an experiment open to validity threats and reduces our ability to make causal inferences.
Hurlbert (1984) distinguishes between two classes of experiments.
He terms the first manipulative experiments.
These are basically true experiments involving random assignment, multiple comparisons (for example, treatment versus control), and the manipulation of independent variables.
He refers to the second as mensurative experiments, which involve simply the measurement of variables in space and time and among a number of comparison groups, without random allocation and the manipulation of experimental factors.
In manipulative experiments, analytical units are randomly allocated to comparative groups, whereas in mensurative experiments selection of units is based on some probability or nonprobability sampling scheme.
Finally, natural experiments are similar to quasi-experiments except that the manipulation of independent variables occurs naturally or is unplanned rather than artificial or directed.
Cook and Campbell (1979) make a similar distinction but refer to these kinds of natural experiments as "passive-observational studies.
True experiments are, of course, rare in anthropology (but see Harris et al.
[1993] for an example of a true experiment in a field setting).
Even in quasi-experiments, it's often difficult to manipulate independent variables directly.
From a statistical standpoint, designs that don't involve random assignment including quasi-experiments-are considered observational (Cook and Campbell 1979).
It is important, though, to contrast quasi-experiments to what Kleinbaum et al.
All knowing is comparative, however phenomenally absolute it appears, and an anthropologist is usually in a very poor position for valid comparison, as their own student experience and their secondhand knowledge of schools involve such different perspectives as to be of little comparative use.
Cook and Campbell (1979) discuss two sampling models for increasing external validity in quasi-experiments.
For example, there was no measurement on which to compare differences in stress experienced by adolescents in Samoa and the United States.
Field Experiments
In field experiments, the experimenter has little control over all possible extraneous factors and the experiment may not involve random assignment of subjects to groups.
Nevertheless, field experiments can be quite informative and, if carefully constructed, can provide formal tests of hypotheses derived from and complementary with ethnography.
He designed a clever experiment in which keys were placed at similar locations during each two-hour field visit over a 24-hour period covering day and night.
Melbin speculates that the variation in results may have been due to the fact that the other three experiments involved direct personal contact among the subjects, while the key experiment involved no such interactions.
Had Melbin conducted only the key experiment, he may have come to very different conclusions regarding the helpfulness and sociability of night-timers.
In all research, but particularly in field experiments like the one described above, there should be a concern for ethics and the well-being of experimental participants.
Unlike studies where informed consent is obtained prior to participation, in experiments like Melbin's, individuals often participate without knowing about it.
This is analogous to treatment and control groups without the random assignment of subjects to experimental units and where the treatment is implied rather than researcher directed (that is, natural differences in experience with fish).
In the comparison, both culture and language were held constant while experience with fish was varied.
Four groups-from North Carolina, East Florida, West Florida, and Texas-were sampled to examine the effects of different kinds of experience since there are regional variation in species abundance.
To ensure that experts were, in fact, experienced recreational fishermen, the rosters of sport fishing clubs in each region were sampled at random.
The selection of control group subjects, by contrast, involved a purposeful selection procedure in which potential subjects were screened for recreational fishing experience.
Using a questionnaire to gain background information, 15 college undergraduates who had the least amount of recreational fishing experience were selected from two introductory anthropology classes.
Beyond these constancies of cultural and marital experience, they were selected to maximize diversity with regard to such obvious differences as their occupations and educational backgrounds, religious affiliations and ethnic and racial identities, their neighborhoods and social networks, and the duration of their marriages.
Folk beliefs surrounding susto attribute loss of a critical substance or force due to a frightening experience.
Many peasant societies in Central America are experiencing dramatic economic and cultural change.
Using her experience as a participant observer, Goldin developed a survey which she applied to a random sample of 10% of the heads of households in the township (n = 57).
Pseudoreplication and Design of Ecological Field Experiments.
Experiments in the Field.
Learning from the Field: A Guide from Experience.
99 experimental
The lessons learned from controlled experimentation are applied today to the policy arena where groups are in conflict over resources or because of social inequalities (Johnson and Pollnac 1989;
So another study was conducted placing experimental treatment and control nets in the same proximity.
Thus, fishers were blind as to which nets were control and which were treatment-a classic double-blind experimental design.
Although the power of experimental design is evident, concern for its application in anthropology-particularly cultural anthropology-has been limited.
More extreme forays into experimental ethnography have blurred this distinction even further, and there is more of a focus on writing strategies that include such approaches as montages, evocative representations, polyvocal texts, and even ethnographic fictions (Denzin and Lincoln 1994).
She contrasts her study with research on cooperation by "experimental" psychologists, emphasizing the cultural and social orientation of her work and the importance of considering context (social, cultural, political, etc.
"Statistical analysis and interpretation," he says, "are the least critical aspects of experimentation, in that if purely statistical or interpretive errors are made, the data can be reanalyzed.
Nevertheless, the principles of experimentation are instructive and are a guide for understanding potential sources of error, even in a nonlaboratory setting.
Random assignment maximizes the probability that experimental groups are equivalent on key variables prior to the introduction of an intervention.
This distinction between experimental and observational approaches is similar to one in ecological field studies.
He refers to the second as mensurative experiments, which involve simply the measurement of variables in space and time and among a number of comparison groups, without random allocation and the manipulation of experimental factors.
Howevert, with careful attention to design and ethnographic context, quasi-experimental and natural experimental designs can be applied to anthropological field settings, particularly in evaluation research and development research.
Johnson and Murray (1997), for example, used a quasi-experimental design to evaluate the use of fish aggregation devices (FADS) in small-scale fisheries development projects.
Most research designs in the explanatory mode, like true experimental designs, are comparative (for example, control versus treatment).
Table 1 describes examples from observational and quasi-experimental study designs discussed by Kleinbaum et al.
They have been hired after (or just as) the experimental program has got under way, and are inevitably studying a mixture of the old and the new under conditions in which it is easy to make the mistake of attributing to the program results which would have been there anyway.
It would help in this if the anthropologists were to spend half of their time studying another school that was similar, except for the new experimental program.
While the purpose of experimental design is to ward off as threats to validity, there are several types of validity-face, construct, statistical conclusion, internal, external, etc.
Tables 2 and 3 give examples of threats to internal and external validity as discussed in Cook and Campbell (1979) for quasi-experimental designs.
In the quasi-experimental case, this means changes between pre- and posttest, but this way of thinking can be expanded to include hypothesized effects dealing with differences, similarities, or associations whether diachronic or synchronic.
Cook and Cambell (1979) detail how each of the quasi-experimental designs in Table 1 are better or worse at dealing with each of the threats to validity that are found in Tables 2 and 3.
The logic behind this model can be extended beyond the quasi-experimental case to observational studies.
In field experiments, the experimenter has little control over all possible extraneous factors and the experiment may not involve random assignment of subjects to groups.
In all research, but particularly in field experiments like the one described above, there should be a concern for ethics and the well-being of experimental participants.
The ramifications and consequences of experimental outcomes must be considered thoroughly before any experimental design is implemented.
In lieu of equalization through randomization, Young and Garro, through extensive ethnographic background research, produced groups that, although nonequivalent in the quasi-experimental sense, shared similarities with regard to a number of important characteristics.
This is analogous to treatment and control groups without the random assignment of subjects to experimental units and where the treatment is implied rather than researcher directed (that is, natural differences in experience with fish).
The newer forays into experimental and other ethnographic forms of presentation are more reflexive in character and more concerned with believable and moving representations rather than the production of valid accounts or conclusions.
Quasi-Experimentation: Design and Analysis for Field Settings.
An Example of Research Design: Experimental Design in the Study of Culture Change.
99 Explanatory
Suffice to say that traditionally, research design and its logic have been associated with science and an underlying belief in objectivity and explanation.
Under the systematic distinction are the two primary categories of exploratory and explanatory approaches, each entailing a specific design strategy.
In its most extreme form, systematic strategies tend to involve the search for explanations of phenomena and the pursuit of theoretical foundations.
Explanatory: Explanatory approaches generally involve testing elements of theory that may already have been proposed in the literature or that have been informed by exploratory research.
In this enterprise, explanation can involve a general search for causality or prediction.
There is a fundamental belief that the intersubjective, everyday meanings and how they are produced, maintained, and changed in any given context often defy objective study and explanation.
Practitioners of almost all interpretive paradigms are searching in one way or another for some understanding (verstehen) rather than for some explanation of social phenomena.
First, it is representative of the genre that rejects explanation in favor of conveying a moral tale.
Its purpose is not a traditional attempt at explanation where facts are considered real, but political interpretation and representation of facts, independent of their "realness.
As against the magic of academic rituals of explanation which, their alchemical promise of yielding system from chaos, do nothing to ruffle the placid surface of this natural order, I choose to work with a different conflation of modernism and the primitivism it conjures into life-namely the carrying over into history of the principle of montage, as I learned that principle not only from terror, but from Putumayo shamanism with its adroit, albeit unconscious, use of the magic of history and its healing power.
Currently, the qualitative analysis of text and discourse is no longer: restricted to either interpretive or exploratory approaches, but can also be used in hypothesis testing and explanatory research.
Figure 2 illustrates the relationship between exploratory and explanatory approaches within the ethnographic context.
This contrast between explanatory and descriptive or exploratory approaches is commonly made in nonexperimental disciplines in both the natural and social sciences.
As with research in community ecology, ethnographic research can be purely exploratory or descriptive involving a research process focused on producing better theory-or purely explanatory, although this is usually not the case.
Rather, the most common model has exploratory research informing and complementing explanatory research.
As we will see in the examples to come, exploratory research is often an essential component of the explanatory research process.
These and other sources of error are all potential rival hypotheses and randomized experiments are best at eliminating the threats of rival explanations.
Random allocation produces equivalent comparison groups, and artificial manipulation of independent variables (also known as explanatory variables or study factors), with all other variables or factors controlled for, allows for the most valid assessment of the causal relationship between the independent and dependent variables or response variables.
Most research designs in the explanatory mode, like true experimental designs, are comparative (for example, control versus treatment).
When one is interested in explanation, the importance of comparative thinking in ethnographic work cannot be overemphasized.
His idea is that "to know is to compare" is fundamental to explanatory work in anthropology:
These examples show how, even for less powerful designs, the interplay of exploratory and explanatory approaches can aid in guarding against threats to validity (Robson 1993).
One explanation views use tied to congruence between a client's medical beliefs and scientific medical theory: the higher the congruence, the more likely the client will choose a physician's treatment.
This isn't perfect, but a greater in-depth exploratory understanding and an explicit discussion of design can enhance our chances for the production of valid explanations.
Issues of validity in this case are not as overriding as they would be in a purely explanatory study.
However, further research in the explanatory mode is now warranted.
As seen in Figure 2, exploratory and descriptive research are often essential components of an overall explanatory research design.
This is a good example of the application of exploratory research in the production of better measures of potentially important explanatory variables.
This provided for an evaluation of the explanatory power of each.
This study is an example of multimethod ethnography in which there was a combination of exploratory and explanatory approaches-that is, qualitative data and tests of models with data collected using a cross-sectional design.
99 Figure
Figure I is a taxonomic characterization of the different types of research strategies found in contemporary cultural anthropology.
The figure distinguishes between strategies within the realm of interpretive studies and those using systematic strategies that have more of the elements of science.
Figure 1.
However, some interpretive work is more similar in nature to the exploratory or descriptive strategies found under the systematic side of Figure 1 than to some of the more radical forays into, for example, postmodernism.
Thus, the rather simple characterization of research strategies found in Figure 1 attempts to recognize the variation inherent in the range of work found in contemporary anthropology by placing "interpretive anthropology" adjacent to "exploratory/descriptive" (see, for example, the work of Zabusky 1995).
Figure 2 illustrates the relationship between exploratory and explanatory approaches within the ethnographic context.
The figure shows that the overall research process is more than just a matter of study design.
Figure 2 shows that the research process involves a simultaneous concern for the development of empirical statements from theory (for example, hypotheses), the operationalization of theoretical concepts (for example, meaningful and reliable measures), design (for example, groups to be studied), data collection (for example, qualitative versus quantitative), and data analysis (for example, multiple regression and text analysis).
In Young and Garro's case, a visual inspection of the graphical representations of the data could lead to no other conclusion than that there was little or no difference in beliefs between the two communities (see Figure 3).
Using statistical and descriptive inference, the authors concluded that whether informants use form or function for classification depends on the knowledge base of the informants and the methods used to test their knowledge (see Figure 4).
Based on an in-depth analysis of informants' discourse about marriage, Quinn produced a cultural model incorporating a number of causal links in informants' reasoning as to a "lasting marriage" (see Figure 5).
As seen in Figure 2, exploratory and descriptive research are often essential components of an overall explanatory research design.
(1996) used these additional distinctions in sharing to look at the prevalence of injection-related HIV risk behaviors among several subpopulations of injection drug users (see Figure 6).
A major component of the study was the comparison of IDUs who engaged in both direct sharing and indirect sharing with IDUs who engaged in indirect sharing only and those who several subpopulations of injection drug users (see Figure 6).
This design allowed for a variety of comparisons, including comparisons by controls and asustados, by gender, and by matched pairs both within and between cultural groups (see Figure 7).
The study shows how the use of multiple methods fosters triangulation that contributes to the production of valid conclusions (see Figure 8).
99 foundations
I use "defensible" in addition to "valid," which I normally use, to make readers aware that I am broadening the traditional application of research design to include the variety of research strategies found in anthropology today.
There is much variation in what funding agencies and foundations expect regarding research design.
Ironically, this concern for context and meaning over methodological rigor, particularly for those in search of theoretical foundations (that is, the Boasian idea of data leading to the construction of theory), would ultimately hinder the comparability of data from different ethnographic sources (see Moran [1995] for a recent discussion of this issue and see Ember and Ember, this volume).
It is a strange paradox in the development of field methods that the scientific study of other cultures has been built upon such a foundation.
A more appropriate term that would encompass the diversity currently found in cultural anthropology might be "research strategy.
Figure I is a taxonomic characterization of the different types of research strategies found in contemporary cultural anthropology.
In its most extreme form, systematic strategies tend to involve the search for explanations of phenomena and the pursuit of theoretical foundations.
In searching for such foundations, there is a need for objectivity, replication, and control over possible sources of error leading to a valid assessment of a given theory.
However, some interpretive work is more similar in nature to the exploratory or descriptive strategies found under the systematic side of Figure 1 than to some of the more radical forays into, for example, postmodernism.
Thus, the rather simple characterization of research strategies found in Figure 1 attempts to recognize the variation inherent in the range of work found in contemporary anthropology by placing "interpretive anthropology" adjacent to "exploratory/descriptive" (see, for example, the work of Zabusky 1995).
Discussions about this debate can be found in Seidman (1994), on the one hand, and Faia (1993), on the other, and, more specifically for anthropology, by Kuznar (1997).
An important implication here is that scholars who follow this line of inquiry are searching for local rationales rather than nomothetic theory or universal foundations and may be more interested in conveying a moral tale of some type rather than a value-free account (Seidman 1994).
Thus, there is less focus on the means of research, such as methods of data collection and analysis as found in the systematic strategies, and more on the ends of research-the ethnographic or literary product.
As Ramos sees it, "I found myself making forays into the self-conscious meanderings of reflexive anthropology in order to shift the axis of analysis from the skeleton-like dissertation to the flesh and blood of ethnography" (p. 6).
Thus, there is little discussion of research design and methods of data collection as might be found in work in the systematic tradition.
Design, however, is the foundation of good research.
Mensurative designs, then, are observational and characteristic of the types of comparative designs found in field studies in anthropology.
More details can be found in these and other sources (for example, Robson 1993).
Cook and Cambell (1979) detail how each of the quasi-experimental designs in Table 1 are better or worse at dealing with each of the threats to validity that are found in Tables 2 and 3.
In one approach, the model of deliberate sampling for heterogeneity, target classes of units, whether classes or categories of persons, places, times, or events, are deliberately chosen to represent the range of such classes found in the population.
Orans (1996) reanalyzed Mead's field notes and correspondence and once again found that her depiction of Samoa as a halcyon society was at odds with his own impression of Samoa as much more agonistic.
But, in common with Brim and Spain, Orans found itemized problems with Mead's research design.
Using a standard chi-square test, the authors found a significant difference in the frequency distribution of treatment alternatives between the two towns, with the exception of folk curers.
Using multidimensional scaling, Young and Garro (1982) compared the belief data and found striking similarities in the medical beliefs of communities.
All the groups were shown cards with artists' renderings and the common names of 43 marine species commonly found from North Carolina to Texas.
Her sample is an attempt to capture the range of diversity found in the region.
(1985) report on a study of a folk illness known as susto, found in many cultural groups throughout North and South America.
found that there was, in fact, an association between susto and an individual's perception of the adequacy of his or her performance of critical social roles.
Scientific Anthropology at the National Science Foundation.
99 HIV
1996) offer excellent examples of the role of participant observation in more clearly defining the set of HIV risk behaviors surrounding injection drug use.
In most earlier research on injection drug users (IDUs) and HIV risk, the primary risk factor was viewed in terms of direct needle sharing.
Termed "indirect sharing," these nine behaviors can promote the transmission of HIV among IDUs who, although not sharing needles directly, often share water for mixing of drugs or for rinsing syringes, share drug-mixing containers (cookers and spoons), share cottons for filtering, and share the actual drug solution itself.
(1996) used these additional distinctions in sharing to look at the prevalence of injection-related HIV risk behaviors among several subpopulations of injection drug users (see Figure 6).
The Process of Drug Injection: Applying Ethnography to the Study of HIV Risk Among IDU's.
The Prevalence of Additional InjectionRelation HIV Risk Behaviors Among Injection Drug Users.
99 IDUs
In most earlier research on injection drug users (IDUs) and HIV risk, the primary risk factor was viewed in terms of direct needle sharing.
Thus, most large epidemiological studies of IDUs focused mainly on direct sharing behaviors in attempts to understand seroconversion rates and other risk factors.
Based on participant observation among IDUs, Koester (1996) identified nine other behaviors that were outside the realm of the direct sharing of a single syringe by two or more IDUs.
Termed "indirect sharing," these nine behaviors can promote the transmission of HIV among IDUs who, although not sharing needles directly, often share water for mixing of drugs or for rinsing syringes, share drug-mixing containers (cookers and spoons), share cottons for filtering, and share the actual drug solution itself.
These findings are undeniably important for larger epidemiological work that examines elements of IDUs' behaviors and such things as producing valid models of seroconversion.
A major component of the study was the comparison of IDUs who engaged in both direct sharing and indirect sharing with IDUs who engaged in indirect sharing only and those who several subpopulations of injection drug users (see Figure 6).
A major component of the study was the comparison of IDUs who engaged in both direct sharing and indirect sharing with IDUs who engaged in indirect sharing only and those who neither shared directly nor indirectly.
The Process of Drug Injection: Applying Ethnography to the Study of HIV Risk Among IDU's.
99 injection
1996) offer excellent examples of the role of participant observation in more clearly defining the set of HIV risk behaviors surrounding injection drug use.
In most earlier research on injection drug users (IDUs) and HIV risk, the primary risk factor was viewed in terms of direct needle sharing.
(1996) used these additional distinctions in sharing to look at the prevalence of injection-related HIV risk behaviors among several subpopulations of injection drug users (see Figure 6).
A major component of the study was the comparison of IDUs who engaged in both direct sharing and indirect sharing with IDUs who engaged in indirect sharing only and those who several subpopulations of injection drug users (see Figure 6).
The Process of Drug Injection: Applying Ethnography to the Study of HIV Risk Among IDU's.
The Prevalence of Additional InjectionRelation HIV Risk Behaviors Among Injection Drug Users.
99 Mead
A good example of this tension between the stated early concerns for the methods of science and the actual use of such methods in ethnography comes from correspondence between Boas and his student Margaret Mead during her first fieldwork in Samoa.
As Orans (1996) describes it, Mead wrote to Boas with her concerns about possible violations of scientific principles in the data she had collected to that point.
The constraints of field research may lead one to stray from the idealized prescriptions of a research design, but Mead was attempting to exert her authority without necessarily following the research procedures advocated by Boas and others.
What is most surprising is Boas's response to Mead.
Boas's final sentence in his response to Mead illustrates that even at this early stage the issue of the subjectivity of ethnographic research was of concern.
Mead's position on these various elements of research design provided fuel for the continuing discussions about the validity of her original findings (Brim and Spain 1974;
The next section shows how concern for the elimination of potential errors and bias through design and attention to methodological detail applies to discussions about the findings of Margaret Mead and Derek Freeman in Samoa.
Mead Versus Freeman: Research Design as Mediator
Derek Freeman's (1983) criticism of Margaret Mead's work and her findings in Samoa has led to reactions from anthropologists who come from different epistemological traditions.
Some have defended Mead (Shankman 1996);
The criticisms and counter-criticisms are difficult to assess, given the time between Mead's and Freeman's studies, the differences in locations of their work, and the differences in their ideological positions (Ember 1985).
Freeman contended that some of Mead's informants lied to her and that Mead's commitment to a particular ideological position caused her to evaluate evidence incorrectly.
We certainly cannot hold Mead to the design standards available today.
Mead used what can be referred to as a static group comparison design with a conjectural treatment group.
Brim and Spain (1974) recognized several problems in the design that could have affected Mead's ability to draw valid conclusions.
In her use of a conjectural treatment group, Mead assumed some things about American adolescents without collecting comparable data.
Mead relied mostly on herself as an instrument to measure the variables of interest.
In lieu of the between-culture comparisons, Mead could have made a within-case comparison that would have suffered less from problems with possible sources of error.
For example, Mead could have compared girls living in the households of native pastors to those who did not.
Some of his experiences were incongruent with Mead's descriptions.
Orans (1996) reanalyzed Mead's field notes and correspondence and once again found that her depiction of Samoa as a halcyon society was at odds with his own impression of Samoa as much more agonistic.
If Samoa was not, during Mead's day, a halcyon society, then her conclusions might have been flawed.
Orans's work was, of course, many years after Mead's, and he worked at different field sites than did Mead.
But, in common with Brim and Spain, Orans found itemized problems with Mead's research design.
There was a lack of comparisons between various sources of data that were crucial to Mead's argument.
For example, Mead made an assertion concerning the relationship between the size of residential units and adolescent troubles.
Similar to the observation by Brim and Spain, Orans points out that Mead made no comparison of sexual behavior between girls living in a native pastor's household and girls living with their own family.
Mead makes assertions about the rarity of events without any knowledge of the frequency distribution of all such events.
For Brim and Spain, and for Orans, Mead's research design limited her ability to draw the conclusions she did.
Evidence and Science in Ethnography: Reflections on the Mead-Freeman Controversy.
Margaret Mead and Samoa: The Making and Unmaking of an Anthropological Myth.
Not Even Wrong: Margaret Mead, Derek Freeman, and the Samoans.
The History of Samoan Sexual Conduct and the Mead-Freeman Controversy.
99 methodological
Design, on the other hand, involves the methodological and analytical details that contribute to the credibility, validity, believability, or plausibility of any study.
Some early exceptions include Brim and Spain's (1974) book on hypothesis-testing designs, Pelto and Pelto's (1978) book on research methodology in cultural anthropology, and Naroll and Cohen's (1973) A Handbook of Method in Cultural Anthropology, which has several chapters that address issues in research design (LeVine 1973;
Malinowski was also concerned with the aims of science and with methodological rigor.
His earliest contributions, however, were more a demonstration of the value of ethnographic writing-his "unusual literary sense" (Lowie 1937:231)--rather than of methodological details of proper ethnographic fieldwork (Ellen 1984).
Ironically, this concern for context and meaning over methodological rigor, particularly for those in search of theoretical foundations (that is, the Boasian idea of data leading to the construction of theory), would ultimately hinder the comparability of data from different ethnographic sources (see Moran [1995] for a recent discussion of this issue and see Ember and Ember, this volume).
In Britain the claims that anthropology not only studied a distinctive body of data but also that it possessed a sophisticated methodology to collect these data, was an important factor in the establishment of anthropology as a discipline.
But in spite of claims to scientific methodology, particularly in the British tradition, there are surprisingly few details about actual methods anthropologists used in the field, beyond a few first principles and illustrative anecdotes.
Even in the late 1960s, when concern for methodological rigor was probably at its peak in anthropology, many treatments of research methods and design in the literature played down the need for more systematic methods and design detail, particularly with respect to hypothesis-testing approaches (LeVine 1973).
This leads to designs that involve concern for a higher degree of methodological and analytical detail, whether quantitative or qualitative.
Phenomenology, hermeneutics, symbolic anthropology, interpretive anthropology, interpretive interactionism, deconstructionism, postmodernism, and constructivism, to name a few, question, in one way or another, some or all of the ontology, epistemology, and methodology of systematic approaches.
Although some of the older interpretive strategies that emerged from the scientific tradition in the social sciences, such as early interpretive anthropology, still adhered to some logical empiricist methodology and maintained a degree of belief in ethnographic authority, more recent approaches, such as postmodernism and constructivism, are more radical in their sweeping rejection of scientific method and design logic (see Schwandt 1994).
In a short methodology section, she discusses the challenge of conducting participant observation research in this rather complex, geographically dispersed, cross-cultural setting.
For some, interpretive work is an exploratory enterprise with an implicit concern for methodological issues.
Rather it creates an open forum that can contribute much to important theoretical and methodological debates.
Without such attention to good design and methodological detail, researchers leave themselves open to one of the worst criticisms of all-of being "not even wrong" (Orans 1996).
In other words, a lack of design and methodological detail makes it next to impossible to fairly and adequately assess the validity of any study's conclusions such that "rightness" or "wrongness" may not even be debatable.
The next section shows how concern for the elimination of potential errors and bias through design and attention to methodological detail applies to discussions about the findings of Margaret Mead and Derek Freeman in Samoa.
In Methodology and Epistomology for Social Science: Selected Papers E. S. Overman, ed.
99 problem
According to Pelto and Pelto (1978:291): "Research design involves combining the essentials of investigation into an effective problem-solving sequence.
A good understanding of the research problem and the research site allows us to plan for some contingencies, but there is no research design crystal ball.
Interpretive, hermeneutic, and postmodern approaches make little explicit reference to ethnographic design issues, but well-written examples from ethnography may provide "moral evidence" to deal with current social problems, moving people (including politicians) in ways that numerical facts can't (Seidman 1994:134).
others may require a description of the research problem and site but require less detail about the methods of data collection and analysis.
This case led to a systematic test of a technology that might ameliorate the problem.
LeVine (1973) and others (Johnson 1990) make the point that the nature of fieldwork, in terms of its requisite huge investments in time and geographical focus, has often limited the attractiveness of more formal research designs because of its commitment to studying specific problems in a specific way.
The realities of fieldwork often dictate the need to change the problem focus or, finding that the proposed hypotheses are inappropriate to the cultural setting under study, the need to somehow salvage the research.
Laboratory and survey researchers have some flexibility to change the problem focus and study populations in light of emerging problems, but field workers are limited in their ability to do so.
Thus, the idea of researchers "putting all their eggs in one basket" may have limited the a priori formulation of problems in fieldwork (LeVine 1973:184).
These approaches are by no means mutually exclusive in approaching a research problem (see section on Research Design in Systematic Research, below).
For any given research problem, it is the purpose of research design to ward off as many threats to validity as possible.
She also discusses the rationales for selecting the site and the group she studied, problems of working in a linguistically and technically diverse social milieu, the use of semistructured and unstructured interviews, and the effect of her role as ethnographer on informant relations and data quality.
Designs of this type lack direct control over independent variables and, thus, have more potential problems with various types of internal validity and with the ability to assess time order effects and causality.
Similarly, threats to external validity, such as problems stemming from biased samples or research in atypical or unique settings, can hamper the generalizability of one's findings.
Many probability and nonprobability sampling designs are available for any given research problem.
Each of these approaches has potential problems, and most do not allow for generalizations about a population since they involve elements of unknown error even if the method involves some form of random selection criteria (for example, random selection of locations in which to intercept respondents).
Recent developments in randomization and computer-intensive methods of statistical analysis involve less restrictive assumptions concerning the data (for example, assumption of a random sample from a population or skewed, sparse, or small sample sizes), opening the way for the development of new test statistics particularly suited for the problem at hand (Noreen 1989;
Brim and Spain (1974) recognized several problems in the design that could have affected Mead's ability to draw valid conclusions.
There were possible problems with interaction between selection and the effects of extraneous variables.
In lieu of the between-culture comparisons, Mead could have made a within-case comparison that would have suffered less from problems with possible sources of error.
But, in common with Brim and Spain, Orans found itemized problems with Mead's research design.
Another problem involved the existence of more social stratification in one community than expected, leading to a lower incidence of.
Symptomology and health problems were operationalized using a panel of physicians.
But the researchers' awareness of the problems, combined with the strength of their multiple case-control study design, increases our confidence in their conclusions.
99 quasi
Included are experiments, quasi-experiments, observational study designs, and what I refer to as natural experiments.
What separates quasi-experiments from true experiments is the lack of random assignment of group members.
Finally, natural experiments are similar to quasi-experiments except that the manipulation of independent variables occurs naturally or is unplanned rather than artificial or directed.
Even in quasi-experiments, it's often difficult to manipulate independent variables directly.
Howevert, with careful attention to design and ethnographic context, quasi-experimental and natural experimental designs can be applied to anthropological field settings, particularly in evaluation research and development research.
Johnson and Murray (1997), for example, used a quasi-experimental design to evaluate the use of fish aggregation devices (FADS) in small-scale fisheries development projects.
From a statistical standpoint, designs that don't involve random assignment including quasi-experiments-are considered observational (Cook and Campbell 1979).
It is important, though, to contrast quasi-experiments to what Kleinbaum et al.
Table 1 describes examples from observational and quasi-experimental study designs discussed by Kleinbaum et al.
Tables 2 and 3 give examples of threats to internal and external validity as discussed in Cook and Campbell (1979) for quasi-experimental designs.
In the quasi-experimental case, this means changes between pre- and posttest, but this way of thinking can be expanded to include hypothesized effects dealing with differences, similarities, or associations whether diachronic or synchronic.
Cook and Cambell (1979) detail how each of the quasi-experimental designs in Table 1 are better or worse at dealing with each of the threats to validity that are found in Tables 2 and 3.
Cook and Campbell (1979) discuss two sampling models for increasing external validity in quasi-experiments.
The logic behind this model can be extended beyond the quasi-experimental case to observational studies.
In lieu of equalization through randomization, Young and Garro, through extensive ethnographic background research, produced groups that, although nonequivalent in the quasi-experimental sense, shared similarities with regard to a number of important characteristics.
Quasi-Experimentation: Design and Analysis for Field Settings.
99 risk
1996) offer excellent examples of the role of participant observation in more clearly defining the set of HIV risk behaviors surrounding injection drug use.
In most earlier research on injection drug users (IDUs) and HIV risk, the primary risk factor was viewed in terms of direct needle sharing.
Thus, most large epidemiological studies of IDUs focused mainly on direct sharing behaviors in attempts to understand seroconversion rates and other risk factors.
(1996) used these additional distinctions in sharing to look at the prevalence of injection-related HIV risk behaviors among several subpopulations of injection drug users (see Figure 6).
Statistical tests of group differences provided a greater understanding of the risk factors associated with the different types of behavior.
The Process of Drug Injection: Applying Ethnography to the Study of HIV Risk Among IDU's.
The Prevalence of Additional InjectionRelation HIV Risk Behaviors Among Injection Drug Users.
99 science
One agency may require a detailed description of the proposed project paying attention to the research design logic of science (for example, validity, reliability, hypotheses, etc.
Boas and most of his students advocated a natural science logic in the collection of ethnographic materials and a true concern for the collection of reliable data that could lead to the production of valid theory.
Malinowski was also concerned with the aims of science and with methodological rigor.
A good example of this tension between the stated early concerns for the methods of science and the actual use of such methods in ethnography comes from correspondence between Boas and his student Margaret Mead during her first fieldwork in Samoa.
Further, the huge investment in time and resources limited another important goal of science, that of replication, since an ethnographer couldn't realistically be expected to replicate someone else's work.
Contemporary Design Issues in Cultural Anthropology There is an ongoing debate in cultural anthropology concerning science and its role in contemporary research.
Suffice to say that traditionally, research design and its logic have been associated with science and an underlying belief in objectivity and explanation.
The historical tension between interpretive and scientific approaches in anthropology has given way to an outright rejection by some anthropologists of science and its logic of design.
To say that the research design logic of science has been replaced by something that is recognizable as the research design logic of, say, postmodernism would, I think, be misleading.
The figure distinguishes between strategies within the realm of interpretive studies and those using systematic strategies that have more of the elements of science.
Geertz (like the other anthropologists) is still directing his efforts to reinvent an anthropological science with the help of textual mediations.
A good example of this is Zabusky's (1995) ethnographic study of cooperation in European space science that she admits "took the form of mutual exploration rather than unidirectional examination" (p. 46).
In some social science disciplines, like psychology, the design of research is driven by features of the analysis.
In Methodology and Epistomology for Social Science: Selected Papers E. S. Overman, ed.
Evidence and Science in Ethnography: Reflections on the Mead-Freeman Controversy.
Scientific Anthropology at the National Science Foundation.
In Anthropology Between Science and the Humanities.
In Constructing Knowledge: Authority and Critique in Social Science.
Social Science and Medicine 16:1453-1465.
Launching Europe: An Ethnography of European Cooperation in Space Science.
99 scientific
If research design gets relatively little attention from anthropologists, other social scientists have written volumes about it.
Boas, Malinowski, and Research Design in the Scientific Tradition
Yet, despite his concern for scientific method, Boas was more explicit about his methods of data analysis than about his methods of fieldwork and data collection (Ellen 1984;
As Orans (1996) describes it, Mead wrote to Boas with her concerns about possible violations of scientific principles in the data she had collected to that point.
First, it demonstrates the differences between the stated scientific objectives of ethnographic work as advocated by Boas and the actual practice of ethnographic research.
Thus, while early British and U.S. anthropologists advocated the scientific method in ethnographic research, there is little evidence that they considered appropriate design issues when they actually did the research.
But in spite of claims to scientific methodology, particularly in the British tradition, there are surprisingly few details about actual methods anthropologists used in the field, beyond a few first principles and illustrative anecdotes.
It is a strange paradox in the development of field methods that the scientific study of other cultures has been built upon such a foundation.
The historical tension between interpretive and scientific approaches in anthropology has given way to an outright rejection by some anthropologists of science and its logic of design.
As a result, the assessment of any theory involves research designs more heavily concerned with the means-the research process, rather than simply the way the study was written or argued-since the validity of study results depends on the scientific soundness of the research design.
A variety of names are used in the lexicon of social scientists that can be associated to varying degrees with an interpretive strategy.
Although some of the older interpretive strategies that emerged from the scientific tradition in the social sciences, such as early interpretive anthropology, still adhered to some logical empiricist methodology and maintained a degree of belief in ethnographic authority, more recent approaches, such as postmodernism and constructivism, are more radical in their sweeping rejection of scientific method and design logic (see Schwandt 1994).
One explanation views use tied to congruence between a client's medical beliefs and scientific medical theory: the higher the congruence, the more likely the client will choose a physician's treatment.
Criteria of Social Scientific Knowledge: Interpretation, Prediction, Praxis.
Reclaiming a Scientific Anthropology.
Scientific Anthropology at the National Science Foundation.
Real World Research: A Resource for Social Scientists and PractitionerResearchers.
Scientific American (September):40-12.
99 shared
In lieu of equalization through randomization, Young and Garro, through extensive ethnographic background research, produced groups that, although nonequivalent in the quasi-experimental sense, shared similarities with regard to a number of important characteristics.
There is a body of literature that views the interaction of culture with the individual as so deeply unique and personal as to not be researchable in terms of cultural universals, coherence, or even sharing.
In contrast, Quinn views culture as being shared-that there are cultural models for a variety of domains that are widely held in common, and that these models can be developed from the discourses of cultural members.
Because she was interested in a model that was shared, it was crucial to interview a wide range of couples who, although of the same culture, were not just from one region of the country or of only one ethnicity, religion, or social class.
That is, finding commonality in the face of diversity provides stronger evidence of a shared cultural model (Johnson and Griffith 1996).
Although the model appeared to be widely shared among informants from Quinn's sample and data collected from other studies on marriage, research still has to be designed to test this model across settings and researchers.
In most earlier research on injection drug users (IDUs) and HIV risk, the primary risk factor was viewed in terms of direct needle sharing.
Thus, most large epidemiological studies of IDUs focused mainly on direct sharing behaviors in attempts to understand seroconversion rates and other risk factors.
Based on participant observation among IDUs, Koester (1996) identified nine other behaviors that were outside the realm of the direct sharing of a single syringe by two or more IDUs.
Termed "indirect sharing," these nine behaviors can promote the transmission of HIV among IDUs who, although not sharing needles directly, often share water for mixing of drugs or for rinsing syringes, share drug-mixing containers (cookers and spoons), share cottons for filtering, and share the actual drug solution itself.
(1996) used these additional distinctions in sharing to look at the prevalence of injection-related HIV risk behaviors among several subpopulations of injection drug users (see Figure 6).
A major component of the study was the comparison of IDUs who engaged in both direct sharing and indirect sharing with IDUs who engaged in indirect sharing only and those who several subpopulations of injection drug users (see Figure 6).
A major component of the study was the comparison of IDUs who engaged in both direct sharing and indirect sharing with IDUs who engaged in indirect sharing only and those who neither shared directly nor indirectly.
99 social
Interpretive, hermeneutic, and postmodern approaches make little explicit reference to ethnographic design issues, but well-written examples from ethnography may provide "moral evidence" to deal with current social problems, moving people (including politicians) in ways that numerical facts can't (Seidman 1994:134).
The lessons learned from controlled experimentation are applied today to the policy arena where groups are in conflict over resources or because of social inequalities (Johnson and Pollnac 1989;
Thus, policy emerges from interactions between groups of differing political, ideological, social, and economic backgrounds.
If research design gets relatively little attention from anthropologists, other social scientists have written volumes about it.
A variety of names are used in the lexicon of social scientists that can be associated to varying degrees with an interpretive strategy.
Although some of the older interpretive strategies that emerged from the scientific tradition in the social sciences, such as early interpretive anthropology, still adhered to some logical empiricist methodology and maintained a degree of belief in ethnographic authority, more recent approaches, such as postmodernism and constructivism, are more radical in their sweeping rejection of scientific method and design logic (see Schwandt 1994).
The core activity is still social description of the other, however modified by new conceptions of discourse, author, or text.
Practitioners of almost all interpretive paradigms are searching in one way or another for some understanding (verstehen) rather than for some explanation of social phenomena.
She contrasts her study with research on cooperation by "experimental" psychologists, emphasizing the cultural and social orientation of her work and the importance of considering context (social, cultural, political, etc.
She also discusses the rationales for selecting the site and the group she studied, problems of working in a linguistically and technically diverse social milieu, the use of semistructured and unstructured interviews, and the effect of her role as ethnographer on informant relations and data quality.
In some social science disciplines, like psychology, the design of research is driven by features of the analysis.
This contrast between explanatory and descriptive or exploratory approaches is commonly made in nonexperimental disciplines in both the natural and social sciences.
Such matters are particularly germane for observational designs using various social network approaches (see Johnson [ 1994] for a review).
Because she was interested in a model that was shared, it was crucial to interview a wide range of couples who, although of the same culture, were not just from one region of the country or of only one ethnicity, religion, or social class.
Beyond these constancies of cultural and marital experience, they were selected to maximize diversity with regard to such obvious differences as their occupations and educational backgrounds, religious affiliations and ethnic and racial identities, their neighborhoods and social networks, and the duration of their marriages.
The ultimate aim of the study was to show the relationship between various social forces and susto susceptibility.
The design involved three communities that differed in history, language, and culture but had similarities in social, demographic, and economic factors.
Another problem involved the existence of more social stratification in one community than expected, leading to a lower incidence of.
Based on earlier ethnographic research, social stress, an important component for understanding an individual's inability to perform social roles, was operationalized using the Social Stress Gauge developed by one of the researchers.
found that there was, in fact, an association between susto and an individual's perception of the adequacy of his or her performance of critical social roles.
The stigma of susto among males and the greater social stratification encountered in one of the communities are possible threats to the validity of their conclusions.
In Methodology and Epistomology for Social Science: Selected Papers E. S. Overman, ed.
Criteria of Social Scientific Knowledge: Interpretation, Prediction, Praxis.
What's Wrong with the Social Sciences?
Social Forces 72(2): 451 X62.
Anthropological Contributions to the Study of Social Networks: A Review.
In Advances in Social Network Analysis.
Development of Social Networks in Preschool Children.
Representations are Social Facts: Modernity and Post-Modernity in Anthropology.
Real World Research: A Resource for Social Scientists and PractitionerResearchers.
The Postmodern Turn: New Perspectives on Social Theory.
Constructing Social Theories.
In Constructing Knowledge: Authority and Critique in Social Science.
Social Science and Medicine 16:1453-1465.
99 strategies
Research Design and Research Strategies
First, research design involves an a priori plan or strategy for all phases of the research (such as data collection and analysis) including, for some researchers, the production of the final product (like an ethnography).
I use "defensible" in addition to "valid," which I normally use, to make readers aware that I am broadening the traditional application of research design to include the variety of research strategies found in anthropology today.
The idea that quantification detracts from context and meaning in the ethnographic endeavor-evident even in the time of Boas-and a failure to understand that systematic methods-whether quantitative or qualitative-help minimize the subjectivity of the investigator have impeded the development of well-delineated research strategies in anthropology.
A more appropriate term that would encompass the diversity currently found in cultural anthropology might be "research strategy.
Figure I is a taxonomic characterization of the different types of research strategies found in contemporary cultural anthropology.
The figure distinguishes between strategies within the realm of interpretive studies and those using systematic strategies that have more of the elements of science.
Under the systematic distinction are the two primary categories of exploratory and explanatory approaches, each entailing a specific design strategy.
In its most extreme form, systematic strategies tend to involve the search for explanations of phenomena and the pursuit of theoretical foundations.
Types of anthropological research strategies and their features.
Interpretive strategies, on the other hand, differ from systematic approaches in that they question a researcher's ability to maintain objectivity, particularly in the ethnographic context where the ethnographer is often the instrument of measurement.
A variety of names are used in the lexicon of social scientists that can be associated to varying degrees with an interpretive strategy.
Although some of the older interpretive strategies that emerged from the scientific tradition in the social sciences, such as early interpretive anthropology, still adhered to some logical empiricist methodology and maintained a degree of belief in ethnographic authority, more recent approaches, such as postmodernism and constructivism, are more radical in their sweeping rejection of scientific method and design logic (see Schwandt 1994).
However, some interpretive work is more similar in nature to the exploratory or descriptive strategies found under the systematic side of Figure 1 than to some of the more radical forays into, for example, postmodernism.
Thus, the rather simple characterization of research strategies found in Figure 1 attempts to recognize the variation inherent in the range of work found in contemporary anthropology by placing "interpretive anthropology" adjacent to "exploratory/descriptive" (see, for example, the work of Zabusky 1995).
Further, the purpose of research strategies under these interpretive paradigms is more focused on the production of a believable or plausible account or story rather than a single depiction of the truth, since it is thought that there are a multitude of plausible accounts rather than just a single true story.
Thus, there is less focus on the means of research, such as methods of data collection and analysis as found in the systematic strategies, and more on the ends of research-the ethnographic or literary product.
More extreme forays into experimental ethnography have blurred this distinction even further, and there is more of a focus on writing strategies that include such approaches as montages, evocative representations, polyvocal texts, and even ethnographic fictions (Denzin and Lincoln 1994).
Instead, Ramos emphasizes the emergent and reflexive nature of data and the literary strategies used in producing the ethnographic product.
These examples offer only a brief glimpse of the range of possible strategies in use by interpretivists in anthropology.
For others, interpretive work is concerned more with the strategies and methods of ethnographic presentation and with the reflexive character of the ethnographic enterprise.
For further discussion of research strategies in the interpretive mode, see Fernandez and Herzfeld (this volume).
Although this process might be equated to a method, it's better to think of ethnography as a strategy in which a variety of methods can be used in the quest for knowledge (Pelto and Pelto 1978).
The levels at which theoretical concepts are measured (for example, nominal or ordinal), the types of sampling strategies used, and the application of appropriate types of analysis must all be considered as a part of the design.
Systematic Research Strategies
This review of research design and strategies in cultural anthropology only scratches the surface of the research designs, hybrid designs, and combinations of designs possible within an ethnographic context.
The strength of the ethnographic approach is its ability to incorporate a wide range of methods, strategies, and designs within a single enterprise, all combining in ways to improve the chances for credible results.
The Discovery of Grounded Theory: Strategies for Qualitative Research.
99 systematic
In such worlds, a systematic argument enjoys a star-spangled legitimacy.
In Agar's statement above, we get the impression that a credible argument should be systematic and based on a process that informs us about how researchers came to know what they know.
This case led to a systematic test of a technology that might ameliorate the problem.
There appears to be a perception that a systematic treatment of the data will have to be abandoned to preserve context and meaning.
The idea that quantification detracts from context and meaning in the ethnographic endeavor-evident even in the time of Boas-and a failure to understand that systematic methods-whether quantitative or qualitative-help minimize the subjectivity of the investigator have impeded the development of well-delineated research strategies in anthropology.
Even in the late 1960s, when concern for methodological rigor was probably at its peak in anthropology, many treatments of research methods and design in the literature played down the need for more systematic methods and design detail, particularly with respect to hypothesis-testing approaches (LeVine 1973).
And since I have indicated here that research in culture involves a great deal of unique personal experience for the anthropologist, I have taken the position that it is probably unlikely there can be a rigorous, systematic, and formal presentation of methods in the study of culture like those of the natural sciences and that there are overriding concerns among many sociologists, psychologists, and economists.
The figure distinguishes between strategies within the realm of interpretive studies and those using systematic strategies that have more of the elements of science.
Under the systematic distinction are the two primary categories of exploratory and explanatory approaches, each entailing a specific design strategy.
These approaches are by no means mutually exclusive in approaching a research problem (see section on Research Design in Systematic Research, below).
In its most extreme form, systematic strategies tend to involve the search for explanations of phenomena and the pursuit of theoretical foundations.
Epistemologically, systematic work is objectivist.
Interpretive strategies, on the other hand, differ from systematic approaches in that they question a researcher's ability to maintain objectivity, particularly in the ethnographic context where the ethnographer is often the instrument of measurement.
Phenomenology, hermeneutics, symbolic anthropology, interpretive anthropology, interpretive interactionism, deconstructionism, postmodernism, and constructivism, to name a few, question, in one way or another, some or all of the ontology, epistemology, and methodology of systematic approaches.
However, some interpretive work is more similar in nature to the exploratory or descriptive strategies found under the systematic side of Figure 1 than to some of the more radical forays into, for example, postmodernism.
Thus, there is less focus on the means of research, such as methods of data collection and analysis as found in the systematic strategies, and more on the ends of research-the ethnographic or literary product.
While systematic analytical paradigms are primarily concerned with threats to validity, recent interpretive paradigms are focused more on threats to believability -as in "Do you believe my story?
As stated, many interpretive studies are closer in character to exploratory and descriptive research in the systematic mode than to some of the more extreme postmodern studies.
In contrast to Zabusky, there is a body of interpretive work in anthropology that is more extreme in its rejection of systematic design issues.
Thus, there is little discussion of research design and methods of data collection as might be found in work in the systematic tradition.
In the following pages, I focus primarily on research designs in systematic research.
Research Design in Systematic Research:
These include systematic sampling, stratified random sampling, cluster sampling, and multistage sampling.
She did not, however, make any systematic comparisons among the different units.
Systematic Research Strategies
Systematic Data Collection.
Systematic Fieldwork, Vol.
99 testing
This case led to a systematic test of a technology that might ameliorate the problem.
Some early exceptions include Brim and Spain's (1974) book on hypothesis-testing designs, Pelto and Pelto's (1978) book on research methodology in cultural anthropology, and Naroll and Cohen's (1973) A Handbook of Method in Cultural Anthropology, which has several chapters that address issues in research design (LeVine 1973;
Even in the late 1960s, when concern for methodological rigor was probably at its peak in anthropology, many treatments of research methods and design in the literature played down the need for more systematic methods and design detail, particularly with respect to hypothesis-testing approaches (LeVine 1973).
Explanatory: Explanatory approaches generally involve testing elements of theory that may already have been proposed in the literature or that have been informed by exploratory research.
This allows for flexibility, multiple tests of a theory, increased chances for various types of validity, triangulation, and the potential for high levels of innovation and creativity.
Currently, the qualitative analysis of text and discourse is no longer: restricted to either interpretive or exploratory approaches, but can also be used in hypothesis testing and explanatory research.
Community ecologists, for example, similarly distinguish between exploratory or descriptive studies that seek to describe and determine patterns in ecological data and those studies that specifically seek to predict or test hypotheses.
Random sampling meets the restrictions of some statistical tests, but it does not afford the same protection as does random assignment of group members against the potential effects of extraneous factors.
In anthropological fieldwork, these designs and others can be used in tandem to test or explore components of a theory (such as combinations of time series and repeated measures designs particularly applicable to long-term fieldwork).
For example, the pretest/posttest nonequivalent groups design controls for some internal threats to validity, but it's problematic with respect to controlling for changes due to how groups members were selected (selection maturation), changes due to how individuals were tested (instrumentation), changes due to the selection of individuals with extreme pretest measures leading to regression toward the mean (regression), and changes due to local events not a part of the study (history).
This is essential if we are to generalize to a whole population and is generally, though not always, a requirement for classical statistical tests.
Thus, testing for a treatment effect across a wide range of classes in the set of all possible classes (including both extremes and the modal class) in the population allows the researcher to say something about how the effect holds in a variety of settings.
In that case, much of the bias in the sample is a matter of the logic used in the original selection of sample seeds and any statistical analysis of the data must be concerned about violations of assumptions for the particular statistical test to be employed (for example, independence of observations or random sample from a population).
If you don't use random sampling, pay careful consideration to possible violations of assumptions for a given statistical test.
Recent developments in randomization and computer-intensive methods of statistical analysis involve less restrictive assumptions concerning the data (for example, assumption of a random sample from a population or skewed, sparse, or small sample sizes), opening the way for the development of new test statistics particularly suited for the problem at hand (Noreen 1989;
The comparison group, Samoan adolescent girls, was compared to a conjectural treatment group, American adolescent girls, to test the proposition that exposure to Western civilization increases adolescent trauma.
She could then have tested the proposition that exposure to competing standards of sexual morality leads to higher levels of emotional distress in adolescents.
Nevertheless, field experiments can be quite informative and, if carefully constructed, can provide formal tests of hypotheses derived from and complementary with ethnography.
Melbin conducted four tests of the feature relating to helpfulness and sociability.
" However, Melbin's three other tests supported the hypothesis of more sociability and helpfulness at night.
This example illustrates nicely the importance of not relying on a single test, but having multiple tests and measures (Stinchcombe 1987).
This example also shows how readily multiple tests can be incorporated into a research design within a field setting.
One of the primary purposes of the research design was the elimination of competing hypotheses-the hallmark of good research design-and the testing of the primary hypothesis is an example of descriptive inference, as opposed to statistical inference.
Young and Garro tested the two main hypotheses in sequence.
Using a standard chi-square test, the authors found a significant difference in the frequency distribution of treatment alternatives between the two towns, with the exception of folk curers.
This established, Young and Garro could then test the second hypothesis relating to the similarity in beliefs between the two communities.
Note that the analysis used to test the hypothesis concerning similarities in beliefs involved descriptive inference, not statistical inference.
This distinction is important, particularly with regard to anthropological research, in that hypothesis-testing research can be done without narrowly restricting it to analytical methods using statistical inference.
Using statistical and descriptive inference, the authors concluded that whether informants use form or function for classification depends on the knowledge base of the informants and the methods used to test their knowledge (see Figure 4).
Often the primary objective of research design is more a matter of discovery and exploration than the testing of hypotheses.
Although the model appeared to be widely shared among informants from Quinn's sample and data collected from other studies on marriage, research still has to be designed to test this model across settings and researchers.
Statistical tests of group differences provided a greater understanding of the risk factors associated with the different types of behavior.
This is an excellent example of a study design that incorporates within-study replication or multiple tests of a theory.
Multiple tests are always much more convincing than a single test (Stinchcombe 1987).
She made an earnest attempt to control for as many biases as possible and, using the data collected during the survey, conducted statistical tests of the four competing models.
This study is an example of multimethod ethnography in which there was a combination of exploratory and explanatory approaches-that is, qualitative data and tests of models with data collected using a cross-sectional design.
With advances in computer technology, qualitative data analysis can now be a powerful mode to test theories.
Similarly, advances in computer-intensive methods for testing hypotheses have the potential to expand the range of designs possible, particularly in the imperfect world of fieldwork (Johnson and Murray 1997).
Research Design in Anthropology: Paradigms and Pragmatics in the Testing of Hypotheses.
Computer-Intensive Methods for Testing Hypotheses.
99 theory
Second, an idealized plan gives guidelines for linking theory to the methods of data collection and analysis that yield either valid or "defensible" results.
Boas and most of his students advocated a natural science logic in the collection of ethnographic materials and a true concern for the collection of reliable data that could lead to the production of valid theory.
Ironically, this concern for context and meaning over methodological rigor, particularly for those in search of theoretical foundations (that is, the Boasian idea of data leading to the construction of theory), would ultimately hinder the comparability of data from different ethnographic sources (see Moran [1995] for a recent discussion of this issue and see Ember and Ember, this volume).
The "my natives" or "my village" mentality of some and the fact that careers were made by discovering new theories or describing exotic less well-known cultures has certainly inhibited replication efforts (Johnson 1990).
In searching for such foundations, there is a need for objectivity, replication, and control over possible sources of error leading to a valid assessment of a given theory.
As a result, the assessment of any theory involves research designs more heavily concerned with the means-the research process, rather than simply the way the study was written or argued-since the validity of study results depends on the scientific soundness of the research design.
This is similar to the grounded theory ideas of Glaser and Strauss (1967), where exploratory descriptive research leads to the development of more meaningful theory and measures.
Explanatory: Explanatory approaches generally involve testing elements of theory that may already have been proposed in the literature or that have been informed by exploratory research.
An important implication here is that scholars who follow this line of inquiry are searching for local rationales rather than nomothetic theory or universal foundations and may be more interested in conveying a moral tale of some type rather than a value-free account (Seidman 1994).
" (Tyler 1991:85)-or, in critical theory, threats to trustworthiness (Kincheloe and McLaren 1994).
This allows for flexibility, multiple tests of a theory, increased chances for various types of validity, triangulation, and the potential for high levels of innovation and creativity.
As with research in community ecology, ethnographic research can be purely exploratory or descriptive involving a research process focused on producing better theory-or purely explanatory, although this is usually not the case.
There is no substitute for a good theory, and there is a critical need to link theory, design, data collection, analysis, and interpretation in a coherent fashion.
Figure 2 shows that the research process involves a simultaneous concern for the development of empirical statements from theory (for example, hypotheses), the operationalization of theoretical concepts (for example, meaningful and reliable measures), design (for example, groups to be studied), data collection (for example, qualitative versus quantitative), and data analysis (for example, multiple regression and text analysis).
(Stinchcombe [1987] provides an excellent discussion of how empirical statements are derived from theory.
It involves constructing a logical plan that links all the elements of research together so as to produce the most valid assessment possible of some theory, given some set of realistic constraints (for example, cost, scope, geographical setting, etc.
In anthropological fieldwork, these designs and others can be used in tandem to test or explore components of a theory (such as combinations of time series and repeated measures designs particularly applicable to long-term fieldwork).
There is a vast literature on sampling theory and random sampling procedures, including discussions of sample sizes (see, for example, Bernard [1994] for a summary and Babbie [1990] for detailed discussion of sampling issues).
But it is critical to remember the connection between theory, design (including sampling), and data analysis from the beginning, because how the data were collected, both in terms of measurement and sampling, is directly related to how they can be analyzed.
One explanation views use tied to congruence between a client's medical beliefs and scientific medical theory: the higher the congruence, the more likely the client will choose a physician's treatment.
Although such designs are less driven by an established theoretical framework, there still is need to pay careful attention to a number of design details in the proper development of new theories and models.
This is an excellent example of a study design that incorporates within-study replication or multiple tests of a theory.
With advances in computer technology, qualitative data analysis can now be a powerful mode to test theories.
The Discovery of Grounded Theory: Strategies for Qualitative Research.
Evaluating FAD Effectiveness in Development Projects: Theory and Praxis.
Rethinking Critical Theory and Qualitative Research.
The History of Ethnological Theory.
The Postmodern Turn: New Perspectives on Social Theory.
Constructing Social Theories.
99 threats
Media campaigns in the U.S. showing pictures of dolphins being caught in nets (generally not in U.S. waters), contributed to Florida's totally banning fishing nets-even though no marine mammals were threatened by the use of nets in Florida waters.
Wildlife conservationists petitioned the U.S. federal government in 1991 to declare harbor porpoises a threatened species.
For any given research problem, it is the purpose of research design to ward off as many threats to validity as possible.
Research designs in this mode are determined a priori and their primary purpose is to eliminate threats to validity, where validity is concerned with whether things are what they appear to be or are the best approximation to the truth (Cook and Campbell 1979).
While systematic analytical paradigms are primarily concerned with threats to validity, recent interpretive paradigms are focused more on threats to believability -as in "Do you believe my story?
" (Tyler 1991:85)-or, in critical theory, threats to trustworthiness (Kincheloe and McLaren 1994).
Although Zabusky doesn't talk specifically about design or about concerns for potential threats to validity, there is implicit concern for such issues throughout the ethnography.
The purpose of research design is to ward off as many threats to validity as possible and to help one eliminate competing hypotheses.
Outside the laboratory, a multitude of influences can threaten the validity of any conclusions.
These and other sources of error are all potential rival hypotheses and randomized experiments are best at eliminating the threats of rival explanations.
Nonrandom assignment lays an experiment open to validity threats and reduces our ability to make causal inferences.
While the purpose of experimental design is to ward off as threats to validity, there are several types of validity-face, construct, statistical conclusion, internal, external, etc.
Here, I stress the importance of thinking through how validity threats have influenced and will influence observations or data (for a more in-depth discussion of how these types of validity can impact study conclusions, see Cook and Campbell 1979).
Tables 2 and 3 give examples of threats to internal and external validity as discussed in Cook and Campbell (1979) for quasi-experimental designs.
The threats in Table 2 deal with extraneous factors that may account for the presence or absence of a hypothesized effect (that is, contrast validity with invalidity).
Cook and Cambell (1979) detail how each of the quasi-experimental designs in Table 1 are better or worse at dealing with each of the threats to validity that are found in Tables 2 and 3.
For example, the pretest/posttest nonequivalent groups design controls for some internal threats to validity, but it's problematic with respect to controlling for changes due to how groups members were selected (selection maturation), changes due to how individuals were tested (instrumentation), changes due to the selection of individuals with extreme pretest measures leading to regression toward the mean (regression), and changes due to local events not a part of the study (history).
Each of these threats may hamper a researcher's ability to assess the contribution of a hypothesized effect to any changes observed.
Similarly, threats to external validity, such as problems stemming from biased samples or research in atypical or unique settings, can hamper the generalizability of one's findings.
(1982) offer a similar discussion of the strengths and weaknesses of observational designs in terms of controlling for threats to both internal and external validity.
The selection of units of analysis, whether settings, events, times, households, or people, is important for understanding a variety of internal and external threats to validity, but it is particularly important for increasing external validity.
These examples show how, even for less powerful designs, the interplay of exploratory and explanatory approaches can aid in guarding against threats to validity (Robson 1993).
There are, of course, threats to validity in this study.
In principle, this is similar to Cook and Campbell's (1979) model of deliberate sampling for heterogeneity as one of several means for warding off threats to external validity.
This study is important because of the authors' candor about the potential threats to validity they encountered in conducting the research.
The stigma of susto among males and the greater social stratification encountered in one of the communities are possible threats to the validity of their conclusions.
98 conclusions
Although some may be swayed by the elegance of a well-written essay, for many it's crucial to know something about the author, his or her motivations, experiences, skills, methods of investigation, and so on before passing judgment on the conclusions.
Outside the laboratory, a multitude of influences can threaten the validity of any conclusions.
In other words, a lack of design and methodological detail makes it next to impossible to fairly and adequately assess the validity of any study's conclusions such that "rightness" or "wrongness" may not even be debatable.
While the purpose of experimental design is to ward off as threats to validity, there are several types of validity-face, construct, statistical conclusion, internal, external, etc.
Here, I stress the importance of thinking through how validity threats have influenced and will influence observations or data (for a more in-depth discussion of how these types of validity can impact study conclusions, see Cook and Campbell 1979).
Attention and concern with all the potential sources of error, whether stemming from how the study was designed, how the data were collected (for example, face-to-face interviews or mail-out surveys), or how the data were analyzed (for example, statistical conclusion validity), will help lead to the production of solid evidence.
Brim and Spain (1974) recognized several problems in the design that could have affected Mead's ability to draw valid conclusions.
If Samoa was not, during Mead's day, a halcyon society, then her conclusions might have been flawed.
For Brim and Spain, and for Orans, Mead's research design limited her ability to draw the conclusions she did.
Had Melbin conducted only the key experiment, he may have come to very different conclusions regarding the helpfulness and sociability of night-timers.
This leads us to the conclusion that the substantial variation apparent in the use of a physician's treatment between the two samples, a consequence of differential access to such treatment, occurs without corresponding degrees of variation in resident's attitudes and beliefs about illness.
In Young and Garro's case, a visual inspection of the graphical representations of the data could lead to no other conclusion than that there was little or no difference in beliefs between the two communities (see Figure 3).
But the in-depth ethnographic background research, the particular structure of the hypothesis, and the overwhelming reliability of informant responses make for more confidence in the possible validity of the study's conclusions.
The stigma of susto among males and the greater social stratification encountered in one of the communities are possible threats to the validity of their conclusions.
But the researchers' awareness of the problems, combined with the strength of their multiple case-control study design, increases our confidence in their conclusions.
My conclusions must be interpreted in terms of these general trends.
The study shows how the use of multiple methods fosters triangulation that contributes to the production of valid conclusions (see Figure 8).
Had Goldin relied exclusively on, say, the life histories of a nonprobabilistic sample of informants without specified selection criteria (Johnson 1990), she might have arrived at a very different, and possibly erroneous, conclusion.
The newer forays into experimental and other ethnographic forms of presentation are more reflexive in character and more concerned with believable and moving representations rather than the production of valid accounts or conclusions.
98 Cook
These micro-level approaches are attempts to get at a better understanding of meaning in everyday life (Cook 1994).
Research designs in this mode are determined a priori and their primary purpose is to eliminate threats to validity, where validity is concerned with whether things are what they appear to be or are the best approximation to the truth (Cook and Campbell 1979).
Cook and Campbell (1979) make a similar distinction but refer to these kinds of natural experiments as "passive-observational studies.
From a statistical standpoint, designs that don't involve random assignment including quasi-experiments-are considered observational (Cook and Campbell 1979).
(1982) and Cook and Campbell (1979).
Here, I stress the importance of thinking through how validity threats have influenced and will influence observations or data (for a more in-depth discussion of how these types of validity can impact study conclusions, see Cook and Campbell 1979).
Tables 2 and 3 give examples of threats to internal and external validity as discussed in Cook and Campbell (1979) for quasi-experimental designs.
Cook and Cambell (1979) detail how each of the quasi-experimental designs in Table 1 are better or worse at dealing with each of the threats to validity that are found in Tables 2 and 3.
Cook and Campbell (1979) discuss two sampling models for increasing external validity in quasi-experiments.
In principle, this is similar to Cook and Campbell's (1979) model of deliberate sampling for heterogeneity as one of several means for warding off threats to external validity.
Termed "indirect sharing," these nine behaviors can promote the transmission of HIV among IDUs who, although not sharing needles directly, often share water for mixing of drugs or for rinsing syringes, share drug-mixing containers (cookers and spoons), share cottons for filtering, and share the actual drug solution itself.
Cook, T. D. 1994.
Cook, T. D., and D. T. Campbell.
98 fieldwork
By definition, a plan cannot deal with the unanticipated or unknown realities of research, such as tragedies or acts of nature that disrupt fieldwork.
Yet, despite his concern for scientific method, Boas was more explicit about his methods of data analysis than about his methods of fieldwork and data collection (Ellen 1984;
His earliest contributions, however, were more a demonstration of the value of ethnographic writing-his "unusual literary sense" (Lowie 1937:231)--rather than of methodological details of proper ethnographic fieldwork (Ellen 1984).
A good example of this tension between the stated early concerns for the methods of science and the actual use of such methods in ethnography comes from correspondence between Boas and his student Margaret Mead during her first fieldwork in Samoa.
It's tempting to explain this as the consequence of the intensely personal nature of fieldwork, and the complexity of a holistic approach.
There was a wide belief among British anthropologists that fieldwork could not be taught to new recruits, but could only be experienced by individuals in the field.
I believe that only someone wholly involved and fully immersed in fieldwork can really communicate the essence of cultural anthropology to students or general readers.
LeVine (1973) and others (Johnson 1990) make the point that the nature of fieldwork, in terms of its requisite huge investments in time and geographical focus, has often limited the attractiveness of more formal research designs because of its commitment to studying specific problems in a specific way.
The realities of fieldwork often dictate the need to change the problem focus or, finding that the proposed hypotheses are inappropriate to the cultural setting under study, the need to somehow salvage the research.
Thus, the idea of researchers "putting all their eggs in one basket" may have limited the a priori formulation of problems in fieldwork (LeVine 1973:184).
In natural settings, particularly fieldwork, there is no perfect design that can control for all possible extraneous effects at once.
Designs of this type, however, are often impossible in anthropological fieldwork.
In anthropological fieldwork, these designs and others can be used in tandem to test or explore components of a theory (such as combinations of time series and repeated measures designs particularly applicable to long-term fieldwork).
More recently, Martin Orans did fieldwork in Samoa.
One subsample was of individuals who complained of susto during the fieldwork or who had admitted their condition to relatives or curers.
Similarly, advances in computer-intensive methods for testing hypotheses have the potential to expand the range of designs possible, particularly in the imperfect world of fieldwork (Johnson and Murray 1997).
Systematic Fieldwork, Vol.
98 statistical
Some environmental groups were still concerned that evidence with more statistical power was needed.
I am very decidedly of the opinion that a statistical treatment of such intricate behavior as the one that you are studying, will not have very much meaning and that the characterization of a selected number of cases must necessarily be the material with which you operate.
Statistical work will require the tearing out of its natural setting, some particular aspects of behavior which, without that setting may have no meaning whatever. A complete elimination of the subjective use of the investigator is of course quite impossible in a matter of this kind but undoubtedly you will try to overcome this so far as that is all possible.
Second, Boas's concern for contextual meaning over the statistical analysis of data was prophetic.
"Statistical analysis and interpretation," he says, "are the least critical aspects of experimentation, in that if purely statistical or interpretive errors are made, the data can be reanalyzed.
Random sampling meets the restrictions of some statistical tests, but it does not afford the same protection as does random assignment of group members against the potential effects of extraneous factors.
From a statistical standpoint, designs that don't involve random assignment including quasi-experiments-are considered observational (Cook and Campbell 1979).
While the purpose of experimental design is to ward off as threats to validity, there are several types of validity-face, construct, statistical conclusion, internal, external, etc.
Attention and concern with all the potential sources of error, whether stemming from how the study was designed, how the data were collected (for example, face-to-face interviews or mail-out surveys), or how the data were analyzed (for example, statistical conclusion validity), will help lead to the production of solid evidence.
This is essential if we are to generalize to a whole population and is generally, though not always, a requirement for classical statistical tests.
In that case, much of the bias in the sample is a matter of the logic used in the original selection of sample seeds and any statistical analysis of the data must be concerned about violations of assumptions for the particular statistical test to be employed (for example, independence of observations or random sample from a population).
If you don't use random sampling, pay careful consideration to possible violations of assumptions for a given statistical test.
Recent developments in randomization and computer-intensive methods of statistical analysis involve less restrictive assumptions concerning the data (for example, assumption of a random sample from a population or skewed, sparse, or small sample sizes), opening the way for the development of new test statistics particularly suited for the problem at hand (Noreen 1989;
One of the primary purposes of the research design was the elimination of competing hypotheses-the hallmark of good research design-and the testing of the primary hypothesis is an example of descriptive inference, as opposed to statistical inference.
Ironically, in statistical terms, the authors have more interest in the null hypothesis of no difference in beliefs than in the alternative hypothesis of a difference in beliefs between the two communities.
Note that the analysis used to test the hypothesis concerning similarities in beliefs involved descriptive inference, not statistical inference.
Despite the authors' claims of finding no "significant difference," there was no real way, at least when the study was conducted, to assess the extent to which any differences were significant in the sense of statistical probability.
Recent developments in statistical procedures allow us to assess the similarities in aggregated judged-similarity matrices between the two communities (see Handwerker and Borgatti, this volume, and Hubert 1987).
This distinction is important, particularly with regard to anthropological research, in that hypothesis-testing research can be done without narrowly restricting it to analytical methods using statistical inference.
Boster and Johnson used statistical and graphical methods to evaluate whether experts' and novices' judgments of fish, at the aggregate and individual levels, were closer to the morphological characteristics of fish (taxonomic distance) or the uses of fish (beliefs about use).
Using statistical and descriptive inference, the authors concluded that whether informants use form or function for classification depends on the knowledge base of the informants and the methods used to test their knowledge (see Figure 4).
Statistical tests of group differences provided a greater understanding of the risk factors associated with the different types of behavior.
Using standard methods of statistical inference, Rubel et al.
She made an earnest attempt to control for as many biases as possible and, using the data collected during the survey, conducted statistical tests of the four competing models.
Using path analytic modeling, she applied different statistical controls in each of the competing models.
97 Boas
I don't think we should make too much of it because the important elements of research design-reliability, informant accuracy, validity, objectivity, and operationalization of theoretical concepts-have been present in the writings of cultural anthropologists even before Boas.
Boas, Malinowski, and Research Design in the Scientific Tradition
Boas and most of his students advocated a natural science logic in the collection of ethnographic materials and a true concern for the collection of reliable data that could lead to the production of valid theory.
Yet, despite his concern for scientific method, Boas was more explicit about his methods of data analysis than about his methods of fieldwork and data collection (Ellen 1984;
Boas 1920).
A good example of this tension between the stated early concerns for the methods of science and the actual use of such methods in ethnography comes from correspondence between Boas and his student Margaret Mead during her first fieldwork in Samoa.
As Orans (1996) describes it, Mead wrote to Boas with her concerns about possible violations of scientific principles in the data she had collected to that point.
She had concerns-and I believe she thought her mentor, Boas, would feel similarly-as to whether a valid comparison of this type could be made given the selection process for her sample of girls.
The constraints of field research may lead one to stray from the idealized prescriptions of a research design, but Mead was attempting to exert her authority without necessarily following the research procedures advocated by Boas and others.
What is most surprising is Boas's response to Mead.
First, it demonstrates the differences between the stated scientific objectives of ethnographic work as advocated by Boas and the actual practice of ethnographic research.
Second, Boas's concern for contextual meaning over the statistical analysis of data was prophetic.
The idea that quantification detracts from context and meaning in the ethnographic endeavor-evident even in the time of Boas-and a failure to understand that systematic methods-whether quantitative or qualitative-help minimize the subjectivity of the investigator have impeded the development of well-delineated research strategies in anthropology.
Boas's final sentence in his response to Mead illustrates that even at this early stage the issue of the subjectivity of ethnographic research was of concern.
Boas, F. 1920.
97 Campbell
Research designs in this mode are determined a priori and their primary purpose is to eliminate threats to validity, where validity is concerned with whether things are what they appear to be or are the best approximation to the truth (Cook and Campbell 1979).
Cook and Campbell (1979) make a similar distinction but refer to these kinds of natural experiments as "passive-observational studies.
From a statistical standpoint, designs that don't involve random assignment including quasi-experiments-are considered observational (Cook and Campbell 1979).
(1982) and Cook and Campbell (1979).
Discussing "common sense knowing" in evaluation research, Campbell (1988) gives an important critique of ethnography.
Here, I stress the importance of thinking through how validity threats have influenced and will influence observations or data (for a more in-depth discussion of how these types of validity can impact study conclusions, see Cook and Campbell 1979).
Tables 2 and 3 give examples of threats to internal and external validity as discussed in Cook and Campbell (1979) for quasi-experimental designs.
Cook and Campbell (1979) discuss two sampling models for increasing external validity in quasi-experiments.
In ethnographic research, no matter the mix of methods, the design of the study should allow for an ethnographers' hypotheses or hunches to be rejected as well as confirmed (Campbell 1975).
In principle, this is similar to Cook and Campbell's (1979) model of deliberate sampling for heterogeneity as one of several means for warding off threats to external validity.
Campbell, D. T. 1975.
Campbell, D. T. 1988.
Cook, T. D., and D. T. Campbell.
97 reliability
One agency may require a detailed description of the proposed project paying attention to the research design logic of science (for example, validity, reliability, hypotheses, etc.
I don't think we should make too much of it because the important elements of research design-reliability, informant accuracy, validity, objectivity, and operationalization of theoretical concepts-have been present in the writings of cultural anthropologists even before Boas.
Boas and most of his students advocated a natural science logic in the collection of ethnographic materials and a true concern for the collection of reliable data that could lead to the production of valid theory.
Although Ramos discusses informant interviewing and various sources of data, her introduction is largely devoted to discussions of her reliance on her own memory in writing the ethnography and the shift in the narrative between synchrony and diachrony.
Exploratory research may contribute to the production of reliable and valid measures, provide information essential for constructing comparison groups, facilitate construction of structured questions or questionnaires, or provide information necessary for producing a sound probability or nonprobability sample.
Figure 2 shows that the research process involves a simultaneous concern for the development of empirical statements from theory (for example, hypotheses), the operationalization of theoretical concepts (for example, meaningful and reliable measures), design (for example, groups to be studied), data collection (for example, qualitative versus quantitative), and data analysis (for example, multiple regression and text analysis).
How reliable are your measures in terms of precision, sensitivity, resolution, and consistency?
But the in-depth ethnographic background research, the particular structure of the hypothesis, and the overwhelming reliability of informant responses make for more confidence in the possible validity of the study's conclusions.
Reliability and Validity in Quantitative Research.
97 tradition
I use "defensible" in addition to "valid," which I normally use, to make readers aware that I am broadening the traditional application of research design to include the variety of research strategies found in anthropology today.
Boas, Malinowski, and Research Design in the Scientific Tradition
But in spite of claims to scientific methodology, particularly in the British tradition, there are surprisingly few details about actual methods anthropologists used in the field, beyond a few first principles and illustrative anecdotes.
In the American tradition texts provided what was regarded as an objective body of data, whereas the British tradition was more a matter of subjective experience.
There is much anecdotal evidence for a belief, across the British and U.S. traditions, in a trial-by-fire method of training for ethnographers.
Suffice to say that traditionally, research design and its logic have been associated with science and an underlying belief in objectivity and explanation.
Although some of the older interpretive strategies that emerged from the scientific tradition in the social sciences, such as early interpretive anthropology, still adhered to some logical empiricist methodology and maintained a degree of belief in ethnographic authority, more recent approaches, such as postmodernism and constructivism, are more radical in their sweeping rejection of scientific method and design logic (see Schwandt 1994).
Following in the "thick description" tradition of Geertz, Zabusky clearly believes in some kind of ethnographic authority.
Thus, there is little discussion of research design and methods of data collection as might be found in work in the systematic tradition.
Its purpose is not a traditional attempt at explanation where facts are considered real, but political interpretation and representation of facts, independent of their "realness.
Thus, traditional methods sections are replaced by discussions on how to read the work or on the particular methods used in writing the ethnography itself (see, for example, Panourgia's discussion on the use of the parerga).
The most common designs used traditionally by anthropologists have been observational in nature.
Derek Freeman's (1983) criticism of Margaret Mead's work and her findings in Samoa has led to reactions from anthropologists who come from different epistemological traditions.
An important element of this position is that traditional medical beliefs are not a barrier to choice of physician treatment.
The research design included the comparison of two Mexican communities that were similar in terms of cultural traditions and economies but varied in terms of access to Western medical services.
96 fishing
The outcry for a ban on nets in tuna fishing is a famous recent example.
Environmental organizations launched campaigns to ban nets in tuna fishing because dolphins are often caught incidentally in that fishery.
Media campaigns in the U.S. showing pictures of dolphins being caught in nets (generally not in U.S. waters), contributed to Florida's totally banning fishing nets-even though no marine mammals were threatened by the use of nets in Florida waters.
There has been similar concern over the incidental catch of harbor porpoises by net fishers in New England (Schneider 1996).
In response, the fishing industry proposed the voluntary use of "pingers" an underwater acoustic device-to keep porpoises from their nets.
Fishers petitioned the federal government to fund a study of pinger effectiveness.
Lobbying efforts by fishers yielded more funds for a larger, more comprehensive study involving more than 10,000 fishing nets.
Thus, fishers were blind as to which nets were control and which were treatment-a classic double-blind experimental design.
Johnson and Murray (1997), for example, used a quasi-experimental design to evaluate the use of fish aggregation devices (FADS) in small-scale fisheries development projects.
Two fixed fishing structures (piers) were pretested for differences in catch rates.
Then, FADS, umbrella-like units suspended in the water column, were alternately placed at the piers and individual fishers were interviewed simultaneously during randomly selected times at both the treatment (the pier with the FADS) and the control (the pier without the FADS) piers.
Boster and Johnson (1989) explored this issue in an ethnobiological study of fish.
This is analogous to treatment and control groups without the random assignment of subjects to experimental units and where the treatment is implied rather than researcher directed (that is, natural differences in experience with fish).
In the comparison, both culture and language were held constant while experience with fish was varied.
To ensure that experts were, in fact, experienced recreational fishermen, the rosters of sport fishing clubs in each region were sampled at random.
The selection of control group subjects, by contrast, involved a purposeful selection procedure in which potential subjects were screened for recreational fishing experience.
Using a questionnaire to gain background information, 15 college undergraduates who had the least amount of recreational fishing experience were selected from two introductory anthropology classes.
Individuals were asked to perform an unconstrained judged similarity of the fish-a free pile sort (see Weller, this volume, and Weller and Romney 1988).
Further, beliefs about the use and functional characteristics of the fish obtained from extensive ethnographic interviews were turned into a sentence-frame completion task described by Weller and Romney (1988).
Finally, a measure of morphological similarities was determined, using taxonomic distances between pairs of fish.
Boster and Johnson used statistical and graphical methods to evaluate whether experts' and novices' judgments of fish, at the aggregate and individual levels, were closer to the morphological characteristics of fish (taxonomic distance) or the uses of fish (beliefs about use).
Form or Function: A Comparison of Expert and Novice Judgments of Similarity Among Fish.
In Fish Aggregation Devices in Developing Fisheries: Potential and Pitfalls.
96 write
Although some may be swayed by the elegance of a well-written essay, for many it's crucial to know something about the author, his or her motivations, experiences, skills, methods of investigation, and so on before passing judgment on the conclusions.
Interpretive, hermeneutic, and postmodern approaches make little explicit reference to ethnographic design issues, but well-written examples from ethnography may provide "moral evidence" to deal with current social problems, moving people (including politicians) in ways that numerical facts can't (Seidman 1994:134).
If research design gets relatively little attention from anthropologists, other social scientists have written volumes about it.
I don't think we should make too much of it because the important elements of research design-reliability, informant accuracy, validity, objectivity, and operationalization of theoretical concepts-have been present in the writings of cultural anthropologists even before Boas.
His earliest contributions, however, were more a demonstration of the value of ethnographic writing-his "unusual literary sense" (Lowie 1937:231)--rather than of methodological details of proper ethnographic fieldwork (Ellen 1984).
As Orans (1996) describes it, Mead wrote to Boas with her concerns about possible violations of scientific principles in the data she had collected to that point.
She wrote of her doubts about the comparability of cases and about her ability, or even the need, to do a quantitative comparison of the similarity of attitudes among the adolescent girls in her study.
He writes:
Williams writes:
As a result, the assessment of any theory involves research designs more heavily concerned with the means-the research process, rather than simply the way the study was written or argued-since the validity of study results depends on the scientific soundness of the research design.
In this line of thinking, the researcher is a field-worker-as-writer.
In contrasting Geertz and early interpretive anthropology with some of the later postmodern turns of such ethnographic writers as James Clifford, Rabinow (1986) observes:
In contrast to the field-worker-as-writer, we find the writer-as-field-worker (Denzin and Lincoln 1994).
For scholars like Geertz, analysis of ethnography has less to do with the methods of observation and description than the inscriptions and writings concerning the meaning of human action.
More extreme forays into experimental ethnography have blurred this distinction even further, and there is more of a focus on writing strategies that include such approaches as montages, evocative representations, polyvocal texts, and even ethnographic fictions (Denzin and Lincoln 1994).
Although Ramos discusses informant interviewing and various sources of data, her introduction is largely devoted to discussions of her reliance on her own memory in writing the ethnography and the shift in the narrative between synchrony and diachrony.
Thus, traditional methods sections are replaced by discussions on how to read the work or on the particular methods used in writing the ethnography itself (see, for example, Panourgia's discussion on the use of the parerga).
No amount of sophisticated statistics, computer intensive text analysis, or elegant writing can salvage a poorly designed study.
In Writing Culture: The Poetics and Politics of Ethnogrophy.
Tales of the Field: On Writing Ethnography.
94 observational
Included are experiments, quasi-experiments, observational study designs, and what I refer to as natural experiments.
Observational studies involve neither random assignment of members to comparison groups nor the manipulation by the observer of independent variables.
This distinction between experimental and observational approaches is similar to one in ecological field studies.
Mensurative designs, then, are observational and characteristic of the types of comparative designs found in field studies in anthropology.
Cook and Campbell (1979) make a similar distinction but refer to these kinds of natural experiments as "passive-observational studies.
From a statistical standpoint, designs that don't involve random assignment including quasi-experiments-are considered observational (Cook and Campbell 1979).
(1982) refer to as observational studies.
The most common designs used traditionally by anthropologists have been observational in nature.
Due to their predominance in anthropology, the examples that follow are comparative observational designs.
Table 1 describes examples from observational and quasi-experimental study designs discussed by Kleinbaum et al.
(1982) offer a similar discussion of the strengths and weaknesses of observational designs in terms of controlling for threats to both internal and external validity.
The logic behind this model can be extended beyond the quasi-experimental case to observational studies.
Such matters are particularly germane for observational designs using various social network approaches (see Johnson [ 1994] for a review).
93 variables
In each successive study, investigators tried to control for as many extraneous variables as possible so that the hypothesized effect could be assessed (that is, the effectiveness of pingers compared to not using pingers).
True experiments involve random assignment and afford the best chances for controlling for things like: the effects of extraneous factors (that is, unmeasured variables that might affect the dependent variable);
Random allocation produces equivalent comparison groups, and artificial manipulation of independent variables (also known as explanatory variables or study factors), with all other variables or factors controlled for, allows for the most valid assessment of the causal relationship between the independent and dependent variables or response variables.
Random assignment maximizes the probability that experimental groups are equivalent on key variables prior to the introduction of an intervention.
Observational studies involve neither random assignment of members to comparison groups nor the manipulation by the observer of independent variables.
These are basically true experiments involving random assignment, multiple comparisons (for example, treatment versus control), and the manipulation of independent variables.
He refers to the second as mensurative experiments, which involve simply the measurement of variables in space and time and among a number of comparison groups, without random allocation and the manipulation of experimental factors.
While random assignment aids in controlling for confounding variables by producing homogeneous comparative groups, random sampling of units produces comparison groups that are representative of such groups.
Finally, natural experiments are similar to quasi-experiments except that the manipulation of independent variables occurs naturally or is unplanned rather than artificial or directed.
Even in quasi-experiments, it's often difficult to manipulate independent variables directly.
Designs of this type lack direct control over independent variables and, thus, have more potential problems with various types of internal validity and with the ability to assess time order effects and causality.
Mead relied mostly on herself as an instrument to measure the variables of interest.
There were possible problems with interaction between selection and the effects of extraneous variables.
She could have chosen comparison groups that were as similar as possible in order to rule out the effects of unmeasured variables as much as possible.
Several examples are reviews of studies that incorporate comparative designs of various types in which nonequivalent groups are constructed in order to control for as many extraneous factors as possible, and the manipulation of independent variables is a function of how comparative groups were chosen.
Because respondents weren't randomly assigned into comparison groups, it's difficult to know the influences of confounding variables on physician utilization and beliefs about illness.
It is unrealistic to suppose that Young and Garro could have randomly assigned community members to the different comparison groups in order to control for confounding variables and then subject their informants to the treatments of interest.
This is a good example of the application of exploratory research in the production of better measures of potentially important explanatory variables.
Her selection of variables allowed a comparison of different levels (for example, Catholic versus Protestant) across the four variables.
The combination helped Goldin in the specification of appropriate variables, in the development of a sound survey instrument, and in the specification and assessment of the four competing models.
92 theoretical
I don't think we should make too much of it because the important elements of research design-reliability, informant accuracy, validity, objectivity, and operationalization of theoretical concepts-have been present in the writings of cultural anthropologists even before Boas.
Ironically, this concern for context and meaning over methodological rigor, particularly for those in search of theoretical foundations (that is, the Boasian idea of data leading to the construction of theory), would ultimately hinder the comparability of data from different ethnographic sources (see Moran [1995] for a recent discussion of this issue and see Ember and Ember, this volume).
I find this stance comfortable, for it is my conviction that so long as prime theoretical concerns in the study of culture are an attempt to record and understand the native's view of his culture and the objective and historical realities of culture, then methods for field study will have to reflect the end purpose of making a whole account of a part of the human experience.
In its most extreme form, systematic strategies tend to involve the search for explanations of phenomena and the pursuit of theoretical foundations.
To her, the original work was "old-fashioned and theoretically unsophisticated" and had to be replaced by a more reflexive work.
Figure 2 shows that the research process involves a simultaneous concern for the development of empirical statements from theory (for example, hypotheses), the operationalization of theoretical concepts (for example, meaningful and reliable measures), design (for example, groups to be studied), data collection (for example, qualitative versus quantitative), and data analysis (for example, multiple regression and text analysis).
Theoretical knowledge is derived either from earlier studies or from exploratory work.
The levels at which theoretical concepts are measured (for example, nominal or ordinal), the types of sampling strategies used, and the application of appropriate types of analysis must all be considered as a part of the design.
For example, the particular structure of an empirical statement or hypothesis will partially determine the manner in which theoretical concepts are operationalized and eventually analyzed.
Rather it creates an open forum that can contribute much to important theoretical and methodological debates.
In one way or another, various study designs, in combination with other considerations such as the operationalization of theoretical constructs and sampling, are better or worse at dealing with each.
Measurement, operationalization of theoretical concepts, and type of analysis used are other important factors.
Implicit in this proposition is the overall theoretical notion that culture is the major factor contributing to human behavior.
Although such designs are less driven by an established theoretical framework, there still is need to pay careful attention to a number of design details in the proper development of new theories and models.
91 findings
In fact, chance factors often lead to great discoveries or unexpected findings.
Mead's position on these various elements of research design provided fuel for the continuing discussions about the validity of her original findings (Brim and Spain 1974;
I find this stance comfortable, for it is my conviction that so long as prime theoretical concerns in the study of culture are an attempt to record and understand the native's view of his culture and the objective and historical realities of culture, then methods for field study will have to reflect the end purpose of making a whole account of a part of the human experience.
The realities of fieldwork often dictate the need to change the problem focus or, finding that the proposed hypotheses are inappropriate to the cultural setting under study, the need to somehow salvage the research.
Its practitioners are ultimately interested in research findings that approximate an external truth.
Epistemologically, interpretive paradigms are subjective, with findings that are value mediated or even created.
In contrast to the field-worker-as-writer, we find the writer-as-field-worker (Denzin and Lincoln 1994).
or interaction effects involving selection (that is, when selection interacts with other factors to create erroneous findings).
External validity is concerned with the approximation to the truth as expanded to other settings-that is, with the generalizability of research findings.
Similarly, threats to external validity, such as problems stemming from biased samples or research in atypical or unique settings, can hamper the generalizability of one's findings.
The next section shows how concern for the elimination of potential errors and bias through design and attention to methodological detail applies to discussions about the findings of Margaret Mead and Derek Freeman in Samoa.
Derek Freeman's (1983) criticism of Margaret Mead's work and her findings in Samoa has led to reactions from anthropologists who come from different epistemological traditions.
According to Melbin, "To find a key is to come across an implied need for help" (p. 75).
Despite the authors' claims of finding no "significant difference," there was no real way, at least when the study was conducted, to assess the extent to which any differences were significant in the sense of statistical probability.
In my view, the consistency of her findings in this diverse sample of informants makes her case stronger (see Johnson 1990).
That is, finding commonality in the face of diversity provides stronger evidence of a shared cultural model (Johnson and Griffith 1996).
These findings are undeniably important for larger epidemiological work that examines elements of IDUs' behaviors and such things as producing valid models of seroconversion.
91 hypothesis
Some early exceptions include Brim and Spain's (1974) book on hypothesis-testing designs, Pelto and Pelto's (1978) book on research methodology in cultural anthropology, and Naroll and Cohen's (1973) A Handbook of Method in Cultural Anthropology, which has several chapters that address issues in research design (LeVine 1973;
Even in the late 1960s, when concern for methodological rigor was probably at its peak in anthropology, many treatments of research methods and design in the literature played down the need for more systematic methods and design detail, particularly with respect to hypothesis-testing approaches (LeVine 1973).
Currently, the qualitative analysis of text and discourse is no longer: restricted to either interpretive or exploratory approaches, but can also be used in hypothesis testing and explanatory research.
For example, the particular structure of an empirical statement or hypothesis will partially determine the manner in which theoretical concepts are operationalized and eventually analyzed.
The hypothesis was that residents of the night would return keys on average more often than those of the day.
" However, Melbin's three other tests supported the hypothesis of more sociability and helpfulness at night.
One of the primary purposes of the research design was the elimination of competing hypotheses-the hallmark of good research design-and the testing of the primary hypothesis is an example of descriptive inference, as opposed to statistical inference.
Termed the "conceptual-incompatibility" hypothesis, a number of studies have suggested that such a congruence was the primary determinant of treatment choice among Third World peoples.
This established, Young and Garro could then test the second hypothesis relating to the similarity in beliefs between the two communities.
Ironically, in statistical terms, the authors have more interest in the null hypothesis of no difference in beliefs than in the alternative hypothesis of a difference in beliefs between the two communities.
On the basis of the data from the triads study and the term-frame interviews, we see little reason to reject the "null hypothesis" of no significant differences between the responses of the two groups of informants.
The authors' careful attention to research design and analytical issues contributed to the production of impressive evidence that casts doubt on the validity of the "conceptual incompatibility" hypothesis.
Note that the analysis used to test the hypothesis concerning similarities in beliefs involved descriptive inference, not statistical inference.
This distinction is important, particularly with regard to anthropological research, in that hypothesis-testing research can be done without narrowly restricting it to analytical methods using statistical inference.
But the in-depth ethnographic background research, the particular structure of the hypothesis, and the overwhelming reliability of informant responses make for more confidence in the possible validity of the study's conclusions.
91 logic
One agency may require a detailed description of the proposed project paying attention to the research design logic of science (for example, validity, reliability, hypotheses, etc.
The logic of the research design contributed to the production of credible results.
Boas and most of his students advocated a natural science logic in the collection of ethnographic materials and a true concern for the collection of reliable data that could lead to the production of valid theory.
Suffice to say that traditionally, research design and its logic have been associated with science and an underlying belief in objectivity and explanation.
The historical tension between interpretive and scientific approaches in anthropology has given way to an outright rejection by some anthropologists of science and its logic of design.
To say that the research design logic of science has been replaced by something that is recognizable as the research design logic of, say, postmodernism would, I think, be misleading.
Although some of the older interpretive strategies that emerged from the scientific tradition in the social sciences, such as early interpretive anthropology, still adhered to some logical empiricist methodology and maintained a degree of belief in ethnographic authority, more recent approaches, such as postmodernism and constructivism, are more radical in their sweeping rejection of scientific method and design logic (see Schwandt 1994).
It involves constructing a logical plan that links all the elements of research together so as to produce the most valid assessment possible of some theory, given some set of realistic constraints (for example, cost, scope, geographical setting, etc.
The logic behind this model can be extended beyond the quasi-experimental case to observational studies.
In that case, much of the bias in the sample is a matter of the logic used in the original selection of sample seeds and any statistical analysis of the data must be concerned about violations of assumptions for the particular statistical test to be employed (for example, independence of observations or random sample from a population).
87 hypotheses
One agency may require a detailed description of the proposed project paying attention to the research design logic of science (for example, validity, reliability, hypotheses, etc.
The realities of fieldwork often dictate the need to change the problem focus or, finding that the proposed hypotheses are inappropriate to the cultural setting under study, the need to somehow salvage the research.
Exploratory: Exploratory approaches are used to develop hypotheses and more generally to make probes for circumscription, description, and interpretation of less well-understood topics.
Community ecologists, for example, similarly distinguish between exploratory or descriptive studies that seek to describe and determine patterns in ecological data and those studies that specifically seek to predict or test hypotheses.
Figure 2 shows that the research process involves a simultaneous concern for the development of empirical statements from theory (for example, hypotheses), the operationalization of theoretical concepts (for example, meaningful and reliable measures), design (for example, groups to be studied), data collection (for example, qualitative versus quantitative), and data analysis (for example, multiple regression and text analysis).
The purpose of research design is to ward off as many threats to validity as possible and to help one eliminate competing hypotheses.
These and other sources of error are all potential rival hypotheses and randomized experiments are best at eliminating the threats of rival explanations.
In ethnographic research, no matter the mix of methods, the design of the study should allow for an ethnographers' hypotheses or hunches to be rejected as well as confirmed (Campbell 1975).
Nevertheless, field experiments can be quite informative and, if carefully constructed, can provide formal tests of hypotheses derived from and complementary with ethnography.
One of the primary purposes of the research design was the elimination of competing hypotheses-the hallmark of good research design-and the testing of the primary hypothesis is an example of descriptive inference, as opposed to statistical inference.
Young and Garro tested the two main hypotheses in sequence.
They had to establish differences in treatment choice behavior in the two communities before they could assess any hypotheses concerning differences in beliefs.
Often the primary objective of research design is more a matter of discovery and exploration than the testing of hypotheses.
The authors were interested in three primary hypotheses relating to role performance and the presence of the illness, psychiatric impairment and susto, and relationship between organic disease and susto.
Similarly, advances in computer-intensive methods for testing hypotheses have the potential to expand the range of designs possible, particularly in the imperfect world of fieldwork (Johnson and Murray 1997).
Research Design in Anthropology: Paradigms and Pragmatics in the Testing of Hypotheses.
Computer-Intensive Methods for Testing Hypotheses.
85 Brim
Some early exceptions include Brim and Spain's (1974) book on hypothesis-testing designs, Pelto and Pelto's (1978) book on research methodology in cultural anthropology, and Naroll and Cohen's (1973) A Handbook of Method in Cultural Anthropology, which has several chapters that address issues in research design (LeVine 1973;
Mead's position on these various elements of research design provided fuel for the continuing discussions about the validity of her original findings (Brim and Spain 1974;
Brim and Spain (1974) recognized several problems in the design that could have affected Mead's ability to draw valid conclusions.
But, in common with Brim and Spain, Orans found itemized problems with Mead's research design.
Similar to the observation by Brim and Spain, Orans points out that Mead made no comparison of sexual behavior between girls living in a native pastor's household and girls living with their own family.
For Brim and Spain, and for Orans, Mead's research design limited her ability to draw the conclusions she did.
Brim, J. A., and D. H. Spain.
85 Samoa
A good example of this tension between the stated early concerns for the methods of science and the actual use of such methods in ethnography comes from correspondence between Boas and his student Margaret Mead during her first fieldwork in Samoa.
The next section shows how concern for the elimination of potential errors and bias through design and attention to methodological detail applies to discussions about the findings of Margaret Mead and Derek Freeman in Samoa.
Derek Freeman's (1983) criticism of Margaret Mead's work and her findings in Samoa has led to reactions from anthropologists who come from different epistemological traditions.
More recently, Martin Orans did fieldwork in Samoa.
Orans (1996) reanalyzed Mead's field notes and correspondence and once again found that her depiction of Samoa as a halcyon society was at odds with his own impression of Samoa as much more agonistic.
If Samoa was not, during Mead's day, a halcyon society, then her conclusions might have been flawed.
For example, there was no measurement on which to compare differences in stress experienced by adolescents in Samoa and the United States.
Margaret Mead and Samoa: The Making and Unmaking of an Anthropological Myth.
85 Spain
Some early exceptions include Brim and Spain's (1974) book on hypothesis-testing designs, Pelto and Pelto's (1978) book on research methodology in cultural anthropology, and Naroll and Cohen's (1973) A Handbook of Method in Cultural Anthropology, which has several chapters that address issues in research design (LeVine 1973;
Mead's position on these various elements of research design provided fuel for the continuing discussions about the validity of her original findings (Brim and Spain 1974;
Brim and Spain (1974) recognized several problems in the design that could have affected Mead's ability to draw valid conclusions.
But, in common with Brim and Spain, Orans found itemized problems with Mead's research design.
Similar to the observation by Brim and Spain, Orans points out that Mead made no comparison of sexual behavior between girls living in a native pastor's household and girls living with their own family.
For Brim and Spain, and for Orans, Mead's research design limited her ability to draw the conclusions she did.
Brim, J. A., and D. H. Spain.
84 postmodernism
Interpretive, hermeneutic, and postmodern approaches make little explicit reference to ethnographic design issues, but well-written examples from ethnography may provide "moral evidence" to deal with current social problems, moving people (including politicians) in ways that numerical facts can't (Seidman 1994:134).
To say that the research design logic of science has been replaced by something that is recognizable as the research design logic of, say, postmodernism would, I think, be misleading.
Phenomenology, hermeneutics, symbolic anthropology, interpretive anthropology, interpretive interactionism, deconstructionism, postmodernism, and constructivism, to name a few, question, in one way or another, some or all of the ontology, epistemology, and methodology of systematic approaches.
Although some of the older interpretive strategies that emerged from the scientific tradition in the social sciences, such as early interpretive anthropology, still adhered to some logical empiricist methodology and maintained a degree of belief in ethnographic authority, more recent approaches, such as postmodernism and constructivism, are more radical in their sweeping rejection of scientific method and design logic (see Schwandt 1994).
In contrasting Geertz and early interpretive anthropology with some of the later postmodern turns of such ethnographic writers as James Clifford, Rabinow (1986) observes:
However, some interpretive work is more similar in nature to the exploratory or descriptive strategies found under the systematic side of Figure 1 than to some of the more radical forays into, for example, postmodernism.
If we talk of an interpretive method, particularly with regard to postmodernism, it more than likely involves both the researcher's immersion into the cultural context of the actor(s) and some means, usually literary, for conveying the understanding gained from such an immersion.
As stated, many interpretive studies are closer in character to exploratory and descriptive research in the systematic mode than to some of the more extreme postmodern studies.
The Perils of the Postmodern.
The Postmodern Turn: New Perspectives on Social Theory.
83 Johnson
Jeffery C. Johnson
The lessons learned from controlled experimentation are applied today to the policy arena where groups are in conflict over resources or because of social inequalities (Johnson and Pollnac 1989;
LeVine (1973) and others (Johnson 1990) make the point that the nature of fieldwork, in terms of its requisite huge investments in time and geographical focus, has often limited the attractiveness of more formal research designs because of its commitment to studying specific problems in a specific way.
The "my natives" or "my village" mentality of some and the fact that careers were made by discovering new theories or describing exotic less well-known cultures has certainly inhibited replication efforts (Johnson 1990).
Part of the confusion stems from a lack of consensus on what ethnography really is (Johnson 1990).
Johnson and Murray (1997), for example, used a quasi-experimental design to evaluate the use of fish aggregation devices (FADS) in small-scale fisheries development projects.
Johnson and Murray compared and determined catch rates.
For example, in their study of preschool children, Johnson et al.
Careful attention to sampling, whether probabilistic (Babbie 1990) or nonprobabilistic (Johnson 1990), is essential.
Johnson 1990;
Such matters are particularly germane for observational designs using various social network approaches (see Johnson [ 1994] for a review).
Johnson and Murray 1997).
Boster and Johnson (1989) explored this issue in an ethnobiological study of fish.
Boster and Johnson used a static group comparison design to compare several groups of expert fishermen with a group of novice fishermen.
Boster and Johnson used statistical and graphical methods to evaluate whether experts' and novices' judgments of fish, at the aggregate and individual levels, were closer to the morphological characteristics of fish (taxonomic distance) or the uses of fish (beliefs about use).
In my view, the consistency of her findings in this diverse sample of informants makes her case stronger (see Johnson 1990).
That is, finding commonality in the face of diversity provides stronger evidence of a shared cultural model (Johnson and Griffith 1996).
Had Goldin relied exclusively on, say, the life histories of a nonprobabilistic sample of informants without specified selection criteria (Johnson 1990), she might have arrived at a very different, and possibly erroneous, conclusion.
Similarly, advances in computer-intensive methods for testing hypotheses have the potential to expand the range of designs possible, particularly in the imperfect world of fieldwork (Johnson and Murray 1997).
Boster, J. S., and J. C. Johnson.
Johnson, J. C. 1990.
Johnson J. C. 1994.
Johnson, J. C., and D. C. Griffith.
Johnson, J. C., M. Ironsmith, A. L. Whitcher, G. M. Poteat, and C. W. Snow.
Johnson, J. C., and J. D. Murray.
Johnson, J. C., and R. Pollnac, eds.
82 Orans
As Orans (1996) describes it, Mead wrote to Boas with her concerns about possible violations of scientific principles in the data she had collected to that point.
Orans says: "What she wants is permission to present data simply as `illustrative material' for the representativeness of which one will simply have to take her word" (p. 127).
(from Orans 1996:128)
Orans 1996).
Without such attention to good design and methodological detail, researchers leave themselves open to one of the worst criticisms of all-of being "not even wrong" (Orans 1996).
More recently, Martin Orans did fieldwork in Samoa.
Orans (1996) reanalyzed Mead's field notes and correspondence and once again found that her depiction of Samoa as a halcyon society was at odds with his own impression of Samoa as much more agonistic.
Orans's work was, of course, many years after Mead's, and he worked at different field sites than did Mead.
But, in common with Brim and Spain, Orans found itemized problems with Mead's research design.
Similar to the observation by Brim and Spain, Orans points out that Mead made no comparison of sexual behavior between girls living in a native pastor's household and girls living with their own family.
For Brim and Spain, and for Orans, Mead's research design limited her ability to draw the conclusions she did.
Orans, M. 1996.
77 measures
This is similar to the grounded theory ideas of Glaser and Strauss (1967), where exploratory descriptive research leads to the development of more meaningful theory and measures.
Exploratory research may contribute to the production of reliable and valid measures, provide information essential for constructing comparison groups, facilitate construction of structured questions or questionnaires, or provide information necessary for producing a sound probability or nonprobability sample.
Figure 2 shows that the research process involves a simultaneous concern for the development of empirical statements from theory (for example, hypotheses), the operationalization of theoretical concepts (for example, meaningful and reliable measures), design (for example, groups to be studied), data collection (for example, qualitative versus quantitative), and data analysis (for example, multiple regression and text analysis).
The levels at which theoretical concepts are measured (for example, nominal or ordinal), the types of sampling strategies used, and the application of appropriate types of analysis must all be considered as a part of the design.
In anthropological fieldwork, these designs and others can be used in tandem to test or explore components of a theory (such as combinations of time series and repeated measures designs particularly applicable to long-term fieldwork).
For example, the pretest/posttest nonequivalent groups design controls for some internal threats to validity, but it's problematic with respect to controlling for changes due to how groups members were selected (selection maturation), changes due to how individuals were tested (instrumentation), changes due to the selection of individuals with extreme pretest measures leading to regression toward the mean (regression), and changes due to local events not a part of the study (history).
Other sources of potential bias include sampling error (that is, chance), nonresponse, the use of imprecise measures, data recording errors, informant inaccuracies, and interviewer effects (see Pelto and Pelto 1978;
How reliable are your measures in terms of precision, sensitivity, resolution, and consistency?
Are they valid, particularly with respect to accuracy and specificity, in that they are actually measuring what they are intended to measure?
Mead relied mostly on herself as an instrument to measure the variables of interest.
This example illustrates nicely the importance of not relying on a single test, but having multiple tests and measures (Stinchcombe 1987).
Finally, a measure of morphological similarities was determined, using taxonomic distances between pairs of fish.
This is a good example of the application of exploratory research in the production of better measures of potentially important explanatory variables.
68 focus
However, this debate has its parallel in sociology where schools such as ethnomethodology and symbolic interactionism developed in response to the largely quantitative macro-level focus of the discipline.
LeVine (1973) and others (Johnson 1990) make the point that the nature of fieldwork, in terms of its requisite huge investments in time and geographical focus, has often limited the attractiveness of more formal research designs because of its commitment to studying specific problems in a specific way.
The realities of fieldwork often dictate the need to change the problem focus or, finding that the proposed hypotheses are inappropriate to the cultural setting under study, the need to somehow salvage the research.
Laboratory and survey researchers have some flexibility to change the problem focus and study populations in light of emerging problems, but field workers are limited in their ability to do so.
Exploratory research can be the primary focus of a given design or just one of many components.
Thus, there is less focus on the means of research, such as methods of data collection and analysis as found in the systematic strategies, and more on the ends of research-the ethnographic or literary product.
More extreme forays into experimental ethnography have blurred this distinction even further, and there is more of a focus on writing strategies that include such approaches as montages, evocative representations, polyvocal texts, and even ethnographic fictions (Denzin and Lincoln 1994).
In the following pages, I focus primarily on research designs in systematic research.
66 population
Laboratory and survey researchers have some flexibility to change the problem focus and study populations in light of emerging problems, but field workers are limited in their ability to do so.
The nature of the groups or characteristics to be compared-in terms of such things as the size of the comparison groups in the overall population, the frequency of characteristics of interest in the population, the availability of a sampling frame, the ability to identify members of the population (for example, hidden or clandestine populations)-all influence the choice of a sample design.
To generalize to a target population, the sample has to be representative of the population of interest.
This is essential if we are to generalize to a whole population and is generally, though not always, a requirement for classical statistical tests.
When generalization to a target population is the objective, you should strive to define a sampling universe or frame using a selection procedure with known error limits and one that represents the population of interest.
In one approach, the model of deliberate sampling for heterogeneity, target classes of units, whether classes or categories of persons, places, times, or events, are deliberately chosen to represent the range of such classes found in the population.
Thus, testing for a treatment effect across a wide range of classes in the set of all possible classes (including both extremes and the modal class) in the population allows the researcher to say something about how the effect holds in a variety of settings.
While this might not be generalized to the population as a whole, it does inform the researcher if an effect holds across wide ranging classes within the population.
For some populations, it may be impossible to develop a sampling frame from which to draw a sample.
Each of these approaches has potential problems, and most do not allow for generalizations about a population since they involve elements of unknown error even if the method involves some form of random selection criteria (for example, random selection of locations in which to intercept respondents).
In some cases, a researcher may not be interested in generalizing to a population but may just want to know whether two subgroups obtained from a snowball sample differ with respect to some variable of interest.
In that case, much of the bias in the sample is a matter of the logic used in the original selection of sample seeds and any statistical analysis of the data must be concerned about violations of assumptions for the particular statistical test to be employed (for example, independence of observations or random sample from a population).
If you are interested in generalizing to a given population, random sampling of some kind is essential.
Recent developments in randomization and computer-intensive methods of statistical analysis involve less restrictive assumptions concerning the data (for example, assumption of a random sample from a population or skewed, sparse, or small sample sizes), opening the way for the development of new test statistics particularly suited for the problem at hand (Noreen 1989;
In addition, she had a tendency to understate the population and overstate the proportion of girls in her study.
An important issue in this area of research concerns the factors influencing the use of Western treatments among non-Western populations.
Although not generally representative of either the regional population or of the population of the United States, Quinn claims that her sample of informants represents the regions' population in terms of the high degree of recent in-migration to the area from regions outside the South.
65 authority
Although some may be swayed by the elegance of a well-written essay, for many it's crucial to know something about the author, his or her motivations, experiences, skills, methods of investigation, and so on before passing judgment on the conclusions.
The constraints of field research may lead one to stray from the idealized prescriptions of a research design, but Mead was attempting to exert her authority without necessarily following the research procedures advocated by Boas and others.
A discussion of the basic arguments as related to epistemology, objectivity, reality, authority, and the like are beyond the scope of this chapter (see Schweizer in this volume).
Although some of the older interpretive strategies that emerged from the scientific tradition in the social sciences, such as early interpretive anthropology, still adhered to some logical empiricist methodology and maintained a degree of belief in ethnographic authority, more recent approaches, such as postmodernism and constructivism, are more radical in their sweeping rejection of scientific method and design logic (see Schwandt 1994).
The core activity is still social description of the other, however modified by new conceptions of discourse, author, or text.
Following in the "thick description" tradition of Geertz, Zabusky clearly believes in some kind of ethnographic authority.
Among other things, they provide escape and opportunity, tolerate a wider range of behaviors, consist of isolated settlements, have fewer status distinctions, involve novel hardships, have decentralized authority, involve lawlessness and peril, have a reputation for helpfulness and sociability, lag in the development of policies to exploit and regulate, and involve a variety of interest group conflicts.
Using a standard chi-square test, the authors found a significant difference in the frequency distribution of treatment alternatives between the two towns, with the exception of folk curers.
Ironically, in statistical terms, the authors have more interest in the null hypothesis of no difference in beliefs than in the alternative hypothesis of a difference in beliefs between the two communities.
The authors' careful attention to research design and analytical issues contributed to the production of impressive evidence that casts doubt on the validity of the "conceptual incompatibility" hypothesis.
Despite the authors' claims of finding no "significant difference," there was no real way, at least when the study was conducted, to assess the extent to which any differences were significant in the sense of statistical probability.
Using statistical and descriptive inference, the authors concluded that whether informants use form or function for classification depends on the knowledge base of the informants and the methods used to test their knowledge (see Figure 4).
The authors were interested in three primary hypotheses relating to role performance and the presence of the illness, psychiatric impairment and susto, and relationship between organic disease and susto.
The proposed design could have been conducted in a single community, but the authors felt that the generalizability of the results would be enhanced with a multiple case-control study design.
This study is important because of the authors' candor about the potential threats to validity they encountered in conducting the research.
In Constructing Knowledge: Authority and Critique in Social Science.
63 LeVine
Some early exceptions include Brim and Spain's (1974) book on hypothesis-testing designs, Pelto and Pelto's (1978) book on research methodology in cultural anthropology, and Naroll and Cohen's (1973) A Handbook of Method in Cultural Anthropology, which has several chapters that address issues in research design (LeVine 1973;
Even in the late 1960s, when concern for methodological rigor was probably at its peak in anthropology, many treatments of research methods and design in the literature played down the need for more systematic methods and design detail, particularly with respect to hypothesis-testing approaches (LeVine 1973).
LeVine (1973) and others (Johnson 1990) make the point that the nature of fieldwork, in terms of its requisite huge investments in time and geographical focus, has often limited the attractiveness of more formal research designs because of its commitment to studying specific problems in a specific way.
Thus, the idea of researchers "putting all their eggs in one basket" may have limited the a priori formulation of problems in fieldwork (LeVine 1973:184).
LeVine, R. A. 1973.
62 Freeman
Freeman 1983;
The next section shows how concern for the elimination of potential errors and bias through design and attention to methodological detail applies to discussions about the findings of Margaret Mead and Derek Freeman in Samoa.
Mead Versus Freeman: Research Design as Mediator
Derek Freeman's (1983) criticism of Margaret Mead's work and her findings in Samoa has led to reactions from anthropologists who come from different epistemological traditions.
others have pointed to the biases and flaws in Freeman's argument (Marcus 1983;
The criticisms and counter-criticisms are difficult to assess, given the time between Mead's and Freeman's studies, the differences in locations of their work, and the differences in their ideological positions (Ember 1985).
Freeman contended that some of Mead's informants lied to her and that Mead's commitment to a particular ideological position caused her to evaluate evidence incorrectly.
Evidence and Science in Ethnography: Reflections on the Mead-Freeman Controversy.
Freeman, D. 1983.
Not Even Wrong: Margaret Mead, Derek Freeman, and the Samoans.
The History of Samoan Sexual Conduct and the Mead-Freeman Controversy.
61 Table
Table 1 describes examples from observational and quasi-experimental study designs discussed by Kleinbaum et al.
Tables 2 and 3 give examples of threats to internal and external validity as discussed in Cook and Campbell (1979) for quasi-experimental designs.
The threats in Table 2 deal with extraneous factors that may account for the presence or absence of a hypothesized effect (that is, contrast validity with invalidity).
Cook and Cambell (1979) detail how each of the quasi-experimental designs in Table 1 are better or worse at dealing with each of the threats to validity that are found in Tables 2 and 3.
60 Margaret
A good example of this tension between the stated early concerns for the methods of science and the actual use of such methods in ethnography comes from correspondence between Boas and his student Margaret Mead during her first fieldwork in Samoa.
The next section shows how concern for the elimination of potential errors and bias through design and attention to methodological detail applies to discussions about the findings of Margaret Mead and Derek Freeman in Samoa.
Derek Freeman's (1983) criticism of Margaret Mead's work and her findings in Samoa has led to reactions from anthropologists who come from different epistemological traditions.
Margaret Mead and Samoa: The Making and Unmaking of an Anthropological Myth.
Not Even Wrong: Margaret Mead, Derek Freeman, and the Samoans.
57 limited
Although the power of experimental design is evident, concern for its application in anthropology-particularly cultural anthropology-has been limited.
Bernard (1994) has elaborated in more detail on issues of design, but his treatment is necessarily limited, given his task of describing the range of methods available to anthropologists.
LeVine (1973) and others (Johnson 1990) make the point that the nature of fieldwork, in terms of its requisite huge investments in time and geographical focus, has often limited the attractiveness of more formal research designs because of its commitment to studying specific problems in a specific way.
Laboratory and survey researchers have some flexibility to change the problem focus and study populations in light of emerging problems, but field workers are limited in their ability to do so.
Thus, the idea of researchers "putting all their eggs in one basket" may have limited the a priori formulation of problems in fieldwork (LeVine 1973:184).
Further, the huge investment in time and resources limited another important goal of science, that of replication, since an ethnographer couldn't realistically be expected to replicate someone else's work.
For Brim and Spain, and for Orans, Mead's research design limited her ability to draw the conclusions she did.
More attention to issues of research design and methods would have improved her chances to make valid claims and possibly limited later criticism of her work.
51 needs
A distinction needs to be made between what's sometimes called the laundry-list component of research and research design.
Some environmental groups were still concerned that evidence with more statistical power was needed.
In most earlier research on injection drug users (IDUs) and HIV risk, the primary risk factor was viewed in terms of direct needle sharing.
Termed "indirect sharing," these nine behaviors can promote the transmission of HIV among IDUs who, although not sharing needles directly, often share water for mixing of drugs or for rinsing syringes, share drug-mixing containers (cookers and spoons), share cottons for filtering, and share the actual drug solution itself.
47 search
Ironically, this concern for context and meaning over methodological rigor, particularly for those in search of theoretical foundations (that is, the Boasian idea of data leading to the construction of theory), would ultimately hinder the comparability of data from different ethnographic sources (see Moran [1995] for a recent discussion of this issue and see Ember and Ember, this volume).
In its most extreme form, systematic strategies tend to involve the search for explanations of phenomena and the pursuit of theoretical foundations.
In searching for such foundations, there is a need for objectivity, replication, and control over possible sources of error leading to a valid assessment of a given theory.
In this enterprise, explanation can involve a general search for causality or prediction.
Practitioners of almost all interpretive paradigms are searching in one way or another for some understanding (verstehen) rather than for some explanation of social phenomena.
An important implication here is that scholars who follow this line of inquiry are searching for local rationales rather than nomothetic theory or universal foundations and may be more interested in conveying a moral tale of some type rather than a value-free account (Seidman 1994).
Participant Observation and the Search for Validity
45 evaluation
The value of empirical evidence can only be properly evaluated by understanding the details of how the research was conducted.
The invention of the simple control/treatment design of clinical trials allowed researchers in this century to evaluate competing therapies and to select the ones that worked best.
" Anthropologists involved in development and evaluation research are most likely to use this design.
Howevert, with careful attention to design and ethnographic context, quasi-experimental and natural experimental designs can be applied to anthropological field settings, particularly in evaluation research and development research.
Johnson and Murray (1997), for example, used a quasi-experimental design to evaluate the use of fish aggregation devices (FADS) in small-scale fisheries development projects.
Discussing "common sense knowing" in evaluation research, Campbell (1988) gives an important critique of ethnography.
It would also help if the anthropologists were to study the school for a year or two prior to the program evaluation.
Freeman contended that some of Mead's informants lied to her and that Mead's commitment to a particular ideological position caused her to evaluate evidence incorrectly.
Boster and Johnson used statistical and graphical methods to evaluate whether experts' and novices' judgments of fish, at the aggregate and individual levels, were closer to the morphological characteristics of fish (taxonomic distance) or the uses of fish (beliefs about use).
This provided for an evaluation of the explanatory power of each.
Then, a survey approach was used to evaluate the viability of these mechanisms in terms of characterizing general trends within Almolonga.
Evaluating FAD Effectiveness in Development Projects: Theory and Praxis.
44 differences
First, it demonstrates the differences between the stated scientific objectives of ethnographic work as advocated by Boas and the actual practice of ethnographic research.
Two fixed fishing structures (piers) were pretested for differences in catch rates.
In the quasi-experimental case, this means changes between pre- and posttest, but this way of thinking can be expanded to include hypothesized effects dealing with differences, similarities, or associations whether diachronic or synchronic.
The criticisms and counter-criticisms are difficult to assess, given the time between Mead's and Freeman's studies, the differences in locations of their work, and the differences in their ideological positions (Ember 1985).
For example, there was no measurement on which to compare differences in stress experienced by adolescents in Samoa and the United States.
They had to establish differences in treatment choice behavior in the two communities before they could assess any hypotheses concerning differences in beliefs.
On the basis of the data from the triads study and the term-frame interviews, we see little reason to reject the "null hypothesis" of no significant differences between the responses of the two groups of informants.
Despite the authors' claims of finding no "significant difference," there was no real way, at least when the study was conducted, to assess the extent to which any differences were significant in the sense of statistical probability.
This is analogous to treatment and control groups without the random assignment of subjects to experimental units and where the treatment is implied rather than researcher directed (that is, natural differences in experience with fish).
Beyond these constancies of cultural and marital experience, they were selected to maximize diversity with regard to such obvious differences as their occupations and educational backgrounds, religious affiliations and ethnic and racial identities, their neighborhoods and social networks, and the duration of their marriages.
Statistical tests of group differences provided a greater understanding of the risk factors associated with the different types of behavior.
36 distinction
A distinction needs to be made between what's sometimes called the laundry-list component of research and research design.
Under the systematic distinction are the two primary categories of exploratory and explanatory approaches, each entailing a specific design strategy.
In many ways, this blurs the distinction between what is anthropological and what is literary.
More extreme forays into experimental ethnography have blurred this distinction even further, and there is more of a focus on writing strategies that include such approaches as montages, evocative representations, polyvocal texts, and even ethnographic fictions (Denzin and Lincoln 1994).
This distinction between experimental and observational approaches is similar to one in ecological field studies.
The primary distinction lies between that of sampling versus allocation.
Cook and Campbell (1979) make a similar distinction but refer to these kinds of natural experiments as "passive-observational studies.
Among other things, they provide escape and opportunity, tolerate a wider range of behaviors, consist of isolated settlements, have fewer status distinctions, involve novel hardships, have decentralized authority, involve lawlessness and peril, have a reputation for helpfulness and sociability, lag in the development of policies to exploit and regulate, and involve a variety of interest group conflicts.
This distinction is important, particularly with regard to anthropological research, in that hypothesis-testing research can be done without narrowly restricting it to analytical methods using statistical inference.
(1996) used these additional distinctions in sharing to look at the prevalence of injection-related HIV risk behaviors among several subpopulations of injection drug users (see Figure 6).
36 experts
Boster and Johnson used a static group comparison design to compare several groups of expert fishermen with a group of novice fishermen.
To ensure that experts were, in fact, experienced recreational fishermen, the rosters of sport fishing clubs in each region were sampled at random.
Each of the four expert groups comprised 15 subjects chosen at random from a larger sample of recreational fishermen.
Thus the groups to be compared consisted of five groups of 15 subjects, four consisting of experts and one of novices.
Boster and Johnson used statistical and graphical methods to evaluate whether experts' and novices' judgments of fish, at the aggregate and individual levels, were closer to the morphological characteristics of fish (taxonomic distance) or the uses of fish (beliefs about use).
Form or Function: A Comparison of Expert and Novice Judgments of Similarity Among Fish.
36 sociability
Orans (1996) reanalyzed Mead's field notes and correspondence and once again found that her depiction of Samoa as a halcyon society was at odds with his own impression of Samoa as much more agonistic.
If Samoa was not, during Mead's day, a halcyon society, then her conclusions might have been flawed.
Among other things, they provide escape and opportunity, tolerate a wider range of behaviors, consist of isolated settlements, have fewer status distinctions, involve novel hardships, have decentralized authority, involve lawlessness and peril, have a reputation for helpfulness and sociability, lag in the development of policies to exploit and regulate, and involve a variety of interest group conflicts.
Melbin conducted four tests of the feature relating to helpfulness and sociability.
" However, Melbin's three other tests supported the hypothesis of more sociability and helpfulness at night.
Had Melbin conducted only the key experiment, he may have come to very different conclusions regarding the helpfulness and sociability of night-timers.
Many peasant societies in Central America are experiencing dramatic economic and cultural change.
The Comparative Analysis of Human Societies: Toward Common Standards for Data Collection and Reporting.
35 operationalization
I don't think we should make too much of it because the important elements of research design-reliability, informant accuracy, validity, objectivity, and operationalization of theoretical concepts-have been present in the writings of cultural anthropologists even before Boas.
Figure 2 shows that the research process involves a simultaneous concern for the development of empirical statements from theory (for example, hypotheses), the operationalization of theoretical concepts (for example, meaningful and reliable measures), design (for example, groups to be studied), data collection (for example, qualitative versus quantitative), and data analysis (for example, multiple regression and text analysis).
For example, the particular structure of an empirical statement or hypothesis will partially determine the manner in which theoretical concepts are operationalized and eventually analyzed.
In one way or another, various study designs, in combination with other considerations such as the operationalization of theoretical constructs and sampling, are better or worse at dealing with each.
Measurement, operationalization of theoretical concepts, and type of analysis used are other important factors.
There is a lack of specificity in the development and operationalization of key concepts.
Symptomology and health problems were operationalized using a panel of physicians.
Psychiatric impairment was operationalized using the 22-item Screening Score for Psychiatric Impairment.
Based on earlier ethnographic research, social stress, an important component for understanding an individual's inability to perform social roles, was operationalized using the Social Stress Gauge developed by one of the researchers.
33 analytical
Design, on the other hand, involves the methodological and analytical details that contribute to the credibility, validity, believability, or plausibility of any study.
This leads to designs that involve concern for a higher degree of methodological and analytical detail, whether quantitative or qualitative.
While systematic analytical paradigms are primarily concerned with threats to validity, recent interpretive paradigms are focused more on threats to believability -as in "Do you believe my story?
In sociology, multiple regression models, structural equation models, and path analytic models (all related analytical techniques) have influenced the design of survey research.
Whereas the analytical techniques most often used in psychology, sociology, and economics often led to rather standard designs, in anthropology the eclectic nature of ethnography leaves the design of research more open ended.
There are generally no ethnographic "analytical techniques" driving the design, although ethnography has been variously associated with a number of qualitative methods.
This is particularly true today, given the large number of computer analytical packages available for analyzing text (see Bernard and Ryan, this volume).
In manipulative experiments, analytical units are randomly allocated to comparative groups, whereas in mensurative experiments selection of units is based on some probability or nonprobability sampling scheme.
The authors' careful attention to research design and analytical issues contributed to the production of impressive evidence that casts doubt on the validity of the "conceptual incompatibility" hypothesis.
This distinction is important, particularly with regard to anthropological research, in that hypothesis-testing research can be done without narrowly restricting it to analytical methods using statistical inference.
Using path analytic modeling, she applied different statistical controls in each of the competing models.
33 nets
The outcry for a ban on nets in tuna fishing is a famous recent example.
Environmental organizations launched campaigns to ban nets in tuna fishing because dolphins are often caught incidentally in that fishery.
Media campaigns in the U.S. showing pictures of dolphins being caught in nets (generally not in U.S. waters), contributed to Florida's totally banning fishing nets-even though no marine mammals were threatened by the use of nets in Florida waters.
There has been similar concern over the incidental catch of harbor porpoises by net fishers in New England (Schneider 1996).
In response, the fishing industry proposed the voluntary use of "pingers" an underwater acoustic device-to keep porpoises from their nets.
The study used the classic control/treatment design in which catch rates for a set of nets with pingers were compared to catch rates for set of nets without pingers.
In the first experiment, the control net caught 10 porpoises while the treatment net caught none.
Some conservationist groups claimed the study was biased in that the treatment nets were placed in areas known not to have large numbers of porpoises.
So another study was conducted placing experimental treatment and control nets in the same proximity.
This time, the treatment net caught only 1 porpoise while the control net caught 32.
Lobbying efforts by fishers yielded more funds for a larger, more comprehensive study involving more than 10,000 fishing nets.
Both control and treatment nets were outfitted with pingers, but only the pingers on treatment nets would activate once placed in the water.
Thus, fishers were blind as to which nets were control and which were treatment-a classic double-blind experimental design.
Again the evidence was impressive: The treatment nets caught 2 porpoises (1 was thought to be deaf), while the control nets caught 25.
Alarming Nets.
30 representation
We need a powerful mode of argumentation, a mode that ensures we can represent our representations in credible ways.
That's what I seek, not as the only possible representation that our field can offer, but as an essential lever to try and move the world.
This is a highly simplified representation.
The other for Clifford is the anthropological representation of the other.
More extreme forays into experimental ethnography have blurred this distinction even further, and there is more of a focus on writing strategies that include such approaches as montages, evocative representations, polyvocal texts, and even ethnographic fictions (Denzin and Lincoln 1994).
First, it is representative of the genre that rejects explanation in favor of conveying a moral tale.
Its purpose is not a traditional attempt at explanation where facts are considered real, but political interpretation and representation of facts, independent of their "realness.
While random assignment aids in controlling for confounding variables by producing homogeneous comparative groups, random sampling of units produces comparison groups that are representative of such groups.
To generalize to a target population, the sample has to be representative of the population of interest.
In Young and Garro's case, a visual inspection of the graphical representations of the data could lead to no other conclusion than that there was little or no difference in beliefs between the two communities (see Figure 3).
Although not generally representative of either the regional population or of the population of the United States, Quinn claims that her sample of informants represents the regions' population in terms of the high degree of recent in-migration to the area from regions outside the South.
However, when a large representative sample of the township was aggressively pursued, the different data sets tended to support model C as the one that characterizes the general tendencies within the township.
The newer forays into experimental and other ethnographic forms of presentation are more reflexive in character and more concerned with believable and moving representations rather than the production of valid accounts or conclusions.
Representations are Social Facts: Modernity and Post-Modernity in Anthropology.
29 British
Thus, while early British and U.S. anthropologists advocated the scientific method in ethnographic research, there is little evidence that they considered appropriate design issues when they actually did the research.
But in spite of claims to scientific methodology, particularly in the British tradition, there are surprisingly few details about actual methods anthropologists used in the field, beyond a few first principles and illustrative anecdotes.
There was a wide belief among British anthropologists that fieldwork could not be taught to new recruits, but could only be experienced by individuals in the field.
In the American tradition texts provided what was regarded as an objective body of data, whereas the British tradition was more a matter of subjective experience.
There is much anecdotal evidence for a belief, across the British and U.S. traditions, in a trial-by-fire method of training for ethnographers.
26 novices
Boster and Johnson used a static group comparison design to compare several groups of expert fishermen with a group of novice fishermen.
Thus the groups to be compared consisted of five groups of 15 subjects, four consisting of experts and one of novices.
Boster and Johnson used statistical and graphical methods to evaluate whether experts' and novices' judgments of fish, at the aggregate and individual levels, were closer to the morphological characteristics of fish (taxonomic distance) or the uses of fish (beliefs about use).
Form or Function: A Comparison of Expert and Novice Judgments of Similarity Among Fish.
25 generalizability
However, if done properly, such designs can have increased external validity and generalizability.
External validity is concerned with the approximation to the truth as expanded to other settings-that is, with the generalizability of research findings.
Similarly, threats to external validity, such as problems stemming from biased samples or research in atypical or unique settings, can hamper the generalizability of one's findings.
When generalization to a target population is the objective, you should strive to define a sampling universe or frame using a selection procedure with known error limits and one that represents the population of interest.
While this might not be generalized to the population as a whole, it does inform the researcher if an effect holds across wide ranging classes within the population.
Each of these approaches has potential problems, and most do not allow for generalizations about a population since they involve elements of unknown error even if the method involves some form of random selection criteria (for example, random selection of locations in which to intercept respondents).
In some cases, a researcher may not be interested in generalizing to a population but may just want to know whether two subgroups obtained from a snowball sample differ with respect to some variable of interest.
If you are interested in generalizing to a given population, random sampling of some kind is essential.
If generalization is not a primary goal, then sampling requirements may be relaxed.
The proposed design could have been conducted in a single community, but the authors felt that the generalizability of the results would be enhanced with a multiple case-control study design.
24 advocated
Boas and most of his students advocated a natural science logic in the collection of ethnographic materials and a true concern for the collection of reliable data that could lead to the production of valid theory.
The constraints of field research may lead one to stray from the idealized prescriptions of a research design, but Mead was attempting to exert her authority without necessarily following the research procedures advocated by Boas and others.
First, it demonstrates the differences between the stated scientific objectives of ethnographic work as advocated by Boas and the actual practice of ethnographic research.
Thus, while early British and U.S. anthropologists advocated the scientific method in ethnographic research, there is little evidence that they considered appropriate design issues when they actually did the research.
24 assessment
For Stinchcombe (1987:23), the observations produced by how a study was designed are fundamental to the proper assessment of empirical evidence: "We always want to reject evidence if it can be explained by the design of the research or by a large number of small, unorganized causes.
Without some unbiased means for assessing the evidence, the truth is only be a matter of who has the most political clout.
In each successive study, investigators tried to control for as many extraneous variables as possible so that the hypothesized effect could be assessed (that is, the effectiveness of pingers compared to not using pingers).
In searching for such foundations, there is a need for objectivity, replication, and control over possible sources of error leading to a valid assessment of a given theory.
As a result, the assessment of any theory involves research designs more heavily concerned with the means-the research process, rather than simply the way the study was written or argued-since the validity of study results depends on the scientific soundness of the research design.
It involves constructing a logical plan that links all the elements of research together so as to produce the most valid assessment possible of some theory, given some set of realistic constraints (for example, cost, scope, geographical setting, etc.
Random allocation produces equivalent comparison groups, and artificial manipulation of independent variables (also known as explanatory variables or study factors), with all other variables or factors controlled for, allows for the most valid assessment of the causal relationship between the independent and dependent variables or response variables.
The combination helped Goldin in the specification of appropriate variables, in the development of a sound survey instrument, and in the specification and assessment of the four competing models.
22 degree
This leads to designs that involve concern for a higher degree of methodological and analytical detail, whether quantitative or qualitative.
A variety of names are used in the lexicon of social scientists that can be associated to varying degrees with an interpretive strategy.
Although some of the older interpretive strategies that emerged from the scientific tradition in the social sciences, such as early interpretive anthropology, still adhered to some logical empiricist methodology and maintained a degree of belief in ethnographic authority, more recent approaches, such as postmodernism and constructivism, are more radical in their sweeping rejection of scientific method and design logic (see Schwandt 1994).
This leads us to the conclusion that the substantial variation apparent in the use of a physician's treatment between the two samples, a consequence of differential access to such treatment, occurs without corresponding degrees of variation in resident's attitudes and beliefs about illness.
Although not generally representative of either the regional population or of the population of the United States, Quinn claims that her sample of informants represents the regions' population in terms of the high degree of recent in-migration to the area from regions outside the South.
"Degrees of Freedom" in the Case Study.
20 Boster
Boster and Johnson (1989) explored this issue in an ethnobiological study of fish.
Boster and Johnson used a static group comparison design to compare several groups of expert fishermen with a group of novice fishermen.
Boster and Johnson used statistical and graphical methods to evaluate whether experts' and novices' judgments of fish, at the aggregate and individual levels, were closer to the morphological characteristics of fish (taxonomic distance) or the uses of fish (beliefs about use).
Boster, J. S., and J. C. Johnson.
Kempton, W., J. S. Boster, and J. A. Hartley.
20 forays
However, some interpretive work is more similar in nature to the exploratory or descriptive strategies found under the systematic side of Figure 1 than to some of the more radical forays into, for example, postmodernism.
More extreme forays into experimental ethnography have blurred this distinction even further, and there is more of a focus on writing strategies that include such approaches as montages, evocative representations, polyvocal texts, and even ethnographic fictions (Denzin and Lincoln 1994).
As Ramos sees it, "I found myself making forays into the self-conscious meanderings of reflexive anthropology in order to shift the axis of analysis from the skeleton-like dissertation to the flesh and blood of ethnography" (p. 6).
The newer forays into experimental and other ethnographic forms of presentation are more reflexive in character and more concerned with believable and moving representations rather than the production of valid accounts or conclusions.
19 characterization
I am very decidedly of the opinion that a statistical treatment of such intricate behavior as the one that you are studying, will not have very much meaning and that the characterization of a selected number of cases must necessarily be the material with which you operate.
Figure I is a taxonomic characterization of the different types of research strategies found in contemporary cultural anthropology.
Thus, the rather simple characterization of research strategies found in Figure 1 attempts to recognize the variation inherent in the range of work found in contemporary anthropology by placing "interpretive anthropology" adjacent to "exploratory/descriptive" (see, for example, the work of Zabusky 1995).
Then, a survey approach was used to evaluate the viability of these mechanisms in terms of characterizing general trends within Almolonga.
Indeed, I interacted with several individuals who had life histories that were inconsistent with my general characterization and who were the basis for suggesting the competing models discussed above.
However, when a large representative sample of the township was aggressively pursued, the different data sets tended to support model C as the one that characterizes the general tendencies within the township.
19 interviewer
She also discusses the rationales for selecting the site and the group she studied, problems of working in a linguistically and technically diverse social milieu, the use of semistructured and unstructured interviews, and the effect of her role as ethnographer on informant relations and data quality.
Although Ramos discusses informant interviewing and various sources of data, her introduction is largely devoted to discussions of her reliance on her own memory in writing the ethnography and the shift in the narrative between synchrony and diachrony.
Behar discusses the multiplexity of roles, in that she was variously involved as "priest, interviewer, collector, transcriber, translator, analyst, academic, connoisseur, editor, and peddler" (p. 12).
Then, FADS, umbrella-like units suspended in the water column, were alternately placed at the piers and individual fishers were interviewed simultaneously during randomly selected times at both the treatment (the pier with the FADS) and the control (the pier without the FADS) piers.
Other sources of potential bias include sampling error (that is, chance), nonresponse, the use of imprecise measures, data recording errors, informant inaccuracies, and interviewer effects (see Pelto and Pelto 1978;
Attention and concern with all the potential sources of error, whether stemming from how the study was designed, how the data were collected (for example, face-to-face interviews or mail-out surveys), or how the data were analyzed (for example, statistical conclusion validity), will help lead to the production of solid evidence.
interviewed members of Earth First (a radical environmentalist group) and dry cleaning shop owners (who depend on toxic chemicals for their business).
On the basis of the data from the triads study and the term-frame interviews, we see little reason to reject the "null hypothesis" of no significant differences between the responses of the two groups of informants.
Further, beliefs about the use and functional characteristics of the fish obtained from extensive ethnographic interviews were turned into a sentence-frame completion task described by Weller and Romney (1988).
Based on in-depth interviews with 22 informants, Quinn (1996) attempted to build a cultural model of Americans' reasoning about marriage.
Because she was interested in a model that was shared, it was crucial to interview a wide range of couples who, although of the same culture, were not just from one region of the country or of only one ethnicity, religion, or social class.
and all were married during the period of their interviews, all in first marriages.
19 judged
Although some may be swayed by the elegance of a well-written essay, for many it's crucial to know something about the author, his or her motivations, experiences, skills, methods of investigation, and so on before passing judgment on the conclusions.
Recent developments in statistical procedures allow us to assess the similarities in aggregated judged-similarity matrices between the two communities (see Handwerker and Borgatti, this volume, and Hubert 1987).
Individuals were asked to perform an unconstrained judged similarity of the fish-a free pile sort (see Weller, this volume, and Weller and Romney 1988).
Boster and Johnson used statistical and graphical methods to evaluate whether experts' and novices' judgments of fish, at the aggregate and individual levels, were closer to the morphological characteristics of fish (taxonomic distance) or the uses of fish (beliefs about use).
Form or Function: A Comparison of Expert and Novice Judgments of Similarity Among Fish.
18 diversity
A more appropriate term that would encompass the diversity currently found in cultural anthropology might be "research strategy.
She also discusses the rationales for selecting the site and the group she studied, problems of working in a linguistically and technically diverse social milieu, the use of semistructured and unstructured interviews, and the effect of her role as ethnographer on informant relations and data quality.
Beyond these constancies of cultural and marital experience, they were selected to maximize diversity with regard to such obvious differences as their occupations and educational backgrounds, religious affiliations and ethnic and racial identities, their neighborhoods and social networks, and the duration of their marriages.
Her sample is an attempt to capture the range of diversity found in the region.
In my view, the consistency of her findings in this diverse sample of informants makes her case stronger (see Johnson 1990).
That is, finding commonality in the face of diversity provides stronger evidence of a shared cultural model (Johnson and Griffith 1996).
18 empirical
For Stinchcombe (1987:23), the observations produced by how a study was designed are fundamental to the proper assessment of empirical evidence: "We always want to reject evidence if it can be explained by the design of the research or by a large number of small, unorganized causes.
The value of empirical evidence can only be properly evaluated by understanding the details of how the research was conducted.
Figure 2 shows that the research process involves a simultaneous concern for the development of empirical statements from theory (for example, hypotheses), the operationalization of theoretical concepts (for example, meaningful and reliable measures), design (for example, groups to be studied), data collection (for example, qualitative versus quantitative), and data analysis (for example, multiple regression and text analysis).
For example, the particular structure of an empirical statement or hypothesis will partially determine the manner in which theoretical concepts are operationalized and eventually analyzed.
(Stinchcombe [1987] provides an excellent discussion of how empirical statements are derived from theory.
17 elements
It is the articulation of this "process by which we came to know it" that reflects the elements of research design.
" This statement illustrates at least two important elements of research design.
In this chapter I concentrate on elements of design related to the production of valid results or a believable ethnographic account.
The issue is still under debate, but this series of studies illustrates how the elements of research design help muster evidence in light of competing beliefs and philosophies.
I don't think we should make too much of it because the important elements of research design-reliability, informant accuracy, validity, objectivity, and operationalization of theoretical concepts-have been present in the writings of cultural anthropologists even before Boas.
Mead's position on these various elements of research design provided fuel for the continuing discussions about the validity of her original findings (Brim and Spain 1974;
The figure distinguishes between strategies within the realm of interpretive studies and those using systematic strategies that have more of the elements of science.
Explanatory: Explanatory approaches generally involve testing elements of theory that may already have been proposed in the literature or that have been informed by exploratory research.
It involves constructing a logical plan that links all the elements of research together so as to produce the most valid assessment possible of some theory, given some set of realistic constraints (for example, cost, scope, geographical setting, etc.
Each of these approaches has potential problems, and most do not allow for generalizations about a population since they involve elements of unknown error even if the method involves some form of random selection criteria (for example, random selection of locations in which to intercept respondents).
These findings are undeniably important for larger epidemiological work that examines elements of IDUs' behaviors and such things as producing valid models of seroconversion.
16 assumptions
In that case, much of the bias in the sample is a matter of the logic used in the original selection of sample seeds and any statistical analysis of the data must be concerned about violations of assumptions for the particular statistical test to be employed (for example, independence of observations or random sample from a population).
If you don't use random sampling, pay careful consideration to possible violations of assumptions for a given statistical test.
Recent developments in randomization and computer-intensive methods of statistical analysis involve less restrictive assumptions concerning the data (for example, assumption of a random sample from a population or skewed, sparse, or small sample sizes), opening the way for the development of new test statistics particularly suited for the problem at hand (Noreen 1989;
These new approaches seem particularly well suited for the imperfect world of ethnographic research, where the rather restrictive assumptions of parametric analysis are often difficult to meet.
16 members
Members of such competing groups-such as large-scale commercial producers, commodity producers, environmental groups, and real estate developers-believe strongly in their positions.
What separates quasi-experiments from true experiments is the lack of random assignment of group members.
Observational studies involve neither random assignment of members to comparison groups nor the manipulation by the observer of independent variables.
Random sampling meets the restrictions of some statistical tests, but it does not afford the same protection as does random assignment of group members against the potential effects of extraneous factors.
For example, the pretest/posttest nonequivalent groups design controls for some internal threats to validity, but it's problematic with respect to controlling for changes due to how groups members were selected (selection maturation), changes due to how individuals were tested (instrumentation), changes due to the selection of individuals with extreme pretest measures leading to regression toward the mean (regression), and changes due to local events not a part of the study (history).
The nature of the groups or characteristics to be compared-in terms of such things as the size of the comparison groups in the overall population, the frequency of characteristics of interest in the population, the availability of a sampling frame, the ability to identify members of the population (for example, hidden or clandestine populations)-all influence the choice of a sample design.
interviewed members of Earth First (a radical environmentalist group) and dry cleaning shop owners (who depend on toxic chemicals for their business).
It is unrealistic to suppose that Young and Garro could have randomly assigned community members to the different comparison groups in order to control for confounding variables and then subject their informants to the treatments of interest.
In contrast, Quinn views culture as being shared-that there are cultural models for a variety of domains that are widely held in common, and that these models can be developed from the discourses of cultural members.
Because the asustados were "sick," control group members must also be sick.
Thus, sick people were compared to sick people and control group members were selected from the pool of patients at the project clinics in each of the communities.
16 recreational
To ensure that experts were, in fact, experienced recreational fishermen, the rosters of sport fishing clubs in each region were sampled at random.
The selection of control group subjects, by contrast, involved a purposeful selection procedure in which potential subjects were screened for recreational fishing experience.
Using a questionnaire to gain background information, 15 college undergraduates who had the least amount of recreational fishing experience were selected from two introductory anthropology classes.
Each of the four expert groups comprised 15 subjects chosen at random from a larger sample of recreational fishermen.
16 susto
Susto, A Folk Illness
(1985) report on a study of a folk illness known as susto, found in many cultural groups throughout North and South America.
Folk beliefs surrounding susto attribute loss of a critical substance or force due to a frightening experience.
The authors were interested in three primary hypotheses relating to role performance and the presence of the illness, psychiatric impairment and susto, and relationship between organic disease and susto.
The ultimate aim of the study was to show the relationship between various social forces and susto susceptibility.
One subsample was of individuals who complained of susto during the fieldwork or who had admitted their condition to relatives or curers.
felt comfortable with the comparability among the susto subsamples from the communities.
found that there was, in fact, an association between susto and an individual's perception of the adequacy of his or her performance of critical social roles.
Although there was no association between susto and psychiatric impairment, there was a relationship between susto and the suffering of more organic disease signs.
The stigma of susto among males and the greater social stratification encountered in one of the communities are possible threats to the validity of their conclusions.
Susto, A Folk Illness.
15 application
I use "defensible" in addition to "valid," which I normally use, to make readers aware that I am broadening the traditional application of research design to include the variety of research strategies found in anthropology today.
Although the power of experimental design is evident, concern for its application in anthropology-particularly cultural anthropology-has been limited.
Thus, ethnography should involve multiple methods, both qualitative and quantitative, and may involve applying more than one research design.
The levels at which theoretical concepts are measured (for example, nominal or ordinal), the types of sampling strategies used, and the application of appropriate types of analysis must all be considered as a part of the design.
Random sampling has been a primary requirement in the proper application of parametric statistics.
The next section shows how concern for the elimination of potential errors and bias through design and attention to methodological detail applies to discussions about the findings of Margaret Mead and Derek Freeman in Samoa.
This is a good example of the application of exploratory research in the production of better measures of potentially important explanatory variables.
The Process of Drug Injection: Applying Ethnography to the Study of HIV Risk Among IDU's.
15 girls
She wrote of her doubts about the comparability of cases and about her ability, or even the need, to do a quantitative comparison of the similarity of attitudes among the adolescent girls in her study.
She had concerns-and I believe she thought her mentor, Boas, would feel similarly-as to whether a valid comparison of this type could be made given the selection process for her sample of girls.
The comparison group, Samoan adolescent girls, was compared to a conjectural treatment group, American adolescent girls, to test the proposition that exposure to Western civilization increases adolescent trauma.
For example, Mead could have compared girls living in the households of native pastors to those who did not.
Similar to the observation by Brim and Spain, Orans points out that Mead made no comparison of sexual behavior between girls living in a native pastor's household and girls living with their own family.
In addition, she had a tendency to understate the population and overstate the proportion of girls in her study.
14 economic
Thus, policy emerges from interactions between groups of differing political, ideological, social, and economic backgrounds.
And since I have indicated here that research in culture involves a great deal of unique personal experience for the anthropologist, I have taken the position that it is probably unlikely there can be a rigorous, systematic, and formal presentation of methods in the study of culture like those of the natural sciences and that there are overriding concerns among many sociologists, psychologists, and economists.
Whereas the analytical techniques most often used in psychology, sociology, and economics often led to rather standard designs, in anthropology the eclectic nature of ethnography leaves the design of research more open ended.
The research design included the comparison of two Mexican communities that were similar in terms of cultural traditions and economies but varied in terms of access to Western medical services.
The design involved three communities that differed in history, language, and culture but had similarities in social, demographic, and economic factors.
Many peasant societies in Central America are experiencing dramatic economic and cultural change.
One consequence of these changes is increasing economic differentiation.
Parallel to these economic changes, there has been a shift in religious preference over the last 70 years.
For some researchers, the shift from Catholicism to Protestantism helps account for economic change, as Protestantism is more compatible with capitalist ideology and the accumulation of wealth.
Goldin (1996) wanted to understand the relationships among religious affiliation, economic ideology, occupation, and economic status in a Guatemalan township (Almolonga).
Models of Economic Differentiation and Cultural Change.
14 Melbin
In his work on "colonizing the night," Melbin (1987) theorized that the night was a frontier, not unlike the western United States in the nineteenth century.
Melbin conducted four tests of the feature relating to helpfulness and sociability.
According to Melbin, "To find a key is to come across an implied need for help" (p. 75).
" However, Melbin's three other tests supported the hypothesis of more sociability and helpfulness at night.
Melbin speculates that the variation in results may have been due to the fact that the other three experiments involved direct personal contact among the subjects, while the key experiment involved no such interactions.
Had Melbin conducted only the key experiment, he may have come to very different conclusions regarding the helpfulness and sociability of night-timers.
Unlike studies where informed consent is obtained prior to participation, in experiments like Melbin's, individuals often participate without knowing about it.
Melbin, M. 1987.
13 computer
This is particularly true today, given the large number of computer analytical packages available for analyzing text (see Bernard and Ryan, this volume).
No amount of sophisticated statistics, computer intensive text analysis, or elegant writing can salvage a poorly designed study.
Recent developments in randomization and computer-intensive methods of statistical analysis involve less restrictive assumptions concerning the data (for example, assumption of a random sample from a population or skewed, sparse, or small sample sizes), opening the way for the development of new test statistics particularly suited for the problem at hand (Noreen 1989;
With advances in computer technology, qualitative data analysis can now be a powerful mode to test theories.
Similarly, advances in computer-intensive methods for testing hypotheses have the potential to expand the range of designs possible, particularly in the imperfect world of fieldwork (Johnson and Murray 1997).
Computer-Intensive Methods for Testing Hypotheses.
12 adolescent
She wrote of her doubts about the comparability of cases and about her ability, or even the need, to do a quantitative comparison of the similarity of attitudes among the adolescent girls in her study.
The comparison group, Samoan adolescent girls, was compared to a conjectural treatment group, American adolescent girls, to test the proposition that exposure to Western civilization increases adolescent trauma.
In her use of a conjectural treatment group, Mead assumed some things about American adolescents without collecting comparable data.
That is, any observed difference between the two groups with respect to the dependent variable, adolescent trauma, might have been due to one or several extraneous (unmeasured) factors and might have had nothing to do with the independent variable, exposure to Western culture.
She could then have tested the proposition that exposure to competing standards of sexual morality leads to higher levels of emotional distress in adolescents.
For example, Mead made an assertion concerning the relationship between the size of residential units and adolescent troubles.
For example, there was no measurement on which to compare differences in stress experienced by adolescents in Samoa and the United States.
11 bias
There was a faith, however, that awareness of the potential biases associated with the subjectivity of the investigator could be dealt with in some reasonable way.
A further irony is that the one thing that might have lessened potential subjectivity biases-the use of standardized methods-was rejected outright because meaning might be compromised.
Along with this shift came the freedom not to be concerned with issues of bias and validity or with the need for working systematically, thus allowing for a less restrictive ethnographic narrative.
Potential errors and bias creep in at various steps in the research process.
Other sources of potential bias include sampling error (that is, chance), nonresponse, the use of imprecise measures, data recording errors, informant inaccuracies, and interviewer effects (see Pelto and Pelto 1978;
In that case, much of the bias in the sample is a matter of the logic used in the original selection of sample seeds and any statistical analysis of the data must be concerned about violations of assumptions for the particular statistical test to be employed (for example, independence of observations or random sample from a population).
This increases the chances of detecting potential bias and also makes replication feasible.
The next section shows how concern for the elimination of potential errors and bias through design and attention to methodological detail applies to discussions about the findings of Margaret Mead and Derek Freeman in Samoa.
others have pointed to the biases and flaws in Freeman's argument (Marcus 1983;
She made an earnest attempt to control for as many biases as possible and, using the data collected during the survey, conducted statistical tests of the four competing models.
11 further
A further irony is that the one thing that might have lessened potential subjectivity biases-the use of standardized methods-was rejected outright because meaning might be compromised.
Further, the huge investment in time and resources limited another important goal of science, that of replication, since an ethnographer couldn't realistically be expected to replicate someone else's work.
Further, the purpose of research strategies under these interpretive paradigms is more focused on the production of a believable or plausible account or story rather than a single depiction of the truth, since it is thought that there are a multitude of plausible accounts rather than just a single true story.
More extreme forays into experimental ethnography have blurred this distinction even further, and there is more of a focus on writing strategies that include such approaches as montages, evocative representations, polyvocal texts, and even ethnographic fictions (Denzin and Lincoln 1994).
For further discussion of research strategies in the interpretive mode, see Fernandez and Herzfeld (this volume).
Further, beliefs about the use and functional characteristics of the fish obtained from extensive ethnographic interviews were turned into a sentence-frame completion task described by Weller and Romney (1988).
However, further research in the explanatory mode is now warranted.
11 key
Random assignment maximizes the probability that experimental groups are equivalent on key variables prior to the introduction of an intervention.
Nonprobability sampling methods have come to be associated with qualitative approaches or for the selection of ethnographic informants, particularly key informants or consultants (Werner and Schoepfle 1987;
There is a lack of specificity in the development and operationalization of key concepts.
He designed a clever experiment in which keys were placed at similar locations during each two-hour field visit over a 24-hour period covering day and night.
The idea was to see if there were a difference in key-returning behaviors among the different times.
According to Melbin, "To find a key is to come across an implied need for help" (p. 75).
The hypothesis was that residents of the night would return keys on average more often than those of the day.
The keys had the request to "Please Return," with an address encased in plastic (keys dropped in the mailbox were delivered by the U.S. Postal Service to the address on the keys with postage due).
Each of the keys were coded so they could be identified as to what time of day they were picked up and from what location.
In all, 326 keys were picked up, of which 220 were returned.
Returned keys were also scored for the manner in which they were sent.
One point was given for keys dropped unwrapped in the mail, two points for keys returned wrapped in an envelope, and three points if the envelope contained a personal note.
Contrary to expectations, night-timers were not more amiable than day-timers in their key-returning behaviors;
Melbin speculates that the variation in results may have been due to the fact that the other three experiments involved direct personal contact among the subjects, while the key experiment involved no such interactions.
Had Melbin conducted only the key experiment, he may have come to very different conclusions regarding the helpfulness and sociability of night-timers.
11 phenomena
In its most extreme form, systematic strategies tend to involve the search for explanations of phenomena and the pursuit of theoretical foundations.
Practitioners of almost all interpretive paradigms are searching in one way or another for some understanding (verstehen) rather than for some explanation of social phenomena.
Thus, comparison groups may be chosen on the basis of different levels of exposure to some naturally occurring or human-induced phenomena (for example, natural disaster, war, or the building of a dam).
All knowing is comparative, however phenomenally absolute it appears, and an anthropologist is usually in a very poor position for valid comparison, as their own student experience and their secondhand knowledge of schools involve such different perspectives as to be of little comparative use.
11 psychologists
And since I have indicated here that research in culture involves a great deal of unique personal experience for the anthropologist, I have taken the position that it is probably unlikely there can be a rigorous, systematic, and formal presentation of methods in the study of culture like those of the natural sciences and that there are overriding concerns among many sociologists, psychologists, and economists.
She contrasts her study with research on cooperation by "experimental" psychologists, emphasizing the cultural and social orientation of her work and the importance of considering context (social, cultural, political, etc.
In some social science disciplines, like psychology, the design of research is driven by features of the analysis.
Whereas the analytical techniques most often used in psychology, sociology, and economics often led to rather standard designs, in anthropology the eclectic nature of ethnography leaves the design of research more open ended.
11 role
Still, while luck plays a role in research, planning for such luck is not within the realm of research design (Kirk and Miller 1986).
Contemporary Design Issues in Cultural Anthropology There is an ongoing debate in cultural anthropology concerning science and its role in contemporary research.
She also discusses the rationales for selecting the site and the group she studied, problems of working in a linguistically and technically diverse social milieu, the use of semistructured and unstructured interviews, and the effect of her role as ethnographer on informant relations and data quality.
Behar discusses the multiplexity of roles, in that she was variously involved as "priest, interviewer, collector, transcriber, translator, analyst, academic, connoisseur, editor, and peddler" (p. 12).
1996) offer excellent examples of the role of participant observation in more clearly defining the set of HIV risk behaviors surrounding injection drug use.
The authors were interested in three primary hypotheses relating to role performance and the presence of the illness, psychiatric impairment and susto, and relationship between organic disease and susto.
carefully selected communities that were as similar as possible in terms of forms of government and genderspecific role expectations.
Based on earlier ethnographic research, social stress, an important component for understanding an individual's inability to perform social roles, was operationalized using the Social Stress Gauge developed by one of the researchers.
found that there was, in fact, an association between susto and an individual's perception of the adequacy of his or her performance of critical social roles.
11 statement
In Agar's statement above, we get the impression that a credible argument should be systematic and based on a process that informs us about how researchers came to know what they know.
Thus the plan of research is a statement that concentrates on the components that must be present in order for the objectives of the study to be realized.
" This statement illustrates at least two important elements of research design.
No amount of sophisticated statistics, computer intensive text analysis, or elegant writing can salvage a poorly designed study.
For example, the particular structure of an empirical statement or hypothesis will partially determine the manner in which theoretical concepts are operationalized and eventually analyzed.
(1996) used a static-group comparative design sampling across a range of groups that varied with respect to their values on environmental issues.
Random sampling has been a primary requirement in the proper application of parametric statistics.
Recent developments in randomization and computer-intensive methods of statistical analysis involve less restrictive assumptions concerning the data (for example, assumption of a random sample from a population or skewed, sparse, or small sample sizes), opening the way for the development of new test statistics particularly suited for the problem at hand (Noreen 1989;
Mead used what can be referred to as a static group comparison design with a conjectural treatment group.
Young and Garro's (1982) investigation of treatment choice in two Mexican communities is an example of a static-group comparison where the presence or absence of the treatment is based on selection criteria not directly under the control of the researchers.
Boster and Johnson used a static group comparison design to compare several groups of expert fishermen with a group of novice fishermen.
10 criticisms
Without such attention to good design and methodological detail, researchers leave themselves open to one of the worst criticisms of all-of being "not even wrong" (Orans 1996).
Derek Freeman's (1983) criticism of Margaret Mead's work and her findings in Samoa has led to reactions from anthropologists who come from different epistemological traditions.
The criticisms and counter-criticisms are difficult to assess, given the time between Mead's and Freeman's studies, the differences in locations of their work, and the differences in their ideological positions (Ember 1985).
Still, it is instructive to review her work through a contemporary design lens, noting how slight modifications in design and method could have thwarted later criticisms.
More attention to issues of research design and methods would have improved her chances to make valid claims and possibly limited later criticism of her work.
Some of the criticisms of the Young and Garro study apply to this example as well.
10 epistemology
A discussion of the basic arguments as related to epistemology, objectivity, reality, authority, and the like are beyond the scope of this chapter (see Schweizer in this volume).
Epistemologically, systematic work is objectivist.
Phenomenology, hermeneutics, symbolic anthropology, interpretive anthropology, interpretive interactionism, deconstructionism, postmodernism, and constructivism, to name a few, question, in one way or another, some or all of the ontology, epistemology, and methodology of systematic approaches.
Epistemologically, interpretive paradigms are subjective, with findings that are value mediated or even created.
This contrast between the old and the new reflects the increased variation in epistemological emphasis in the field that has developed over the last 30 years.
Derek Freeman's (1983) criticism of Margaret Mead's work and her findings in Samoa has led to reactions from anthropologists who come from different epistemological traditions.
10 Geertz
In contrasting Geertz and early interpretive anthropology with some of the later postmodern turns of such ethnographic writers as James Clifford, Rabinow (1986) observes:
At first glance James Clifford's work, like that of others in this volume, seems to follow naturally in the wake of Geertz's interpretive turn.
Geertz (like the other anthropologists) is still directing his efforts to reinvent an anthropological science with the help of textual mediations.
For scholars like Geertz, analysis of ethnography has less to do with the methods of observation and description than the inscriptions and writings concerning the meaning of human action.
Following in the "thick description" tradition of Geertz, Zabusky clearly believes in some kind of ethnographic authority.
10 school
However, this debate has its parallel in sociology where schools such as ethnomethodology and symbolic interactionism developed in response to the largely quantitative macro-level focus of the discipline.
The anthropologists have never studied a school system before.
It would help in this if the anthropologists were to spend half of their time studying another school that was similar, except for the new experimental program.
It would also help if the anthropologists were to study the school for a year or two prior to the program evaluation.
(This would be hard to schedule, but we might regard the current school ethnographies as pre-studies for new innovations still to come.
All knowing is comparative, however phenomenally absolute it appears, and an anthropologist is usually in a very poor position for valid comparison, as their own student experience and their secondhand knowledge of schools involve such different perspectives as to be of little comparative use.
9 error
In searching for such foundations, there is a need for objectivity, replication, and control over possible sources of error leading to a valid assessment of a given theory.
These and other sources of error are all potential rival hypotheses and randomized experiments are best at eliminating the threats of rival explanations.
Nevertheless, the principles of experimentation are instructive and are a guide for understanding potential sources of error, even in a nonlaboratory setting.
Other sources of potential bias include sampling error (that is, chance), nonresponse, the use of imprecise measures, data recording errors, informant inaccuracies, and interviewer effects (see Pelto and Pelto 1978;
Attention and concern with all the potential sources of error, whether stemming from how the study was designed, how the data were collected (for example, face-to-face interviews or mail-out surveys), or how the data were analyzed (for example, statistical conclusion validity), will help lead to the production of solid evidence.
When generalization to a target population is the objective, you should strive to define a sampling universe or frame using a selection procedure with known error limits and one that represents the population of interest.
Each of these approaches has potential problems, and most do not allow for generalizations about a population since they involve elements of unknown error even if the method involves some form of random selection criteria (for example, random selection of locations in which to intercept respondents).
In lieu of the between-culture comparisons, Mead could have made a within-case comparison that would have suffered less from problems with possible sources of error.
9 individuals
There was a wide belief among British anthropologists that fieldwork could not be taught to new recruits, but could only be experienced by individuals in the field.
For example, the pretest/posttest nonequivalent groups design controls for some internal threats to validity, but it's problematic with respect to controlling for changes due to how groups members were selected (selection maturation), changes due to how individuals were tested (instrumentation), changes due to the selection of individuals with extreme pretest measures leading to regression toward the mean (regression), and changes due to local events not a part of the study (history).
Unlike studies where informed consent is obtained prior to participation, in experiments like Melbin's, individuals often participate without knowing about it.
Individuals were asked to perform an unconstrained judged similarity of the fish-a free pile sort (see Weller, this volume, and Weller and Romney 1988).
One subsample was of individuals who complained of susto during the fieldwork or who had admitted their condition to relatives or curers.
Indeed, I interacted with several individuals who had life histories that were inconsistent with my general characterization and who were the basis for suggesting the competing models discussed above.
9 overall
The light line connecting the two categories indicates their complementarity and interrelatedness in that a design may include both within an overall research design framework.
The figure shows that the overall research process is more than just a matter of study design.
The selection of any of these designs or the development of some hybrid design depends on the overall design of the research itself.
The nature of the groups or characteristics to be compared-in terms of such things as the size of the comparison groups in the overall population, the frequency of characteristics of interest in the population, the availability of a sampling frame, the ability to identify members of the population (for example, hidden or clandestine populations)-all influence the choice of a sample design.
Implicit in this proposition is the overall theoretical notion that culture is the major factor contributing to human behavior.
As seen in Figure 2, exploratory and descriptive research are often essential components of an overall explanatory research design.
Her study design incorporates quantitative and qualitative methods in the overall ethnographic enterprise.
9 pingers
In response, the fishing industry proposed the voluntary use of "pingers" an underwater acoustic device-to keep porpoises from their nets.
Fishers petitioned the federal government to fund a study of pinger effectiveness.
The study used the classic control/treatment design in which catch rates for a set of nets with pingers were compared to catch rates for set of nets without pingers.
Both control and treatment nets were outfitted with pingers, but only the pingers on treatment nets would activate once placed in the water.
In each successive study, investigators tried to control for as many extraneous variables as possible so that the hypothesized effect could be assessed (that is, the effectiveness of pingers compared to not using pingers).
9 program
They have been hired after (or just as) the experimental program has got under way, and are inevitably studying a mixture of the old and the new under conditions in which it is easy to make the mistake of attributing to the program results which would have been there anyway.
It would help in this if the anthropologists were to spend half of their time studying another school that was similar, except for the new experimental program.
It would also help if the anthropologists were to study the school for a year or two prior to the program evaluation.
9 units
In manipulative experiments, analytical units are randomly allocated to comparative groups, whereas in mensurative experiments selection of units is based on some probability or nonprobability sampling scheme.
While random assignment aids in controlling for confounding variables by producing homogeneous comparative groups, random sampling of units produces comparison groups that are representative of such groups.
Then, FADS, umbrella-like units suspended in the water column, were alternately placed at the piers and individual fishers were interviewed simultaneously during randomly selected times at both the treatment (the pier with the FADS) and the control (the pier without the FADS) piers.
But it's not always easy to know who or what you want to sample and to know enough about these sampling units to derive a valid sample.
The selection of units of analysis, whether settings, events, times, households, or people, is important for understanding a variety of internal and external threats to validity, but it is particularly important for increasing external validity.
We mostly think of selection in terms of some type of sample units.
In one approach, the model of deliberate sampling for heterogeneity, target classes of units, whether classes or categories of persons, places, times, or events, are deliberately chosen to represent the range of such classes found in the population.
No matter what the sampling method, you should be explicit about how you chose the sampling units.
For example, Mead made an assertion concerning the relationship between the size of residential units and adolescent troubles.
She did not, however, make any systematic comparisons among the different units.
This is analogous to treatment and control groups without the random assignment of subjects to experimental units and where the treatment is implied rather than researcher directed (that is, natural differences in experience with fish).
8 background
Thus, policy emerges from interactions between groups of differing political, ideological, social, and economic backgrounds.
In lieu of equalization through randomization, Young and Garro, through extensive ethnographic background research, produced groups that, although nonequivalent in the quasi-experimental sense, shared similarities with regard to a number of important characteristics.
Using a questionnaire to gain background information, 15 college undergraduates who had the least amount of recreational fishing experience were selected from two introductory anthropology classes.
But the in-depth ethnographic background research, the particular structure of the hypothesis, and the overwhelming reliability of informant responses make for more confidence in the possible validity of the study's conclusions.
Beyond these constancies of cultural and marital experience, they were selected to maximize diversity with regard to such obvious differences as their occupations and educational backgrounds, religious affiliations and ethnic and racial identities, their neighborhoods and social networks, and the duration of their marriages.
8 classic
The study used the classic control/treatment design in which catch rates for a set of nets with pingers were compared to catch rates for set of nets without pingers.
Thus, fishers were blind as to which nets were control and which were treatment-a classic double-blind experimental design.
Hurlbert (1984) emphasizes this in a classic paper on the design of field experiments in ecology.
This is essential if we are to generalize to a whole population and is generally, though not always, a requirement for classical statistical tests.
Using statistical and descriptive inference, the authors concluded that whether informants use form or function for classification depends on the knowledge base of the informants and the methods used to test their knowledge (see Figure 4).
8 face
While the purpose of experimental design is to ward off as threats to validity, there are several types of validity-face, construct, statistical conclusion, internal, external, etc.
Attention and concern with all the potential sources of error, whether stemming from how the study was designed, how the data were collected (for example, face-to-face interviews or mail-out surveys), or how the data were analyzed (for example, statistical conclusion validity), will help lead to the production of solid evidence.
That is, finding commonality in the face of diversity provides stronger evidence of a shared cultural model (Johnson and Griffith 1996).
8 Handbook
Some early exceptions include Brim and Spain's (1974) book on hypothesis-testing designs, Pelto and Pelto's (1978) book on research methodology in cultural anthropology, and Naroll and Cohen's (1973) A Handbook of Method in Cultural Anthropology, which has several chapters that address issues in research design (LeVine 1973;
In Handbook of Qualitative Research.
In Handbook of Qualitative Research.
In A Handbook of Methods in Cultural Anthropology.
A Handbook of Method in Cultural Anthropology.
In Handbook of Qualitative Research.
In A Handbook of Methods in Cultural Anthropology.
In A Handbook of Method in Cultural Anthropology.
8 helpfulness
Among other things, they provide escape and opportunity, tolerate a wider range of behaviors, consist of isolated settlements, have fewer status distinctions, involve novel hardships, have decentralized authority, involve lawlessness and peril, have a reputation for helpfulness and sociability, lag in the development of policies to exploit and regulate, and involve a variety of interest group conflicts.
Melbin conducted four tests of the feature relating to helpfulness and sociability.
" However, Melbin's three other tests supported the hypothesis of more sociability and helpfulness at night.
Had Melbin conducted only the key experiment, he may have come to very different conclusions regarding the helpfulness and sociability of night-timers.
8 ideological
Thus, policy emerges from interactions between groups of differing political, ideological, social, and economic backgrounds.
The criticisms and counter-criticisms are difficult to assess, given the time between Mead's and Freeman's studies, the differences in locations of their work, and the differences in their ideological positions (Ember 1985).
Freeman contended that some of Mead's informants lied to her and that Mead's commitment to a particular ideological position caused her to evaluate evidence incorrectly.
For some researchers, the shift from Catholicism to Protestantism helps account for economic change, as Protestantism is more compatible with capitalist ideology and the accumulation of wealth.
Goldin (1996) wanted to understand the relationships among religious affiliation, economic ideology, occupation, and economic status in a Guatemalan township (Almolonga).
8 literary
His earliest contributions, however, were more a demonstration of the value of ethnographic writing-his "unusual literary sense" (Lowie 1937:231)--rather than of methodological details of proper ethnographic fieldwork (Ellen 1984).
Thus, there is less focus on the means of research, such as methods of data collection and analysis as found in the systematic strategies, and more on the ends of research-the ethnographic or literary product.
In many ways, this blurs the distinction between what is anthropological and what is literary.
If we talk of an interpretive method, particularly with regard to postmodernism, it more than likely involves both the researcher's immersion into the cultural context of the actor(s) and some means, usually literary, for conveying the understanding gained from such an immersion.
Instead, Ramos emphasizes the emergent and reflexive nature of data and the literary strategies used in producing the ethnographic product.
8 literature
The effectiveness of the device, however, was in question, and there was no firm evidence in the literature about it.
Even in the late 1960s, when concern for methodological rigor was probably at its peak in anthropology, many treatments of research methods and design in the literature played down the need for more systematic methods and design detail, particularly with respect to hypothesis-testing approaches (LeVine 1973).
Explanatory: Explanatory approaches generally involve testing elements of theory that may already have been proposed in the literature or that have been informed by exploratory research.
There is a vast literature on sampling theory and random sampling procedures, including discussions of sample sizes (see, for example, Bernard [1994] for a summary and Babbie [1990] for detailed discussion of sampling issues).
There is a body of literature that views the interaction of culture with the individual as so deeply unique and personal as to not be researchable in terms of cultural universals, coherence, or even sharing.
8 texts
In the American tradition texts provided what was regarded as an objective body of data, whereas the British tradition was more a matter of subjective experience.
Geertz (like the other anthropologists) is still directing his efforts to reinvent an anthropological science with the help of textual mediations.
The core activity is still social description of the other, however modified by new conceptions of discourse, author, or text.
he must constantly feed off others' texts.
More extreme forays into experimental ethnography have blurred this distinction even further, and there is more of a focus on writing strategies that include such approaches as montages, evocative representations, polyvocal texts, and even ethnographic fictions (Denzin and Lincoln 1994).
This is particularly true today, given the large number of computer analytical packages available for analyzing text (see Bernard and Ryan, this volume).
Currently, the qualitative analysis of text and discourse is no longer: restricted to either interpretive or exploratory approaches, but can also be used in hypothesis testing and explanatory research.
No amount of sophisticated statistics, computer intensive text analysis, or elegant writing can salvage a poorly designed study.
Figure 2 shows that the research process involves a simultaneous concern for the development of empirical statements from theory (for example, hypotheses), the operationalization of theoretical concepts (for example, meaningful and reliable measures), design (for example, groups to be studied), data collection (for example, qualitative versus quantitative), and data analysis (for example, multiple regression and text analysis).
8 ward
For any given research problem, it is the purpose of research design to ward off as many threats to validity as possible.
The purpose of research design is to ward off as many threats to validity as possible and to help one eliminate competing hypotheses.
While the purpose of experimental design is to ward off as threats to validity, there are several types of validity-face, construct, statistical conclusion, internal, external, etc.
In principle, this is similar to Cook and Campbell's (1979) model of deliberate sampling for heterogeneity as one of several means for warding off threats to external validity.
7 ed
Behar discusses the multiplexity of roles, in that she was variously involved as "priest, interviewer, collector, transcriber, translator, analyst, academic, connoisseur, editor, and peddler" (p. 12).
Survey Research Methods, 2d ed.
Research Methods in Anthropology: Qualitative and Quantitative Approaches, 2d ed.
In Methodology and Epistomology for Social Science: Selected Papers E. S. Overman, ed.
N. K. Denzin and Y. S. Lincoln, eds.
R F. Ellen, ed.
S. Wasserman and J. Galaskiewicz, eds.
R. Pollnac and J. Poggie, eds.
Johnson, J. C., and R. Pollnac, eds.
N. K. Denzin and Y. S. Lincoln, eds.
T. Rhodes and R. Hartnoll, eds.
R. Naroll and R. Cohen, eds.
Qualitative Data Analysis, 2d ed.
Moran, E. F., ed.
Naroll, R., and R. Cohen, eds.
Anthropological Research: The Structure of Inquiry, 2d ed.
C. Furlow, ed.
J. Clifford and G. E. Marcus, eds.
N. K. Denzin and Y. S. Lincoln, eds.
R. Naroll and R. Cohen, eds.
R. Naroll and R. Cohen, eds.
L. Nencel and P. Pels, eds.
R. F. Ellen ed.
In the series Studies in Anthropological Method, George Spindler and Louise Spindler, eds.
7 enterprise
In this enterprise, explanation can involve a general search for causality or prediction.
For some, interpretive work is an exploratory enterprise with an implicit concern for methodological issues.
For others, interpretive work is concerned more with the strategies and methods of ethnographic presentation and with the reflexive character of the ethnographic enterprise.
Her study design incorporates quantitative and qualitative methods in the overall ethnographic enterprise.
The strength of the ethnographic approach is its ability to incorporate a wide range of methods, strategies, and designs within a single enterprise, all combining in ways to improve the chances for credible results.
7 Garro
Young and Garro's (1982) investigation of treatment choice in two Mexican communities is an example of a static-group comparison where the presence or absence of the treatment is based on selection criteria not directly under the control of the researchers.
Young and Garro took a different stance, stressing physician accessibility as the most important determinant of physician use.
From a random sample of approximately 10% of the households in each of the towns, Young and Garro collected data on the number of illnesses that had occurred during the previous two months and the treatment each had received.
Young and Garro tested the two main hypotheses in sequence.
This established, Young and Garro could then test the second hypothesis relating to the similarity in beliefs between the two communities.
Using multidimensional scaling, Young and Garro (1982) compared the belief data and found striking similarities in the medical beliefs of communities.
In Young and Garro's case, a visual inspection of the graphical representations of the data could lead to no other conclusion than that there was little or no difference in beliefs between the two communities (see Figure 3).
It is unrealistic to suppose that Young and Garro could have randomly assigned community members to the different comparison groups in order to control for confounding variables and then subject their informants to the treatments of interest.
In lieu of equalization through randomization, Young and Garro, through extensive ethnographic background research, produced groups that, although nonequivalent in the quasi-experimental sense, shared similarities with regard to a number of important characteristics.
Some of the criticisms of the Young and Garro study apply to this example as well.
Young, J. C., and L. Y. Garro.
7 Murray
Johnson and Murray (1997), for example, used a quasi-experimental design to evaluate the use of fish aggregation devices (FADS) in small-scale fisheries development projects.
Johnson and Murray compared and determined catch rates.
Johnson and Murray 1997).
Similarly, advances in computer-intensive methods for testing hypotheses have the potential to expand the range of designs possible, particularly in the imperfect world of fieldwork (Johnson and Murray 1997).
Johnson, J. C., and J. D. Murray.
7 Networks
Such matters are particularly germane for observational designs using various social network approaches (see Johnson [ 1994] for a review).
Beyond these constancies of cultural and marital experience, they were selected to maximize diversity with regard to such obvious differences as their occupations and educational backgrounds, religious affiliations and ethnic and racial identities, their neighborhoods and social networks, and the duration of their marriages.
Anthropological Contributions to the Study of Social Networks: A Review.
In Advances in Social Network Analysis.
Development of Social Networks in Preschool Children.
7 porpoises
There has been similar concern over the incidental catch of harbor porpoises by net fishers in New England (Schneider 1996).
Wildlife conservationists petitioned the U.S. federal government in 1991 to declare harbor porpoises a threatened species.
In response, the fishing industry proposed the voluntary use of "pingers" an underwater acoustic device-to keep porpoises from their nets.
In the first experiment, the control net caught 10 porpoises while the treatment net caught none.
Some conservationist groups claimed the study was biased in that the treatment nets were placed in areas known not to have large numbers of porpoises.
Again the evidence was impressive: The treatment nets caught 2 porpoises (1 was thought to be deaf), while the control nets caught 25.
7 regression
In sociology, multiple regression models, structural equation models, and path analytic models (all related analytical techniques) have influenced the design of survey research.
Figure 2 shows that the research process involves a simultaneous concern for the development of empirical statements from theory (for example, hypotheses), the operationalization of theoretical concepts (for example, meaningful and reliable measures), design (for example, groups to be studied), data collection (for example, qualitative versus quantitative), and data analysis (for example, multiple regression and text analysis).
For example, the pretest/posttest nonequivalent groups design controls for some internal threats to validity, but it's problematic with respect to controlling for changes due to how groups members were selected (selection maturation), changes due to how individuals were tested (instrumentation), changes due to the selection of individuals with extreme pretest measures leading to regression toward the mean (regression), and changes due to local events not a part of the study (history).
7 relationship
Figure 2 illustrates the relationship between exploratory and explanatory approaches within the ethnographic context.
Random allocation produces equivalent comparison groups, and artificial manipulation of independent variables (also known as explanatory variables or study factors), with all other variables or factors controlled for, allows for the most valid assessment of the causal relationship between the independent and dependent variables or response variables.
For example, Mead made an assertion concerning the relationship between the size of residential units and adolescent troubles.
One of the central concerns of medical anthropologists has been to better understand the relationship between health-related behaviors and native perceptions about illness.
The authors were interested in three primary hypotheses relating to role performance and the presence of the illness, psychiatric impairment and susto, and relationship between organic disease and susto.
The ultimate aim of the study was to show the relationship between various social forces and susto susceptibility.
Although there was no association between susto and psychiatric impairment, there was a relationship between susto and the suffering of more organic disease signs.
Goldin (1996) wanted to understand the relationships among religious affiliation, economic ideology, occupation, and economic status in a Guatemalan township (Almolonga).
First, qualitative approaches were used to suggest different mechanisms and relationships that might be operating within Almolonga.
7 sciences
And since I have indicated here that research in culture involves a great deal of unique personal experience for the anthropologist, I have taken the position that it is probably unlikely there can be a rigorous, systematic, and formal presentation of methods in the study of culture like those of the natural sciences and that there are overriding concerns among many sociologists, psychologists, and economists.
Although some of the older interpretive strategies that emerged from the scientific tradition in the social sciences, such as early interpretive anthropology, still adhered to some logical empiricist methodology and maintained a degree of belief in ethnographic authority, more recent approaches, such as postmodernism and constructivism, are more radical in their sweeping rejection of scientific method and design logic (see Schwandt 1994).
This contrast between explanatory and descriptive or exploratory approaches is commonly made in nonexperimental disciplines in both the natural and social sciences.
What's Wrong with the Social Sciences?
7 similarly
She had concerns-and I believe she thought her mentor, Boas, would feel similarly-as to whether a valid comparison of this type could be made given the selection process for her sample of girls.
Community ecologists, for example, similarly distinguish between exploratory or descriptive studies that seek to describe and determine patterns in ecological data and those studies that specifically seek to predict or test hypotheses.
Similarly, threats to external validity, such as problems stemming from biased samples or research in atypical or unique settings, can hamper the generalizability of one's findings.
Similarly, advances in computer-intensive methods for testing hypotheses have the potential to expand the range of designs possible, particularly in the imperfect world of fieldwork (Johnson and Murray 1997).
7 site
Some things, like site selection, sampling, measurement, and recording are at least partly within our control.
A good understanding of the research problem and the research site allows us to plan for some contingencies, but there is no research design crystal ball.
others may require a description of the research problem and site but require less detail about the methods of data collection and analysis.
It involves details about getting into and out of the field situation, travel arrangements, getting proper government permissions, making contacts at the field site, arranging for living accommodations, and so on.
She also discusses the rationales for selecting the site and the group she studied, problems of working in a linguistically and technically diverse social milieu, the use of semistructured and unstructured interviews, and the effect of her role as ethnographer on informant relations and data quality.
Orans's work was, of course, many years after Mead's, and he worked at different field sites than did Mead.
7 sociology
However, this debate has its parallel in sociology where schools such as ethnomethodology and symbolic interactionism developed in response to the largely quantitative macro-level focus of the discipline.
And since I have indicated here that research in culture involves a great deal of unique personal experience for the anthropologist, I have taken the position that it is probably unlikely there can be a rigorous, systematic, and formal presentation of methods in the study of culture like those of the natural sciences and that there are overriding concerns among many sociologists, psychologists, and economists.
In sociology, multiple regression models, structural equation models, and path analytic models (all related analytical techniques) have influenced the design of survey research.
Whereas the analytical techniques most often used in psychology, sociology, and economics often led to rather standard designs, in anthropology the eclectic nature of ethnography leaves the design of research more open ended.
7 stories
In such worlds, a systematic argument enjoys a star-spangled legitimacy.
Agar (1980) and Bernard (1994) relate stories about Kroeber's recommendations regarding the teaching and conduct of ethnographic research.
In the stories, one concerning Wagley's teaching of a field methods course and one concerning a graduate student at Berkeley asking for advice before going to the field, Kroeber's response was a terse, one liner that reflected the attitude of the times.
I find this stance comfortable, for it is my conviction that so long as prime theoretical concerns in the study of culture are an attempt to record and understand the native's view of his culture and the objective and historical realities of culture, then methods for field study will have to reflect the end purpose of making a whole account of a part of the human experience.
Further, the purpose of research strategies under these interpretive paradigms is more focused on the production of a believable or plausible account or story rather than a single depiction of the truth, since it is thought that there are a multitude of plausible accounts rather than just a single true story.
While systematic analytical paradigms are primarily concerned with threats to validity, recent interpretive paradigms are focused more on threats to believability -as in "Do you believe my story?
Other examples in this vein include Panourgia's (1995) use of we and they in her "Athenian Anthropography" and Behar's (1993) use of montage in her collaboration with a single woman in the telling of that woman's life story.
Young and Garro took a different stance, stressing physician accessibility as the most important determinant of physician use.
A Post-modern In-stance.
6 American
In the American tradition texts provided what was regarded as an objective body of data, whereas the British tradition was more a matter of subjective experience.
The comparison group, Samoan adolescent girls, was compared to a conjectural treatment group, American adolescent girls, to test the proposition that exposure to Western civilization increases adolescent trauma.
In her use of a conjectural treatment group, Mead assumed some things about American adolescents without collecting comparable data.
An example of research in this mode is Naomi Quinn's (1996) development of Americans' cultural models concerning marriage.
Based on in-depth interviews with 22 informants, Quinn (1996) attempted to build a cultural model of Americans' reasoning about marriage.
all were native-born Americans who spoke English as a first language;
American Anthropologist 22(4):311-321.
American Anthropologist 91(4): 866-889.
American Anthropologist 87(4):906-910.
Environmental Values in American Culture.
Scientific American (September):40-12.
American Anthropologist 98(3):555-567.
6 characteristics
Mensurative designs, then, are observational and characteristic of the types of comparative designs found in field studies in anthropology.
The nature of the groups or characteristics to be compared-in terms of such things as the size of the comparison groups in the overall population, the frequency of characteristics of interest in the population, the availability of a sampling frame, the ability to identify members of the population (for example, hidden or clandestine populations)-all influence the choice of a sample design.
In lieu of equalization through randomization, Young and Garro, through extensive ethnographic background research, produced groups that, although nonequivalent in the quasi-experimental sense, shared similarities with regard to a number of important characteristics.
Ethnobiologists have long debated whether folk biological classifiers are natural historians who compare animals on the basis of their morphological characteristics or pragmatists who compare on the basis of the utility of organisms.
Further, beliefs about the use and functional characteristics of the fish obtained from extensive ethnographic interviews were turned into a sentence-frame completion task described by Weller and Romney (1988).
Boster and Johnson used statistical and graphical methods to evaluate whether experts' and novices' judgments of fish, at the aggregate and individual levels, were closer to the morphological characteristics of fish (taxonomic distance) or the uses of fish (beliefs about use).
6 Denzin
In contrast to the field-worker-as-writer, we find the writer-as-field-worker (Denzin and Lincoln 1994).
More extreme forays into experimental ethnography have blurred this distinction even further, and there is more of a focus on writing strategies that include such approaches as montages, evocative representations, polyvocal texts, and even ethnographic fictions (Denzin and Lincoln 1994).
Denzin, N. K., and Y. S. Lincoln.
N. K. Denzin and Y. S. Lincoln, eds.
N. K. Denzin and Y. S. Lincoln, eds.
N. K. Denzin and Y. S. Lincoln, eds.
6 folk
Using a standard chi-square test, the authors found a significant difference in the frequency distribution of treatment alternatives between the two towns, with the exception of folk curers.
Ethnobiologists have long debated whether folk biological classifiers are natural historians who compare animals on the basis of their morphological characteristics or pragmatists who compare on the basis of the utility of organisms.
Susto, A Folk Illness
(1985) report on a study of a folk illness known as susto, found in many cultural groups throughout North and South America.
Folk beliefs surrounding susto attribute loss of a critical substance or force due to a frightening experience.
Susto, A Folk Illness.
6 frame
The nature of the groups or characteristics to be compared-in terms of such things as the size of the comparison groups in the overall population, the frequency of characteristics of interest in the population, the availability of a sampling frame, the ability to identify members of the population (for example, hidden or clandestine populations)-all influence the choice of a sample design.
When generalization to a target population is the objective, you should strive to define a sampling universe or frame using a selection procedure with known error limits and one that represents the population of interest.
For some populations, it may be impossible to develop a sampling frame from which to draw a sample.
Later, the researchers collected triads data and what they call term-frame data on informants' perceived similarity of illnesses.
On the basis of the data from the triads study and the term-frame interviews, we see little reason to reject the "null hypothesis" of no significant differences between the responses of the two groups of informants.
Further, beliefs about the use and functional characteristics of the fish obtained from extensive ethnographic interviews were turned into a sentence-frame completion task described by Weller and Romney (1988).
6 Lincoln
In contrast to the field-worker-as-writer, we find the writer-as-field-worker (Denzin and Lincoln 1994).
More extreme forays into experimental ethnography have blurred this distinction even further, and there is more of a focus on writing strategies that include such approaches as montages, evocative representations, polyvocal texts, and even ethnographic fictions (Denzin and Lincoln 1994).
Denzin, N. K., and Y. S. Lincoln.
N. K. Denzin and Y. S. Lincoln, eds.
N. K. Denzin and Y. S. Lincoln, eds.
N. K. Denzin and Y. S. Lincoln, eds.
6 links
Second, an idealized plan gives guidelines for linking theory to the methods of data collection and analysis that yield either valid or "defensible" results.
There is no substitute for a good theory, and there is a critical need to link theory, design, data collection, analysis, and interpretation in a coherent fashion.
It involves constructing a logical plan that links all the elements of research together so as to produce the most valid assessment possible of some theory, given some set of realistic constraints (for example, cost, scope, geographical setting, etc.
Based on an in-depth analysis of informants' discourse about marriage, Quinn produced a cultural model incorporating a number of causal links in informants' reasoning as to a "lasting marriage" (see Figure 5).
6 Quinn
An example of research in this mode is Naomi Quinn's (1996) development of Americans' cultural models concerning marriage.
In contrast, Quinn views culture as being shared-that there are cultural models for a variety of domains that are widely held in common, and that these models can be developed from the discourses of cultural members.
Based on in-depth interviews with 22 informants, Quinn (1996) attempted to build a cultural model of Americans' reasoning about marriage.
Although not generally representative of either the regional population or of the population of the United States, Quinn claims that her sample of informants represents the regions' population in terms of the high degree of recent in-migration to the area from regions outside the South.
Based on an in-depth analysis of informants' discourse about marriage, Quinn produced a cultural model incorporating a number of causal links in informants' reasoning as to a "lasting marriage" (see Figure 5).
Although the model appeared to be widely shared among informants from Quinn's sample and data collected from other studies on marriage, research still has to be designed to test this model across settings and researchers.
Quinn was careful and diligent in her selection of informants, and her diligence certainly contributes to the potential validity of her model.
Quinn, N. 1996.
6 replication
Further, the huge investment in time and resources limited another important goal of science, that of replication, since an ethnographer couldn't realistically be expected to replicate someone else's work.
The "my natives" or "my village" mentality of some and the fact that careers were made by discovering new theories or describing exotic less well-known cultures has certainly inhibited replication efforts (Johnson 1990).
In searching for such foundations, there is a need for objectivity, replication, and control over possible sources of error leading to a valid assessment of a given theory.
This increases the chances of detecting potential bias and also makes replication feasible.
Replication is extremely important to external and other types of validity, such as construct validity.
This is an excellent example of a study design that incorporates within-study replication or multiple tests of a theory.
5 argument
In such worlds, a systematic argument enjoys a star-spangled legitimacy.
We need a way to argue what we know based on the process by which we came to know it.
In a complex world of competing arguments, who is to be believed or trusted?
In Agar's statement above, we get the impression that a credible argument should be systematic and based on a process that informs us about how researchers came to know what they know.
A discussion of the basic arguments as related to epistemology, objectivity, reality, authority, and the like are beyond the scope of this chapter (see Schweizer in this volume).
As a result, the assessment of any theory involves research designs more heavily concerned with the means-the research process, rather than simply the way the study was written or argued-since the validity of study results depends on the scientific soundness of the research design.
others have pointed to the biases and flaws in Freeman's argument (Marcus 1983;
There was a lack of comparisons between various sources of data that were crucial to Mead's argument.
5 believable
Design, on the other hand, involves the methodological and analytical details that contribute to the credibility, validity, believability, or plausibility of any study.
In this chapter I concentrate on elements of design related to the production of valid results or a believable ethnographic account.
Further, the purpose of research strategies under these interpretive paradigms is more focused on the production of a believable or plausible account or story rather than a single depiction of the truth, since it is thought that there are a multitude of plausible accounts rather than just a single true story.
While systematic analytical paradigms are primarily concerned with threats to validity, recent interpretive paradigms are focused more on threats to believability -as in "Do you believe my story?
The newer forays into experimental and other ethnographic forms of presentation are more reflexive in character and more concerned with believable and moving representations rather than the production of valid accounts or conclusions.
5 character
As stated, many interpretive studies are closer in character to exploratory and descriptive research in the systematic mode than to some of the more extreme postmodern studies.
For others, interpretive work is concerned more with the strategies and methods of ethnographic presentation and with the reflexive character of the ethnographic enterprise.
The bad news is that the open-ended character of ethnography contributes to a less well-focused discussion of research design issues in ethnographic approaches.
The newer forays into experimental and other ethnographic forms of presentation are more reflexive in character and more concerned with believable and moving representations rather than the production of valid accounts or conclusions.
5 Ellen
Nevertheless, a well-articulated project design helps "to promote the effective conduct of research," whether one starts from a positivist or humanist perspective (Ellen 1984:158).
Yet, despite his concern for scientific method, Boas was more explicit about his methods of data analysis than about his methods of fieldwork and data collection (Ellen 1984;
His earliest contributions, however, were more a demonstration of the value of ethnographic writing-his "unusual literary sense" (Lowie 1937:231)--rather than of methodological details of proper ethnographic fieldwork (Ellen 1984).
Ellen, R. F. 1984.
R F. Ellen, ed.
R. F. Ellen ed.
5 fishermen
Boster and Johnson used a static group comparison design to compare several groups of expert fishermen with a group of novice fishermen.
To ensure that experts were, in fact, experienced recreational fishermen, the rosters of sport fishing clubs in each region were sampled at random.
Each of the four expert groups comprised 15 subjects chosen at random from a larger sample of recreational fishermen.
5 focused
Further, the purpose of research strategies under these interpretive paradigms is more focused on the production of a believable or plausible account or story rather than a single depiction of the truth, since it is thought that there are a multitude of plausible accounts rather than just a single true story.
While systematic analytical paradigms are primarily concerned with threats to validity, recent interpretive paradigms are focused more on threats to believability -as in "Do you believe my story?
The bad news is that the open-ended character of ethnography contributes to a less well-focused discussion of research design issues in ethnographic approaches.
As with research in community ecology, ethnographic research can be purely exploratory or descriptive involving a research process focused on producing better theory-or purely explanatory, although this is usually not the case.
Thus, most large epidemiological studies of IDUs focused mainly on direct sharing behaviors in attempts to understand seroconversion rates and other risk factors.
5 funding
There is much variation in what funding agencies and foundations expect regarding research design.
All funding agencies expect a well-organized outline of the proposed project-one that meets the design expectations of peer reviewers and agency personnel.
Fishers petitioned the federal government to fund a study of pinger effectiveness.
Lobbying efforts by fishers yielded more funds for a larger, more comprehensive study involving more than 10,000 fishing nets.
5 households
The selection of units of analysis, whether settings, events, times, households, or people, is important for understanding a variety of internal and external threats to validity, but it is particularly important for increasing external validity.
For example, Mead could have compared girls living in the households of native pastors to those who did not.
Similar to the observation by Brim and Spain, Orans points out that Mead made no comparison of sexual behavior between girls living in a native pastor's household and girls living with their own family.
From a random sample of approximately 10% of the households in each of the towns, Young and Garro collected data on the number of illnesses that had occurred during the previous two months and the treatment each had received.
Using her experience as a participant observer, Goldin developed a survey which she applied to a random sample of 10% of the heads of households in the township (n = 57).
5 human
Nevertheless, a well-articulated project design helps "to promote the effective conduct of research," whether one starts from a positivist or humanist perspective (Ellen 1984:158).
I find this stance comfortable, for it is my conviction that so long as prime theoretical concerns in the study of culture are an attempt to record and understand the native's view of his culture and the objective and historical realities of culture, then methods for field study will have to reflect the end purpose of making a whole account of a part of the human experience.
For scholars like Geertz, analysis of ethnography has less to do with the methods of observation and description than the inscriptions and writings concerning the meaning of human action.
Thus, comparison groups may be chosen on the basis of different levels of exposure to some naturally occurring or human-induced phenomena (for example, natural disaster, war, or the building of a dam).
Implicit in this proposition is the overall theoretical notion that culture is the major factor contributing to human behavior.
Human Ecology 24(1):87-110.
Journal of Acquired Immune Deficiency Syndromes and Human Retrovirology 12:202-207.
The Comparative Analysis of Human Societies: Toward Common Standards for Data Collection and Reporting.
In Anthropology Between Science and the Humanities.
Constructivist, Interpretivist Approaches to Human Inquiry.
5 Kleinbaum
I borrow terminology from Kleinbaum et al.
It is important, though, to contrast quasi-experiments to what Kleinbaum et al.
Table 1 describes examples from observational and quasi-experimental study designs discussed by Kleinbaum et al.
Kleinbaum et al.
Kleinbaum, D. G., L. L. Kupper, and H. Morgenstern.
5 question
The effectiveness of the device, however, was in question, and there was no firm evidence in the literature about it.
Interpretive strategies, on the other hand, differ from systematic approaches in that they question a researcher's ability to maintain objectivity, particularly in the ethnographic context where the ethnographer is often the instrument of measurement.
Phenomenology, hermeneutics, symbolic anthropology, interpretive anthropology, interpretive interactionism, deconstructionism, postmodernism, and constructivism, to name a few, question, in one way or another, some or all of the ontology, epistemology, and methodology of systematic approaches.
He can invent his questions with few constraints;
Exploratory research may contribute to the production of reliable and valid measures, provide information essential for constructing comparison groups, facilitate construction of structured questions or questionnaires, or provide information necessary for producing a sound probability or nonprobability sample.
5 Ramos
Ramos (1995), for example, has recently published an ethnography based on a rewrite of her 1972 ;
As Ramos sees it, "I found myself making forays into the self-conscious meanderings of reflexive anthropology in order to shift the axis of analysis from the skeleton-like dissertation to the flesh and blood of ethnography" (p. 6).
Although Ramos discusses informant interviewing and various sources of data, her introduction is largely devoted to discussions of her reliance on her own memory in writing the ethnography and the shift in the narrative between synchrony and diachrony.
Instead, Ramos emphasizes the emergent and reflexive nature of data and the literary strategies used in producing the ethnographic product.
Ramos, A. R. 1995.
5 Stinchcombe
For Stinchcombe (1987:23), the observations produced by how a study was designed are fundamental to the proper assessment of empirical evidence: "We always want to reject evidence if it can be explained by the design of the research or by a large number of small, unorganized causes.
(Stinchcombe [1987] provides an excellent discussion of how empirical statements are derived from theory.
This example illustrates nicely the importance of not relying on a single test, but having multiple tests and measures (Stinchcombe 1987).
Multiple tests are always much more convincing than a single test (Stinchcombe 1987).
Stinchcombe, A. L. 1987.
5 water
Media campaigns in the U.S. showing pictures of dolphins being caught in nets (generally not in U.S. waters), contributed to Florida's totally banning fishing nets-even though no marine mammals were threatened by the use of nets in Florida waters.
Both control and treatment nets were outfitted with pingers, but only the pingers on treatment nets would activate once placed in the water.
Then, FADS, umbrella-like units suspended in the water column, were alternately placed at the piers and individual fishers were interviewed simultaneously during randomly selected times at both the treatment (the pier with the FADS) and the control (the pier without the FADS) piers.
Termed "indirect sharing," these nine behaviors can promote the transmission of HIV among IDUs who, although not sharing needles directly, often share water for mixing of drugs or for rinsing syringes, share drug-mixing containers (cookers and spoons), share cottons for filtering, and share the actual drug solution itself.
5 Western
The comparison group, Samoan adolescent girls, was compared to a conjectural treatment group, American adolescent girls, to test the proposition that exposure to Western civilization increases adolescent trauma.
That is, any observed difference between the two groups with respect to the dependent variable, adolescent trauma, might have been due to one or several extraneous (unmeasured) factors and might have had nothing to do with the independent variable, exposure to Western culture.
In his work on "colonizing the night," Melbin (1987) theorized that the night was a frontier, not unlike the western United States in the nineteenth century.
An important issue in this area of research concerns the factors influencing the use of Western treatments among non-Western populations.
The research design included the comparison of two Mexican communities that were similar in terms of cultural traditions and economies but varied in terms of access to Western medical services.
Thus, the two communities seemed to differ in their use of Western medical services.
5 Zabusky
Thus, the rather simple characterization of research strategies found in Figure 1 attempts to recognize the variation inherent in the range of work found in contemporary anthropology by placing "interpretive anthropology" adjacent to "exploratory/descriptive" (see, for example, the work of Zabusky 1995).
A good example of this is Zabusky's (1995) ethnographic study of cooperation in European space science that she admits "took the form of mutual exploration rather than unidirectional examination" (p. 46).
Following in the "thick description" tradition of Geertz, Zabusky clearly believes in some kind of ethnographic authority.
Although Zabusky doesn't talk specifically about design or about concerns for potential threats to validity, there is implicit concern for such issues throughout the ethnography.
In contrast to Zabusky, there is a body of interpretive work in anthropology that is more extreme in its rejection of systematic design issues.
Zabusky, S. E. 1995.
4 choice
The nature of the groups or characteristics to be compared-in terms of such things as the size of the comparison groups in the overall population, the frequency of characteristics of interest in the population, the availability of a sampling frame, the ability to identify members of the population (for example, hidden or clandestine populations)-all influence the choice of a sample design.
Young and Garro's (1982) investigation of treatment choice in two Mexican communities is an example of a static-group comparison where the presence or absence of the treatment is based on selection criteria not directly under the control of the researchers.
Termed the "conceptual-incompatibility" hypothesis, a number of studies have suggested that such a congruence was the primary determinant of treatment choice among Third World peoples.
An important element of this position is that traditional medical beliefs are not a barrier to choice of physician treatment.
They had to establish differences in treatment choice behavior in the two communities before they could assess any hypotheses concerning differences in beliefs.
Variation in the Choice of Treatment in Two Mexican Communities.
4 Clifford
In contrasting Geertz and early interpretive anthropology with some of the later postmodern turns of such ethnographic writers as James Clifford, Rabinow (1986) observes:
At first glance James Clifford's work, like that of others in this volume, seems to follow naturally in the wake of Geertz's interpretive turn.
The other for Clifford is the anthropological representation of the other.
This means that Clifford is simultaneously more firmly in control of his project and more parasitical.
J. Clifford and G. E. Marcus, eds.
4 discipline
However, this debate has its parallel in sociology where schools such as ethnomethodology and symbolic interactionism developed in response to the largely quantitative macro-level focus of the discipline.
In Britain the claims that anthropology not only studied a distinctive body of data but also that it possessed a sophisticated methodology to collect these data, was an important factor in the establishment of anthropology as a discipline.
In some social science disciplines, like psychology, the design of research is driven by features of the analysis.
This contrast between explanatory and descriptive or exploratory approaches is commonly made in nonexperimental disciplines in both the natural and social sciences.
4 earlier
His earliest contributions, however, were more a demonstration of the value of ethnographic writing-his "unusual literary sense" (Lowie 1937:231)--rather than of methodological details of proper ethnographic fieldwork (Ellen 1984).
Theoretical knowledge is derived either from earlier studies or from exploratory work.
In most earlier research on injection drug users (IDUs) and HIV risk, the primary risk factor was viewed in terms of direct needle sharing.
Based on earlier ethnographic research, social stress, an important component for understanding an individual's inability to perform social roles, was operationalized using the Social Stress Gauge developed by one of the researchers.
4 ecology
Community ecologists, for example, similarly distinguish between exploratory or descriptive studies that seek to describe and determine patterns in ecological data and those studies that specifically seek to predict or test hypotheses.
As with research in community ecology, ethnographic research can be purely exploratory or descriptive involving a research process focused on producing better theory-or purely explanatory, although this is usually not the case.
Hurlbert (1984) emphasizes this in a classic paper on the design of field experiments in ecology.
This distinction between experimental and observational approaches is similar to one in ecological field studies.
Pseudoreplication and Design of Ecological Field Experiments.
Ecological Monographs 54(2):187-211.
Human Ecology 24(1):87-110.
4 fade
Johnson and Murray (1997), for example, used a quasi-experimental design to evaluate the use of fish aggregation devices (FADS) in small-scale fisheries development projects.
Then, FADS, umbrella-like units suspended in the water column, were alternately placed at the piers and individual fishers were interviewed simultaneously during randomly selected times at both the treatment (the pier with the FADS) and the control (the pier without the FADS) piers.
4 Guide
Nevertheless, the principles of experimentation are instructive and are a guide for understanding potential sources of error, even in a nonlaboratory setting.
In Ethnographic Research: A Guide to General Conduct.
In Ethnographic Research: A Guide to General Conduct.
Learning from the Field: A Guide from Experience.
4 historical
I find this stance comfortable, for it is my conviction that so long as prime theoretical concerns in the study of culture are an attempt to record and understand the native's view of his culture and the objective and historical realities of culture, then methods for field study will have to reflect the end purpose of making a whole account of a part of the human experience.
The historical tension between interpretive and scientific approaches in anthropology has given way to an outright rejection by some anthropologists of science and its logic of design.
The idea of a montage as an organizing principle was also central to Taussig's (1987) historical and ethnographic account of shamanism, colonialism, and terror in South America.
" Second, Taussig uses the "principle of montage" as a means, at least in his view, for better relating the lessons of history.
As against the magic of academic rituals of explanation which, their alchemical promise of yielding system from chaos, do nothing to ruffle the placid surface of this natural order, I choose to work with a different conflation of modernism and the primitivism it conjures into life-namely the carrying over into history of the principle of montage, as I learned that principle not only from terror, but from Putumayo shamanism with its adroit, albeit unconscious, use of the magic of history and its healing power.
For example, the pretest/posttest nonequivalent groups design controls for some internal threats to validity, but it's problematic with respect to controlling for changes due to how groups members were selected (selection maturation), changes due to how individuals were tested (instrumentation), changes due to the selection of individuals with extreme pretest measures leading to regression toward the mean (regression), and changes due to local events not a part of the study (history).
The design involved three communities that differed in history, language, and culture but had similarities in social, demographic, and economic factors.
Indeed, I interacted with several individuals who had life histories that were inconsistent with my general characterization and who were the basis for suggesting the competing models discussed above.
Had Goldin relied exclusively on, say, the life histories of a nonprobabilistic sample of informants without specified selection criteria (Johnson 1990), she might have arrived at a very different, and possibly erroneous, conclusion.
The History of Ethnological Theory.
The History of Samoan Sexual Conduct and the Mead-Freeman Controversy.
A History of Field Methods.
4 hypothesized
In each successive study, investigators tried to control for as many extraneous variables as possible so that the hypothesized effect could be assessed (that is, the effectiveness of pingers compared to not using pingers).
The threats in Table 2 deal with extraneous factors that may account for the presence or absence of a hypothesized effect (that is, contrast validity with invalidity).
In the quasi-experimental case, this means changes between pre- and posttest, but this way of thinking can be expanded to include hypothesized effects dealing with differences, similarities, or associations whether diachronic or synchronic.
Each of these threats may hamper a researcher's ability to assess the contribution of a hypothesized effect to any changes observed.
4 illness
One of the central concerns of medical anthropologists has been to better understand the relationship between health-related behaviors and native perceptions about illness.
This leads us to the conclusion that the substantial variation apparent in the use of a physician's treatment between the two samples, a consequence of differential access to such treatment, occurs without corresponding degrees of variation in resident's attitudes and beliefs about illness.
Because respondents weren't randomly assigned into comparison groups, it's difficult to know the influences of confounding variables on physician utilization and beliefs about illness.
Susto, A Folk Illness
(1985) report on a study of a folk illness known as susto, found in many cultural groups throughout North and South America.
The authors were interested in three primary hypotheses relating to role performance and the presence of the illness, psychiatric impairment and susto, and relationship between organic disease and susto.
reported cases of the illness (higher-income people recognized the condition but felt that belief in it was more superstitious than real).
Susto, A Folk Illness.
4 large
For Stinchcombe (1987:23), the observations produced by how a study was designed are fundamental to the proper assessment of empirical evidence: "We always want to reject evidence if it can be explained by the design of the research or by a large number of small, unorganized causes.
Members of such competing groups-such as large-scale commercial producers, commodity producers, environmental groups, and real estate developers-believe strongly in their positions.
Some conservationist groups claimed the study was biased in that the treatment nets were placed in areas known not to have large numbers of porpoises.
This is particularly true today, given the large number of computer analytical packages available for analyzing text (see Bernard and Ryan, this volume).
Thus, most large epidemiological studies of IDUs focused mainly on direct sharing behaviors in attempts to understand seroconversion rates and other risk factors.
However, when a large representative sample of the township was aggressively pursued, the different data sets tended to support model C as the one that characterizes the general tendencies within the township.
4 marriage
An example of research in this mode is Naomi Quinn's (1996) development of Americans' cultural models concerning marriage.
Based on in-depth interviews with 22 informants, Quinn (1996) attempted to build a cultural model of Americans' reasoning about marriage.
and all were married during the period of their interviews, all in first marriages.
Beyond these constancies of cultural and marital experience, they were selected to maximize diversity with regard to such obvious differences as their occupations and educational backgrounds, religious affiliations and ethnic and racial identities, their neighborhoods and social networks, and the duration of their marriages.
Based on an in-depth analysis of informants' discourse about marriage, Quinn produced a cultural model incorporating a number of causal links in informants' reasoning as to a "lasting marriage" (see Figure 5).
Although the model appeared to be widely shared among informants from Quinn's sample and data collected from other studies on marriage, research still has to be designed to test this model across settings and researchers.
Culture Contradictions: The Case of America's Reasoning about Marriage.
4 measurement
Some things, like site selection, sampling, measurement, and recording are at least partly within our control.
Interpretive strategies, on the other hand, differ from systematic approaches in that they question a researcher's ability to maintain objectivity, particularly in the ethnographic context where the ethnographer is often the instrument of measurement.
the effects of reactive measurement (that is, the measurement procedure itself caused a change in the dependent variable);
He refers to the second as mensurative experiments, which involve simply the measurement of variables in space and time and among a number of comparison groups, without random allocation and the manipulation of experimental factors.
Measurement, operationalization of theoretical concepts, and type of analysis used are other important factors.
But it is critical to remember the connection between theory, design (including sampling), and data analysis from the beginning, because how the data were collected, both in terms of measurement and sampling, is directly related to how they can be analyzed.
There were no equivalent measurement procedures for the two groups.
For example, there was no measurement on which to compare differences in stress experienced by adolescents in Samoa and the United States.
4 project
Nevertheless, a well-articulated project design helps "to promote the effective conduct of research," whether one starts from a positivist or humanist perspective (Ellen 1984:158).
One agency may require a detailed description of the proposed project paying attention to the research design logic of science (for example, validity, reliability, hypotheses, etc.
All funding agencies expect a well-organized outline of the proposed project-one that meets the design expectations of peer reviewers and agency personnel.
This means that Clifford is simultaneously more firmly in control of his project and more parasitical.
Analysis-of-variance models and multigroup comparisons (factorial designs) may dictate the whos, whats, and wheres of a given project.
Johnson and Murray (1997), for example, used a quasi-experimental design to evaluate the use of fish aggregation devices (FADS) in small-scale fisheries development projects.
Thus, sick people were compared to sick people and control group members were selected from the pool of patients at the project clinics in each of the communities.
Evaluating FAD Effectiveness in Development Projects: Theory and Praxis.
4 religious
Because she was interested in a model that was shared, it was crucial to interview a wide range of couples who, although of the same culture, were not just from one region of the country or of only one ethnicity, religion, or social class.
Beyond these constancies of cultural and marital experience, they were selected to maximize diversity with regard to such obvious differences as their occupations and educational backgrounds, religious affiliations and ethnic and racial identities, their neighborhoods and social networks, and the duration of their marriages.
Parallel to these economic changes, there has been a shift in religious preference over the last 70 years.
Goldin (1996) wanted to understand the relationships among religious affiliation, economic ideology, occupation, and economic status in a Guatemalan township (Almolonga).
4 Seidman
Interpretive, hermeneutic, and postmodern approaches make little explicit reference to ethnographic design issues, but well-written examples from ethnography may provide "moral evidence" to deal with current social problems, moving people (including politicians) in ways that numerical facts can't (Seidman 1994:134).
Discussions about this debate can be found in Seidman (1994), on the one hand, and Faia (1993), on the other, and, more specifically for anthropology, by Kuznar (1997).
An important implication here is that scholars who follow this line of inquiry are searching for local rationales rather than nomothetic theory or universal foundations and may be more interested in conveying a moral tale of some type rather than a value-free account (Seidman 1994).
Seidman, S. 1994.
4 standard
Whereas the analytical techniques most often used in psychology, sociology, and economics often led to rather standard designs, in anthropology the eclectic nature of ethnography leaves the design of research more open ended.
We certainly cannot hold Mead to the design standards available today.
She could then have tested the proposition that exposure to competing standards of sexual morality leads to higher levels of emotional distress in adolescents.
Using a standard chi-square test, the authors found a significant difference in the frequency distribution of treatment alternatives between the two towns, with the exception of folk curers.
Using standard methods of statistical inference, Rubel et al.
The Comparative Analysis of Human Societies: Toward Common Standards for Data Collection and Reporting.
4 Weller
Individuals were asked to perform an unconstrained judged similarity of the fish-a free pile sort (see Weller, this volume, and Weller and Romney 1988).
Further, beliefs about the use and functional characteristics of the fish obtained from extensive ethnographic interviews were turned into a sentence-frame completion task described by Weller and Romney (1988).
Weller, S. C., and A. K. Romney.
3 Academic
Behar discusses the multiplexity of roles, in that she was variously involved as "priest, interviewer, collector, transcriber, translator, analyst, academic, connoisseur, editor, and peddler" (p. 12).
As against the magic of academic rituals of explanation which, their alchemical promise of yielding system from chaos, do nothing to ruffle the placid surface of this natural order, I choose to work with a different conflation of modernism and the primitivism it conjures into life-namely the carrying over into history of the principle of montage, as I learned that principle not only from terror, but from Putumayo shamanism with its adroit, albeit unconscious, use of the magic of history and its healing power.
New York: Academic Press.
London: Academic Press.
London: Academic Press.
3 asustados
These groups will be referred to as the asustados groups.
The researchers were careful to make the control group as comparable to the asustados groups as possible.
Because the asustados were "sick," control group members must also be sick.
In addition to the control group being sick, males were matched with males and females with females and asustados and controls were matched in terms of age.
This design allowed for a variety of comparisons, including comparisons by controls and asustados, by gender, and by matched pairs both within and between cultural groups (see Figure 7).
3 CA
Belmont, CA: Wadsworth.
Walnut Creek, CA: Alta Mira Press.
Thousand Oaks, CA: Sage Publications.
Thousand Oaks, CA: Sage Publications.
Thousand Oaks, CA: Sage Publications.
Thousand Oaks, CA: Sage Publications.
Thousand Oaks, CA: Sage Publications.
Belmont, CA: Lifetime Learning Publications.
Beverly Hills, CA: Sage Publications.
Walnut Creek, CA: Alta Mira Press.
Thousand Oaks, CA: Sage Publications.
Novato, CA: Chandler and Sharp.
Walnut Creek, CA: Alta Mira Press.
Thousand Oaks, CA: Sage Publications.
Thousand Oaks, CA: Sage Publications.
Thousand Oaks, CA: Sage Publications.
Newbury Park, CA: Sage Publications.
3 Cambridge
Cambridge: Harvard University Press.
Cambridge: M.I.T. Press.
Cambridge: Cambridge University Press.
Cambridge: Cambridge University Press.
3 cases
She wrote of her doubts about the comparability of cases and about her ability, or even the need, to do a quantitative comparison of the similarity of attitudes among the adolescent girls in her study.
I am very decidedly of the opinion that a statistical treatment of such intricate behavior as the one that you are studying, will not have very much meaning and that the characterization of a selected number of cases must necessarily be the material with which you operate.
In these cases, there are a variety of solutions, including intercept sampling, snowball sampling, random walks, quota sampling, and purposive sampling.
In some cases, a researcher may not be interested in generalizing to a population but may just want to know whether two subgroups obtained from a snowball sample differ with respect to some variable of interest.
In most cases, if you can use a random sample, do it!
reported cases of the illness (higher-income people recognized the condition but felt that belief in it was more superstitious than real).
3 Chicago
Chicago: University of Chicago Press.
Chicago: Rand McNally.
Chicago: Aldine.
Chicago: University of Chicago Press.
Chicago: University of Chicago Press.
Chicago: University of Chicago Press.
3 Cohen
Some early exceptions include Brim and Spain's (1974) book on hypothesis-testing designs, Pelto and Pelto's (1978) book on research methodology in cultural anthropology, and Naroll and Cohen's (1973) A Handbook of Method in Cultural Anthropology, which has several chapters that address issues in research design (LeVine 1973;
R. Naroll and R. Cohen, eds.
Naroll, R., and R. Cohen, eds.
R. Naroll and R. Cohen, eds.
R. Naroll and R. Cohen, eds.
3 Columbia
New York: Columbia University Press.
New York: Columbia University Press.
New York: Columbia University Press.
New York: Columbia University Press.
3 constraints
The constraints of field research may lead one to stray from the idealized prescriptions of a research design, but Mead was attempting to exert her authority without necessarily following the research procedures advocated by Boas and others.
He can invent his questions with few constraints;
It involves constructing a logical plan that links all the elements of research together so as to produce the most valid assessment possible of some theory, given some set of realistic constraints (for example, cost, scope, geographical setting, etc.
The results of my study, of course, must be interpreted within the constraints of the data collection methods.
3 effectiveness
The effectiveness of the device, however, was in question, and there was no firm evidence in the literature about it.
Fishers petitioned the federal government to fund a study of pinger effectiveness.
In each successive study, investigators tried to control for as many extraneous variables as possible so that the hypothesized effect could be assessed (that is, the effectiveness of pingers compared to not using pingers).
Evaluating FAD Effectiveness in Development Projects: Theory and Praxis.
3 established
In Britain the claims that anthropology not only studied a distinctive body of data but also that it possessed a sophisticated methodology to collect these data, was an important factor in the establishment of anthropology as a discipline.
This was less necessary in America where, by the late nineteenth century, anthropology was already established in universities, museums and government agencies.
This established, Young and Garro could then test the second hypothesis relating to the similarity in beliefs between the two communities.
Although such designs are less driven by an established theoretical framework, there still is need to pay careful attention to a number of design details in the proper development of new theories and models.
3 Goldin
Goldin (1996) wanted to understand the relationships among religious affiliation, economic ideology, occupation, and economic status in a Guatemalan township (Almolonga).
Based on extensive participant observation, Goldin constructed four plausible models that might account for what she observed while in the field.
Using her experience as a participant observer, Goldin developed a survey which she applied to a random sample of 10% of the heads of households in the township (n = 57).
The combination helped Goldin in the specification of appropriate variables, in the development of a sound survey instrument, and in the specification and assessment of the four competing models.
Had Goldin relied exclusively on, say, the life histories of a nonprobabilistic sample of informants without specified selection criteria (Johnson 1990), she might have arrived at a very different, and possibly erroneous, conclusion.
Goldin, L. R. 1996.
3 introduction
Introduction
Although Ramos discusses informant interviewing and various sources of data, her introduction is largely devoted to discussions of her reliance on her own memory in writing the ethnography and the shift in the narrative between synchrony and diachrony.
Random assignment maximizes the probability that experimental groups are equivalent on key variables prior to the introduction of an intervention.
Using a questionnaire to gain background information, 15 college undergraduates who had the least amount of recreational fishing experience were selected from two introductory anthropology classes.
The Professional Stranger: An Informal Introduction to Ethnography.
Introduction.
3 London
London: Academic Press.
London: Routledge Press.
London: Sage Publications.
London: Academic Press.
3 matched
In addition to the control group being sick, males were matched with males and females with females and asustados and controls were matched in terms of age.
Matched pairs were made within communities only.
This design allowed for a variety of comparisons, including comparisons by controls and asustados, by gender, and by matched pairs both within and between cultural groups (see Figure 7).
3 montage
More extreme forays into experimental ethnography have blurred this distinction even further, and there is more of a focus on writing strategies that include such approaches as montages, evocative representations, polyvocal texts, and even ethnographic fictions (Denzin and Lincoln 1994).
Other examples in this vein include Panourgia's (1995) use of we and they in her "Athenian Anthropography" and Behar's (1993) use of montage in her collaboration with a single woman in the telling of that woman's life story.
The idea of a montage as an organizing principle was also central to Taussig's (1987) historical and ethnographic account of shamanism, colonialism, and terror in South America.
" Second, Taussig uses the "principle of montage" as a means, at least in his view, for better relating the lessons of history.
As against the magic of academic rituals of explanation which, their alchemical promise of yielding system from chaos, do nothing to ruffle the placid surface of this natural order, I choose to work with a different conflation of modernism and the primitivism it conjures into life-namely the carrying over into history of the principle of montage, as I learned that principle not only from terror, but from Putumayo shamanism with its adroit, albeit unconscious, use of the magic of history and its healing power.
3 Naroll
Some early exceptions include Brim and Spain's (1974) book on hypothesis-testing designs, Pelto and Pelto's (1978) book on research methodology in cultural anthropology, and Naroll and Cohen's (1973) A Handbook of Method in Cultural Anthropology, which has several chapters that address issues in research design (LeVine 1973;
R. Naroll and R. Cohen, eds.
Naroll, R., and R. Cohen, eds.
R. Naroll and R. Cohen, eds.
R. Naroll and R. Cohen, eds.
3 nonprobability
Exploratory research may contribute to the production of reliable and valid measures, provide information essential for constructing comparison groups, facilitate construction of structured questions or questionnaires, or provide information necessary for producing a sound probability or nonprobability sample.
In manipulative experiments, analytical units are randomly allocated to comparative groups, whereas in mensurative experiments selection of units is based on some probability or nonprobability sampling scheme.
Many probability and nonprobability sampling designs are available for any given research problem.
Nonprobability sampling methods have come to be associated with qualitative approaches or for the selection of ethnographic informants, particularly key informants or consultants (Werner and Schoepfle 1987;
3 Oaks
Thousand Oaks, CA: Sage Publications.
Thousand Oaks, CA: Sage Publications.
Thousand Oaks, CA: Sage Publications.
Thousand Oaks, CA: Sage Publications.
Thousand Oaks, CA: Sage Publications.
Thousand Oaks, CA: Sage Publications.
Thousand Oaks, CA: Sage Publications.
Thousand Oaks, CA: Sage Publications.
Thousand Oaks, CA: Sage Publications.
3 organisms
Ethnobiologists have long debated whether folk biological classifiers are natural historians who compare animals on the basis of their morphological characteristics or pragmatists who compare on the basis of the utility of organisms.
Were individual informants classifying organisms on the basis of form or function?
The authors were interested in three primary hypotheses relating to role performance and the presence of the illness, psychiatric impairment and susto, and relationship between organic disease and susto.
Although there was no association between susto and psychiatric impairment, there was a relationship between susto and the suffering of more organic disease signs.
3 paradigms
Practitioners of almost all interpretive paradigms are searching in one way or another for some understanding (verstehen) rather than for some explanation of social phenomena.
Further, the purpose of research strategies under these interpretive paradigms is more focused on the production of a believable or plausible account or story rather than a single depiction of the truth, since it is thought that there are a multitude of plausible accounts rather than just a single true story.
Epistemologically, interpretive paradigms are subjective, with findings that are value mediated or even created.
While systematic analytical paradigms are primarily concerned with threats to validity, recent interpretive paradigms are focused more on threats to believability -as in "Do you believe my story?
Research Design in Anthropology: Paradigms and Pragmatics in the Testing of Hypotheses.
3 Pelto
According to Pelto and Pelto (1978:291): "Research design involves combining the essentials of investigation into an effective problem-solving sequence.
Some early exceptions include Brim and Spain's (1974) book on hypothesis-testing designs, Pelto and Pelto's (1978) book on research methodology in cultural anthropology, and Naroll and Cohen's (1973) A Handbook of Method in Cultural Anthropology, which has several chapters that address issues in research design (LeVine 1973;
Although this process might be equated to a method, it's better to think of ethnography as a strategy in which a variety of methods can be used in the quest for knowledge (Pelto and Pelto 1978).
Other sources of potential bias include sampling error (that is, chance), nonresponse, the use of imprecise measures, data recording errors, informant inaccuracies, and interviewer effects (see Pelto and Pelto 1978;
Pelto, P. J., and G. H. Pelto.
3 physician
One explanation views use tied to congruence between a client's medical beliefs and scientific medical theory: the higher the congruence, the more likely the client will choose a physician's treatment.
Young and Garro took a different stance, stressing physician accessibility as the most important determinant of physician use.
An important element of this position is that traditional medical beliefs are not a barrier to choice of physician treatment.
This leads us to the conclusion that the substantial variation apparent in the use of a physician's treatment between the two samples, a consequence of differential access to such treatment, occurs without corresponding degrees of variation in resident's attitudes and beliefs about illness.
Because respondents weren't randomly assigned into comparison groups, it's difficult to know the influences of confounding variables on physician utilization and beliefs about illness.
Given a lack of pretest observations, we can only assume that beliefs were similar prior to the availability of physicians in Uricho.
Symptomology and health problems were operationalized using a panel of physicians.
3 prior
Behar discusses the multiplexity of roles, in that she was variously involved as "priest, interviewer, collector, transcriber, translator, analyst, academic, connoisseur, editor, and peddler" (p. 12).
Random assignment maximizes the probability that experimental groups are equivalent on key variables prior to the introduction of an intervention.
It would also help if the anthropologists were to study the school for a year or two prior to the program evaluation.
Unlike studies where informed consent is obtained prior to participation, in experiments like Melbin's, individuals often participate without knowing about it.
Given a lack of pretest observations, we can only assume that beliefs were similar prior to the availability of physicians in Uricho.
3 procedures
The constraints of field research may lead one to stray from the idealized prescriptions of a research design, but Mead was attempting to exert her authority without necessarily following the research procedures advocated by Boas and others.
the effects of reactive measurement (that is, the measurement procedure itself caused a change in the dependent variable);
When generalization to a target population is the objective, you should strive to define a sampling universe or frame using a selection procedure with known error limits and one that represents the population of interest.
There is a vast literature on sampling theory and random sampling procedures, including discussions of sample sizes (see, for example, Bernard [1994] for a summary and Babbie [1990] for detailed discussion of sampling issues).
There were no equivalent measurement procedures for the two groups.
Recent developments in statistical procedures allow us to assess the similarities in aggregated judged-similarity matrices between the two communities (see Handwerker and Borgatti, this volume, and Hubert 1987).
The selection of control group subjects, by contrast, involved a purposeful selection procedure in which potential subjects were screened for recreational fishing experience.
3 Psychiatric
The authors were interested in three primary hypotheses relating to role performance and the presence of the illness, psychiatric impairment and susto, and relationship between organic disease and susto.
Psychiatric impairment was operationalized using the 22-item Screening Score for Psychiatric Impairment.
Although there was no association between susto and psychiatric impairment, there was a relationship between susto and the suffering of more organic disease signs.
3 sick
Because the asustados were "sick," control group members must also be sick.
Thus, sick people were compared to sick people and control group members were selected from the pool of patients at the project clinics in each of the communities.
In addition to the control group being sick, males were matched with males and females with females and asustados and controls were matched in terms of age.
3 township
Goldin (1996) wanted to understand the relationships among religious affiliation, economic ideology, occupation, and economic status in a Guatemalan township (Almolonga).
Using her experience as a participant observer, Goldin developed a survey which she applied to a random sample of 10% of the heads of households in the township (n = 57).
However, when a large representative sample of the township was aggressively pursued, the different data sets tended to support model C as the one that characterizes the general tendencies within the township.
3 University
This was less necessary in America where, by the late nineteenth century, anthropology was already established in universities, museums and government agencies.
When generalization to a target population is the objective, you should strive to define a sampling universe or frame using a selection procedure with known error limits and one that represents the population of interest.
There is a body of literature that views the interaction of culture with the individual as so deeply unique and personal as to not be researchable in terms of cultural universals, coherence, or even sharing.
Chicago: University of Chicago Press.
Lanham, MD: University Press of America.
Cambridge: Harvard University Press.
New York: Columbia University Press.
New York: Columbia University Press.
Madison: University of Wisconsin Press.
Cambridge: Cambridge University Press.
Princeton: Princeton University Press.
Berkeley: University of California Press.
Madison: University of Wisconsin Press.
Berkeley: University of California Press.
New York: Columbia University Press.
Cambridge: Cambridge University Press.
New York: Columbia University Press.
Chicago: University of Chicago Press.
Chicago: University of Chicago Press.
Chicago: University of Chicago Press.
Princeton: Princeton University Press.
3 Vol
Qualitative Research Methods Series, Vol.
Qualitative Research Methods Series, Vol.
Qualitative Research Methods Series, Vol.
Systematic Fieldwork, Vol.
3 York
New York: Academic Press.
New York: Holt, Rinehart and Winston.
New York: Marcel Dekker.
New York: Columbia University Press.
New York: Rinehart.
New York Times Book Review, March 27:3, 22-23.
New York: The Free Press.
New York: Columbia University Press.
New York: John Wiley.
New York: Columbia University Press.
New York: Columbia University Press.
New York: Holt, Rinehart and Winston.