Criteria for Judging a Research Study

Review of Previous Research

• 1. How closely is the literature reviewed in the study related to previous literature?
• 2. Is the review recent? Are there any outstanding references you know about that were left out?

Problem and the Purpose

• 3. Can you understand the statement of the problem?
• 4. Is the purpose of the study clearly stated?
• 5. Does the purpose seem to be tied to the literature that is reviewed?
• 6. Is the objective of the study clearly stated?
• 7. Is there a conceptual rationale to which the hypotheses are grounded?
• 8. Is there a rationale for why the study is an important one to do?

Hypotheses

• 9. Are the research hypotheses clearly stated?
• 10. Are the research hypotheses explicitly stated?
• 11. Do the hypotheses state a clear association between variables?
• 12. Are the hypotheses grounded in theory or in a review and presentation of relevant literature?
• 13. Are the hypotheses testable?

Method

• 14. Are the independent and dependent variables clearly defined?
• 15. Are the definition and description of the variables complete?
• 16. Is it clear how the study was conducted?

Sample

• 17. Was the sample selected in such a way that you think it is representative of the population?
• 18. Is it clear where the sample came from and how it was selected?
• 19. How similar are the subjects in the study to those that have been used in other similar studies?

Results and Discussion

• 20. Does the author relate the results to the review of the literature?
• 21. Are the results related to the hypotheses?
• 22. Is the discussion of the results consistent with the results?
• 23. Does the discussion provide closure to the initial hypotheses presented by the author?

References

• 24. Is the list of references current?
• 25. Are the references consistent in their format?
• 26. Are the references complete?
• 27. Does the list of references reflect some of the most important reference sources in the field?
• 28. Does each reference cited in the body of the paper appear in the reference list?

General Comments About the Report

• 29. Is the report clearly written and understandable?
• 30. Is the language unbiased (nonsexist and relatively culture free)?
• 31. What are the strengths and weaknesses of the research?
• 32. What are the primary implications of the research?
• 33. What would you do to improve the research?

In my class, students are required to answer all 33 of these questions for a research article that reports about an experimental study in their discipline.

Planning the Actual Research

You are well on your way to formulating good, workable hypotheses, and you now know at least how to start reviewing the literature and making sense out of the hundreds of available resources. But what you may not know, especially if you have never participated in any kind of research endeavor, is how much time it will take you to progress from your very first visit to the library to your final examination or submission of the finished research report. That is what you will learn here.Although you still have plenty to learn about the research process, now is a good time to get a feel for the other activities you will have to undertake in order to complete your research project. It is also helpful to get a sense of how much time these activities might take.First the activities. Table 13.1 shows an example of a checklist of activities you probably need to complete in order to complete your proposal (or research). The activities are grouped by the general headings previously discussed.Now for computing how much time the process will take. One effective way to do this is to estimate how much time each individual activity (writing the literature review, collecting data, etc.) will require, using some standard measure, such as days, keeping in mind that sometimes things go

• • Just as planned
• • Not as well as planned
• • Not well at all (which usually is the rule, rather than the exception).

Now take the average of these values. To be more precise, let’s break workdays into 4-hour chunks (for morning and evening) and call each chunk one unit of time. There are then 10 units of time in 1 week. If you enter Table 13.1 as a spreadsheet (using a program such as Excel), you can easily sum the columns as you fiddle and tinker with the amount of necessary time.

Table 13.1 A checklist of activities to help you complete your proposal or research

ActivityTime EstimatesWhen Things Go Just as PlannedWhen Things Don’t Go Exactly as PlannedWhen Things Don’t Go Well at AllIntroduction

• • Search general sources and come up with an idea
• • Formulate a research question
• • Present a preliminary hypothesis

______________________________________________________Review of the Literature

• • Search through secondary sources
• • Search through primary sources
• • Reconsider the literature and state the research hypothesis

______________________________________________________Methodology

• • Identify and describe the dependent variables
• • Identify and describe the independent variables
• • Field test the dependent variables
• • Create data entry forms
• • Locate a suitable sample
• • Pilot test the research hypothesis
• • Distribute permission forms
• • Collect data

__________________________________________________________________________________________________________________________________________________________________Results

• • Analyze the data
• • Report the results accompanied by tables and graphs, if useful

______________________________________________________Discussion

• • Review the nature and purpose of the research
• • Refer to the results in light of the question being asked
• • Draw the appropriate conclusions about the confirmation or refutation of the research hypothesis
• • Discuss the limitations of the study
• • Discuss the implications of the study
• • Discuss topics and directions for future research

________________________________________________________________________________________________________________________________________________For example, let’s look at a search through primary sources (as part of the literature review) and estimate that it will take you

• • Four days, or 8 time units, if things go great
• • Six days, or 12 units, if things do not go exactly as planned
• • A whopping 8 days, or 16 units, if things do not go well at all

Once you have these estimates, average them for the activity, and you will have a singular estimate of how long any one activity should take, such as,( 8 + 12 + 16 )3  = 12  unitsor 6 days, which is about one very full week’s work (if you work on Saturday or Sunday).If you want to be even more precise, weight the estimates. For example, let’s say that you anticipate having trouble finding a sample, and at best you can expect things to go only okay. Writing the descriptive section, though, should be a snap. You should weight the “not as well as planned” estimate two or three times greater than the others.These estimates can be computed for all the activities you see in Table 13.1 and then summed to get an estimate for the overall activity. Keep in mind that everything takes longer than you initially think, so be generous, even for your most optimistic estimate.