When Research Goes Off the Rails
Why It Happens and What You Can Do About It
Edited by David L. Streiner and Souraya Sidani
Request a free digital professor copy on VitalSource ?Winner—American Journal of Nursing Book of the Year Award!
“An eye-opening text that pushes the reader to think about solutions to real world problems that occur in the field as one performs research....An essential read for new researchers and will be thought-provoking for those in the field....As a graduate student new to the field, I found this book useful for exploring the realities of research....The text reads more like a novel than a traditional scholarly text, which is a welcome addition to the reading list of most doctoral students. This text would be excellent material to pair with a traditional textbook for a research design course....Regardless of your position, social researchers will find the honest discussion of research problems and solutions to be an enlightening and interesting read.”
—Canadian Journal of Program Evaluation
“This unique, thorough guide offers inspiring testimony about how to persevere in the face of unexpected setbacks when designing or implementing a study. Contributors describe the untold story of research—the real-world intrusions that almost never make it into the publication of record. Topics include the ethical review process, recruitment contingencies with formal and informal gatekeepers, missteps in data collection and analysis, roadblocks and detours when implementing the study, and conflicts and personality factors associated with collaboration and intervention. The cautionary yet inspiring vignettes will resonate with experienced researchers and will appeal to graduate students just learning the ropes. This text will enliven a standard research methods course with a wonderful collection of stories from the front lines.”
—Gregory J. Meyer, PhD, Department of Psychology, University of Toledo
“Most books for social and behavioral science researchers assume that faithfully following certain protocols will produce useful results. In contrast, this book shows that the unexpected almost always strikes. There is as much to learn from these real-world situations of research gone awry as from textbook examples of 'perfect' designs that lead to straightforward results. The underlying message of all of the chapters is that serious mishaps are best avoided by focusing on prevention. Drawing on diverse studies using different designs in multiple disciplines, the book illustrates broadly applicable approaches to navigating the vicissitudes of research and evaluation practice.”
—Henry M. Levin, PhD, William Heard Kilpatrick Professor of Economics and Education, Teachers College, Columbia University; David Jacks Professor of Education and Economics (Emeritus), Stanford University
“Nobody told us this stuff! This is the information that is missing in graduate courses and professional texts on research methods, experimental design, and statistics. From distinguished interdisciplinary editors and contributors, this book fills an extremely important niche. It should be required reading for all graduate students considering a career in the health and social sciences. It will be extremely valuable to experienced researchers, knowledge brokers, and policy makers in the health and social service fields.”
—Charles E. Cunningham, PhD, Department of Psychiatry and Behavioural Neurosciences and Jack Laidlaw Chair in Patient-Centred Health Care, McMaster University, Canada
“In typical 'Streineresque' style, this book is easy to read, grounded in reality, and solution oriented. The contributors use accessible language, which makes reading the book feel like you are having an informal dialogue with colleagues. Social science researchers experiencing challenges at various stages of project implementation, graduate students who want to avoid and learn from mistakes made by others, and reviewers of grant proposals or manuscripts who are in a position to help others keep their research on track should seriously consider making this book a 'must read.'”
—Lucyna M. Lach, MSW, PhD, School of Social Work, McGill University, Canada
“This book is a useful reference that could help you anticipate potential problems when planning your thesis or dissertation study.”
—Amanda L. Garrett, graduate student, Department of Educational Psychology, University of Nebraska-Lincoln
Table of Contents
Going Off the Rails: An Introduction, Souraya Sidani and David L. Streiner
I. Ethics Approval
1. When Mountains Move Too Slowly, Melanie A. Hwalek and Victoria L. Straub
2. The Ethics of Sex Research on the Internet, Alissa Sherry and Amy Amidon
3. When Safeguards Become Straitjackets: How Ethics Research Board Requirements Might Contribute to Ethical Dilemmas in Studies with Marginalized Populations, Mechthild Meyer, Alma Estable, Lynne MacLean, and Nancy Edwards
4. Going Off the Rails for “Love or Money”: Implementation Issues Related to Payment of Research Participants in an Addiction-Research Project, Brian R. Rush and Dominique Morisano
II. Accessing the Participants
5. Frailty, Thy Name Is Macho, José Quirino dos Santos
6. Power in Numbers: Research with Families in Long-Term Care, Julie M. Dergal Serafini
7. Getting the Wrong Gatekeeper, Lynne MacLean
8. Breaking into Court, Mandeep K. Dhami and Karen A. Souza
9. The RDC Archipelago, Scott Veldhuizen, John Cairney, and David L. Streiner
III. Recruitment and Retention
10. Small Colleges and Small n’s, Christopher Koch and Anna Tabor
11. Mitigating the Impact of External Forces, Souraya Sidani, David L. Streiner, and Chantale Marie LeClerc
12. A Trip to the School of Hard Knocks: Recruiting Participants from Health Service Agencies for Qualitative Studies of Aging, Kathleen W. Piercy
13. All Aboard!: Using Community Leaders to Keep Clinical Researchers on Track, Philippe Barrette
14. Changing Horses in Midstream: Transforming a Study to Address Recruitment Problems, Anthony S. Joyce
15. When Cost Meets Efficiency: Rethinking Ways to Sample a Rare Population, Julian Montoro-Rodriguez and Gregory C. Smith
16. The Story Is in the Numbers, Robert van Reekum
17. Strategies for Retaining Participants in Longitudinal Research with Economically Disadvantaged and Ethnically Diverse Samples, Elizabeth A. Goncy, Michelle E. Roley, and Manfred H. M. van Dulmen
18. Culturally Specific Strategies for Retention and Adherence to Physical Activity Interventions in Hispanic Women, Colleen Keller, Julie Fleury, and Adrianna Perez
IV. Study Implementation
19. When a Beautiful Intervention Meets Ugly Reality: Implementing an Intervention in the Real World, Souraya Sidani, David L. Streiner, and Chantale Marie LeClerc
20. When Saving Blood Goes Wrong, Claudio S. Cinà and Catherine M. Clase
21. PDA = Pretty Darned Awful: The Trials and Tribulations of Running Trials of PDAs, Geoffrey R. Norman
22. When Sugar Is Not So Sweet: Camera Shyness and Intentional Cointervention Almost Derail a Study, Françoise Filion and C. Celeste Johnston
23. Placebo Problems: Power and Persecution, or Paranoia?, Robert van Reekum
V. Data Collection
24. Revisiting Traditional Survey Methodology to Recruit and Survey Lesbian, Gay, and Bisexual Older Adults, S. Melinda Spencer and Julie Hicks Patrick
25. Technology: Help or Hindrance?, Nasreen Roberts
26. Hoist on Our Own Postcard, David L. Streiner
27. On the Finer Points of Handling Googlies: Reflections on Hits, Near Misses, and Full-Blown Swings at the Air in Large, Population-Based Studies Involving School, Parents, and Children, John Cairney, John A. Hay, and Brent E. Faught
28. Pets, Pies, and Videotape: Conducting In-Home Observational Research with Late-Life Intergenerational Families, Brian D. Carpenter and Steve Balsis
29. Underfunded but Not Undone, Dianne Bryant
30. Community-Based Participatory Research: A Lesson in Humility, Dennis Watson
31. Where Did All the Bodies Go?, Harry S. Shannon
32. Measures for Improving Measures, Katherine McKnight, and Patrick E. McKnight
VI. Data Analysis
33. Drowsing Over Data: When Less Is More, Lynne MacLean, Alma Estable, Mechthild Meyer, Anita Kothari, Nancy Edwards, and Barb Riley
34. Bigger Is Not Always Better: Adventures in the World of Survey Data Analysis, Sylvia Kairouz and Louise Nadeau
35. Taking Aim at a Moving Target: When a Study Changes in the Middle, Arturo Martí-Carvajal
36. Lack of Normative Data as an Obstacle to Neuropsychological Assessment, F. Richard Ferraro and Kaylee Trottier-Wolter
37. These Data Do Not Compute, Lynne MacLean, Mechthild Meyer, Alma Estable, Anita Kothari, and Nancy Edwards
38. Avoiding Data Disasters and Other Pitfalls, Melinda F. Davis
39. When Interpretation Goes Awry: The Impact of Interim Testing, Dale Glaser
VII. Collaboration
40. What Happened to Cooperation and Collaboration?, Nasreen Roberts
41. Presto! It’s Gone: When a Study Ceases to Exist Right before Your Eyes, Katrina L. Bledsoe
42. Building Stakeholder Capacity to Enhance Effectiveness in Participatory Program Evaluation, Debazou Y. Yantio
VIII. Final Thoughts
43. Sometimes It Is the Researcher, Not the Research, That Goes “Off the Rails”: The Value of Clear, Complete, and Precise Information in Scientific Reports, Joseph A. Durlak, Christine I. Celio, Molly K. Pachan, and Kriston B. Schellinger
44. A Healthy Dose of Realism, Souraya Sidani and David L. Streiner
About the Editors
David L. Streiner is Senior Scientist at the Baycrest Centre for Geriatric Care and Professor of Psychiatry at the University of Toronto.Souraya Sidani is Canada Research Chair, Tier One, in Health Interventions Design and Evaluation at Toronto Metropolitan University.
Contributors
Amy Amidon, PhD, is a Postdoctoral Fellow at the Veterans Medical Research Foundation in San Diego.Steve Balsis, PhD, is Assistant Professor in the Department of Psychology at Texas A&M University in College Station, Texas.
Philippe Barrette, PhD, is a psychotherapist and a workplace consultant in Ancaster, Ontario, Canada.
Katrina L. Bledsoe, PhD, is Project Director of the national evaluation of the federally funded initiative Comprehensive Community Mental Health Services for Children and Their Families Program" and a senior research manager at Walter R. McDonald & Associates, Inc., in Rockville, Maryland.
Dianne Bryant, MSc, PhD, is a clinical epidemiologist; Director of Empower, a data management and methods center; Assistant Professor in the School of Physical Therapy, Faculty of Health Sciences, and Department of Surgery, Division of Orthopaedic Surgery, Faculty of Medicine and Dentistry, at the University of Western Ontario in London, Ontario, Canada; and part-time Associate Professor in the Department of Clinical Epidemiology and Biostatistics at McMaster University in Hamilton, Ontario, Canada.
John Cairney, PhD, is McMaster Family Medicine Professor of Child Health Research and Associate Professor in the Department of Psychiatry and Behavioural Neuroscience at McMaster University. He is also Associate Director of Research in the Department of Family Medicine at McMaster University in Hamilton, Ontario, Canada, and adjunct Senior Research Scientist in the Health Systems Research and Consulting Unit at the Centre for Addiction and Mental Health.
Brian D. Carpenter, PhD, is Associate Professor in the Department of Psychology at Washington University in St. Louis, Missouri.
Christine I. Celio, MA, is a doctoral student at Loyola University in Chicago.
Claudio S. Ciná , MD, MSc, FRCSC, is Professor in the Department of Surgery at the University of Toronto and an associate member of the Department of Clinical Epidemiology and Biostatistics at McMaster University in Hamilton, Ontario, Canada.
Catherine M. Clase, MB, BChir, MSc, FRCPC, is Associate Professor in the Department of Medicine, and an associate member of the Department of Clinical Epidemiology and Biostatistics, at McMaster University in Hamilton, Ontario, Canada.
Melinda F. Davis, PhD, is a Research Assistant Professor in Psychology at the University of Arizona in Tucson.
Mandeep K. Dhami, PhD, is a Senior Lecturer at the Institute of Criminology, University of Cambridge.
Joseph A. Durlak, PhD, is Professor of Psychology at Loyola University in Chicago.
Nancy Edwards, RN, PhD, is Professor, School of Nursing and Department of Epidemiology and Community Medicine, University of Ottawa. She holds a Nursing Chair funded by the Canadian Health Services Research Foundation, the Canadian Institutes of Health Research, and the Government of Ontario. She is Scientific Director, Institute of Population and Public Health, Canadian Institutes of Health Research.
Alma Estable, MSW, is a Qualitative Research Consultant with the University of Ottawa Community Health Research Unit and a principal of Gentium Consulting, a community-based social research firm in Ottawa, Ontario, Canada.
Brent E. Faught, PhD, is Associate Professor in the Department of Community Health Sciences at Brock University in St. Catharines, Ontario, Canada.
F. Richard Ferraro, PhD, is Director of the General/Experimental PhD program at the University of North Dakota and in 2007 was awarded the Chester Fritz Distinguished Professorship.
Françoise Filion, RN, MScN, is Research Coordinator and Professional Associate for pediatric pain research projects at McGill University and a faculty lecturer in the Faculty of Nursing at the Université de Montréal.
Julie Fleury, PhD, RN, FAAN, is Associate Dean for Research and Director of the PhD Program at Arizona State University, Phoenix.
Dale Glaser, PhD, is a principal of Glaser Consulting and adjunct faculty/lecturer at San Diego State University, Alliant International University, and the University of San Diego.
Elizabeth A. Goncy, MA, is a doctoral student in clinical child psychology at Kent State University.
John A. Hay, PhD, is Professor in the Department of Community Health Sciences at Brock University in St. Catharines, Ontario, Canada.
Melanie A. Hwalek, PhD, is an applied social psychologist and the Founder and CEO of SPEC Associates in Detroit, Michigan.
C. Celeste Johnston, RN, DEd, FCAHS is James McGill Professor at McGill University and Codirector of the Groupe de Recherche Interuniversitaire en Interventions en Sciences Infirmières à Québec (GRIISIQ).
Anthony S. Joyce, PhD, is Professor in the Department of Psychiatry at the University of Alberta and Director of the Psychotherapy Research and Evaluation Unit in the outpatient service at the Walter Mackenzie Health Sciences Centre in Edmonton, Alberta, Canada.
Sylvia Kairouz, PhD, is Assistant Professor in the Department of Sociology and Anthropology at Concordia University in Montreal.
Colleen Keller, PhD, RN-C, FNP, is Professor and Director of Hartford Center for Geriatric Nursing Excellence at Arizona State University, Phoenix.
Christopher Koch, PhD, is Professor of Psychology and Director of Assessment at George Fox University in Newberg, Oregon.
Anita Kothari, PhD, is Assistant Professor in the Faculty of Health Sciences at the University of Western Ontario in London, Ontario, Canada. She also holds a Career Scientist award from the Ontario Ministry of Health and Long Term Care.
Chantale Marie LeClerc, RN, MSc, GNC(C), is Senior Director of Planning, Integration, and Community Engagement at Champlain Local Health Integration Network in Ottawa, Ontario, Canada.
Lynne MacLean, PhD, is Research Associate at the Community Health Research Unit, University of Ottawa, where she leads the Qualitative Team.
Lynne MacLean, PhD, is Research Associate at the Community Health Research Unit, University of Ottawa, where she leads the Qualitative Team. She has previously worked as a government researcher and a mental health practitioner.
Arturo Martí-Carvajal, MD, MSc (ClinEpi), is Professor in the Department of Public Health at the University of Carabobo, Valencia, Venezuela.
Katherine McKnight, PhD, is Director of Evaluation for Pearson K-12 Solutions. She is also Adjunct Assistant Professor at George Mason University in Fairfax, Virginia.
Patrick E. McKnight, PhD, is Assistant Professor in the Department of Psychology at George Mason University in Fairfax, Virginia.
Mechthild Meyer, MEd, is Qualitative Research Consultant with the University of Ottawa Community Health Research Unit and a researcher with Gentium Consulting, a community-based research firm in Ottawa, Ontario, Canada.
Julian Montoro-Rodriguez, PhD, is Professor of Sociology and Executive Member of the Institute of Child Development and Family Relations at California State University, San Bernardino.
Dominique Morisano, PhD, is a Postdoctoral Clinical Research Fellow, Schizophrenia and Addiction Psychiatry Programs, Centre for Addiction and Mental Health (CAMH) and the Department of Psychiatry, Faculty of Medicine, University of Toronto, Toronto, Ontario.
Louise Nadeau, PhD, is Professor in the Department of Psychology at the Université de Montréal, in addition to being Scientific Director of the Centre Dollard-Cormier / University Institute on Dependencies and a researcher affiliated with the Douglas Hospital Research Center.
Geoffrey R. Norman, PhD, is Professor and Assistant Dean in the Department of Clinical Epidemiology and Biostatistics at McMaster University in Hamilton, Ontario, Canada.
Molly K. Pachan, MA, is a doctoral student in Clinical Psychology at Loyola University in Chicago.
Julie Hicks Patrick, PhD, is Woodburn Associate Professor and Director of Undergraduate Training at the West Virginia University Department of Psychology in Morgantown, West Virginia.
Adriana Perez, MS, RN, is a PhD student in Nursing and Healthcare Innovation at Arizona State University in Phoenix.
Kathleen W. Piercy, MSW, PhD, is Associate Professor in the Department of Family, Consumer, and Human Development at Utah State University in Logan, Utah.
José Quirino dos Santos, PhD, is Professor of Anthropology at Universidade de São Paulo, Brazil, and is a Visiting Professor in the Department of Preventive Medicine at the Universidade Federal de São Paulo.
Barb Riley, PhD, is a Scientist at the Centre for Behavioural Research and Program Evaluation at Waterloo University in Waterloo, Ontario, Canada, and Advisor in Knowledge Development and Exchange for the Public Health Agency of Canada.
Nasreen Roberts, MD, FRCPC, is Director of the Child and Adolescent Psychiatric Emergency Service and Associate Professor at Queens University in Kingston, Ontario, Canada.
Michelle E. Roley, BA, is currently employed as Lab Manager for the AAKOMA Project and CBT-RP study in the Department of Psychiatry and Behavioral Sciences at Duke University Medical Center in Durham, North Carolina.
Brian R. Rush, PhD, is Codirector of the Health Systems Research and Consulting Unit, Centre for Addiction and Mental Health and Professor in the Department of Psychiatry, Faculty of Psychiatry, University of Toronto, Toronto, Ontario.
Kriston B. Schellinger, BS, is a doctoral student in Clinical Psychology at Loyola University in Chicago.
Julie M. Dergal Serafini, MSc, is a doctoral candidate in the Faculty of Social Work, University of Toronto, and a graduate student in the Kunin-Lunenfeld Applied Research Unit (KLARU), at the Baycrest Centre, Toronto.
Harry S. Shannon, PhD, is Professor in the Department of Clinical Epidemiology and Biostatistics and the Program in Occupational Health and Environmental Medicine at McMaster University, Hamilton, Ontario, and Adjunct Scientist at the Institute for Work and Health in Toronto.
Alissa Sherry, PhD, is Associate Professor in the Counseling Psychology Program at the University of Texas at Austin.
Souraya Sidani, PhD, serves as Canada Research Chair, Tier One, in Health Interventions Design and Evaluation at Ryerson University, Toronto, Ontario.
Gregory C. Smith, EdD, is Professor and Director of the Human Development Center at Kent State University.
Karen A. Souza, MA, is a research assistant at the Institute of Criminology, University of Cambridge.
S. Melinda Spencer, PhD, is a 2006-2008 Kellogg Health Scholar and Assistant Professor in the Department of Health Promotion, Education, and Behavior at the University of South Carolina.
Victoria L. Straub, MA, CSSBB, is Chief Operating Officer of SPEC Associates in Detroit, Michigan, and a specialist in quality information and evaluation systems for nonprofit organizations.
David L. Streiner, PhD, CPsych, was Assistant Vice President of Research at the Baycrest Centre and Director of the Kunin-Lunenfeld Applied Research Unit. He also holds faculty appointments at the University of Toronto as a Full Professor in the Departments of Psychiatry and in Public Health Sciences, the Faculties of Nursing and Social Work, and in the School of Rehabilitation. Currently, he is Senior Scientist at Baycrest, and Guru-in-Residence with the Child Health Research Institute at McMaster University, Hamilton, Ontario.
Anna Tabor, MA, is a doctoral student in clinical psychology at George Fox University in Newberg, Oregon.
Kaylee Trottier-Wolter, BA, is a graduate student in the Clinical Psychology program at the University of North Dakota.
Manfred H. M. van Dulmen, PhD, is Assistant Professor in the Department of Psychology and Affiliate Faculty Member of the Institute for the Study and Prevention of Violence at Kent State University.
Robert van Reekum, MD, FRCPC, is a neuropsychiatrist and Assistant Professor in the Department of Psychiatry at the University of Toronto.
Scott Veldhuizen, BA, is a research analyst at the Centre for Addiction and Mental Health in Toronto and a graduate student in the Department of Applied Health Sciences at Brock University in St. Catharines, Ontario, Canada.
Dennis Watson, MA, CSP, is currently a PhD candidate in Sociology at Loyola University in Chicago, and University-Community Research Coordinator at the Center for Urban Research and Learning.
Debazou Y. Yantio, MA, is Ingénieur Agronome with the Ministry of Agriculture and Rural Development in Cameroon. He is currently a task team member of the Network of Networks on Impact Evaluation (NONIE) Initiative, chaired by the Department for International Development (DFID), United Kingdom, and is the founding member of the Cameroon Development Evaluation Association (CaDEA) and serves on various committees of the African Evaluation Association (AfrEA)."