1. INTRODUCTION
Natural disasters, and pandemics, mostly occur without warning. The threat to public health, the environment, and social wellbeing warrants scientific investigation to inform future preventative or rehabilitative measures. However, the panic and disruption to normal life caused by disasters and pandemics, and government responses to them, poses extraordinary challenges for researching such events and their effects.
The SARS-COV-2 (COVID-19) pandemic and quarantine measures implemented to reduce the incidence of morbidity and mortality associated with COVID-19, triggered a large, rapid wave of research on the topic globally (Riccaboni & Verginer, 2022). Positively, many research ethical review boards implemented rapid assessment processes for studies on a COVID-19 related topic. However, social distancing, work-from-home orders, fear of COVID-19 infection, and other lockdown restrictions caused the termination of much face-to-face data collection (Marsden et al., 2020).
COVID-19 triggered a shift to a range of online, and innovative, data collection methods. There are, however, some context specific challenges that arise when recruiting participants during a pandemic.
Research involving participants who might be vulnerable to unintended negative consequences as a result of a study’s focus or method, such as being asked questions about sensitive topics, usually needs adaptation to minimize the potential for harm (Gordon, 2020). A global pandemic could introduce context specific vulnerabilities, for example, if a participant was experiencing bereavement, which has been recognized as a source of vulnerability (Alexander, 1993).
One context specific vulnerability of relevance to our study, regardless of the pandemic or methodology, is the recruitment of people who are stigmatized because they smoke tobacco (Riley et al., 2017), vape (Fraser et al., 2018) or use drugs, some of which are illicit in some countries (Alexander, 1993). Innovation to collect data online from these groups could face extra obstacles.
In the absence of pandemic conditions, a number of barriers to conducting research with stigmatized or disenfranchised groups exist. A review of barriers to participation in randomized trials for colonized Indigenous peoples identified some generic barriers that could apply to other methodologies (Glover et al., 2015). These include barriers that disadvantage, social exclude or limit access to participation in research; barriers such as not having a telephone or internet access, distrust of research due to concerns about privacy and confidentiality, or because an unfamiliar methodology was being used (ibid). Facilitators to enrolment included building relationships with organizations that could represent the concerns and needs of the population being recruited, having Indigenous co-investigators or staff, respecting cultural differences, ensuring data collection tools and methods are not culturally inappropriate or offensive, and targeted recruitment methods (ibid).
This paper discusses some of the methodological and operational challenges encountered during the conduct of an exploratory, cross-sectional, self-completed, and online survey of substance use in four countries (India, United States (USA), Russia, New Zealand (NZ)) that were in various stages of lockdown restrictions due to COVID 19 in 2020 (Glover et al., 2020). The challenges experienced at each stage of the research are presented in the chronological order in which they were encountered. Adaptions made during COVID-19 are considered and recommendations for improving research-related pandemic preparedness are identified.
2. METHOLODOLICAL CHALLENGES
2.1 Conceptualizing A Survey In Response To A Pandemic
That natural and human-initiated disasters increase stress (Alexander, 1993) and can have negative effects on a range of social and health behaviors (Khanna & Fujii, 2020) was established in the literature prior to the COVID-19 pandemic. After the 2010-2011 Christchurch earthquakes in NZ, 24% of ex-smokers relapsed to smoking and 34% of current smokers increased their smoking behavior (Erskine et al., 2013). Substance use also increased (Phibbs et al., 2015). Within 10–20 months after Japan’s 9.0 earthquake in 2011, 11% quit smoking, yet 1.4% who never smoked started smoking (Nakano et al., 2018). The 9/11 terrorist attack on New York in 2001 was also associated with increased substance use (Dynes, 2002; Neria et al., 2008; Updegraff et al., 2008). Economic crises had been found to increase trends in alcohol consumption (Moskalewicz, 2019).
The trauma or psychological adaptation associated with the COVID-19 pandemic was widely expected to trigger increased alcohol, tobacco, and other substance consumption, and increased prevalence as people relapsed or initiated use to relieve stress, anxiety, and grief. A gap in the literature was how people cope with substance use withdrawals during the period of a disaster. Determining the immediate effects of pandemic-related grief, trauma, and negative effects of lockdown early in the pandemic could usefully inform pandemic preparedness for primary health, drug and alcohol services.
2.1.1 Challenges and Facilitators
The idea for the study was conceived during the second week of March 2020. Ideally formative work, such as a literature review, is conducted to inform the refinement of a research question and the study’s design. A robust systematic review is a study in itself. It has been estimated that a systematic review can take from 243 to 1752 person-hours (Nussbaumer-Streit et al., 2021). Wanting to launch our survey as quickly as possible, even a less-than-robust narrative review would have caused a significant delay. Two facilitators of our being able to conceptualise the study rapidly, identified in Nussbaumer-Streit et al.’s review of resource use during systematic reviews, were the relevant expert content knowledge some of us had and that we had completed a cursory search of the literature on the effects of disasters on smoking and alcohol consumption for a prior study proposal.
2.2 Securing Research Funding
None of the investigators had prior funding approval for the study, however the principal study center had a program grant and many of the center’s approved research projects requiring travel internationally or domestically, and face-to-face data collection, had to be suspended due to the threat of COVID-19 infection, and lockdown restrictions. This facilitated the rapid consideration of the study concept and funding approval.
2.3 Forming an International Study Team
For expediency’s sake, and to stay within budgetary restraints, the number of countries was limited to four. The country selection criteria were (i) implementation of some form of government-ordered COVID-19 prevention lockdown, (ii) existing relationship with researchers in the country to join the project as co-investigators/regional level interest from community groups to assist recruitment, (iii) ability to have the survey translated into local languages, and (iv) local assistance with obtaining local independent ethical review
In addition to meeting the above criteria, NZ was selected first as it was the resident country of the study center coordinating the research. Subsequently, the USA, India, and Russia were assessed for inclusion against the above criteria principally because existing collaborations with researchers in those countries enabled the research design, ethics review and recruitment to be expedited.
2.4 Preparation of Ethics Applications
Ethics review boards still operating were mostly limiting their activities to the consideration of COVID-19 related studies requiring urgent assessment. In NZ, the ethics committees established processes for rapid review to ensure COVID-19 related studies were prioritized and responded to immediately (Health and Disability Ethics Committee, 2020).
Most research ethics assessment processes require completion of an application form and appended study protocol and all the study documents, including study information for participants, consent forms, the questionnaire, and advertisements that would be used for recruitment. An additional requirement when applying for research ethics approval in NZ is attachment of an independent reviewer’s report. The rapid development of these is presented in the next section.
Ethics approval was sought in NZ first. Although most ethical considerations and review board requirements were common across all four countries, some differences imposed unexpected extra work and expenditure (Table 1). For example, the US Solutions IRB board required all of the co-investigators to complete an online training course in research ethics. The Indian ethics committee wanted the survey to be available in Tamil and Hindi, as well as English. Tamil is a regional language which is widely spoken among 68 million people in south India, particularly in the state of Tamil Nadu. English is the second-most widely spoken language after Hindi, which has an estimated 260 million native speakers. While we had a Tamil speaking co-investigator, we were unable to find someone to translate the survey into Hindi in time.
The survey was designed to be anonymous. No identifying personal information (such as name or address) was collected from participants. Thus, no individual can be identified at any stage in the publication or presentation of the findings. Participants were informed that all information collected would be stored securely.
Some information about the study was displayed at the beginning of the questionnaire. The amount varied by country (see Table 1). A first question sought affirmation from the participant that they had had an opportunity to review the study information and they were asked to indicate their consent. Instructions advised participants that they could stop completing the survey at any time without having to give a reason.
2.5 Questionnaire Development
An initial survey was created in English for NZ. This was then adapted for the USA, Russia, and India. Though most changes were small, it did result in four slightly different questionnaires. Translation into Russian and Tamil could have introduced some variance in how participants interpreted a question. Demographic questions needed to be checked against cultural knowledge, local practices, and common vocabulary. For example, ethnic groups the participant might identify with varied by country. The common types of residential facilities and words for describing them varied. The description for the type of area people lived in varied. Common NZ descriptors, such as rural, small town, metropolitan city, and urban were insufficient for the other countries. For example, there are no “suburbs” in Russian cities.
2.5.1 Research with Vulnerable Participants
As we decided that some participants might be vulnerable, as discussed above, we set up a website (https://coping-in-lockdown.com) to provide participants with tips for coping in lockdown, how to cope with withdrawals from smoking, and access to an experienced therapist if needed. As the website was available in English only, it could not be promoted in Russia. Since the website included information on alternatives to smoking, such as smokeless tobacco, snus, oral nicotine pouches, and nicotine replacement products (depending on the availability of these in the country and region), we also did not promote it in India. Russian and Indian recruits were encouraged to contact the researchers by email if they had any questions or needed support.
2.6 Participant Recruitment
We expected that recruitment during a lockdown might be challenging. In NZ, printed newspapers and magazines were not being distributed. We also felt rushed to place recruitment adverts before lockdown lifted. Securing advertising space in NZ was difficult because all businesses and government departments wanting to advertise at the time were also turning to online and radio advertising. Some media outlets were only interested in booking high-priced campaigns that would run over a longer period of time than we planned to keep the study open for. Recruitment methods were largely dictated by expediency.
A media release was issued announcing the study in NZ, the USA and India. Adverts were run on two NZ national news provider websites, a radio station whose audience was mainly Pacific Island peoples and Māori (the indigenous people) and social media platforms. In the USA and India, the media release attracted no coverage. An advertisement was run on a USA indigenous magazine website and an article was published by a digital harm reduction magazine. The principal investigator was interviewed for an English-speaking tobacco reduction focused podcast and the co-ordinating centre aired one podcast of its own on the study. Recruitment notices were distributed by email to the co-investigators’ networks in all four countries. Facebook and Twitter were also used and posts were shared in NZ and the USA.
In NZ and the USA, advertisements directed interested people to the website for information and to access the questionnaire. Recruitment in Russia was conducted by the in-country investigators. In India, an independent advertising agency was engaged to assist with recruitment. No information on media outlets used in India by the sub-contractor was obtained. After digital means of recruitment did not attract many participants in Russia, the investigators adapted to face-to-face recruitment in Moscow’s hospitals, approaching people in professional and hobby groups and a snowball method was employed whereby participants were asked to encourage others to also participate. If people were interested in participating, they were handed a tablet loaded with the questionnaire to self-complete. When bought advertising in the USA did not attract many participants, a large national vaping consumer advocacy group, and other vaping advocates, promoted the study via their social media platforms.
The varied methods of recruitment introduced further variance in the way the study was conducted by country.
2.7 Data Collection
One facilitator of the rapid development and launch of the study was that the coordinating research center had an existing license to use Zoho, a survey hosting platform, and experience in using Zoho.
The study ran from 22 April 2020 (NZ) to 22 July 2020 (Russia).
2.8 Data Analysis
Data analysis was beset by several challenges. Countries implemented different lockdown measures and restrictions on access to tobacco, vaping and alcohol products. There were also differences between the questionnaires and methods of recruitment in each country.
Only one participant completed the survey in Tamil. Due to the work involved to translate the data into English for analysis, and potential lost-in-translation differences that might arise in comprehension of questions, this response was considered an outlier and excluded from the analysis.
2.8.2 Types of Lockdown Provisions
Restrictions across the four countries studied included local and international travel bans; cancellation of sports and social events; pedestrian restrictions and curfews; social distancing; self-isolation of individuals, families, and whole communities; closure of shops, restricted business hours, and temporary restrictions on trade; library, school, and university closures; restrictions or closure of hospitality and food provision services; work from home rules; exclusion from shared work environments if ill; and home confinement.
However, the type of restrictions, degree of stringency, and duration of interventions varied between countries and sometimes within countries by region. The stringency index of government lockdown interventions, created by Hale et al. (Hale et al., 2021) (Figure 1) illustrates the differences between India, USA, Russia, and NZ from slightly before the survey period, 22 April, to 8 June 2020. During most of March and April, the most stringent pandemic policies were implemented in India, followed by NZ, Russia, and the USA.
A further difference of relevance to the focus of our study was the level of economic assistance provided to people in lockdown, which would impact on stress, and therefore substance use.
2.8.3 Restrictions on Tobacco Products, Vaping, Alcohol and Drugs
Some countries imposed restrictions on the sale of tobacco products, alcohol, vaping products, and other licit products we were asking about. In Russia, there was a restriction or ban on the sale of alcohol in some regions. In NZ, vape stores were not allowed to stay open. Other countries stopped short of imposing legal restrictions but promoted strong persuasive messages to discourage consumption of alcohol, vaping or tobacco products. Many public health academics and health agencies, including the World Health Organisation, utilized news media to warn people who smoked that they had a higher risk of COVID-19 infection (Bonevski, 2020) and illness severity (Organisation, 2020). Such warnings were based on limited evidence that displayed a high risk of bias (Simons et al., 2022) and led to critiques, debate, and a stream of research seeking to confirm the hypotheses, much of which were disseminated in the media. The varied extent and reach of the prohibitions and health-risk campaigns in each country could have had an influence on the substance use changes we were attempting to measure. Our results therefore would be further weakened in terms of representativeness.
2.8.4 Implications for Data Analysis
Heterogeneity among participants, methodological variables, range, and severity of lockdown measures within and between countries meant that we conducted four related, but different, surveys. This invalidated the intent to conduct cross-country quantitative analysis. As a result, the presentation of the results had to be organized to clearly discourage such an expectation among readers.
The study goal was to recruit a total of 500 participants made up of at least 125 participants from each country. As we were unable to combine the data from the four countries, the total number of participants per country became another limitation, preventing some of the planned analyses about the effects of lockdown across many variables.
2.8.5 COVID-19 Infection
While research team members can become ill or need bereavement leave at any time during non-pandemic times, having team members become ill due to infection with COVID-19, or have close family members die due to COVID-19, was not anticipated. That is, we had no contingency plan in place for replacing key investigators and this introduced some unexpected delays.
3. DISCUSSION
Studying the pandemic to inform preparedness plans and improve the efficacy of harm prevention and rehabilitation efforts is highly valuable. While science should always aspire for the highest standard of robustness, concerns have been raised about the overall quality of the COVID-19-related science that has been published (Raynaud et al., 2021). One study of peer reviews conducted on articles appearing early in the pandemic found that journal editors, peer reviewers, and readers appeared to lower their criteria for quality (Horbach, 2021). The health consequences if policy analysis and practice decisions are based on poor or flawed science can be dire. Thus, it has been suggested that health researchers have a “more significant responsibility” (Raynaud et al., 2021) to pursue robustness and report scientific results, along with a clear explication of the strength of the evidence presented and the limitations.
This paper reflected upon some of the challenges to robustness we experienced in our rush to conduct a cross-country cross-sectional survey of coping in lockdown among people who smoked tobacco, consumed alcohol, vaped or used other smokeless products, and who may have used other drugs. Some challenges we encountered and adaptations to the study protocols were not specific to conducting research during a pandemic. Some sources of bias are usual for web-based surveys: they are susceptible to non-response; are less representative of the population being recruited due to self-selection and access is restricted to people with access to the internet, the necessary literacy and digital literacy skills to participate in online surveys (Dema et al., 2022). Potentially specific to surveying during the pandemic, Dema et al., (Dema et al., 2022) found a higher proportion of participants reported being in ‘fair health’ than a previous Health Survey for England. The authors considered that this difference may have been due to the impact of the pandemic on perceived health status (ibid).
Some short-comings we identified represent learnings that may be useful for informing future pandemic-focused research design.
One learning was that our hypothesis relied on research on the negative effects of isolated disasters or terrorist attacks. Pre-pandemic research on the health and social effects of natural disasters and isolated terrorist attacks led us to expect that similar effects on substance use might be experienced by communities or groups who experienced lockdown. There were, however, too many differences between these types of disasters to support that hypothesis.
The pandemic was experienced globally. Governments of different countries responded in different ways. NZ, for instance, implemented a severe and sustained lockdown of its borders aimed at preventing large numbers of COVID-19 related deaths. Some governments implemented economic relief packages that may have mitigated some of the severity of negative effects on mental health. However, in some countries financial or other support was not equitably distributed. Disaster relief included provision of food and housing, free health care and counselling (including by phone or video calls), and financial compensation. In addition, such events are often accompanied by a social outpouring of support at the community level, with aid workers and volunteers collaborating to help the victims. This type of social help forges bonds and encourages community cohesion (Kenney & Phibbs, 2015; Paton & Johnston, 2015). In contrast, lockdowns are by their nature isolating. Mass media campaigns urged people to avoid contact, and not only socially. The way many people worked, and their work environment, was radically altered. Family members who did not live in the same accommodation were forbidden proximal and physical contact with loved ones in some countries. Culturally significant practices – such as how people worshipped, how cultures celebrated life milestones, and their traditional grieving rituals – had to be missed, deferred, or significantly altered. For many people globally, internet access was unaffected. This enabled some familial and social contact to continue. Largely via mass media, including the internet, global and national agencies launched campaigns aimed at managing beliefs and opinions about COVID-19 and the response measures implemented by those agencies and governments.
Future pandemic-focused studies will have an immense literature of pandemic-related research to draw on for planning research on pandemics in the future. However, as with our study, much of the early pandemic-focused research was likely rushed. Risk of bias, especially early in the pandemic, was likely to be high. One large meta-analysis of COVID-19-related medical research published in the first six months of the pandemic found many publications were not peer-reviewed, intervention studies were limited by small case numbers, and the risk of bias was overly high (Raynaud et al., 2021). Of 7468 peer-reviewed articles assessed for risk of bias by Raynaud et al., (2021), 56.1% were opinion pieces.
3.1 Pandemic Preparedness Plans for Research
The COVID-19 pandemic highlighted the grave need for all countries to have pandemic response plans. Not just countries, but industries, companies, universities, government agencies, and institutions may benefit from having pandemic preparedness plans. Research sectors also should identify priorities for research. Concerns have been raised that the high level of displacement of research activity from non-COVID related topics, including the suspension of many medically-focused clinical trials, could have future negative consequences for people needing improved treatments for non-COVID related diseases (Riccaboni & Verginer, 2022). Waste of research resources was another concern (ibid).
To prevent a worsening of inequities, the United Kingdom National Institute for Health Research Innovations in Clinical Trial Design and Delivery for the Underserved project predicted that traditionally underserved groups would also likely be disproportionately affected by COVID-19. They called for COVID-19 research to ensure underserved groups be included and they published guidance on how to ensure that happens (Witman et al., 2020).
Research ethics boards fast-tracked assessment of COVID-19 related study protocols, lockdown measures curtailed the use of common research methods. Although the online cross-sectional methodology used in this study is well-established, establishing collaborations with researchers in other countries was difficult to achieve in the absence of an existing relationship. Pandemic research preparedness should include the formation of networks of potential co-investigators in advance. As Witham et al. (Witman et al., 2020) recommend, consumers and representatives of underserved groups should also be identified in advance and involved in the process of research priority setting.
Different economic factors, cultural considerations and languages meant there were significant differences in living conditions in each country. Laws restricting the use or access to the various products or substances we were studying were also different. The country-specific adaptations to the survey questions further invalidated the intent to compare experiences across the countries.
Different recruitment methods in each country, added to usual limitations of online survey methods, undermined representativeness. Groups with poor internet access and lower digital literacy would have been under-represented.
The four countries also varied in the range of lockdown measures implemented and intensity with which these were enforced. NZ established early severe border control restrictions to effectively mitigate the introduction of COVID-19 and minimize the spread. A small number of deaths were experienced. By contrast, there were no significant border control restrictions for Russian citizens and residents travelling home from abroad and lockdown measures and COVID-19 incidence and mortality varied across regions and territories. As a result, the incidence of COVID-19 infection, illness, and deaths experienced in the different countries would have had a very different incidence and intensity of “COVID stress syndrome” (Taylor et al., 2020). Research teams may face similar challenges to the populations they do research with and as recommended by Mobaraka, Elkazzaz and Rizkalla (Mobaraka et al., 2022) pandemic research preparedness plans need to factor in practical, emotional and compassionate support for team members adversely affected by COVID-19.
Responses may have been variably affected by the different intensity and stage of dissemination of non-science-based suppositions linking increased risk of COVID-19 infection and potential death from a COVID-19 infection, with smoking tobacco, alcohol use, vaping, and other substance use (Mallet et al., 2021).
Recommendations for pandemic preparedness tasks for researchers are summarized in Table 2.
Table 2. Pandemic prepreparedness checklist for research
· Identify and build relationships with potential co-investigators and consumer or other stakeholder representatives.
· Discuss and agree on authorship and waiver of authorship in the case of investigators becoming unavailable at the time of article submission.
· Identify and establish memorandums of agreement with substitute investigators for key roles.
· Identify and prioritize potential research questions, including input from underserved groups.
· Conduct scoping reviews to develop a preliminary science-based rationale and method for each study idea, including calculation of sample size.
· Scope out the potential ethical concerns, including consideration of vulnerabilities specific to a pandemic context.
· Identify translators if relevant.
· Identify independent protocol reviewers, ethics review boards, and their application requirements.
4. CONCLUSION
Studying unprecedented disasters in modern human history is important for identifying how to prevent or mitigate harm. Evidence-based policy decisions cannot be made without it. Research, however, needs to be robust and alert to a higher risk of bias. Pandemic preparedness should include having teams, study protocols, and ethics pre-approved for potential future events, and access to rapid-response type funding when they occur.
ACKNOWLEDGEMENTS
The authors thank co-investigators and coordinating center research staff associated with the conduct of the main study but who were not involved in the development of this paper: Carl V Phillips, Igor Nikiforov, Olga Karpenko, Saravana Kumar Paramanathen, Sree T Sucharitha, Aleksey Pavlichenko, Alexander Komarov, Vitalij Kazin, and George Kostyuk. We are also grateful to Michael Morris for early review of the draft paper and Dennis Viehland for editing assistance.
CONFLICT OF INTEREST DISCLOSURE STATEMENT
The funding for this study was obtained following submission of a researcher-initiated application to the Foundation for a Smoke-Free World, Inc. (FSFW), a US non-profit 501(c)(3) private foundation. This study is, under the terms of the grant agreement with FSFW, editorially independent of FSFW. The contents, selection and presentation of facts, as well as any opinions expressed herein, are the sole responsibility of the authors and under no circumstances should they be regarded as reflecting the positions of FSFW. Neither the authors nor the Centre of Research Excellence: Indigenous Sovereignty & Smoking have any commercial interests in products associated with any of the substance use behaviors investigated. M.G. has, over 10 years ago, received fees from pharmaceutical companies for consultancy re cessation medicines.
REVIEW COMMENTS
R1: Simon Luck
The paper makes a welcome effort to analyse the pitfalls and limitations of conducting cross-country research in times of pandemic, and provides a useful checklist of points to consider for future research in emergency or pandemic contexts.
The various factors limiting representativeness of the data collected and possibilities of comparing different countries are presented. However, as the difficulty to recruit participants in the context of the Covid-19 pandemic is one of the central aspects, it would be useful to have more information on the opportunities, constraints and choices made in advertising, and the results obtained (estimate of the number of people reached, number of respondents).
The authors could also make it clearer/more apparent to what extent the methodological difficulties faced are relative to their object of enquiry (alcohol/tobacco/substance use) or the pandemic context, and therefore to what extent the conclusions are relevant to other types of studies (this could be reflected in a more generic title).
R2: Maxi Heitmayer
This paper provides an interesting insight into the planning of a study on smoking and substance use habits during the wave of lockdowns that occurred during the Covid-19 pandemic. The authors share valuable insights not just into the processes and practicalities, but also the many challenges that arise during the planning and deployment of an empirical study - not least in times of crisis. More than an empirical study, though, the paper also constitutes an important science-historical document giving a snasphot of how one of the many research teams around the world coordinated and worked towards developing a means of inquiry to understand how people managed to cope with the impacts of the crisis.
Author’s Response
Thank you for your positive and constructive feedback.