Guideline on Good Pharmacovigilance Practices (GVP)

1.A. INTRODUCTION

This Module contains guidance for the establishment and maintenance of quality assured pharmacovigilance systems for marketing authorization holders (MAHs). How the systems of these organizations interact while undertaking specific pharmacovigilance processes is described in each respective Module of GVP.

The definition of a pharmacovigilance system is a system used by the MAH and by The Saudi Food and Drug Authority (SFDA) to fulfil the tasks and responsibilities listed in this guideline and designed to monitor the safety of authorized medicinal products and detect any change to their benefit-risk balance. The SFDA likewise maintains a pharmacovigilance system to fulfil its pharmacovigilance activities.

For performing their pharmacovigilance activities, MAHs, and the SFDA shall establish and use quality systems that are adequate and effective for this performance.

By following the overall quality objectives in I.B.4 and the guiding principle in I.B.5 to meet the needs of patients, healthcare professionals and the public in relation to the safety of medicines, the application of the quality system should be adapted to how crucial each pharmacovigilance task is for fulfilling the quality objectives for each medicinal product covered by a quality system.

In this Module, all applicable legal requirements are usually identifiable by the modal verb “shall". Guidance for the implementation of legal requirements is provided using the modal verb “should".​​

II.A. INTRODUCTION

The PSMF defined as a detailed description of the pharmacovigilance system used by the MAH with respect to one or more authorized medicinal products, and the minimum requirements for its content and maintenance are set out in the annex 1. The detailed requirements provided by the Commission Implementing Regulation are further supported by the guidance in this Module of the Good Vigilance Practice(s).

The PSMF shall be located either at the site in the KSA where the main pharmacovigilance activities of the MAH are performed or at the site in the KSA where the QPPV operates.

It is a requirement of the marketing authorization application that summary information about the pharmacovigilance system is submitted to the SFDA. This summary includes information on the location of the PSMF (see II.B.2.1). There is no requirement for variations for changes in the content of the PSMF, refer to the variation guideline.

This Module provides detailed guidance regarding the requirements for the PSMF, including its maintenance, content and associated submissions to SFDA.​​

III.A. INTRODUCTION

This Module contains guidance on the planning, conduct, reporting and follow-up of pharmacovigilance inspections in the KSA and outlines the role of the different parties involved. General guidance is provided under III.B., while III.C. covers the overall operation of pharmacovigilance inspections in the KSA. 

In order to determine that MAHs comply with pharmacovigilance obligations in the KSA, and to facilitate compliance, the SFDA shall conduct pharmacovigilance inspections of MAHs or any firms employed to fulfil MAH’s pharmacovigilance obligations. Such inspections shall be conducted by inspectors appointed by the SFDA and empowered to inspect the premises, records, documents and PSMF of the MAH or any firms employed by the MAH to perform the activities including third party organizations. In particular, MAHs are required to provide, on request, the PSMF, which will be used to inform inspection conduct (see Module II). 

The objectives of pharmacovigilance inspections are: 

● To determine that the MAH has personnel, systems and facilities in place to meet their pharmacovigilance obligations; 

● To identify, record and address non-compliance, which may pose a risk to public health; 

● To use the inspection results as a basis for enforcement action, where considered necessary. 

For MAHs of products, it is the responsibility of the SFDA to ensure that the MAH for the medicinal product satisfies the pharmacovigilance requirements. The PSMF shall be located either where the main pharmacovigilance activities of the MAH are performed or where the QPPV operates. The SFDA may conduct pre-authorization inspections to verify the accuracy and successful implementation of the existing or proposed pharmacovigilance system. 

Pharmacovigilance inspection programs will be implemented, which will include routine inspections scheduled according to a risk-based approach and will also incorporate “for cause” inspections, which have been triggered to examine suspected non-compliance or potential risks, usually with impact on a specific product(s).

Any non-compliance should also be rectified by the MAH in a timely manner through the implementation of a corrective and preventive action plan. 

If the outcome of the inspection is that the MAH does not comply with the pharmacovigilance obligations, the SFDA shall take the necessary measures to ensure that a MAH is subject to effective, proportionate and dissuasive penalties. ​

​IV.A. INTRODUCTION

The overall description and objectives of pharmacovigilance systems and quality systems for pharmacovigilance activities are referred to in Module I, while the specific pharmacovigilance processes are described in each respective Module of GVP.

This Module provides guidance on planning and conducting the legally required audits, the role, context and management of pharmacovigilance audit activity. This Module is intended to facilitate the performance of pharmacovigilance audits, especially to promote harmonization, and encourage consistency and simplification of the audit process. The principles in this Module are aligned with internationally accepted auditing standards, issued by relevant international auditing standardization organizations (the Institute of Internal Auditors; the International Organization for Standardisation (ISO); the Information Systems Audit and Control Association (ISACA); the International Auditing and Assurance Standards Board (IAASB); the International Organization of Supreme Audit Institutions (INTOSAI) and the Committee of Sponsoring Organizations (COSO)) and support a risk-based approach to pharmacovigilance audits.

Section IV.B. outlines the general structures and processes that should be followed to identify the most appropriate pharmacovigilance audit engagements and describes the steps which can be undertaken by MAHs or the SFDA, to plan, conduct and report upon an individual pharmacovigilance audit engagement. This section also provides an outline of the general quality system and record management practices for pharmacovigilance audit processes. ​

V.A. INTRODUCTION 

It is recognized that at the time of authorization, information on the safety of a medicinal product is relatively limited. This is due to many factors including the relatively small numbers of subjects in clinical trials compared with the intended treatment population, restricted population in terms of age, gender and ethnicity, restricted co-morbidities, restricted co-medications, restricted conditions of use, relatively short duration of exposure and follow up, and the statistical problems associated with looking at multiple outcomes.

A medicinal product is authorized on the basis that in the specified indication(s), at the time of authorization, the benefit-risk balance is judged to be positive for the target population. A typical medicinal product will have multiple risks attached to it and individual risks will vary in terms of severity, effect on individual patients and public health impact. However, not all actual or potential risks will have been identified at the time when an initial authorization is sought and many of the risks associated with the use of a medicinal product will only be discovered and characterized in the post-authorization phase. The aim of a risk management plan (RMP) is to document the risk management system considered necessary to identify, characterize and minimize a medicinal product’s important risks. To this end, the RMP contains: 

1. The identification or characterization of the safety profile of the medicinal product, with emphasis on important identified and important potential risks and missing information, and also on which safety concerns need to be managed proactively or further studied (the ‘safety specification’); 

2. The planning of pharmacovigilance activities to characterise and quantify clinically relevant risks, and to identify new adverse reactions (the ‘pharmacovigilance plan’);   

3. The planning and implementation of risk minimisation measures, including the evaluation of the effectiveness of these activities (the ‘risk minimisation plan’).  

As knowledge regarding a medicinal product’s safety profile increases over time, so will the risk management plan change.  

This module includes the principles of risk minimization, it should be read in conjunction with GVP Module XVI and GVP Module XVI Addendum I on educational materials.​

VI.A. INTRODUCTION

This Module addresses the legal requirements by SFDA, as regards the collection, data management and submission of individual reports of suspected adverse reactions (serious and non-serious) associated with medicinal products for human use authorized in the KSA.

The guidance provided in this Module does not address the collection, management and reporting of events or patterns of use, which do not result in suspected adverse reactions (e.g. asymptomatic overdose, abuse, off-label use, misuse or medication error) or which do not require to be reported as ICSR. This information may however need to be collected and presented in PSUR/PBRERs for the interpretation of safety data or for the benefit risk evaluation of medicinal products. With regard to this, guidance provided in Module VII applies. ​​

VII.A. INTRODUCTION 

PSUR/PBRERs are pharmacovigilance documents intended to provide an evaluation of the benefit-risk balance of a medicinal product for submission by MAHs at defined time points during the post-authorization phase. 

The format of PSUR/PBRERs shall follow the structure described in the annex 1. This Module provides guidance on the preparation, submission and assessment of PSUR/PBRERs. 

The scope, objectives, format and content of the PSUR/PBRER are described in VII.B.. The required format and content of PSUR/PBRERs in the KSA are based on those for the Periodic Benefit Risk Evaluation Report (PBRER) described in the ICH-E2C(R2) guideline (see Annex IV ICH-E2C(R2)). The PBRER replaces the PSUR/PBRER format previously described in the ICH-E2C(R1). In the KSA, the report shall be described and named as PSUR/PBRER. 

Further details and guidance for the submission of PSUR/PBRERs in the KSA, including the frequency of submission are provided in VII.C.

MAHs should submit PSUR/PBRERs to the SFDA according to the following timelines: 

• within 70 calendar days of the data lock point for PSUR/PBRERs covering intervals up to 12 months (including intervals of exactly 12 months); and 

• within 90 calendar days of the data lock point for PSUR/PBRERs covering intervals in excess of 12 months; 

• the timeline for the submission of ad hoc PSUR/PBRERs requested by the SFDA will normally be specified in the request, otherwise the ad hoc PSUR/PBRERs should be submitted within 90 calendar days of the data lock point. 

It should be noted that detailed listings of individual cases shall not be included systematically. The PSUR/PBRER should focus on summary information, scientific safety assessment and integrated benefit-risk evaluation.  

The obligations imposed in respect of PSUR/PBRERs should be proportionate to the risks posed by medicinal products. PSUR/PBRER reporting should therefore be linked to the RMPs (RMPs) of a medicinal product (see Module V). The “modular approach” of the PSUR/PBRER described in VII.B.5. aims to minimize duplication and improve efficiency during the preparation and review of PSUR/PBRERs along with other regulatory documents such as the development safety update report (DSUR) or the safety specification in the RMP, by enabling the common content of particular sections where appropriate to be utilized interchangeably across different PSUR/PBRERs, DSURs and RMPs. 

The new legislation also waives the obligation to submit PSUR/PBRERs routinely for generic medicinal products, well-established use medicinal products, homeopathic medicinal products and traditional herbal medicinal products. For such products, PSUR/PBRERs shall be submitted where there is a condition in the marketing authorization or when requested by the SFDA on the basis of concerns relating to pharmacovigilance data or due to the lack of PSUR/PBRERs for an active substance after its authorization. However, if the generic product is the first registered in Saudi Arabia, the MAH should submit the PSUR/PBRER to SFDA according to timelines provided in VII.C.

SFDA shall assess PSUR/PBRERs to determine whether there are new risks or whether risks have changed or whether there are changes to the benefit-risk balance of medicinal products. 

As part of the assessment, it should be considered whether further investigations need to be carried out and whether any action concerning the marketing authorizations of products containing the same active substance or the same combination of active substances, and their product information is necessary. 

This GVP Module VII may be reviewed and updated following further development and finalization of the ICH-E2C(R2) guideline on PBRER. ​

​VIII.A. INTRODUCTION

A post-authorization safety study (PASS) is defined as any study relating to an authorized medicinal product conducted with the aim of identifying, characterizing or quantifying a safety hazard, confirming the safety profile of the medicinal product, or of measuring the effectiveness of risk management measures. 

A PASS may be initiated, managed or financed by a MAH voluntarily, or pursuant to an obligation imposed by the SFDA. This Module concerns PASS which are interventional or non-interventional studies and does not address pre-clinical safety studies. 

A PASS is non-interventional if the following requirements are cumulatively fulfilled: 

• the medicinal product is prescribed in the usual manner in accordance with the terms of the marketing authorization; 

• the assignment of the patient to a particular therapeutic strategy is not decided in advance by a trial protocol but falls within current practice and the prescription of the medicine is clearly separated from the decision to include the patient in the study; and 

• no additional diagnostic or monitoring procedures are applied to the patients and epidemiological methods are used for the analysis of collected data. 

Non-interventional studies are defined by the methodological approach used and not by its scientific objectives. Non-interventional studies include database research or review of records where all the events of interest have already happened (this may include case-control, cross-sectional, cohort or other study designs making secondary use of data). Non-interventional studies also include those involving primary data collection (e.g. prospective observational studies and registries in which the data collected derive from routine clinical care), provided that the conditions set out above are met. In these studies, interviews, questionnaires and collection of blood samples may be performed as part of normal clinical practice. 

If a PASS is a clinical trial, other SFDA guidelines should be followed (Regulation and Requirements for Conducting Clinical Trials on Drug, Good Clinical Practice Guidelines). 

The purposes of this Module are to:  

• provide general guidance for the transparency, scientific standards and quality standards of non-interventional PASS conducted by MAHs (VIII.B.); 

• describe procedures whereby the SFDA may impose to a MAH an obligation to conduct a clinical trial or a non-interventional study (VIII.C.), and the impact of this obligation on the risk management system; 

• describe procedures that apply to non-interventional PASS imposed as an obligation for the protocol oversight and reporting of results and for changes to the marketing authorization following results (VIII.C.). 

The guidance in VIII.B. applies to non-interventional PASS which are initiated, managed or financed by a MAH and conducted in the KSA. This guidance should also be used for studies conducted outside the KSA which have been imposed or required by the SFDA. In VIII.B., some legal requirements which are applicable to studies conducted pursuant to an obligation are recommended to all PASS in order to support the same level of transparency, scientific standards and quality standards for all PASS. This applies, for example, to the format and content of study protocols, abstracts and final study reports. A distinction is made in the text between situations where the provision of the guidance represents a legal requirement or a recommendation.

This guidance applies to studies initiated, managed or financed by a MAH as well as those conducted by a third party on behalf of the MAH. This guidance applies to studies that involve primary collection of safety data directly from patients and healthcare professionals and those that make secondary use of data previously collected from persons and healthcare professionals for another purpose. ​

IX.A. INTRODUCTION

The objectives of this Module are:

• To provide general guidance and requirements on structures and processes involved in signal management (section IX.A.);

• To describe how these structures and processes are applied in the setting of the pharmacovigilance and regulatory network (section IX.B.).

Individual organisations may follow alternative signal management processes and terminology but should encompass the general principles outlined in this Module.​

​​X.A. INTRODUCTION

Pharmacovigilance is a vital public health function with the aim of rapidly detecting and responding to potential safety hazards associated with the use of medicinal products. 

A medicinal product is authorized on the basis that, its benefit-risk balance is considered to be positive at that time for a specified target population within its approved indication(s). However, not all risks can be identified at the time of initial authorization and some of the risks associated with the use of a medicinal product emerge or are further characterized in the post-authorization phase of the product’s lifecycle. To strengthen the safety monitoring of medicinal products, , a framework for enhanced risk proportionate post-authorization data collection for medicinal products has been introduced, including the concept of additional monitoring for certain medicinal products.

The SFDA shall set up, maintain and make public a list of medicinal products that are subject to additional monitoring (hereafter referred to as “the list”). These medicinal products will be readily identifiable by an inverted equilateral black triangle. That triangle will be followed by an explanatory statement in the SPC as follows:

“This medicinal product is subject to additional monitoring. This will allow quick identification of new safety information. Healthcare professionals are asked to report any suspected adverse reactions to the NPC in Saudi Arabia. Reporting information is provided in “report any side effects” section.

A similar statement will also be included in the package leaflet. This explanatory statement should encourage healthcare professionals and patients to report all suspected adverse reactions. 

This GVP takes into account the new provisions relating to the list of products which require additional monitoring.

Post-authorization spontaneous reports remain a cornerstone of pharmacovigilance. Data from spontaneous reports is a key source of information for signal detection activities (see Module IX). Increasing the awareness of healthcare professionals and patients of the need to report suspected ADRs and encouraging their reporting is therefore an important means of monitoring the safety profile of a medicinal product.

The concept of additional monitoring originates primarily from the need to enhance the ADR reporting rates for newly authorized products for which the safety profile might not be fully characterized or for products with newly emerging safety concerns that also need to be better characterized. The main goals are to collect additional information as early as possible to further elucidate the risk profile of products when used in clinical practice and thereby informing the safe and effective use of medicinal products.​

XV.A. INTRODUCTION

This Module provides guidance to MAHs and the SFDA on how to communicate and coordinate safety information in the KSA. Communicating safety information to patients and healthcare professionals is a public health responsibility and is essential for achieving the objectives of pharmacovigilance in terms of promoting the rational, safe and effective use of medicines, preventing harm from adverse reactions and contributing to the protection of patients' and public health (see Module I).

Safety communication is a broad term covering different types of information on medicines, including statutory information as contained in the product information (i.e. the SPC, PIL and the labelling of the packaging) and public assessment reports. Although some principles in this Module (i.e. Section XV.B.1 and B.2.) apply to all types of safety communication, the module itself focuses on the communication of 'new or emerging safety information', which means new information about a previously known or unknown risk of a medicine which has or may have an impact on a medicine's benefit-risk balance and its condition of use. Unless otherwise stated, the term 'safety communication' in this module should be read as referring to emerging safety information.

Communication of important new safety information on medicinal products should consider the views and expectations of concerned parties, including patients and healthcare professionals, with due consideration given to relevant legislation. This Module addresses some aspects of the interaction with concerned parties and supplements the specific guidance given in Module XI on public participation.

Communication is distinct from transparency, which aims to provide public access to information related to data assessment, decision-making and safety monitoring performed by the SFDA.

Section XV.B. of this Module describes principles, means of safety communication, dissemination of safety communications and guidance on the coordination. This section provides particular consideration to direct healthcare professional communications (DHPCs) and provide specific guidance for preparing them. This is because of the importance of DHPCs in targeting healthcare professionals and because of the level of coordination required between MAHs and the SFDA in their preparation.

XVI.A. INTRODUCTION

Risk management includes the identification, characterisation (including quantification), prevention and minimisation of risks. Risk management systems consist of pharmacovigilance activities and interventions relating to individual medicinal products for this purpose, including the assessment of the effectiveness of those activities and interventions. The objectives of risk minimisation are achieved through the implementation of risk minimisation measures (RMM) required by the SFDA and generation of evidence that these measures are effective.

Monitoring RMM outcomes refers to adherence to RMM by healthcare professionals and patients and achieving the objectives of RMM. Monitoring and amending RMM, if warranted, aim at ensuring that the benefits of a particular medicinal product continue to exceed the risks by the greatest achievable margin. The assessment of the effectiveness of RMM is important for risk management with an iterative process of evaluation, correction and re-evaluation of RMM, which is integral to the lifecycle benefit-risk assessment of medicinal products.

This GVP Module should be read together with GVP Module V on risk management systems as documented through risk management plans (RMPs) and on details of routine RMM, GVP Module VIII on post-authorisation safety studies (PASS), GVP Module XV on safety communication and the Addenda of this GVP Module as referenced.

XVI.B. describes criteria for selection, development, implementation and co-ordination of RMM, in particular of additional RMM, and the principles and concepts of the evaluation of RMM effectiveness. XVI.C. describes the related roles and responsibilities of marketing authorisation holders and the SFDA. It also reflects the contribution of healthcare professional and patient representatives.

The term 'patient' in this guidance covers patients using or considering the use of a medicine, parents and other carers, and patient and consumer representatives. It also includes the (unborn) child in the case of exposure during pregnancy.

​XVI.ADD.I​.1. INTRODUCTION

Risk Minimization Measures are public health interventions intended to prevent or reduce the occurrence of adverse reactions associated with the exposure to a medicine, or to reduce their severity or impact on the patient.

Risk Minimization Measures include:

• Educational materials such as healthcare provider guides, prescriber checklists, or patient alert cards.

Educational programs are additional risk minimization measures (RMM) (see GVP Module XVI) and usually require educational materials based on targeted communication with the aim to supplement the information in the summary product characteristics (SPC) and package Information leaflet (PIL). 

When the development and distribution of educational material is recommended by the SFDA, a draft of educational materials should be submitted to the SFDA and these educational materials shall implement the key elements. Alternatively, the exact content of educational materials could be agreed and also become part of the summary of product characteristics (SPC) and/or the package Information leaflet (PIL), as applicable.

• Direct Healthcare Professional Communications (DHPCs) which are communication interventions aimed to deliver important information directly to individual healthcare professionals by the marketing authorization holder (MAH) to inform them of the need to take certain actions or adapt their practices in relation to a medicinal product. 

The aim of this Addendum is to provide general guidance on the local requirements for drafting educational materials and Direct Healthcare Professional Communication (DHPC) letters that are part of Risk Minimization Measures.​​

XVI.ADD.II.1. Introduction

This Addendum to GVP Module XVI provides additional guidance for marketing authorisation holders and SDFA on data sources and methodologies for monitoring outcomes of risk minimisation measures (RMM) in line with the principles for RMM effectiveness evaluation laid down in GVP Module XVI.

Depending on the risk minimisation objective, studies evaluating RMM effectiveness may integrate different quantitative measurements and qualitative research approaches to evaluate risk minimisation outcomes for individual tools or sets of RMM described in GVP Module XVI. Risk knowledge, behavioural changes and health outcomes may be considered, and in this respect the guidance on objectives of effectiveness evaluation in GVP Module XVI should be followed. The Addendum also provides guidance on the reporting of the results of studies evaluating the effectiveness of RMM.

1. Vora P, Artime E, Soriano-Gabarró M, Qizilbash N, Singh V, Asiimwe A. A review of studies evaluating the effectiveness of risk minimisation measures in Europe using the European Union electronic Register of Post-Authorization Studies.Pharmacoepidemiol Drug Saf [Internet]. 2018 Apr 16 [cited 2018 Apr 19]; Available from: http://doi.wiley.com/10.1002/pds.4434 

2. Farcas A, Huruba M, Mogosan C. Study design, process and outcome indicators of post‐authorization studies aimed at evaluating the effectiveness of risk minimization measures in the EU PAS Register. Br J Clin Pharmacol. 2019 Mar;85(3):476–91.  

3. Goedecke T, Morales DR, Pacurariu A, Kurz X. Measuring the impact of medicines regulatory interventions - Systematic review and methodological considerations: Methods for measuring impact of medicines regulatory interventions. Br J Clin Pharmacol. 2018 Mar;84(3):419–33.  

4. Pacurariu A, Plueschke K, McGettigan P, Morales DR, Slattery J, Vogl D, et al. Electronic healthcare databases in Europe: descriptive analysis of characteristics and potential for use in medicines regulation. BMJ Open. 2018 Sep;8(9):e023090.  

5. Flick U. An Introduction to Qualitative Research. 3rd ed. London: Sage Publications Ltd.; 2006.  

6. Lingard L, Albert M, Levinson W. Grounded theory, mixed methods, and action research. BMJ. 2008 Aug 7;337:a567.  

7. Creswell JW, Plano Clark VL. Designing and conducting mixed methods research. 3rd ed. London: Sage Publications Ltd.; 2017.  

8. Silvermann David, editor. Qualitative Research. 4th ed. London: Sage Publications Ltd.; 2016.  

9. Kuper A, Lingard L, Levinson W. Critically appraising qualitative research. BMJ. 2008 Aug 7;337:a1035.  

10. Campbell A, Taylor BJ, McGlade A. Research Design in Social Work: Qualitative and Quantitative Methods Transforming Social Work Practice Series. Learning Matters; 2016. 

11. Bouvy JC, Blake K, Slattery J, De Bruin ML, Arlett P, Kurz X. Registries in European post-marketing surveillance: a retrospective analysis of centrally approved products, 2005-2013. Pharmacoepidemiol Drug Saf. 2017 Dec;26(12):1442–44950.  

12. McGettigan P, Alonso Olmo C, Plueschke K, Castillon M, Nogueras Zondag D, Bahri P, et al. Patient Registries: An Underused Resource for Medicines Evaluation: Operational proposals for increasing the use of patient registries in regulatory assessments. Drug Saf. 2019 Jul 13;

13. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015 Mar 19;350:h1258.  

14. Taylor BJ, Killick C, McGlade A. Understanding & Using Research in Social Work [Internet]. 1 Oliver’s Yard, 55 City Road London EC1Y 1SP: SAGE Publications, Inc.; 2015 [cited 2020 Apr 22]. Available from:  http://sk.sagepub.com/books/understanding-and-using-research-in-social-work 

15. Creswell JW. Qualitative Inquiry and Research Design: Choosing Among Five Approaches. 2012th ed. London: Sage Publications Ltd.; 2012.  

16. Tindall L. J.A. Smith, P. Flower and M. Larkin (2009), Interpretative Phenomenological Analysis: Theory, Method and Research: London: Sage. Qual Res Psychol. 2009 Nov 25;6(4):346–7. 

17. Bahri P, Fogd J, Morales D, Kurz X, ADVANCE consortium. Application of real-time global media monitoring and “derived questions” for enhancing communication by regulatory bodies: the case of human papillomavirus vaccines. BMC Med. 2017 02;15(1):91.  

18. Charmaz Kathy. Introducing Qualitative Methods series. 2nd ed. London: Sage Publications Ltd; 2014.  

19. Newman PA, Seiden DS, Roberts KJ, Kakinami L, Duan N. A small dose of HIV? HIV vaccine mental models and risk communication. Health Educ Behav Off Publ Soc Public Health Educ. 2009 Apr;36(2):321–33.  

20. Omedo M, Ogutu M, Awiti A, Musuva R, Muchiri G, Montgomery SP, et al. The effect of a health communication campaign on compliance with mass drug administration for schistosomiasis control in western Kenya--the SCORE project. Am J Trop Med Hyg. 2014 Nov;91(5):982–8.  

21. Yin RK. Case Study Research: Design and Methods. 5th ed. London: Sage Publications Ltd.; 2013.  

22. Bahri P, Morales DR, Inoubli A, Dogné JM, Straus SMJM. Proposals for engaging patients and healthcare professionals in risk minimisation from an analysis of stakeholder input to the EU valproate assessment using the novel Analysing Stakeholder Safety Engagement Tool (ASSET). Drug Saf. 2020;(accepted).  

23. Bradbury H. The SAGE Handbook of Action Research [Internet]. 1 Oliver’s Yard, 55 City Road London EC1Y 1SP: SAGE Publications Ltd; 2015 [cited 2020 Apr 7]. Available from: http://methods.sagepub.com/book/the-sage-handbook-of-action-research-3e 

24. Taylor BJ, Moorhead SA. The social sciences. In: In Bahri P (ed) Communicating about risks and safe use of medicines: real life and applied research. Singapore: Springer Nature; 2020.  

25. Noyes J, Booth A, Cargo M, Flemming K, Harden A, Harris J, Garside R, Hannes K, Pantoja T, Thomas J. Chapter 21: Qualitative evidence. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors) Cochrane Handbook for Systematic Reviews of Interventions version 60 (updated July 2019). Cochrane; 2019.   

26. Prieto L, Spooner A, Hidalgo-Simon A, Rubino A, Kurz X, Arlett P. Evaluation of the effectiveness of risk minimization measures: effectiveness of risk minimisation. Pharmacoepidemiol Drug Saf. 2012 Aug;21(8):896–9.  

27. Weatherburn CJ, Guthrie B, Dreischulte T, Morales DR. Impact of medicines regulatory risk communications in the United Kingdom on prescribing and clinical outcomes: systematic review, time series analysis and meta-analysis. Br J Clin Pharmacol. 2019 Aug 29; 

28. Cochrane Effective Practice and Organisation of Care (EPOC). Interrupted time series (ITS) analyses. EPOC Resources for review authors, 2017. [Internet]. Available from: https://epoc.cochrane.org/sites/epoc.cochrane.org/files/public/uploads/Resources-for authors2017/interrupted_time_series_analyses.docx  

29. Zhang F, Wagner AK, Ross-Degnan D. Simulation-based power calculation for designing interrupted time series analyses of health policy interventions. J Clin Epidemiol. 2011 Nov;64(11):1252–61.  

30. Bernal JL, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: a tutorial. Int J Epidemiol. 2017 01;46(1):348–55.  

31. Kim HJ, Fay MP, Feuer EJ, Midthune DN. Permutation tests for joinpoint regression with applications to cancer rates. Stat Med. 2000 Feb 15;19(3):335–51.  

32. Hedenmalm K, Kurz X, Morales D. Effect of withdrawal of fusafungine from the market on prescribing of antibiotics and other alternative treatments in Germany: a pharmacovigilance impact study. Eur J Clin Pharmacol [Internet]. 2019 Mar 5 [cited 2019 Apr 24]; Available from: http://link.springer.com/10.1007/s00228-019-02650-z 

33. Hernandez-Santiago V, Marwick CA, Patton A, Davey PG, Donnan PT, Guthrie B. Time series analysis of the impact of an intervention in Tayside, Scotland to reduce primary care broad-spectrum antimicrobial use. J Antimicrob Chemother. 2015 Aug;70(8):2397–404.  

34. Hedenmalm K, Blake K, Donegan K, Macia M-A, Gil M, Williams J, et al. A European multicenter drug utilisation study of the impact of regulatory measures on prescribing of codeine for pain in children. Pharmacoepidemiol Drug Saf [Internet]. 2019 Jun 20 [cited 2019 Jul 23]; Available from: http://doi.wiley.com/10.1002/pds.4836 

35. Zomerdijk IM, Ruiter R, Houweling LMA, Herings RMC, Sturkenboom MCJM, Straus SMJM, et al. Isotretinoin exposure during pregnancy: a population-based study in The Netherlands. BMJ Open. 2014 Nov 12;4(11):e005602.  

36. Morales DR, Morant SV, MacDonald TM, Mackenzie IS, Doney ASF, Mitchell L, et al. Impact of EMA regulatory label changes on systemic diclofenac initiation, discontinuation, and switching to other pain medicines in Scotland, England, Denmark, and The Netherlands. Pharmacoepidemiol Drug Saf [Internet]. 2020 Jan 3 [cited 2020 Jan 28]; Available from: https://onlinelibrary.wiley.com/doi/abs/10.1002/pds.4955 

37. Sultana J, Fontana A, Giorgianni F, Tillati S, Cricelli C, Pasqua A, et al. Measuring the Effectiveness of Safety Warnings on the Risk of Stroke in Older Antipsychotic Users: A Nationwide Cohort Study in Two Large Electronic Medical Records Databases in the United Kingdom and Italy. Drug Saf [Internet]. 2019 Sep 25 [cited 2019 Oct 8]; Available from: http://link.springer.com/10.1007/s40264-019-00860-z 

38. Eccles M, Grimshaw J, Campbell M, Ramsay C. Research designs for studies evaluating the effectiveness of change and improvement strategies. Qual Saf Healthcare. 2003 Feb;12(1):47–52.   

39. Smith MY, Russell A, Bahri P, Mol PGM, Frise S, Freeman E, et al. The RIMES Statement: A Checklist to Assess the Quality of Studies Evaluating Risk Minimization Programs for Medicinal Products. Drug Saf.2018 Apr;41(4):389–401​