Authors
Liberty A. Walton, Brandie K. Taylor, and Larry S. Solomon
Abstract
The National Institute of Allergy and Infectious Diseases (NIAID) of the National Institutes of Health launched a six-year grant in 2013 to develop, design, implement, and manage a clinical research agenda to increase knowledge of antibacterial resistance. This program—the Leadership Group for a Clinical Research Network on Antibacterial Resistance (ARLG)—represents a substantial NIAID investment and brings together researchers from around the world to tackle the increase in antibacterial resistance (AR). As the current funding for the ARLG expires, NIAID sought to examine this large-scale program to assess outcomes to-date and to identify ways to improve the complex program structure and implementation. The mixed-methods approach to this study included a complex portfolio analysis, in-depth interviews, and a survey of the research community. This poster illustrates the process and outcome evaluation methodology, key study findings, and study strengths and limitations.
Evaluation Purpose
ARLG focuses on four research areas: multidrug resistant (MDR) gram-positive bacteria, MDR gram-negative bacilli, diagnostics, and antimicrobial stewardship. As the ARLG neared the end of its current funding cycle, NIAID requested a comprehensive review of the program, including the program design and outcomes to-date. Evaluation findings were to be used to identify areas to improve the design of the program for its next funding cycle.
The evaluation focused on:
- ARLG’s facilitation of the research process for AR clinical researchers
- Impact of ARLG’s program structure on its scientific contribution
- Optimization of available clinical sites
- ARLG’s contributions to AR clinical research
- Prioritization of future scientific priorities of ARLG
Evaluation Design
- Portfolio Analysis
- Bibliometric analysis
- Data pulled from NIH grants database, PubMed/Medline, SPIRES, USPTO Database, NIH ExPORTER/REPORTER, Web of Science, iCite, CrossRef, and other internal NIAID and ARLG documentation
- Network analysis
- Analysis of other outcomes data
- Content analysis of publications and grant documents
- Bibliometric analysis
- In-Depth Interviews (n=20)
- 10 NIH program staff engaged in ARLG grant oversight
- 10 ARLG personnel, including committee members and principal investigators (PIs)
- Conducted in-person and via phone
- Average interview was 51 minutes
- Analyzed qualitatively using Dedoose software
- Surveys with ARLG “Users” and Community Members (n=165)
- ARLG community satisfaction survey distributed to 401 potential respondents
- Response rate of 41%
- Diverse target audience including investigators, committee members, fellows, mentees/early stage investigators, collaborators, clinical site personnel and study applicants
Key Findings
Recommendations
- Program Areas to Maintain
- Strong collaborations
- Focus on clinical design and problem-solving
- High levels of expertise
- Research funding
- Effective leadership
- Program Areas for Improvement
- Interest survey and application process
- Improved communications
- Adjustments to some study review and implementation processes
- Increase funding
- Enhance collaborations
- Improve mentoring
- Invite a broader membership
Limitations
- Given ARLG’s unique role, no comparison group options were available with accessible outcomes data
- The views of the survey and interview participants may not be generalizable and may overestimate ARLG’s value
- Publication quality was assessed via RCR, journal impact factor, and number of citations; however, these are imperfect metrics
- Data may be limited by the quality and completeness of the data sources
Contact Information
NIAIDEvaluation2@niaid.nih.gov
Acknowledgements
DMID Staff; Ripple Effect Communications, Inc.
This work has been funded in whole or in part with Federal funds from NIAID, under contract HHSN272201600016I.
Poster Presented at the 2018 American Evaluation Association (AEA) Annual Meeting:
- Walton, L., Taylor, B., & Solomon, L. (2018, October). Process and Outcome Evaluation of the Antibacterial Resistance Leadership Group at the National Institute of Allergy and Infectious Diseases. Presented at the Annual AEA Conference Meeting, Cleveland, OH.