Authors
Liberty A. Walton, Brandie K. Taylor, and Larry S. Solomon
Abstract
The National Institute of Allergy and Infectious Diseases (NIAID) of the National Institutes of Health launched a six-year grant in 2013 to develop, design, implement, and manage a clinical research agenda to increase knowledge of antibacterial resistance. This program—the Leadership Group for a Clinical Research Network on Antibacterial Resistance (ARLG)—represents a substantial NIAID investment and brings together researchers from around the world to tackle the increase in antibacterial resistance (AR). As the current funding for the ARLG expires, NIAID sought to examine this large-scale program to assess outcomes to-date and to identify ways to improve the complex program structure and implementation. The mixed-methods approach to this study included a complex portfolio analysis, in-depth interviews, and a survey of the research community. This poster illustrates the process and outcome evaluation methodology, key study findings, and study strengths and limitations.
Evaluation Purpose
ARLG focuses on four research areas: multidrug resistant (MDR) gram-positive bacteria, MDR gram-negative bacilli, diagnostics, and antimicrobial stewardship. As the ARLG neared the end of its current funding cycle, NIAID requested a comprehensive review of the program, including the program design and outcomes to-date. Evaluation findings were to be used to identify areas to improve the design of the program for its next funding cycle.
The evaluation focused on:
- ARLG’s facilitation of the research process for AR clinical researchers
- Impact of ARLG’s program structure on its scientific contribution
- Optimization of available clinical sites
- ARLG’s contributions to AR clinical research
- Prioritization of future scientific priorities of ARLG
Evaluation Design
- Portfolio Analysis
- Bibliometric analysis
- Data pulled from NIH grants database, PubMed/Medline, SPIRES, USPTO Database, NIH ExPORTER/REPORTER, Web of Science, iCite, CrossRef, and other internal NIAID and ARLG documentation
- Network analysis
- Analysis of other outcomes data
- Content analysis of publications and grant documents
- Bibliometric analysis
- In-Depth Interviews (n=20)
- 10 NIH program staff engaged in ARLG grant oversight
- 10 ARLG personnel, including committee members and principal investigators (PIs)
- Conducted in-person and via phone
- Average interview was 51 minutes
- Analyzed qualitatively using Dedoose software
- Surveys with ARLG “Users” and Community Members (n=165)
- ARLG community satisfaction survey distributed to 401 potential respondents
- Response rate of 41%
- Diverse target audience including investigators, committee members, fellows, mentees/early stage investigators, collaborators, clinical site personnel and study applicants
Key Findings
The chart provides data on perceptions of the ARLG program structure. The total survey population was 165 participants.
- 48% of respondents agreed ARLG was receptive to antibacterial resistance community feedback and 9% disagreed. However, 43% of respondents stated the did not have any knowledge of this survey element.
- 49% of respondents agreed ARLG organizational structure was effective and 13% disagreed. However, 38% of respondents stated the did not have any knowledge of this survey element.
- 55% of respondents agreed ARLG was accessible to a wide variety of disciplines and 15% disagreed. However, 30% of respondents stated the did not have any knowledge of this survey element.
- 57% of respondents agreed ARLG communicated effectively with the antibacterial resistance community and 23% disagreed. However, 20% of respondents stated the did not have any knowledge of this survey element.
- 65% of respondents agreed ARLG was accessible to investigators with varying experience and 16% disagreed. However, 19% of respondents stated the did not have any knowledge of this survey element.
The chart provides data on perceptions of the ARLG program structure. The total survey population was 165 participants.
- 48% of respondents agreed ARLG was receptive to antibacterial resistance community feedback and 9% disagreed. However, 43% of respondents stated the did not have any knowledge of this survey element.
- 49% of respondents agreed ARLG organizational structure was effective and 13% disagreed. However, 38% of respondents stated the did not have any knowledge of this survey element.
- 55% of respondents agreed ARLG was accessible to a wide variety of disciplines and 15% disagreed. However, 30% of respondents stated the did not have any knowledge of this survey element.
- 57% of respondents agreed ARLG communicated effectively with the antibacterial resistance community and 23% disagreed. However, 20% of respondents stated the did not have any knowledge of this survey element.
- 65% of respondents agreed ARLG was accessible to investigators with varying experience and 16% disagreed. However, 19% of respondents stated the did not have any knowledge of this survey element.
ARLG has collaborated with 59 unique organizations.
- Top figure presents a summary of the location and type of collaborating organizations and their locations. Overall, 25% (15 of 59) of the organizations that collaborated with ARLG were based at international locations. Similar proportions of internationally based collaborating organizations were found within diagnostic companies (23%) and biotechnology or biochemistry companies (20%), but a higher proportion of pharmaceutical or biopharmaceutical companies (46%) were internationally based. Consultants and government organization collaborators were all based in the United States.
- Bottom figure presents number of respondents that indicated that the ARLG has helped them collaborate with each of five types of collaborators. The two most common collaboration partners were other AR investigators (33%, 50 of 151) and ARLG members (24%, 36 of 151). Collaborations with industry, international collaborators, and investigators outside the AR field were reported less frequently. A total of 42% (63 of 151) of respondents indicated that the ARLG has not helped them collaborate with others
ARLG has collaborated with 59 unique organizations.
- Top figure presents a summary of the location and type of collaborating organizations and their locations. Overall, 25% (15 of 59) of the organizations that collaborated with ARLG were based at international locations. Similar proportions of internationally based collaborating organizations were found within diagnostic companies (23%) and biotechnology or biochemistry companies (20%), but a higher proportion of pharmaceutical or biopharmaceutical companies (46%) were internationally based. Consultants and government organization collaborators were all based in the United States.
- Bottom figure presents number of respondents that indicated that the ARLG has helped them collaborate with each of five types of collaborators. The two most common collaboration partners were other AR investigators (33%, 50 of 151) and ARLG members (24%, 36 of 151). Collaborations with industry, international collaborators, and investigators outside the AR field were reported less frequently. A total of 42% (63 of 151) of respondents indicated that the ARLG has not helped them collaborate with others
- Top figure presents publication trends from 2014 to 2017. There were 16 publications in 2014; 19 in 2015; 22 in 2016; 15 in 2017. There was a total of 72 ARLG-affiliated scientific articles through September 2017. ARLG publications have been cited 908 times, with an average of 13 citations per publication
- Bottom figure presents citations per publication, Journal Impact Factor, and Relative Citation Ratio (RCR) data. Of the 71 ARLG publications, the mean Journal Impact Factor score was 71 with a Relative Citation Rate of 35. When evaluating the mean scores, ARLG publications were cited 3.2 times more often than other NIH-funded publications in the same field and averaged a journal Impact Factor of 6.2.
- Top figure presents publication trends from 2014 to 2017. There were 16 publications in 2014; 19 in 2015; 22 in 2016; 15 in 2017. There was a total of 72 ARLG-affiliated scientific articles through September 2017. ARLG publications have been cited 908 times, with an average of 13 citations per publication
- Bottom figure presents citations per publication, Journal Impact Factor, and Relative Citation Ratio (RCR) data. Of the 71 ARLG publications, the mean Journal Impact Factor score was 71 with a Relative Citation Rate of 35. When evaluating the mean scores, ARLG publications were cited 3.2 times more often than other NIH-funded publications in the same field and averaged a journal Impact Factor of 6.2.
Future Priority Areas: Figure displaying survey respondents’ priority ratings for areas of future scientific priorities. Of the current research focus areas, respondents most often endorsed Gram-negative infections as the highest priority for the future (63%, 90 of 142). Approximately one-third of respondents indicated that antimicrobial stewardship (35%, 49 of 141) and diagnostics (33%, 47 of 142) should be the highest priority focus for the future. Only 14% of respondents (20 of 140) indicated that Gram-positive infections should be the highest priority for the future. All three of the special emphasis areas received similar ratings for future priority. About 40% of respondents endorsed pediatrics, special populations, and pharmacokinetics in the top two highest rating categories for the future.
Future Priority Areas: Figure displaying survey respondents’ priority ratings for areas of future scientific priorities. Of the current research focus areas, respondents most often endorsed Gram-negative infections as the highest priority for the future (63%, 90 of 142). Approximately one-third of respondents indicated that antimicrobial stewardship (35%, 49 of 141) and diagnostics (33%, 47 of 142) should be the highest priority focus for the future. Only 14% of respondents (20 of 140) indicated that Gram-positive infections should be the highest priority for the future. All three of the special emphasis areas received similar ratings for future priority. About 40% of respondents endorsed pediatrics, special populations, and pharmacokinetics in the top two highest rating categories for the future.
Recommendations
- Program Areas to Maintain
- Strong collaborations
- Focus on clinical design and problem-solving
- High levels of expertise
- Research funding
- Effective leadership
- Program Areas for Improvement
- Interest survey and application process
- Improved communications
- Adjustments to some study review and implementation processes
- Increase funding
- Enhance collaborations
- Improve mentoring
- Invite a broader membership
Limitations
- Given ARLG’s unique role, no comparison group options were available with accessible outcomes data
- The views of the survey and interview participants may not be generalizable and may overestimate ARLG’s value
- Publication quality was assessed via RCR, journal impact factor, and number of citations; however, these are imperfect metrics
- Data may be limited by the quality and completeness of the data sources
Contact Information
NIAIDEvaluation2@niaid.nih.gov
Acknowledgements
DMID Staff; Ripple Effect Communications, Inc.
This work has been funded in whole or in part with Federal funds from NIAID, under contract HHSN272201600016I.
Poster Presented at the 2018 American Evaluation Association (AEA) Annual Meeting:
- Walton, L., Taylor, B., & Solomon, L. (2018, October). Process and Outcome Evaluation of the Antibacterial Resistance Leadership Group at the National Institute of Allergy and Infectious Diseases. Presented at the Annual AEA Conference Meeting, Cleveland, OH.