Liberty A. Walton, Brandie K. Taylor, Larry S. Solomon, Stuart Z. Shapiro & Angela Malaspina
The National Institute of Allergy and Infectious Diseases (NIAID) of the National Institutes of Health uses several approaches to support scientific research. One method that NIAID recently deployed is the large-scale, high-resource, “big science” approach. This method allows the government to fund and manage the research of large, complex issues that require input from several disciplines. NIAID uses the approach to support the Centers for HIV/AIDS Vaccine Immunology and Immunogen Discovery (CHAVI-ID). Funding for CHAVI-ID ends in fiscal year 2019, at which point NIAID aims to support a third iteration of the CHAVI program. For this reason, NIAID supported a comprehensive program evaluation to assess CHAVI-related outcomes and identify areas for program improvement. This evaluation also aimed to understand the value of CHAVI’s “big science” approach within the larger field of HIV vaccine research. This poster illustrates the process and outcome evaluation methodology, key study findings, and strengths and limitations.
- Assess program outcomes
- Identify areas to improve the program implementation and design
- Understand the impact of this large-scale program within the greater HIV vaccine research field
Areas of Interest
- Areas for program improvement
- Scientific outcomes of the program to-date
- Impact of the program on the HIV vaccine field
- Focus of future program initiatives
- Role of large-scale program for advancement of program goals
- Development of infrastructure and knowledge base to help sustain or expand the vaccine field
- Collaborations of the program scientists with other researchers in the vaccine field
Program Logic Model
- Portfolio Analysis
- Included a bibliometric analysis, network analysis, analysis of other outcomes data (e.g., patents, subsequent NIH funding, achievements of junior personnel), and content analysis
- Data sources included Internal NIH Databases (e.g., IMPAC II), iCite, NIH ExPORTER & RePORTER, PubMed, SPIRES, USPTO Database, Web of Science, and CrossRef
- 8 NIH program staff engaged in CHAVI/CHAVI-ID grant oversight
- 8 Non-federal employees currently or previously affiliated with CHAVI/CHAVI-ID
- 3 HIV vaccine subject matter experts
- 54-minute interviews, on average
Grants were selected for the comparison groups based on: grant activity code, year of initial program funding, supporting NIAID Branch, the number of grants and subprojects, percentage of grants with Early Stage Investigator and New Investigator principal investigators, as well as the percent of funding for each category along the product development pathway. In addition, investigators were excluded from the comparison groups if they were associated with CHAVI/CHAVI-ID.
- R01 HIV Vaccine: A collection of HIV vaccine-related NIH R01 grant funding that are not collaborative, co-located, or co-funded.
- Comparison of traditional R01 funding versus big science funding
- Identify the impact of big science on outcome measures (e.g., productivity)
- P01 HIV Vaccine: Co-located, organized, or integrated programs of HIV vaccine-related research funded by NIH P01 grant funding (i.e., not funded by a big science mechanism).
- Takes co-location and natural collaboration outside of the big science approach into account
- Identify the unique effects attributable to the big science mechanism (e.g., collaboration)
- Malaria Vaccine: A collection of NIH grants (e.g., R, P, U) in a non-HIV vaccine-related field that has not received big science funding.
- Comparison of big science approach and traditional funding among two different vaccine research fields
- Identify the unique effects attributable to the big science approach on larger field of vaccine research
|Group||CHAVI/CHAVI-ID||R01 HIV Vaccine||P01 HIV Vaccine||Malaria Vaccine|
|# of Distinct Publications||817||300||149||150|
|Average Publication Cost||$539,373||$235,597||$417,974||$239,266|
|Publications per $10M||18.5||42.4||23.9||41.8|
|Impact Factor per Publication (Mean)||10.1||8.9||8.8||5.5|
|Impact Factor per Publication (Median)||5.2||4.6||4.9||3.5|
|Relative Citation Ratio (RCR - Mean)||3.20||2.07||2.23||1.61|
Top HIV Vaccine Research Articles
Success of Junior-level Investigators
|Metric||CHAVI/CHAVI-ID||R01 HIV Vaccine||P01 HIV Vaccine||Malaria Vaccine|
|Any grant-specific publications||269 (56%)||104 (56%)||57 (35%)||43 (50%)|
|Average time to first publication||3.4 years||2.7 years||2.8 years||3.3 years|
|Publications per person (among those with any publications) (median)||4||2||2||2|
|Junior investigators with Subsequent NIH Funding||31 (6.4%)||6 (3.2%)||10 (6.2%)||3 (3.5%)|
- Differences in funding levels between groups
- Metrics only accounted for NIAID funding
- Overlap between groups (i.e., collaborate on other grants)
- Bibliometric measures were used to assess publication quality, but they have limitations. For example, impact factor may not necessarily indicate the impact of individual articles within a journal.
- Interview findings may not be generalizable
- Interview participants were selected by NIAID and most were currently or previously affiliated with CHAVI/CHAVI-ID
- Continue to offer a flexible funding mechanism
- Maintain the current program structure; consider minor changes
- Engage with HIV vaccine researchers outside of CHAVI/CHAVI-ID
- Maintain balance of current program focus, but shift toward development activities and beyond broadly neutralizing antibodies
- Continue to share resources with the HIV vaccine research field
- Prepare for manufacturing challenges
- Sustain junior investigator involvement; focus on collaborative opportunities
DAIDS Staff; Ripple Effect Communications, Inc.
This work has been funded in whole or in part with Federal funds from NIAID, under contract HHSN272201600016I.
Poster presented at the 2018 American Evaluation Association Annual Meeting
- Walton, L., Taylor, B., Solomon, L., Shapiro, S., & Malaspina, A. (2018, October). Assessment of the Centers for HIV/AIDS Vaccine Immunology and Immunogen Design at the National Institute of Allergy and Infectious Diseases. Presented at the Annual AEA Conference Meeting, Cleveland, OH.