FY2017 IUCRC Evaluation Project June 15, 2018 Lindsey McGowen, PI - PowerPoint PPT Presentation

Highlights of f Survey Data Industry/University Cooperative Research Centers FY2017 IUCRC Evaluation Project June 15, 2018 Lindsey McGowen, PI Olena Leonchuck & Angela Stoica North Carolina State University

Overview Industry/University Cooperative Research • Response rate Centers • Industry Findings • Faculty Findings • Student Findings • Questions & Discussion • Draft New Student Survey: For Evaluator Feedback IUCRC Evaluation Project at NCSU 2

FY2017 Response Rates Industry/University Cooperative Research Center Level Individual Level Centers Pulse Benefits Faculty Student Pulse Benefits Faculty Student Continuing Population from CD report 70 70 70 70 1214 1214 900 1432 1st Year Reporting Population from CD +2 +1 +1 +0 +12 +9 +12 +0 report Retired/Defunct Centers 9 9 9 9 127 127 88 241 Retired/Defunct Centers Reporting [1] +2 +2 +0 +1 +4 +6 +0 +5 Population [2] 65 64 62 62 1103 1102 824 1196 Centers That Did Not Return Data [3] 7 15 28 45 245 393 402 883 Available Population [4] 58 49 34 17 858 709 422 313 Data Received 58 49 34 17 457 317 269 108 Received / Population 87.69% 76.56% 54.84% 29.03% 41.43% 28.77% 32.64% 9.03% Received / Available Population 100% 100% 100% 100% 53.26% 44.71% 63.74% 34.50% [1] Retired/defunct Centers are not required to submit data, but some do. If so, those data were included in the analysis. This year, a previously graduated Center that was not included in the FY2017 population from the CD report also reported data. Their data are counted in the individual counts, but not the center level. [2] Population was defined as centers that were at least 1 year old. [3] Centers were excused for reasons such as being in the midst of center restructuring, high respondent turnover, and respondent refusal to complete surveys. [4] Numbers based on population minus excused and not returned counts. IUCRC Evaluation Project at NCSU 3

Industry Response Rate 100% Industry/University Cooperative Research Centers 80% 60% 40% 20% 0% 07 08 09 10 11 12 13 14 15 16 17 Center Level Received/Available Population Center Level Received/Population = Pulse Individual Level Received/Available Population Individual Level Received/Population = Benefits IUCRC Evaluation Project at NCSU 4

Industry/University Cooperative Research Centers Industry Pulse Survey Select Results IUCRC Evaluation Project at NCSU 5

Industry Satisfaction Industry/University Very Satisfied 5 Cooperative Research Centers 4 Quite Satisfied Somewhat 3 Satisfied Slightly 2 Satisfied Not Satisfied 1 07 08 09 10 11 12 13 14 15 16 17 Center Research* Center Administration Center Meetings * Previous years data reflect ratings of research quality 6/19/2018 IUCRC Evaluation Project at NCSU 6

Areas for Improvement Industry/University Cooperative Research 30% Centers 26.6% “Academic leads and IAB members need to work 25% more closely together to 19.8% 17.1% 18.2% 20% 17.4% recruit, especially at 16.7% 16.0% conferences. It may 15% 12.7% become easier when the 10.3% 9.7% 10% results get presented at 5.5% conferences .” 5% 0.4% 0% “The group can work together to improve project results reporting and archiving, access to results, and technology transfer” Research Plan & Selection Communication & Tech Transfer Operations IUCRC Evaluation Project at NCSU 7

Renewal Intentions Industry/University Definitely Yes 5 Cooperative Research Centers 4.28 4.21 4.22 4.17 4.13 4.14 4.09 4.07 4.05 4.01 4.00 Probably Yes 4 3 Uncertain Probably Not 2 1 Definitely Not 07 08 09 10 11 12 13 14 15 16 17 *Response Categories include: Definitely Not (1), Probably Not (2), Uncertain (3), Probably Yes (4), Definitely Yes (5) IUCRC Evaluation Project at NCSU 8

Predicting Renewal Intentions Industry/University Likelihood to Renew Cooperative Research Centers Satisfaction: Center Research .305** Satisfaction: Center Administration .231* Satisfaction: Center Meetings .175** Improve: Project Selection -.139** Note. *p<.05, **p<.01 6/19/2018 IUCRC Evaluation Project at NCSU 9

Comments for NSF Industry/University • “This is one of the greatest opportunities to Cooperative Research 100% Centers bring industry, academia and government 90% together and we have to work to keep it together .” 76.20% 80% • “I recently learned that NSF stopped providing 70% assistance to STTR/SBIR companies for membership fees. I think it's a great mistake. 60% The centers are excellent opportunities for 50% SBIR/STTR companies to outreach their technology to seek cost-effective technical 40% partnership for further R&D beyond original SOW as well as commercial partnership 30% opportunities with other member companies .“ 16.40% 20% • “The center is great but still needs to improve 10% 4.40% integrating industries' interests. Otherwise it 3.10% is hard to recruit and convince additional 0% members to join the center. “ No Comment Positive Area of Area of Comment Improvement for Improvement for NSF Center 6/19/2018 IUCRC Evaluation Project at NCSU 10

Industry Feedback: Pulse Survey Industry/University “Might be good to include metrics that you can track Cooperative Research 100% Centers for improvement of the center over time. This is useful for pulse surveys. One suggestion is a Net Promoter 90% 83.70% Score.” 80% “Consider using numerical ranking system. Also 70% include a space for 'wish list '.” 60% “Query relevance level of overall research to the 50% organization/industry in the survey .” 40% “The question, "Have you received benefits of 30% participation" should have drop-down choices and/or scale of benefits” 20% 13.30% 10% “Have a section asking the users for additional 3.00% questions that they would like to see included in the 0% survey. These can be reviewed and used to update the good suggestion none structure of the survey in line with users' views .” 6/19/2018 IUCRC Evaluation Project at NCSU 11

Industry/University Cooperative Research Centers Industry Benefits Inventory Select Results 6/19/2018 IUCRC Evaluation Project at NCSU 12

Networking Benefits “Our membership has been a critical part of Industry/University almost every business relationship we currently Cooperative Research 100% Centers enjoy. Absolutely irreplaceable in the growth 89.00% 90% and maturation of our company.” 80% 74.81% “We have gained improved access to University 70% faculty and staff for grant collaborations above and beyond the [Center] projects .” 60% 52.80% 50% “The interns provide fresh, state -of-the-art knowledge to our organization.” 40% 30.03% 30% “We have leveraged human resources and expertise from industry otherwise unavailable to 18.05% 20% us . ” 10% 5.91% “We have two collaborations with IAB members 0% (large companies) that have directly emerged New New Industry New University Hired Students Other Any of these Connections Partnerships Partnerships from prior center research and win-win opportunity assessment .” 6/19/2018 IUCRC Evaluation Project at NCSU 13

Students Hired Industry/University Cooperative Research Centers Average Students Hired per Member Firm 1 0.9 0.8 0.7 0.15/Member 0.6 0.54 0.95/Center 0.5 45 Program Wide 0.39 0.38 0.4 0.34 0.33 0.31 0.31 0.30 0.26 0.3 0.24 0.2 0.15 0.1 0 07 08 09 10 11 12 13 14 15 16 IUCRC Evaluation Project at NCSU 14

Research Relevance of the Average Member Industry/University Cooperative Research Centers Not Relevant Research: % of projects that are 16.23% probably not relevant to your organization's current or future needs 29.73% Adjacent Research: % of projects that are potentially relevant to your organization's current or future needs, but in an area that is outside your organization's current focus Core Research: % of projects so relevant to your organization's current or future needs that your 26.50% organization would almost certainly have conducted or contracted out a similar project within the next couple years Transformational Research: % of projects that are potentially relevant to your organization's current or future needs, but too risky/blue sky for internal investment 27.54% 6/19/2018 IUCRC Evaluation Project at NCSU 15

Research Cost Avoidance Industry/University • Definition: Research cost avoidance is savings a firm obtains by Cooperative Research Centers having “necessary” research projects performed by a center rather than performing them internally. • Example: If a firm reports that a particular “necessary” project would cost $100,000 to carry out internally (counterfactual estimate) but that project was actually carried out by a center to which they pay a $50,000 membership fee that firm has avoided $50,000 of R&D costs. • RCA = N of Proj. Avoid x Scien. Months x $/Scien. Months (Gray & Steenhuis, 2003) • N of Proj. Avoid = N of Center projects (CD report) X % Core projects (Benefits Inventory) • N Scientist months = 5 year median IUCRC Evaluation Project at NCSU 16

Mean Research Cost Avoidance (in thousands) Industry/University Cooperative Research $800.00 Centers $698.38 $700.00 $658.09 $631.15 $573.66 $600.00 $571.17 $487.55 $500.00 $400.00 $300.00 $200.00 $100.00 $0.00 12 13 14 15 16 17 IUCRC Evaluation Project at NCSU 17