Knowledge Development Task Force I: Progress in Assessing the - PDF document

Knowledge Development Task Force I: Progress in Assessing the Literature George Smeaton 1 , Frederick Burrack 2 , David Dirlam 3 , Yuerong Sweetland 4 , and Teresa Flateby 5 The Knowledge Development Task Force (KDTF) was established by the AALHE Board at its 2017 Conference Meeting at the same meeting where it expanded the mission statement to include “Our association supports the generation of theory and information about effective assessment” and added a sixth strategic goal to “Contribute to the research and literature on assessing student learning in higher education.” The KDTF is contributing to these changes in two ways. First, there is its initiative to create, test, and use developmental rubrics for expertise in the assessment of learning in higher education. The second initiative is to conduct case studies of institutions who have demonstrated impacts on learning of their assessment activities. This session included presentations by five of the most active KDTF members. First, George Smeaton introduced the KDTF Charter, a useful idea which he introduced the group to in our first meetings. Next Fred Burrack described how we conducted our developmental interviews. KDTF Chair, David Dirlam then outlined the analysis that turned 107 developmental dimensions from the interviews into developmental rubrics. Next, Yuerong Sweetland described how we used the rubrics to rate articles and refine the definitions to improve reliability. Finally, KDTF Co-Chair, Terri Flateby described progress of the case-studies sub-group. KDTF Charter From the outset of this project, the potential breadth of its scope became a matter of concern for the project team. Studying the development of knowledge even in the past 40 years could lead to limitless potential directions for research. Coordinating the efforts of the project team required establishing a shared vision for the project’s goals, objectives, and timeline. In the field of project management, the document that delineates this vision is known as a project charter (Shlomo & Yotam, 2018). There are numerous project charter templates available on the internet, but most contain sections outlining the scope, goals, and deliverables. The charter developed for the Knowledge Development Task Force (KDTF) contained a section that provided detailed information on the project and a section that contained information relating to project planning. Project Information Content included in this portion of the charter established the shared vision for the project, delineated its scope, and specified the deliverables that the project proposes to achieve. 1 Keene State College, Keene NH, 2 Kansas State University, Manhatten, KS 3 www.changingwisdoms.com 4 Franklin University, Columbus, OH 5 Georgia Southern University, Statesville, GA

Organizational mission case. This section opened by making the case for the need for research on knowledge development as it relates to the field of assessment. Specifically, it noted the following two research questions. 1. Are current assessment practices really contributing to student learning in higher education? 2. What can be done to identify these and facilitate advancement in knowledge of how to use assessment to improve student learning. As a means of addressing these questions, this portion of the charter provided the following overall mission for the KDTF, “To identify and facilitate ways to advance the development of a body of knowledge devoted to assessing and improving student learning in higher education.” Proposed Solution. Although the research questions and mission specified in the case made for the project’s need greatly narrowed the scope of the study of knowledge development, numerous approaches for achieving the project mission are possible. As a means of providing additional clarity for the direction envisioned for the project, a Proposed Solution section provided a) the overall concept of knowledge development that serves as the project’s foundation, b) the project’s goals, and c) the project’s deliverables. Using Dirlam’s (2017b) framework, the foundational concept distinguishes the incremental knowledge development resulting from numerous small contributions from transformational knowledge development, which involves dramatic change from a single or a few contributions. Goals for the project are as follows: 1. Identify advances recorded in journals within library databases in the last four decades. 2. Conduct content analyses. Include disciplinary journals and other sources with information on the assessment of learning in higher education (ALHE). 3. Identify problems that could be solved in the next decade 4. Facilitate implementation of the selected solutions As a final component of the charter’s Proposed Solution sub-section, Deliverables to be completed by June of 2019 include: 1. A selected reference list of high centrality KD sources (see note 2 on this page) relevant to AALHE indexed by key strategies identified from them (see concept of knowledge development above). 2. An AALHE Database of Learning Identifiers (ADLI; see Dirlam, Wehlburg, and Perry, 2017). Learning identifiers describe for students and the public what is expected of learners and include statements of outcomes, competencies, goals, and objectives of programs at all levels of higher education..

3. A bibliography of articles that have been among the top hundred centrality values in any decade from 1970 on. Include centrality changes over time. 4. A set of developmental rubrics for rating randomly selected articles on the ALHE. 5. Analysis of rubrics ratings by committee members of the articles (see the section called “Concept of knowledge development” above). 6. A description of procedures for supporting “the generation of theory and information about effective assessment” that a successive standing committee could follow if the AALHE Board chooses to create a standing committee on knowledge development. 7. Three analyses of the gap between what is and needs to be known about the following questions: a. What could show impact on student learning? b. What assessment leadership does with the process in relation to student learning and faculty development? c. What institutional leadership does with the results, including reporting to the institutional board and public, as well as inclusion in strategic planning and budgeting? Additional Project Information. Other information pertaining to the project specified in the charter include its consistency with the strategic goals of the sponsoring organization, the Association for the Assessment of Learning in Higher Education (AALHE), alternative courses of action and the basis of their rejection, and known project limitations. The project advances AALHE strategic goal #2, “ Provide professional development for advanced assessment practitioners,” by identifying assessment research publications high in centrality indexed by key strategies identified from them. Alternative approaches identified include a) doing nothing, and b) providing only an ADLI. Doing nothing was rejected because inaction would extend current uncertainty regarding progress in assessment and could foster the use of weak and indirect measures of learning such as rates of retention and graduation. Providing only an ADLI without grounding it in the findings from research on learning would result in a largely speculative approach to the analysis of learning indicators. In the final component of this section, the charter identifies potential budgetary, schedule, and resource constraints to achievement of the project’s deliverables. The potential impact of each constraint was evaluated and found to be minimal. Information for Project Planning. In addition to clarifying the vision and scope of a project, a charter can serve as an important planning tool by noting key milestones and deadlines for their achievement. Content of this nature included in the Information for Project Planning section of the KDTF charter includes a communication plan that specifies the task force’s meeting schedule, a high-level schedule of the tasks required for achieving the project’s deliverables, and a set of specific project milestones that include target dates for completion.