Think back to your first day at your current job. What did you need to know to be able to hit the ground running?
What kinds of things did you do regularly, and what were the most important things you did—even if you did them infrequently? What skills or abilities did you need to accomplish your work?
A practice analysis systematically assesses these kinds of questions by asking employees across a certain profession about the types of tasks they perform and the knowledge, skills, and abilities (KSAs) required to perform them. The results of a practice analysis survey provide an accurate representation of what employees in a particular sector do. In a previous blog post, we discussed the importance of conducting a practice analysis to help determine the job activities of newly licensed lawyers. In this post, we’ll take a closer look at the future-focused practice analysis that the Testing Task Force is undertaking.
It is important, first of all, to distinguish between a practice or job analysis of the type that the Task Force is conducting and other types of job-related research. Some individuals and organizations have conducted important studies for the legal profession seeking to determine what competencies, qualities, etc., are necessary to be successful as a new lawyer. In contrast to this, the Testing Task Force’s practice analysis—like the job analysis conducted by NCBE in 2012—is focused on determining the tasks and activities performed by newly licensed lawyers and the KSAs required for competence to practice law. It is minimum competence, not (potential) success, that the bar examination tests, with the purpose of admitting only those who demonstrate the necessary competencies to be licensed lawyers.
The Task Force’s practice analysis seeks to assess the current and future state of the profession by determining in detail which KSAs new practitioners need most frequently and which are most critical to their work, even if infrequently used or performed. It will attempt to highlight not only the current requirements of the job, as the 2012 job analysis did, but also future ones, by exploring emerging trends and technologies. The practice analysis will also seek to determine which other personal characteristics, such as leadership, are important for new lawyers to possess.
To accomplish all this, a significant amount of preparation is required. American Institutes for Research (AIR), the consultant conducting the future-focused practice analysis for the Task Force, facilitated focus groups in February and March 2019 with newly licensed lawyers and lawyers who supervise newly licensed lawyers to review and revise draft lists of tasks, KSAs, other personal characteristics, and technology used to perform tasks. (These draft lists were developed by AIR during its environmental scan research, which began in October 2018.) The results of these focus group sessions will help AIR, in turn, develop a future-focused practice analysis survey that will be distributed nationwide beginning in June 2019. It will gather quantitative data about the work done by newly licensed lawyers today and in the near future. The Task Force is currently reaching out to state bar associations across all jurisdictions to request assistance in distributing the survey to their members.
And all this is just one part of the process. Following the conclusion of the practice analysis survey, AIR will perform a linkage exercise with subject matter experts from around the United States to connect key KSAs to the tasks that new lawyers perform and then rate the importance of each KSA to performing those tasks. This step will inform the decisions about what should be tested—the KSAs that are most important for performing the most frequent tasks and critical tasks (regardless of frequency) generally should be given more consideration for inclusion in test content. The results of this exercise will guide the Testing Task Force as it tackles the bar examination program and test components redesign in 2020.