If you were designing the bar exam from the ground up, how would you begin?
Would you try to determine what type of questions to ask, how many questions the exam should include, how hard the questions should be? These are all important topics to consider. But you wouldn’t be able to get far without answering more fundamental questions, including what exactly the bar exam should be testing, and why—in other words, what do you want to be able to do with bar exam scores?
The bar exam, like all licensure tests, is intended first and foremost to assist in protecting the public by helping ensure that those who are licensed to practice a particular profession—in this case, the legal profession—are competent to do so. For the legal profession, public protection also includes protecting the rule of law and promoting the values of the profession, given that lawyers are officers of the court and play an important role in the judicial branch of government. The legal licensure process, of which the bar exam is just one part, must determine whether applicants to the bar possess the knowledge, skills, and abilities that are necessary for minimal competence as a new lawyer.
But how do we know which knowledge, skills, and abilities new lawyers will need? This requires, first of all, that we reach out to stakeholders to gather their opinions and input. Additionally, any claim that the knowledge or skills being assessed by an exam are necessary for the safe or effective performance of a job should typically be supported through a job or practice analysis.* NCBE conducted a national job analysis in 2012, and a new practice analysis will be a significant component of the Testing Task Force’s study of the bar examination.
A practice analysis is a systematic process for collecting and analyzing information about work activities and requirements using tools such as focus groups and surveys. A typical practice analysis begins by identifying the activities required to perform the relevant job, as well as the environmental context in which those activities are performed. It then determines the knowledge, skills, and abilities necessary to perform those activities. Because a license to practice a profession authorizes the individual to work in a variety of settings and positions or practice areas, it must include an appropriately broad sample of respondents and job positions. Because jobs can and do change over time, it is appropriate to repeat a practice analysis periodically, with the period for doing so informed by the pace and extent of change taking place within the job or profession in question. The Task Force’s practice analysis will identify the current job activities of newly licensed lawyers, but it will also consider how those activities are changing or are reasonably expected to change in the next five to ten years.
The Task Force’s research consultants have already begun conducting stakeholder research and laying the groundwork for a practice analysis. As always, we encourage you to sign up to receive updates from the Task Force as this work progresses.
* AERA et al, Standards for Educational and Psychological Testing 182 (2014).