Research

The National Conference of Bar Examiners’ (NCBE’s) Testing Task Force (TTF) is undertaking a comprehensive, future-focused study to ensure that the bar examination continues to test the knowledge, skills, and abilities required for competent entry-level legal practice in a changing legal profession. The collaborative study involves input from stakeholders at multiple phases and considers the content, format, timing, and delivery method for NCBE’s current tests, which make up all or part of the bar examination in most U.S. jurisdictions: the Multistate Bar Examination (MBE), the Multistate Essay Examination (MEE), and the Multistate Performance Test (MPT). The study also includes the Multistate Professional Responsibility Examination (MPRE), which is administered by NCBE and required for admission in most U.S. jurisdictions.

This Executive Summary provides a synthesis of the stakeholder listening sessions conducted during Phase 1 of the study between November 2018 and June 2019. The purpose of these sessions was to solicit input from various stakeholder groups about characteristics and considerations for the next generation of the bar examination. The sessions were facilitated by ACS Ventures LLC (ACS), an independent psychometric consulting firm. To read a summary of each of the ten listening sessions, download the full report.

Most of the listening sessions were conducted in-person in conjunction with scheduled meetings or conferences of stakeholder groups. In addition, virtual sessions using Zoom, an online meeting platform, were held for two stakeholder groups. Multiple sessions were offered in each case to maximize opportunities for stakeholders to provide input. Comments were also received by email.

The stakeholders represented included bar administrators, bar examiners, and justices; law school deans and doctrinal, clinical, and academic support faculty; and lawyer members of the American Bar Association (ABA). The meetings or conferences at which in-person sessions were conducted and the virtual sessions are listed in the following table.

It is estimated that a total of more than 400 stakeholders participated. An exact count of participants is not available because at some events the listening sessions were offered as part of the conference or meeting agenda and participants were not required to register or sign in to attend a session.

In this section we highlight some of the key points that emerged from participants’ input across all listening sessions. It is important to note that the purpose of the Phase 1 activities was not to draw conclusions about what the bar examination of the future should or will be. Rather, the sessions were intended to serve as a starting point by identifying possible options for redesign and to gather input on where greater emphasis might be placed. With this in mind, the key points set out in this Executive Summary should not be viewed as signaling any decisions regarding future changes, but simply as the main ideas voiced with some degree of frequency by stakeholders; the appended session summaries provide greater detail and additional comments that emerged from each. The TTF will continue to consult with stakeholders during Phase 3 to build upon the valuable input received during Phase 1.

Very few, if any, opinions were universally shared by stakeholders. In fact, some suggestions for change were potentially in conflict with one another (e.g., add simulations or mock client interviews and do not increase costs of exams). Additionally, while the intended focus of the sessions was on changes that could be made, in most sessions there were comments supporting various aspects of the current exam program/model, but with suggested opportunities for continued evolution and improvement.

• The MPT is a strength of the current bar exam.
  • The MPT was widely viewed as the component that is most representative of the skills needed for NLLs at the point of entry to practice.
• The MBE generally represents core content NLLs need to know but tests too deeply within the content.
  • The subject areas measured on the MBE were generally viewed as representative of subjects that would be applicable to all NLLs. However, the target level for items on the MBE was viewed by many as going beyond the point of entry-level competency by testing nuanced issues and “exceptions to exceptions to rules.”
• Lawyering skills should be emphasized over subject matter knowledge.
  • Content that focuses on skills such as issue spotting, critical thinking, legal analysis, written and oral communication, and reasoning was considered more applicable to all NLLs. In contrast, content that focuses on subject matter knowledge was viewed by some as requiring memorization of legal rules that lawyers can look up in practice.
• Consider adding legal research skills by providing access to an electronic database of legal resources.
• There are too many subjects covered on the bar examination when considering the MBE and the MEE in combination, and especially when considering the additional subjects tested in some jurisdictions that draft their own essay questions.

• Constructed-response items are preferred over multiple-choice items.
  • The constructed-response format of the MEE and the MPT was viewed as more representative of what NLLs do in practice (i.e., written analysis of legal and factual issues) than the multiple-choice format of the MBE.
  • The MPRE content could be assessed using essays or MPT-like questions as opposed to, or in addition to, the current multiple-choice format.
  • Multiple-choice items are not reflective of the way law is practiced.
  • While multiple-choice items were not widely supported, many stakeholders recognized the benefits that the MBE contributes: objective scoring, reliability of scores, and scaled scores that have consistent meaning over time and across jurisdictions because the exam is equated.
• Consider adding formats such as multimedia videos or mock clients/simulations to provide additional, and more realistic, assessment of important skills needed by NLLs.
  • While using simulations was suggested by stakeholders, the associated downsides of greater subjectivity in grading, the potential for bias, and increased costs were also noted.

• Consider a “step testing” approach for the bar examination.
  • The MBE could be the first step (test of legal knowledge) and could be completed while candidates are still in law school, when it would be more proximate to when they take core foundational courses.
  • The second step (test of legal skills), taken upon graduation, could be an enhanced MPT that is content-neutral and/or an essay exam that covers a limited range of content or for which legal resources are provided.
  • While the notion of step testing was frequently mentioned by stakeholders, the downsides of step testing were also discussed, such as taking time away from summer employment opportunities or semester coursework and potentially requiring candidates to incur associated costs of hotel, travel, and prep courses for each step. Further, some stakeholders commented that a step exam after the first year of law school would be too early in students’ training because many students would still be processing what they have learned.
• Consider more frequent administrations of the bar examination.
  • Additional administrations of the bar examination could permit candidates to sit for the exam when they are ready, permit failing candidates to retake the exam sooner, and reduce the time to employment after graduation, which would help graduates with student debt.
  • Reducing the time it takes to grade the constructed-response components (essays and performance tests) could also allow passing candidates to begin practicing sooner.
  • It was also acknowledged that more frequent administrations of the exam could require jurisdictions to need more staff and other resources, which could increase costs.

• Mixed feedback on how the exam should be administered/delivered.
  • There was varied support for paper-based testing, computer-based testing, or some combination of these delivery modes. However, it was mentioned that the delivery of the exam should align with law school, training, and practice environments.

Results from Phase 1 will be used to inform discussions that are part of Phase 3 on the design of the next generation of the bar examination and the test(s) associated with it. In Phase 2 of the study, a nationwide practice analysis is being conducted to collect information about the importance and frequency of tasks performed by NLLs, as well as the importance of related knowledge, technologies, skills, abilities, and other characteristics necessary for competent entry-level practice. The TTF is committed to continuing to seek input from stakeholders during Phase 3 as it develops recommendations for the next generation of the bar examination and MPRE.