2003 to 2008

1991 to 2002

1981 to 1990

1972 to 1980

ANALYSIS OF JULY 2004 TEXAS BAR EXAM RESULTS BY GENDER AND RACIAL/ETHNIC GROUP

Stephen P. Klein, Ph.D. and Roger Bolus, Ph.D.
GANSK & Associates

(This analysis was updated in June 2006. Click here for details.)
December 15, 2004

Texas Government Code Sec. 82.0291 directed the Texas Board of Law Examiners to “compile a report indicating the number of applicants who fail the July 2004 bar examination. The data shall be aggregated by gender, ethnicity, and race. The report shall also include an analysis of the identifiable causes of failure and recommendations, if any, to address the causes of failure.”

The analyses described below were conducted to respond to this legislation. As background for what follows, we begin by summarizing the major features of the Texas bar exam and how the scores on it are computed and pass/fail decisions are made. We then describe the procedures that were used to gather and process the data for our analyses. Finally, we provide information about the size and nature of the differences in bar exam scores and passing rates among gender and racial/ethnic groups as well as the results of our analyses of certain factors that are and are not related to these differences. The specific questions we address in our analyses are as follows:

1. Do men and women have comparable bar exam passing rates and test scores? Findings
2. Do different racial/ethnic groups have comparable bar exam passing rates and scores? Findings
3. Do the differences in bar exam passing rates and scores among racial/ethnic group correspond to the differences in their admissions credentials and law school grades? Findings
4. Were some bar exam preparation activities associated with higher scores? Findings
5. As a group, do the students at some law schools generally score higher or lower on the bar exam than what would be expected on the basis of their mean LSAT scores? Findings

Texas Bar Exam Components

The Texas Bar Exam is a two and a half-day test. There is one day for the Texas essay section, one day for the Multistate Bar Exam, and a half-day for the combination of the Multistate Performance Test and the Texas Procedure and Evidence test. The major features of these four components are as follows:

Multistate Bar Exam (MBE). The MBE is a six-hour, 200-question, multiple-choice test. MBE questions (or “items”) are prepared and scored by American College Testing (ACT) under the general direction of the National Conference of Bar Examiners. The MBE has an approximately equal number of items in each of the following six subjects: Constitutional Law, Contracts, Criminal Law, Evidence, Property, and Torts. An applicant’s MBE “raw” score is the number of questions answered correctly.

Roughly 30 percent of the MBE questions that are asked on one administration (such as July 2004) have been used previously. ACT uses the data on these repeated items to adjust MBE raw scores for possible differences in average question difficulty across administrations. As a result of this calibration process (which is called “equating” or “scaling”), a given MBE “scale” score indicates about the same level of proficiency regardless of the administration on which it was earned.

Multistate Performance Test (MPT). Texas administers one 90-minute MPT question or “task”, consisting of a legal analysis and writing assignment. This task is developed under the direction of the National Conference of Bar Examiners. There is a new task for each administration. Texas readers grade the answers on a 1 to 6 scale in half-point intervals.

An MPT task assesses certain practice oriented legal research, analysis, and writing skills. A task consists of a File that looks like a typical lawyer file (e.g., letters, memos, reports, and the like) and includes relevant and irrelevant materials and a Library with all the case law, statutes, and secondary materials needed to deal with various matters in a hypothetical case. Candidates use the File and Library to complete a realistic task, such as drafting a memo to a senior lawyer, a letter to a client or opposing counsel, or a brief to be filed with a court.

Texas Essay Test. The six-hour essay portion of the Texas exam consists of 12 questions in such areas as Business Associations, Wills, Real Property, and Family Law. Members of the Texas Board of Law Examiners, with the assistance of professional editors, draft the questions. Board members and experienced attorney graders then score the answers to each question on a 1 to 25-point scale. The maximum possible essay raw score is 12 x 25 = 300 points.

Texas Procedure and Evidence (P&E) Test. The 90-minute P&E test contains 20 short- answer civil questions and 20 short-answer criminal questions. The Texas Board of Law Examiners is responsible for creating these questions and they and their associates grade the responses to each question on a 0 to 5 scale. Texas divides the total P&E raw score on each section by 2 (and rounds the result to a whole number) so that the maximum possible total P&E raw score across the two sections is 100 points.

Total Scores and Pass/Fail Decisions

Texas converts total essay raw scores to the same scale of measurement as that used for the MBE. This is done to adjust the essay scores for possible differences in average question difficulty across administrations. Scaling involves assigning the highest total essay raw score the same value as the highest MBE scale score in Texas, the second highest total essay raw score the same value as the second highest MBE scale score, and so on until the lowest total essay raw score is the assigned the same value as the lowest MBE scale score. The converted scores are called essay “scale” scores. This same procedure is used to convert MPT and P&E raw scores to scale scores.

Texas uses the following formula to compute each applicant’s total scale score so that the weights assigned to the MBE, Essay, MPT, and P&E tests are 40, 40, 10, and 10 percent, respectively:

Total scale = 2(MBE Scale) + 2(Essay Scale) + (MPT Scale)/2 + (P&E Scale)/2

Applicants with total scale scores of 675 or higher pass. All others fail. This pass/fail standard (which corresponds to a 135 on the MBE scale of measurement) is comparable to the standards used by most other states.

Analysis Sample Data

The application form for the July 2004 exam contained a section in which candidates indicated their gender and racial/ethnic group. Almost all of those taking the exam also completed a questionnaire that was distributed during a break in the test session, although everyone did not answer every question. A copy of this survey is attached to the end of this report.

The 2003 applicants whose data are used in this report graduated from the nine law schools in Texas that are accredited by the American Bar Association. These schools provided us with the Law School Admission Test (LSAT) scores, undergraduate grade point averages (UGPAs), and law school grade point averages (LGPAs) for their graduates who took the July 2004 Texas bar exam. None of the data furnished by the law schools has been disclosed to the Texas Board of Law Examiners. Such data are considered the property of the individual law schools and will not be disclosed at any time. The Texas Board of Law Examiners provided us with these applicants’ bar exam scores and repeater status data. All data reported by the Texas Board of Law Examiners has been disclosed to the law schools pursuant to Texas Government Code Sec. 82.029 and cannot be further disclosed in accordance with that statute.

All data were furnished to us without revealing the identity of any candidate and have been linked through a common study ID number for each candidate. The confidentiality of these data was preserved by employing procedures that prevented us from having access to applicant names and prevented the Texas Board from having access to the data provided by the law schools.

Preliminary Analyses

It is well recognized that grading standards vary across law schools. A 3.00 LGPA at one school may correspond to a substantially higher or lower level of proficiency than a 3.00 at another school. However, several analyses require combining LGPAs across schools. Thus, to adjust for possible differences in grading standards among schools, we converted the LGPAs within a school to a score distribution that had the same mean and standard deviation as the distribution of the LSAT scores at that school. This conversion is used throughout this report.

Applicants indicated their gender and racial/ethnic group on their application form. The analysis sample contained applicants from 19 racial/ethnic groups, but there were only a few candidates in some of these groups. This led us to form the following five clusters for our analyses:

Asian = Asian, Chinese, Japanese, Korean, Pacific Islander, Polynesian, and Vietnamese
Black = African American, African, and Black
Hispanic = Hispanic, Mexican, Cuban, Puerto Rican, Latin, and Central/South American
White = Caucasian and White
Other = All others (includes 5 Native Americans and racial/ethnic group omitted)

Research Questions and Answers

1) Do men and women have comparable bar exam passing rates and test scores?

Yes. Men and women had virtually identical passing rates (74.73 percent and 74.85 percent, respectively). There also were nearly identical numbers of men and women in our nine-school sample (1005 and 998, respectively). Table 1a shows that men had a slightly higher mean LSAT score than women whereas the reverse was true on UGPA. Men and women had very similar mean LGPAs.

Table 1a
Mean UGPAs, LSAT Scores, and LGPAs By Gender

Standard deviations for all takers for UGPA, LSAT, and LGPA were 0.47, 7.34, and 10.41, respectively.

On the average, men scored slightly higher than women on the MBE whereas the reverse was true on other sections (see Table 1b). This finding is consistent with the results regarding gender effects that are presented in technical reports regularly published by the California Committee of Bar Examiners. The differences in mean scores between gender groups balanced out so that overall, men and women had virtually identical mean total scale scores.

Table 1b
Mean Bar Exam Scale Scores By Gender and For All Takers

Standard deviations for the MBE, Essay, MPT, and P&E scores were all 13.0. The Total scale score standard deviation was 54.5.

2) Do different racial/ethnic groups have comparable bar exam passing rates and scores?

No. Using the racial/ethnic group designations noted above, Whites and Asians had statistically significantly higher bar exam passing rates and mean bar exam test scores than their classmates. This held for first time takers and repeaters.

Table 2a shows the number of applicants by racial/ethnic group and repeater status. Table 2b shows their passing rates. These data indicate that 75 percent of the 2003 applicants in the analysis sample passed. The passing rate for first timers (81 percent) was nearly double the rate for repeaters (42 percent). Together, Blacks and Hispanics comprised 18 percent of the first timers, but 36 percent of the repeaters.

Table 2a
Number of Takers By Racial/Ethnic Group and Repeater Status

Table 2b
Percent Passing By Racial/Ethnic Group And Repeater Status

Table 3a (which uses the data on all 2003 of the candidates in the analysis sample) shows each group’s mean UGPA, LSAT score, and LGPA.

Table 3b shows their mean bar exam scores. These data indicate that a group’s mean scale score was very similar across the four sections of the exam. The sole exception was that Asian applicants did especially well on the MPT, but this could easily be due to chance given the comparatively low score reliability of a single MPT task. Hispanics and Blacks did about as well on the MBE as they did on the written portions of the exam. Thus, overall, exam format had no effect on the differences in mean total scores between groups.

                              Table 3a
Mean UGPA, LSAT, and LGPA By Racial/Ethnic Group

                                Table 3b
Mean Texas Bar Examination Scores By Racial/Ethnic Group

3) Do the differences in bar exam passing rates and scores among racial/ethnic group correspond to the differences in their admissions credentials and law school grades?

Yes. We found that on the average, the applicants in different racial/ethnic groups performed as well on the bar exam as would be expected on the basis of their law school admission credentials and law school grades.

We examined this matter in two ways. First, we noted that the 8-point difference in mean LGPA between Whites and Blacks was equivalent to 0.78 standard deviation units. This was nearly identical to the difference (in standard deviation units) between these groups’ mean total scale scores. The size of the difference between Whites and Hispanics on LGPA also was very similar to the difference (in standard deviation units) between these groups in total scale scores. Asians were the only group that did not do quite as well on the bar exam as would be predicted on the basis of their LGPAs.

Our second (and more statistically sophisticated and precise) approach involved constructing two “multiple regression” equations to predict an applicant’s total bar exam scale score. One of these equations included the applicant’s UGPA, LSAT score, and LGPA. The other equation contained these same variables plus the applicant’s gender and racial/ethnic group. This analysis found that racial/ethnic group had almost no relationship with bar exam scores once there is control on the applicant’s admissions credentials and law school grades.

Specifically, the first equation explained 37.2 percent of the variance in total bar exam scores whereas the second explained 37.8 percent, i.e., just 0.6 percent more. Thus, the addition of gender and racial/ethnic group to the equation did not have any practical effect on predictive accuracy. All the groups (including Asians) performed as expected. In short, minority and non-minority bar exam scores were very consistent with what would be expected given the differences in their admissions credentials and law school grades. The exam did not increase or decrease the differences between groups that were present before they sat down to take the test.

Our analyses within and across schools also indicated that there is a great deal of variance in bar exam scores that is not explained by UGPA, LSAT, LGPA, gender, and racial/ethnic group. A substantial portion of the differences in bar exam scores among candidates must therefore be due to other factors, such as how the applicants prepared for the exam.

4) Were some bar exam preparation activities associated with higher scores?

Yes. Almost all the applicants in our analysis sample reported having participated in one or more commercial bar review courses in the six months prior to taking the exam. To investigate whether some of these activities were more helpful than others, we constructed a regression equation that contained the applicant’s LSAT score, LGPA, and their response to each of the questions in the student survey (see attached copy of this questionnaire).

This analysis found that applicants tended to receive 4 to 10 more total scale score points if they did one or more of the following during the six months prior to taking the exam: attend lecture and discussion sessions, use Internet lessons, and use hard copy study guides and books. The percentage of candidates using these methods were: 85, 28, and 95, respectively (many applicants used more than one strategy).

We were surprised that the use of hard copy study materials had a statistically significant effect because almost all the candidates used them. It is evident that those who did not use them were not well served. The effect of using Internet lessons was not as strong as the effects of using the other two methods.

About 21 percent of the Black and Hispanic applicants (and 15 percent of all of the other applicants) reported that they worked 20 or more hours during the five weeks prior to taking the July 2004 exam (excluding any paid leave time they may have received from their employer to study for the exam). On the average, the applicants who worked earned about 15 total scale score points less than their classmates with comparable LSAT scores and LGPAs. To put this statistically significant 15-point difference in total scale scores in perspective, it is comparable to the unique effect (i.e., after controlling on LSAT and LGPA) of being a repeater rather than a first time taker.

5) As a group, do the students at some law schools generally score higher or lower on the bar exam than what would be expected on the basis of their mean LSAT scores?

Generally No. As noted on the Texas Board of Law Examiners’ Website (www.ble.state.tx.us), there are large differences in bar exam passing rates among schools. We found that almost all of these differences can be explained by differences in the admissions scores of the students they graduate. For example, there is a nearly perfect relationship between a law school’s mean total bar exam scale score and its mean LSAT score (the correlation is .98 out of a possible 1.00). Many of a law school’s graduates do better or worse on the bar exam than what would be expected on the basis of their own LSAT scores, but these differences almost entirely balance out when the data are analyzed by school. Nevertheless, one school’s mean total bar exam scale score was 10 points higher than what would be expected on the basis of its mean LSAT score (the odds of a difference of this size occurring by chance are less than 5 in 100).

Conclusions and Recommendations

Men score slightly higher than women on the MBE while the reverse is true on the rest of the exam so that overall, they have nearly identical total scores and passing rates on the Texas bar examination. Men and women candidates in Texas also have comparable admissions credentials.

Black and Hispanic candidates are not spread evenly across the nine Texas law schools. They are much more likely to attend some schools than others. There also are large differences in passing rates among schools. However, the large differences in passing rates among racial/ethnic groups are not related to which law schools they attend because almost all the schools do about as well on the bar exam as would be expected on the basis of the mean LSAT scores of their graduates. That is what is driving the differences in bar scores among groups.

The differences in scores among racial/ethnic groups were quite similar across the different sections of the exam. With the possible exception of Asians who did especially well on the MPT, no section stood out as being unusually easy or difficult for a particular racial/ethnic group. In addition, total bar exam scores essentially mirror the differences in these groups’ admissions credentials and law school grades. Thus, the bar exam does not appear to widen or narrow the gap in scores that was present between the groups before they sat for the exam.

We also found that a significant portion of the differences in bar exam scores between applicants is not attributable to differences in their admissions credentials, law school grades, gender, or racial/ethnic group. A small but statistically significant piece of this remaining variance is related to whether the candidate worked for more than 20 hours during the five weeks leading up to the exam. And, Black and Hispanic applicants were about 1.5 times more likely to be among those who worked during this period than were other applicants. A few other preparation factors also were related to scores, such as participation in lecture and discussion sessions presented by a commercial bar review course.

Given the findings above, we see no reason to make any changes in the nature of the exam itself. It appears to be well balanced and fair to all takers. Moreover, the results on it correspond closely to the law schools’ own evaluations of their graduates’ abilities (as reflected by the generally high correlations between law school grades and bar exam scores at each school). Nevertheless, the findings about preparation factors suggest that something might be done in this area to improve minority bar passage rates. This might involve providing funding (and perhaps scholarships to bar review courses) to students who did well in law school but may not have all the financial resources they need to prepare for the exam in the same way as their classmates.

July 2004 Texas Bar Examination Examinee Survey

Please put a checkmark (√) in the Yes or No box in response to each question below:

Dr. Stephen Klein is the Senior Partner in the consulting firm of GANSK and Associates. In that capacity, he has done research and consulted on a wide range of issues for the National Conference of Bar Examiners, more than two dozen state boards of bar examiners, over a dozen law schools, and the Association of American Law Schools. He also has consulted for certification boards in accounting, acupuncture, actuarial science, dentistry, medicine, podiatry, psychology, and teaching. He has testified as an expert in state and federal courts and at legislative and administrative hearings regarding criminal justice, testing, educational, personnel, voting rights, and other matters. He served as the federal court's appointed technical advisor on a large class-action suit involving measurement issues and consulted for the National Academy of Sciences, the National Science Foundation, the Knight Commission on Intercollegiate Athletics, the Little Hoover Commission, and many other public and private agencies and organizations. Dr. Klein also is a Senior Research Scientist at the RAND Corporation in Santa Monica, California where he has led studies on educational, health, military manpower, and criminal justice issues.

Dr. Klein received his BS from Tufts University and his M.S. and Ph.D. in Industrial Psychology from Purdue University. Before coming to RAND in 1975, he was a Research Psychologist with the Educational Testing Service in Princeton and Associate Professor in Residence at UCLA where he chaired the Research Methods division in the Graduate School of Education. Dr. Klein has over 250 publications, he is on the editorial board of the Review of Educational Research, and he is a member of the American Statistical Association, the American Psychological Association, the American Educational Research Association, and the National Council on Measurement in Education.

Dr. Roger Bolus serves as Senior Partner, Research Solutions Group, a company providing technical and analytical services to research endeavors in the areas of education, healthcare and large scale testing. Roger also has an appointment as Director, Psychometrics at the Center For Neurovisceral Sciences in the Department of Medicine at UCLA. For the past 25 years, he, in collaboration with Dr. Stephen Klein, has provided statistical, psychometric and data management consultation to state bar examination boards throughout the country. Among current clients are the state bars of California, Illinois, Maryland, Ohio, Massachusetts, Texas, Delaware and Nevada. In this work, Roger has developed an expertise in the design, management and analysis of large, complex databases related to legal testing. Dr. Bolus’ current interests are in the areas of adaptability of the Internet to the administration and scoring of open-ended responses in high stakes testing. Dr. Bolus received his M.A. and Ph.D. from the University of California, Los Angeles School of Education with a specialization in Educational Testing, Measurement and Evaluation (1981). He is the author or co-author of over 30 articles and technical reports, and has spoken at several national conferences on the topics of testing and outcome assessment.