Levi, Ray & Shoup, Inc.

Evaluating Umpires: If You Can’t Measure It, You Can’t Improve It

8/26/2021 by Steve Cavolick

By Steve Cavolick

One of my favorite baseball stories comes from A.J. Pierzynski. He recalled a game where he was behind the plate and struggled with how home plate umpire Quinn Wolcott was calling balls and strikes. That led to the following exchange:

Pierzynski: Excuse me, can I have a new ball?
Wolcott: For what?
Pierzynski: One you can see.

Umpires are sensitive to comments about their vision, and A.J. was ejected from the game. But it does call to light how difficult an umpire’s job is and how we view their performance.

Umpiring is a hard job because you have to make split-second decisions, many times when the pitch in real-time looks as if it could be called either way. With television showing a virtual strike zone on the screen and replays of pitches, it is easy to scrutinize every called pitch as right or wrong. But did you know that you can see how correctly an umpire called a game you watched? The website www.umpscorecards.com (and Twitter feed @UmpScorecards) uses advanced analytics and Python to grade the performance of every home plate umpire in every game.

MLB shares pitch data from every game, and UmpScorecards uses a Python library to collect it. Each pitch is described with 89 attributes, but UmpScorecards uses only 5: the pitch’s horizontal (plate_x) and vertical position (plate_z) as it crosses the plate, the top (sx_top) and bottom (sz_bot) of the strike zone (adjusted for each batter’s height and unique batting stance), and whether it was called a ball or strike.  Based on the first four attributes and taking margin of error from the pitch tracking system into account, UmpScorecards determines the probability of a pitch being strike, then compares it to the actual call. For a more detailed description of the collision geometry, simulation, and statistical analysis involved, please continue reading here.

The output from the analysis are scorecards of the home plate umpire’s performance by game. Let’s look at two recent ones and see what they show.

The bottom of the scorecard indicates the accuracy of called balls and strikes inside the true strike zone. Todd Tichenor was extremely accurate on August 10th with 100% accuracy calling balls and 97% accuracy calling strikes. Greg Gibson was less accurate calling balls on August 15th, and much less accurate in calling strikes (only 80% correct). Another key measurement I consider is Overall Consistency. Each umpire sees the strike zone differently, and UmpScorecards groups all called strikes into an “established strike zone” (represented by the red polygon in the lower left). It tracks balls called outside of his own established strike zone to measure consistency. For these two games, Todd Tichenor was 100% consistent in his established zone, while Greg Gibson was only 92% consistent in his established zone. This tells me that when Tichenor is behind the plate, both hitters and pitchers will get a strike zone that is similar to the true strike zone. Greg Gibson looks to have an expanded strike zone, which would benefit pitchers, but the consistency of calls in his established zone is low, which makes for a frustrating night for both hitters and pitchers.

What’s true in business is true in baseball: if you can’t measure it, you can’t improve it. Don’t get caught with the bat on your shoulder and don’t lose the opportunity to use your data to discover new revenue streams or create new business models.

If you are ready to build advanced analytical and AI applications to measure and improve your business, our data science team can help. Not sure how to get started? Our strategic offerings can help you align business and technology teams, discover the right use case, and determine an ROI. Please fill out the form below to request a meeting.

About the author

Steve Cavolick is a Senior Solution Architect with LRS IT Solutions. With over 20 years of experience in enterprise business analytics and information management, Steve is 100% focused on helping customers find value in their data to drive better business outcomes. Using technologies from best-of-breed vendors, he has created solutions for the retail, telco, manufacturing, distribution, financial services, gaming, and insurance industries.