?

Log in

No account? Create an account
Something genuinely scary - Input Junkie
October 31st, 2016
09:10 am

[Link]

Previous Entry Share Next Entry
Something genuinely scary
Here's a very frightening thing that I've been meaning to do as a Halloween post. I realize that Halloween is for things that are fun scary, but it's not night yet, so this is isn't really Halloween.

You may have heard of Goodhart's Law. It has nothing to do with being good-hearted, Goodhart's just the name of its author.

Here's my current formulation: Any measurement which is used to guide policy will become corrupt.

You understand? Not might become corrupt. Not won't become corrupt if it's well designed. WILL BECOME CORRUPT.

Are you shaking? You are living in a culture which gives more and more trust to using measurement to guide policy.

Good long discussion of metrics (metrics are measurements used to guide policy)

The Wells Fargo accounts scandal is a classic example of measuring the worng thing-- top management set up impossible demands for new accounts, and staff both pressured customers into getting accounts they didn't need and also added accounts to customers who didn't ask for them. This has turned into a disaster for Wells Fargo.

This sort of thing doesn't just happen in business. If there's a demand for crime to go down, it might just get translated into make crime statistics go down, so the public is discouraged from reporting crime.

Does GDP measure somehting important? Do unemployment statistics?

I think you can get a rough measure of which places are better to live in than others by looking at immigration/emigration and adjusting it for the level of risk and cost people are willing to endure in order to move.... but if this were used for policy it would go wrong some way or other.

This entry was posted at http://nancylebov.dreamwidth.org/1085632.html. Comments are welcome here or there. comment count unavailable comments so far on that entry.

(4 comments | Leave a comment)

Comments
 
[User Picture]
From:kalimac
Date:October 31st, 2016 03:42 pm (UTC)
(Link)
This may happen to all metrics over a sufficiently long time, but the way to make it happen quickly and grotesquely is to set impossible standards and then enforce them by putting people's jobs on the line. This is supposed to make them try harder legitimately, but in fact it puts them in an impossible position where they either have to quit or cheat. And so you get bankers creating fake accounts, or teachers erasing and fixing their students' test sheets, etc.
[User Picture]
From:nancylebov
Date:October 31st, 2016 03:46 pm (UTC)
(Link)
That reminds me, I meant to post something about the SNAFU Principle, which is that information doesn't move in a hierarchy.

Corollary: the harsher the hierarchy is, the less the information moves.
[User Picture]
From:selenite
Date:October 31st, 2016 06:28 pm (UTC)
(Link)
This is my day job. My personal version of Goodhart's Law is "it's always easier to game the metric than improve what you're trying to measure."

I had to explain this when a senior manager retired. His replacement asked "If it only takes us [time] to process a [thingie], why do we have such a big backlog?" I went through and explained how the start to finish span time metric had been trimmed to exclude various stages that were "not our fault" such as preparation by other departments and approval by the customer. Which could lead to something that took over 200 days being reported as 34 days in the metric.
[User Picture]
From:dpolicar
Date:November 1st, 2016 08:08 pm (UTC)
(Link)
Yup, all of this is true.

And, just to state the obvious: failing to measure anything, while it doesn't invoke Goodhart's Law, is also an unreliable way of guiding policy.

Having a secret metric -- that is, a measurement I use to guide policy, where you don't know that I use it to guide policy -- can work pretty well if I can pull it off, but has other consequences, including opportunity costs, and isn't always possible anyway.

This is a difficult problem.
nancybuttons.com Powered by LiveJournal.com