The short version is that some very smart people didn't have quite enough sense and/or enough virtue.
I used to have a theory that working excessively long hours (90 to 100 hour weeks) at the top of the financial system was a major contributor to crisis, because if you're running that low on sleep, it's hard to think about whether what you're doing makes sense. Besides, I thought it was an amusing irony that if the analysts had been protected by a union or regulations, things would have gone better.
I abandon that theory-- those guys were so convinced they knew what they were doing and so focused on money that they wouldn't have done any better if they'd been getting eight hours of sleep every night. One or two of them might have been hit by inspiration, but it wouldn't have been enough to do any good.
A lot of this starts with Edward Thorp (Beat the Dealer, Beat the Market). He invented card counting (finding a tiny flaw in the rules of blackjack which used to make it possible for someone with a good memory to come out ahead).
The thing about Thorp is that he knew he was operating in a real world-- he knew things could go wrong and he used an equation to be sure of how much to bet safely. (Sorry I don't have time to look up the equation, I'm typing in haste-- the book has to go back to the library and I should be doing ICON prep.) He kept using the equation and thinking about risks when he applied computer analysis to the market.
There were others (notably Taleb, author The Black Swan and Soros) who thought about what they were doing.
However, there were a bunch of guys who thought you could understand the market by just studying the market. They forgot there was a real world which was bigger than their minds. In particular, a common strategy was to find something they thought would go up, and bet big on it. Then they'd figure out something they thought would go down if that went up, and they'd short sell the second thing. Short selling has unlimited possibilities for loss.
This kind of thing isn't nonsense-- some of those funds were bringing in returns of 40% or nearly for multiple years.
I learned about short squeezes from the book. For some reason, my mind doesn't quite wrap itself around short selling. Fortunately, it's possible to treat short selling as a black box. You make money if the price goes down. You lose money if the price goes up.
A short squeeze is what happens when a lot of people sell something short, and then see the price going up. So they all want to buy it now because they want to not lose more money as it goes higher. The price goes up very fast.
Anyway, it was all chugging along nicely and then things happened and both the bets were wrong. This is a way to lose a lot of money very fast.
None of these guys had experience in the market, and I suspect this contributed to their not knowing that things occasionally get very weird.
bradhicks has written about the process of someone finding a new good investment, and then other people trying to piggyback on the insight without understanding it, and buying things which look kind of the good investment, but which are really less and less like it. And, of course, there are plenty of people who'll sell to the unthoughtful buyers.
If this book is correct, there was a lot more wishful thinking than conscious fraud.
I have a notion that part of what went wrong was a shift in the felt definition of "risk". Instead of seeing risk as a region which included some unknown unknowns, quants came to see risk as a perfectly well-defined chance to make money.
I'm planning to read Michael Lewis' The Big Short-- it's an account of three people who figured out by different routes that the financial system was going sour, resisted the social pressure (one of them is autistic) and made a lot of money thereby. All of them placed their bets and then tried to warn people. No one listened.
 I'm adding a concept to my mental toolbox. Knowing about a thing isn't enough-- it's worth asking what happens if you have a whole lot of that thing.
Addendum: In the very popular Rich Dad, Poor Dad (published in 2000), the underlying premise was that doing something useful for your living was for suckers. Nothing wrong with doing something useful as a hobby, but you should be making your living in some way that you shuffle the risk off on to someone else. There were government guarantees which made it possible to do this in real estate.
This looked sociopathic to me at the at the time, and no one else seemed to see it that way.
Addendum 2: I just found something I wanted to include Paul Wilmott could see that the quants didn't understand enough, and they were going to blow up the financial system. In 2003, he started a training program so that there would be people who understood what the equations meant, but it was too late.