🎲 Claude mixes up who said what, and that's not OK
Gareth Dwyer, in a post about LLMs not keeping track of who said what:
Yes, of course AI has risks and can behave unpredictably, but after using it for months you get a ‘feel’ for what kind of mistakes it makes, when to watch it more closely, when to give it more permissions or a longer leash.
Sure, you can get a “feel” for what kind of mistakes LLMs make, but that feeling means absolutely nothing and cannot be trusted. Randomness is literally built in. The temperature setting of an LLM determines how random the token selection process will be, and every token selection is a fork in the road for the rest of the output.
This kind of widespread misunderstanding about how LLMs work terrifies me. To be clear, I’m not picking on Gareth specifically—this is just a good example of the misplaced trust in LLMs I see running rampant these days. These systems are intentionally designed to be nondeterministic, with randomness as a key component of how they work. Thinking you can get a “feel” for when you can trust them is like getting a “feel” for how the dice are rolling or a “feel” for how the cards are being shuffled.