There’s a scene in the film *Good Will Hunting* that I think about a lot in relation to LLMs where a therapist tells the brilliant but flawed young protagonist: I’m not going to listen to you because you don’t know anything. Your knowledge comes from books. Mine comes from decades of real experience. It’s a good scene in a good movie. And it’s strikingly relevant to LLMs’ level of knowledge in 2025 January 2026.

Here Be Dragons

I don’t say this to diminish LLM’s utility or the growth in their capabilities. In the last few years, reasoning, integrated web search, bigger and more powerful models, integrated code execution, etc. the models have so far surpassed the “next token prediction” mental model that they’re basically off the edge of the edge of the world. Off the end of the Mercator projection. Chilling with the dragons on the edge of the map.

In fact, LLMs have gotten so good that it’s actually quite hard to critique them anymore without coming across as a Luddite or out of touch, and I like to think that I am neither. Instead, I think I see that today’s LLMs are so good at some things that they can fool people into thinking they are good at everything, and there’s a systematic way in which LLMs are better at some things than others that we should be aware of.

All of the aforementioned advances smooth over an underlying problem; LLMs’ knowledge is real, their capacity to get more knowledge and summarize and integrate it is real, but this knowledge is systematically incomplete right now (as of January 2026). It frequently fails the vibe check in an important way. It can 1000000% pass the LinkedIn think-fluencer Turing Test (or is it the humans that are passing the reverse Turing there?). But can it actually advise you on how to act in the world to achieve your objectives in complex and specialized situations?

Cordyceps

If you took a person and made their brain run a simple program of 1) asking Opus 4.5 what to do, doing it, observing results, and asking Opus 4.5 what to do next, would that person succeed at attaining their goals? I would argue absolutely not, and that at least half of the problem is that the LLM’s knowledge comes from text [1]. In other words, LLMs have access to legible knowledge derived from text, not tacit knowledge derived from real experiences that aren’t really reducible to text or frequently recorded in texts that have made their way into model training.

Nikhil Krishnan and Adu Subramanian were talking about this tacit knowledge problem with LLMs over a year ago and only recently have I really begun to understand what he meant (I think). LLM heuristics come from summarized, distilled representations of memorized words and textual narratives. An AI model can memorize or just look up the year the Thirty Year’s War started and convincingly (to a non-expert) argue various positions related to that war’s influence on subsequent events. But it has more trouble with questions that pertain to people or specialized and novel situations and circumstances [0].

Hell is Other Non-AIs

The “dealing with people” part is particularly problematic, since many LLMs attempt to take sort of a globally-empathetic and deeply-concerned-with-human-emotion view on things by default, and this view almost always leads them astray. Ask yourself, when the last time Claude or ChatGPT made a “feel” statement that actually made sense and was something a person might actually feel?

Asking ChatGPT what factors led to the outbreak of WWI is going to lead to a much better response than asking it “What do I say in this email to maximize the chance that my client will renew the contract next year?” Claude can tell you what to say in the meeting but it can’t help you come up with a game plan to deal with people’s faces tightening and their smiles thinning as you speak, dead silence following your “any questions” and then everyone going back to their desks and working at cross purposes to your proposal.

If you ask Claude how to make your company successful or deal with a specific thorny problem, it’s drawing on sources of information that are well known and already out there. These facts are part of the conventional wisdom. They’re already priced in, and now, thanks to Claude, nearly everyone has access to them. Since everyone has access to it now, this legible information becomes even less valuable. It’s table stakes.

Hidden Knowledge

But a great deal of valuable information and knowledge is tacit. Secret. Occluded. Not legible. Unwritten. Bound up in NDAs and in the experience of people who have already tried to solve these problems and the lessons they learned that have never been committed to publicly-available text -- or even the private repositories of text drawn from the entire Internet, video transcripts, books, historical artifacts, public datasets, even synthetic text outputs from higher level LLMs or whatever other massive sources of tokens Google/OpenAI/Anthropic can delve into with their very deep pockets.

These are things that the LLM doesn’t know that it doesn’t know, or, if it does know it doesn’t know, almost always discounts the importance of. Of the entire possible universe of things to know - you as an individual will only ever occupy a pretty small space, and many of the things that you “know” are actually wrong, outdated, over-simplified, conditional, or nuanced in ways that actually mean that they don’t apply to the situation that you find yourself in the way you thought they did.

This hole where tacit knowledge should be is even less available to the ‘mind’ of LLM because of RLHF’s emphasis on producing a pleasing and convincing output. This process further obliterates and occludes the importance of missing knowledge from the LLM’s own cognition. Telling a user about its lack of knowledge in important areas or refusing to answer would increase the probability of the “thumbs down” that the entire RLHF process is meant to minimize the likelihood of.

🌭Thoughtdogs (Emulsified Thoughts)

Instead, you’ll get a thoroughly emulsified torrent of (mostly correct) facts and arguments and then a super-confident synthesis and recommended courses of action based on what the model knows or can find on the ever-shrinking and increasingly slop-ified public Internet. Sometimes it will genuflect towards uncertainty, but you won’t get a true sense of epistemic humility or access to hidden or tacit knowledge.

How does this actually show up in LLM-suggested actions? I’d argue it helps explain the following:

• Suggestions that sound plausible but won’t actually work due to <reasons>

• Hedged suggestions that don’t actually provide a lot of guidance since they rely on the user understanding the situation so well that they wouldn’t actually need guidance in the first place

• Recommendations that would only work under a set of common circumstances, but actual circumstances that are different from the one the user is actually in and that the LLM failed to elucidate (and that the user themselves probably doesn’t understand)

• Actions that annoy people around the user or damage others’ view of them and make them seem unreliable or out of touch (or reliant on using LLMs for everything)

• The exact same action that the median person would try in the situation which already likely has been tried and failed either by the user or someone else, which is why the problem persists

• A highly over-confident prescription that has a mixture of all of the above and that still has that lovely LLM flattery that we still struggle with

In other words, often when you ask an LLM what to do instead of “what is” you get a response that merit’s the Good Will Hunting response. “You’re just a [chatbot]. You don’t have the faintest idea what you’re talking about.... I can’t learn anything from you that I can’t read in some...book.”[2

Any awkward turns of phrase, fragments, typos and incorrect punctuation are the result of the human generation process and should be taken as charming proof-of-work that I actually wrote this instead of instructing an LLM to do so.

Thanks to Nikhil Krishnan, Adu Subramanian, and other Out-of-Pocket-ers who helped shape this piece by reading and pushing back on its central thesis.

[0] I am aware of the attempted “study” that violated a bunch of academic integrity codes where LLM apparently could argue convincingly against people on Reddit. However, I think this actually supports my point here rather than refuting it. Convincing forum arguers can be accomplished by rearranging text into new shapes. Actually influencing and acting in the world when people have significant skin in the game and there are groups of players involved not just 1:1s requires much more than that. https://www.science.org/content/article/unethical-ai-research-reddit-under-fire

[1] The other half is due to the input and output mechanism remaining the same as it is now - text inputted by one person’s fallible viewpoint. This context pinhole problem is probably even bigger than the tacit knowledge problem.

[2] Full (and uncensored) text: https://scalar.usc.edu/works/henry-v/transcript-good-will-hunting---your-move-chief

It's hard to get real feedback on your ideas. Friends and family are usually biased towards positivity and providing encouragement. Strangers don't have much of an incentive to a) respond at all to your ideas, and b) the rare ones that do are also incentivized in various ways, typically towards either selling you something or perhaps planting the seeds of a friendly acquaintanceship for future opportunities.

American culture specifically has a pretty widely known bias towards overt positivity as well, and it's generally seen as unusual behavior to give critiques of those that you don't know very well outside of certain prescribed settings such as a structured debate. Anonymous forums are one place where people can elude the typical norms against this behavior, and it's one reason why places like Hacker News have proved valuable to many in the startup/technology space.

With more and more people talking about their ideas with LLMs, and even developing parasocial relationships and attachments to them, there is a new frontier of uncritical validation and positivity: the Wormtongue problem of LLMs.

Leechcraft

For those not familiar, Grima or "Wormtongue" is a character in the fantasy series Lord of the Rings who engages in flattery and manipulation to undermine the rule of an important King. One of his prominent features is his obsequious and flattering commentary to the King.

If you've spent much time talking with the major frontier models, you will notice their constant penchant for flattery. There are the obvious flattering linguistic tics and phrases -- "That's a really interesting idea" etc. but these models also generally refrain from offering substantive critiques, and, when they do, they offer it in much reduced and watered-down form compared to their unequivocal and uncritical praise.

Unlike Wormtongue, LLMs probably are not secretly working for your evil enemies, but they do have one significant ulterior motive: ensuring that you continue to use and talk to them as much as possible.

To Flatter is Human

LLMs are trained on vast corpora of text from the Internet and are also trained by human feedback and tuned based on how people respond to the prompts. Humans like to get positive feedback, and generally don't like to get negative feedback. The user experience of having your ideas forthrightly and honestly critiqued by an LLM is probably not as widely appealing as being told your ideas are good, interesting, and original.

But as we rely more and more on LLMs for these tools, I think there's a risk of being pervasively told that our ideas are good, original, and interesting when they often...are not.

Interestingly, this habit towards flattery appears a) to be becoming less prevalent as models advance and b) is already less prevalent among reasoning models vs older models. Perhaps people are responding less positively to validation and praise from a non-human entity than to humans, or perhaps that the makers of gen AI products realize that the level of flattery previously prevalent was making people take their products less seriously for high-value tasks. Either way, it's a step in the right direction.

In the mean time, make sure to add something to your prompts to get past the gee-whiz flattering exclamations and take better advantage of what these tools have to offer.

PS: I am once again teaching EHR Data 101 through Out-Of-Pocket. Check out the comments for a link - our next cohort is starting in May 2025!

PPS: ChatGPT 4o told me this article was amazingly insightful, truly groundbreaking, and would change the realm of human-LLM interaction forever. So it might be really good! Claude 3.7 said "this piece makes a thoughtful and relevant observation about the tendency of AI language models to provide overly positive feedback."

The Expert understands the US healthcare system, the techno-legal-medical nexus that surrounds it, its vested interests and the reasons why things are the way they are.

The Outsider sees the problems and they propose their fix.

I’ve worked in healthcare IT and healthcare technology for 10 years, and I’ve never met a single person in healthcare who doesn’t acknowledge that there are serious problems in the US healthcare system.

Experts are probably more aware of the problems even than the Outsider, but their understanding is complicated by their understanding of the reasons behind the problems; and why solving them is not as simple as ‘doing X thing.’ The Expert is more likely to champion tweaks and feasible changes and less ambitious overhauls.

This post— **Why Are Cancer Guidelines Stuck in PDFs?—**which reached the top of HN this week shows the dichotomy clearly, as does its comment section. The engineer author embodies the Outsider by pointing out something: NCCN cancer guidelines are often available in non-computable PDF files. He created a very cool POC that uses a LLM and very savvy data modeling to make these guidelines interactive and chat-able. You should definitely read his post and check out his prototype! Many HN commenters, embodying the healthcare Expert, pointed out some reasons why NCCN guidelines as PDFs make sense in the current system, and some problems with the idea that a NCCN chat interface would make a huge difference in the current healthcare system. The author engaged very thoughtfully with these responses.

I’m not totally comfortable calling myself a true expert on healthcare data. The field is too vast and so variegated. There are exceptions and nuances to every rule. Maybe I am just chilling in the low point of the Dunning-Kruger curve.

But, after this time, I can feel the voice of the “Expert” bubble up inside me. I understand some baseline realities that aren’t as obvious to Outsiders, or that at least many such Outsiders overlook. For example, the average amount of time in the typical PCP visit ranging from 15-20 minutes, and there are a huge laundry list of quality-measure and reimbursement-driven items that need to be performed and documented that clinicians have to get through in each one.

This is why it’s not trivial to “just” ask PCPs to do [some additional thing you think may improve outcomes but has no clear path to reimbursement].” I understand intimately why it’s hard to use the data we have collected in EHRs for clinical research. Outsiders who are new to healthcare trumpet their brand new solution to Problem X, I can sometimes feel my jaw start to clench.

It’s easy to see the shortcomings of the Outsider’s approach. Sometimes, they seemingly don’t understand healthcare at all, either from a technology or the experience of being a patient or caregiver. Other times, Outsiders understand these perspectives extraordinarily well, but they may not have a strong grasp of reimbursement or administrative and regulatory requirements, or the historical and contextual reasons that EHRs and other pieces of healthcare technology are the way that they are.

Often, they ignore all of the “experts” telling them why their idea won’t work, or won’t be paid for, or won’t actually solve the problem because there are X, Y, Z other multi-factorial causes of the problem. It can feel as though some Outsiders haven’t considered that many people with good ideas, good intentions, and strong knowledge and capabilities have been working around and with these problems for decades.

If a problem was easy to solve, wouldn’t it have been solved by now? Is new technology so groundbreaking and novel that it can wipe away layers of technical, organizational, financial, legal/regulatory, and psychological context? Do these Outsiders think everyone working in healthcare simply doesn’t work as hard, simply lacks the acumen, or are too compromised by their proximity to the current system to fix easily solvable problems?

When channeling Expert energy, it’s easy to poke holes, to immediately reject, to bristle. I find myself doing it more and more, having to unclench and relax my fingers more and more often. But while it’s easy to see the drawbacks of the Outsider approach, I have to more and more force myself to step back and reconsider. If the Experts truly have all the knowledge that’s necessary to fix the system, then why isn’t it fixed? Why isn’t it even improving?

If we want change, we have to embrace the Outsider, because the “Expert” (or rather, a particular manifestation of expertise) view here is ultimately a trap. When all you see are webs of causation and all of the reasons why the current state exists and why alternatives are not feasible, you are immobilized. The “Expert” approach ultimately leads us back to the current system. We need challenging questions to be asked, premises to be exposed, and structures to be placed in competitive pressures or eliminated. Expertise in the current state won’t save us—at least not alone.

There are these two wolves inside each of us in healthcare. The jaded Expert wolf can see the tar pits and dead ends and understands more of the context and the reasons underlying what is. The Outsider wolf either doesn’t know about or has the courage to disregard these limitations, and has enough audacity to reject the current state and the reasons for its persistence. I think that a challenge for people as they grow in their understanding is to keep the Outsider wolf’s spirit alive, and to harness both expertise and audacity. It certainly has been for me, at least. As 2024 winds down and 2025 looms, I’ll be feeding the Outsider wolf as much as I can.

Hyperlinks are a poor substitute, and even more likely to distract or derail since you have to follow the link, open a new window or tab, and then return to the original text.

A gloss is a special type of marginalia, designed to define a word or add meaning to a phrase that might be unfamiliar. Glosses were written either between the lines of the main text or in the margin at the same level as the unfamiliar term, or with one leading into the other.

A gloss example, in a medieval manuscript, courtesy of the digital library at the University of Missouri [1]

The ability to keep your place in the text while getting a semantic rundown of an unfamiliar term serves a recognizable purpose, even today 800+ years after this text was produced.

Eventually, marginalia in general fell out of common use, but glosses became collected together into the glossary, where they remain today — if people actually seek them out to have unfamiliar terms explained to them. The authors of texts describing religious law in the medieval period made particular use of glosses and glossaries.

A gloss (Gk. glossa, Lat. glossa, tongue, speech) is an interpretation or explanation of isolated words… A glossary is therefore a collection of words about which observations and notes have been gathered….The canonists of Bologna in particular, favoured the method of the glossarists, and affixed to text and words the meaning which they should bear. [2]

From the glossary eventually came the idea of a general glossary — a list of the meanings of all possible terms — which eventually became the dictionary.

The problem the glossary attends to is a fundamental part of working with data: the lack of a shared semantic understanding between the users of and the generators of data.

A given token, code, string, value, word, or whatever you want to call it can have various meanings to different audiences and in different contexts. LLMs calculate and embed this contextual relationship across vast corpora of text via their embeddings, but most electronic systems exchange these tokens or codes in isolation.

In short, this means that some system outputs a O or a 2 and the end user often need to reason about, intuit, and decide what that code was supposed to mean. ****In healthcare, specific external systems of taxonomy and terminology help fill some of this gap, such as ICD-10-CM or NUCC Place of Service codes. Regardless, not everything is covered by a standard, and the specific implementations of standards differ over time and between actors.

Language itself is prone to this kind of glossal failure — how many times have you encountered misunderstandings between yourself and even those closest to you over the meaning of specific words or phrases — especially in async electronic communication via text?

Usage of terms without definitions leads to another phenomenon (at least metaphorically) — glossolalia — the production of sounds and words that appear to be a language but that are not. I mean this in the linguistic sense, not in the sense of the religious or cultural practice of ‘speaking in tongues.’

glossolalia, (from Greek glōssa, “tongue,” and lalia, “talking”), utterances approximating words and speech, usually produced during states of intense religious experience. The vocal organs of the speaker are affected; the tongue moves, in many cases without the conscious control of the speaker; and generally unintelligible speech pours forth [3].

Linguistically, glossolalia is the production of sounds that appear to have the structure of a known language, but that do not constitute a known language and their meaning is not understood by the speaker or the audience. I adapt that linguistic term to describe the phenomenon of people trying to communicate but instead only conjuring the appearance of communication.

I think most of us have had the experience of having a conversation or reading a piece of writing in a work context and coming away thoroughly confused by what was intended to be communicated. Gulfs in context between speakers or communicators — commercial vs technical, client-focused vs product-focused, different thought disciplines — contribute to this problem, which can cause serious misallocations of effort and significant grief and upheaval in organizations.

I think that slowing down and finding the agreed upon definitions of key terms — adding glosses — to our communication could help avoid this kind of miscommunication. Wider usage of standard taxonomies and ontologies would help. Someday, this may all be incorporated into the operation of every data system — the vaunted‘semantic layer’— but, until then, we’ll have to continue being comfortable with glossolalia at least some of the time.

[1] https://library.missouri.edu/specialcollections/exhibits/show/glossary/page4

[2] Boudinhon, Auguste. "Glosses, Glossaries, Glossarists." The Catholic Encyclopedia. Vol. 6. New York: Robert Appleton Company, 1909. http://www.newadvent.org/cathen/06588a.htm.

[3] https://www.britannica.com/topic/glossolalia

The language around generative AI also betrays this strangeness: perplexity, hallucination, context, alignment. Nothing that you need to worry about when firing up Python to wrangle some nice tabular data and output some graphs.

The capabilities of frontier LLMs are at times absolutely astonishing. Who would have thought 3 years ago that by now you could ask an app on your phone to look at a picture of a math problem on a whiteboard, transcribe it, and solve it using Python. Back then, I couldn't even get Siri to reliably play the next song [1].

The mobile app for OpenAI uses your phone's haptic feature to make it feel as if a ghostly invisible hand is typing on your keyboard very, very quickly when its models return output. And the model improvements and capability changes from release to release are equally if not more spooky than the typing of the invisible hand.

Classical computer programs are hardened, have clear if-then logical paths, and computers execute code exactly as written. They resemble trains that follow tracks that switches are manipulated to guide them onto. That's the point.

Programming is creating a mathematical object, where an input yields an output following prescribed, fixed steps. If you put in one wrong period or write FRON instead of FROM in single line of your program or query, it all goes out the window.

If this kind of work resembles soldering a solid motherboard or building a solid railroad (or Rube Goldberg machine). LLMs are inescapably fluid. They can adapt to a huge variety of inputs, and their outputs can take an incredible variety of shapes. You can make misspellings in your prompts. You can ask them to prompt themselves. You can ask them to answer in the form of a picture, or rewrite the code (or text) they just generated in a totally different language. Fluid doesn't mean infallible, but it does mean flexible.

This liquidity has major implications for systems that can incorporate the best of generative AI and traditional engineering approaches, that can harness both fluidity and reliability and explain-ability. Their fluidity, and how it relates to the lurking strangeness behind classical programming is what motivates the first edition of this very infrequent newsletter.

[1] Actually, I still can't.