Artificial Part of the Artificial series

Attention Is the Bottleneck Now

  • artificial
  • ai-ethics

For as long as written words have existed, writing them required the writer to do the reading, not as a discipline, but as a mechanical fact. To contribute to a topic, you would likely have to educate yourself first, and whatever you finally wrote, you had to read at least once in the act of writing it. Reading and comprehension weren't virtues that good writers brought to the work; they were table stakes, the toll at the gate. They were the bottleneck, slowing down the creation of new ideas and products.

Now, the toll is optional, the traffic through the gate has increased exponentially, and new bottlenecks have emerged: attention and understanding.

The argument I want to make isn't "AI is drowning us in slop," though that is probably true, it's that the reading didn't disappear, it just moved downstream of the writing, when before it was upstream. I don't believe the amount of understanding required by the world has changed, but where it happens in the process has. And in a world so focused on measuring productivity (unreasonably, IMHO), we need to redefine that metric because every productivity claim I have come across measures the writing, which is now cheap, rather than the reading.

Two Requirements That Are Gone Now

Human-created writing quietly provides two assurances, and while they are similar in many ways, I think they break down differently.

The first assurance is that to contribute to a topic, a person first had to read up on it. Entry to the space cost time, and probably reading, to comprehend the subject well enough to add something worth adding, or at least something that would pass muster with others knowledgeable on the subject. Anything a reader encountered, therefore, came with an implicit guarantee that at least one person understood it well enough to make it.

That may still be true to an extent because LLMs are trained on content still (mostly) written by humans. I posit that this assurance holds for now, but may be weakening. (The implications of the debate about whether LLMs are only capable of derivative ideas, the nature of hallucinationsWhen a language model states something false or made up with total confidence. Not lying; the model has no intent.Read the glossary entry →, and the specter of model collapseModels degrading when they are trained on output from earlier models instead of human-made material.Read the glossary entry → may or may not change the calculus here, but those thoughts will have to wait for another post.)

The second assurance is that all written words had been read at least once because, at the very least, whoever wrote the words had to read them in the act of writing. This held for bad writing, lazy writing, and even plagiarized writing. Somebody's eyes saw every sentence at the very beginning of those words' history.

This second assurance has broken more dramatically. We have made it possible for novel prose to exist that no human has ever read -- not skimmed, not proofread, not glanced at before hitting send -- never read, by anyone. I don't think that has ever been true before.

There may be some asterisks on that point in situations where the second guarantee was a little leaky. Contracts can be assembled out of clause libraries, and mail merge has allowed people to templatize messages. While the final product may never have been read in a particular configuration, a person would have had to write (and therefore read) all of the individual components; therefore, I think the second assurance holds, at least in spirit.

The Cost Got Externalized

The assurances above were the product we got for the price of doing the required reading before writing something new. We all implicitly understand, and even rely on, the effort required to generate a new piece of content. That effort, and that cost, lay with the producer.

Not only are humans no longer the ones doing the bulk of the generating, in many cases, we are not even trying anymore. We often spend time waiting on the AI to answer questions, generate content, and write. Only once that work is done do we engage: we read it and understand it as best we can (or as much, or as little, as we feel we need to) before responding and sending the AI off to generate another round of content.

Now, the lion's share of the cost sits with the consumer. I can generate in seconds something that will cost you 40 minutes, and nothing in the transaction makes me feel the 40 minutes.

There may always have been some asymmetry in the exchange between production and comprehension, but never before has the exchange rate been so lopsided.

Productivity Is the Wrong Metric

If we try to measure value added to a process, economy, or other environment, we often discuss "productivity." (I am going to completely ignore the question of what "value" means for now, even though I think it is a fascinating and important question. For the purposes of this argument, I mean something like "economic value," which is probably measured in dollars or something similar.) Productivity is typically defined as some version of output over input, and it has been a pretty good proxy for value for about two centuries.

Productivity was a reasonable measure of value because output and input were always bundled together: more input tends to lead to more output, higher-quality input tends to lead to higher-quality output, etc. The ability to output something that was desirable required someone to put time, energy, and thought into inventing something, creating something, or designing a system that created more output with less input. If someone produced a report, they had understood the material. The artifact count and/or quality (i.e., the output) increased in line with comprehension and effort, a correlation that couldn't be avoided.

Now, that correlation has come apart, but the measure of "productivity" hasn't noticed. A report that nobody reads has produced nothing. Code that nobody understands isn't an asset, it's a liability. Counting artifacts now measures the half of the process that became effectively free, which means our "productivity" metric goes up substantially at precisely the moment the thing it was built to track is going down.

For one week, try a thought experiment and count both sides: count the words you caused to exist (drafts, emails, code, agent output, etc.), and then count the words you read closely enough to actually understand. When I tried it, it was enlightening, bizarre, and almost a little frightening to see how much the ratio has flipped. I read, let alone understand, only a fraction of what I cause to exist, and that fraction is probably shrinking. What I create used to be the product of a lot of consumption and processing, and while not all of that has gone away (my education and years of experience didn't suddenly cease to exist), the math has changed.

"A Wealth of Information Creates a Poverty of Attention"

In 1971, Herbert Simon wrote the sentence that gets quoted on every slide about information overload:

What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention and a need to allocate that attention efficiently among the overabundance of information sources that might consume it.

The quoting usually stops at "poverty of attention," which is a shame, because the back half is the operative part. Simon wasn’t lamenting. He was identifying a scarce resource and saying the job is now allocation.

He was writing about an abundance that hadn’t yet arrived. It has now, and it arrived from a direction he didn’t anticipate: not more people writing and creating more things, but the same number of people able to produce arbitrarily many. The prediction was right, but the mechanism is new and was likely unforeseen at the time. That is usually the point at which an old idea becomes worth rereading, and worth allocating a little attention to understand.

Where That Leaves Me

The bottleneck didn’t get removed. It got relocated, and the first rule of a bottleneck is that relocating it is all you ever do. Speed up the slow station and the line doesn’t become fast; it becomes fast up to the next slow station. For the entire history of knowledge work, the slow station was production. It isn’t anymore. It’s the finite, non-parallelizable capacity of a human being to understand something well enough to be responsible for it. I can buy more generation with a subscription. There is no subscription to increase my own reading speed.

One feature of artificial intelligence that continually strikes me is how well it does parallelize understanding. That leads me to one key question: is machine understanding good enough, or will we continue to define understanding as human understanding?

While I believe that question is important and worth consideration, I would argue that the growing ability of machines to reliably understand increasingly complex and nuanced scenarios frees us to allocate our attention where we really need to or where we choose to. That choice might be very difficult at times, but as a techno-optimist (I fully acknowledge that optimism might be a bit of a hot take in today's environment), I believe that is exactly what these emerging and improving technologies give us: flexibility and choice, which are such important pieces of a fulfilling human existence.

Certainly, there are situations when thorough human understanding and judgment are still critical, but when we choose to rely on AI, even when humans remain in the loopA person reviews or approves an AI system's output before it takes effect.Read the glossary entry →, nearly every version of that process seems to end with a human signing off on a summary of a summary. The assurance that used to come with decision-making, that somebody understood this, has been replaced by the claim that something understood this.

Maybe that’s fine. It isn’t obviously fine. Sitting with that is part of the work.


I will leave it as an exercise to the reader to decide how much I read before writing this article.