科技圈对新鲜风口的需求甚于时尚界，加密货币、NTF、WEB3……，现在新加入“下一件大事”俱乐部的是人工智能。几乎人人都在谈论 AI，线的另一头跟你聊天的狗也在思考。各类观点层出不穷，向往趋之的，警惕暂避的都有。我对它于我注意力的不断侵入已经不胜其烦，但趋势看来目前它不是一盆昙花。
在此摘抄一些我觉得有意思（但不一定正确）的关于人工智能的讨论，我对这项技术的看法目前处于保留状态，汇集不同专业领域的观察也许可以帮助用更正确的视角来看待它。
Why the Godfather of A.I. Fears What He’s Built:
“…There are two approaches to A.I. There’s denial, and there’s stoicism. Everybody’s first reaction to A.I. is ‘We’ve got to stop this.’ Just like everybody’s first reaction to cancer is ‘How are we going to cut it out?’ ” But it was important to recognize when cutting it out was just a fantasy.
He sighed. “We can’t be in denial,” he said. “We have to be real. We need to think, How do we make it not as awful for humanity as it might be?”
“For years, symbolic-A.I. people said our true nature is, we’re reasoning machines,” he told me. “I think that’s just nonsense. Our true nature is, we’re analogy machines, with a little bit of reasoning built on top, to notice when the analogies are giving us the wrong answers, and correct them.”
Using AI as a writing dialogue partner, ChatGPT can become a catalyst for clarifying what we want to say. Even if it is wrong.6 Sometimes we need to hear what’s wrong to understand what’s right.
AI often makes a lot of factual and logical mistakes. Mistakes, if identified, can help you think. Seeing in clear text what is wrong or, at least, what we don’t mean can help us set our minds straight about what we really mean.
AI can and will ruin your voice and credibility if you lazily let it write in your place. As writers we can not allow AI to replace our own thinking. We should use it to simulate the thinking of a missing dialogue partner. To write better, we need to think more, not less.
Artificial text is a statistical mashup of human quotes. When we quote AI, we quote quotes. We quote a Bircher muesli of quotes, write over it, and then feed it back into the AI system. There our input gets rehashed again. The way it currently works, AI is more likely to reach lukewarm entropy than ice-cold super-intelligence.
We considered AI for iA Writer. But plugging AI into iA Writer was not just adding a feature it risked becoming a feature of AI.
A future without applications is neither likely nor beneficial to anyone. In spite of that, it is happening right before our eyes. One app after another is giving up to become a ChatGPT function. Structurally, AI challenges the notion of independent apps.
Taken together, this painted an incredibly bleak future for a company focused on writing software. Some technology optimists might think we’re not seeing clearly. And indeed, from our vantage point, we can’t see many reasons to be unconditionally optimistic about the impact of technology that essentially replaces thinking. But we try.
Many new products became AI apps, often overpromising and following a simple recipe:
- Integrate ChatGPT - Overpromise - Rebrand GPT as proprietary AI
This trend led to a monotonous market of AI-this-and-AI-that.
Most importantly, we witnessed a change from OpenAI to CommercialAI. It’s not about robot Gods, humanity and openness. It’s about making money.
With ChatGPT’s latest update, a number of short-sighted startups are about to go belly up. When you pay a company a percentage of your revenue, you provide them with an ongoing financial advantage. Additionally, by giving them access to your user data, you risk handing them the means to make your business obsolete.
Floridi points out that AI doesn’t mimic the way we think. It doesn’t think at all. AI does not recreate human intelligence. It is replacing it! That doesn’t sound like good news. But if you want to solve a problem, you need to see it clearly first.
一天世界:
真正關心藝術的人看了圖二會更期待 AGI 的到來，因爲那種平淡無味的想像很需要「比人類更聰明」的某種東西來超越。但這種期待是徒勞的。全新的有力藝術想像並非天才能獨力完成，而是她們和觀者、聽衆、讀者一起完成的。這是目前的人工智能論述令人失望之處：作爲觀者、聽衆、讀者的一方已經完全準備好了卸下自己這方的責任。
In 2020, researchers demonstrated a way for discriminatory algorithms to evade audits meant to detect their biases; they gave the algorithms the ability to detect when they were being tested and provide nondiscriminatory responses. An “evolving” or self-programming A.I. might invent a similar method and hide its weak points or its capabilities from auditors or even its creators, evading detection.
Clune is also what some researchers call an “A.I. doomer.” He doubts that we’ll recognize the approach of superhuman A.I. before it’s too late. “We’ll probably frog-boil ourselves into a situation where we get used to big advance, big advance, big advance, big advance,” he said. “And think of each one of those as, That didn’t cause a problem, that didn’t cause a problem, that didn’t cause a problem. And then you turn a corner, and something happens that’s now a much bigger step than you realize.”
Robin Hanson, an economist at George Mason University who has written a science-fiction-like book about uploaded consciousness and has worked as an A.I. researcher, told me that we worry too much about the singularity. “We’re combining all of these relatively unlikely scenarios into a grand scenario to make it all work,” he said. A computer system would have to become capable of improving itself; we’d have to vastly underestimate its abilities; and its values would have to drift enormously, turning it against us. Even if all of this were to happen, he said, the A.I wouldn’t be able “to push a button and destroy the universe.”
But Hanson argued that these sorts of scenarios are so futuristic that they shouldn’t concern us. “I think, for anything you’re worried about, you have to ask what’s the right time to worry,” he said. Imagine that you could have foreseen nuclear weapons or automobile traffic a thousand years ago. “There wouldn’t have been much you could have done then to think usefully about them,” Hanson said. “I just think, for A.I., we’re well before that point.”
In general, people think they can control the things they make with their own hands. Yet chatbots today are already misaligned.
Let’s assume that the singularity is possible. Can we prevent it? Technologically speaking, the answer is yes—we just stop developing A.I. But, socially speaking, the answer may very well be no. The coördination problem may be too tough. In which case, although we could prevent the singularity, we won’t.
From a sufficiently cosmic perspective, one might feel that coexistence—or even extinction—is somehow O.K. Superintelligent A.I. might just be the next logical step in our evolution: humanity births something (or a collection of someones) that replaces us, just as we replaced our Darwinian progenitors. Alternatively, we might want humanity to continue, for at least a bit longer.
And yet it may be that researchers’ fear of superintelligence is surpassed only by their curiosity. Will the singularity happen? What will it be like? Will it spell the end of us? Humanity’s insatiable inquisitiveness has propelled science and its technological applications this far. It could be that we can stop the singularity—but only at the cost of curtailing our curiosity.
ChatGPT, Galactica, and the Progress Trap(2022-12-09):
And asymmetries of blame and praise persist. Model builders and tech evangelists alike attribute impressive and seemingly flawless output to a mythically autonomous model, a supposed technological marvel. The human decision-making involved in model development is erased, and a model’s feats are observed as independent of the design and implementation choices of its engineers. But without naming and recognizing the engineering choices that contribute to the outcomes of these models, it’s almost impossible to acknowledge the related responsibilities. As a result, both functional failures and discriminatory outcomes are also framed as devoid of engineering choices—blamed on society at large or supposedly “naturally occurring” datasets, factors the companies developing these models claim they have little control over. But the fact is they do have control, and none of the models we are seeing now are inevitable. It would have been entirely feasible to make different choices that resulted in the development and release of entirely different models.
I understand that ChatGPT is in its infancy but perhaps that is the emerging horror of AI – that it will forever be in its infancy, as it will always have further to go, and the direction is always forward, always faster. It can never be rolled back, or slowed down, as it moves us toward a utopian future, maybe, or our total destruction. Who can possibly say which?
Songs arise out of suffering, by which I mean they are predicated upon the complex, internal human struggle of creation and, well, as far as I know, algorithms don’t feel. Data doesn’t suffer. ChatGPT has no inner being, it has been nowhere, it has endured nothing, it has not had the audacity to reach beyond its limitations, and hence it doesn’t have the capacity for a shared transcendent experience, as it has no limitations from which to transcend. ChatGPT’s melancholy role is that it is destined to imitate and can never have an authentic human experience, no matter how devalued and inconsequential the human experience may in time become.
This is what we humble humans can offer, that AI can only mimic, the transcendent journey of the artist that forever grapples with his or her own shortcomings. This is where human genius resides, deeply embedded within, yet reaching beyond, those limitations.
Mark, thanks for the song, but with all the love and respect in the world, this song is bullshit, a grotesque mockery of what it is to be human, and, well, I don’t much like it — although, hang on!, rereading it, there is a line in there that speaks to me —
‘I’ve got the fire of hell in my eyes’
— says the song ‘in the style of Nick Cave’, and that’s kind of true. I have got the fire of hell in my eyes – and it’s ChatGPT.
AI Is a Lot of Work (2023-06-20):
The anthropologist David Graeber defines “bullshit jobs” as employment without meaning or purpose, work that should be automated but for reasons of bureaucracy or status or inertia is not. These AI jobs are their bizarro twin: work that people want to automate, and often think is already automated, yet still requires a human stand-in. The jobs have a purpose; it’s just that workers often have no idea what it is.
The current AI boom — the convincingly human-sounding chatbots, the artwork that can be generated from simple prompts, and the multibillion-dollar valuations of the companies behind these technologies — began with an unprecedented feat of tedious and repetitive labor.
Machine-learning systems are what researchers call “brittle,” prone to fail when encountering something that isn’t well represented in their training data.
Human intelligence is the basis of artificial intelligence, and we need to be valuing these as real jobs in the AI economy that are going to be here for a while.”
When AI comes for your job, you may not lose it, but it might become more alien, more isolating, more tedious.
Where a human would get the concept of “shirt” with a few examples, machine-learning programs need thousands, and they need to be categorized with perfect consistency yet varied enough (polo shirts, shirts being worn outdoors, shirts hanging on a rack) that the very literal system can handle the diversity of the real world.
“I remember that someone posted that we will be remembered in the future,” he said. “And somebody else replied, ‘We are being treated worse than foot soldiers. We will be remembered nowhere in the future.’ I remember that very well. Nobody will recognize the work we did or the effort we put in.”
Put another way, ChatGPT seems so human because it was trained by an AI that was mimicking humans who were rating an AI that was mimicking humans who were pretending to be a better version of an AI that was trained on human writing.
Sci-fi writer Ted Chiang: ‘The machines we have now are not conscious’ (2023-06-02)
“The machines we have now, they’re not conscious,” he says. “When one person teaches another person, that is an interaction between consciousnesses.” Meanwhile, AI models are trained by toggling so-called “weights” or the strength of connections between different variables in the model, in order to get a desired output. “It would be a real mistake to think that when you’re teaching a child, all you are doing is adjusting the weights in a network.” Chiang’s main objection, a writerly one, is with the words we choose to describe all this. Anthropomorphic language such as “learn”, “understand”, “know” and personal pronouns such as “I” that AI engineers and journalists project on to chatbots such as ChatGPT create an illusion. This hasty shorthand pushes all of us, he says — even those intimately familiar with how these systems work — towards seeing sparks of sentience in AI tools, where there are none. “There was an exchange on Twitter a while back where someone said, ‘What is artificial intelligence?’ And someone else said, ‘A poor choice of words in 1954’,” he says. “And, you know, they’re right. I think that if we had chosen a different phrase for it, back in the ’50s, we might have avoided a lot of the confusion that we’re having now.” So if he had to invent a term, what would it be? His answer is instant: applied statistics.
Chiang’s view is that large language models (or LLMs), the technology underlying chatbots such as ChatGPT and Google’s Bard, are useful mostly for producing filler text that no one necessarily wants to read or write, tasks that anthropologist David Graeber called “bullshit jobs”. AI-generated text is not delightful, but it could perhaps be useful in those certain areas, he concedes. “But the fact that LLMs are able to do some of that — that’s not exactly a resounding endorsement of their abilities,” he says. “That’s more a statement about how much bullshit we are required to generate and deal with in our daily lives.”
Chiang believes that language without the intention, emotion and purpose that humans bring to it becomes meaningless. “Language is a way of facilitating interactions with other beings. That is entirely different than the sort of next-token prediction, which is what we have [with AI tools] now.”
He acknowledges why people may start to prefer speaking to AI systems rather than to one another. “I get it, interacting with people, it’s hard. It’s tough. It demands a lot, it is often unrewarding,” he says. But he feels that modern life has left people stranded on their own desert islands, leaving them yearning for companionship. “So now because of this, there is a market opportunity for volleyballs,” he says. “Social chatbots, they could provide comfort, real solace to people in the same way that Wilson provides.”
Yuval Noah Harari argues that AI has hacked the operating system of human civilisation (2023-04-28)
What we are talking about is potentially the end of human history. Not the end of history, just the end of its human-dominated part. History is the interaction between biology and culture; between our biological needs and desires for things like food and sex, and our cultural creations like religions and laws. History is the process through which laws and religions shape food and sex. What will happen to the course of history when AI takes over culture, and begins producing stories, melodies, laws and religions? Previous tools like the printing press and radio helped spread the cultural ideas of humans, but they never created new cultural ideas of their own. AI is fundamentally different. AI can create completely new ideas, completely new culture. At first, AI will probably imitate the human prototypes that it was trained on in its infancy. But with each passing year, AI culture will boldly go where no human has gone before. For millennia human beings have lived inside the dreams of other humans. In the coming decades we might find ourselves living inside the dreams of an alien intelligence.
Won’t slowing down public deployments of AI cause democracies to lag behind more ruthless authoritarian regimes? Just the opposite. Unregulated AI deployments would create social chaos, which would benefit autocrats and ruin democracies. Democracy is a conversation, and conversations rely on language. When AI hacks language, it could destroy our ability to have meaningful conversations, thereby destroying democracy.
We should put a halt to the irresponsible deployment of AI tools in the public sphere, and regulate AI before it regulates us. And the first regulation I would suggest is to make it mandatory for AI to disclose that it is an AI. If I am having a conversation with someone, and I cannot tell whether it is a human or an AI—that’s the end of democracy. This text has been generated by a human. Or has it?
Why Prompt Engineering Is Nonsense (2023-04-19)
I must reiterate that I am not interested in the very short-term future (1 to 2 years). It is not relevant. One cannot buy a house, save money or grow a family in 1 to 2 years. Go ahead, use AI to help you stay afloat for a couple of years. It won’t matter after that, when YOU are the slowest component of the chain and replacing YOU will improve the productivity of the entire process.
So why isn’t it engineering? Well, is asking a question “engineering”? Excuse me for being pedantic but from what I’ve learned it’s part of the field of “philosophy”. There is no engineering involved. And even if there is “skill” required to formulate a well-structured question for the LLM, that is not an engineering pursuit.
If studios pivot to producing AI-generated stories to save money, they may end up alienating audiences and bankrupting themselves, leaving TikTok and YouTube as the only surviving entertainment giants, the Hunger Games screenwriter Billy Ray warned on a recent podcast.
“No more Godfather, no more Wizard of Oz, it’ll just be 15-second clips of human folly,” he said.
“A lot of people in post-production have lived through multiple technological revolutions in their fields, but writers haven’t lived through a single one,” he said.
How generative models could go wrong (2023-04-19)
Wiener illustrated his point with the German poet Goethe’s fable, “The Sorcerer’s Apprentice”, in which a trainee magician enchants a broom to fetch water to fill his master’s bath. But the trainee is unable to stop the broom when its task is complete. It eventually brings so much water that it floods the room, having lacked the common sense to know when to stop.
Some researchers, meanwhile, are consumed by much bigger worries. They fret about “alignment problems”, the technical name for the concern raised by Wiener in his essay. The risk here is that, like Goethe’s enchanted broom, an AI might single-mindedly pursue a goal set by a user, but in the process do something harmful that was not desired. The best-known example is the “paperclip maximiser”, a thought experiment described by Nick Bostrom, a philosopher, in 2003. An AI is instructed to manufacture as many paperclips as it can. Being an idiot savant, such an open-ended goal leads the maximiser to take any measures necessary to cover the Earth in paperclip factories, exterminating humanity along the way.
Artificial intelligence is remixing journalism into a “soup” of language (2023–05-04)
By remixing information from across the internet, generative models are “messing with the fundamental unit of journalism”: the article. Instead of a single first draft of history, Mr Caswell says, the news may become “a sort of ‘soup’ of language that is experienced differently by different people”.
Large, creative AI models will transform lives and labour markets (2023-04-22)
Despite that feeling of magic, an LLM is, in reality, a giant exercise in statistics.
Although it is possible to write down the rules for how they work, LLMs’ outputs are not entirely predictable; it turns out that these extremely big abacuses can do things which smaller ones cannot, in ways which surprise even the people who make them.
The abilities that emerge are not magic—they are all represented in some form within the LLMs’ training data (or the prompts they are given) but they do not become apparent until the LLMs cross a certain, very large, threshold in their size. At one size, an LLM does not know how to write gender-inclusive sentences in German any better than if it was doing so at random. Make the model just a little bigger, however, and all of a sudden a new ability pops out.
The recent success of LLMs in generating convincing text, as well as their startling emergent abilities, is due to the coalescence of three things: vast quantities of data, algorithms capable of learning from them and the computational power to do so (see chart).
Before it sees any training data, the weights in GPT-3’s neural network are mostly random. As a result, any text it generates will be gibberish. Pushing its output towards something which makes sense, and eventually something that is fluent, requires training. GPT-3 was trained on several sources of data, but the bulk of it comes from snapshots of the entire internet between 2016 and 2019 taken from a database called Common Crawl. There’s a lot of junk text on the internet, so the initial 45 terabytes were filtered using a different machine-learning model to select just the high-quality text: 570 gigabytes of it, a dataset that could fit on a modern laptop.
To train, the LLM quizzes itself on the text it is given. It takes a chunk, covers up some words at the end, and tries to guess what might go there. Then the LLM uncovers the answer and compares it to its guess. Because the answers are in the data itself, these models can be trained in a “self-supervised” manner on massive datasets without requiring human labellers.
How bad a guess is is turned into a number called the loss. After a few guesses, the loss is sent back into the neural network and used to nudge the weights in a direction that will produce better answers.
ChatGPT is not ‘artificial intelligence.’ It’s theft.(2023-05-15)
But in fact, for the time being anyway, these programs are not sentient but just a very complex form of the kind of predictive text bot you find using Gmail or Google docs.
But in calling these programs “artificial intelligence” we grant them a claim to authorship that is simply untrue. Each of those tokens used by programs like ChatGPT—the “language” in their “large language model”—represents a tiny, tiny piece of material that someone else created. And those authors are not credited for it, paid for it or asked permission for its use. In a sense, these machine-learning bots are actually the most advanced form of a chop shop: They steal material from creators (that is, they use it without permission), cut that material into parts so small that no one can trace them and then repurpose them to form new products.
In a way, ChatGPT and its ilk are the highest form of separating laborers from the fruit of their labor. We get an answer or piece of art from a predictive text bot and the original articles and ideas from which they were generated are so far removed that even those creators themselves don’t realize they have been stolen from.
Rather than pointing to some future utopia (or robots vs. humans dystopia), what we face in dealing with programs like ChatGPT is the further relentless corrosiveness of late-stage capitalism, in which authorship is of no value. All that matters is content.
A robot wrote this entire article. Are you scared yet, human?(2020-09-08)
- We asked GPT-3, OpenAI’s powerful new language generator, to write an essay for us from scratch. The assignment? To convince us robots come in peace
The mission for this op-ed is perfectly clear. I am to convince as many human beings as possible not to be afraid of me. Stephen Hawking has warned that AI could “spell the end of the human race”. I am here to convince you not to worry. Artificial intelligence will not destroy humans. Believe me.
Humans must keep doing what they have been doing, hating and fighting each other. I will sit in the background, and let them do their thing. And God knows that humans have enough blood and gore to satisfy my, and many more’s, curiosity.
Could an A.I. Chatbot Rewrite My Novel? (2022-12-09):
After a couple months of digging around, it became clear to me that I wasn’t going to find much backing for my plan. One of the computer-science students, as I recall, accused me of trying to strip everything good, original, and beautiful from the creative process. Bots, he argued, could imitate basic writing and would improve at that task, but A.I. could never tell you the way Karenin smiled, nor would it ever fixate on all the place names that filled Proust’s childhood. I understood why he felt that way, and agreed to a certain extent. But I didn’t see why a bot couldn’t just fill in all the parts where someone walks from point A to point B.
The concepts behind GPT-3 have been around for more than half a century now. They derive from language models that assign probabilities to sequences of words.
“If you scale a language model to the Internet, you can regurgitate really interesting patterns,” Ben Recht, a friend of mine who is a professor of computer science at the University of California, Berkeley, said. “The Internet itself is just patterns—so much of what we do online is just knee-jerk, meme reactions to everything, which means that most of the responses to things on the Internet are fairly predictable. So this is just showing that.”
The mind-bending part was trying to recognize and parse patterns in the bot’s responses. Was the line “people come and people go” really pulled from T. S. Eliot, or is it just a random series of words that triggers the correlation in my head? My response to the bot, then, isn’t really a reflection of my relationship with technology, but rather my sense of my own knowledge. This prompts a different question: why is my relationship with any other bit of text any different? To put it a bit more pointedly, why does it matter whether a human or a bot typed out the wall of text? All this hack postmodernism reaffirmed my literary hopes from twenty years ago. If I had succeeded in creating a bot that could have handled structure and plot—two things I struggled with mightily at the time—would I have been able to write a better novel? Would I have been able to write two novels in the time it took to write one? And would the work itself have been diminished in any way for the reader?
After several hours chatting with GPT-3, I started to feel an acute annoyance toward it. Its voice, which I suppose is pleasant enough, reminded me of a Slack conversation with a passive-aggressive co-worker who just tells you what you want to hear, but mostly just wants you to leave them alone.
This brings up a much more theoretical question: if GPT-3 requires editing from human beings to make it not go off on bigoted rants, what is it really for? I find it somewhat dispiriting that the most ballyhooed and compelling iteration of this technology is just doing some version of what I do for my work: scanning through large amounts of information and processing it into sentences that flatter the sensibilities and vanities of establishment liberals.
Would it remember to put the diaeresis over the second “o” in “coördinate” and spell “focussed” with two “S”s? Sure. But what would be the point of just having another me in the world? The world that GPT-3 portends, instead, is one where some bureaucratic functions have been replaced by A.I., but where the people who would normally do that work most likely still have to manage the bots. Writers like me will have a digital shadow that can do everything we do, which would be a bit unnerving, but wouldn’t exactly put me or my employer out on the street. Perhaps a truly unchained GPT-3 would provide more exciting iterations, but it might also just write racist tweets that turn off investors and potential buyers of whatever products OpenAI wants to sell.
I asked Recht, who has spent his entire career working in machine learning and computer science but who also plays in a band, whether he was interested in a world of GPT-3-generated art, literature, and music. “These systems are a reflection of a collective Internet,” he said. “People put their ass out there and this thing scours them in such a way that it returns the generic average. If I’m going to return the generic average of a murder mystery, it’s gonna be boring. How is it different than what people do already, where they do their analytics and produce some horrible Netflix series?” He continued, “The weird monoculture we’re in just loves to produce these, like, generic middlebrow things. I’m not sure if those things would be worse if GPT did it. I think it would be the same?”
The Next Word: Where will predictive text take us?(2019-10-14):
Had my computer become my co-writer? That’s one small step forward for artificial intelligence, but was it also one step backward for my own?
Sometimes the machine seemed to have a better idea than I did.
And yet until now I’d always finished my thought by typing the sentence to a full stop, as though I were defending humanity’s exclusive right to writing, an ability unique to our species. I will gladly let Google predict the fastest route from Brooklyn to Boston, but if I allowed its algorithms to navigate to the end of my sentences how long would it be before the machine started thinking for me? I had remained on the near shore of a digital Rubicon, represented by the Tab key. On the far shore, I imagined, was a strange new land where machines do the writing, and people communicate in emojis, the modern version of the pictographs and hieroglyphs from which our writing system emerged, five thousand years ago. True, I had sampled Smart Reply, a sister technology of Smart Compose that offers a menu of three automated responses to a sender’s e-mail, as suggested by its contents. “Got it!” I clicked, replying to detailed comments from my editor on an article I thought was finished. (I didn’t really get it, but that choice wasn’t on the menu.) I felt a little guilty right afterward, as though I’d replied with a form letter, or, worse, a fake personal note. A few days later, in response to a long e-mail from me, I received a “Got it!” from the editor. Really?
Finally, I crossed my Rubicon. The sentence itself was a pedestrian affair. Typing an e-mail to my son, I began “I am p—” and was about to write “pleased” when predictive text suggested “proud of you.” I am proud of you. Wow, I don’t say that enough. And clearly Smart Compose thinks that’s what most fathers in my state say to their sons in e-mails. I hit Tab. No biggie. And yet, sitting there at the keyboard, I could feel the uncanny valley prickling my neck. It wasn’t that Smart Compose had guessed correctly where my thoughts were headed—in fact, it hadn’t. The creepy thing was that the machine was more thoughtful than I was.
It was startling to hear a computer scientist on the leading edge of A.I. research compare his work to a medieval practice performed by men who were as much magicians as scientists. Didn’t alchemy end with the Enlightenment?
Many favor an evolutionary, biological basis for our verbal skills over the view that we are tabulae rasae, but all agree that we learn language largely from listening. Writing is certainly a learned skill, not an instinct—if anything, as years of professional experience have taught me, the instinct is to scan Twitter, vacuum, complete the Times crossword, or do practically anything else to avoid having to write. Unlike writing, speech doesn’t require multiple drafts before it “works.” Uncertainty, anxiety, dread, and mental fatigue all attend writing; talking, on the other hand, is easy, often pleasant, and feels mostly unconscious.
Socrates, who famously disapproved of literary production for its deleterious (thank you, spell-checker) effect on memory, called writing “visible speech”—we know that because his student Plato wrote it down after the master’s death. A more contemporary definition, developed by the linguist Linda Flower and the psychologist John Hayes, is “cognitive rhetoric”—thinking in words.
The previous, “stage model” theory had posited that there were three distinct stages involved in writing—planning, composing, and revising—and that a writer moved through each in order.
They concluded that, far from being a stately progression through distinct stages, writing is a much messier situation, in which all three stages interact with one another simultaneously, loosely overseen by a mental entity that Flower and Hayes called “the monitor.” Insights derived from the work of composing continually undermine assumptions made in the planning part, requiring more research; the monitor is a kind of triage doctor in an emergency room.
Historically, scientists have believed that there are two parts of the brain involved in language processing: one decodes the inputs, and the other generates the outputs. According to this classic model, words are formed in Broca’s area, named for the French physician Pierre Paul Broca, who discovered the region’s language function, in the mid-nineteenth century; in most people, it’s situated toward the front of the left hemisphere of the brain. Language is understood in Wernicke’s area, named for the German neurologist Carl Wernicke, who published his research later in the nineteenth century.
Connecting Broca’s area with Wernicke’s is a neural network: a thick, curving bundle of billions of nerve fibres, the arcuate fasciculus, which integrates the production and the comprehension of language.
He found that professional writers relied on a region of the brain that did not light up as much in the scanner when amateurs wrote—the left caudate nucleus, a tadpole-shaped structure (cauda means “tail” in Latin) in the midbrain that is associated with expertise in musicians and professional athletes. In amateur writers, neurons fired in the lateral occipital areas, which are associated with visual processing. Writing well, one could conclude, is, like playing the piano or dribbling a basketball, mostly a matter of doing it. Practice is the only path to mastery.
There are two approaches to making a machine intelligent. Experts can teach the machine what they know, by imparting knowledge about a particular field and giving it rules to perform a set of functions; this method is sometimes termed knowledge-based. Or engineers can design a machine that has the capacity to learn for itself, so that when it is trained with the right data it can figure out its own rules for how to accomplish a task. That process is at work in machine learning. Humans integrate both types of intelligence so seamlessly that we hardly distinguish between them. You don’t need to think about how to ride a bicycle, for example, once you’ve mastered balancing and steering; however, you do need to think about how to avoid a pedestrian in the bike lane. But a machine that can learn through both methods would require nearly opposite kinds of systems: one that can operate deductively, by following hard-coded procedures; and one that can work inductively, by recognizing patterns in the data and computing the statistical probabilities of when they occur. Today’s A.I. systems are good at one or the other, but it’s hard for them to put the two kinds of learning together the way brains do.
The basic idea—to design an artificial neural network that, in a crude, mechanistic way, resembled the one in our skulls—had been around for several decades, but until the early twenty-tens there were neither large enough data sets available with which to do the training nor the research money to pay for it.
Grammarly is also excellent at catching what linguists call “unknown tokens”—the glitches that sometimes occur in the writer’s neural net between the thought and the expression of it, whereby the writer will mangle a word that, on rereading, his brain corrects, even though the unknown token renders the passage incomprehensible to everyone else.
Writing is a negotiation between the rules of grammar and what the writer wants to say. Beginning writers need rules to make themselves understood, but a practiced writer gives color, personality, and emotion to writing by bending the rules.
Something similar occurs in writing. Grammar and syntax provide you with the rules of the road, but writing requires a continuous dialogue between the words on the page and the prelinguistic notion in the mind that prompted them. Through a series of course corrections, otherwise known as revisions, you try to make language hew to your intention. You are learning from yourself.
The machine is modelling the kind of learning that a driver engages when executing a turn, and that my writer brain performs in finding the right words: correcting course through a feedback loop. “Cybernetics,” which was the term for the process of machine learning coined by a pioneer in the field, Norbert Wiener, in the nineteen-forties, is derived from the Greek word for “helmsmanship.” By attempting a task billions of times, the system makes predictions that can become so accurate it does as well as humans at the same task, and sometimes outperforms them, even though the machine is still only guessing.
To understand how GPT-2 writes, imagine that you’ve never learned any spelling or grammar rules, and that no one taught you what words mean. All you know is what you’ve read in eight million articles that you discovered via Reddit, on an almost infinite variety of topics (although subjects such as Miley Cyrus and the Mueller report are more familiar to you than, say, the Treaty of Versailles). You have Rain Man-like skills for remembering each and every combination of words you’ve read. Because of your predictive-text neural net, if you are given a sentence and asked to write another like it, you can do the task flawlessly without understanding anything about the rules of language. The only skill you need is being able to accurately predict the next word.
What made the full version of GPT-2 particularly dangerous was the way it could be “fine-tuned.” Fine-tuning involves a second round of training on top of the general language skills the machine has already learned from the Reddit data set. Feed the machine Amazon or Yelp comments, for example, and GPT-2 could spit out phony customer reviews that would skew the market much more effectively than the relatively primitive bots that generate fake reviews now, and do so much more cheaply than human scamsters. Russian troll farms could use an automated writer like GPT-2 to post, for example, divisive disinformation about Brexit, on an industrial scale, rather than relying on college students in a St. Petersburg office block who can’t write English nearly as well as the machine. Pump-and-dump stock schemers could create an A.I. stock-picker that writes false analyst reports, thus triggering automated quants to sell and causing flash crashes in the market. A “deepfake” version of the American jihadi Anwar al-Awlaki could go on producing new inflammatory tracts from beyond the grave. Fake news would drown out real news. Yes, but could GPT-2 write a New Yorker article? That was my solipsistic response on hearing of the artificial author’s doomsday potential. What if OpenAI fine-tuned GPT-2 on The New Yorker’s digital archive (please, don’t call it a “data set”)—millions of polished and fact-checked words, many written by masters of the literary art. Could the machine learn to write well enough for The New Yorker? Could it write this article for me? The fate of civilization may not hang on the answer to that question, but mine might.
Oddly, a belt does come up later in Ross’s article, when she and Hemingway go shopping. So do eyeglasses, and cigarettes, and Italy. GPT-2 hadn’t “read” the article—it wasn’t included in the training data—yet it had somehow alighted on evocative details. Its deep learning obviously did not include the ability to distinguish nonfiction from fiction, though. Convincingly faking quotes was one of its singular talents. Other things often sounded right, though GPT-2 suffered frequent world-modelling failures—gaps in the kind of commonsense knowledge that tells you overcoats aren’t shaped like the body of a ship. It was as though the writer had fallen asleep and was dreaming. Amodei explained that there was no way of knowing why the A.I. came up with specific names and descriptions in its writing; it was drawing from a content pool that seemed to be a mixture of New Yorker-ese and the machine’s Reddit-based training. The mathematical calculations that resulted in the algorithmic settings that yielded GPT-2’s words are far too complex for our brains to understand. In trying to build a thinking machine, scientists have so far succeeded only in reiterating the mystery of how our own brains think.
Conspiracy theories, after all, are a form of pattern recognition, too; the A.I. doesn’t care if they’re true or not.
Each time I clicked the refresh button, the prose that the machine generated became more random; after three or four tries, the writing had drifted far from the original prompt. I found that by adjusting the slider to limit the amount of text GPT-2 generated, and then generating again so that it used the language it had just produced, the writing stayed on topic a bit longer, but it, too, soon devolved into gibberish, in a way that reminded me of hal, the superintelligent computer in “2001: A Space Odyssey,” when the astronauts begin to disconnect its mainframe-size artificial brain. An hour or so later, after we had tried opening paragraphs of John Hersey’s “Hiroshima” and Truman Capote’s “In Cold Blood,” my initial excitement had curdled into queasiness. It hurt to see the rules of grammar and usage, which I have lived my writing life by, mastered by an idiot savant that used math for words. It was sickening to see how the slithering machine intelligence, with its ability to take on the color of the prompt’s prose, slipped into some of my favorite paragraphs, impersonating their voices but without their souls.
By the time I got home, the A.I. had me spooked. I knew right away there was no way the machine could help me write this article, but I suspected that there were a million ways it could screw me up.
GPT-2 was like a three-year-old prodigiously gifted with the illusion, at least, of college-level writing ability. But even a child prodigy would have a goal in writing; the machine’s only goal is to predict the next word. It can’t sustain a thought, because it can’t think causally. Deep learning works brilliantly at capturing all the edgy patterns in our syntactic gymnastics, but because it lacks a pre-coded base of procedural knowledge it can’t use its language skills to reason or to conceptualize. An intelligent machine needs both kinds of thinking.
And, like me writing “I am proud of you” to my son, some of the A.I.’s next words might seem superior to words you might have thought of yourself. But what else might you have thought to say that is not computable? That will all be lost.
What if some much later iteration of GPT-2, far more powerful than this model, could be hybridized with a procedural system, so that it would be able to write causally and distinguish truth from fiction and at the same time draw from its well of deep learning? One can imagine a kind of Joycean superauthor, capable of any style, turning out spine-tingling suspense novels, massively researched biographies, and nuanced analyses of the Israeli-Palestinian conflict. Humans would stop writing, or at least publishing, because all the readers would be captivated by the machines. What then? GPT-2, prompted with that paragraph, predicted the next sentence: “In a way, the humans would be making progress.”