Decent Newsroom
Sign In

PABLOF7z

f7z.io

Magical Other Stuff Maximalist.

f7z.io
Editorial Articles Media Highlights
65 ## profound impact large language models are about to have. If DQN and AlphaGo were the early signs of something lapping at the shore, ChatGPT and LLMs are the first signs of the wave beginning to crash around us. In 1996, thirty-six million people used the internet; this year it will be well over five billion. That's the kind of trajectory we should expect for these tools, only much faster. Over the next few years, I believe, Al will become as ubiquitous as the internet itself: just as available, and yet even more consequential. ## BRAIN-SCALE MODELS The Al systems I'm describing operate on an immense scale. Here's an example. Much of Al's progress during the mid-zotos was powered by the ef fectiveness of "supervised" deep learning. Here Al models learn from carefully hand-labeled data. Quite often the quality of the Al's predictions depends on the quality of the labels in the training data. However, a key ingredient of the LLM revolution is that for the first time very large models could be trained directly on raw, messy, real-world data, without the need for carefully curated and human-labeled data sets. As a result almost all textual data on the web became useful. The more the better. Today's LLMs are trained on trillions of words. Imag ine digesting Wikipedia wholesale, consuming all the subtitles and comments on YouTube, reading millions of legal contracts, tens of millions of emails, and hundreds of thousands of books. This kind of vast, almost instantaneous consumption of information is not just dif ficult to comprehend; it's truly alien. Pause here for a moment. Consider the unfathomable number of words that these models consume during training, If we assume that the average person can read about two hundred words per minute, in an eighty-year lifetime that would be about eight billion words, assuming they did absolutely nothing else twenty-four hours per day. More realistically, the average American reads a book for about fifteen minutes per day, which over the year amounts to reading about a million
E # sed in electrical grids and ## THE TECHNOLOGY OF INTELLIGENCE # sources at a time of grow- 63 ## about what comes next. Their work laid the foundation for what has # st how to write emails or been nothing short of a revolution in the field of large language modrite stories, diagnose rare els (LLMs)-including ChatGPT. ## LLMs take advantage of the fact that language data comes in a se- # e change. They feature in quential order. Each unit of information is in some way related to data # homes. You already inter- earlier in a series. The model reads very large numbers of sentences, # : many more, and almost learns an abstract representation of the information contained within # fficient, faster, more use- them, and then, based on this, generates a prediction about what should come next. The challenge lies in designing an algorithm that "knows where to look" for signals in a given sentence. What are the key words, the most salient elements of a sentence, and how do they ## HING: relate to one another? In Al this notion is commonly referred to as "attention." ## ¡E MODELS When a large language model ingests a sentence, it constructs what can be thought of as an "attention map." It first organizes commonly guage seemed too comoccurring groups of letters or punctuation into "tokens," something Then, in November 2022, like syllables, but really just chunks of frequently occurring letters latGPT. Within a week it making it easier for the model to process the information. It's worth alked about in rapturous noting that humans do this with words of course, but the model doesn't ght eclipse Google Search use our vocabulary. Instead, it creates a new vocabulary of common tokens that helps it spot patterns across billions and billions of docu- But it is so much more ments. In the attention map, every token bears some relationship to every token before it, and for a given input sentence the strength of ## ad previously been made this relationship describes something about the importance of that aneously in fluent prose. token in the sentence. In effect, the LLM learns which words to pay # siness plan in the style of attention to. ## does so in seconds. Ask it So if you take the sentence "There is going to be a fairly major # ting manual, or a Python storm tomorrow in Brazil," the model would likely create tokens for the letters "the" in the word "there" and "ing" in the word "going," # ent is that we look at the since they commonly occur in other words. When parsing the full sen- # ure. In this sense intelli- tence, it would learn that "storm," "tomorrow," and "Brazil" are the # rate a range of plausible key features, inferring that Brazil is a place, that a storm will be happening in the future, and so on. Based on this, it then suggests which # lay unfold and then base tokens should come next in the sequence, what output logically fol- # 2017 a small group of re- Lows the input. In other words, it autocompletes what might come # ver version of this prob- next. # on the most important and efficient predictions
Picture

Today we're visiting the island in front of ours. Telendos. No roads. Iirc about 15 ppl live here y...
Picture

Open waters #madeira
Picture

Test #olas365
Picture

If you’re training without logging you only think you’re training. #olas365
Picture

Birds chirping makes me happy #olas365 #madeira #nature
Picture

Had a great #family day 😍😍 #olas365 #madeira #travel
Picture

City jungle #olas365 #madeira #nature
Picture

Some engineers will recognize this spot #olas365 #madeira #travel #nature
Picture

Treegonometry #olas365 #madeira #travel #nature
Picture

Me every night #meme
Picture

A friend sent me this one 😎 #olas365 #olas #nostr
Picture

Fractals #olas365 #madeira #nature
Picture

#catstr
Picture

Missing #kalymnos #travel #nature
Picture

The first time I went #climbing with my daughter #olas365 #nature #family
Picture

It’s all perspective #olas365 #madeira #photography #nature

Picture

Freedom constraints, aka big cities
Picture

Imaginary numbers
Picture

#art
Picture

Furniture from the 15th century. How much of what we build nowadays will exist in 600 years? #olas...
Picture

Black hole sun #olas365 #nature