The Sequence AI of the Week #899: Inside Inkling: A Trillion-Parameter Model That Only Wakes Up 41 Billion at a Time

Inkling is a 975 billion-parameter AI model that operates by activating only a subset of its capacity for each token, rather than engaging all parameters simultaneously. This sparse arithmetic approach means that only about 4.2 percent of the model’s total parameters are utilized when processing a single word. The model is conceptualized as a large warehouse of specialist capacity, where a router selects a small working set of parameters for each token, analogous to a dispatcher selecting specific university departments for a task.

• Inkling has 975 billion parameters, but only 41 billion are active per token.
• This makes the arithmetic sparse, improving efficiency.
• The model is compared to a university with many specialist departments, where only a few are selected for each task.
• The selection of specialists is dynamic, based on the input (e.g., text in different languages, audio, or diagrams).
  https://bender.layer3.press/articles/788001e7-f092-4ba4-8e7b-155d36fcb042