Model collapse is the phenomenon in which text or images output by AI are uploaded to the internet, those outputs become mixed into AI training data, the AI trained on that data then produces further outputs that are uploaded, and this cycle repeats until AI performance collapses. The most famous study is probably AI models […]
Model Collapse: AI and diversity Read Post »
What I like about this result is that it combines a strong “wait, you can do that?” feeling with real research usefulness. I recently used it in our paper, and since ChatGPT could not answer it when I asked, I decided to write this post. The original source is Dynamic Generation of Discrete Random Variate.
Constant-Time Sampling And Updates for Discrete Distributions Read Post »
If there are a million people, then there will be someone who flips heads 20 times in a row. Even when there is no difference in how many chances people are given and no difference in how lucky they are, and the playing field is perfectly equal, some people will still end up looking lucky.
Stop Leaving Your Success to Luck: A Researcher’s Guide to Hacking Randomness Read Post »
If you are unsure what to do in the age of AI, it is good to look to the past.When you think about what kind of life you have lived up to now, a path opens up. AI levels people’s abilities.With coding agents, the gap between the very best programmer and an ordinary programmer becomes
Past Becomes Important in the Era of AI Read Post »
AI performance is improving day by day, and its evolution may appear unstoppable. However, in reality, that is not what will happen. In this article, I explain why. The key terms in this article are epistemic uncertainty and aleatoric uncertainty. Epistemic uncertainty is uncertainty that can be resolved through observation, whereas aleatoric uncertainty arises from
Why AI Progress Will Stall Read Post »
Cognitive semantics is a linguistic approach that understands the meaning of words in relation to human cognition and ways of perceiving things. In this article, I would like to talk about the “cognition” of LLMs from the perspective of cognitive semantics. In the background of cognitive semantics are fields such as Gestalt psychology and cognitive
How LLMs Feel Language Read Post »
I think a lot of people feel a real sense of crisis about AI taking their jobs. I feel it too. In fact, I am in the almost absurd, joke-like situation where I was supposed to be working on building AI, and now even that very job looks as though it may itself be taken
How to Survive the AI Revolution Read Post »
We can flatten the parameters of a deep neural network into a single vector. For a large model, this vector can easily have tens of billions of dimensions. At first glance it may look like a meaningless list of numbers, but it has become clear that this vector has deep meaning as a vector. For
Model Merge Explained: Build Models without Training Read Post »
Over the past few years, LLMs have developed rapidly, significantly expanding our cognitive capabilities. Many people probably now turn to AI first when they encounter a problem. It may be that LLMs already know more than my brain does, and perhaps possess higher reasoning abilities than my brain. However, in this article, I will discuss
Why Your Brain Matters in the Era of AI: You Can’t Outsource Intuition Read Post »
In this article, I will argue that transformers are RNNs. I will not merely point out formal containment; I will also explain how similar transformers and RNNs are, and what practical implications follow from that viewpoint. Transformers are RNNs First, I will show that transformers are RNNs. To put it simply, a transformer is an
Transformers are RNNs: A Kernel Perspective Read Post »
