Arbitraging language models for gains?
Posting this as an intellectual exercise. And calling it intellectual is probably a stretch, as I haven’t really thought much about it.
We often say that LLMs answers are average, as in, they will tell you what is the most likely answer among all the data they have been trained on. It usually won’t give you the outlier (and even if you prompt specifically for such outliers, you’re getting the mean outcome among that outlier field).
A few days ago, someone told me that if you ask your LLM to ask for a random number between 1 and 30, there is a very high probability it’ll answer 17 (unless you prompt it to invoke a random number generator, ofc). Regardless of the specific LLM you ask.
My wife has been prompting Gemini a lot these days asking it whether a certain stock will go up or down. It often answers her by giving her a probability that it will go up. I got bothered by how much fait she puts in these numbers, and then, to try to prove a point (yes, silly, I know… happy wife, happy life), I asked her to ask the same question, but in the negative (using a new session). To illustrate that
- What is the probability that SK Hynix’s stock goes up? Answer: 20%
- What is the probability that SK Hynix’s stock goes down? Answer: 70%
Simplifying by ignoring the case that it stays completely flat, there is a clear mismatch between these two answers. If I could trade against the LLM, there is a 10% arbitrage opportunity here.
Of course, I can’t trade against an LLM.
But, putting all the above together, would it be possible to take advantage of the fact that my wife is likely not the only one to ask such questions to their favorite LLM? And retail people using LLMs to ask such questions likely use the mainstream ones, so one could check which probabilities these LLMs converge on.
[Insert plenty of other simplifications (small variations to the above prompts, delays in training, etc) and their impact here]
And maybe, just maybe, one could then figure out how to leverage this retail asymmetry?
Then again, I hate the stock market, and I keep telling my wife that day trading is a losing game, so it’ll just remain a silly thought on SN.
Write a comment