o1 – What is Going On? Why o1 is a 3rd Paradigm of Model + 10 Things You Might Not Know

@sickandtired6156 says:

Exactly was about to comment

@user93237 says:

This is to be expected because growth in accuracy becomes exponentially hard with higher accuracy as for each item there is a chance of the model not knowing the correct answer. This corresponds to a conjunction of many events with probability <1 resulting in a product that tends toward zero probability. Accuracy, hence, does not measure capabilities very well in the lower and upper ranges. I think you would actually have to transform the y-axis by an inverse sigmoid to get something more intuitive. However, accuracy overall can of course be a misleading measure still due to things like data leakage, overfitting, ceiling effect, lacking generalization to other datasets and so on.

@TobiasWeg says:

I removed my comment making the same point, it’s good point and you are right. OpenAI sentence, below the graph, is correct, but misleading.

@CellarDoorCS says:

I used Yan LeCuns current llm bench question – walking along the sphere. It failed

@ThreeChe says:

@@r-saint No Terminators until we get integrated spatial reasoning. Although 1 billion raised in 4 months for that startup gives me a feeling it’ll come sooner rather than later.

@alansmithee419 says:

@@CellarDoorCS
I have so far not been happy with the fact that pretty much everyone using that test seems to be completely unaware of how straight lines work on spheres. That said, I also haven’t found an AI model that gives a good answer. But the problem isn’t limited to AI models, most humans can’t answer it either, or more often answer it confidentially incorrectly. So I wouldn’t use a model’s failure to answer it in an attempt to demonstrate that it’s any lesser than humans.

To make this quick as I can (which, it’s me, so that’s probably not gonna be very quick):

A straight line on a sphere is not a line of latitude. People often look at the question as meaning you walk some way around the earth’s curvature south, and then because you turn to face east for example and start walking in a straight line they assume that means you will continue to travel exactly east on a line of latitude, a constant distance away from the pole. That is not the case. To see why, imagine only walking one meter away from the north pole in a physical space (like literally imagine yourself standing at the north pole and walking away 1 meter). Then turn 90 degrees left, and walk in a straight line. The assumption made in most people’s attempts implies that you will now walk in a 1 meter radius circle (a line of latitude with very close to a 90 degree north angle) around the north pole. That is clearly not a straight line as the question requires, as you are actively turning the entire time. Unless you walk all the way to the equator, you cannot think of the problem like this.

Straight lines on spheres are called great circles. On earth, they loop all away around the planet in a circle and have the core of the earth at their centre. All of them have an approximate length of 40,000km and radius roughly equal to the standard radius of the earth (varies slightly depending on your starting distance from the equator and direction).

@Hexanitrobenzene says:

Yeah, and his manner of speech is very distinctive – perfect for ominous statements 🙂