I can't see the downvoted comment in your comment history. Did you delete it? 

By the way, did you use an LLM such as ChatGPT or Claude to help write this post? It has the markings of LLM writing. I think when people detect that, they are turned off. They want to read what you wrote, not what an LLM wrote.

Another factor is that if you are a new poster, you get less benefit of the doubt and you need to work harder to state your points in plain English and make them clear as day. If it's not immediately clear what you're saying, and especially if your writing seems LLM-generated/LLM-assisted, people will not put in the time and effort to engage deeply. 

I don't think you can have it both ways: A superhuman coder (that is actually competent, which you don't think AI assistants are now) is relatively narrow AI, but would accelerate AI progress. A superhuman AI researcher is more general (which would drastically speed up AI progress), but is not fully general.

I definitely disagree with this. Hopefully what I say below will explain why.

I would argue that LLMs now are more general than AI researcher tasks (though LLMs are currently not good at all of those tasks), because LLMs can competently discuss philosophy, economics, political science, art, history, engineering, science, etc.

The general in artificial general intelligence doesn't just refer to having a large repertoire of skills. Generality is about the ability to learn to generalize beyond what a system has seen in its training data. An artificial general intelligence doesn't just need to have new skills, it needs to be able to acquire new skills, and to acquire new skills that have never existed in history before by developing them itself — just as humans do. 

If a new video game comes out today, I'm able to play that game and develop a new skill that has never existed before.^[1] I will probably get the hang of it in a few minutes, with a few attempts. That's general intelligence. 

AlphaStar was not able to figure out how to play StarCraft using pure reinforcement learning. It just got stuck using its builders to attack the enemy, rather than figuring out how to use its builders to make buildings that produce units that attack. To figure out the basics of the game, it needed to do imitation learning on a very large dataset of human play. Then, after imitation learning, to get as good as it did, it needed to do an astronomical amount of self-play, around 60,000 years of playing StarCraft. That's not general intelligence. If you need to copy a large data of human examples to acquire a skill and millennia of training on automatically gradable, relatively short time horizon tasks (which often don't exist in the real world), that's something, and it's even something impressive, but it's not general intelligence. 

Let's say you wanted to apply this kind of machine learning to AI R&D. The necessary conditions don't apply. You don't have a large dataset of human examples to train on. You don't have automatically gradable, relatively short time horizon tasks with which to do reinforcement learning. And if the tasks require real world feedback and can't be simulated, you certainly don't have 60,000 years.

I like what the AI researcher François Chollet has to say about this topic in this video from 11:45 to 20:00. He draws the distinction between crystallized behaviours and fluid intelligence, between skills and the ability to learn skills. I think this is important. This is really what the whole topic of AGI is about.

Why have LLMs absorbed practically all text on philosophy, economics, political science, art, history, engineering, science, and so on and not come up with a single novel and correct idea of any note in any of these domains? They are not able to generalize enough to do so. They can generalize or interpolate a little bit beyond their training data, but not very much. It's that generalization ability that's the holy grail. 

I'm claiming that they could approach overall staff to vehicle ratio of 1:10 if the number of real-time helpers (which don't have to be engineers) and vehicles were dramatically scaled up, and that's enough for profitability.

There are two concepts here. One is remote human assistance, which Waymo calls fleet response. The other is Waymo's approach to the engineering problem. I was saying that I suspect Waymo's approach to the engineering problem doesn't scale. I think it probably relies on engineers doing too much special casing that doesn't generalize well when a modest amount of novelty is introduced. So, Waymo currently has something like 1,500 engineers to operate in the comparatively small geofenced areas where it currently operates. If it wanted to expand where it drives to a 10x larger area, would its techniques generalize to that larger area, or would it need to hire commensurately more engineers? 

I suspect that Waymo faces the problem of trying to do far too much essentially by hand, just adding incremental fix after fix after fix as problems arise. The ideal would be to, instead, apply machine learning techniques that can learn from data and generalize to new scenarios and new driving conditions. Unfortunately, current machine learning techniques do not seem to be up to that task. 

The 2023 LessWrong survey was median 2040 for singularity, and 2030 for "By what year do you think AI will be able to do intellectual tasks that expert humans currently do?". The second question was ambiguous, and some people put it in the past. 

Thank you. Well, that isn't surprising at all.

1. ^{^}

   Okay, well maybe the play testers and the game developers have developed the skill before me, but then at some point one of them had to be the first person in history to ever acquire the skill of playing that game.

I think the uncertainty is often just irreducible. Someone faces the choice of either becoming an oncologist who treats patients or a cancer researcher. They don't know which option has higher expected value because they don't know the relevant probabilities. And there is no way out of that uncertainty, so they have to make a choice with the information they have. 

Very good thought experiment. The point is correct. The problem is that in real life we almost never know the probability of anything. So, it will almost never happen that someone knows a long shot bet has 10x better expected value than a sure thing. What will happen in almost every case is that a person faces irreducible uncertainty and takes a bet. That’s life and it ain’t such a bad gig. 

Can you clarify what you mean by this? I don't understand what you mean by "continuous altruistic expenditure", "each cycle transfers away control to others", or "a movement permanently spending itself down". What do these phrases mean? 

I think this post needs to be re-stated in plain English, in words that like an average high school student who has never heard of effective altruism could understand.

A good rule of thumb I try to follow is to try to write so that what I'm saying could be understood by a broader and less knowledgeable audience than the main audience I actually have in mind when I'm writing. So, if I'm writing a post that I think will mainly read by people who read a lot of posts on the EA Forum, I'll try to write it so that if it's the first post someone has ever read on the EA Forum, and they're not very familiar with EA, that person will have a better chance of understanding it too.

This has two benefits: 1) it broadens the audience who can understand what I write and 2) it makes my writing clearer and easier to understand for everyone, including my main audience of frequent EA Forum users.

Hope that all makes sense, and helps. 

Hi Soe Lin. This post is very sweet. But I think this part is wrong:

if all of humanity will change their personal/religious/political/philosophical views and identities to post-humanist views, many of the world’s current problems will be solved immediately

I don’t think transhumanist or posthumanist philosophy would solve anything immediately. This philosophy would only help at all insofar as this philosophy promoted long-term investment in scientific and technological research and development. 

I find this post incisive overall. I love this paragraph in particular:

What even is a worldview? It is the set of beliefs about fundamental aspects of Reality that ground and influence all one's perceiving, thinking, knowing, and doing (source). For example, a social justice worldview primarily sees the world as dynamics between the oppressed and the oppressor. In any worldview, there is a central organizing theme on what the world is “about” — everything else is just commentary.

I really like the way you said this. I think "a central organizing theme on what the world is ‘about’" is a clearer articulation than previous formulations I’ve heard. I find this insight.

I don’t know if this is a good definition of worldviews in general. For example, maybe someone’s worldview is that the world is full of diverse and heterogeneous problems and conflicts, each of which requires their own special attention and thinking. That seems like it could be a worldview but there wouldn’t be one thing the whole world is about. I think maybe there should be a term specifically worldviews that are characterized by reliance on a central organizing theme. 

A related concept is the sociologist James Hughes’ treatment of millennialism. He has a great talk about that called "Avoiding Millennialist Cognitive Biases". (If you prefer to read, he has a paper called "Millennial Tendencies in Response to Apocalyptic Threats" that was published in the essay anthology Global Catastrophic Risks.) Millennialist beliefs can be totalizing — they can make the world "about" some coming utopia or apocalypse, or utopia that will follow an apocalypse. But millennialist beliefs don’t encompass all totalizing worldviews. For example, someone could have a Christian worldview that is totalizing but not believe in the rapture or the end times or anything like that. Or someone could have a social justice worldview that is fairly totalizing but doesn’t involve a utopia or an apocalypse.

The superintelligence worldview is definitely millennialist. The advent of superintelligence is presented as bringing either apocalypse and utopia. 

Identifying a belief or a worldview as millennialist isn’t a refutation of it. It is merely a classification. But that classification that should draw concern because of the psychological biases associated with millennialism. It warrants caution and scrutiny. It’s similar to any situation where you identify bias — detecting bias isn’t a refutation, it’s just a reason for skepticism.

I guess there could have recently been a major breakthrough in RL at any of the major AI companies that the public doesn’t know about yet. Or there could be one soon that we wouldn’t know about right away. But why think that is the case? And why think that is more like at this particular point in time than any other time within the last 10 years or so?

Can you explain what "LLMs as reward models" and "verifiable domains for self-play" mean and why these would make RL dramatically more compute efficient? I’m guessing that "LLMs as reward models" means that the representational power of LLMs is far greater than for RL agents in the past. But hasn’t RLHF been used on LLMs since before the first version of ChatGPT? So wouldn’t our idea of how quickly LLMs learn or improve using RL from the past 3 years or so already account for LLMs as world models?

By "verifiable domains for self-play", do you we have benchmarks or environments that automatically gradable and can provide a reward signal without a human manually taking any action? If so, again, that seems like something that should already be accounted for in the last 3 years or so of data.

If what you’re saying is that LLMs as reward models or verifiable domains for self-play could contribute to research or innovation in RL such that a major breakthrough in RL compute efficiency is more likely, I don’t follow the reasoning there.

You also mentioned "unprecedented compute for experiments", which maybe could be a factor that will contribute to the likelihood of such a breakthrough, but who knows. Why couldn’t you test an idea for more compute efficient RL on a small number of GPUs first and see if you get early results? Why would having a lot more GPUs help? With a lot of GPUs, you could test more ideas in parallel, but is the limiting factor really the ability to test ideas or is it coming up with new ideas in the first place? 

I’ll just mention that, for what it’s worth, the AI researcher and former OpenAI Chief Scientist thinks the scaling of pre-training for LLMs has run out of steam. Dario Amodei, the CEO of Anthropic, has also said things that seem to indicate the scaling of pre-training no longer has the importance it once did. 

Other evidence would be reporters talking to anonymous engineers inside OpenAI and Meta who have expressed disappointment with the results of scaling pre-training. Toby mentioned this in another blog post and I quoted the relevant paragraph in a comment here.