I couldn't find any reliable information for Waymo, but I found a 2023 article where a Cruise spokesperson said there was one human doing remote assistance for every 15 to 20 autonomous vehicles. That article cites a New York Times article that says a human intervention was needed every 2.5 to 5 miles.

That's very helpful! I'm guessing that since 2023, a person could manage more vehicles than that, and it will continue to improve. But let's just work with that number. I don't think the pay for a person solving problems for Waymos would need to be that much higher than taxi drivers but even if it is 1.5x or 2x, that would still be an order of magnitude reduction in labor cost. Of course there is additional hardware cost for autonomous vehicles, but I think that can be paid off with a reasonable duty cycle. So then if you grant that the geofencing is less than a 20x safety advantage, I think there is an economic case for the chimeras, as you say.

I don't understand the details of the coding, math, or question and answer (Q&A or QA) benchmarks, but is the "time" dimension of these not just LLMs producing larger outputs, i.e., using more tokens? And, if so, why would LLMs performing tasks that use more tokens, although it may well be a sign of LLM improvement, indicate anything about AGI progress?

My understanding is that 50% success on a 1 hour task means the human expert takes ~1 hour, but the computer time is not specified.

As for AGI, I do think that some people conflate the 50% success time horizon at coding tasks reaching ~one month as AGI, when it really only means superhuman coder (I think something like 50% success rate is appropriate for long coding tasks, because humans rarely produce code that works on the first try). However, once you get to superhuman coder, the logic goes that the progress in AI research will dramatically accelerate. In AI 2027, I think there was some assumption like it would take 90 years at current progress to get something like AGI, but when things accelerate, it ends up happening in a year or two. Another worry is that you might not need AGI to have something catastrophic, because it could become superhuman in a limited number of tasks, like hacking.

Incidentally, how good is o3 (or any other LLM) at chess, shogi, go, and Atari games? Or at StarCraft? If we're making progress toward artificial general intelligence, shouldn't one system be able to do all of these things? 

You're right it's not fully general, but AI systems are much more general than they were 7 years ago.

I think EA is in this weird, crazy filter bubble/echo chamber when it comes to AGI where if there's any expert consensus on AGI, it's against the assertions many people in EA make about AGI.

LessWrong is also known for having very short timelines, but I think the last survey indicated median time for AGI was 2040. So I do think that it is a vocal minority in EA and LW that have median timelines before 2030. However, I do agree with them that we should be spending significant effort on AI safety because it's plausible that AGI comes in the next five years. 

Though it's true that the expert surveys you cite have much longer timelines, I would guess that if you took a poll of people in the AI labs (which I think would qualify as a group of experts - biased though they may be, they do have inside information), their median timelines would be before 2030.

So, if Waymos are prone to suddenly breaking due to harmless obstacles like plastic bags or simply because of mistakes made by the perception system, then they could get rear-ended a lot, but rear-ending is always the fault of the car in the rear as a matter of law.

Interesting, but AI says:

"Exceptions exist where the lead driver may share or bear full fault, such as if they:

• Stop suddenly without a valid reason
• Reverse unexpectedly
• Have malfunctioning brake lights or turn signals
• Make unsafe lane changes without signaling
• Were driving negligently (e.g., distracted driving)"

So if plastic bags are not a valid reason to stop, it sounds like the Waymo would be at fault for the rear-end accident.

I agree other company failures are evidence for your point. I think Waymo is trying to scale up, and they are limited by cars at this point. 

It sounds like Tesla solves the problem of unreliability by having the driver take over. And Waymo solves it by the car stopping and someone remotely taking over. So if that has to happen every 464 miles (say 10 hours) and it takes the remote operator five minutes to deal with (probably less but there would be downtime), then that means you would only need one remote operator for 120 cars. I think that's pretty scalable. It would not be as safe when it can drive anywhere as Waymo's claim of roughly 20 times as safe (it's five times fewer accidents but then the remaining accidents were mostly the other drivers' faults), but I still think it would be safer than the average human driver.

On a final note, I'm bothered by METR's focus on tasks with a 50% success rate. I mean, it's fine to track whatever you want, but I disagree with how people interpret what this means for overall AI progress. Humans do many, many tasks with a 99.9%+ success rate. Driving is the perfect example. If you benchmarked AI progress against a 99.9% success rate, I imagine that graph would just into one big, flat line on the bottom at 0.0, and what story would that tell about AI progress?

I agree that 50% is not realistic for many tasks. But they do plot some data for higher percent success:

Roughly I think going from 50% to 99.9% would be 2 hours to 4 seconds, not quite 0, but very bad!

• Disruption of global synchronization and navigation: Global navigation satellite systems (things like GPS) could be disrupted for several days. This would be annoying for navigation in general, but more importantly, we also use it for global synchronization for all kinds of things, like our financial system. This would not be possible during the storm.

 

I thought some satellites would get their cumulative dose in a bad solar storm, so then they would be permanently damaged (though that might just be with EMP particles).

• Aviation problems: Airplanes might be exposed to higher radiation doses than normal. This would not have any immediate effects, but would condemn a random collection of people on the airplanes to cancer later in their life.

Have they thought about flying lower in an event?

Thanks-I appreciate the skeptical take. I have updated, but not all the way to thinking self driving is less safe than humans. Another thing I've been concerned about is that the self driving cars I assume would be relatively new, so they would have fewer mechanical problems than the average car. But mechanical problems make up a very small percent of total accidents, so that wouldn't change things much. I agree that if there is a lot of babysitting required, then it would not be scalable. But I'm estimating that even without the babysitting and geofencing, it would still be safer than humans with all the drunk/sleepy/angry/teen/over 80 drivers.
I agree this does have some lessons for AGI. Though interestingly the progress on self driving cars is the slowest of all the areas measured in the graph below:

Waymo’s self-driving cars are geofenced down to the street level, e.g., they can drive on this street but not on that street right next to it. Waymo carves up cities into the places where it’s easiest for its cars to drive and where the risk is lowest if they make a mistake. 

Interesting. But if the vast majority of the serious accidents that did occur were the fault of other drivers, wouldn't that be strong evidence that the Waymos were safer (apples to apples at least for those streets)?

Zvi has criticized both Amodei's and Cowen's claims.

Self-driving cars are geofenced to the safest areas (and the human drivers who make up the collision statistics they are compared against are not). 

I wouldn't say that the cities Waymo are in are the safest areas. But I was critical of the claim that Teslas were ~4x safer a few years back because they were only driving autonomously on highways (which are much safer). What's the best apples-to-apples comparison you have seen?

I think there is essentially no chance that self-driving cars will substitute for human drivers on a large scale within the next 5 years. By a large scale, I mean scaling up to the size of something like 1% or 10% of the overall U.S. vehicle fleet.

Metalculus says Dec 31 for half of new cars sold being autonomous, which would pretty much guarantee 1% of US fleet in 5 years, so you might find some people there to bet...

My understanding is that in the US and the UK, employers do not have to hire the most qualified, and they can take into account cost-effectiveness. It is hard to get an EA job, so I think it does make sense to indicate you are willing to work for less, but maybe keep it qualitative. Or only say 20% less and then just donate the rest. Then that should cause fewer of the potential problems you and others note.

That's interesting that people may still be kind of in the loop through teleoperations centers, but I doubt it would be one to one, and the fact that Waymo is 5x safer (actually much safer than that because almost all the serious accidents were caused by humans, not the Waymos) is mostly due to the tech.

self-driving cars remain at roughly the same level of deployment as 5 years ago or even 10 years ago, nowhere near substituting for human drivers on a large scale

The first part is false, e.g. "Driverless taxi usage in California has grown eight fold in just one year". The second part is true, but the fact that self-driving cars are safer and cheaper (once they pay off the R&D) means that they most likely will soon substitute for human drivers on a large scale absent banning.

With dismay, I have to conclude that the bulk of EA is in Camp A "Race to superintelligence "safely"”. I'd love to see some examples where this isn't the case (please share in the comments), but at least the vast majority of money, power and influence in EA seems to be on the wrong side of history here.

I made a poll to get at what the typical EA Forum user thinks, though that is not necessarily representative of the money, power, or influence. My takeaway was:

"For the big picture, it received 39 votes. 13% want AGI never to be built, 26% said to pause AI now in some form, and another 21% would like to pause AI if there is a particular event/threshold. 31% want some other regulation, 5% are neutral and 5% want to accelerate AI in a safe US lab. So if I had to summarize the median respondent, it would be strong regulation for AI or pause if a particular event/threshold is met. There appears to be more evidence for the claim that EA wants AI to be paused/stopped than for the claim that EA wants AI to be accelerated."