Why aren't there more people like him, and what is he doing or planning to do about that?

Maybe if/how his thinking about AI governance has changed over the last year?

Nice to know, Rob! I have really liked the podcasts Carl did. You may want to link to Carl's (great!) blog in your post too.

In general, I would be curious to know more about how Carl thinks about determining how much resources should go into each cause area, which I do not recall being discussed much in Carl's 3 podcasts. Some potential segways:

• Open Phil Should Allocate Most Neartermist Funding to Animal Welfare. Carl shared in the comments his thoughts on Rethink Priorities' moral weight project.
• Rethink Priorities' CURVE sequence and cross-cause cost-effectiveness model.
• How would Carl allocate Open Philanthropy's funding? I am not sure how easy it would be to discuss this given Carl has been advising Open Phil, but I like your policy of letting guests decide which questions to answer.
• Which areas are under or overrated, and why.

Carl has knowledge about lots of topics, very much like Anders Sandberg. So I think the questions I shared to ask Anders are also good questions for Carl:

• Should more resources be directed towards patient philanthropy at the margin? How much more/less?
• How binary is longterm value? Relevant to the importance of the concept of existential risk.
• Should the idea that more global warming might be good to mitigate the food shocks caused by abrupt sunlight reduction scenarios (ASRSs) be taken seriously? (Anders is at the board of ALLFED, and therefore has knowledge about ASRSs.)
• Which fraction of the expected effects of neartermist interventions (e.g. global health and development, and animal welfare) flow through longtermis considerations (e.g. longterm effects of changing population size, or expansion of the moral circle)? [Is it unclear whether Against Malaria Foundation is better/worse than Make-A-Wish Foundation, as argued in section 4.1 of Maximal cluelessness?]
• Under moral realism, are we confident that superintelligent artificial intelligence disempowering humans would be bad?
• Should we be uncertain about whether saving lives is good/bad because of the meat eater problem [my related calculations; related post]?
• What is the chance that the time of perils hypothesis is true (e.g. how does the existential risk this century compare to that over the next 1 billion years)? How can we get more evidence for/against it? Relevant because, if existential risk is spread out over a long time, reducing existential risk this century has a negligible effect on total existential risk, as discussed by David Thorstad. [See also Rethink's post on this question.]
• How high is the chance of AGI lock-in this century?
• What can we do to ensure a bright future if there are advanced aliens on or around Earth (Magnus Vinding's thoughts)? More broadly, should humanity do anything differently due to the possibility of advanced civilisations which did not originite on Earth? [Another speculative question, which you covered a little in the podcast with Joe, is what should we do differently to improve the world if we were in a simulation?]
• How much weight should one give to the XPT's forecasts? The ones regarding nuclear extinction seem way too pessimistic to be accurate [the reasons I think this are in this thread]. Superforecasters and domain experters predicted a likelihood of nuclear extinction by 2100 of 0.074 % and 0.55 %. My guess would be something like 10^-6 (10 % of a global nuclear nuclear war involving tens of detonations, 10 % of it escalating to thousands of detonations, and 0.01 % of that leading to extinction), in which case superforecasters would be off by 3 orders of magnitude.

Importance of the digital minds stuff compared to regular AI safety; how many early-career EAs should be going into this niche? What needs to happen between now and the arrival of digital minds? In other words, what kind of a plan does Carl have in mind for making the arrival go well? Also, since Carl clearly has well-developed takes on moral status, what criteria he thinks could determine whether an AI system deserves moral status, and to what extent.

Additionally—and this one's fueled more by personal curiosity than by impact—Carl's beliefs on consciousness. Like Wei Dai, I find the case for anti-realism as the answer to the problem of consciousness weak, yet this is Carl's position (according to this old Brian Tomasik post, at least), and so I'd be very interested to hear Carl explain his view.

I'd like to hear his advice for smart undergrads who want to build their own similarly deep models in important areas which haven't been thought about very much e.g. take-off speeds, the influence of pre-AGI systems on the economy, the moral value of insects, preparing for digital minds (ideally including specific exercises/topics/reading/etc.).

I'm particularly interested in how he formed good economic intuitions, as they seem to come up a lot in his thinking/writing.

IIRC Carl had a $5M discretionary funding pot from OpenPhil. What has he funded with it?

My understanding is that he believes that full non-indexical conditioning has solved many (most? all?) problems in anthropics. It might be interesting to hear his views on what has been solved, and what is remaining.

Can you ask him whether or not it's rational to assume AGI comes with significant existential risk as a default position, or if one has to make a technical case for it coming with x-risk?