Announcing: 2026 MIRI Technical Governance Team Research Fellowship.
MIRI’s Technical Governance Team plans to run a small research fellowship program in early 2026. The program will run for 8 weeks, and include a $1200/week stipend. Fellows are expected to work on their projects 40 hours per week. The program is remote-by-default, with an in-person kickoff week in Berkeley, CA (flights and housing provided). Participants who already live in or near Berkeley are free to use our office for the duration of the program.
Fellows will spend the first week picking out scoped projects from a list provided by our team or designing independent research projects (related to our overall agenda), and then spend seven weeks working on that project under the guidance of our Technical Governance Team. One of the main goals of the program is to identify full-time hires for the team.
If you are interested in participating, please fill out this application as soon as possible (should take 45-60 minutes). We plan to setdates for participation based on applicant availability, but we expect the fellowship to begin after February 2, 2026 and end before August 31, 2026 (i.e., some 8 week period in spring/summer, 2026).
Strong applicants care deeply about existential risk, have existing experience in research or policy work, and are able to work autonomously for long stretches on topics that merge considerations from the technical and political worlds.
Unfortunately, we are not able to sponsor visas for this program.
Just to clarify, is the 8-week period for all participants? And if so, will you still accept some applications after the date has be decided?
I might apply but I could only participate if the program was organized in July-August. But given that it could occur any time between February and August, I probably won't apply since it's only like 1/7th chance it will start in July.
We may be running multiple smaller cohorts rather than one big one, if that's what maximizes the ability of strong candidates to participate.
The single most important factor in deciding the timing is the window in which strong candidates are available, and the target size for the cohort is small enough (5-20 depending on strength of applicants) that the availability of a single applicant is enough to sway the decision. It's specifically cases like yours that we're intending to accommodate. Please apply!
I have a bunch of disagreements with Good Ventures and how they are allocating their funds, but also Dustin and Cari are plausibly the best people who ever lived.
I want to agree, but "best people who ever lived" is a ridiculously high bar! I'd imagine that both of them would be hesitant to claim anything quite that high.
Yeah, sorry: it was obvious to me that this was the intended meaning, after I realized it could be interpreted this way. I noted it because I found the syntactic ambiguity mildly interesting/amusing.
For example, Norman Borlaug is often called "the father of the Green Revolution", and is credited with saving a billion people worldwide from starving to death. Stanislav Petrov and Vasily Arkhipov prevented a probable nuclear war from happening.
I made an AI generated podcast of the 2021 MIRI Conversations. There are different voices for the different participants, to make it easier and more natural to follow along with.
This was done entirely in my personal capacity, and not as part of my job at MIRI.[1] I did this because I like listening to audio and there wasn’t a good audio version of the conversations.
Imagine a world populated by many, many (trillions) of people. These people's lives aren't purely full of joy, and do have a lot of misery as well. But each person thinks that their life is worth living. Their lives might be a be bit boring or they might be full of huge ups and downs, but on the whole they are net-positive.
From this view it seems really strange to think that it would be good for every person in this world to die/not exist/never have existed in order to allow a very small number of privileged people to live spectacular lives. It seems bad to stop many people from living a life that they mostly enjoy, in order to allow the flourishing of the few.
I think this hypothetical is a decent intuition pump for why the Repugnant Conclusion isn't actually repugnant. But I do think it might be a little bit dishonest or manipulative. It frames the situation in terms of fairness and equality; we can sympathize with the many slightly happy people who are maybe being denied the right to exist, and think of the few extremely happy people as the privileged elite. It also takes advantage of status quo bias; by beginning with the many slightly happy people it seems worse to then 'remove' them.
I've always thought the Repugnant Conclusion was mostly status quo bias, anyway, combined with the difficulty of imagining what such a future would actually be like.
I think the Utility Monster is a similar issue. Maybe it would be possible to create something with a much richer experience set than humans, which should be valued more highly. But any such being would actually be pretty awesome, so we shouldn't resent giving it a greater share of resources.
Humans seem like (plausible) utility monsters compared to ants, and many religious people have a conception of god that would make Him a utility monster ("maybe you don't like prayer and following all these rules, but you can't even conceive of the - 'joy' doesn't even do it justice - how much grander it is to god if we follow these rules than even the best experiences in our whole lives!"). Anti-utility monster sentiments seem to largely be coming from a place where someone imagines a human that's pretty happy by human standards, and thinks the words "orders of magnitude happier than what any human feels", and then they notice their intuition doesn't track the words "orders of magnitude".
Imagine there is a box with a ball inside it, and you believe the ball is red. But you also believe that in the future you will update your belief and think that the ball is blue (the ball is a normal, non-color-changing ball). This seems like a very strange position to be in, and you should just believe that the ball is blue now.
This is an example of how we should deal with beliefs in general; if you think in the future you will update a belief in a specific direction then you should just update now.
I think the same principle applies to moral beliefs. If you think that in the future you'll believe that it's wrong to do something, then you should believe that it's wrong now.
As an example of this, if you think that in the future you'll believe eating meat is wrong, then you sort of already believe eating meat is wrong. I was in exactly this position for a while, thinking in the future I would stop eating meat, while also continuing to eat meat. A similar case to this is deliberately remaining ignorant about something because learning would change your moral beliefs. If you're avoiding learning about factory farming because you think it would cause you to believe eating factory farmed meat is bad, then you already on some level believe that.
Another case of this is in politics when a politician says it's 'not the time' for some political action but in the future it will be. This is 'fine' if it's 'not the time' due to political reasons, such as the electorate not reelecting the politician. But I don't think it's consistent to say an action is currently not moral, but will be moral in the future. Obviously this only works if the action now and in the future are actually equivalent.
Announcing: 2026 MIRI Technical Governance Team Research Fellowship.
MIRI’s Technical Governance Team plans to run a small research fellowship program in early 2026. The program will run for 8 weeks, and include a $1200/week stipend. Fellows are expected to work on their projects 40 hours per week. The program is remote-by-default, with an in-person kickoff week in Berkeley, CA (flights and housing provided). Participants who already live in or near Berkeley are free to use our office for the duration of the program.
Fellows will spend the first week picking out scoped projects from a list provided by our team or designing independent research projects (related to our overall agenda), and then spend seven weeks working on that project under the guidance of our Technical Governance Team. One of the main goals of the program is to identify full-time hires for the team.
If you are interested in participating, please fill out this application as soon as possible (should take 45-60 minutes). We plan to set dates for participation based on applicant availability, but we expect the fellowship to begin after February 2, 2026 and end before August 31, 2026 (i.e., some 8 week period in spring/summer, 2026).
Strong applicants care deeply about existential risk, have existing experience in research or policy work, and are able to work autonomously for long stretches on topics that merge considerations from the technical and political worlds.
Unfortunately, we are not able to sponsor visas for this program.
See here for examples of potential projects
Just to clarify, is the 8-week period for all participants? And if so, will you still accept some applications after the date has be decided?
I might apply but I could only participate if the program was organized in July-August. But given that it could occur any time between February and August, I probably won't apply since it's only like 1/7th chance it will start in July.
We may be running multiple smaller cohorts rather than one big one, if that's what maximizes the ability of strong candidates to participate.
The single most important factor in deciding the timing is the window in which strong candidates are available, and the target size for the cohort is small enough (5-20 depending on strength of applicants) that the availability of a single applicant is enough to sway the decision. It's specifically cases like yours that we're intending to accommodate. Please apply!
I have a bunch of disagreements with Good Ventures and how they are allocating their funds, but also Dustin and Cari are plausibly the best people who ever lived.
I want to agree, but "best people who ever lived" is a ridiculously high bar! I'd imagine that both of them would be hesitant to claim anything quite that high.
“Plausibly best people who have ever lived” is a much lower bar than “best people who have ever lived”.
If you are like me, this comment will leave you perplexed. After a while, I realized that it should not be read as
but as
fwiw i instinctively read it as the 2nd, which i think is caleb's intended reading
I was going for the second, adding some quotes to make it clearer.
Yeah, sorry: it was obvious to me that this was the intended meaning, after I realized it could be interpreted this way. I noted it because I found the syntactic ambiguity mildly interesting/amusing.
For example, Norman Borlaug is often called "the father of the Green Revolution", and is credited with saving a billion people worldwide from starving to death. Stanislav Petrov and Vasily Arkhipov prevented a probable nuclear war from happening.
It's true how many people actually give away so much money as they make it?
AI Generated Podcast for 2021 MIRI Conversations.
I made an AI generated podcast of the 2021 MIRI Conversations. There are different voices for the different participants, to make it easier and more natural to follow along with.
This was done entirely in my personal capacity, and not as part of my job at MIRI.[1] I did this because I like listening to audio and there wasn’t a good audio version of the conversations.
Spotify link: https://open.spotify.com/show/6I0YbfFQJUv0IX6EYD1tPe
RRS: https://anchor.fm/s/1082f3c7c/podcast/rss
Apple Podcasts: https://podcasts.apple.com/us/podcast/2021-miri-conversations/id1838863198
Pocket Casts: https://pca.st/biravt3t
I do think you probably should (pre-)order If Anyone Builds It, Everyone Dies though.
Flipping the Repugnant Conclusion
Imagine a world populated by many, many (trillions) of people. These people's lives aren't purely full of joy, and do have a lot of misery as well. But each person thinks that their life is worth living. Their lives might be a be bit boring or they might be full of huge ups and downs, but on the whole they are net-positive.
From this view it seems really strange to think that it would be good for every person in this world to die/not exist/never have existed in order to allow a very small number of privileged people to live spectacular lives. It seems bad to stop many people from living a life that they mostly enjoy, in order to allow the flourishing of the few.
I think this hypothetical is a decent intuition pump for why the Repugnant Conclusion isn't actually repugnant. But I do think it might be a little bit dishonest or manipulative. It frames the situation in terms of fairness and equality; we can sympathize with the many slightly happy people who are maybe being denied the right to exist, and think of the few extremely happy people as the privileged elite. It also takes advantage of status quo bias; by beginning with the many slightly happy people it seems worse to then 'remove' them.
I've always thought the Repugnant Conclusion was mostly status quo bias, anyway, combined with the difficulty of imagining what such a future would actually be like.
I think the Utility Monster is a similar issue. Maybe it would be possible to create something with a much richer experience set than humans, which should be valued more highly. But any such being would actually be pretty awesome, so we shouldn't resent giving it a greater share of resources.
Humans seem like (plausible) utility monsters compared to ants, and many religious people have a conception of god that would make Him a utility monster ("maybe you don't like prayer and following all these rules, but you can't even conceive of the - 'joy' doesn't even do it justice - how much grander it is to god if we follow these rules than even the best experiences in our whole lives!"). Anti-utility monster sentiments seem to largely be coming from a place where someone imagines a human that's pretty happy by human standards, and thinks the words "orders of magnitude happier than what any human feels", and then they notice their intuition doesn't track the words "orders of magnitude".
I like this perspective. I've never really understood why people find the repugnant conclusion repugnant!
Updating Moral Beliefs
Imagine there is a box with a ball inside it, and you believe the ball is red. But you also believe that in the future you will update your belief and think that the ball is blue (the ball is a normal, non-color-changing ball). This seems like a very strange position to be in, and you should just believe that the ball is blue now.
This is an example of how we should deal with beliefs in general; if you think in the future you will update a belief in a specific direction then you should just update now.
I think the same principle applies to moral beliefs. If you think that in the future you'll believe that it's wrong to do something, then you should believe that it's wrong now.
As an example of this, if you think that in the future you'll believe eating meat is wrong, then you sort of already believe eating meat is wrong. I was in exactly this position for a while, thinking in the future I would stop eating meat, while also continuing to eat meat. A similar case to this is deliberately remaining ignorant about something because learning would change your moral beliefs. If you're avoiding learning about factory farming because you think it would cause you to believe eating factory farmed meat is bad, then you already on some level believe that.
Another case of this is in politics when a politician says it's 'not the time' for some political action but in the future it will be. This is 'fine' if it's 'not the time' due to political reasons, such as the electorate not reelecting the politician. But I don't think it's consistent to say an action is currently not moral, but will be moral in the future. Obviously this only works if the action now and in the future are actually equivalent.