This is a special post for quick takes by Chris Leong. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Looks like Mechanize is choosing to be even more irresponsible than we previously thought. They're going straight for automating software engineering. Would love to hear their explanation for this.
"Software engineering automation isn't going fast enough"[1] - oh really?
This seems even less defensible than their previous explanation of how their work would benefit the world.
Some useful context is that I think a software singularity is unlikely to occur; see this blog post for some arguments. Loosely speaking, under the view expressed in the linked blog post, there aren't extremely large gains from automating software engineering tasks beyond the fact that these tasks represent a significant (and growing) fraction of white collar labor by wage bill.
Even if I thought a software singularity will likely happen in the future, I don't think this type of work would be bad in expectation, as I continue to think that accelerating AI is likely good for the world. My main argument is that speeding up AI development will hasten large medical, technological, and economic benefits to people alive today, without predictably causing long-term harms large enough to outweigh these clear benefits. For anyone curious about my views, I've explained my perspective on this issue at length on this forum and elsewhere.
Note: Matthew's comment was negative just now. Please don't vote it into the negative and use the disagree button instead. Even though I don't think Matthew's defense is persuasive, it deserves to be heard.
I wrote a critique of that article here. TLDR: "It has some strong analysis at points, but unfortunately, it's undermined by some poor choices of framing/focus that mean most readers will probably leave more confused than when they came".
"A software singularity is unlikely to occur" - Unlikely enough that you're willing to bet the house on it? Feels like you're picking up pennies in front of a steamroller.
AI is already going incredibly fast. Why would you want to throw more fuel on the fire?
Is it that you honestly think AI is moving too slow at the moment (no offense, but seems crazy to me) or is your worry that current trends are misleading and AI might slow in the future?
Regarding the latter, I agree that once timelines start to get sufficiently long, there might actually be an argument for accelerating them (but in order to reach AGI before biotech causes a catastrophe, rather than the more myopic reasons you've provided). But if your worry is stagnation, why not actually wait until things appear to have stalled and then perhaps consider doing something like this?
Or why didn't you just stay at Epoch, which was a much more robust and less fragile theory of action? (Okay, I don't actually think articles like this are high enough quality to be net-positive, but you were 90% of the way towards having written a really good article. The framing/argument just needed to be a little bit tighter, which could have been achieved with another round of revisions).
The main reason not to wait is... missing the opportunity to cash in on the current AI boom.
I bet the strategic analysis for Mechanize being a good choice (net-positive and positive relative to alternatives) is paper-thin, even given his rough world view.
6
Ryan Greenblatt
Might be true, doesn't make that not a strawman. I'm sympathetic to thinking it's implausible that mechanize would be the best thing to do on altruistic grounds even if you share views like those of the founders. (Because there is probably something more leveraged to do and some weight on cooperativeness considerations.)
Sometimes the dollar signs can blind someone and cause them not to consider obvious alternatives. And they will feel that they made the decision for reasons other than the money, but the money nonetheless caused the cognitive distortion that ultimately led to the decision.
I'm not claiming that this happened here. I don't have any way of really knowing. But it's certainly suspicious. And I don't think anything is gained by pretending that it's not.
I guess orgs need to be more careful about who they hire as forecasting/evals researchers in light of a recently announced startup.
Sometimes things happen, but three people at the same org...
This is also a massive burning of the commons. It is valuable for forecasting/evals orgs to be able to hire people with a diversity of viewpoints in order to counter bias. It is valuable for folks to be able to share information freely with folks at such forecasting orgs without having to worry about them going off and doing something like this.
However, this only works if those less worried about AI risks who join such a collaboration don't use the knowledge they gain to cash in on the AI boom in an acceleratory way. Doing so undermines the very point of such a project, namely, to try to make AI go well. Doing so is incredibly damaging to trust within the community.
Now let's suppose you're an x-risk funder considering whether to fund their previous org. This org does really high-quality work, but the argument for them being net-positive is now significantly weaker. This is quite likely to make finding future funding harder for them.
This is less about attacking those three folks and more just no... (read more)
I'd like to suggest a little bit more clarity here. The phrases you use refer to some knowledge that isn't explicitly stated here. "in light of a recently announced startup" and "three people at the same org" make sense to someone who already knows the context of what you are writing about, but it is confusing to a reader who doesn't have the same background knowledge that you do.
Once upon a time, some people were arguing that AI might kill everyone, and EA resources should address that problem instead of fighting Malaria.
So OpenPhil poured millions of dollars into orgs such as EpochAI (they got 9 million).
Now 3 people from EpochAI created a startup to provide training data to help AI replace human workers.
Some people are worried that this startup increases AI capabilities, and therefore increases the chance that AI will kill everyone.
100 percent agree. I dont understand the entire post because I don't know the context. I don't think alluding to something helps, better to say it explicitly.
7
Henry Stanley 🔸
I tend to agree; better to be explicit especially as the information is public knowledge anyway.
It refers to this: https://forum.effectivealtruism.org/posts/HqKnreqC3EFF9YcEs/
Also, it is worrying if the optimists easily find financial opportunities that depend on them not changing their minds. Even if they are honest and have the best of intentions, the disparity in returns to optimism is epistemically toxic.
I agree that we need to be careful about who we are empowering.
"Value alignment" is one of those terms which has different meanings to different people. For example, the top hit I got on Google for "effective altruism value alignment" was a ConcernedEAs post which may not reflect what you mean by the term. Without knowing exactly what you mean, I'd hazard a guess that some facets of value alignment are pretty relevant to mitigating this kind of risk, and other facets are not so important. Moreover, I think some of the key factors are less cognitive or philosophical than emotional or motivational (e.g., a strong attraction toward money will increase the risk of defecting, a lack of self-awareness increases the risk of motivated reasoning toward goals one has in a sense repressed).
So, I think it would be helpful for orgs to consider what elements of "value alignment" are of particular importance here, as well as what other risk or protective factors might exist outside of value alignment, and focus on those specific things.
3
Chris Leong
Agreed. "Value alignment" is a simplified framing.
5
Chris Leong
Short update - TLDR - mechanise is going straight for automating software engineering.
-1
Yarrow Bouchard🔸
If you only hire people who you believe are intellectually committed to short AGI timelines (and who won’t change their minds given exposure to new evidence and analysis) to work in AGI forecasting, how can you do good AGI forecasting?
One of the co-founders of Mechanize, who formerly worked at Epoch AI, says he thinks AGI is 30 to 40 years away. That was in this video from a few weeks ago on Epoch AI’s YouTube channel.
He and one of his co-founders at Mechanize were recently on Dwarkesh Patel’s podcast (note: Dwarkesh Patel is an investor in Mechanize) and I didn’t watch all of it but it seemed like they were both arguing for longer AGI timelines than Dwarkesh believes in.
I also disagree with the shortest AGI timelines and found it refreshing that within the bubble of people who are fixated on near-term AGI, at least a few people expressed a different view.
I think if you restrict who you hire to do AGI forecasting based on strong agreement with a predetermined set of views, such as short AGI timelines and views on AGI alignment and safety, then you will just produce forecasts that re-state the views you already decided were the correct ones while you were hiring.
0
Chris Leong
I wasn't suggesting only hiring people who believe in short-timelines. I believe that my original post adequately lays out my position, but if any points are ambiguous, feel free to request clarification.
1
Yarrow Bouchard🔸
I don’t know how Epoch AI can both "hire people with a diversity of viewpoints in order to counter bias" and ensure that your former employees won’t try to "cash in on the AI boom in an acceleratory way". These seem like incompatible goals.
I think Epoch has to either:
1. Accept that people have different views and will have different ideas about what actions are ethical, e.g., they may view creating an AI startup focused on automating labour as helpful to the world and benign
or
1. Only hire people who believe in short AGI timelines and high AGI risk and, as a result, bias its forecasts towards those conclusions
Is there a third option?
4
David Mathers🔸
Presumably there are at least some people who have long timelines, but also believe in high risk and don't want to speed things up. Or people who are unsure about timelines, but think risk is high whenever it happens. Or people (like me) who think X-risk is low* and timelines very unclear, but even a very low X-risk is very bad. (By very low, I mean like at least 1 in 1000, not 1 in 1x10^17 or something. I agree it is probably bad to use expected value reasoning with probabilities as low as that.)
I think you are pointing at a real tension though. But maybe try to see it a bit from the point of view of people who think X-risk is real enough and raised enough by acceleration that acceleration is bad. It's hardly going to escape their notice that projects at least somewhat framed as reducing X-risk often end up pushing capabilities forward. They don't have to be raging dogmatists to worry about this happening again, and it's reasonable for them to balance this risk against risks of echo chambers when hiring people or funding projects.
*I'm less surely merely catastrophic biorisk from human misuse is low sadly.
Unfortunately, the EA Communications Fellowship and the EA Blog prize shut down[1]. Any new project needs to be adapted to the new funding environment.
If someone wanted to start something in this vein, what I'd suggest would be something along the lines of AI Safety Camp. People would apply with a project to be project leads and then folk could apply to these projects. Projects would likely run over a few months, part-time remote[2].
Something like this would be relatively cheap as it would be possible for someone to run this on a volunteer basis, but it might also make sense for there to be a paid organiser at a certain point.
For the record, I see the new field of "economics of transformative AI" as overrated.
Economics has some useful frames, but it also tilts people towards being too "normy" on the impacts of AI and it doesn't have a very good track record on advanced AI so far.
I'd much rather see multidisciplinary programs/conferences/research projects, including economics as just one of the perspectives represented, then economics of transformative AI qua economics of transformative AI.
(I'd be more enthusiastic about building economics of transformative AI as a field if we were starting five years ago, but these things take time and it's pretty late in the game now, so I'm less enthusiastic about investing field-building effort here and more enthusiastic about pragmatic projects combining a variety of frames).
(Could you elaborate on ‘economics doesn’t have a very good track record on advanced AI so far’? I haven’t heard this before)
6
Chris Leong
Things in AI have been moving fast, most economists seem to have expected it to have moved slower. Sorry, I don't really want to get into more detail as writing a proper response would end up taking me more time than I want to spend defending this "Quick take".
As an example, I expect political science and international relations to be better for looking at issues related to power distribution rather than economics (though the economic frame adds some value as well). Historical studies of coups seems pretty relevant as well.
When it comes to predicting future progress, I'd be much more interested in hearing the opinions of folks who combine knowledge of economics with knowledge of ML or computer hardware, rather than those who are solely economists. Forecasting seems like another relevant discipline, as is future studies and history of science.
I think "economics of transformative AI" only matters in the narrow slice of worlds (maybe 20% of my probability?) where AI is powerful enough to transform the economy, but not powerful enough to kill everyone or to create a post-scarcity utopia. So I think you're right.
4
Chris Leong
It has some relevance to strategy as well, such as in terms of how fast we develop the tech and how broadly distributed we expect it to be, however there's a limit to how much additional clarity we can expect to gain over short time period.
I’m pretty bullish on having these kinds of debates. While EA is doing well at having an impact in the world, the forum has started to feel intellectually stagnant in some ways. And I guess I feel that these debates provide a way to move the community forward intellectually. That's something I've been feeling has been missing for a while.
Having a space that is intellectually-edgy, but not edge-lord maxing seems extremely valuable. Especially given how controversial some EA ideas were early on (and how controversial wild animal welfare and AI welfare still are).
In fact, I'd go further and suggest that it would be great if they were to set up their own forum. This would allow us to nudge certain discussions into an adjacent, not-explicitly EA space instead of discussing it here.
Certain topics are a poor fit for the forum because they rate high on controversy + low-but-non-zero on relevance to EA. It's frustrating having these discussions on the forum as it may turn some people off, but at the same time declaring these off-topic risks being intellectually stiffling. Sometimes things turn out to be more important than you thought when you dive into the details. So I guess I'd really love to see another non-EA space end up being the first port of call for such discussions, with the hope that only the highest quality and most relevant ideas would make it over to the EA forum.
Although I have mixed feelings on the proposal, I'm voting insightful because I appreciate that you are looking toward an actual solution that at least most "sides" might be willing to live with. That seems more insightful than what the Forum's standard response soon ends up as: rehashing fairly well-worn talking points every time an issue like this comes up.
2[anonymous]
Considering how much skepticism there is in EA about forecasting being a high priority cause area anyway, this seems like an ok idea :)
1
harfe
Manifold already has a highly active discord, where they can discuss all the manifold-specific issues. This did not prevent the EA Forum from discussing the topic, and I doubt it would be much different if Manifold had a proper forum instead of a discord.
It might seem low on importance for EA to you, but I suspect some people who are upset about Manifest inviting right-wing people do not consider it low-importance.
4
Chris Leong
Oh, I wasn't referring to redirecting the discussions about Manifest onto a new forum. More discussions about pro-natalism or genetic engineering to improve welfare. To be clear, I was suggesting a forum associated with Manifest rather than one more narrowly associated with Manifold.
I'd love to see the EA forum add a section titled "Get Involved" or something similar.
There is the groups directory, but it's one of only many ways that folks can get more involved, from EAGx Conferences, to Virtual Programs, 80,000 Hours content/courses to donating.
A list like that could be added to the EA Handbook, which is linked on the forum sidebar
2
Sarah Cheng 🔸
Thanks for the suggestion Chris! I'd be really excited for the Forum (or for EA.org) to have a nice page like that, and I think others at CEA agree. We did a quick experiment in the past by adding the "Take action" sidebar link that goes to the Opportunities to take action topic page, and the link got very few clicks. We try not to add clutter to the site without good reason so we removed that link for logged in users (it's still visible for logged out users since they're more likely to get value from it). Since then we've generally deprioritized it. I would like us to pick it back up at some point, though first we'd need to decide where it should live (EA.org or here) and what it should look like, design-wise.
For now, I recommend people make updates to the Opportunities to take action wiki text to help keep it up-to-date! I've done so myself a couple times but I think it would be better as a team effort. :)
5
Gemma 🔸
Have the forum team considered running an online event to collaborate on improving wikis? I think wikis are a deeply underrated forum feature and a fantastic way for people who aren't new but aren't working in EA to directly contribute to the EA project.
I wrote a quick take a while ago about how it's probably too hard for people to edit wikis atm - I actually can't link to it but here are my quick takes: Gemma Paterson's Quick takes — EA Forum (effectivealtruism.org)
2
Sarah Cheng 🔸
I'm glad that you like the wiki! ^^ I agree that it's a nice way for people in the community to contribute.
I believe no one on the team has focused on the wiki in a while, and I think before we invest time into it we should have a more specific vision for it. But I do like the idea of collaborative wiki editing events, so thanks for the nudge! I'll have a chat with @Toby Tremlett🔹 to see what he thinks. For reference, we do have a Wiki FAQ page, which is a good starting point for people who want to contribute.
About your specific suggestion, thank you for surfacing it and including detailed context — that's quite helpful. I agree that ideally people could contribute to the wiki with lower karma. I'll check if we can lower the minimum at least. Any more substantive changes (like making a "draft" change and getting it approved by someone else) would take more technical work, so I'm not sure when we would prioritize it.
(It looks like your link to a specific quick take did work, but if you think there's a bug then let me know!)
3
Gemma 🔸
Ah glad the link worked. Not sure why it looked like it didn't.
Let me know if you do end up interested in doing an editing event - happy to host an in person coworking session for it in London.
2
Chris Leong
Interesting. I still think it could be valuable even with relatively few clicks. You might only even need someone to click on it once.
4
Sarah Cheng 🔸
Yeah I agree, it does feel like a thing that should exist, like there's some obvious value to it even though I got some evidence that there was low demand for it on the Forum. I think it would be faster to add to EA.org instead so perhaps we should just add a static page there.
I like that we have a list in the wiki, so that people in the EA community can help us keep the info up-to-date by editing it, but practically speaking people don't spend much time doing that.
I'll post some extracts from the commitments made at the Seoul Summit. I can't promise that this will be a particularly good summary, I was originally just writing this for myself, but maybe it's helpful until someone publishes something that's more polished:
Frontier AI Safety Commitments, AI Seoul Summit 2024
The major AI companies have agreed to Frontier AI Safety Commitments. In particular, they will publish a safety framework focused on severe risks: "internal and external red-teaming of frontier AI models and systems for severe and novel threats; to work toward information sharing; to invest in cybersecurity and insider threat safeguards to protect proprietary and unreleased model weights; to incentivize third-party discovery and reporting of issues and vulnerabilities; to develop and deploy mechanisms that enable users to understand if audio or visual content is AI-generated; to publicly report model or system capabilities, limitations, and domains of appropriate and inappropriate use; to prioritize research on societal risks posed by frontier AI models and systems; and to develop and deploy frontier AI models and systems to help address the world’s greatest challenges"
If this were a story, there'd be some kind of academy taking in humanity's top talent and skilling them up in alignment.
Most of the summer fellowships seem focused on finding talent that is immediately useful. And I can see how this is tempting given the vast numbers of experienced and talented folks seeking to enter the space. I'd even go so far as to suggest that the majority of our efforts should probably be focused on finding people who will be useful fairly quickly.
Nonetheless, it does seem as though there should be at least one program that aims to f... (read more)
There is a world that needs to be saved. Saving the world is a team sport. All we can do is to contribute our part of the puzzle, whatever that may be and no matter how small, and trust in our companions to handle the rest. There is honor in that, no matter how things turn out in the end.
Zachary Robinson recently stated that CEA would choose to emphasize a principles-first approach to EA. Here are my thoughts on the kinds of strategic decisions that naturally synergies with this high-level strategy:
Growth strategy: Less focused on fast growth, more focus on attracting value-aligned talent:
Eternal September effects make it hard to both grow fast and maintain high-fidelity transmission of EA principles.
Recruiting from audiences that are capable of engaging in nuanced discussions of what these principl
I just created a new Discord server for generated AI safety reports (ie. using Deep Research or other tools). Would be excited to see you join (ps. Open AI now provides uses on the plus plan 10 queries per month using Deep Research).
Maybe I'm missing something, but I think it's a negative sign that mirror bacteria seems to have pretty much not been discussed within the EA community until now (that said, what really matters is the percent of biosecurity folk in the community who have heard of this issue).
I'm not really focused on animal rights nor do I spend much time thinking about it, so take this comment with a grain of salt.
However, if I wanted to make the future go well for animals I'd be offering free vegan meals in the Bay Area or running a conference on how to ensure that the transition to advanced AI systems goes well for animals in the Bay Area.
Reality check: Sorry for being harsh, but you're not going to end factory farming before the transition to advanced AI technologies. Max 1-2% chance of that happening. So the best thing to do is to ensure that this goes well for animals and not just humans.
There is an AI, Animals, & Digital Minds conference that's being planned in the Bay Area for earlyish 2025! Updates will be announced in the AI & Animals newsletter.
3
Jason
I'm confused about the theory of impact for "free vegan meals in the Bay Area" idea. A few recipients might work in AI, but I don't see the link between eating a vegan meal offered for free and making more animal-friendly AI development choices.
3
Chris Leong
Presumably you’d be doing outreach at the same time to influence values.
If we run any more anonymous surveys, we should encourage people to pause and consider whether they are contributing productively or just venting. I'd still be in favour of sharing all the responses, but I have enough faith in my fellow EAs to believe that some would take this to heart.
One underrated factor in whether to engage in community-building[1] is how likely you are to move to a hub.
I suspect that in most cases people can achieve more when they are part of a group, rather than when they are by themselves. Let's assume that your local community doesn't already provide what you need. Let's further assume that an online community isn't sufficient for your needs either:
Then you have two main options:
• If there's already a hub that provides the community that you need, then you could move there • You could... (read more)
One of the vague ideas spinning around in my head is that maybe in addition to EA which is a fairly open, loosely co-ordinated, big-tent movement with several different cause areas, there would also in value in a more selective, tightly co-ordinated, narrow movement focusing just on the long term future. Interestingly, this would be an accurate description of some EA orgs, with the key difference being that these orgs tend to rely on paid staff rather than volunteers. I don't have a solid idea of how this would work, but just thought I'd put this out there...
Oh, I would've sworn that was already the case (with the understanding that, as you say, there is less volunteering involved, because with the "inner" movement being smaller, more selective, and with tighter/more personal relationships, there is much less friction in the movement of money, either in the form of employment contracts or grants).
I think I posted in one of the threads that I have no knowledge of what private evidence Nonlinear may have, but I just realised that I actually do. I don't think it's a big enough deal for me to go back and try to track down the actual comments and edit them, but I thought it was good practise to note this on short form nonetheless.
Maybe EA should try to find a compromise on the unpaid internship issue? For example, unpaid internships up to a maximum of 2 days/week being considered acceptable with the community?
This would provide additional opportunities for people to skill up, whilst ensuring that these opportunities would still be broadly accessible.
You say "find a compromise" as if this is a big and contentious issue, but I... don't really see it coming up a lot? I know Kat Woods has recently posted elsewhere about how lots of unpaid internships are being suppressed because random bystanders on the internet object to them, but I just don't actually see that happening. I would imagine that often management capacity is more of a bottleneck than pay anyway?
I suspect that it could be impactful to study say a masters of AI or computer science even if you don't really need it. University provides one of the best opportunities to meet and deeply connect with people in a particular field and I'd be surprised if you couldn't persuade at least a couple of people of the importance of AI safety without really trying. On the other hand, if you went in with the intention of networking as much as possible, I think you could have much more success.
Someone needs to be doing mass outreach about AI Safety to techies in the Bay Area.
I'm generally more of a fan of niche outreach over mass outreach, but Bay Area tech culture influences how AI is developed. If SB 1047 is defeated, I wouldn't be surprised if the lack of such outreach ended up being a decisive factor.
There's now enough prominent supporters of AI Safety and AI is hot enough that public lectures or debates could draw a big crowd. Even though a lot of people have been exposed to these ideas before, there's something about in-person events that make ideas seem real.
Weak-downvoted; I think it's fair game to say an org acted in an untrustworthy way, but I think it's pretty essential to actually sketch the argument rather than screenshotting their claims and not specifying what they've done that contradicts the claims. It seems bad to leave the reader in a position of being like, "I don't know what the author means, but I guess Epoch must have done something flagrantly contradictory to these goals and I shouldn't trust them," rather than elucidating the evidence so the reader can actually "form their own judgment." Ben_West then asked in two comments for these specifics, and I still don't know what you mean (and I think I'm pretty high-percentile among forum readers on the dimension of "familiar with drama/alleged bad behavior of AI safety orgs").
Would remove the downvote if you fill in the implicit part of the argument here: what information/explanation would a reader need to know what you mean by "it certainly seems to me that the AI Safety community was too ready to trust Epoch" in the context of these screenshots?
Honestly, I don't care enough to post any further replies. I've spent too much time on this whole Epoch thing already (not just through this post, but through other comments). I've been reflecting recently on how I spend my time and I've realised that I often make poor decisions here. I've shared my opinion, if your opinion is different, that's perfectly fine, but I'm out.
Trust has never been just about whether someone technically lied.
8
Ben_West🔸
Sure, but I just genuinely don't know what you are complaining about here. I can make a few guesses but it seems better to just ask what you mean.
3
Chris Leong
It can be a mistake to have trusted someone without there necessarily having been misbehavior. I'm not saying there wasn't misbehavior, that's just not my focus here.
Looks like Mechanize is choosing to be even more irresponsible than we previously thought. They're going straight for automating software engineering. Would love to hear their explanation for this.
"Software engineering automation isn't going fast enough"[1] - oh really?
This seems even less defensible than their previous explanation of how their work would benefit the world.
Not an actual quote
Some useful context is that I think a software singularity is unlikely to occur; see this blog post for some arguments. Loosely speaking, under the view expressed in the linked blog post, there aren't extremely large gains from automating software engineering tasks beyond the fact that these tasks represent a significant (and growing) fraction of white collar labor by wage bill.
Even if I thought a software singularity will likely happen in the future, I don't think this type of work would be bad in expectation, as I continue to think that accelerating AI is likely good for the world. My main argument is that speeding up AI development will hasten large medical, technological, and economic benefits to people alive today, without predictably causing long-term harms large enough to outweigh these clear benefits. For anyone curious about my views, I've explained my perspective on this issue at length on this forum and elsewhere.
This is a clear strawman. Matthew has given reasons why he thinks acceleration is good which aren't this.
Sometimes the dollar signs can blind someone and cause them not to consider obvious alternatives. And they will feel that they made the decision for reasons other than the money, but the money nonetheless caused the cognitive distortion that ultimately led to the decision.
I'm not claiming that this happened here. I don't have any way of really knowing. But it's certainly suspicious. And I don't think anything is gained by pretending that it's not.
I guess orgs need to be more careful about who they hire as forecasting/evals researchers in light of a recently announced startup.
Sometimes things happen, but three people at the same org...
This is also a massive burning of the commons. It is valuable for forecasting/evals orgs to be able to hire people with a diversity of viewpoints in order to counter bias. It is valuable for folks to be able to share information freely with folks at such forecasting orgs without having to worry about them going off and doing something like this.
However, this only works if those less worried about AI risks who join such a collaboration don't use the knowledge they gain to cash in on the AI boom in an acceleratory way. Doing so undermines the very point of such a project, namely, to try to make AI go well. Doing so is incredibly damaging to trust within the community.
Now let's suppose you're an x-risk funder considering whether to fund their previous org. This org does really high-quality work, but the argument for them being net-positive is now significantly weaker. This is quite likely to make finding future funding harder for them.
This is less about attacking those three folks and more just no... (read more)
I'd like to suggest a little bit more clarity here. The phrases you use refer to some knowledge that isn't explicitly stated here. "in light of a recently announced startup" and "three people at the same org" make sense to someone who already knows the context of what you are writing about, but it is confusing to a reader who doesn't have the same background knowledge that you do.
Once upon a time, some people were arguing that AI might kill everyone, and EA resources should address that problem instead of fighting Malaria. So OpenPhil poured millions of dollars into orgs such as EpochAI (they got 9 million). Now 3 people from EpochAI created a startup to provide training data to help AI replace human workers. Some people are worried that this startup increases AI capabilities, and therefore increases the chance that AI will kill everyone.
Also, it is worrying if the optimists easily find financial opportunities that depend on them not changing their minds. Even if they are honest and have the best of intentions, the disparity in returns to optimism is epistemically toxic.
EA needs more communications projects.
Unfortunately, the EA Communications Fellowship and the EA Blog prize shut down[1]. Any new project needs to be adapted to the new funding environment.
If someone wanted to start something in this vein, what I'd suggest would be something along the lines of AI Safety Camp. People would apply with a project to be project leads and then folk could apply to these projects. Projects would likely run over a few months, part-time remote[2].
Something like this would be relatively cheap as it would be possible for someone to run this on a volunteer basis, but it might also make sense for there to be a paid organiser at a certain point.
Likely due to the collapse of FTX
Despite the name, AI Safety Camp is now remote.
For the record, I see the new field of "economics of transformative AI" as overrated.
Economics has some useful frames, but it also tilts people towards being too "normy" on the impacts of AI and it doesn't have a very good track record on advanced AI so far.
I'd much rather see multidisciplinary programs/conferences/research projects, including economics as just one of the perspectives represented, then economics of transformative AI qua economics of transformative AI.
(I'd be more enthusiastic about building economics of transformative AI as a field if we were starting five years ago, but these things take time and it's pretty late in the game now, so I'm less enthusiastic about investing field-building effort here and more enthusiastic about pragmatic projects combining a variety of frames).
As an example, I expect political science and international relations to be better for looking at issues related to power distribution rather than economics (though the economic frame adds some value as well). Historical studies of coups seems pretty relevant as well.
When it comes to predicting future progress, I'd be much more interested in hearing the opinions of folks who combine knowledge of economics with knowledge of ML or computer hardware, rather than those who are solely economists. Forecasting seems like another relevant discipline, as is future studies and history of science.
I’m pretty bullish on having these kinds of debates. While EA is doing well at having an impact in the world, the forum has started to feel intellectually stagnant in some ways. And I guess I feel that these debates provide a way to move the community forward intellectually. That's something I've been feeling has been missing for a while.
Let Manifest be Manifest.
Having a space that is intellectually-edgy, but not edge-lord maxing seems extremely valuable. Especially given how controversial some EA ideas were early on (and how controversial wild animal welfare and AI welfare still are).
In fact, I'd go further and suggest that it would be great if they were to set up their own forum. This would allow us to nudge certain discussions into an adjacent, not-explicitly EA space instead of discussing it here.
Certain topics are a poor fit for the forum because they rate high on controversy + low-but-non-zero on relevance to EA. It's frustrating having these discussions on the forum as it may turn some people off, but at the same time declaring these off-topic risks being intellectually stiffling. Sometimes things turn out to be more important than you thought when you dive into the details. So I guess I'd really love to see another non-EA space end up being the first port of call for such discussions, with the hope that only the highest quality and most relevant ideas would make it over to the EA forum.
Looks like you double posted this
I'd love to see the EA forum add a section titled "Get Involved" or something similar.
There is the groups directory, but it's one of only many ways that folks can get more involved, from EAGx Conferences, to Virtual Programs, 80,000 Hours content/courses to donating.
I'll post some extracts from the commitments made at the Seoul Summit. I can't promise that this will be a particularly good summary, I was originally just writing this for myself, but maybe it's helpful until someone publishes something that's more polished:
Frontier AI Safety Commitments, AI Seoul Summit 2024
The major AI companies have agreed to Frontier AI Safety Commitments. In particular, they will publish a safety framework focused on severe risks: "internal and external red-teaming of frontier AI models and systems for severe and novel threats; to work toward information sharing; to invest in cybersecurity and insider threat safeguards to protect proprietary and unreleased model weights; to incentivize third-party discovery and reporting of issues and vulnerabilities; to develop and deploy mechanisms that enable users to understand if audio or visual content is AI-generated; to publicly report model or system capabilities, limitations, and domains of appropriate and inappropriate use; to prioritize research on societal risks posed by frontier AI models and systems; and to develop and deploy frontier AI models and systems to help address the world’s greatest challenges"
"Risk a... (read more)
If this were a story, there'd be some kind of academy taking in humanity's top talent and skilling them up in alignment.
Most of the summer fellowships seem focused on finding talent that is immediately useful. And I can see how this is tempting given the vast numbers of experienced and talented folks seeking to enter the space. I'd even go so far as to suggest that the majority of our efforts should probably be focused on finding people who will be useful fairly quickly.
Nonetheless, it does seem as though there should be at least one program that aims to f... (read more)
There is a world that needs to be saved. Saving the world is a team sport. All we can do is to contribute our part of the puzzle, whatever that may be and no matter how small, and trust in our companions to handle the rest. There is honor in that, no matter how things turn out in the end.
Someone really needs to make Asterisk meetup groups a thing.
What could principles-first EA look like?
Zachary Robinson recently stated that CEA would choose to emphasize a principles-first approach to EA. Here are my thoughts on the kinds of strategic decisions that naturally synergies with this high-level strategy:
- Growth strategy: Less focused on fast growth, more focus on attracting value-aligned talent:
- Eternal September effects make it hard to both grow fast and maintain high-fidelity transmission of EA principles.
- Recruiting from audiences that are capable of engaging in nuanced discussions of what these principl
... (read more)I just created a new Discord server for generated AI safety reports (ie. using Deep Research or other tools). Would be excited to see you join (ps. Open AI now provides uses on the plus plan 10 queries per month using Deep Research).
https://discord.gg/bSR2hRhA
Maybe I'm missing something, but I think it's a negative sign that mirror bacteria seems to have pretty much not been discussed within the EA community until now (that said, what really matters is the percent of biosecurity folk in the community who have heard of this issue).
I'm not really focused on animal rights nor do I spend much time thinking about it, so take this comment with a grain of salt.
However, if I wanted to make the future go well for animals I'd be offering free vegan meals in the Bay Area or running a conference on how to ensure that the transition to advanced AI systems goes well for animals in the Bay Area.
Reality check: Sorry for being harsh, but you're not going to end factory farming before the transition to advanced AI technologies. Max 1-2% chance of that happening. So the best thing to do is to ensure that this goes well for animals and not just humans.
Anyway, that concludes my hot-take.
If we run any more anonymous surveys, we should encourage people to pause and consider whether they are contributing productively or just venting. I'd still be in favour of sharing all the responses, but I have enough faith in my fellow EAs to believe that some would take this to heart.
To Community Build or Not
One underrated factor in whether to engage in community-building[1] is how likely you are to move to a hub.
I suspect that in most cases people can achieve more when they are part of a group, rather than when they are by themselves. Let's assume that your local community doesn't already provide what you need. Let's further assume that an online community isn't sufficient for your needs either:
Then you have two main options:
• If there's already a hub that provides the community that you need, then you could move there
• You could... (read more)
One of the vague ideas spinning around in my head is that maybe in addition to EA which is a fairly open, loosely co-ordinated, big-tent movement with several different cause areas, there would also in value in a more selective, tightly co-ordinated, narrow movement focusing just on the long term future. Interestingly, this would be an accurate description of some EA orgs, with the key difference being that these orgs tend to rely on paid staff rather than volunteers. I don't have a solid idea of how this would work, but just thought I'd put this out there...
I think I posted in one of the threads that I have no knowledge of what private evidence Nonlinear may have, but I just realised that I actually do. I don't think it's a big enough deal for me to go back and try to track down the actual comments and edit them, but I thought it was good practise to note this on short form nonetheless.
Is anyone doing broad AI Safety outreach to techies in the Bay Area?
It seems very important to have a group doing this given how much opinions within Bay Area tech influence how AI is developed.
If SB 1047 doesn't pass, this ball being dropped may be partially to blame.
Maybe EA should try to find a compromise on the unpaid internship issue? For example, unpaid internships up to a maximum of 2 days/week being considered acceptable with the community?
This would provide additional opportunities for people to skill up, whilst ensuring that these opportunities would still be broadly accessible.
(In countries where this is legally allowed)
You say "find a compromise" as if this is a big and contentious issue, but I... don't really see it coming up a lot? I know Kat Woods has recently posted elsewhere about how lots of unpaid internships are being suppressed because random bystanders on the internet object to them, but I just don't actually see that happening. I would imagine that often management capacity is more of a bottleneck than pay anyway?
I suspect that it could be impactful to study say a masters of AI or computer science even if you don't really need it. University provides one of the best opportunities to meet and deeply connect with people in a particular field and I'd be surprised if you couldn't persuade at least a couple of people of the importance of AI safety without really trying. On the other hand, if you went in with the intention of networking as much as possible, I think you could have much more success.
Someone needs to be doing mass outreach about AI Safety to techies in the Bay Area.
I'm generally more of a fan of niche outreach over mass outreach, but Bay Area tech culture influences how AI is developed. If SB 1047 is defeated, I wouldn't be surprised if the lack of such outreach ended up being a decisive factor.
There's now enough prominent supporters of AI Safety and AI is hot enough that public lectures or debates could draw a big crowd. Even though a lot of people have been exposed to these ideas before, there's something about in-person events that make ideas seem real.
I took a look at the post announcing Epoch.
It was interesting noting this comment by Ofer:
Jaime Sevilla replied:
Additionally, looking at the post itself:
It's up to the reader to form their own judgement, but it certainly seems to me that the AI Safety community was too ready to trust Epoch.
Weak-downvoted; I think it's fair game to say an org acted in an untrustworthy way, but I think it's pretty essential to actually sketch the argument rather than screenshotting their claims and not specifying what they've done that contradicts the claims. It seems bad to leave the reader in a position of being like, "I don't know what the author means, but I guess Epoch must have done something flagrantly contradictory to these goals and I shouldn't trust them," rather than elucidating the evidence so the reader can actually "form their own judgment." Ben_West then asked in two comments for these specifics, and I still don't know what you mean (and I think I'm pretty high-percentile among forum readers on the dimension of "familiar with drama/alleged bad behavior of AI safety orgs").
Would remove the downvote if you fill in the implicit part of the argument here: what information/explanation would a reader need to know what you mean by "it certainly seems to me that the AI Safety community was too ready to trust Epoch" in the context of these screenshots?
Why do you think we were too ready to trust them? Are you implying that they later violated what Jaime says here?