Very nice, thank you for writing. 

It seems plausible that p(annual collapse risk) is in part a function of the N friction as well? I think you may cover some of this here but can't really remember. 

e.g. a society with less non-renewables will have to be more sustainable survive/grow -> resulting population is somehow more value aligned -> reduced annual collapse risk. 

or on the other side

nukes still exist and we can still launch them -> we have higher N friction in pre nuclear age in the next society -> increased annual collapse risk. 

(i have a bad habit of just adding slop onto models and think this isn't at all something that need be in the scope of original post just a curiousity). 

I think monied prediction markets are negative EV. The original reasons the CFTC were not allowing binary event contracts on most things are/were actually good reasons. It's quite clear that our elected officials can get away with insider trading (and probably to a certain extent market manipulation). My intuition is that in the current admin I expect this behavior to increasingly not be punished and maybe actively encouraged. Importantly, insider trading on the existing financial instruments doesn't really work. My take here is just that the marginal value of a piece of information is pretty low for traditionally financial markets, so while it might allow a high-tech research shop to beat the market it doesn't even cover slippage/the lack of complex modeling done when in the hands of a dumb paper congress person. This is not the case for most/all binary event markets, where a single marginal piece of information can credibly flip probabilities from <1 -> >99. 

Insider trading, which I expect to be much more likely to happen at scale with these markets, is not that bad. It degrades the integrity of the prediction markets themselves and might lower volume but probably not that relevant for EA. However market manipulation could get really really bad in terms of EV (I think significant market manipulation is currently quite unlikely at the current levels of liquidity on these markets). In particular, the less likely something is to happen, the more incentive a prediction market provides for manipulating the market. And like in insider trading, I think market manipulation is going to be much more accessible and effective for generating alpha vs traditional markets. 

low stakes example: Trump's speech writer puts random phrase ( Lock x up, I hate x country, Sperm)

medium stakes: Every election market is an assassination market, and a bounty for either candidate to drop out

High stakes: "will x leader/country bomb  y"

Of course with the medium and high stakes examples, we are far from the liquidity on these markets to make the incentives provided to be worth seriously considering. But what if we 100x? what if e.g. mamdani could make 50 million dollars by dropping out instead of 500k? What if trump could make 20m by delaying a ceasefire a few days?

To some extent we will just have to see how the enforcement of this stuff plays out as that will shape the incentives. It might seem obvious that the stuff above will be easily sorted out by the relevant federal bodies. Again, my intuition is that (1) I don't actually expect this to be a huge priority right now and (2) the regulatory burden to properly regulate this stuff is ridiculously high (both in terms of just tracking everything that is happening and actually trying to apply the law to decide if various things constitute as "security fraud"). 

It's already hard for the SEC to enforce behavior the NYSE or CME, and that's with mostly big institutional players who cooperate and record all their communications, etc. I'd have to imagine especially in the short term the vast majority of questionable or illegal behavior on prediction markets will slip through the cracks. The space of potential "securities fraud" is just so ridiculously big and confusing when you have 1000s of random binary event contracts. 

Then you put that with the encouragement of gambling and incentivizing the spread of fake news, I just think we should be very very skeptical of going head first into this.

And the alternatives are pretty good! Non monied prediction markets like manifold and metaculus are huge improvements over previous epistemics, and to be fair the monied prediction markets are quite literally crowding them out. It's not really fair to compare polymarkets accuracy to manifold, when a bunch of people who are on poly would probably use manifold if they made poly illegal (and that being said I'm pretty sure manifold doesn't stack up to poorly but don't have up to date stats). And some of the most important binary event contracts were already being traded on CME because they didn't suffer from the issues I listed above. 

Given the public good nature of prediction markets, the government should be quite willing to front $10s-100s of millions to improve their quality. And there are lots of ways to improve the markets without directly providing personal monetary incentives. This could mean improving the site ui/ux, creating a public leaderboard with some sort of recognization/award, improving the question statements/resolution, or even providing prizes that can be donated to a charity of the winners choice. 
 

Yea my original framing was a little confused wrt the "vs"  dichotomy you present in paragraph one, good shout. I guess I actually meant a little bit of each, though. My interpretation of the post is basically, (1) in so forth as we need to defeat powerful people or thought patterns we (ea or humans) haven't proven it (2) it's somewhat likely we will need to do this to create the world we want. 

I.e. Given that future s-risk efforts are probably not going to be successful, current extinction-risk efforts are therefore also less useful. 

I am saying aligning AI is in the best interests of AI companies
 

If you define it in a specifically narrow AI Takeover way yes. Making sure it doesn't allow a dictator to take power or gradually disempowerment scenarios, not really. Or to the extent that ensuring alignment requires slowing down progress.

Anyway mostly in agreement with your points/world, I definitely think we should be focusing on AI right now and I think that our goals and the AI companies/US gov are sufficiently aligned atm that we aren't swimming up stream, but I resonante with OP that it would allievate some concerns if we actually racked up some hard fought politically unpopular battles before trying to steer the whole future. 

It certainly seems possible (>1%) that in the next 2 US admins (current plus next) AI safety becomes so toxic that all the EA -adj ai safety people in the gov get purged and they stop listeing to most ai safety researchers. If this co-occurs with some sort of AI nationalization most of our TOC is cooked. 

Many, if not most, longtermists believe we're living near a hinge of history

Right but this requires believing the future will be better if humans survive. I take ops point as saying she doesn't agree or is at least skeptical. 

and a small group of AI companies adopting research that is in their best interests to use.

I think again, the point of OP is trying to make is we have very little proof of concept of getting people to go against their best interests. And so if doing what's right isn't in the ai companies best interest op wouldn't believe we can get them to do what we think they should.  

Yea it's kinda like what they tell you not to do when building a startup. Every founder wants to build a beautiful, hyperscaling tech-heavy product before they have even confirmed that they have a few single real customers. In this case we are gonna write out our entire plans for the future of the universe before we win a single congressional seat. 

Anyway this community isn't set up to end something like veganism I don't think. That requires large scale evangelizing and coalition building (unless we can solve it with tech). This movement is investing mostly into research and policy, I.E. we are betting on lots of the most important issues of our time not being politically toxic/salient. I think there is a lot of truth to the notion that most federal policy is written by people in think tanks and OMB - and that as long as it doesn't piss off the electorate then the policymaker rather than the elected politician effectively gets to write the law. 

But for stuff that obviously is in the mainstream overton window, e.g. veganism that is going to require large behavioral changes from ordinary citizens , you need an actual coalition of hard power. 

Sure that's a fair point. I'd guess I hope you would feel at least a little pushed in the direction after this thread that AIs need not take a similar route to humans to automating large amounts of our current work. 

"novel idea" means almost nothing to me. A math proof is simply a->b. It doesn't matter how you figure out a->b. If you can figure it out by reading 16 million papers and clicking them together that still counts. There are many ways to cook an egg. 

https://x.com/slow_developer/status/1979157947529023997
I would bet a lot of money you are going to see exactly what I described for math in the next two years. The capabilities literally just exploded. It took us like 20 years to start using the lightbulb but you are expecting results from products that came out in the last few weeks/months. 

I can also confidently say because I am working on a project with doctors that the work I described for clinical medicine is being tested and happening right now. It's exact usefulness remains to be seen but like people are trying exactly what I described, there will be some lag as people need to learn how to use the tools best and then distribute their results. 

Again, I don't think most of this stuff was particularly useful with the tools available to use >1 year ago. 

>Would an AI system that can't learn new ideas from one example or a few examples count as AGI?

https://www.anthropic.com/news/skills
you are going to need to be a lot more precise in your definitions imo otherwise we are going to talk past each other. 

i'm fleshing out nunos point a bit. Basically AI have so many systematic advantages with their cost/speed/seemless integration into the digital world that they can afford to be worse than humans at a variety of things and still automate (most/all/some) work. Just as a plane doesn't need to flap it's wings. Of course I wasn't saying I solved automating the economy. I'm just showing you ways in which something lacking some top level human common sense/iq/whatever could replace still. 

FWIW I basically disagree with every point you made in the summary. This mostly just comes from using these tools every day and getting utility out of them + seeing how fast they are improving + seeing how many different routes there are to improvement (i was quite skeptical a year ago, not so anymore). But I wanted to keep the argument contained and isolate a point of disagreement. 

For example, how can AI automate the labour of scientists, philosophers, and journalists if it can’t understand novel ideas?

The bar is much lower because they are 100x faster and 1000x cheaper than me. They open up a bunch of brute forceable techniques in the same way that you can open up  https://projecteuler.net/ solve many of eulers discoveries with little math knowledge but basic python and for loops. 

Math -> re read every arxiv paper -> translate them all into lean -> aggregate every open well specificied math problem -> use the database of all previous learnings to see if you can chain chunks of previous problems together to solve. 

clinical medicine -> re-read every RCT ever done and comprehensively rank intervention effectiveness by disease -> find cost data where available and rank the cost/qaly of all disease/intervention space

Econometrics -> aggregate every natural experiment and instrumental variable ever used in an econometrics paper -> think about other use cases for these tools -> search if other use cases have available data -> reapply the general theory of the original paper with the new data.