L

Lucia

4 karmaJoined

Comments
1

Thank you, Marcus and Austin - I found this post fascinating and a really useful exercise. 
I have some thoughts on how you could develop this further.

My main critique is that the current calculations focus a lot on consumption (watching a video) rather than impact (change in the world) - of course this is arguably pretty hard to do without loads of assumptions, but given that the current calcs still use qualitative assumptions, it might be worth trying this other angle.

I'd start sketching this out with a light touch logic model/theory of change, to map where input (money, time) lead to an outcome (hopefully, reduced AI risk). I think the main metric used here (quality-adjusted viewer-minutes) captures the beginning of this process, but would be interesting to go further. A simple logic model could look like:

Inputs: Cost (money, time) --> 
Outputs:Video created --> 
Outcomes:  

  • Reach (views, viewer-minutes)
  • Engagement (likes, shares, Comments)
  • Change in knowledge/attitude (Comprehension, shift in prioriites)
  • Change in behaviour (clicks links, donates, starts a project, discusses with policymakers) --> 

Impact: Reduced AI risk

I think currently you're using some assumptions to essentially derive comprehension, but there's not much in your calcs about engagement or behaviour change. I think it would be useful to try to measure these. 

For instance, you could add in:

  1. A way to calculate network effects/virality:
    1. Watching is not the only way to engage with a video - sharing with a wider network creates a number of positive externalities, reaching people who wouldn't have already been in the niche sphere of the youtube channel (but might equally be high-quality, interested in the topic, just missed by the algorithm), increasing the channel's social stance/prominence, growing the audience, etc.
    2. To measure this you could use the API to include counts of shares, likes, comments as a measure of engagement.
      1. I'd argue this would be a better way to measure quality and resonance as well, rather than using more subjective measures for things like message fidelity, quality of audience - you could use shares, likes, comments as a way to measure a highly engaged audience / audience activation.
      2. You could also argue that views from shares cost a channel less than direct views because the audience did the marketing for them.. but maybe that complicates things too much.
  2. Account for the fact that videos keep being watched:
    1. In theory, successful videos have an upfront cost but then could keep being watched over time, even for a number of years, and so continue to generate a stream of value (views and influence) with marginal cost.
    2. For this, you could include the video viewership decay rate over time - is it just a short burst or does it have a long tail for consistent views? Can you use this to assign a longevity score to videos? This helps minimise the fact that your current analysis is a snapshot, but many of these videos (though not all) will keep racking up views.
  3. Measure the impact of spillovers/brand reputation:
    1. Videos are of course intended to reach an audience, but they also increase brand reputation (e.g. your example of Rob Miles known by everyone). Building up the creator's influence means that it can be used for other high-impact activities e.g. podcast appearances, authoring news articles, briefings with policymakers.
    2. I think for this it would be good to track creator-level outcomes. How many collaborations/speaking gigs/media mentions result from the channel's success (i.e. after videos were made)?
  4. The hardest thing to measure is behaviour change..
    1. While I think shares/likes/overtime engagement/public standing of youtuber could all be some sort of proxy for macro-behaviour change, ideally you'd try to measure things like..
      1. Did the video drive clicks to AI safety career resource (e.g. for 80K)?
      2. Anyone doing post-video surveys that could be factored in?
      3. Did the video creator get more DMs about offers to help?
    2. Given that you reached out to the creators with other questions, you could have included some of these to get a better sense of behaviour change.
       

Anyway, these are just some thoughts, but hope you'll find at least some of them useful. 

Thanks again for sharing this post, really interesting and I'm excited to see how it evolves.