[Back to Computing and IT]
Introduction
The ‘Center for Applied Rationality’, or ‘CFAR’ is one of a group of organizations, run by a fairly small group of individuals, in the fields of artificial intelligence, rationality and global philanthropy. This is an attempt to identify some of the key players and their connections.
Core Timeline
When used in the context of AI, ‘singularity’ refers to an event – either to the moment when artificial intelligence becomes able to improve itself, leading to a positive feedback loop of increasingly rapid self improvement, or to the point soon after this when it surpasses human intelligence. Many in the AI community believe that this will happen, perhaps soon, and will rapidly lead to irreversible changes, but the nature of those changes are debated; some believe it will result in the end of the human race. While the details are debated, belief in the singularity is almost universal within the AI community.
Eliezer Yudkowsky is an AI researcher who, along with Brian and Sabine Atkins, set up the ‘Singularity Institute for Artificial Intelligence’, or ‘SIAI’, in 2000; initially, this was for the purpose of accelerating the development of artificial intelligence, but since 2005 the main focus of its activity was on identifying and managing potential existential risks from artificial general intelligence.
In 2006, SIAI cooperated with Stanford University to organize the the first Singularity Summit to discuss the future of AI; it was funded by Peter Thiel and described by the San Francisco Chronicle as a “Bay Area coming-out party for the tech-inspired philosophy called transhumanism”. These summits continued each year until 2012.
Also in 2006, ‘Overcoming Bias’ (www.overcomingbias.com/) was established: this was a cognitive and social science blog sponsored by the ‘Future of Humanity Institute’ of Oxford University; the main contributors were Yudkowsky and economist Robin Hanson. The main focus now is on human rationality. “This is a blog on why we believe and do what we do, why we pretend otherwise, how we might do better, and what our descendants might do, if they don't all die.”
In 2008, ‘Singularity’ (su.org/) was founded by Peter Diamandis and Ray Kurzweil, inspired by Ray’s book, ‘The Singularity is Near’. Singularity is often called ‘Singularity Group’ and trades as ‘Singularity University’, although it is not a university. It describes its vision as “a world of abundance created by futuremakers working together to solve the world’s greatest challenges”; “we believe technology and entrepreneurship can solve the world’s greatest challenges”.
In 2009, Yudkowsky and Hanson went their separate ways; Hanson retained ‘Overcoming Bias’ as his personal blog, and Yudkowsky established ‘LessWrong’ (lesswrong.com/), starting with much of his material from Overcoming Bias. “LessWrong is an online forum and community dedicated to improving human reasoning and decision-making. We seek to hold true beliefs and to be effective at accomplishing our goals. Each day, we aim to be less wrong about the world than the day before.”
The ‘Center for Applied Rationality’, or ‘CFAR’ (rationality.org/) grew out of LessWrong in 2012 to deliver workshops that improve participants’ rationality using “a set of techniques from math and decision theory for forming your beliefs about the world as accurately as possible”; they run workshops dedicated to “developing clear thinking for the sake of humanity's future.”
In December 2012 (a few months after CFAR was established) SIAI sold its name, web domain, and the Singularity Summit to Singularity University. The following month it changed its name to the ‘Machine Intelligence Research Institute’, or ‘MIRI’ (intelligence.org/).
In December 2015, OpenAI was founded as a non-profit, with Sam Altman and Elon Musk as the co-chairs, with promised funding by various sources including Peter Thiel; the founding mission was to ensure that AGI benefits all of humanity, with the promise that it would make its patents and research open to the public. In 2018, Musk resigned; the following year, it changed from non-profit to ‘capped for-profit’. In December 2022, OpenAI launched ChatGPT; this gained over a million users within the first five days and over 100 million in the first two months.
Charities
In 2012, Toby Ord and William MacAskill formed the ‘Centre for Effective Altruism’, or ‘CEA’ (centreforeffectivealtruism.org/) to provide an umbrella organization for several charities (‘Giving What We Can’ and ‘80,000 Hours’), with support from SIAI, LessWrong and the Future of Humanity Institute. Sam Bankman-Fried became its Director of Development in 2017 for a short while, but this is before he was convicted of fraud in November 2023.
CEA is a project of two charities: ‘Effective Ventures Foundation’ in the UK, and a similar body in the USA. IN 2021-22, the UK charity had an income of just over £140m; the figures for the US charity are not yet available, but in the previous year it had an income of nearly $17½m.
In 2006, two people who worked at a hedge fund (Holden Karnofsky and Elie Hassenfeld ) set up an informal group to apply the techniques they used in the hedge fund and apply them to charities In 2007, they formed GiveWell (www.givewell.org) to rate charities according to how much money it cost them to save a life. According to one observer, they “try to figure out which global poverty charities are most cost effective and funnel money to them.” Karnofsky was on the OpenAI board between 2017 and 2021; in 2025, he joined the staff of Anthropic, a company which his wife coi-founded.
Also worth mentioning is Good Ventures (www.goodventures.org), a philanthropic organization co-founded in 2011 by Cari Tuna and Dustin Moskovitz (a co-founder of Facebook). It distributes money from Moskovitz in accordance with the principles of effective altruism. The partnership between GiveWell and Good Ventures was called the ‘Open Philanthropy Project’ in 2014, and was established as Open Philanthropy (www.openphilanthropy.org) in 2017 to distribute the money from Good Ventures.
Other Background
In December 1998, Peter Thiel and two others co-founded the company that became PayPal. In March 2000, it merged with X.com (a company set up in March 1999 by Elon Musk and three others), and was renamed PayPal in June 2001.
MIRI now refers to the Singularity as ‘artificial superintelligence’, or ‘ASI’. According to its website, the default consequence of the creation of ASI is human extinction; our survival depends on delaying the creation of ASI, as soon as we can, for as long as necessary.
The ‘Future of Humanity Institute’ of Oxford University no longer exists. It was founded in 2005 as an interdisciplinary research centre within the Faculty of Philosophy, with the philosopher Nick Bostrom as its director. Among the staff was Toby Ord, the founder of ‘Giving What We Can’; from 2012 they shared an office with CEA. Tensions with the Faculty of Philosophy led to the Institute being closed in April 2024.
See Also
Comments