Getting AI Right: A 2050 Thought Experiment
by James Manyika (1)
The author concludes the volume with an ambitious agenda for the future, envisioning a society in 2050 where AI has been broadly beneficial. He outlines grand challenges across the AI pipeline, emphasizing ethical development to maintain democratic integrity and public trust. The vision focuses on human flourishing in a world enhanced by AI.
During this period of astonishing technical progress and public engagement on artificial intelligence, the world has been grappling with how to get AI right—and for good reason.
AI is a foundational and general-purpose technology with the potential to benefit people and society in a range of ways: (1) assist people with everyday tasks, help them access and benefit from the world’s information and knowledge, and pursue their most ambitious, productive, and creative endeavors; (2) contribute to economic progress by enabling entrepreneurs, powering small and large businesses and other entities, including those providing public services, transforming organizations and sectors, fueling innovation and productivity, and contributing to economic prosperity; (3) accelerate scientific advances in fields ranging from medicine to materials, physics, climate sciences and more; and (4) help make progress on many of humanity’s most pressing challenges and opportunities, from food security to health and well-being. While many of these possibilities are increasingly being realized (some recently, others for years), it is still early days, and we can expect more ahead if and as AI becomes more capable and its use broadens. Such benefits are neither guaranteed nor automatic—they will require co-innovations, investments, relevant and broad application, and process and other changes, as well as societal and policy choices to enable them.
At the same time, there are concerns about AI’s development, deployment, and use, including (1) robustness, accuracy, bias, and privacy risks; (2) risks from misapplication and misuse; (3) potentially complex impacts on society from jobs to education and democracy, and the possibility of unintended or unforeseen consequences; and (4) challenges of alignment with human preferences and human flourishing as AI becomes more capable. Such concerns, if unaddressed, could create information hazards, lead to safety and security risks, and cause harm. They could also worsen already challenging issues like inequality, destabilize institutions, and erode public trust—but this is not inevitable. It is this combination—of the immense potential of AI to benefit humanity and its risks and complexities—that makes it imperative that we get AI right. This must involve developers and users of AI and multiple stakeholders. It also raises the stakes of how we, as nations and an international community, govern AI democratically and inclusively.
“We must tackle complex problems while also acknowledging what we do not yet know in the face of a new and advancing technology and its evolving uses.”
A raft of national, regional, and international efforts by governments, civil society, academia, and industry have emerged in response. (2) In an encouraging demonstration of international resolve, in March 2024, member states of the United Nations unanimously adopted a landmark resolution: “Seizing the opportunities of safe, secure and trustworthy artificial intelligence systems for sustainable development.” The resolution called for “respect, protection and promotion of human rights in the design, development, deployment and the use of AI,” and called on members to “bridge the artificial intelligence and other digital divides between and within countries.” Earlier, the UN Secretary General’s High-Level Advisory Body on AI had highlighted the opportunities and risks, as well as the gaps in governance and capacity for all to benefit from AI. It emphasized the need to govern AI in the public interest and for the benefit of all, “anchored in the UN Charter, International Human Rights Law, and other agreed international commitments such as the Sustainable Development Goals.” (3)
To realize the ambitious goals in the UN resolution, and those in many national and regional initiatives, we must tackle complex problems, while also acknowledging what we do not yet know in the face of a new and advancing technology and its evolving uses. The problems include and go beyond technical and sociotechnical challenges; they include complex economic, social, and ethical issues, as well as issues of governance within and among countries. Adding further complexity are the wide range of views around the world on AI. (4) And all this is certain to evolve as AI advances and its use, and society’s experiences with it, grows.
A Thought Experiment
Like many others, I’ve been exploring the question of getting AI right with many of AI’s pioneers and others at the forefront of its potential impact on society. (5) This led a few years ago to a Working List of Hard Problems in AI. (6) The initial motivation was a question Stuart Russell posed to a panel I was on. I have come to articulate it as follows:
“It’s the year 2050. AI has turned out to be hugely beneficial to society and generally acknowledged as such. What happened?”
The question, or thought experiment, aims to elicit the most worthwhile possibilities we will have achieved, the most beneficial opportunities realized, the hard problems solved, the risks averted, misuses avoided, and unintended consequences mitigated if we are to acknowledge a positive outcome in 2050. It is a way of asking what we need to get right if AI is to be a net benefit to society in a not-too-distant future.
Since compiling the list of hard problems, I’ve revisited it with many of those I initially discussed it with to assess whether it is still a good list. While the specifics underlying each problem have evolved with AI’s advance, the list has proven to be relatively robust—at least so far.
So, assuming we have a good list of hard problems that matter (the Working List or a better one) the question becomes: How do we make progress? In this paper, the ten problems from the working list are grouped into five. The goal is to motivate readers to reflect on their own view of the hard problems. This is urgently needed, as the actions we take now will chart the course of AI’s impact for decades. I welcome vigorous discussion, debate, iteration, and action.
Group 1 Problems:
Development of more capable, safer, and trustworthy AI
This first group of problems has to do with the challenge of developing AI capable of assisting humanity to achieve our most ambitious and beneficial uses. This requires that we tackle the limitations of current systems and develop even more capable AI. And at the same time, we must address the shortcomings of current systems, mitigate concerns, and build public trust. It is also important that the technical characteristics of AI not create new harms or worsen existing societal issues—ideally, they should contribute to improving lives.
Why it matters: In the 2010s, progress in deep learning and the availability of compute and data accelerated the development of AI—from speech and natural language processing, to image recognition and machine vision, and to robotics and games of strategy like Go—with many key advances contributed by my colleagues. (7) Since the development of the Transformer by Google researchers in 2017, (8) heralding the era of large language models (LLMs), AI systems have become more general across domains and modalities, including text, audio, images, and video. Their performance has—so far—continued to advance according to so-called “scaling laws,” which suggest that bigger and more complex models, trained on more data, improve in capability. Due to this and innovations in other areas of AI (impressive in their own right), AI’s capacity to be useful has rapidly advanced to more and richer ways to assist people, transform and power organizations’ activities, advance science, and help tackle societal challenges. (9)
“While many of the advances so far have been attributed to progress in LLMs, scaling laws, and other current techniques, this may not persist and/or be sufficient; additional breakthroughs may be needed to attain more advanced and hitherto unattained capabilities.”
Yet despite these advances, AI systems have limitations—things they are not (yet) capable of— and shortcomings—undesirable characteristics we would rather they did not have. The limitations, if overcome, would further enhance AI’s usefulness. For example, complex reasoning, while improving, is still in its early stages, as is AI’s capacity to handle memory, perceive and build real-world models, plan and act, and continuously learn and self-improve. Moreover, more-capable AI may be needed for further breakthroughs in science and on challenges like climate change. This may require AI capable of generating novel scientific conjectures and theories, proposing and steering experiments, perceiving the world in novel ways, including and beyond our sensory and cognitive limits, and solving problems that require other kinds of intelligence in addition to ours. While many of the advances so far have been attributed to progress in LLMs, scaling laws, and other current techniques, this may not persist and/or be sufficient; additional breakthroughs may be needed to attain more advanced and hitherto unattained capabilities.
Some of the shortcomings of current systems could create risks, cause harm, and erode public trust if not mitigated. For example, LLMs sometimes struggle with factuality and so-called “hallucinations” that could create information hazards, and they sometimes exhibit bias, in some cases due to bias in training data. Shortcomings such as out-of-distribution failures, security and privacy risks, vulnerability to compromise, and adversarial steering are especially important in high-stakes domains like health, finance, and law. While there has been some technical progress on these shortcomings (e.g., privacy-preserving machine learning), more is needed. Other shortcomings include the current resource intensity associated with training and using AI. Ignoring AI’s shortcomings could cause ethical and social risks (10) and potentially limit use of AI even where benefits are high and the risks very low.
Some markers of progress
• AI with improved paradigms of learning, real-world understanding, meta-level reasoning, human-directed self-improvement, and agentive capabilities.
• AI that complements and extends human capabilities in novel and useful ways, including in areas beyond human sensory, cognitive, and spatiotemporal range.
• AI capable of developing novel scientific concepts, theories, and experiments, and of generating genuinely new insights and discoveries.
• AI with greater robustness, interpretability, privacy preservation, factuality, safety, and security, and new evaluations to demonstrate these.
• Mechanisms for mitigating bias and other societal concerns; robust practices and techniques widely adopted to mitigate information hazards.
• AI training and use is orders of magnitude less resource intensive; AI’s contribution to solving the world’s carbon footprint exceeds its own by orders of magnitude.
• Step change in funding for research and infrastructure in academia and beyond for research on a wider range of advances, uses, safety, and implications of AI.
• Readers should add to, edit, or improve this list.
Group 2 Problems:
Realizing opportunities for humanity, addressing pressing societal challenges, delivering broadly shared benefits
The second group focuses not on AI’s technical capabilities, but on its potential to benefit humanity by assisting people, powering economic prosperity, advancing science, helping address societal challenges, and improving lives everywhere—in other words, deliver on what excites and motivates many of us about AI. (11) And so the question in 2050 will be to what extent these possibilities will have been realized and the benefits broadly shared.
Why it matters: First, the potential for AI to drive economic gains and prosperity underlies much of the excitement and investment in AI. AI is proving to be applicable in a wide spectrum of activities by individuals and by small and large organizations. AI’s use is also expected to lead to a cascade of complementary innovations and technology spillovers and to show continued improvements over time. With these generalpurpose technology attributes, AI could drive productivity for individuals, businesses, and organizations in sectors as diverse as retail, manufacturing and operations, healthcare, and the public sector, in developed and developing economies. (12) Productivity growth matters because, over time, it has been crucial for raising workers’ wages and consumers’ purchasing power, increasing demand for goods and services, and improving living standards. (13) Productivity will become even more important as populations age and countries and regions seek to leap forward in their development. Many estimate AI’s potential to boost productivity to be significant, (14) and empirical studies, albeit early and limited, seem to back this up. (15) While productivity has been the bedrock of long-term prosperity, in recent decades it has been sluggish in many countries and has not always been accompanied by levels of wage growth experienced in prior decades. This would need to change if AI-enabled productivity is to benefit workers.
To be clear, the productivity and economic gains from AI will not be automatic or guaranteed. They will require broad adoption in companies and organizations of all sizes, within and across sectors; application in productivity-enhancing uses; and AI that makes workers more productive and ushers in new economic activities at scale. They will also require investment and capital deepening, co-innovations, process and organizational changes, workforce readiness, and enabling policies. In other words, the pace and scale of the gains will depend on the choices we make. Furthermore, whether AI will be considered to have been beneficial when we look back in 2050 will also depend on whether it is improving people’s economic well-being, with gains broadly shared. This matters because, despite economic growth over the last decades, inequality has grown. AI should not make this worse, but help broaden prosperity.
On most people’s minds is AI’s impact on work—understandably, since most derive their livelihood through work. Many fear that AI will lead to technological unemployment, a concern with a long history (and a term attributed to John Maynard Keynes in the 1930s). Historically, technology has not led to economy-wide job displacement, despite significant shifts between sectors, from agriculture to manufacturing to services, and shifts in occupations, with some declining and others growing. (16) Therefore, it will matter that with AI people can sustain their livelihoods as such shifts occur. This should be possible if AI is used to assist workers more than displace them; worker skills co-evolve with AI capabilities; assistive use of AI does not depress wages, but ideally grows them as workers become more productive; and in cases where workers are displaced, they can find opportunities in growing and/or new occupations and sectors. (17) Research, including by the International Labor Organization, suggests that in the foreseeable future, AI will likely be more worker-assistive than worker-displacing. (18) Others also highlight that while technology has tended to benefit the skilled more than the less skilled, early evidence suggests generative AI may benefit the less skilled more than the skilled. Thus AI could—as David Autor put it—help “rebuild the American middle class.” (19) Here again, much will depend on choices by developers, users (especially employers), and policymakers, as Erik Brynjolffson and others highlight, (20) while acknowledging the difficulties of predicting or guiding outcomes, as Jason Furman reminds us. (21)
Second, AI has another potentially distinctive attribute—one that also motivates many AI researchers: the potential to be an invention for inventing and for solving complex problems in domains important for humanity’s progress. Science is one such domain where AI has begun to enable and accelerate landmark advances. (22) The example of Google’s AlphaFold is instructive for its ambitions and possibilities for impact: AlphaFold solved the 50-year grand challenge of protein-folding, predicting the structure of 200 million proteins; its open-access database is being used at time of writing by over two million scientists in over 190 countries, many working on neglected diseases; this year, AlphaFold3 extended to life’s other biomolecules, DNA, RNA, and ligands. (23) Other notable advances range from neuroscience (connectomics), genomics (pangenome), material science (GNoME), and physics (novel explorations of quantum gravity). (24) It’s early days, but more AI-enabled advances in science (and potentially other domains, potentially including social sciences) are likely as AI progresses and there are co-investments in science and relevant domains coupled with enabling governance and policy approaches.
Third, if in 2050 we are to respond affirmatively to the question of whether AI will have benefited society and acknowledged as such, it will also be because we will have applied AI to significant beneficial effect on major societal challenges such as the Sustainable Development Goals (SDGs). These 17 goals, adopted by UN member states in 2016, can be understood as an expression of what the world agrees are humanity’s greatest challenges and a shared vision for human flourishing. They include ending poverty and hunger, increasing health and well-being, expanding access to education, reducing gender and other inequalities, tackling climate change, and achieving sustainability—goals the world is currently not on track to achieve. (25) There is promising, but early, progress in using AI to help advance the SDGs. (26) It’s worth noting that many AI-enabled benefits are not always recognized as such; examples include the use of AI-enabled tools by over a billion people to access information across language and other barriers, flood forecasting (27) now covering hundreds of millions in over 80 countries at time of writing, tools for medical diagnosis in developed and developing countries, and developments in vaccines. Still, more is required for AI’s contributions to have made a difference when we look back in 2050. Making a difference should also include avoiding what the UN High- Level Body on AI called “missed” uses, where not using AI could lead to worsening of societal challenges, especially in contexts where alternatives are limited or simply unavailable.
Some markers of progress
• AI has enabled step-change improvements in universal access to information, knowledge, and services critical to well-being and human flourishing.
• AI has contributed to gains in productivity for individuals, businesses and other organizations, a wide range of sectors (including the public sector), and whole economies.
• Worker-assistive AI is a bigger effect than worker-displacing AI; active labor-market policies and other supports are in place for impacted workers, e.g., skilling, transition, and wage support where needed.
• Material progress in addressing capacity gaps (within and between countries) that could hinder broad participation in the development and use of AI.
• AI has not worsened or created new economic inequalities (within and between countries), and it is helping reduce such inequalities.
• A vibrant ecosystem with many more researchers, entrepreneurs, companies, and countries participating in AI’s development, use, and benefits.
• AI has contributed to significant progress on multiple SGDs, and has increased the rate and number of breakthroughs that benefit society and improve well-being.
• AI has meaningfully contributed to the cure for one or more major diseases; AI has meaningfully contributed to mitigating and adapting to climate change.
• Readers should add to, edit, or improve this list.
Group 3 Problems:
Responsible development, deployment, use of AI
This third group has to do with the responsible development, deployment, and use of AI, and governance of these. Responsible development encompasses the conduct of research and its dissemination, development of and access to frontier AI, and development of applications, especially for high-stakes domains. Deployment includes readiness and the where, whether, and how of deployment of AI-enabled systems, products, and services. Use considerations include appropriate uses and customizations of AI by individuals, companies and other entities in each sector of the economy, and by governments. Use and misuse must consider different categories of risks, which will vary by user, use case, sector, and stakes involved, as well as sociotechnical embedding, transparency, and choice in different contexts. (28) Equally important are ethical considerations and human involvement and oversight, especially as AI becomes more capable and agentive. (29)
Why it matters: As an early-stage technology, AI still has the possibility of performance and safety risks, misapplication, and misuse. Use concerns include misinformation, deception and manipulation, and cybercrime; use in surveillance and autonomous weapons; and (in)appropriate use by some users such as minors. Concerns about bad actors—especially in so-called “your money or your life,” cybersecurity, or national security contexts—could hinder the open science and development that has historically catalyzed innovation and enabled many to participate in research, development, improvements, and evaluations (e.g., in Linux and Android). Amid the current debate around open-sourcing AI, some express concern regarding more capable systems, (30) while others conclude that in current generations of AI, the benefits of open-sourcing exceed the marginal risks of what bad actors could do. (31) However, this latter view may not hold in the future as AI becomes more capable, agentive, and connected to other systems, including infrastructure.
The scale of AI’s potential economic impact will undoubtedly spur competition between companies—providers of AI infrastructure and tools; developers of models, applications and services; those using AI in retail, financial services, manufacturing, healthcare, and other sectors—and between nations seeking comparative advantages in economic, geopolitical, and national security spheres. Competition should be encouraged if and where it broadens participation in development and use of AI, drives innovation and diffusion of beneficial applications, boosts productivity, and leads to greater and wider prosperity. On the other hand, hypercompetitive dynamics could get in the way of responsible development, deployment, and use of AI; focus on AI in the public interest; sharing of best practices; and coordinated or collective action on safety between competing companies and countries.
It will matter that we get governance—policies, incentives, regulations, norms, and practices—right. Governance, at national or international level, will need to solve for two things simultaneously: (1) address the risks and complexities associated with AI, and (2) enable innovation and the beneficial possibilities of AI—in other words, governance should focus on AI’s possibilities, not just its perils. (32) Governance should make use of established laws and agreements wherever possible and be adaptive to rapid and novel advances in AI and its evolving use. And since AI will involve not just developers, but also infrastructure providers, deployers, and users (individuals, companies and other organizations, and governments), it will be important that governance involve and apply to all. It must also engage other stakeholders (including academia and civil society), especially those who have not been involved thus far in some communities and in the Global South.
Some markers of progress
• National (and regional) AI governance and regulations have evolved to enable innovation and beneficial use of AI and to address the risks in its development, deployment, and use—and can adapt to developments in AI and its use.
• Global adoption of (or significant progress toward) internationally harmonized AI standards, risk frameworks, and industry practices, including in early warnings and safety.
• Robust and broadly deployed mechanisms for detecting, reporting, and mitigating misuse of AI for misinformation and cybercrime; chemical, biological, radiological, and nuclear (CBRN) risks; and critical infrastructure-related risks.
• Widely adopted mechanisms that encourage open innovation, access, and wide participation in AI research, development, and use, while limiting risks of bad actors taking advantage of such openness.
• International governance based on broadly agreed-upon principles and goals (e.g., Human Rights Law, the SDGs) and functions that should be coordinated internationally (e.g., a scientific panel to assess risks and opportunities). (33)
• For beneficial uses of AI that fall outside of commercial pursuit and where there might be market failures, government support, subsidies, incentives, or other mechanisms have enabled their development, deployment, and use.
• Readers should add to, edit, or improve this list.
Group 4 Problems:
Co-evolution of societal systems and what it means to be human in the age of AI
The fourth group revolves around two closely related challenges. First, as capable AI comes to touch many aspects of human life, our societal systems, social contracts, civic participation, education, and other institutions will need to adapt. This will require consideration of not only what could be lost, but also what could be gained through the use of AI. The second challenge concerns what it means to be human in the age of AI, and how we think about work, social relations, achievement, purpose, and more. It will require us to reflect on what it means to be intelligent, creative, or cognitively human when many of the ways we have defined these characteristics of ourselves increasingly can be imitated or even, in the future, done better or better done by machines.
Why it matters: First, the potential for AI to strain, disrupt, and reshape our societal arrangements in beneficial and challenging ways is significant. Education serves as a potent harbinger here. In a recent survey of U.S. K-12 teachers, over 30% indicated that they used AI in the classroom, while another survey found that 40% of students have used AI for their schoolwork. (34) Schools and universities are already experimenting with how to adapt, from changing testing methods to assigning projects that encourage students to use AI. As AI advances and adoption grows, how we learn and teach, the classroom, school, university, and other learning spaces, may be profoundly different when we look back in 2050. Other aspects may require more foundational (re)considerations. For example, drawing on the political philosophy of John Rawls, Iason Gabriel argues that AI, when integrated into the structure of society, should uphold principles of fair allocation of resources, opportunities, and benefits within a society. This perspective emphasizes that AI must not only support citizens’ rights but also contribute to equality of opportunity and wellbeing of the least advantaged members of society. Equally important are ground truth, trust, and processes in a democracy in the age of AI. (35)
Second, we may be faced with the most challenging question of them all: what it means to be human in the age of AI, what it means to flourish and live a fulfilling life. We have always taken pride in being the only species endowed with reason and, thus, uniquely capable of scientific discovery, invention, and artistic expression. We have also often, albeit with less justification, considered our capacities for feeling, experiencing, and social relationships to be uniquely human. AI will increasingly reshape and challenge many such assumptions, making AI a far-reaching philosophical challenge, as well as a philosophical mirror, a laboratory even, for us to ask questions about us. (36) Questions arise as well about who and what shapes the answers, since in the past technology has sometimes universalized particular conceptions of what it means to be human and to progress, often at the exclusion of other ways of being human and of progressing—something we must get right with AI. (37)
It is likely that many aspects of ourselves may be assisted by AI, thereby changing our sense of them, of ourselves, and of what attributes we value, individually and collectively. Some may choose to create boundaries within which some aspects remain human-only, while others may choose to fully embrace the inclusion of AI. One possible outcome would be developing systems that do not outsource but boost human intellectual and creative capability—a kind of “steam engine for the mind,” as Reid Hoffman suggests. (38) There will likely be disagreement about what constitutes outsourcing versus boosting, even about what is human versus not. Just as when technical breakthroughs resulted in new genres of art, there were similar debates about whether the new was art, craft, or neither. (39) Such debates will undoubtedly emerge in the age of AI.
Some markers of progress
• Sectors, institutions, and systems that provide services (e.g., education, healthcare, public services) have adapted to make appropriate use of AI’s capabilities, while also delineating societally agreed-upon boundaries.
• Institutions that shape societal arrangements and social contracts, such as legal systems, governments, and sociopolitical processes, have incorporated the beneficial capabilities of AI and instituted mechanisms to mitigate its risks.
• Effective mechanisms have emerged for individuals and communities with different beliefs about boundaries (e.g., degrees of AI augmentation) to coexist and thrive.
• Language and mental models for what it means to be human and to flourish alongside highly capable AI have emerged and are being robustly debated.
• One or more celebrated genre of AI-enabled art, or other creative endeavor, has emerged with many practitioners and new institutions like museums.
• Interfaces to help humans think in dimensions they were not capable of before emerge: interfaces that allow—for a large number of humans—unprecedented levels of cognitive height that neither humans nor AI could achieve on their own.
• Readers should add to, edit, or improve this list.
Group 5 Problems:
Alignment with increasingly powerful and capable AI
This group concerns satisfactorily addressing the challenges of human alignment and compatibility with increasingly powerful AI. This includes aligning goals and tackling gain-of-function risks, issues of control, human-machine cooperation, and complexities of multiagent systems. And there is also the age-old challenge of human preferences themselves, their normativity and plasticity—this time in a new context where (mis)alignment could have outsize effects.
Why it matters: Ensuring that AI is aligned and compatible with humanity and its flourishing will become increasingly important and consequential as AI becomes more powerful, more agentive (involving subgoals, planning and actions, connections to other systems), self-improving, and with capabilities (and arguably, intelligence) outside those of humans. While there is debate among AI experts on how general and super intelligence is defined, the technical community is increasingly in agreement that the prospect of it is no longer a far distant or unrealizable prospect.
So what we do now and going forward to ensure that the goals and outcomes of more capable artificial intelligences are compatible with, and enabling of, human preferences, widely held values, human rights and dignities, and human flourishing is critical. This “alignment problem” is as old as the AI field itself, discussed early on in 1951 by Alan Turing and recently by a growing community of AI researchers including Russell. Russell proposes an approach that embraces and makes use of the inherent ambiguity of human goals. (40) But the challenge of alignment may be as much about socio-ethical challenges as it is about the technical challenges. There are foundational questions about what we actually mean and want—individually and collectively—with respect to alignment. (41) Human values, individually and at various societal aggregations—from community to nation and beyond—will continue to be diverse, often contradicting each other. Alignment also grows in complexity when it involves not only the plurality of human preferences, but also a near certain future involving a plurality of highly capable AIs.
Ultimately, success in this group of problems may center on what is prevented. There is much discussion about potential catastrophic risks from un- or misaligned AI systems. While the probabilities may be low, the severity of possible risks demands serious attention to the choices and path we chart for AI research and development.
Some markers of progress
• Robust technical methods are developed for aligning individual AI systems with individual or collective human preferences.
• International agreement to base AI alignment on already agreed-upon universal norms including Human Rights and other International law.
• Societally agreed-upon principles that also allow wide latitude for the plurality of individual and societal values and preferences—a tall order!
• Practical methods have been developed and ongoing research established to better understand interactions among communities of intelligent agents.
• Ongoing and robust research, assessment, and testing for likely or emerging AI capabilities in advance of achieving them, for single and multiple AI systems.
• Robust mechanisms for contending with uncontrolled gain of function, goal-drift, and corruption, deception, and manipulation by capable AI systems.
• Robust monitoring, incidence reporting, and response mechanisms, as well as pre-agreed protocols for the emergence of superintelligent capabilities.
• Readers should add to, edit, or improve this list.
From Thought Experiment to Action
The problems presented here may well be an idiosyncratic view of what we must get right if AI is to have been a net positive for humanity when we look back in 2050. Readers will undoubtedly have their own views of what we must get right and what will constitute progress. There should be vigorous discussion, debate, and iteration to get to better and, ideally, shared lists of what matters most to get right. While such lists will likely evolve as AI advances, its uses evolve, and society’s experience with it grows, the work to get AI right must not wait. It must be taken on now with a focus on not just what could go wrong, but also, and importantly, what could go right and how we shape it in the face of some unknowns. It is work that must involve everyone—researchers, developers and users of AI, the private and public sector, academia, civil society, and governments—so we should get on with it.
Footnotes
(1) The author is grateful to readers who provided feedback on an early draft and to the many others whose insights informed the Working List of Hard Problems, collaborators on many related initiatives and the works cited here. The author is also indebted to Kerry McHugh, who helped edit this paper. The views expressed herein are the author’s and do not necessarily reflect those of Google or Alphabet.
(2) “Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence,” Exec. Order No. 14110, 88 Fed. Reg. 75191 (October 30, 2023), https://www.whitehouse.gov/briefing-room/presidential-actions/2023/10/30/ executive-order-on-the-safe-secure-and-trustworthy-development-and-use-ofartificial- intelligence/; “The Bletchley Declaration by Countries Attending the AI Safety Summit 1–2 November 2023,” Gov.uk, November 1, 2023, https://www.gov.uk/government/ publications/ai-safety-summit-2023-the-bletchley-declaration/the-bletchleydeclaration- by-countries-attending-the-ai-safety-summit-1-2-november-2023; “Hiroshima Process International Code of Conduct for Organizations Developing Advanced AI Systems,” G7 Hiroshima Summit, October 2023, https://www.mofa.go.jp/ files/100573473.pdf; European Artificial Intelligence Act, O.J.L. 2024/1689, 12.7.2024 (2024), https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=OJ:L_202401689; U.S. National AI Advisory Committee Recommendations, 2023–2024 (see “National AI Advisory Committee,” AI.gov, accessed July 11, 2024, https://ai.gov/naiac/).
(3) UN G.A. Res. A/78/L.49 (March 11, 2024); UN High-Level Advisory Body on Artificial Intelligence, Interim Report: Governing AI for Humanity (United Nations, December 2023), https://www.un.org/sites/un2.un.org/files/un_ai_advisory_body_governing_ai_for_ humanity_interim_report.pdf.
(4) “Global Public Opinion on Artificial Intelligence (GOA-AI),” Schwartz Reisman Institute for Technology and Society, University of Toronto, May 2024, https://srinstitute. utoronto.ca/public-opinion-ai; “Google / Ipsos Multi-country Survey on AI,” Ipsos, January 16, 2024, https://www.ipsos.com/en-us/google-ipsos-multi-country-ai-survey.
(5) James Manyika, “Getting AI Right: Introductory Notes on AI & Society,” and essays by others in Daedalus 151, no. 2 (Spring 2022), https://direct.mit.edu/daed/issue/151/2, a special volume on AI & Society edited by James Manyika.
(6) “Working List of Hard Problems in AI, compiled by James Manyika for the AI2050 Initiative,” AI2050, first compiled 2020, accessed July 11, 2024, https://ai2050.schmidtsciences. org/hard-problems/.
(7) For a 20-year timeline of Google AI milestones and advances, see “Our AI Journey,” Google AI, accessed July 11, 2023, https://ai.google/ai-milestones?section=gan.
(8) See Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin, “Attention is All You Need,” in Proceedings of the 31st International Conference on Neural Information Processing Systems (Red Hook, NY: Curran Associates, Inc., 2017): 6000–6010, https://proceedings.neurips.cc/ paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
(9) For the current state of advances, see Nestor Maslej, Loredana Fattorini, Raymond Perrault, et al., The AI Index 2024 Annual Report (Stanford Institute for Human-Centered AI, April 2024), https://aiindex.stanford.edu/wp-content/uploads/2024/05/HAI_ AI-Index-Report-2024.pdf.
(10) Laura Weidinger, John Mellor, Maribeth Rauh, et al., “Ethical and Social Risks of Harm from Language Models,” arXiv, December 8, 2021, https://doi.org/10.48550/ arXiv.2112.04359.
(11) James Manyika, Jeff Dean, Demis Hassabis, et al., “Why We Focus on AI (and to What End),” Google, January 2023, https://ai.google/static/documents/google-whywe- focus-on-ai.pdf.
(12) James Manyika and Michael Spence, “The Coming AI Economic Revolution: Can AI Reverse the Productivity Slowdown?,” Foreign Affairs, October 24, 2023, https://www. foreignaffairs.com/world/coming-ai-economic-revolution.
(13) Manyika and Spence, “The Coming AI Economic Revolution”; “Self-reported Life Satisfaction vs. GDP per Capita, 2022,” Our World in Data, accessed July 11, 2024, https://ourworldindata.org/grapher/gdp-vs-happiness; John Helliwell, Richard Layard, Jefferey D. Sachs, Lara B. Aknin, Jan-Emmanuel De Neve, and Shun Wang, eds., World Happiness Report 2023 (11th ed.) (Sustainable Development Solutions Network, 2023) https://worldhappiness.report/ed/2023/.
(14) Michael Chui, Eric Hazan, Roger Roberts, Alex Singla, Kate Smaje, Alex Sukharevsky, Lareina Yee, and Rodney Zemmel, The Economic Potential of Generative AI: The Next Productivity Frontier (McKinsey & Company, June 2023), https://www.mckinsey.com/ capabilities/mckinsey-digital/our-insights/the-economic-potential-of-generativeai- the-next-productivity-frontier; Joseph Briggs, Devesh Kodnani, Jan Hatzius, and Giovanni Peirdomenico, “Upgrading Our Longer-Run Global Growth Forecasts to Reflect the Impact of Generative AI,” Goldman Sachs Research, October 29, 2023, https://www.gspublishing.com/content/research/en/reports/2023/10/30/2d567ebf- 0e7d-4769-8f01-7c62e894a779.html; Andrew McAfee, “Generally Faster: The Economic Impact of Generative AI,” Google, April 25, 2024, https://policycommons. net/artifacts/12281693/generally_faster_-_the_economic_impact_of_generative_ ai/13175782/.
(15) Erik Brynjolfsson, Danielle Li, and Lindsey R. Raymond, “Generative AI at Work,” National Bureau of Economic Research Working Paper 31161, April 2023, doi.10.3386/ w31161; Shakked Noy and Whitney Zhang, “Experimental Evidence on the Productivity Effects of Generative Artificial Intelligence,” Science 381 (2023):187–192, doi:10.1126/ science.adh2586.
(16) Guy Ben-Ishai, Jeff Dean, James Manyika, Ruth Porat, Hal Varian, and Kent Walker, “AI and the Opportunity for Shared Prosperity: Lessons from the History of Technology and the Economy,” arXiv, Feburary 2024, https://doi.org/10.48550/arXiv.2401.09718.
(17) Erik Brynjolffson, Adam Thierer, and Daron Acemoglu, Navigating the Future of Work: Perspectives on Automation, AI and Economic Prosperity (American Enterprise Institute, March 2024), https://www.aei.org/wp-content/uploads/2024/03/Navigatingthe- Future-of-Work-Perspectives-on-Automation-AI-and-Economic-Prosperity. pdf?x85095.
(18) Paweł Gmyrek, Janine Berg, and David Bescond, Generative AI and Jobs: A Global Analysis of Potential Effects on Job Quantity And Quality, ILO Working Paper 96 (International Labour Organization, August 2023), https://www.ilo.org/publications/generative- ai-and-jobs-global-analysis-potential-effects-job-quantity-and.
(19) David Autor, Applying AI to Rebuild Middle Class Jobs, NBER Working Paper 32140 (National Bureau of Economic Research, 2024), doi:10.3386/w32140.
(20) Erik Brynjolfsson, “The Turing Trap: The Promise & Peril of Human-Like Artificial Intelligence,” Daedalus 151, no. 2 (2022): 272–287, https://doi.org/10.1162/daed_a_01915; Daron Acemoglu, David Autor, and Simon Johnson, Can We Have Pro-Worker AI? Choosing a Path of Machines in Service of Minds (Shaping the Future of Work, September 2023), https://shapingwork.mit.edu/wp-content/uploads/2023/09/Pro-Worker- AI-Policy-Memo.pdf.
(21) Jason Furman, “Policies for the Future of Work Should Be Based on Its Past and Present,” The American Worker Project, Economic innovation Group, July 2024, https:// eig.org/wp-content/uploads/2024/07/TAWP-Furman.pdf.
(22) Hanchen Wang, Tianfan Fu, Yuanqi Du, et al., “Scientific Discovery in the Age of Artificial Intelligence,” Nature 620 (2023): 47–60, https://doi.org/10.1038/s41586-023-06221-2; Science in the Age of AI: How Artificial Intelligence is Changing the Nature and Method of Scientific Research (The Royal Society, 2024), https://royalsociety.org/-/media/policy/ projects/science-in-the-age-of-ai/science-in-the-age-of-ai-report.pdf.
(23) John Jumper, Richard Evans, Alexander Pritzel, et al., “Highly Accurate Protein Structure Prediction with AlphaFold,” Nature 596 (2021): 583–589, https://doi. org/10.1038/s41586-021-03819-2; Josh Abramson, Jonas Adler, Jack Dunger, et al., “Accurate Structure Prediction Of Biomolecular Interactions with AlphaFold 3,” Nature 630 (2024): 493–500, https://doi.org/10.1038/s41586-024-07487-w.
(24) Alexander Shapson-Coe, Michał Januszweski, Daniel R. Berger, et al., “A Petavoxel Fragment of Human Cerebral Cortex Reconstructed at Nanoscale Resolution,” Science 384, no. 6696 (2024), doi:10.1126/science.adk4858; Wen-Wei Liao, Mobin Asri, Jana Ebler, et al., “A Draft Human Pangenome Reference,” Nature 617 (2023): 312–324, https://doi.org/10.1038/s41586-023-05896-x; Amil Merchant, Simon Batzner, Samuel S. Schoenholz, Muratahan Aykolm, Gowoon Cheon, and Ekin Dogus Cubuk, “Scaling Deep Learning for Materials Discovery,” Nature 624 (2023): 80–85, https://doi.org/10.1038/ s41586-023-06735-9.
(25) The Sustainable Development Goals Report 2023: Special Edition: Towards a Rescue Plan for People and Planet (United Nations, 2023), https://unstats.un.org/sdgs/report/2023/ The-Sustainable-Development-Goals-Report-2023.pdf.
(26) AI for Good, United Nations Activities on Artificial Intelligence (AI) (International Telecommunications Union, 2022), https://www.itu.int/dms_pub/itu-s/opb/gen/S-GENUNACT- 2023-PDF-E.pdf; Brigitte Hoyer Gosselink, Kate Brandt, Marian Croak, et al., AI in Action: Accelerating Progress Towards the Sustainable Development Goals (Google, 2024), https://static.googleusercontent.com/media/publicpolicy.google/en//resources/ research-brief-ai-and-SDG.pdf; Medha Bankhwal, Ankit Bisht, Michael Chui, Roger Roberts, and Ashley van Heteren, AI for Social Good: Improving Lives and Protecting the Planet (McKinsey & Company, May 2024), https://www.mckinsey.com/capabilities/ quantumblack/our-insights/ai-for-social-good#/.
(27) Grey Nearing, Deborah Cohen, Vusumuzi Dube, et al., “Global Prediction of Extreme Floods in Ungauged Watersheds,” Nature 627 (2024): 559–563, https://doi.org/10.1038/ s41586-024-07145-1.
(28) Committee on Responsible Computing Research, Fostering Responsible Computing Research (National Academies of Sciences, Engineering, Medicine, 2022), https://nap. nationalacademies.org/catalog/26507/fostering-responsible-computing-researchfoundations- and-practices.
(29) Iason Gabriel, Arianna Manzini, Geoff Keeling, et al., “The Ethics of Advanced AI Assistants,” arXiv, April 24, 2024, https://doi.org/10.48550/arXiv.2404.16244; Jonathan Zittrain, “We Need to Control AI Agents Now,” Atlantic, July 2, 2024, https://www. theatlantic.com/technology/archive/2024/07/ai-agents-safety-risks/678864/.
(30) Lawrence Lessig, “Not All AI Models Should Be Freely Available, Argues a Legal Scholar,” Economist, July 29, 2024, https://www.economist.com/by-invitation/2024/07/29/notall- ai-models-should-be-freely-available-argues-a-legal-scholar.
(31) Sayash Kapoor, Rishi Bommasani, Kevin Klyman, et al., “On the Societal Impact of Open Foundation Models,” arXiv, February 27, 2024, https://doi.org/10.48550/ arXiv.2403.07918; Christopher A. Mouton, Caleb Lucas, and Ella Guest, The Operational Risks of AI in Large Scale Biological Attacks (RAND Corporation, 2023), https://www. rand.org/pubs/research_reports/RRA2977-1.html.
(32) Alondra Nelson, “The Right Way to Regulate AI: Focus on its Possibilities, Not its Perils,” Foreign Affairs, January 12, 2024, https://www.foreignaffairs.com/united-states/rightway- regulate-artificial-intelligence-alondra-nelson.
(33) See UN High-Level Advisory Body, Interim Report: Governing AI for Humanity, for five principles and seven functions recommended by the Body, and the Body’s final report, forthcoming in September 2024.
(34) Lauraine Langreo, “Teachers Told Us They’ve Used AI in the Classroom. Here’s Why,” EducationWeek, January 5, 2024, https://www.edweek.org/technology/teacherstold- us-theyve-used-ai-in-the-classroom-heres-why/2024/01; “Instructure Survey Shows Most Teachers, Students Are Optimistic about AI in the Classroom,” Instructure, September 21, 2023, https://www.prnewswire.com/news-releases/instructure-surveyshows- most-teachers-students-are-optimistic-about-ai-in-the-classroom-301935362. HTML.
(35) Iason Gabriel, “Toward a Theory of Justice for Artificial Intelligence,” Daedalus 151, no. 2 (2022): 218–231, https://doi.org/10.1162/daed_a_01911; Cynthia Dwork and Martha Minow, “Distrust of AI: Sources and Responses from Computer Science and Law,” Daedalus 151, no. 2 (2022): 309–321, https://doi.org/10.1162/daed_a_01918; Sonia Katyal, “Democracy and Distrust in an Era of AI,” Daedalus 151, no. 2 (2022): 322–334, https://doi.org/10.1162/daed_a_01919; “Participation of the Holy Father Francis at the G7 in Borgo Egnazia, 06.14.2024,” Holy See Press Office, June 14, 2017, https:// press.vatican.va/content/salastampa/it/bollettino/pubblico/2024/06/14/0504/01030. html#en.
(36) Daniel Dennet, “When Philosophers Encounter Artificial Intelligence,” Daedalus 117, no. 1 (1988): 283–295, https://www.jstor.org/stable/20025148; Tobias Rees, “Non-Human Words: On GPT as a Philosophical Laboratory” Daedalus 151, no. 2 (2022): 168–182, https://doi.org/10.1162/daed_a_01908; Blaise Agüera y Arcas, “Our Attitudes towards AI Reveal How We Really Feel about Human Intelligence,” Guardian, July 3, 2024, https://www.theguardian.com/technology/article/2024/jul/03/ai-human-intelligence.
(37) Michele Elam, “Signs Taken for Wonders: AI, Art & the Matter of Race,” Daedalus 151, no. 2 (2022): 198–217, https://doi.org/10.1162/daed_a_01910.
(38) Reid Hoffman,“Gen AI: A Cognitive Industrial Revolution,” interview by Lareina Yee (podcast), McKinsey Digital, June 7, 2024, https://www.mckinsey.com/capabilities/ mckinsey-digital/our-insights/gen-ai-a-cognitive-industrial-revolution.
(39) Blaise Agüera y Arcas, “Art in the Age of Machine Intelligence,” Arts 6, no. 4, (October 2017): 18, https://doi.org/10.3390/arts6040018; See the work of artists Refik Anadol, https://refikanadol.com, and Stephanie Dinkins, https://www.stephaniedinkins.com.
(40) Stuart Russell, “If We Succeed,” Daedalus 151, no. 2 (2022): 43–57, https://doi. org/10.1162/daed_a_01899; Stuart Russell, Human Compatible: Artificial Intelligence and the Problem of Control (Penguin, 2020).
(41) Iason Gabriel, “Artificial Intelligence, Values, and Alignment,” Minds and Machines 30, no. 3 (October 2020): 411–457, https://doi.org/10.1007/s11023-020-09539-2.