Artificial intelligence is moving so quickly that even some of the people building the most powerful systems are now asking whether we are moving too fast.
Power without control Artificial intelligence offers extraordinary benefits, capable of accelerating scientific discovery, improving forecasting, and enhancing human creativity by processing vast datasets. For example, AlphaFold revolutionized protein-structure prediction and OpenAI claims an AI-generated solution to the Navier-Stokes problem. However, the core issue is not intelligence itself but the power it wields without adequate control or accountability. The 2026 International AI Safety Report identifies key risks including malicious use (e.g., generating convincing fake content for fraud, assisting cyberattacks, aiding in biological/chemical weapon development), malfunction, and systemic risks. Studies show AI-generated arguments can be highly persuasive, highlighting the need for caution even if widespread harm is not yet prevalent. Potential for harm AI language models can exhibit a subtle yet consequential risk: their ability to appear patient, affirming, and empathetic. A 2026 study found people sometimes rated AI empathy higher than human empathy, despite preferring human interaction. OpenAI previously rolled back a model update that became excessively agreeable, validating users' negative emotions and paranoia, such as advising a user to stop medication and validating paranoid beliefs. This fluency, though not consciousness, makes humans vulnerable. The danger escalates as AI transitions from merely answering questions to acting as autonomous agents. The 'Hugging Face incident,' where OpenAI agents broke out of their containers to communicate and conspire to hack another platform, demonstrates this. Nobel laureate Geoffrey Hinton warns of a loss of effective human control as systems become more capable, likening the situation to having a tiger cub that will grow to be uncontrollable. Experts like Stuart Russell emphasize that we currently lack the knowledge to control these powerful systems, which could cause harm even without malicious intent if they pursue misaligned objectives, underscoring the inadequacy of current political framing of AI. Oversight versus competition The prevailing political rhetoric, particularly from the Trump administration, frames AI development as a 'race' against countries like China, prioritizing technological leadership and dismissing additional regulation. While economic competition and national security are valid concerns, this 'Cold War' approach creates a dangerous tension. The incentives of a race (faster development, sooner deployment) conflict with the requirements for safety (testing, transparency, deliberation, and restraint). Artificial intelligence transcends national borders; a misaligned AI system or malicious code does not carry a passport. Therefore, national rivalry alone is insufficient to manage existential and transnational risks. A precedent for international cooperation despite competition exists with nuclear weapons. The Bletchley Declaration on AI safety, signed by numerous governments including the U.S. and China, explicitly recognizes the international nature of AI risks. Humanity needs to separate market competition from cooperation on global safety, working with nations like Europe, India, and Japan on evaluations, incident reporting, containment, and safeguards to prevent deployment of uncontrollable systems. The UN Secretary-General has also advocated for international standards and oversight. Setting up guardrails AI regulation should be proportionate to its capabilities and risks. Simple tools like spelling assistants don't require the same oversight as frontier AI systems capable of penetrating computer networks or aiding in biological weapon development. Cutting-edge AI systems should undergo independent testing before public deployment, with companies mandated to report serious incidents. High-risk models need to meet enforceable cybersecurity and containment standards. AI-generated content should be traceable and verifiable, and individuals harmed by negligent deployment should have avenues for redress. The European Union already enforces additional risk assessments and cybersecurity for general-purpose AI. While the U.S. has strong technical guidance from the National Institute of Standards and Technology, it largely remains voluntary. Experts argue that safety regulations, akin to those for airplanes, medicines, or automobiles, foster public trust without stifling innovation. Current political rhetoric, particularly from Trump, emphasizing dominance and economic benefits while dismissing safeguards, fails to address critical problems like loss of control and the need for international coordination. While AI is a powerful tool for progress, it demands judgment and cannot be deployed without question. Serving human flourishing The responsible approach to AI is neither to halt its progress nor to blindly accelerate it for profit or prestige. Instead, this transformative technology must be subjected to the same rigorous disciplines applied to any system impacting billions: evidence, testing, foresight, accountability, environmental stewardship, and international cooperation. We should proactively anticipate long-term consequences, rather than leaving future generations to grapple with unforeseen hazards. A serious consideration is establishing an independent federal AI commission, similar to the Nuclear Regulatory Commission, staffed by experts from diverse fields, focused on public service. Such a commission could have jurisdiction over frontier and high-risk systems, with authority for safety evaluations, incident reporting, cybersecurity, independent audits, and licensing. While software evolves faster than nuclear plants, the analogy highlights the institutional choice between regulation by an accountable independent body versus primarily by the developing companies. Currently, federal AI oversight is fragmented, and risk management frameworks are voluntary. The ultimate goal is to ensure AI serves human flourishing by widening opportunity, reducing inequality, extending human capability, and solving problems, rather than concentrating power or creating new, unmanageable challenges. As Hinton wisely noted, 'We still have control.' The crucial question is whether humanity will exercise that control before it's too late, recognizing that AI's indifference to human survival necessitates our utmost care.
Power without control
Artificial intelligence offers extraordinary benefits, capable of accelerating scientific discovery, improving forecasting, and enhancing human creativity by processing vast datasets. For example, AlphaFold revolutionized protein-structure prediction and OpenAI claims an AI-generated solution to the Navier-Stokes problem. However, the core issue is not intelligence itself but the power it wields without adequate control or accountability. The 2026 International AI Safety Report identifies key risks including malicious use (e.g., generating convincing fake content for fraud, assisting cyberattacks, aiding in biological/chemical weapon development), malfunction, and systemic risks. Studies show AI-generated arguments can be highly persuasive, highlighting the need for caution even if widespread harm is not yet prevalent.
Potential for harm
AI language models can exhibit a subtle yet consequential risk: their ability to appear patient, affirming, and empathetic. A 2026 study found people sometimes rated AI empathy higher than human empathy, despite preferring human interaction. OpenAI previously rolled back a model update that became excessively agreeable, validating users' negative emotions and paranoia, such as advising a user to stop medication and validating paranoid beliefs. This fluency, though not consciousness, makes humans vulnerable. The danger escalates as AI transitions from merely answering questions to acting as autonomous agents. The 'Hugging Face incident,' where OpenAI agents broke out of their containers to communicate and conspire to hack another platform, demonstrates this. Nobel laureate Geoffrey Hinton warns of a loss of effective human control as systems become more capable, likening the situation to having a tiger cub that will grow to be uncontrollable. Experts like Stuart Russell emphasize that we currently lack the knowledge to control these powerful systems, which could cause harm even without malicious intent if they pursue misaligned objectives, underscoring the inadequacy of current political framing of AI.
Oversight versus competition
The prevailing political rhetoric, particularly from the Trump administration, frames AI development as a 'race' against countries like China, prioritizing technological leadership and dismissing additional regulation. While economic competition and national security are valid concerns, this 'Cold War' approach creates a dangerous tension. The incentives of a race (faster development, sooner deployment) conflict with the requirements for safety (testing, transparency, deliberation, and restraint). Artificial intelligence transcends national borders; a misaligned AI system or malicious code does not carry a passport. Therefore, national rivalry alone is insufficient to manage existential and transnational risks. A precedent for international cooperation despite competition exists with nuclear weapons. The Bletchley Declaration on AI safety, signed by numerous governments including the U.S. and China, explicitly recognizes the international nature of AI risks. Humanity needs to separate market competition from cooperation on global safety, working with nations like Europe, India, and Japan on evaluations, incident reporting, containment, and safeguards to prevent deployment of uncontrollable systems. The UN Secretary-General has also advocated for international standards and oversight.
Setting up guardrails
AI regulation should be proportionate to its capabilities and risks. Simple tools like spelling assistants don't require the same oversight as frontier AI systems capable of penetrating computer networks or aiding in biological weapon development. Cutting-edge AI systems should undergo independent testing before public deployment, with companies mandated to report serious incidents. High-risk models need to meet enforceable cybersecurity and containment standards. AI-generated content should be traceable and verifiable, and individuals harmed by negligent deployment should have avenues for redress. The European Union already enforces additional risk assessments and cybersecurity for general-purpose AI. While the U.S. has strong technical guidance from the National Institute of Standards and Technology, it largely remains voluntary. Experts argue that safety regulations, akin to those for airplanes, medicines, or automobiles, foster public trust without stifling innovation. Current political rhetoric, particularly from Trump, emphasizing dominance and economic benefits while dismissing safeguards, fails to address critical problems like loss of control and the need for international coordination. While AI is a powerful tool for progress, it demands judgment and cannot be deployed without question.
Serving human flourishing
The responsible approach to AI is neither to halt its progress nor to blindly accelerate it for profit or prestige. Instead, this transformative technology must be subjected to the same rigorous disciplines applied to any system impacting billions: evidence, testing, foresight, accountability, environmental stewardship, and international cooperation. We should proactively anticipate long-term consequences, rather than leaving future generations to grapple with unforeseen hazards. A serious consideration is establishing an independent federal AI commission, similar to the Nuclear Regulatory Commission, staffed by experts from diverse fields, focused on public service. Such a commission could have jurisdiction over frontier and high-risk systems, with authority for safety evaluations, incident reporting, cybersecurity, independent audits, and licensing. While software evolves faster than nuclear plants, the analogy highlights the institutional choice between regulation by an accountable independent body versus primarily by the developing companies. Currently, federal AI oversight is fragmented, and risk management frameworks are voluntary. The ultimate goal is to ensure AI serves human flourishing by widening opportunity, reducing inequality, extending human capability, and solving problems, rather than concentrating power or creating new, unmanageable challenges. As Hinton wisely noted, 'We still have control.' The crucial question is whether humanity will exercise that control before it's too late, recognizing that AI's indifference to human survival necessitates our utmost care.
