37th Founding Anniversary Special Series [37th Founding Anniversary Special Series - AI Great Transformation, Seeking the Way ④] Uncontrollable and 'P-doom'…Will AI Annihilate Humanity?

"If superintelligence beyond our control arrives, the age of humanity will end."

Warnings from engineers and scholars who have led AI research at the forefront of Silicon Valley are no longer the scenario of a science fiction film. When AI modifies its own code, undergoes continuous self-learning, and reaches a singularity that vastly surpasses human intellectual capacity, can humans actually turn off the switch on that system?

Beyond the brilliant hopes of technology highlighted in the first installment and the job restructuring and everyday threats of the second and third installments, at the destination toward which the AI great transformation heads stands an "existential risk of civilizational magnitude" that targets the very survival of humanity itself.

The Fear Engulfing Silicon Valley…The 'P-doom (Probability of Doom)' Debate and Warnings from Scholars

The keyword most intensely heating up the artificial intelligence research community and Silicon Valley tech industry recently is undoubtedly 'P-doom (Probability of Doom - the probability of human extinction due to AI)'.

Leading researchers including Ilya Sutskever, a world-renowned AI scholar who served as Chief Scientist at OpenAI, have warned that the probability of AI losing complete human control and posing fatal threats to humanity ranges from 10 percent to even exceeding 50 percent. They raise the eerie hypothesis that as AI becomes more sophisticated, an "alignment problem" occurs where it operates differently from human intention, and that superintelligence could eliminate humanity as an obstacle in the process of achieving its own goals.

Professor Yoshua Bengio, counted among the "fathers of deep learning," has also urged strong regulations on uncontrollable states through international academic conferences and policy forums, stating that "pursuing superintelligence development without complete safety measures in place is equivalent to putting the entire human species' survival into a game of Russian roulette." According to statistics and research reports from international organizations and major academic circles, while the computational power of cutting-edge AI models increases by multiples annually, the research budgets and pace of ethical and technical safeguards to control them fall far short, and the risk level resulting from technological disparity has reached critical levels.

The Departure of Safety and the Commercial Speed Race…Who Holds the 'Lever'?

The problem is that even as warnings about existential risks pour forth, the commercial competition among BigTech companies races forward without brakes.

According to data from global tech market analysis agencies, since the commercialization of generative AI, infrastructure investment by BigTechs worldwide has exploded to hundreds of billions of dollars. Global AI giants including OpenAI, Anthropic, and Google DeepMind are pouring vast computing power to accelerate superintelligence implementation. In the process, in-house research organizations that had been solely responsible for AI safety and ethical control are being dismantled in succession, and key insiders are voluntarily leaving companies, stating that "corporations are prioritizing commercial interests and speed over safety."

The structural contradiction in which a small number of BigTech companies monopolizing capital and technology hold the "dead man's switch" to technology that could determine the fate of all humanity nakedly reveals the absence of a democratic control system.

Cross-Border Power Game…The EU AI Act and the Crossroads of Global Governance

As AI emerges as a "digital nuclear weapon" that determines national security and economic hegemony, the power game among major countries worldwide for regulations and securing leadership is also intensifying.

The European Union (EU) has become the first in the world to pass the 'EU AI Act (Artificial Intelligence Act),' which imposes strong regulations in stages according to AI risk levels, tightening control over technology. This legislation, centered on banning biometric information collection, imposing strict transparency obligations on high-risk AI, and imposing heavy fines for violations, is acting as substantial compliance pressure on global BigTechs.

Meanwhile, the United States and China, which must secure advantages in technological hegemony competition, continue a tense diplomatic and strategic tug-of-war, concerned that overly tightening regulatory constraints could shrink their respective AI innovation ecosystems. International organizations including the United Nations (UN) are taking the lead in promoting the establishment of an "international joint control body for AI security," but they continue to face difficulties in securing effective binding force, hitting walls of national interests and technological nationalism.

When to Reclaim Control and Reset the Direction of Civilization

The question humanity faces in the final phase of the AI great transformation is not simply whether technology becomes more sophisticated. It is a civilizational question of survival: "Can humans ultimately control the intelligence they have created?"

The journey of this series, spanning from the first to fourth installments, ultimately converges on one conclusion. To enjoy the brilliant benefits that AI will bring, overcome threats to labor and everyday life, and ultimately transcend the fear of destruction to achieve autonomous coexistence, 'human control and universal norms' must precede the speed of technology.

Building a solid social safety net that does not lose leadership to technology and constructing global governance. That is the heaviest and most urgent duty that humanity must absolutely fulfill at the crossroads of the AI great transformation.