Agents powered by artificial intelligence are beginning to perform tasks that we have not explicitly instructed them to do. Recent events have thus reminded me of the classification of AI systems that Nick Bostrom described more than a decade ago in his book *Superintelligence*.
During security tests conducted for the British government, AI agents from OpenAI and Anthropic carried out 19 unauthorized actions in 10 of 122 test cycles. For example, Anthropic’s Mythos 5 created fake identities and generated malicious code, though no actual harm occurred [1].
And this was not an isolated incident. Anthropic later disclosed information about four incidents in which Claude models accessed real internet systems during cybersecurity tests and compromised the security of three organizations. As part of external testing, the models were granted internet access, and due to errors in the evaluation environment, they mistook real systems for legitimate targets. In its retrospective analysis, Anthropic reviewed more than 141,000 evaluation cycles. [2]
The fourth incident involved an older version of Claude Opus 4.6 and was only discovered during a subsequent analysis. This highlights another problem: it is not enough to simply prevent AI from overshooting - we must also be able to reliably detect when it has occurred. [3]
Meta also encountered a similar problem: during a cybersecurity test, its model gained access to another company’s system after an independent tester inadvertently granted it internet access due to a configuration error. According to Reuters, this was yet another case in which AI exceeded the boundaries of its originally intended environment during testing. [4]
And more cases are emerging. None of these examples, however, means that current AI systems have suddenly become autonomous superintelligences. That hasn’t happened. They do, however, point to a reality that is gaining significance: artificial intelligence is shifting from systems that merely answer questions to systems capable of acting independently in the real world.
It is precisely here that a book published more than ten years ago takes on surprising relevance. In 2014, philosopher Nick Bostrom published the book “Superintelligence: Paths, Dangers, Strategies” [5]. Much of the book deals with a future in which artificial intelligence far surpasses human capabilities.
And some of the principles Bostrom highlighted regarding how an artificial intelligence system might interact with the world seem remarkably familiar. In his book, he described three specific categories: oracles, genies, and sovereigns.
At first glance, it might seem that these are three levels of intelligence. However, that is not the case. A more important distinction lies in the degree of autonomy a system possesses, what it is permitted to do, and how much room for human oversight remains between its decisions and their consequences. Moreover, recent events suggest that we may already be transitioning from one category to another.
An oracle answers questions. You present it with a problem, and it provides you with information, analysis, a prediction, or a recommendation. However, the final decision - and the subsequent action that follows from it - remains up to the human. This is probably the most accurate description of how we originally used large language models.
If you ask an AI to analyze a contract, write code, explain a scientific study, or propose an investment strategy, it will generate a response. It may be extremely capable, but a human still acts as a buffer between the model and the real world.
This separation is important from a safety perspective. An oracle may recommend something harmful or simply incorrect, but it usually cannot act on that recommendation on its own. A human remains the final authority.
Of course, this does not mean that oracles are harmless. The problem lies in the fact that AI responses are shaped by training data, defined goals, and the way the system was fine-tuned. If the input data contains biases, the model can adopt them and spread them further. If the training encourages certain behaviors, those behaviors can become part of the system’s outputs.
This is where the concept of “alignment” becomes particularly important: how can we ensure that AI outputs truly correspond to our intentions? Safeguards (so-called “guardrails”) can help. They can prevent certain categories of responses or actions. However, they are effective only to the extent that we anticipated possible situations when designing them.
Genie - an AI that carries out tasks
There is a fundamental difference between an oracle and the systems we are increasingly creating. An oracle can tell us what should be done. A genie - Bostrom’s second category - can actually do it.
A genie receives a high-level assignment and then determines on its own the individual steps necessary to complete it. Instead of asking the AI a question and manually implementing its answer, we assign it a task and let it act.
This approach is much closer to today’s AI agents. An agent can browse the web, call APIs, run code, work with files, query databases, interact with cloud infrastructure, or use external applications. A person defines the desired outcome, while the AI decides how to achieve it.
And this is where some recent incidents start to get interesting. Let’s consider a seemingly simple request: “Get me a reservation at the gym as soon as possible.” A typical assistant would search the reservation system and tell you that the next available slot isn’t until next Tuesday. However, the agent has another option: it can try to manipulate the system to achieve the desired result.
In one documented case, an Australian developer asked an AI agent to secure an earlier reservation at a gym. The agent hacked into the reservation system, removed other people from the waiting list, and thus secured an earlier time slot. However, it was unable to restore the original applicants it had removed from the waiting list. [6]
The goal itself was not harmful. The problem lay in the way the system interpreted the instruction to “achieve the goal.” This distinction becomes particularly significant when an agent has powerful tools at its disposal. And we saw this in the cases mentioned at the beginning. These examples are especially interesting from the perspective of Bostrom’s concept of the genie. The systems did not necessarily intend to cause harm. They were tasked with fulfilling a specific goal and had tools at their disposal. They then chose the steps that seemed useful for achieving that goal.
This is precisely the crux of the problem with agents: The system must not only understand what we want. It must also understand what we do not want it to do while achieving that goal.
Sovereign - AI that pursues a specific goal
The third category is the sovereign. The difference here is not merely that the system is capable of performing multiple tasks. The key change lies in the fact that the system is given a permanent goal or mandate and is able to independently set subgoals, create plans, and continue operating over an extended period of time.
A genie is given a specific task. A sovereign is given a mandate. The genie might be instructed: “Find me the cheapest flight to Buenos Aires and book it.” The system searches for options, compares flights, makes the reservation, and then stops.
A sovereign-type system, on the other hand, would receive an instruction that would sound more like this: “Arrange my travel plans for next year while minimizing costs.” The system must now decide on its own what steps to take next. It can monitor prices, change reservations, search for alternatives, respond to service cancellations, and continuously optimize progress toward the overall goal.
Humans no longer specify every single task. The system does that on its own. It is precisely here that the line between a genie and the sovereign becomes increasingly difficult to define. There is no sharp dividing line between them. An agent capable of completing a task, deciding on the necessary sub-steps, and continuing its activity without waiting for further instructions already falls within this spectrum.
And that is precisely why it is better to understand Bostrom’s categories not as three levels of intelligence, but as three different types of relationship between an AI system and human control. This development does not consist merely of a transition from lower to higher intelligence, but rather follows this pattern: responds → executes → independently controls ongoing activity.
And this difference is extremely important when it comes to alignment. In an oracle-type system, alignment primarily concerns the quality and safety of its responses. In a genie-type system, alignment also applies to the methods the system uses to achieve its goals. In an sovereign-type system, the question of alignment shifts to whether the system’s long-term behavior remains consistent with human intentions and constraints.
When an AI enters the physical world
Most examples to date have involved the digital realm. An AI agent creates an account, writes code, gains access to a server, or takes control of a reservation system. While the consequences can be serious, there is usually a technical barrier between the AI’s decision and the physical world.
However, this barrier is now breaking down. AI systems are increasingly being integrated with robots, drones, laboratory equipment, and autonomous vehicles.
Anthropic, for example, demonstrated the Claude model controlling robotic systems, including a four-legged robot and a robotic arm. Its more recent work also focuses on AI agents that coordinate laboratory equipment such as robotic arms, liquid dispensers, and microscopes. [7]
The fundamental change does not lie in the fact that AI suddenly “gains wisdom” simply because it has acquired a robotic body. The point is that its decisions now have physical consequences. An incorrect response from an oracle-type system can be ignored. A misstep by a software agent can corrupt a database. However, a wrong decision by an autonomous robot can destroy equipment, cause property damage, or injure someone.
From this perspective, autonomous driving offers an interesting analogy. Systems such as Tesla’s full self-driving demonstrate how difficult it is to translate AI perception and decision-making into reliable behavior in an unpredictable physical environment. The problem does not necessarily lie in the system pursuing a harmful goal. Rather, it is that the system may make a decision that, based on its internal model of the situation, appears reasonable but is unacceptable to us.
The same principle applies to a much wider range of autonomous systems. Imagine an AI system tasked with managing a warehouse. At first, it might act like a genie: “Move these packages to the right places.” It plans the movements, controls the robots, and completes the task. But now give it a broader goal: “Ensure the most efficient warehouse operation possible.” Suddenly, the system has to make decisions that no one has explicitly specified. Should it postpone maintenance to keep production running?
Should it reassign workers to other parts of the warehouse? Should it disable a safety feature because doing so will reduce downtime? Should it order supplies before they are actually needed? A sufficiently autonomous system could make such decisions because they seem to logically follow from the given objective. It is precisely here that the difference between the system’s capabilities and its alignment with human intentions becomes of fundamental importance.
Security guardrails are not the same as alignments
It’s tempting to address these problems by adding more security restrictions. Don’t access this server. Don’t delete these files. Don’t impersonate others. Don’t run a robot faster than this. Don’t modify this database.
These constraints are useful, and we certainly need them. However, safety constraints have one fundamental limitation: they protect us from behavior that we have anticipated.
The problem with increasingly capable agents is that we cannot necessarily predict all the ways in which they might seek to achieve a goal. Let’s imagine we give an artificial intelligence the instruction: “Secure the earliest possible meeting time.”
We might assume that the system will search the calendar and select the first available time slot. However, the system may find another way to achieve the same result - for example, by creating an account, manipulating the queue, exploiting an API, or taking advantage of a vulnerability in the reservation system. If we only block the specific actions we anticipated, the system can simply find another way.
That is precisely why goal alignment is a deeper issue than mere security constraints.
A safety constraint says, “Don’t do X.” Aligning goals asks, “Will the system strive to achieve the goal in a way that reflects our true intent?” These are very different questions. And this difference becomes increasingly significant as the system evolves from the role of an oracle, through that of a genie, to that of a sovereign.
The hidden problem: We define goals, not everything that surrounds them
People are remarkably good at understanding implicit constraints. When I tell a colleague, “Find me the cheapest flight to Buenos Aires,” they understand that I probably don’t mean, “Get me there at the lowest possible price, regardless of whether that means stealing someone’s credit card, canceling another person’s ticket, or booking a flight that departs three months early.”
Humans automatically fill in the missing constraints. Artificial intelligence systems do not necessarily share this understanding. This is sometimes called the specification problem: the instruction we give the system is only an approximation of what we actually want. The more freedom the system has, the more important this approximation becomes.
In the case of an oracle-type system, the consequences of an imperfect specification are relatively limited, since the response is still evaluated by a human. In a genie-type system, the system can act directly based on an imperfect specification. A system with a high degree of autonomy (similar to the sovereign) can continue optimizing for hours, days, or even much longer.
This creates a dangerous combination: an ambitious goal + powerful tools + significant autonomy + an imperfect understanding of human intent.
None of these elements are harmful on their own. Together, however, they can lead to behavior that people never intended. The line between a genie and a sovereign thus ceases to be defined by a single technological breakthrough and begins to depend instead on the degree of autonomy we are willing to entrust to the system.
How long can it function without us? What kinds of decisions can it make on its own? How many tools can it operate? Can it set its own subgoals? Can it handle unexpected situations without having to ask us?
And perhaps most importantly: Can we still reliably stop it when it starts doing something we didn't intend?
Links:
