If you pay enough attention, there comes a time when a choice you thought you made seems a little strange. It’s like opening a door you think you did on your own, only to discover later that something was already pulling it from the other side. The majority of people miss that opportunity. The underlying technology is built to prevent them from doing so.
Since the late 1990s, when behavioral psychologist B.J. Fogg sat down at a computing conference and came up with the term “captology,” persuasive technology—the vast field of computational systems designed to change human attitudes and behaviors—has existed in one form or another. Back then, it sounded almost innocent. Computers as tools for persuasion. A calorie counter encouraging you to eat a salad. An app for fitness that honors your success. However, the current version, which operates covertly within news aggregators, e-commerce sites, and social media feeds, is far less gentle.
The fact that these systems are more intelligent is not the only thing that makes the present era unique. They are knowledgeable. An AI-powered persuasive system is more than just aware that displaying a baby’s photo can emotionally affect some customers. It can determine that a particular user is probably pregnant, deduce from her posts and searches how she feels about that pregnancy, keep track of the number of times such imagery has specifically influenced a purchase, and modify the entire visual texture of a website to target that window at precisely the right time. It’s not nudging. That is more in line with cognitive mapping.
There’s a feeling that the majority of users would find it genuinely unsettling if they were presented with a detailed account of what’s currently occupying their attention on the internet. The manipulation isn’t taking place on the surface; it’s not a flashing banner or a pop-up. It functions through what researchers refer to as “second-generation dark patterns” and “hypernudges,” which are configurations that are so deeply ingrained in an experience’s architecture that the interference blends in with the surroundings. You don’t feel pressured. You have the impression that you are moving in a straight line.
The distinction between System 1 and System 2 thinking—the quick, instinctive brain versus the slower, more deliberate brain—is the one that researchers keep bringing up. In the academic sense that is most pertinent here, manipulation does not refer to deceiving people with inaccurate information. It involves avoiding the area of the mind that would otherwise stop, consider, and ask questions. According to the philosopher Raz, it distorts how a person develops preferences and goals. It still seems inadequate for what AI systems are now able to accomplish on a large scale.
Numerous organizations, including international human rights organizations, European regulators, and AI ethics panels, have identified computational manipulation as a threat to consumer welfare as well as something more fundamental: an individual’s capacity to form their own opinions. Although freedom of expression and privacy law have traditionally covered some of this, there is a growing contention that these frameworks weren’t intended for this. The right to form an idea without the invisible hand of a third party directing it differs significantly from the right to hold and express one. Whether the current legal framework can truly make that distinction or if a new one needs to be named is still up for debate.
Observing all of this progress, it’s remarkable how little friction it creates in daily life. In a regulatory committee or boardroom, the issue doesn’t seem abstract. However, there is a sort of ambient acceptance at the street level in the way people discuss their phones and feeds—a shrug toward the notion that, yes, the algorithm knows what you want. Perhaps the most convincing result of all was that resignation. The goal was, of course, to gradually and subtly relinquish autonomy rather than in a dramatic manner.
Moving forward, the problem is not just technical or even legislative. It’s theoretical. If the process of making a decision determines what constitutes a free choice, then the discussion of digital freedom must extend beyond the point at which a person clicks. It must return to the circumstances that initially influenced what they desired to click on. That’s unsettling ground. We might only be beginning to realize how far along we are already.
