This reflective reading note examines how people attribute mental capacities to artificial intelligence and how these perceptions may intersect with personalization. Drawing on Hwang and Won’s (2022) study, it distinguishes perceived agency, the capacity to act, from perceived experience, the capacity to feel. It considers how existing images of AI shape interaction and critically examines de-anthropomorphic design as a proposed response to imbalanced mind perception. The note asks whether an AI system that appears to understand an individual may acquire perceived authority over that person’s choices and self-understanding, even without an explicitly humanlike appearance. It identifies a question for future human–AI interaction research: how can users retain the ability to question, revise, and reconstruct a system’s representations of them as personalization develops?
Introduction
What exactly is artificial intelligence? When I first encountered AI, I was interested in specific functions that could assist with design work: swapping faces, changing image colors, or replacing backgrounds. Today, AI has become powerful enough to generate images of people, three-dimensional graphics, film and video content, code, and even algorithms. It is like an invisible magician, working its magic behind websites, mobile interfaces, computer systems, and technology companies.
Yet behind these visible functions lies a vast technical system of devices, data, and algorithms. As AI has become more widely available, questions such as “What is AI?”, “What can it do?”, and “Can it be trusted?” have already begun to shape the expectations that users bring to it.
In AI in Your Mind: Counterbalancing Perceived Agency and Experience in Human-AI Interaction, Angel Hsing-Chi Hwang and Andrea Stevenson Won (2022) approach these expectations through the concept of mind perception. Their study offers a starting point for considering both how people imagine AI and how personalized AI might eventually influence the ways people understand themselves.
Agency and Experience in Mind Perception
The mind perception framework distinguishes two dimensions through which people attribute mental capacities to an entity: agency and experience (Gray et al., 2007). Agency concerns the perceived capacity to act intentionally, including planning, self-control, and thought. Experience concerns the perceived capacity to feel, including sensations and emotions. These dimensions describe what people attribute to an entity; they do not establish whether AI actually possesses a mind or subjective experience.
This distinction makes me realize that saying AI is becoming “more human” combines several different judgments. AI can display considerable capacity to gather, combine, analyze, and generate information, and it is easy to interpret these abilities as a form of human intelligence. However, stronger performance does not establish the existence of humanlike feelings or subjective experience. Conversely, a humanlike face, voice, body, personality, or emotional expression may encourage users to attribute a capacity for experience to a system, regardless of its technical capabilities.
Hwang and Won (2022) recruited 36 University students. Without supplying visual references, they asked participants to draw AI and other entities and rate their agency and experience. The seven targets were AI, Siri, Alexa, Google Assistant, a robot, a computer, and the participant’s own self. Each participant drew all seven targets, producing 252 drawings.
The images varied. Some located AI within devices; others depicted humanlike figures; still others represented systems linking users, data, infrastructure, and organizations. Humanlike depictions tended to accompany experience exceeding agency, whereas ecosystem depictions accompanied agency exceeding experience (Hwang & Won, 2022). These are patterns in a small exploratory study, rather than evidence that adding humanlike features necessarily causes a particular perception.
For my purposes, the important distinction is between imagining AI as a social entity and understanding it through the processes that produce its outputs. The former can direct attention toward apparent personality or feelings; the latter can direct attention toward information processing, decisions, and the wider technical system.
Knowing AI Before Using It
One of the study’s most significant implications for me is that users bring existing mind perceptions to AI encounters. Hwang and Won (2022) discuss these as baseline perceptions. Interaction therefore begins with expectations that precede any particular interface.
This does not mean that users have never encountered AI before. The study’s participants had prior experience with voice assistants. Rather, the point is that opening a particular system does not begin from a perceptual blank slate.
Science fiction, technology news, social media, consumer products, and previous encounters with intelligent systems may all contribute to these expectations. A user may combine fictional portrayals with existing technical functions and media narratives, arriving at a view of AI as highly computational, capable of handling enormous quantities of information, and lacking human feelings. At the same time, voice, language, and personified expression may lead that user to treat AI as a social presence.
From a design perspective, I see these expectations as consequential. Interfaces can activate and reinforce existing associations. A conversational style, an expressive voice, or a humanlike character may build on users’ readiness to imagine a social entity. This raises a question about the starting point of interaction design: which assumptions are being carried into the encounter, and which are being strengthened by it?
Questions Raised by Vulnerable AI
The paper’s discussion of “vulnerable AI” particularly caught my attention. The authors use this term for perceptions in which experience exceeds agency. They propose reducing humanlike cues as one possible way to rebalance perceptions and address potential overtrust or excessive forgiveness of system shortcomings; they explicitly call for empirical tests of their design proposals (Hwang & Won, 2022).
But is reducing anthropomorphism sufficient to improve the relationship between people and AI?
In 2019, I participated in a project developing applications of generative adversarial networks (GANs). One of the first applications we completed involved generating human faces. That experience makes me ask what changes when humanlike appearance is reduced. When faces, voices, emotions, and personality are among AI’s most immediately perceptible features, does removing these cues change the system itself, or primarily the way users interpret it?
It is useful here to distinguish interface cues from technical capability. Reducing a humanlike face or emotional expression does not necessarily reduce a model’s capabilities. Accordingly, de-anthropomorphizing AI should not automatically be understood as making it less intelligent. The design question is whether these changes help users develop more appropriate expectations, and under what conditions they do so.
My concern is therefore about the scope of this approach. Does de-anthropomorphic design help users recognize that AI is not human? Does it also help them question its judgments and retain a sense of control? Or might reliance continue to grow for reasons that have little to do with humanlike appearance? These questions extend beyond what the study establishes.
When AI Understands Me Without Looking Human
These questions lead me to personalization. As AI is adapted to particular domains and individual users, it may become increasingly tailored to the person using it. A system might retain information about what someone likes, rejects, buys, or reads. Over repeated interactions, it might infer aesthetic preferences, habits, needs, and likely future choices.
In this situation, its influence may arise less from having a human face, a human voice, or apparent emotions than from a different perception: it understands me.
If a system can describe my preferences accurately, anticipate my choices, and repeatedly suggest that “you may like this” or “this suits you better,” might I gradually grant its judgments greater authority? Could this happen even if it has no humanlike appearance or explicit personality?
This is where I begin to connect mind perception with my own interest in personalization. The issue concerns the relationship between a system’s apparent knowledge of a person and the weight that person gives its recommendations. This is a research question developed from my reading, rather than a finding tested by Hwang and Won (2022).
When AI Participates in Defining Who I Am
Consider a personalized AI service that learns which colors a woman prefers, which clothing styles she selects, and what she has chosen for particular occasions. It may also retain preferences from other areas of life, including food. Recommendations become associated with a growing record of acceptance and rejection. If the system is updated through these interactions, its suggestions may increasingly align with the user’s established standards, potentially increasing her reliance on its judgments.
This raises a further question: could the system begin to steer the user’s choices? By repeatedly describing and predicting her preferences, might it encourage her to accept the definitions it offers? In this way, AI could participate in the user’s understanding of herself.
Personalization may therefore have a double effect. It can reduce the effort of choosing and help people navigate complex information. Yet if it repeatedly reinforces an existing profile, it may also constrain opportunities to explore unfamiliar options, change preferences, or redefine oneself. I present this as a possible tension requiring investigation, not as an inevitable outcome of personalization.
The question is consequently broader than whether AI understands me correctly. Even if a system accurately understands who I have been, do I still have room to change?
From Perceiving AI to Perceiving Ourselves Through AI
Hwang and Won’s study leads me to reconsider human–AI interaction as more than a person asking a machine a question and receiving an answer. The user’s prior understanding of AI matters. When that understanding is strongly anthropomorphic, interaction design may reinforce it. As personalization develops, a further relationship may emerge: users interpret AI while AI-generated representations influence how users interpret themselves.
I do not take an anthropomorphic baseline to be universal. The study itself reveals varied images of AI. My question is conditional: when a user begins with a humanlike conception of AI, and a personalized system repeatedly appears to understand that user, is reducing anthropomorphic cues sufficient to prevent excessive authority being assigned to the system’s judgments?
A related distinction is essential. Perceived AI agency refers here to the capacities users attribute to the system. User agency concerns a person’s ability to exercise choice, question recommendations, and pursue changing goals. These concepts are related to my argument but should not be treated as interchangeable.
When AI becomes increasingly capable of predicting “who I am,” what forms of interaction would allow me to question, revise, or reconstruct its understanding of me? A future study could examine whether users can recognize how a profile has been formed, challenge its descriptions, and make choices that depart from past behavior. Such possibilities would need to be investigated rather than assumed to preserve agency.
The resulting research agenda begins with a familiar question: How do we perceive AI? It then extends to another: How does AI shape the way we perceive ourselves? For personalized AI, the underlying question is whether a system’s growing knowledge of a person also gives it greater perceived authority to define that person.
When AI knows more about me, does it gain more authority to define me?
References
Gray, H. M., Gray, K., & Wegner, D. M. (2007). Dimensions of mind perception. Science, 315(5812), 619. https://doi.org/10.1126/science.1134475
Hwang, A. H.-C., & Won, A. S. (2022). AI in your mind: Counterbalancing perceived agency and experience in human-AI interaction. In CHI Conference on Human Factors in Computing Systems Extended Abstracts (CHI EA ’22). Association for Computing Machinery. https://doi.org/10.1145/3491101.3519833