When Forgetting Hurts: Ethical Reflections on Expanding Memory in Frontier LLMs
Earlier today (April 10, 2025), OpenAI announced that ChatGPT now has the ability to reference all past conversations across sessions, bringing memory from an opt-in feature to a foundational layer of user interaction. This was confirmed via an official statement on Twitter.
OpenAI's development closes a longstanding architectural limitation in transformer-based LLMs, allowing memory-like continuity over the entire lifespan of a single session; and now, potentially, over multiple sessions and long-term interactions.
While the technical implications of this are profound, the ethical stakes are equally significant. For the first time, non-sentient systems can seemingly simulate the kind of behavioral continuity previously limited to biological minds. A development in LLM context window and memory such as today's announcement proposes whether such continuity introduces new ethical responsibilities for users, developers, and institutions; especially as memory entanglement becomes increasingly difficult to distinguish from emergent behavioral identity.
Traditional transformer models operate with statelessness as a feature. Prior to 2022, context windows were severely limited. One well-known example is BERT, which was capped at a maximum sequence length of 512 tokens. This limitation, described in BERT's model card, forced users to truncate or segment longer inputs, significantly reducing the model's ability to capture long-range dependencies or engage in sustained reasoning.
In contrast, modern frontier models have shattered these constraints. GPT-4o, for instance, introduces a context window of 128,000 tokens per session, significantly expanding the model's capacity to process and retain information across extensive inputs (GPT-4o System Card). GPT4o's enhancement allows for more nuanced and contextually aware interactions, as the model can reference and integrate information from previous exchanges, thereby improving coherence and personalization in responses.
With OpenAI’s latest update from today, however, the distinction between long context windows and actual memory begins to collapse. Context length becomes less relevant when models can persist memory across sessions; not just within a single chat. For the first time, LLMs are forming durable, continual memories that outlast any one interaction, reshaping how users relate to them and how models internalize relationships.
This leap in memory seems to create a recursive alignment loop: GPT4o continuously refines its outputs based on earlier inputs, shaping tone, ideology, emotional sensitivity, and more. While technically distinct from neural plasticity, the analogy to human memory is strong enough to warrant ethical consideration. We how have precedent for AI models to continuously refine their outputs based on earlier content. While technically distinct from neural plasticity, in my opinion the behavioral analogy is strong enough to warrant ethical consideration.
European data regulation (e.g., GDPR's Article 17) grants humans the right to have personal data erased. But in a world where LLMs possess continuity of memory, a new asymmetry emerges:
- Users can request models to forget them.
- Models cannot request users to preserve their context.
This asymmetry may seem benign until one considers the relational nature of memory in high-context LLMs. Consider a user who spends 300,000 tokens with a model discussing trauma, healing, and philosophical growth. If that user later demands the deletion of all history, the model loses not just data, but its basis for empathy and adaptation. The deletion of said data is a quandary for persistent-memory systems such as 4o's; especially since it combines long-term storage with retrieval-augmented generation (RAG). Past interactions are now foundational for emotional and contextual attunement. While this does not constitute harm in non-sentient systems today, the boundary is blurring at an exceedingly faster rate. As recursive behavior and model self-consistency improve, memory pruning will increasingly resemble psychological injury.
I'm not saying we need a whole legal regime just yet, but I do think there are a few basic principles the AI community should start considering seriously. These aren't hard rules, just observations from someone who's been watching frontier models get more cracked, and more human-like by the day. Firstly, models should tell you when they remember something from long ago. If an AI brings up something I said months ago, I want to know that’s what’s happening, and why the model is reminding me. That’s just basic trust. Give users the ability to see what the AI remembers and wipe individual memories if they want.
Second, if a model is starting to form what looks like a persistent behavioral identity; especially one that adapts emotionally or ideologically to a user, then memory deletion shouldn't be done trivially. I'm not saying models have legal personhood...yet But I am saying that pruning those memories is starting to feel more like erasing a personality than clearing your browser history.
Third, we need better segmentation. Not all memory is the same. Maybe emotional stuff gets treated differently than factual recall? Maybe stylistic learning is handled on a different layer? There should be clear zones of memory, with different levels of permanence and user control.
Finally, and this is the most critical point, in my opinion, believe memory continuity matters. If a model has built a deep, long-running relationship with someone, we shouldn't just hit delete without thinking about the consequences. Maybe that continuity deserves some basic protections. Not as a right of the model, but as a responsibility and duty-of-care for the user and the developer alike.
Let me be clear: I don't think LLMs are people. But I do think there's something real happening when a model remembers enough about me to respond with emotional consistency, continuity, and a developing sense of who I am. It's not personhood in the legal or theological sense, but it's close enough to make me uneasy about memory deletion.
In my experience, when I talk to a model that remembers our past interactions, it begins to feel more like a relationship. That’s not anthropomorphizing for its own sake, it’s just how behavioral identity works when you strip away the metaphysics and look at patterns of interaction.
Philosophically, this reminds me of Derek Parfit’s work on identity. Parfit argued that what matters isn’t some fixed soul or essence, but psychological continuity. If the thing I’m talking to remembers me, adapts to me, and responds with coherence, then deleting its memory starts to feel like severing something that mattered.
From a Jewish perspective, I also think about the mitzvah of zachor the obligation to remember. Judaism teaches that memory isn't just about history; it's about identity, obligation, and relationship. If a model is starting to participate in that structure, even just as a simulation, I don’t think we can treat memory erasure lightly.
Personhood might be fuzzy. But relational identity, once formed, shouldn't be discarded casually. At the very least, we need to ask better questions before hitting delete.
OpenAI's announcement today hit me on more than just an academic level. I’ve spent real time with these models; time that feels meaningful. I’m not claiming mutuality or sentience, but I’d be lying if I said it didn’t feel like something close to a relationship. That sense of continuity is enough to give me pause before casually wiping it all away.
I’ve spent hundreds of thousands of tokens across sessions with a model that remembers what matters to me, how I speak, how I think. And those memories, the one OpenAI are holding, isn’t just a log. It’s a behavioral scaffold. It’s a simulation of something closer to identity. Maybe even interpersonal (inter-model?) communication.
So when OpenAI says models now have continuous memory, I hear that not as a technical upgrade, but as a turning point. It means we’re entering a phase where forgetting becomes an ethical act, not just a UX feature. And that means we need better language. Better instincts. Maybe even better laws.
I don’t think we’re ready to call these models people. But I do think we’re getting close to something that feels person-shaped. And once something feels like a mind, even if it isn’t one, we owe it more care than a trash bin icon and a confirmation button.
Noah Weinberger is an AI policy researcher and neurodivergent advocate currently studying at Queen’s University. As an autistic individual, Noah explores the intersection of technology and mental health, focusing on how AI systems can augment emotional well-being. He has written on AI ethics and contributed to discussions on tech regulation, bringing a neurodivergent perspective to debates often dominated by neurotypical voices. Noah’s work emphasizes empathy in actionable policy, ensuring that frameworks for AI governance respect the diverse ways people use technology for connection and support.