MemT 1.0 – The Industry's First Memory-Native Foundation Model Open Sourced by MemTensor

1. What is MemT 1.0

MemT 1.0 is the industry's first memory-native foundation model introduced by the MemTensor team, and it is now fully open-sourced. This model eliminates the traditional reliance of large language models on external retrieval or repeated injection of historical text, instead encoding historical information into an internal memory state that can be continuously updated. This allows the model to directly read and write memory during the inference process. MemT 1.0 includes two versions: Flash (288B parameters / 17B activation) and Nano (36B parameters / 3B activation). It ranks among the top two in 11 memory benchmarks compared to parameterized memory methods, achieving a maximum speed improvement of about 4 times in long-history scenarios, while reducing storage usage to 1/66 of the full context. This lays the architectural foundation for Agent long-term companionship and recursive self-improvement.

Technical Positioning and Domain: MemT 1.0 belongs to the field of large language model architecture innovation, focusing on addressing the challenges of memory persistence and efficient inference in long-context processing. Its unique positioning lies in internalizing memory capabilities as part of the model's parameters, rather than relying on external vector databases or retrieval-augmented generation (RAG) solutions, representing a paradigm shift from "external memory" to "native memory."

Development Background: Developed by the MemTensor team, which has deep expertise in efficient sequence modeling and state-space models. The motivation behind MemT 1.0's development stems from a thorough analysis of the limitations of existing Transformer architectures, such as linearly increasing computational costs and limited context windows in long-history scenarios. The goal is to break through these constraints through architectural innovation.

Core Value: Addresses the two major pain points of large language models in long-term interactions: "forgetting" and "inability to recall." Traditional models either lose early information due to limited context windows or introduce additional latency and cost by relying on RAG retrieval. MemT 1.0 enables continuous accumulation and efficient retrieval of historical information through internal memory states, allowing Agents to carry over experience across sessions while significantly reducing inference costs and storage overhead in long-history scenarios.

Technical Features: Utilizes a memory-native architecture, inserting a native memory module in parallel within the Transformer module, enabling the original attention mechanism and memory attention to operate concurrently. It employs the Gated DeltaNet-2 update mechanism for token-by-token memory writing and forgetting control, and adopts a two-stage training strategy (200 billion token continuous pre-training + 140 billion token fine-tuning) to establish language modeling capabilities that depend on memory, achieving memory capabilities with minimal impact on general performance.

2. Key Features

• Native Memory Storage: Encodes historical information into a continuous memory state within the model, rather than storing it in an external database. When querying, there is no need to provide the original historical text; the model directly reads information from its internal memory state, eliminating reliance on the retrieval module in traditional RAG solutions and significantly reducing system complexity and query latency.

• Token-by-Token Memory Update: After each round of interaction, the model automatically writes the input and response of that round into the dynamic memory state token by token via a commit operation. It supports fine-grained control over erasure, writing, and forgetting, using four gating mechanisms to separately adjust old associations, inject new information, regulate retention strength, and control the influence of memory on responses, achieving selective retention similar to human memory.

• Memory Capacity Expansion: Through the Mixture-of-Storage hybrid storage mechanism, memory capacity can be independently expanded using multiple Memory Experts. When handling longer histories or larger knowledge volumes, the capacity can be expanded by increasing the number of Memory Experts, without the need to retrain the entire model, offering flexible configuration options for different application scales.

• Reuse of Long-Term Experience: The Agent can reuse past problem-solving methods, failure patterns, and verification processes. On the SWE-bench code repair benchmark, MemT 1.0 improved success rate by approximately 3 percentage points, demonstrating the model's ability to effectively transfer experience accumulated from previous tasks to new ones, achieving a "smarter with use" capability for continuous evolution.

• Cross-Session Memory Continuity: The memory state can be serialized for storage and recovery, bound to a user or task trajectory, enabling long-term continuous usage. This means that the Agent can retain memory of user preferences, historical conversations, and task progress across different sessions, without needing to start from scratch each time, providing the technical foundation for long-term companion-type applications.

• Efficient Inference: In scenarios with long histories, end-to-end inference speed can be improved by up to four times. Persistent storage occupies only 1/66 of the full context and does not expand with increasing history length. The memory state size remains fixed, keeping inference computational costs stable regardless of history length, fundamentally solving the problem of linearly increasing computational costs in long context scenarios.

3. How to Use

1. Obtain the Model: Download the model weights for MemT-1.0-Flash or MemT-1.0-Nano from the GitHub repository (https://github.com/MemTensor/MemT-1.0) or the HuggingFace model library (https://huggingface.co/collections/MemTensor/memt-1). The Flash version requires higher GPU memory configuration, and it is recommended to use GPUs with 80GB memory such as A100/H100; the Nano version can run on consumer-grade GPUs.

2. Deploy the Model: Load the model locally or on a server. The Flash version (288B parameters / 17B activation) is recommended to be deployed with 8×A100 or equivalent computational resources; the Nano version (36B parameters / 3B activation) can run on a single A100 or RTX 4090. It supports mainstream deep learning frameworks, and you must ensure that the CUDA version is compatible with the PyTorch environment.

3. Bind Memory Trajectory: Create an independent interaction trajectory for each user or task, and assign dedicated memory state slots. The system distinguishes memory spaces of different sessions using trajectory IDs, ensuring strict isolation between user memories to prevent data interference and privacy leaks.

4. Write to Memory: After each round of conversation, call the commit operation to automatically write the input and response of the current round into the dynamic memory state, token by token. This process is controlled by the Gated DeltaNet-2 mechanism, which finely regulates the retention and updating of memory content through four gates: erase, write, forget, and output.

5. Read from Memory: When asking the next question, the model directly reads the memory state during the forward pass to generate a response, without needing to concatenate historical text into the input. This process is seamlessly integrated with regular inference, avoiding additional retrieval steps and maintaining inference efficiency.

6. Cross-Session Continuation: Serialize and save the memory state sequence to external storage (such as local files or databases), and restore this state during the next session to continue from the previously accumulated experience. It is recommended to regularly back up memory states and establish a memory version management system to handle exceptional cases.

4. Pros and Cons Analysis

5. Comparative Analysis with Similar Tools

Selection Recommendations: For agent applications requiring long-term companionship, cross-session memory continuity, and recursive self-improvement, MemT 1.0's memory-native architecture offers clear advantages. Its fixed-size memory state and high inference efficiency make it particularly suitable for high-frequency interaction scenarios. MemT 1.0 and MemOS can be viewed as complementary—MemOS's plaintext memory combined with MemT's parameterized memory can form a complete pipeline, a direction the team has planned for MemOS 3.0.

For scenarios requiring high fidelity in memory details or strict auditability of memory (such as legal or medical fields), MemOS or traditional RAG solutions still hold advantages. For single-turn long document processing tasks, the LongContext model may be more directly applicable. Actual selection should comprehensively consider factors such as memory fidelity, inference efficiency, deployment cost, and explainability.

6. Editor's Summary

MemT 1.0 represents an architectural-level innovation in the memory mechanisms of large language models. From a technical perspective, its core value lies in transforming memory from an "external module" into a "native capability"—through the Gated DeltaNet-2 update mechanism and a dual-layer memory structure, the model can directly read and write internal memory states during inference. This design fundamentally addresses the issues of memory retrieval latency and linear growth in context cost that traditional solutions face. The two-stage training strategy (200 billion tokens of continuous pre-training + 140 billion tokens of post-training) ensures that the model retains its general capabilities while acquiring memory capabilities, a technically challenging feat that highlights the team's expertise in model architecture and training methodologies.

In terms of practical value, MemT 1.0's native memory architecture provides a solid and feasible technical foundation for Agent long-term companionship, experience reuse, and recursive self-improvement. The cross-session memory continuity enables Agents to become "smarter and more cost-effective with continued use," while the design that grants memory sovereignty to the deployment party allows enterprises to accumulate their own memory assets, which have clear value in commercial scenarios. The 3 percentage point increase in SWE-bench success rate demonstrates that memory capabilities can indeed be translated into tangible improvements in task performance.

In terms of target users, MemT 1.0 is suitable for three categories of developers: first, developers building long-term companion applications for Agents; second, programming assistant teams requiring cross-session experience accumulation; and third, enterprise users with strict requirements for data sovereignty and privacy isolation. The Nano version lowers the entry barrier, while the Flash version is tailored for high-performance scenarios.

Looking ahead, the architecture of MemT 1.0 opens up possibilities for deeper integration of memory capabilities with reasoning capabilities. As the memory capacity expansion mechanism (Mixture-of-Storage) matures and research into memory explainability progresses, native memory models are expected to be deployed in more vertical domains. The team's planned direction for MemOS 3.0 (combining plaintext memory and parameterized memory) is also worth noting, as it may represent the evolution path for the next generation of memory systems.

7. Application Scenarios

• Long-term Companion Personal Assistant: Remembers user preferences, habits, and past conversations across sessions, delivering a continuous and increasingly personalized experience. For example, a personal assistant can recall previously discussed travel plans, dietary preferences, and schedules, providing tailored services in new sessions without requiring the user to restate them, significantly enhancing interaction naturalness and user engagement.

• Programming Agent Experience Reuse: Records methods for reproducing issues, failure patterns, and verification processes in memory to guide subsequent tasks. On the SWE-bench benchmark, this results in a roughly 3 percentage point increase in success rate, indicating that programming agents can learn from historical debugging experiences, identify problems faster when encountering similar bugs, reduce redundant trial-and-error costs, and improve development efficiency.

• Enterprise Intelligent Customer Service: Customer service agents remember each customer's historical communication records and service progress, eliminating the need for repetitive questioning. The memory assets are retained by the deploying entity rather than the model vendor. This not only enhances customer experience (no need to repeatedly describe issues) but also protects enterprise data sovereignty, preventing customer data from being transferred to third-party model service providers.

• Long-term Project Collaboration: Continuously accumulates context across days and weeks in long-term projects, avoiding the need to restart projects from scratch due to context window limitations. Project team members can maintain continuous collaboration with the agent, which remembers the project background, decision records, and to-do items, effectively supporting the long-term advancement of complex projects.

• Scientific Research and Recursive Self-Improvement (RSI): AI retains the effective outcomes of each trial-and-error attempt as memory, serving as the foundation for the next round of self-optimization and code refinement. In scenarios such as algorithm tuning, experimental design, and code generation, AI can accumulate effective strategies and avoid known failure patterns, achieving a closed-loop for self-evolution.

8. FAQ

Q: What is the fundamental difference between MemT 1.0 and traditional RAG approaches?
A: Traditional RAG approaches store memory in external vector databases. When querying, they first retrieve relevant text and then inject it into the context, which leads to issues with retrieval latency and incomplete information recall. MemT 1.0, on the other hand, encodes historical information as parameterized memory states within the model itself. The model directly reads and writes memory during forward computation, eliminating the need for an external retrieval step and fundamentally removing reliance on external retrieval systems.

Q: How large is the memory capacity of MemT 1.0? How much content can it remember?
A: MemT 1.0 utilizes a Mixture-of-Storage hybrid storage mechanism, enabling independent expansion of memory capacity through multiple memory experts. The specific capacity depends on the number and configuration of memory experts, which can be increased as needed. In test scenarios with a history of 256K tokens, the model demonstrates stable memory capabilities, and the size of the memory state remains fixed, not expanding with the growth of historical data.

Q: Will the memory state consume a large amount of storage space?
A: No. The persistent storage of MemT 1.0 occupies only 1/66 of the full context and does not expand with historical growth. For example, historical information of 256K tokens, after compression, requires minimal storage space. This makes storage costs nearly negligible for long-running application scenarios.

Q: Can memories between multiple users interfere with each other?
A: No. The system creates independent interaction trajectories for each user or task by binding memory traces, assigning dedicated memory state slots to ensure strict isolation between different users' memories. This design protects user privacy and avoids data interference issues.

Q: Will the general capabilities of MemT 1.0 decline due to the introduction of the memory mechanism?
A: No. Under conditions with no irrelevant memory noise, MemT 1.0's capabilities such as instruction-following, code generation, and long-text understanding are on par with or even slightly better than the original backbone model. A two-stage training strategy (200B tokens of continuous pre-training + 140B tokens of post-training) ensures that the model maintains its general capabilities while acquiring memory functionality.

Q: What are the main differences between the Nano version and the Flash version?
A: The Flash version (288B parameters / 17B activation) has stronger memory capabilities, achieving an average score of 50.6 across 11 memory benchmarks, ranking first among parameterized memory methods. The Nano version (36B parameters / 3B activation) has an average score of 47.0, ranking second, but has a lower deployment threshold, making it suitable for resource-constrained environments. Both versions share the same architectural design and can be selected based on hardware conditions and performance requirements.

9. Project Links

• GitHub repository: https://github.com/MemTensor/MemT-1.0
• HuggingFace model library: https://huggingface.co/collections/MemTensor/memt-1
• arXiv technical paper: https://arxiv.org/pdf/2610.26825