Storage Solution Empowering AI

Kioxia has been conducting research and development in the field of AI for many years.

Kioxia has authored research on the mechanisms for building and evaluating systems that combine Large Language Models (LLMs) with search.

Large Language Models (LLMs), which emerged at the end of 2022 and captured global attention, have triggered a major movement by “Big Tech” companies—companies developing world-changing technologies — in the field of learning and inference, which led the dawn of generative AI. The services provided by LLMs are expected to be integrated into a wide range of business models, industries, and society.

To increase accuracy, relevance, and timeliness of inference, a novel approach known as Retrieval-Augmented Generation (RAG) has emerged, designed to maximize the effectiveness of inference.

An essential element for any generative AI system is data. SSDs are becoming a more important way to store data with their characteristics of non-volatility, large capacity, and high speed. SSDs are evolving with the different stages of generative AI, and as a result, the demand for SSDs is rapidly increasing. It is expected that between 2024 and 2030, the market for SSD capacity will grow significantly, approximately 40 to 50 times.

Kioxia believes the medium to long term outlook through 2030, growth of the generative AI market will be driven by the following four trends.

1. Development of learning
2. Development of inference
3. Development of RAG for enhanced inference
4. Development of generative AI services

SSDs are expected to play a pivotal role in all of these trends.

In generative AI systems, expensive GPUs are used for both training and inference, and it is difficult to address the large quantities of data required for the generative AI process using only volatile DRAM. Therefore, data processing becomes essential for tasks such as supplying training data from non-volatile storage devices, referencing data for RAG during inference, and storing generated data.

An important requirement for storage devices now is to help maximize GPU utilization and get the most out expensive GPUs, as well as to maximize power and space efficiency. HDDs come to mind when thinking of large storage capacities, however, they are limited by their mechanical characteristics, such as media rotation delay and the movement of read/write heads. This results in read/write latencies that can be over 100 times higher than SSDs, which impacts overall power efficiency. SSDs will support AI storage services to maximize processing efficiency of high-performance GPUs, and will meet the demand for increasingly larger capacities in the future.

LLMs are already being used by many companies to improve operational efficiency and productivity.

But the knowledge possessed by LLMs is limited to the data set used during training, which may be out of date. An older LLM might not generate accurate results for questions or instructions requiring the latest, company proprietary, or confidential information not available to the LLM. These inaccurate results, which are sometimes way off track, are called hallucinations.

RAG solves the hallucination problem by incorporating more accurate knowledge and data outside of the LLM in the form of a vector database. RAG uses database software with a data retrieval function based on vector similarity, called a vector database. Many companies are interested in RAG because it can enhance LLM capabilities utilizing non-public information to improve the accuracy of the AI model.

In RAG, locating the vector closest to the query vector (and the corresponding data among the stored knowledge data) involves calculating the similarity between high-dimensional vectors that can range from hundreds to thousands of dimensions. It is an intensive task even for the latest AI servers. To address this challenge, approximate nearest neighbor (ANN) search can be used with an index to narrow down the comparison vectors to a small number of vectors, to find an approximate solution.

There are various methods for ANN searches, including DRAM-based Hierarchical Navigable Small World (HNSW) and DiskANN, developed by Microsoft, which is primarily storage-based. Currently, HNSW, which stores vectors and indices in DRAM, is widely used. In this method, the number of vectors, which is the volume of knowledge it can utilize, is limited by the amount of DRAM in the server, thus limiting the ability to build increasingly effective RAG systems built on large volumes of knowledge.

Kioxia has developed KIOXIA AiSAQ™ software as a new ANN storage method utilizing SSDs. To further leverage SSDs, especially in the rapidly growing generative AI market, Kioxia will continue advancing its research and development of this important storage technology.

KIOXIA AiSAQ™ (All-in-Storage ANNS with Product Quantization) software, as the name suggests, transfers information that quantizes data characteristics (PQ data) and all ANN elements to SSD storage to optimize approximate nearest neighbor search algorithms for vector search processing in the RAG phase. It has been released as open source to enable higher quality generative AI applications for the market.

In an AI workflow, indexed data is typically stored in DRAM to enhance the efficiency of vector searches. KIOXIA AiSAQ™ technology places indexed data on SSDs.

For more accurate AI queries, RAG vector databases will grow and scale. For the DRAM-based approach, larger databases require more DRAM (graph on the left in the figure below). From a performance perspective, SSD-based search outperforms DRAM-based search in terms of query accuracy and queries per second (graph on the right in the figure below).

This innovative RAG technology is promising because it uses less expensive SSDs, rather than costly DRAM to improve ANNS without compromising performance.