1. Data Loading
Creating LLM applications that can act on your data requires processing and loading that data. LlamaIndex achieves this through data connectors, also known as Reader. These connectors can ingest data from various sources and format it into Document objects, which are collections of text data and metadata.
LlamaIndex's SimpleDirectoryReader is a straightforward class that creates documents from every file in a specified directory. It can read from a variety of file types, including Markdown, PDFs, Word documents, PowerPoint decks, images, audio, and video.
LlamaHub is a registry provided by LlamaIndex where users can download data connectors for their specific use cases.
LlamaParse is a state-of-the-art tool for loading PDF files, available as a managed API. It is designed to create RAG systems over complex PDFs containing embedded tables and charts. LlamaIndex offers a proprietary parsing service that excels at converting PDFs with complex tables into well-structured markdown formats. This representation can be further parsed using advanced Markdown parsing and recursive retrieval algorithms available in the open-source library. LlamaParse enables users to build RAG systems over complex documents that can answer questions involving both tabular and unstructured data.
Additionally, LlamaIndex offers various methods for transforming loaded data, such as node parsers and text splitters, making it easier for LLMs to process the data.
Are you ready to harness the power of AI in your business? Get ahead with our 𝗙𝗥𝗘𝗘 𝗠𝗶𝗰𝗿𝗼𝘀𝗼𝗳𝘁 𝗖𝗼𝗽𝗶𝗹𝗼𝘁 𝗮𝗻𝗱 𝗔𝗜 𝗥𝗲𝗮𝗱𝗶𝗻𝗲𝘀𝘀 𝗖𝗵𝗲𝗰𝗸𝗹𝗶𝘀𝘁!
2. Data Indexing
To quickly fetch relevant information from a user query, we need to index our data. An index is a data structure that is a core component for RAG use cases. Indexing is the step that follows the loading and transformation of data. Indexes store data in Node objects, which are created after splitting the original documents into chunks. There are various types of indexes:
-
Summary Index: Stores nodes as a sequential chain. Retrieved nodes can be filtered with, for example, a keyword filter or a top-k neighbor parameter, and are then loaded into the Response Synthesis module that generates a response.
-
Vector Store Index: Stores each node and its corresponding embedding in a Vector Store. When querying, the top-k most similar nodes are passed into the Response Synthesis module.
-
Tree Index: Represents a hierarchical structure of nodes. Querying a tree index involves traversing from root nodes down to leaf nodes. The child_branch_factor can be set to decide how many child nodes should be selected for a given parent node. The selected leaf nodes are then passed into the Response Synthesis module.
-
Keyword Table Index: Maps keywords found in the nodes to the corresponding nodes. When querying, relevant keywords are extracted from the query and matched with the keywords in the table index. The nodes that map to these keywords are passed into the Response Synthesis module.
3. Engines
Indexed data can be queried using the Query and Chat Engines. The Query Engine is a generic interface that allows users to ask questions about their data. The Chat Engine is an interface for having a conversation with your data.
Query Pipelines
At the beginning of 2024, LlamaIndex introduced Query Pipelines, an API within LlamaIndex that allows users to orchestrate simple-to-advanced query workflows over their data.
QueryPipeline
QueryPipeline is an abstraction that can integrate various LlamaIndex modules, such as LLMs, prompts, query engines, and retrievers. It can create a computational graph over these modules, forming either a Sequential Chain or a Directed Acyclic Graph (DAG). Retrieval Augmented Generation (RAG) systems involve a lot of query orchestration. Building an advanced RAG pipeline, consisting of query transformations, multi-stage retrieval algorithms, and the use of prompts and LLMs optimized for performance, can be a complex task.
The advantages of QueryPipeline are:
- Creating query workflows with less code: The outputs of some modules do not have to be manually converted into the inputs of other modules. This means that QueryPipeline handles the exact typing arguments for each module.
- Better readability: Reduced complexity results in better readability.
- End-to-end observability: Integrate callback functions across the entire pipeline.
Sequential Chains
Sequential Chains are simple pipelines where the output of the previous module directly goes into the input of the next module. Some examples:
- Prompt -> LLM -> Output parsing
- Retriever -> Response synthesizer
DAG for Advanced RAG Workflow
To build an advanced RAG workflow, consisting of rewriting, retrieval, reranking, synthesis, etc., users need to define the modules they want to use. Furthermore, they need to define the query pipeline and integrate the defined modules. Finally, the edges between the modules must be added to build the relationships between the modules.
Are you ready to harness the power of AI in your business? Get ahead with our 𝗙𝗥𝗘𝗘 𝗠𝗶𝗰𝗿𝗼𝘀𝗼𝗳𝘁 𝗖𝗼𝗽𝗶𝗹𝗼𝘁 𝗮𝗻𝗱 𝗔𝗜 𝗥𝗲𝗮𝗱𝗶𝗻𝗲𝘀𝘀 𝗖𝗵𝗲𝗰𝗸𝗹𝗶𝘀𝘁!
4. Evaluation and Observability
Evaluation and benchmarking are fundamental to the development of LLM applications. In this context, LlamaIndex provides essential tools for assessing and enhancing the performance of LLM applications, particularly RAG systems. The two main components of evaluation are Response Evaluation and Retrieval Evaluation.
Response Evaluation
Assessing the quality of generated responses presents unique challenges due to the nuanced nature of language understanding. LlamaIndex addresses this with LLM-based evaluation modules that measure the quality of results against various criteria, including:
- Correctness: Does the generated response align with the reference answer given the query?
- Semantic Similarity: How closely does the predicted answer resemble the reference answer in meaning?
- Faithfulness: Is the generated answer faithful to the retrieved contexts, avoiding the introduction of false information?
- Context and Answer Relevancy: Are the retrieved context and generated answer relevant to the query?
- Guideline Adherence: Does the generated response adhere to predefined guidelines or standards?
Question Generation
Beyond response evaluation, LlamaIndex facilitates question generation using the provided data. This feature enables the automatic creation of questions for evaluation purposes, allowing developers to assess the LLM's accuracy in answering queries based on the available dataset.
Retrieval Evaluation
LlamaIndex also offers modules for independent retrieval evaluation. By leveraging established ranking metrics like mean-reciprocal rank (MRR), hit rate, and precision, these modules enable a comprehensive assessment of retriever performance. The retrieval evaluation process includes two core steps:
- Dataset Generation: Synthetic creation of (question, context) pairs from unstructured text corpora.
- Retrieval Evaluation: Assessment of retrieved results using ranking metrics to evaluate the retriever against a given set of questions.
Instrumentation
The new instrumentation module fundamentally transforms how developers can track events within LlamaIndex applications. It includes several key classes:
- Event: Represents a singular moment in the execution of the application, capturing specific occurrences.
- EventHandler: Listens to events and executes custom logic in response to these occurrences.
- Span: Represents the execution flow within the application, encapsulating multiple events.
- SpanHandler: Manages the lifecycle of spans, including their entry, exit, and potential early termination due to errors.
- Dispatcher: Acts as the central hub, transmitting events and span-related signals to the appropriate handlers.
LlamaIndex's instrumentation module empowers developers to gain deeper insights into their applications' behavior by effectively tracking events and spans. By leveraging this powerful tool, developers can enhance application monitoring, debugging, and performance optimization, ultimately leading to more robust and efficient LLM applications.
Are you ready to harness the power of AI in your business? Get ahead with our 𝗙𝗥𝗘𝗘 𝗠𝗶𝗰𝗿𝗼𝘀𝗼𝗳𝘁 𝗖𝗼𝗽𝗶𝗹𝗼𝘁 𝗮𝗻𝗱 𝗔𝗜 𝗥𝗲𝗮𝗱𝗶𝗻𝗲𝘀𝘀 𝗖𝗵𝗲𝗰𝗸𝗹𝗶𝘀𝘁!
5. llama-index-networks
llama-index-networks is a llama-index library extension that enables the creation of a network of RAG systems built on external data sources and supplied by external actors. This network paradigm facilitates context augmentation, allowing for collaboration across various data sources and actors. Users can easily access these RAG systems with just a few lines of code.
Specifically, actors with their own set of documents and a RAG system must build a ContributorService around their QueryEngine and expose it via the standard llama-index Network Contributor Service.
Actors who wish to connect to the RAG systems in the llama-index Network Contributor Service can build a NetworkQueryEngine to connect to a list of ContributorServices.
Download AI Readiness Checklist
Are you ready to harness the power of AI in your business? Get ahead with our 𝗙𝗥𝗘𝗘 𝗠𝗶𝗰𝗿𝗼𝘀𝗼𝗳𝘁 𝗖𝗼𝗽𝗶𝗹𝗼𝘁 𝗮𝗻𝗱 𝗔𝗜 𝗥𝗲𝗮𝗱𝗶𝗻𝗲𝘀𝘀 𝗖𝗵𝗲𝗰𝗸𝗹𝗶𝘀𝘁!
![AI Readiness Checklist Valprovia](https://www.valprovia.com/hubfs/AI%20Readiness%20Checklist%20Valprovia.png)
Conclusion
In conclusion, LlamaIndex stands out as a robust and comprehensive framework for building production-ready LLM applications. By serving as a bridge between data and language models, it simplifies the creation of Retrieval Augmented Generation (RAG) systems through its versatile features. The framework's efficient data loading capabilities, including the use of various data connectors and tools like LlamaParse, enable seamless ingestion and transformation of diverse data types.
LlamaIndex's sophisticated indexing methods, such as Summary Index, Vector Store Index, Tree Index, and Keyword Table Index, ensure rapid and accurate information retrieval. The framework's query engines and innovative Query Pipelines further streamline complex query workflows, enhancing both functionality and usability. Evaluation and observability tools within LlamaIndex, including advanced response and retrieval evaluation metrics, provide critical insights into system performance, fostering continuous improvement.
Additionally, the instrumentation module offers detailed monitoring and debugging capabilities, contributing to the development of more reliable and efficient LLM applications. The extension through llama-index-networks introduces a collaborative dimension, allowing users to build and query networks of RAG systems, thereby augmenting context and broadening the scope of data accessibility.
Overall, LlamaIndex equips developers with powerful tools and features, significantly reducing the complexity of developing advanced LLM applications and ensuring robust, efficient, and scalable solutions for diverse use cases.