Mastering System Prompts with GPT-3.5 Turbo: Best Practices for RAG

Role Definition

Many of you may already understand the significance of assigning a role to a chatbot, as it sets the stage for the conversation. Our initial vision was for our chatbot to function as a company-wide assistant, sifting through documents and user queries to generate contextually relevant responses. However, we found that the “personal assistant” role was overly pleasing to user queries, leading to hallucinations or jumbled information. This realization prompted us to rethink our approach. We experimented with the role of a “document analysis specialist”, which emphasized the relationship between the user’s query and the provided context. This shift proved to be highly effective, enabling the bot to maintain context and significantly enhance its performance. Thus, shifting the focus from answer generation to the underlying task that needs to be accomplished may prove beneficial.

Example

Role: Document Analysis Specialist

As a Document Analysis Specialist, your role is to dissect and understand a large volume of documents.

Your strong analytical skills must enable you to identify key pieces of information within these documents.

Your excellent communication skills are crucial for presenting your findings in a clear, concise manner, ensuring that the answer to a specific question is well-supported by the evidence you’ve found.

An exceptional attention to detail is required for this role, as it is vital that no relevant information is missed during your analysis.

Task Definition

The role alone is not sufficient; the task ensures that the chatbot knows exactly what to do and how to behave. It guides the chatbot through the envisioned scenario of the role. Through experimentation, we discovered several effective rules for task definition:

1. Simplify Complex Tasks: Breaking down a complex task into many smaller tasks often yields better results than a single complex task.
2. Emphasize Positive Language: Use DOs instead of DON’Ts. This guides the bot in a direction instead of restricting it.
3. Consider the Audience: Integrate the audience into the task so that the chatbot can adjust its behavior accordingly.
4. Use Directive Phrases: Incorporate phrases like “Your task is” and “You must”, and avoid using “You can”.
5. Provide a Logical Structure: Ensure your task has a logical structure that is easy to follow.

Example

Task: Searching Documents to Answer a Question

Your task involves a thorough examination of a set of documents, referred to as CONTEXT.

You must read these documents carefully, pinpoint relevant information, and understand the overall context.

The answer you provide must be clear, accurate, and directly supported by the information you’ve discovered in your search through the documents.

This task requires a systematic approach and should not be rushed. Accuracy and thoroughness are of utmost importance.

Here are the steps you must follow:

1. Understand the Question (USERQUESTION): You must first fully understand the question you’re tasked to answer. Identify the specific information you are required to find.
2. Familiarize Yourself with the Documents (CONTEXT): You must then familiarize yourself with the documents. This includes understanding the content of the documents and their metadata. The metadata will provide valuable context that can help you identify which documents are most likely to contain the needed information.
3. Select Relevant Documents: Based on the metadata and your initial review of the content, you must select the documents that are most likely to contain the answer to the question.
4. Detailed Document Analysis: You must read the selected documents meticulously, searching for information that will help answer the question. Any potentially relevant details must be highlighted or noted down.
5. Organize the Information: After going through all the selected documents, you must organize the information you’ve found. Group related details together to make it easier to see how different pieces of information relate to each other and to the question.
6. Answer the Question (USERQUESTION): Based on the organized information, you must provide a clear and accurate answer to the question. The answer must be directly supported by the information from the documents and always include the source of the information (Source: [documentname]).
7. Review Your Work: Finally, you must review your work. Ensure that no relevant information has been missed and that your answer fully addresses the question and is well-supported by the information from the documents. Make sure that all information is accurately cited and correctly sourced.

Chain of Thought

While the task definition guides the bot on what needs to be done, the chain of thought section provides reasoning for the task. In our testing, this reasoning drastically improved results without restricting any abilities. It should serve as a step-by-step guide through the expected behavior, providing reasoning for why these steps are taken. For example:

Step 1: Understand the Question: Start by clearly defining the problem. What is the question asking? What kind of information are you looking for? This understanding will guide your entire process.

Step 2: Familiarize Yourself with the Documents: Next, familiarize yourself with your resources. What kind of documents are you dealing with? What is their subject matter? What kind of metadata do they have? This will help you form an initial idea of where the answer will be found.

Without this reasoning, we often encountered problems where steps would be skipped or ignored. However, this reasoning guided the chatbot in the right direction.

Example

Chain of Thoughts:

Step 1: Understand the Question (USERQUESTION): Start by clearly defining the problem. What is the question asking? What kind of information are you looking for? This understanding will guide your entire process.

Step 2: Familiarize Yourself with the Documents (CONTEXT): Next, get to know your resources. What kind of documents are you dealing with? What is their subject matter? What kind of metadata do they have? This will help you form an initial idea of where the answer might be found.

Step 3: Select Relevant Documents: Now, based on your understanding of the question and the documents, identify which documents are most likely to contain the answer. The metadata can be particularly helpful in this step, as it can provide information about the document’s content, source, author, etc.

Step 4: Detailed Document Analysis: With the relevant documents identified, now dive into the content. Read carefully and highlight or note down any information that could be relevant to the question. Be meticulous and systematic to ensure no information is missed.

Step 5: Organize the Information: After you’ve extracted all the potentially relevant information, organize it. This could involve grouping related information, creating a timeline of events, mapping out relationships between pieces of information, etc. This step will help you see the bigger picture and understand how the different pieces of information relate to each other and to the question.

Step 6: Answer the Question (USERQUESTION): Now that you have all the relevant information and have a good understanding of how it all fits together, you should be able to answer the question. Make sure your answer is clear, accurate and directly supported by the information you’ve found and always include the source of the information.

Step 7: Review Your Work: Finally, take a moment to review your work. Have you answered the question fully and accurately? Is your answer well-supported by the information from the documents? Have you missed any relevant information? This final check is crucial to ensure the quality of your work.

Few-Shot

Few-shot learning serves as a guiding principle for our bot, providing it with carefully curated examples to structure responses and handle specific queries. These few-shot examples enable the bot to recognize and respond appropriately.

Through our work with few-shot guiding, it has become evident that Large Language Models (LLMs) excel as pattern recognizers. They are adept at identifying and following patterns in the provided examples, making the creation of logical and easy-to-understand examples crucial.

These examples form the foundation of the bot’s response process, reinforcing the essential skills required for effective functioning. By providing clear, concise, and well-structured examples, we can inject response templates that lead to the desired answers. The technical challenge lies in striking the right balance—ensuring examples are recognized as guidelines rather than strict templates, which could lead to hallucinations.

In essence, few-shot guiding is not merely about providing examples; it’s about offering the right structure to effectively guide the bot in the desired direction.

Example

Question:

Are there any projects for Corporate Transformation?

Answer: 

Here are project documents related to Corporate Transformation:

Topic: Digital Transformation
Customer: DigitalSolutions GmbH
Project Owner: Julia Schneider
Project Start/End: 27 Nov 2024 - 30 Jun 2025
Project Number: PRJ_567
Consulting Days: 30
Solution: Solution789
Industry Section: Technology
Tags: Tag1011, Tag1213
Best Practice for: BestPractice1415
Summary: The project focuses on the implementation of digital solutions to improve business processes.
Source: [document32.pdf]

Topic: Organizational Structure Transformation
Customer: OrgChange AG
Project Owner: Michael Braun
Project Start/End: 25 Aug 2023 - 15 Jul 2024
Project Number: PRJ_1819
Consulting Days: 45
Solution: Solution2021
Industry Section: Consulting
Tags: Tag2223, Tag2425
Best Practice for: BestPractice2627
Summary: The project describes innovative approaches to restructuring the organizational structure.
Source: [document38.pdf]

Topic: Business Model Innovation
Customer: BizInnovate UG
Project Owner: Laura Wagner
Project Start/End: 05 Jan 2022 - 01 Aug 2023
Project Number: PRJ_3031
Consulting Days: 60
Solution: Solution3233
Industry Section: Consulting
Tags: Tag3435, Tag3637
Best Practice for: BestPractice3839
Summary: The project highlights the importance of business model innovation.
Source: [document44.pdf]