Prompt Engineering, RAG, or Fine-Tuning? Start With the Problem

When an LLM gives you a bad answer, it is easy to assume that the model is simply not good enough. But that is not always the case. Before changing the model, we should first understand why the answer is bad. In many LLM applications, the problem usually comes down to one of three things: the model does not understand what you want, does not have the information it needs, or is not capable enough to perform the task well. These three problems lead to three different solutions: Prompt Engineering, RAG, and Fine-Tuning.

Problem 1: The Model Does Not Understand Your Request

Imagine you ask an LLM: Help me analyze this product. The request is very broad. What should the model analyze? Price? Customer reviews? Competition? Profitability? Market demand? Even a powerful model may give you a weak answer because your instructions are not clear enough. This is where Prompt Engineering helps. Instead of changing the model, you can improve the instructions you give it. For example, you can ask it to analyze a product based on customer demand, competition, price, and potential risks, then summarize the results in four sections and provide a final recommendation. Now the model has a much clearer understanding of the task. In this case, the problem was not missing knowledge or weak model capabilities. The problem was simply unclear communication. So a useful rule is that if the model does not understand what you want, improve the prompt first.

Problem 2: The Model Does Not Have the Knowledge

Now imagine your company has hundreds of internal documents, including PDF files, Word documents, TXT files, and spreadsheets. You ask the LLM: What is our company’s policy for approving a new supplier? The model may understand the question perfectly, but it may not know your company’s internal policy. Making the prompt longer will not solve this problem. The model needs access to information that is outside its existing knowledge. This is where RAG, or Retrieval-Augmented Generation, becomes useful. With RAG, the system first searches your documents for relevant information. It then provides that information to the LLM as context so the model can generate an answer based on the retrieved content. The important point is that RAG does not primarily change the model itself. Instead, it gives the model access to the information it needs when answering a question. So another useful rule is that if the model understands the task but does not have the required knowledge, consider RAG.

Problem 3: The Model Cannot Perform the Task Well Enough

There is another situation. The model understands your instructions. It also has access to the necessary information. But its performance is still not good enough. For example, you may want the model to consistently classify customer messages into a specialized set of business categories. You provide clear instructions and examples, but the model still makes too many mistakes. At this point, the problem may no longer be about the prompt or external knowledge. You may need to improve how the model performs the task itself. This is where Fine-Tuning can help. Fine-tuning trains a model using examples that demonstrate the behavior or capability you want. Instead of repeatedly explaining everything through prompts, you adjust the model so that it becomes better suited to a particular task or pattern. So the third rule is that if the model has the information and understands the task but still cannot perform it well enough, fine-tuning may be worth considering.

Start With Diagnosis, Not Technology

Prompt Engineering, RAG, and Fine-Tuning are not three competing technologies where you simply choose the “best” one. They solve different problems.

In real applications, you may also combine them. For example, a fine-tuned model can still use RAG to access company documents, while carefully designed prompts control how it responds to the user. In practice, you can combine prompt engineering, RAG, and a fine-tuned model to build a better LLM application. The key is to avoid starting with a technology. Do not begin by asking: Should we use RAG or fine-tuning? Instead, start by asking: Why is the model giving us a bad answer? Once you understand the problem, choosing the right technique becomes much easier.