• Ratings
• Pros/cons
• Description
  • General
  • Quantization
  • Model parameters
• Finding models
  • How to find suitable models
  • Overview of models and benchmarks
• About the developers
  • Llama
  • Qwen
  • Mistral
  • Gemma
  • DeepSeek
• Alternatives

Accuracy / Quality: ★★★☆☆

Flexibility / Features: ★★★★☆

Data security / Privacy: ★★★★★

Pros

• Customizable for many different (language) models.
• Easily updated to use newly released models.
• Data-secure and private.
• Low technical difficulty.
• Model parameters can be (partially) modified.

Cons

• Typically lower in quality than commercially hosted models.
• Slower than commercially hosted models due to lower hardware quality.
• Fewer features than commercial models.
• At least 5-10 GB local storage space required (smaller models).

Using commercially hosted models (e.g. ChatGPT, Gemini or Claude) can have some significant drawbacks. Whenever you enter information into these models there is a risk that the information provided ends up in the training data of the model. This means that there can be no safe usage of these tools when working with either personal sensitive information (e.g. names, phone numbers, (mail) addresses, etc.), but also not with academically sensitive information (research questions, hypotheses, new methodologies, funding information, etc.). Unless a formal (written) agreement is made between the company in question and the WUR, using an open-source model on your own device can be a solution to this problem.

LM Studio (or alternatively: Ollama) is an example of software that allows the user to install language models on their own device. Doing so would allow the model to run on the hardware of the user, rather than the large servers of external companies. Any data you enter into models that are operating locally (on your own device) is not sent to any external servers, and is therefore data secure. Even if the internet connection would fall away, these models would continue to be usable.

When installing models in LM Studio (or Ollama) you do so via GGUF-files. These are quantized versions of released open source models. The information in the model resulting from the training of the model is stored as large numbers, often in 32- or 64-bit lengths, such that the information is maintained with high precision. In the quantization process these lengths are reduced to e.g. 8-bit lengths. This shrinks down the size of the model to a more manageable volume for local devices, but comes at the cost of the accuracy of the model. This does not necessarily need to be a problem for the functioning of the model, as the accuracy provided by the high precision of the data is not always required. But if the precision is affected too much, the model will increasingly hallucinate or be unable to answer. In simple terms this process can be compared to the value for Pi: Having more decimals makes the calculation more precise, but not necessarily better.
The level of quantization can be found in the model name, with Q8 (8-bit) referring to a model in which a large degree of precision has been kept, whereas a Q2 (2-bit) version has lost a lot of precision but is significantly smaller in size. Conventionally the Q4 or Q5 quantized versions of models are an acceptable balance between size and accuracy for most tasks.

To find GGUF-versions of open source models the best place to start is the Huggingface website (https://huggingface.co/). This website serves as a repository of open source models developed by both large companies as well as fine-tuned versions of those models released by individuals. To find suitable models for LM Studio you can look on the Models page. On this page, apply the 'Text generation' filter in the 'Tasks' tab, and the 'GGUF' filter in the 'Libraries' tab. What you are left with are all potentially usable open source language models. 
You will rarely find GGUF-versions of models together with the originally released models. Hence we recommend looking at the versions released by Bartowski or the LM Studio Community. Inside the model page you can download the version of the model you desire via the 'Files' tab. Alternatively, you can also use the 'Discover' functionality within LM Studio itself, though this is a more limited list of available model versions.

Next, you will need to find a suitable model from the list of available ones. When comparing the available models, you will find that the larger the model (indicated by the B-value in the name, referring to the billions of parameters in the model) the higher the accuracy. However, this also means that the model is significantly larger to install and run. In this guide we will list a handful of versions, but will leave out the largest versions of models as these are unlikely to be usable on commonly used devices. We will only list 'Instruct' or 'It' versions of models, as these have been specifically fine-tuned to operate as a chatbot, making them more suitable for the application in LM Studio.

Each model has its own strengths and weaknesses, and their own sizes and quality of the training data. Luckily this can be assessed via independent benchmarks or via blind tests. Benchmarks are datasets containing questions (and answers) which can be posed to a language model to test its capabilities or accuracy on various tasks. These questions are (ideally) not present in the training data of the model, allowing the ranking of the models by the percentage of questions correctly answered. In this guide we will use the Livebench (https://livebench.ai/) benchmark website. Alternatively, models can also be tested blindly by users by presenting users with the output of two or more models to a prompt they submitted, after which the user indicates which model (without knowing the name of the model) performed best. Based on this information a ranking can be established from real user experience. In this guide we use the LM Arena (https://lmarena.ai/) website for this ranking. Note that both the benchmark and blind test rankings were done on the full versions of the models, not the quantized versions. Hence a small decrease in quality compared to the listed ranking can be expected when using the model locally via LM Studio.

When using language models you should be aware of the biases inherent in them, and the limitations of the models. Some of these are tied to the developers of the models. Hence we will briefly go over each developer and provide some background information on them to provide context for the use of their respective models.

• Ollama: https://ollama.com/
• Installing the full model and operating it via programming language (commonly Python). This requires more technical knowledge and is less user-friendly.