Local LLM

How to setup a local LLM

Intro

Since I wanted to experiment with AI and Large Language Models. I made a local LLM system.

With the popularity of AI and LLMs and the costs that comes with them, it might also be an advantage to have a local LLM.

The free tiers could become restricted in such a way that one has to pay to use it in a normal way.


The Virtual Machine

- vCores: 4
- RAM: 12GB
- GPU: RTX 2060 6GB
- OS: Ubuntu/Debian

Setup Ollama

Installing Ollama is very easy. Go to Ollama then click on the download button and select your Operating System. 
sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
Environment="OLLAMA_CONTEXT_LENGTH=16384"
        

sudo systemctl daemon-reload
sudo systemctl restart ollama
Ensure ample system RAM (e.g., 32GB+ for 128k contexts). Use OLLAMA_NUM_PARALLEL=1 and OLLAMA_MAX_LOADED_MODELS=1 env vars to minimize fragmentation.

Setup Open WebUI

Preparing the Open WebUI with the following commands: 
#!/usr/bin/env bash
mkdir openwebui
cd openwebui
python3 -m venv venv
source venv/bin/activate
pip install open-webui
Setting up the daemon 
#!/usr/bin/env bash
which open-webui
sudo useradd --system --create-home --shell /usr/sbin/nologin openwebui
ls -l /home/user/openwebui/venv/bin/open-webui
/home/user/openwebui/venv/bin/open-webui --help
nano /etc/systemd/system/openwebui.service
Content of the daemon 
[Unit]
Description=Open WebUI (venv)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=user
WorkingDirectory=/home/user/openwebui
Environment=PATH=/home/user/openwebui/venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=HOST=0.0.0.0
Environment=WEBUI_URL=https://llm.odicia.net
ExecStart=/home/user/openwebui/venv/bin/open-webui serve --port 8880
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target
Enable the daemon 
sudo systemctl daemon-reload
sudo systemctl enable --now openwebui.service
sudo systemctl status openwebui.service
After all that enter the IP of the server and the port of the server like this: IP:PORT to access Open WebUI.