Tag: ollama

Run ollama on linux

Ollama it has similar pattern as docker.

  • Download 
curl -L https://ollama.com/download/ollama-linux-amd64.tgz -o ollama-linux-amd64.tgz
sudo tar -C /usr -xzf ollama-linux-amd64.tgz

ollama serve

# OR host on different ip
OLLAMA_HOST=192.168.29.13:11435 ollama serve

ollama -v

#check graphics card
nvidia-smi


#port http://127.0.0.1:11434/

###Podman/Docker - https://ollama.com/blog/ollama-is-now-available-as-an-official-docker-image

podman run -d --gpus=all --device nvidia.com/gpu=all --security-opt=label=disable -v ollama:/root/.ollama -p 11434:11434  ollama/ollama
  • Run
user@home:~$ ollama list
NAME               ID              SIZE      MODIFIED     
gemma2:latest      ff02c3702f32    5.4 GB    11 hours ago    
llama3.2:latest    a80c4f17acd5    2.0 GB    12 hours ago 

user@home:~$ ollama run llama3.2
>>> hola
Hola! ¿En qué puedo ayudarte hoy?

>>> hey
What's up? Want to chat about something in particular or just shoot the breeze?
  • Service with environment – /etc/systemd/system/ollama.service
[Unit]
Description=Ollama Service
After=network-online.target

[Service]
ExecStart=/usr/local/bin/ollama serve
User=ollama
Group=ollama
Restart=always
RestartSec=3
Environment="PATH=/usr/local/vitess/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/root/cardano:/root/cardano/bin"
Environment="OLLAMA_HOST=http://192.168.29.13:11434"
[Install]
WantedBy=default.target

podman run -d -p 3000:8080 --gpus all --device nvidia.com/gpu=all --security-opt=label=disable -e OLLAMA_BASE_URL=http://192.168.29.13:11434 -e WEBUI_AUTH=False -v open-webui:/app/backend/data --name open-webui ghcr.io/open-webui/open-webui:main

https://github.com/ollama/ollama/blob/main/docs/linux.md

for podman GPU access cdi- https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html

docker-compose-web.yml

services:
  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    restart: always
    container_name: open-webui
    environment:
      - OLLAMA_BASE_URL=http://192.168.29.142:11434
      - WEBUI_AUTH=False
    volumes:
      - open-webui:/app/backend/data
    ports:
      - 3000:8080
volumes:
  open-webui:
  • Verify nvidia driver 
docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi
user@home:~$ cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  550.120  Fri Sep 13 10:10:01 UTC 2024

apt install nvidia-utils-550

nvtop
  • Problem with 550 that it’s older and ubuntu 24.04 is try to install the latest version
  • Remove everything
sudo apt purge 'nvidia-*' 'linux-modules-nvidia-*'
sudo apt autoremove --purge

sudo rm -rf /lib/modules/$(uname -r)/updates/dkms/*
sudo rm -rf /var/lib/dkms/nvidia*

sudo apt install --reinstall linux-headers-$(uname -r) linux-image-$(uname -r)

#check secure boot, disable it in BIOS
mokutil --sb-state

sudo apt install nvidia-driver-535

reboot

#check with smi command
docker run --rm --gpus all nvidia/cuda:12.2.0-base-ubuntu22.04 nvidia-smi
  • As of now 535 is working fine now.
  • Install driver for docker to communicate with nvidia
distribution=$(. /etc/os-release;echo $ID$VERSION_ID)

curl -s -L https://nvidia.github.io/libnvidia-container/gpgkey | sudo apt-key add -

curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update
sudo apt install -y nvidia-container-toolkit