We have LLM at home!

This will install Local LLM to your server. Basic installation. You will need to do a lot before get benefits from having LLM at home.

Prepare server:

Ensure Docker is installed on your server. Use the following commands to install Docker and docker-compose if not already installed:

apt update
apt install -y docker.io
systemctl start docker
systemctl enable docker
apt install -y docker-compose

Ensure sufficient CPU and memory are allocated for running the Ollama server and Open WebUI.

Create docker-compose.yml file and add following configuration to it:

services:
  ollama-server:
    image: ollama/ollama:latest
    container_name: ollama-server
    ports:
      - "11434:11434" # Ollama API port
    environment:
      - USE_CPU=1 # Force CPU usage
    restart: unless-stopped
    networks:
      inside:
        ipv4_address: 10.100.10.10 # Static IP for Ollama server

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "80:8080" # Open WebUI port
    environment:
      - OLLAMA_BASE_URL=http://10.100.10.10:11434 # Reference Ollama server via static IP
    volumes:
      - ./data:/app/data # Mount folder for manuals
    restart: unless-stopped
    depends_on:
      - ollama-server # Ensure Ollama server starts before Open WebUI
    networks:
      inside:
        ipv4_address: 10.100.11.10 # Static IP for Open WebUI

networks:
  inside:
    driver: bridge
    ipam:
      config:
        - subnet: 10.100.10.0/23 # Combined subnet to include both servers

Start it

docker-compose up -d

Script to feed your LLM (upload all text files from ~/feed_data to LLM)

        for file in ~/feed_data/*.txt; do
            curl -X POST http://<your_server_ip>:11434/embed \
                 -d "{\"text\": \"$(cat $file)\"}"
        done