Running AI on a PC using docker with GPU support

 

             I've successfully run an AI model on my PC using docker, wsl, and other software and packages. It is the easiest way to do the former and other methods were unyielding for me (well some of you may figure it out). Without wasting much time with some unnecessary sentences let's get on to it. 

Prerequisites:

• WSL
• Docker Desktop (If you need GUI and the easiest)
• Good Network connection (If you don't want to be bored to death)
• PC (Windows or linux(well i'm not doing it for mac))
• Time

Tutorial (If you need those):

Instructions for CLI:

    Without GPU:

             Without GPU you won't have the speed that is expected for an AI model, but it will work. 

Open wsl directly or using cmd

wsl

Update the repositories

sudo apt update

Install necessary packages

sudo apt-get install \ apt-transport-https \ ca-certificates \ gnupg-agent \ software-properties-common

Fetch Docker GPG key

curl -fsSl https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Add docker repository

sudo add-apt-repository \ "deb [arch=amd64] https://download.docker.com/linux/ubuntu \ $(lsb_release -cs) \ stable"

Update repositories again

sudo apt update

Install docker

sudo apt-get install docker-ce docker-ce-cli containerd.io

Start docker services(for old systems(init))

sudo service docker start

Start docker services(for new systems)

sudo systemctl start docker

Add user to docker group to use it without su privileges(optional)

sudo usermod -aG docker $USER

Check if docker is installed(it will show docker help section)

docker

Command to run webui image(and download it if it is not downloaded).
You can give any name you want to the container by renaming pensieve.

docker run --name pensieve -p 3000:8000 --detach --restart always ghcr.io/open-webui/open-webui

Command to run ollama image(and download it if it is not downloaded).
You can give any name you want to the container by renaming dumbledore.

docker run --name dumbledore -p 11434:11434 --detach --restart always ollama/ollama

You can use any llm from ollama(i'm using qwen:0.5b).
Replace the name of the model with the one you need.

docker exec -it dumbledore ollama pull qwen2:0.5b

Open web browser and type:

127.0.0.1:3000

If all goes well in first try(which never does) you will see

Sign up if you are using it for the first time.

        With GPU(Nvidia):

You may need to install GPU drivers for linux and/or docker containers

GPG key for the official NVIDIA Docker repository

curl -fsSL https://nvidia.github.io/nvidia-docker/gpgkey | sudo apt-key add -

Set up the repository

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | sudo tee /etc/apt/sources.list.d/nvidia-docker.list

Update

sudo apt-get update

Install NVIDIA Container Toolkit

sudo apt-get install -y nvidia-docker2

Restart the Docker service(new systems)

sudo systemctl restart docker

Restart the Docker service(old systems)

sudo service docker restart

You may need to install nvidia cuda toolkit

Install nvidia cuda toolkit

sudo apt install nvidia-cuda-toolkit -y

Command to run webui image(and download it if it is not downloaded). You can give any name you want to the container by renaming pensieve.

docker run --name pensieve -p 3000:8000 --detach --restart always --gpus all ghcr.io/open-webui/open-webui

Command to run ollama image(and download it if it is not downloaded). You can give any name you want to the container by renaming dumbledore.

docker run --name dumbledore -p 11434:11434 --detach --restart always --gpus all ollama/ollama

All other commands are same as that for the commands without GPU

Instructions for GUI (well partially):

You need to follow the part were you should install the GPU drivers (without running the containers) if you need to use GPU.
Install docker desktop and open it.

Pull ollama and webui images by searching in the search bar.

Get Ollama by searching

ollama

Get webui by searching

dyrnq/open-webui

Run ollama image and open optional settings and type the host port number

11434

Run webui image and open optional settings and type the host port number

3000

Open container which uses ollama.

Go to exec section. You can use any llm from ollama(i'm using deepseek-coder). Replace the name of the model with the one you need.

ollama pull deepseek-coder

Go to browser and type

127.0.0.1:3000

Sign up if you doing it for the first time.
Congratulations for anyone who have finished it especially if you have completed it in first try (well i haven't). This took me months (atleast weeks i think) to find the procedure that works. I haven't shown you how to install the drivers and some other things in my first video so don't die.

You can give any suggestions in youtube comments.