{"id":21032,"date":"2023-11-20T14:15:10","date_gmt":"2023-11-20T22:15:10","guid":{"rendered":"https:\/\/www.pugetsystems.com\/?post_type=hpc_post&#038;p=21032"},"modified":"2024-02-05T13:06:59","modified_gmt":"2024-02-05T21:06:59","slug":"llm-server-setup-part-2-container-tools","status":"publish","type":"hpc_post","link":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/","title":{"rendered":"LLM Server Setup Part 2 &#8212; Container Tools"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_80 counter-hierarchy ez-toc-counter ez-toc-transparent ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Introduction\" >Introduction<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Docker\" >Docker<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#NVIDIA_Enroot\" >NVIDIA Enroot<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Container_Tools\" >Container Tools<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_1_%E2%80%93_Install_the_NVIDIA_Container_Toolkit\" >Step 1 &#8211; Install the NVIDIA Container Toolkit<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_2_%E2%80%93_Install_Docker\" >Step 2 &#8211; Install Docker<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_21_%E2%80%93_Uninstall_old_versions\" >Step 2.1 &#8211; Uninstall old versions<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-8\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_22_%E2%80%93_Add_Dockers_official_GPG_key\" >Step 2.2 &#8211; Add Docker&#8217;s official GPG key:<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-9\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_23_%E2%80%93_Add_the_Docker_repository_to_Apt_sources\" >Step 2.3 &#8211; Add the Docker repository to Apt sources:<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-10\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_24_%E2%80%93_Install_the_Docker_packages\" >Step 2.4 &#8211; Install the Docker packages:<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-11\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_25_%E2%80%93_Verify_that_Docker_Engine_is_installed_correctly_by_running_the_hello-world_image\" >Step 2.5 &#8211; Verify that Docker Engine is installed correctly by running the hello-world image.<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-12\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_3_%E2%80%93_Docker_Post_Install\" >Step 3 &#8211; Docker Post Install<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-13\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_31_Create_a_docker_user_access_group\" >Step 3.1 Create a docker user access group.<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-14\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_32_Add_your_user_account_to_the_docker_group\" >Step 3.2 Add your user account to the docker group.<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-15\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_33_Activate_the_changes_to_the_system_groups\" >Step 3.3 Activate the changes to the system groups.<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-16\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_34_Verify_that_you_can_run_docker_commands_without_sudo\" >Step 3.4 Verify that you can run docker commands without sudo.<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-17\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_35_Enable_docker_service_daemons_to_start_on_boot\" >Step 3.5 Enable docker service daemons to start on boot<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-18\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_36_%E2%80%9CArrrrggg%E2%80%9D_%E2%80%93_What_to_do_if_Docker_breaks_your_LAN_hopefully_optional\" >Step 3.6 &#8220;Arrrrggg!!&#8221; &#8211; What to do if Docker breaks your LAN! (hopefully optional)<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-19\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_37_%E2%80%93_Configure_Docker_to_use_the_NVIDIA_Container_Toolkit\" >Step 3.7 &#8211; Configure Docker to use the NVIDIA Container Toolkit<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-20\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_38_Fix_Docker_default_shm_size_and_other_runtime_configuration_env_parameters\" >Step 3.8 Fix Docker default shm size and other runtime configuration env parameters<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-21\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Step_4_%E2%80%93_Install_NVIDIA_Enroot_optional_but_recommended\" >Step 4 &#8211; Install NVIDIA Enroot (optional but recommended)<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-22\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#Next_Steps\" >Next Steps<\/a><\/li><\/ul><\/nav><\/div>\n\n<h2 class=\"wp-block-heading\" id=\"h-introduction\"><span class=\"ez-toc-section\" id=\"Introduction\"><\/span>Introduction<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p><strong>This post is Part 2 in a series on configuring<\/strong> a system for LLM deployments and development usage. The configuration will be suitable for multi-user deployments and also useful for smaller development systems. I will describe configuration steps in detail with commands suitable for scripts to automate the process. Part 2 is about installing and configuring container tools, Docker,<strong> and NVIDIA Enroot.<\/strong><\/p>\n\n\n\n<p><strong>These individual HowTo posts will be useful on their own and not just for LLM server configs!<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Part 1<\/strong> &#8212; <a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/\">LLM Server Setup Part 1 \u2013 Base OS<\/a> Describes installing Ubuntu Server 22.04, applying configuration changes\/fixes, and installing the NVIDIA driver package.<\/li>\n<\/ul>\n\n\n\n<p>Containerization has become the most sensible server application deployment methodology for deploying any complex application code. This is equally true for deployments in the cloud, on-prem server hardware, or even personal computers for development work.<\/p>\n\n\n\n<p>For reference, the machines that are my deployment targets are a couple of these: <a href=\"https:\/\/www.pugetsystems.com\/solutions\/scientific-computing-workstations\/machine-learning-ai\/buy-404\/\">Puget Systems Qudad NVIDIA 6000Ada 5U system<\/a>. These can serve Llama2-70b, and derivatives for multi-user use with good performance. I also do this at home with a much more modest personal system.<\/p>\n\n\n\n<p>We will go through two container development and runtime environments, Docker and NVIDIA Enroot.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-docker\"><span class=\"ez-toc-section\" id=\"Docker\"><\/span>Docker<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p><a href=\"https:\/\/www.docker.com\/\">Docker<\/a> is essentially the default containerization framework. It is becoming common for applications to include a Dockerfile or Compose file for deployment. This is a very welcomed trend! When dealing with packages with non-trivial setup requirements and dependencies, if they do not include access to a docker container or build file, you are usually best served to build one yourself.<\/p>\n\n\n\n<p>The biggest downside of using Docker is that its installation digs itself deep into your system. It runs as a privileged daemon process and makes significant changes to your system. However, you likely need to have Docker available, at least for development purposes. For application deployment, you may be able to use a simpler user-space container runtime application.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-nvidia-enroot\"><span class=\"ez-toc-section\" id=\"NVIDIA_Enroot\"><\/span>NVIDIA Enroot<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p><a href=\"https:\/\/github.com\/NVIDIA\/enroot\">NVIDIA Enroot<\/a> is my favorite container runtime tool! It is not well known, but it is a true gem. I have written a blog post introducing Enroot that I recommend checking out if you are curious and want to see the arguments for using it. <a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/run-docker-containers-with-nvidia-enroot-2142\/\">Run \u201cDocker\u201d Containers with NVIDIA Enroot<\/a><\/p>\n\n\n\n<p>One of my principal reasons for including Enroot in this post is that it can be used to create <strong>container bundles<\/strong>. These container bundles are self-contained, portable, minimal dependency, runnable container applications that do not require any container runtime to be installed on the target system. Enroot will include itself in the bundle to run the container! I have a post describing this: <a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/self-contained-executable-containers-using-enroot-bundles-2181\/\">Self-Contained Executable Containers Using Enroot Bundles<\/a>.<\/p>\n\n\n\n<p>Here is a clip from the <a href=\"https:\/\/github.com\/NVIDIA\/enroot\">Enroot GitHub page<\/a>,<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-key-concepts\">Key Concepts<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Adheres to the KISS principle and Unix philosophy<br>Standalone (no daemon)<\/li>\n\n\n\n<li>Fully unprivileged and multi-user capable (no setuid binary, cgroup inheritance, per-user configuration\/container store\u2026)<\/li>\n\n\n\n<li>Easy to use (simple image format, scriptable, root remapping\u2026)<\/li>\n\n\n\n<li>Little to no isolation (no performance overhead, simplifies HPC deployments)<\/li>\n\n\n\n<li>Entirely composable and extensible (system-wide and user-specific configurations)<\/li>\n\n\n\n<li>Fast Docker image import (3x to 5x speedup on large images)<\/li>\n\n\n\n<li>Built-in GPU support with libnvidia-container<\/li>\n\n\n\n<li>Facilitate collaboration and development workflows (bundles, in-memory containers\u2026)<\/li>\n<\/ul>\n\n\n\n<p>Most of this post will be concerned with installing and configuring Docker since it is much more complicated than Enroot (which doesn&#8217;t require configuration.) After the Docker setup steps, I&#8217;ll give instructions for installing Enroot. I will use Enroot in later posts (in addition to Docker.)<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-container-tools\"><span class=\"ez-toc-section\" id=\"Container_Tools\"><\/span>Container Tools<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>We are picking up from where we left off in <a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/\">Part 1<\/a> after the OS setup and NVIDIA driver install.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-step-1-install-the-nvidia-container-toolkit\"><span class=\"ez-toc-section\" id=\"Step_1_%E2%80%93_Install_the_NVIDIA_Container_Toolkit\"><\/span>Step 1 &#8211; Install the NVIDIA Container Toolkit<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>This set of tools will allow us to run containers with NVIDIA GPU support. [Note that this has been recently updated, and as of this writing, there are mixed instructions on the web.] <a href=\"https:\/\/docs.nvidia.com\/datacenter\/cloud-native\/container-toolkit\/install-guide.html\">Current reference guide.<\/a><\/p>\n\n\n\n<p>Add the signing key and install the repository information.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>curl -fsSL https:\/\/nvidia.github.io\/libnvidia-container\/gpgkey | sudo gpg --dearmor -o \/usr\/share\/keyrings\/nvidia-container-toolkit-keyring.gpg \\\n  &amp;&amp; curl -s -L https:\/\/nvidia.github.io\/libnvidia-container\/stable\/deb\/nvidia-container-toolkit.list | \\\n    sed 's#deb https:\/\/#deb &#091;signed-by=\/usr\/share\/keyrings\/nvidia-container-toolkit-keyring.gpg] https:\/\/#g' | \\\n    sudo tee \/etc\/apt\/sources.list.d\/nvidia-container-toolkit.list <\/code><\/pre>\n\n\n\n<p>Update and install the toolkit.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt update\nsudo apt install -y nvidia-container-toolkit<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_2_%E2%80%93_Install_Docker\"><\/span>Step 2 &#8211; Install Docker<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Docker is ubiquitous and well-supported. I use it for creating\/distributing new containers. I do prefer user-space container applications when I can use them. <a href=\"https:\/\/podman.io\/\">Podman<\/a>, for example, is an excellent alternative to docker, but it is unfortunately not well supported (kept current) by Canonical\/Ubuntu. (I may post separately about installing an up-to-date Podman on Ubuntu.) Also, there are times when you just really need to use docker.<\/p>\n\n\n\n<p>I will mostly follow the official Docker docs for the base Install.<br>Reference: <a href=\"https:\/\/docs.docker.com\/engine\/install\/ubuntu\/\">https:\/\/docs.docker.com\/engine\/install\/ubuntu\/<\/a><\/p>\n\n\n\n<p>For GPU setup, we will take guidance from NVIDIA.<br>Reference: <a href=\"https:\/\/docs.nvidia.com\/datacenter\/cloud-native\/container-toolkit\/install-guide.html#docker\">https:\/\/docs.nvidia.com\/datacenter\/cloud-native\/container-toolkit\/install-guide.html#docker<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_21_%E2%80%93_Uninstall_old_versions\"><\/span>Step 2.1 &#8211; Uninstall old versions<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>This post assumes a fresh Ubuntu server install, but if you have an old version of Docker installed for any reason, you will want to remove it first.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt-get remove docker docker-engine docker.io docker-compose docker-doc containerd runc<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_22_%E2%80%93_Add_Dockers_official_GPG_key\"><\/span>Step 2.2 &#8211; Add Docker&#8217;s official GPG key:<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt-get update\nsudo apt-get install ca-certificates curl gnupg\nsudo install -m 0755 -d \/etc\/apt\/keyrings\ncurl -fsSL https:\/\/download.docker.com\/linux\/ubuntu\/gpg | sudo gpg --dearmor -o \/etc\/apt\/keyrings\/docker.gpg\nsudo chmod a+r \/etc\/apt\/keyrings\/docker.gpg<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_23_%E2%80%93_Add_the_Docker_repository_to_Apt_sources\"><\/span>Step 2.3 &#8211; Add the Docker repository to Apt sources:<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>echo \\\n  \"deb &#091;arch=\\\"$(dpkg --print-architecture)\\\" signed-by=\/etc\/apt\/keyrings\/docker.gpg] https:\/\/download.docker.com\/linux\/ubuntu \\\n  \\\"$(. \/etc\/os-release &amp;&amp; echo \\\"$VERSION_CODENAME\\\")\\\" stable\" | \\\n  sudo tee \/etc\/apt\/sources.list.d\/docker.list &gt; \/dev\/null\n\nsudo apt-get update<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_24_%E2%80%93_Install_the_Docker_packages\"><\/span>Step 2.4 &#8211; Install the Docker packages:<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_25_%E2%80%93_Verify_that_Docker_Engine_is_installed_correctly_by_running_the_hello-world_image\"><\/span>Step 2.5 &#8211; Verify that Docker Engine is installed correctly by running the hello-world image.<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo docker run hello-world<\/code><\/pre>\n\n\n\n<p>That should run successfully as root. We will want to run docker as a non-root user and will configure that next.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_3_%E2%80%93_Docker_Post_Install\"><\/span>Step 3 &#8211; Docker Post Install<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p><a href=\"https:\/\/docs.docker.com\/engine\/install\/linux-postinstall\/\">Reference: https:\/\/docs.docker.com\/engine\/install\/linux-postinstall\/<\/a><\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_31_Create_a_docker_user_access_group\"><\/span>Step 3.1 Create a docker user access group.<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo groupadd docker<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_32_Add_your_user_account_to_the_docker_group\"><\/span>Step 3.2 Add your user account to the docker group.<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo usermod -aG docker $USER<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_33_Activate_the_changes_to_the_system_groups\"><\/span>Step 3.3 Activate the changes to the system groups.<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>newgrp docker<\/code><\/pre>\n\n\n\n<p>After these steps (no re-login required), you should be able to run docker as a regular user. The containers still run as root, though.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_34_Verify_that_you_can_run_docker_commands_without_sudo\"><\/span>Step 3.4 Verify that you can run docker commands without sudo.<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>docker run hello-world<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_35_Enable_docker_service_daemons_to_start_on_boot\"><\/span>Step 3.5 Enable docker service daemons to start on boot<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo systemctl enable --now docker.service \nsudo systemctl enable --now containerd.service<\/code><\/pre>\n\n\n\n<p>The <code>--now<\/code> option will start the service immediately.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_36_%E2%80%9CArrrrggg%E2%80%9D_%E2%80%93_What_to_do_if_Docker_breaks_your_LAN_hopefully_optional\"><\/span>Step 3.6 &#8220;Arrrrggg!!&#8221; &#8211; What to do if Docker breaks your LAN! (hopefully optional)<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Docker is a fairly intrusive application. Mysterious network problems occasionally happen. The one I have to deal with most often is docker starting its internal network pool on the same subnet as an already-in-use LAN at my location. This causes all kinds of problems, and it is not well documented.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Check the docker network config.<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>docker network inspect bridge<\/code><\/pre>\n\n\n\n<p>If the docker bridge network is on the same subnet as one of your LAN segments, you will need to change it.<br>DO NOT DO THIS IF YOU DON&#8217;T HAVE TO!<br>You can change the docker default network pools by editing \/etc\/docker\/daemon.json. With a fresh Docker install, the daemon.json file won&#8217;t exist yet. The following script section will create the file and change the default network pool to 10.13.0.0\/16 (assuming that it does not interfere with your LAN.)<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo '{\n  \"default-address-pools\": &#091;\n    {\"base\":\"10.13.0.0\/16\",\"size\":24}\n  ]\n}' | sudo tee \/etc\/docker\/daemon.json &gt; \/dev\/null<\/code><\/pre>\n\n\n\n<p>Hopefully, you won&#8217;t need any of that!<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_37_%E2%80%93_Configure_Docker_to_use_the_NVIDIA_Container_Toolkit\"><\/span>Step 3.7 &#8211; Configure Docker to use the NVIDIA Container Toolkit<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>In step 1, we installed the NVIDIA Container Toolkit. Now, we need to configure Docker to use it.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo nvidia-ctk runtime configure --runtime=docker<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_38_Fix_Docker_default_shm_size_and_other_runtime_configuration_env_parameters\"><\/span>Step 3.8 Fix Docker default shm size and other runtime configuration env parameters<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>The shared memory size for docker is too small for containers doing heavy computation. We will fix this by editing the \/etc\/docker\/daemon.json file. If the file does not exist yet, create it. We will also add a few other heavy-load performance environment changes. See my 2018 post for details (it&#8217;s still relevant.) <a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/how-to-setup-nvidia-docker-and-ngc-registry-on-your-workstation-part-5-docker-performance-and-resource-tuning-1119\/\">How-To Setup NVIDIA Docker and NGC Registry on your Workstation \u2013 Part 5 Docker Performance and Resource Tuning<\/a><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>echo '{\n    \"default-shm-size\": \"8G\",\n    \"default-ulimits\": {\n         \"memlock\": { \"name\":\"memlock\", \"soft\":  -1, \"hard\": -1 },\n         \"stack\"  : { \"name\":\"stack\", \"soft\": 67108864, \"hard\": 67108864 }\n    }\n}' | sudo tee -a \/etc\/docker\/daemon.json &gt; \/dev\/null<\/code><\/pre>\n\n\n\n<p><strong>WARNING! The echo command above adds an extra {} and leaves out a comma in the daemon.json file since an earlier step has already written to the file. Please edit the file by hand! It should end up looking like,<\/strong><\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n    \"runtimes\": {\n        \"nvidia\": {\n            \"args\": &#091;],\n            \"path\": \"nvidia-container-runtime\"\n        }\n    },\n    \"default-shm-size\": \"8G\",\n    \"default-ulimits\": {\n         \"memlock\": { \"name\":\"memlock\", \"soft\":  -1, \"hard\": -1 },\n         \"stack\"  : { \"name\":\"stack\", \"soft\": 67108864, \"hard\": 67108864 }\n    }\n}\n<\/code><\/pre>\n\n\n\n<h4 class=\"wp-block-heading\">Restart Docker<\/h4>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo systemctl restart docker<\/code><\/pre>\n\n\n\n<p>You should now have a Docker install with good support for demanding GPU-accelerated applications. If you want a helpful container tool that is much lighter weight than Docker, then check out Enroot next.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Step_4_%E2%80%93_Install_NVIDIA_Enroot_optional_but_recommended\"><\/span>Step 4 &#8211; Install NVIDIA Enroot (optional but recommended)<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>My favorite container runtime is <a href=\"https:\/\/github.com\/NVIDIA\/enroot\">NVIDIA Enroot<\/a> and I use it whenever I can. I will cover its installation in this next section. You can consider this optional but I do recommend that you give Enroot a try if you want a simple container runtime tool that can be surprisingly useful. See the reference links near the top of this post for some usage examples. I will include some Enroot usage examples in later parts of this series of blog posts.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\">Take a look at the Enroot GitHub pages.<\/h4>\n\n\n\n<p><a href=\"https:\/\/github.com\/NVIDIA\/enroot\/\">https:\/\/github.com\/NVIDIA\/enroot\/<\/a><\/p>\n\n\n\n<p>I will do this section as an install script section for Enroot. You will want to have the NVIDIA driver installed and I recommend having the NVIDIA container toolkit installed.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Install dependencies and useful tools\nsudo apt-get update\nsudo apt-get install --yes curl gawk jq squashfs-tools parallel\nsudo apt-get install --yes fuse-overlayfs pigz squashfuse\n\n# Get the latest release tag from the GitHub API and remove the 'v'\nRELEASE=$(curl -s https:\/\/api.github.com\/repos\/NVIDIA\/enroot\/releases\/latest | grep -oP '\"tag_name\": \"\\K(.*)(?=\")' | tr -d 'v')\n\n# Print the release tag\necho \"Installing enroot release: $RELEASE\"\n\n\n## Install\narch=$(dpkg --print-architecture)\n# arch is amd64\ncurl -fSsL -O https:\/\/github.com\/NVIDIA\/enroot\/releases\/download\/v${RELEASE}\/enroot_${RELEASE}-1_${arch}.deb\ncurl -fSsL -O https:\/\/github.com\/NVIDIA\/enroot\/releases\/download\/v${RELEASE}\/enroot+caps_${RELEASE}-1_${arch}.deb \n\nsudo apt-get install --yes .\/enroot_${RELEASE}-1_${arch}.deb\nsudo apt-get install --yes .\/enroot+caps_${RELEASE}-1_${arch}.deb\n\n# Clean up\nrm .\/enroot_${RELEASE}-1_${arch}.deb\nrm .\/enroot+caps_${RELEASE}-1_${arch}.deb<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\"><span class=\"ez-toc-section\" id=\"Next_Steps\"><\/span>Next Steps<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>With <a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/\">Part 1<\/a> and Part 2 of this post series implemented, you will have a good base server platform suitable for AI development and deployment of a robust AI server platform.<\/p>\n\n\n\n<p>The next posts in this series will be LLM server and model deployments. This will include,<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><a href=\"https:\/\/github.com\/huggingface\/text-generation-inference\">Hugging Face Text Generation Inference (TGI) server<\/a><\/li>\n\n\n\n<li><a href=\"https:\/\/github.com\/huggingface\/chat-ui\">Hugging Face Chat-UI<\/a><\/li>\n\n\n\n<li>Other server and user interfaces, such as <a href=\"https:\/\/ollama.ai\/\">Ollama<\/a>, NVIDIA <a href=\"https:\/\/www.nvidia.com\/en-us\/ai-data-science\/products\/triton-inference-server\/\">triton<\/a> server, <a href=\"https:\/\/github.com\/NVIDIA\/TensorRT\">tensorRT<\/a> optimizations, <a href=\"https:\/\/github.com\/NVIDIA\/NeMo\">NeMo<\/a>, etc.<\/li>\n\n\n\n<li>Example model deployments, Llama-2, CodeLama, etc.<\/li>\n<\/ul>\n\n\n\n<p>\u2026 more to come \u2026<\/p>\n\n\n\n<p><strong>Happy computing &#8211;dbk @dbkinghorn<\/strong><\/p>\n\n\n\n<div class=\"wp-block-columns border rounded p-2 is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<h3 class=\"wp-block-heading has-text-align-center\">Related Content<\/h3>\n\n\n \n<div class=\"related-content\">\n\t<ul class=\"related-content-list\">\n\t\t\t\t\t\t<li class=\"related-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/\" title=\"LLM Server Setup Part 2 &#8212; Container Tools\">LLM Server Setup Part 2 &#8212; Container Tools<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<li class=\"related-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/\" title=\"LLM Server Setup Part 1 &#8211; Base OS\">LLM Server Setup Part 1 &#8211; Base OS<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<li class=\"related-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/note-how-to-setup-apache-on-ubuntu-22-04-for-user-public_html\/\" title=\"Note: How To Setup Apache on Ubuntu 22.04 For User public_html\">Note: How To Setup Apache on Ubuntu 22.04 For User public_html<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<li class=\"related-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/how-to-use-linux-kernel-boot-options\/\" title=\"How To Use Linux Kernel Boot Options\">How To Use Linux Kernel Boot Options<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t<\/ul>\n\t \n\t<a class=\"view-term-link\" href=\"\/all_articles?filter=how-to-guide\">View\n\t\tAll Related Content<\/a>\n\t<\/div><\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<h3 class=\"wp-block-heading has-text-align-center\">Latest Content<\/h3>\n\n\n \n<div class=\"latest-content\">\n\t<ul class=\"latest-content-list\">\n\t\t\t\t\t\t<li class=\"latest-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/articles\/2025-professional-gpu-engineering-roundup\/\" title=\"2025 Professional GPU Engineering Roundup\">2025 Professional GPU Engineering Roundup<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<li class=\"latest-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/articles\/2025-professional-gpu-content-creation-roundup\/\" title=\"2025 Professional GPU Content Creation Roundup\">2025 Professional GPU Content Creation Roundup<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<li class=\"latest-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/articles\/a-quick-look-at-rendering-performance-in-windows-vs-linux\/\" title=\"A Quick Look at Rendering Performance in Windows vs Linux\">A Quick Look at Rendering Performance in Windows vs Linux<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<li class=\"latest-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/standing-up-ai-development-quickly-for-supercomputing-2025\/\" title=\"Standing Up AI Development Quickly for Supercomputing 2025\">Standing Up AI Development Quickly for Supercomputing 2025<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t<\/ul>\n\t \n\t\t<a href=\"\/all_posts\" class=\"view-posts-link\">View All<\/a>\n\t<\/div><\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>This post is Part 2 in a series on how to configure a system for LLM deployments and development usage.  Part 2 is about installing and configuring container tools, Docker and NVIDIA Enroot.<\/p>\n","protected":false},"author":145,"featured_media":20696,"menu_order":0,"comment_status":"open","ping_status":"closed","template":"","meta":{"_acf_changed":false,"content-type":"","classic-editor-remember":"","legacy_id":"","redirect_url":[],"expire_date":"","alert_message":"","alert_link":[],"configure_ids":"","system_grid_title":"","system_grid_ids":"","footnotes":""},"hpc_categories":[8881],"hpc_tags":[8762,9093,8770,8940],"coauthors":[9057],"class_list":["post-21032","hpc_post","type-hpc_post","status-publish","has-post-thumbnail","hentry","hpc_category-how-to-guide","hpc_tag-linux","hpc_tag-llms","hpc_tag-ml-ai","hpc_tag-sys-admin"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.7 (Yoast SEO v26.7) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>LLM Server Setup Part 2 -- Container Tools | Puget Systems<\/title>\n<meta name=\"description\" content=\"This post is Part 2 in a series on how to configure a system for LLM deployments and development usage. Part 2 is about installing and configuring container tools, Docker and NVIDIA Enroot.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"LLM Server Setup Part 2 -- Container Tools\" \/>\n<meta property=\"og:description\" content=\"This post is Part 2 in a series on how to configure a system for LLM deployments and development usage. Part 2 is about installing and configuring container tools, Docker and NVIDIA Enroot.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/\" \/>\n<meta property=\"og:site_name\" content=\"Puget Systems\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/PugetSystems\" \/>\n<meta property=\"article:modified_time\" content=\"2024-02-05T21:06:59+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.pugetsystems.com\/wp-content\/uploads\/2023\/11\/post-image.png\" \/>\n\t<meta property=\"og:image:width\" content=\"768\" \/>\n\t<meta property=\"og:image:height\" content=\"768\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@PugetSystems\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"7 minutes\" \/>\n\t<meta name=\"twitter:label2\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data2\" content=\"Dr. Donald Kinghorn\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/\",\"url\":\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/\",\"name\":\"LLM Server Setup Part 2 -- Container Tools | Puget Systems\",\"isPartOf\":{\"@id\":\"https:\/\/www.pugetsystems.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/wp-cdn.pugetsystems.com\/2023\/11\/post-image.png\",\"datePublished\":\"2023-11-20T22:15:10+00:00\",\"dateModified\":\"2024-02-05T21:06:59+00:00\",\"description\":\"This post is Part 2 in a series on how to configure a system for LLM deployments and development usage. Part 2 is about installing and configuring container tools, Docker and NVIDIA Enroot.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#primaryimage\",\"url\":\"https:\/\/wp-cdn.pugetsystems.com\/2023\/11\/post-image.png\",\"contentUrl\":\"https:\/\/wp-cdn.pugetsystems.com\/2023\/11\/post-image.png\",\"width\":768,\"height\":768},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.pugetsystems.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"HPC Posts\",\"item\":\"https:\/\/www.pugetsystems.com\/all-hpc\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"LLM Server Setup Part 2 &#8212; Container Tools\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.pugetsystems.com\/#website\",\"url\":\"https:\/\/www.pugetsystems.com\/\",\"name\":\"Puget Systems\",\"description\":\"Workstations for creators.\",\"publisher\":{\"@id\":\"https:\/\/www.pugetsystems.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.pugetsystems.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.pugetsystems.com\/#organization\",\"name\":\"Puget Systems\",\"url\":\"https:\/\/www.pugetsystems.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.pugetsystems.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.pugetsystems.com\/wp-content\/uploads\/2022\/08\/Puget-Systems-2020-logo-color-full.png\",\"contentUrl\":\"https:\/\/www.pugetsystems.com\/wp-content\/uploads\/2022\/08\/Puget-Systems-2020-logo-color-full.png\",\"width\":2560,\"height\":363,\"caption\":\"Puget Systems\"},\"image\":{\"@id\":\"https:\/\/www.pugetsystems.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/PugetSystems\",\"https:\/\/x.com\/PugetSystems\",\"https:\/\/www.instagram.com\/pugetsystems\/\",\"https:\/\/www.linkedin.com\/company\/puget-systems\",\"https:\/\/www.youtube.com\/user\/pugetsys\",\"https:\/\/en.wikipedia.org\/wiki\/Puget_Systems\"],\"telephone\":\"(425) 458-0273\",\"legalName\":\"Puget Sound Systems, Inc.\",\"foundingDate\":\"2000-12-01\",\"duns\":\"128267585\",\"naics\":\"334111\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"LLM Server Setup Part 2 -- Container Tools | Puget Systems","description":"This post is Part 2 in a series on how to configure a system for LLM deployments and development usage. Part 2 is about installing and configuring container tools, Docker and NVIDIA Enroot.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/","og_locale":"en_US","og_type":"article","og_title":"LLM Server Setup Part 2 -- Container Tools","og_description":"This post is Part 2 in a series on how to configure a system for LLM deployments and development usage. Part 2 is about installing and configuring container tools, Docker and NVIDIA Enroot.","og_url":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/","og_site_name":"Puget Systems","article_publisher":"https:\/\/www.facebook.com\/PugetSystems","article_modified_time":"2024-02-05T21:06:59+00:00","og_image":[{"width":768,"height":768,"url":"https:\/\/www.pugetsystems.com\/wp-content\/uploads\/2023\/11\/post-image.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_site":"@PugetSystems","twitter_misc":{"Est. reading time":"7 minutes","Written by":"Dr. Donald Kinghorn"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/","url":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/","name":"LLM Server Setup Part 2 -- Container Tools | Puget Systems","isPartOf":{"@id":"https:\/\/www.pugetsystems.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#primaryimage"},"image":{"@id":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#primaryimage"},"thumbnailUrl":"https:\/\/wp-cdn.pugetsystems.com\/2023\/11\/post-image.png","datePublished":"2023-11-20T22:15:10+00:00","dateModified":"2024-02-05T21:06:59+00:00","description":"This post is Part 2 in a series on how to configure a system for LLM deployments and development usage. Part 2 is about installing and configuring container tools, Docker and NVIDIA Enroot.","breadcrumb":{"@id":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#primaryimage","url":"https:\/\/wp-cdn.pugetsystems.com\/2023\/11\/post-image.png","contentUrl":"https:\/\/wp-cdn.pugetsystems.com\/2023\/11\/post-image.png","width":768,"height":768},{"@type":"BreadcrumbList","@id":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.pugetsystems.com\/"},{"@type":"ListItem","position":2,"name":"HPC Posts","item":"https:\/\/www.pugetsystems.com\/all-hpc\/"},{"@type":"ListItem","position":3,"name":"LLM Server Setup Part 2 &#8212; Container Tools"}]},{"@type":"WebSite","@id":"https:\/\/www.pugetsystems.com\/#website","url":"https:\/\/www.pugetsystems.com\/","name":"Puget Systems","description":"Workstations for creators.","publisher":{"@id":"https:\/\/www.pugetsystems.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.pugetsystems.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.pugetsystems.com\/#organization","name":"Puget Systems","url":"https:\/\/www.pugetsystems.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.pugetsystems.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.pugetsystems.com\/wp-content\/uploads\/2022\/08\/Puget-Systems-2020-logo-color-full.png","contentUrl":"https:\/\/www.pugetsystems.com\/wp-content\/uploads\/2022\/08\/Puget-Systems-2020-logo-color-full.png","width":2560,"height":363,"caption":"Puget Systems"},"image":{"@id":"https:\/\/www.pugetsystems.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/PugetSystems","https:\/\/x.com\/PugetSystems","https:\/\/www.instagram.com\/pugetsystems\/","https:\/\/www.linkedin.com\/company\/puget-systems","https:\/\/www.youtube.com\/user\/pugetsys","https:\/\/en.wikipedia.org\/wiki\/Puget_Systems"],"telephone":"(425) 458-0273","legalName":"Puget Sound Systems, Inc.","foundingDate":"2000-12-01","duns":"128267585","naics":"334111"}]}},"_links":{"self":[{"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/hpc_posts\/21032","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/hpc_posts"}],"about":[{"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/types\/hpc_post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/users\/145"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/comments?post=21032"}],"version-history":[{"count":0,"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/hpc_posts\/21032\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/media\/20696"}],"wp:attachment":[{"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/media?parent=21032"}],"wp:term":[{"taxonomy":"hpc_category","embeddable":true,"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/hpc_categories?post=21032"},{"taxonomy":"hpc_tag","embeddable":true,"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/hpc_tags?post=21032"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/coauthors?post=21032"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}