{"id":20669,"date":"2023-11-15T15:42:42","date_gmt":"2023-11-15T23:42:42","guid":{"rendered":"https:\/\/www.pugetsystems.com\/?post_type=hpc_post&#038;p=20669"},"modified":"2023-11-16T15:44:31","modified_gmt":"2023-11-16T23:44:31","slug":"llm-server-setup-part-1-base-os","status":"publish","type":"hpc_post","link":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/","title":{"rendered":"LLM Server Setup Part 1 &#8211; Base OS"},"content":{"rendered":"<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_80 counter-hierarchy ez-toc-counter ez-toc-transparent ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of Contents<\/p>\n<span class=\"ez-toc-title-toggle\"><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1 ' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#Introduction\" >Introduction<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#Hardware_requirementsrecommendations\" >Hardware requirements\/recommendations<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#Base_Ubuntu_Server_Install_and_Config\" >Base Ubuntu Server Install and Config<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-4\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#Step_1_%E2%80%93_Install_Ubuntu_Server\" >Step 1 &#8211; Install Ubuntu Server<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-5\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#Step_11_%E2%80%93_Fix_Ubuntu_Server_Annoyances_optional\" >Step 1.1 &#8211; Fix Ubuntu Server Annoyances (optional?)<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-6\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#Step_2_%E2%80%93_Install_NVIDIA_drivers\" >Step 2 &#8211; Install NVIDIA drivers<\/a><\/li><\/ul><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-7\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#Next_Steps\" >Next Steps<\/a><\/li><\/ul><\/nav><\/div>\n\n<h2 class=\"wp-block-heading\" id=\"h-introduction\"><span class=\"ez-toc-section\" id=\"Introduction\"><\/span>Introduction<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p><strong>This post is Part 1 in a series on how to configure a system for LLM deployments and development usage.<\/strong> The configuration will be suitable for multi-user deployments and also useful for smaller development systems. I will describe configuration steps in detail with commands suitable for scripts to automate the process. <strong>Part 1 is about the base Linux server setup.<\/strong><\/p>\n\n\n\n<p>I have recently had the pleasure of diving into the world of modern AI with Large Language Models. Now is truly a history-changing time in computer-human interaction. The pace of development is almost frantic. It&#8217;s been challenging staying informed of new developments and trying new ideas.<br>Fortunately, I have had on-prem access to capable hardware for development and experimentation. I&#8217;ve been mostly working with the nicely designed <a href=\"https:\/\/www.pugetsystems.com\/solutions\/scientific-computing-workstations\/machine-learning-ai\/buy-404\/\">Puget Systems Qudad NVIDIA 6000Ada 5U system<\/a>.<\/p>\n\n\n\n<p><strong>Of course, working with bare-metal hardware requires a system setup!<\/strong> Linux system configuration is something I enjoy doing. The system I&#8217;m using has proven capable of supporting many simultaneous users with state-of-the-art LLMs here at Puget Systems. This is in large part due to the great LLM server software from <a href=\"https:\/\/huggingface.co\/\">Hugging Face<\/a>.<\/p>\n\n\n\n<p>There are lots of details! To keep things manageable (for me and you), I will break this into several posts. This post will cover the base system setup. At the end of the post series, I should be able to provide a script to automate the entire process.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-hardware-requirements-recommendations\"><span class=\"ez-toc-section\" id=\"Hardware_requirementsrecommendations\"><\/span>Hardware requirements\/recommendations<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>The main system I&#8217;ve been working with is configured as follows:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>CPU: Intel Xeon w9-3475X 36-core Sapphire Rapids<\/li>\n\n\n\n<li>RAM: 512GB DDR5 4800MHz Reg ECC<\/li>\n\n\n\n<li>Motherboard: ASUS PRO WS W790E-SAGE SE<\/li>\n\n\n\n<li>2 x Sabrent Rocket 4 Plus 2TB PCIe Gen 4 M.2 SSD<\/li>\n\n\n\n<li>GPUs: 4 x NVIDIA RTX 6000 Ada Generation 48GB<\/li>\n\n\n\n<li>Ubuntu 22.04 LTS<\/li>\n<\/ul>\n\n\n\n<p>This system is capable of serving Llama2-70b and derivatives with good performance. I recommend a configuration like this for small-scale on-prem deployment. With 20-50 end users for inference or a small development work group working on applications and fine-tuning with smaller models.<\/p>\n\n\n\n<p>However, <strong>the instructions in this post (and its follow-ups) are suitable for use with much more modest hardware!<\/strong> Almost any system with one or more NVIDIA GPUs with 16GB or more of VRAM will get you started.<\/p>\n\n\n\n<p>Note: at this point, I am only recommending NVIDIA GPU acceleration. But, soon AMD GPUs will be supported enough to be useful. I have already started experimenting with them. CPU-only deployment is also possible, and I will investigate the feasibility of that in the future.<\/p>\n\n\n\n<p>For hardware configurations see <a href=\"https:\/\/www.pugetsystems.com\/solutions\/scientific-computing-workstations\/machine-learning-ai\/\">Puget Systems &#8220;Workstations for Machine Learning \/ AI&#8221;<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-base-ubuntu-server-install-and-config\"><span class=\"ez-toc-section\" id=\"Base_Ubuntu_Server_Install_and_Config\"><\/span>Base Ubuntu Server Install and Config<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>I will describe a server config. However, you can do this setup for a personal AI dev system too, possibly using Ubuntu Desktop instead of server. The tips and fixes below should be helpful for Desktops, too.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-step-1-install-ubuntu-server\"><span class=\"ez-toc-section\" id=\"Step_1_%E2%80%93_Install_Ubuntu_Server\"><\/span>Step 1 &#8211; Install Ubuntu Server<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>Get a fresh copy of Ubuntu Server Live Installer for &#8220;Jammy&#8221; (22.04).<br><a href=\"https:\/\/cdimage.ubuntu.com\/ubuntu-server\/jammy\/daily-live\/current\/\">https:\/\/cdimage.ubuntu.com\/ubuntu-server\/jammy\/daily-live\/current\/<\/a><\/p>\n\n\n\n<p>Burn that to a USB and do a standard install. You will probably want to enable ssh. Don&#8217;t try to add &#8220;extra drivers&#8221; at this point. I would also skip installing any snaps for things like docker since I will cover that install in the next post. Just do a simple install following the install prompts.<\/p>\n\n\n\n<p>After the first boot, update and upgrade the system. It should already be mostly up to date if you grabbed a fresh install ISO.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt update\nsudo apt upgrade<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-step-1-1-fix-ubuntu-server-annoyances-optional\"><span class=\"ez-toc-section\" id=\"Step_11_%E2%80%93_Fix_Ubuntu_Server_Annoyances_optional\"><\/span>Step 1.1 &#8211; Fix Ubuntu Server Annoyances (optional?)<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-annoyance-1-a-start-job-is-running-for-wait-for-network-to-be-configured\">Annoyance 1 &#8211; &#8220;A start job is running for wait for network to be configured&#8221;<\/h4>\n\n\n\n<p><strong>How to prevent unused network interfaces from waiting for DHCP on boot.<\/strong><\/p>\n\n\n\n<p>Ubuntu Server by default uses <code>networkd<\/code> with <code>netplan<\/code> to configure networking. <a href=\"https:\/\/gist.github.com\/dbkinghorn\/ed923bbcb7ec3f53bd2da5fe5e9b49b2\">I will often configure NetworkManager as the renderer<\/a> however in this guide I will stay with networkd but change the config to prevent the annoyance of unused interfaces waiting for DHCP on boot.<\/p>\n\n\n\n<p>You will need to edit the <code>netplan<\/code> config file in <code>\/etc\/netplan<\/code>.<br>First create a backup of the original install config file.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo cp \/etc\/netplan\/00-installer-config.yaml \/etc\/netplan\/00-installer-config.yaml.bak<\/code><\/pre>\n\n\n\n<p>Then, edit the file with your favorite editor. For example, the system I&#8217;m using while writing this post has two network interfaces, and only one is being used. This was configured by the installer and looks like this: (yours will probably be different.)<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># This is the network config written by 'subiquity'\nnetwork:\n  ethernets:\n    enp5s0:\n      dhcp4: true\n    enp6s0:\n      dhcp4: true\n  version: 2<\/code><\/pre>\n\n\n\n<p>Using <code>ip addr<\/code> I see that enp6s0 is an interface I&#8217;m not using on this system. On boot, networkd will wait (over 2min!) for this interface to get an address from DHCP even though it is not connected, i.e., has NO-CARRIER. To fix this, we will make the interface &#8220;optional&#8221; by adding the <code>optional: true<\/code> line to the config. The config will now look like this:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>network:\n  ethernets:\n    enp5s0:\n      dhcp4: true\n    enp6s0:\n      dhcp4: true\n      optional: true\n  version: 2<\/code><\/pre>\n\n\n\n<p>Now apply the config with: <code>sudo netplan apply<\/code>.<br>On the next boot, there will be no more waiting for unused interfaces to get an address from DHCP! Yay!<\/p>\n\n\n\n<p><strong>You (or ask your network administrator) should assign a &#8220;static lease&#8221; from your DHCP server to the interface you want to run<\/strong> your services on. You can get your current assigned IP address and interface MAC address with: ip addr. Alternatively, you can arrange to use a static IP address and configure <code>netplan<\/code> accordingly. <a href=\"https:\/\/netplan.readthedocs.io\/en\/stable\/\">See the docs<\/a>.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-annoyance-2-grub-menu-timeout-is-set-to-0-seconds-optional\">Annoyance 2 &#8211; Grub menu timeout is set to 0 seconds (optional)<\/h4>\n\n\n\n<p><strong>How to get the grub boot menu to show during boot in Ubuntu.<\/strong><\/p>\n\n\n\n<p>In the file \/etc\/default\/grub change the GRUB_TIMEOUT value to 6 (the number of seconds you want it to wait for a key press) and change &#8220;hidden&#8221; to &#8220;menu&#8221;. Then run update-grub. Do this manually or with the following scriptable commands:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo sed -i 's\/GRUB_TIMEOUT=0\/GRUB_TIMEOUT=6\/g' \/etc\/default\/grub\nsudo sed -i 's\/GRUB_TIMEOUT_STYLE=hidden\/GRUB_TIMEOUT_STYLE=menu\/g' \/etc\/default\/grub\nsudo update-grub<\/code><\/pre>\n\n\n\n<p>Now, you will see the grub menu for 6 seconds and have a chance to select a different kernel or set kernel boot options to fix problems if needed! (You can use [esc] during the start of boot to get to grub menu, too, if you don&#8217;t want to make this change. I prefer to see the menu by default.)<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"h-annoyance-3-stop-unscheduled-automatic-updates-from-regularly-breaking-your-server-setup\">Annoyance 3 &#8211; Stop unscheduled automatic updates from regularly breaking your server setup.<\/h4>\n\n\n\n<p><strong>How to disable &#8220;some&#8221; Ubuntu server automatic updates<\/strong><\/p>\n\n\n\n<p>On a server OS deployment, you want to keep packages current with security updates. However, you should be doing this on your schedule, not letting some default config do it without your knowledge or control. When important packages like the kernel, driver modules, and packages from 3rd party repositories are updated, there is a chance your system will break and require a reboot or more to get things working again. That can make users and administrators very unhappy! Ubuntu server will happily update and break your system automatically at irregular intervals if you let it. Let&#8217;s change that for automatic kernel and NVIDIA driver updates.<\/p>\n\n\n\n<p><strong>Think carefully about this since you will need to handle security updates for the packages you exclude from auto updating.<\/strong><\/p>\n\n\n\n<p>You will need to edit <code>\/etc\/apt\/apt.conf.d\/50unattended-upgrades<\/code>.<\/p>\n\n\n\n<p>This a config file with many options. We add a couple of simple excludes to the <code>Unattended-Upgrade::Package-Blacklist<\/code> section. The section starts as,<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/\/ Python regular expressions, matching packages to exclude from upgrading\nUnattended-Upgrade::Package-Blacklist {\n    \/\/ The following matches all packages starting with linux-\n\/\/  \"linux-\";\n...\n...\n};<\/code><\/pre>\n\n\n\n<p>Excluding Linux kernel updates and NVIDIA driver updates will probably be all you need to keep your system from breaking from automatic updates. We will exclude all packages that start with <code>linux-<\/code> and <code>nvidia-<\/code><\/p>\n\n\n\n<p>To see what those pagkages are you can do,<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>apt list --installed | grep '^linux-'\napt list --installed | grep '^nvidia-'<\/code><\/pre>\n\n\n\n<p>The following script lines should do it.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo cp \/etc\/apt\/apt.conf.d\/50unattended-upgrades \/etc\/apt\/apt.conf.d\/50unattended-upgrades.bak\n\nsudo sed -i 's|\/\/  \"linux-\";|\"  linux-\";\\n  \"nvidia-\";|' \/etc\/apt\/apt.conf.d\/50unattended-upgrades<\/code><\/pre>\n\n\n\n<p>With those applied that section of the config file will look like this,<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/\/ Python regular expressions, matching packages to exclude from upgrading\nUnattended-Upgrade::Package-Blacklist {\n    \/\/ The following matches all packages starting with linux-\n  \"linux-\";\n  \"nvidia-\";\n...\n...\n};<\/code><\/pre>\n\n\n\n<p>The next time the <code>unattended-upgrades<\/code> service is run, those changes will be read.<\/p>\n\n\n\n<p>Those are the main annoyances I would feel remiss if I didn&#8217;t include fixes for in this post.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\" id=\"h-step-2-install-nvidia-drivers\"><span class=\"ez-toc-section\" id=\"Step_2_%E2%80%93_Install_NVIDIA_drivers\"><\/span>Step 2 &#8211; Install NVIDIA drivers<span class=\"ez-toc-section-end\"><\/span><\/h3>\n\n\n\n<p>I recommend installing the latest NVIDIA driver from the standard Ubuntu repos. This will hopefully be stable and working correctly.<\/p>\n\n\n\n<p>You can check the driver versions available with:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>apt search nvidia-driver-* | grep ^nvidia-driver-<\/code><\/pre>\n\n\n\n<p>You will want the highest-numbered version. Don&#8217;t worry about &#8220;server&#8221; or &#8220;open&#8221;. <code>--no-install-recommends<\/code> will do the right thing.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo apt install --no-install-recommends nvidia-driver-535<\/code><\/pre>\n\n\n\n<p>Restart to load the new driver.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>sudo shutdown -r now<\/code><\/pre>\n\n\n\n<p>After restarting, check that the driver is loaded with:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>nvidia-smi<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"h-next-steps\"><span class=\"ez-toc-section\" id=\"Next_Steps\"><\/span>Next Steps<span class=\"ez-toc-section-end\"><\/span><\/h2>\n\n\n\n<p>I&#8217;m going to stop this Part 1 post here. The configuration steps above should provide a stable base server platform suitable for the next configuration steps toward a robust AI server platform.<\/p>\n\n\n\n<p>The next post will cover container runtime setups. This will include,<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>The ubiquitous Docker<\/li>\n\n\n\n<li>My favorite container tool, NVIDIA Enroot<\/li>\n\n\n\n<li>And possibly Podman, which is, in nearly all cases, a good alternative to Docker. And it can be used in user space without elevated privileges.<\/li>\n<\/ul>\n\n\n\n<p>After the container runtime configuration post, we will move on to <a href=\"https:\/\/github.com\/huggingface\/text-generation-inference\">Hugging Face TGI server<\/a> and <a href=\"https:\/\/github.com\/huggingface\/chat-ui\">Chat-ui interface.<\/a><\/p>\n\n\n\n<p>\u2026 to be continued \u2026<\/p>\n\n\n\n<p><strong>Happy computing &#8211;dbk @dbkinghorn<\/strong><\/p>\n\n\n\n<div class=\"wp-block-columns border rounded p-2 is-layout-flex wp-container-core-columns-is-layout-9d6595d7 wp-block-columns-is-layout-flex\">\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<h3 class=\"wp-block-heading has-text-align-center\">Related Content<\/h3>\n\n\n \n<div class=\"related-content\">\n\t<ul class=\"related-content-list\">\n\t\t\t\t\t\t<li class=\"related-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-2-container-tools\/\" title=\"LLM Server Setup Part 2 &#8212; Container Tools\">LLM Server Setup Part 2 &#8212; Container Tools<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<li class=\"related-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/\" title=\"LLM Server Setup Part 1 &#8211; Base OS\">LLM Server Setup Part 1 &#8211; Base OS<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<li class=\"related-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/note-how-to-setup-apache-on-ubuntu-22-04-for-user-public_html\/\" title=\"Note: How To Setup Apache on Ubuntu 22.04 For User public_html\">Note: How To Setup Apache on Ubuntu 22.04 For User public_html<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<li class=\"related-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/how-to-use-linux-kernel-boot-options\/\" title=\"How To Use Linux Kernel Boot Options\">How To Use Linux Kernel Boot Options<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t<\/ul>\n\t \n\t<a class=\"view-term-link\" href=\"\/all_articles?filter=how-to-guide\">View\n\t\tAll Related Content<\/a>\n\t<\/div><\/div>\n\n\n\n<div class=\"wp-block-column is-layout-flow wp-block-column-is-layout-flow\">\n<h3 class=\"wp-block-heading has-text-align-center\">Latest Content<\/h3>\n\n\n \n<div class=\"latest-content\">\n\t<ul class=\"latest-content-list\">\n\t\t\t\t\t\t<li class=\"latest-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/articles\/2025-professional-gpu-engineering-roundup\/\" title=\"2025 Professional GPU Engineering Roundup\">2025 Professional GPU Engineering Roundup<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<li class=\"latest-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/articles\/2025-professional-gpu-content-creation-roundup\/\" title=\"2025 Professional GPU Content Creation Roundup\">2025 Professional GPU Content Creation Roundup<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<li class=\"latest-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/articles\/a-quick-look-at-rendering-performance-in-windows-vs-linux\/\" title=\"A Quick Look at Rendering Performance in Windows vs Linux\">A Quick Look at Rendering Performance in Windows vs Linux<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t\t\t\t<li class=\"latest-content-list-item\">\n\t\t\t\t\t<a href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/standing-up-ai-development-quickly-for-supercomputing-2025\/\" title=\"Standing Up AI Development Quickly for Supercomputing 2025\">Standing Up AI Development Quickly for Supercomputing 2025<\/a>\n\t\t\t\t<\/li>\n\t\t\t\t<\/ul>\n\t \n\t\t<a href=\"\/all_posts\" class=\"view-posts-link\">View All<\/a>\n\t<\/div><\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>This post is Part 1 in a series on how to configure a system for LLM deployments and development usage. The configuration will be suitable for multi-user deployments and also useful for smaller development systems. Part 1 is about the base Linux server setup.<\/p>\n","protected":false},"author":145,"featured_media":20696,"menu_order":0,"comment_status":"open","ping_status":"closed","template":"","meta":{"_acf_changed":false,"content-type":"","classic-editor-remember":"","legacy_id":"","redirect_url":[],"expire_date":"","alert_message":"","alert_link":[],"configure_ids":"","system_grid_title":"","system_grid_ids":"","footnotes":""},"hpc_categories":[8881],"hpc_tags":[8762,9093,8770],"coauthors":[9057],"class_list":["post-20669","hpc_post","type-hpc_post","status-publish","has-post-thumbnail","hentry","hpc_category-how-to-guide","hpc_tag-linux","hpc_tag-llms","hpc_tag-ml-ai"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v26.7 (Yoast SEO v26.7) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>LLM Server Setup Part 1 - Base OS | Puget Systems<\/title>\n<meta name=\"description\" content=\"This post is Part 1 in a series on how to configure a system for LLM deployments and development usage. The configuration will be suitable for multi-user deployments and also useful for smaller development systems. Part 1 is about the base Linux server setup.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"LLM Server Setup Part 1 - Base OS\" \/>\n<meta property=\"og:description\" content=\"This post is Part 1 in a series on how to configure a system for LLM deployments and development usage. The configuration will be suitable for multi-user deployments and also useful for smaller development systems. Part 1 is about the base Linux server setup.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/\" \/>\n<meta property=\"og:site_name\" content=\"Puget Systems\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/PugetSystems\" \/>\n<meta property=\"article:modified_time\" content=\"2023-11-16T23:44:31+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/www.pugetsystems.com\/wp-content\/uploads\/2023\/11\/post-image.png\" \/>\n\t<meta property=\"og:image:width\" content=\"768\" \/>\n\t<meta property=\"og:image:height\" content=\"768\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@PugetSystems\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"7 minutes\" \/>\n\t<meta name=\"twitter:label2\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data2\" content=\"Dr. Donald Kinghorn\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/\",\"url\":\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/\",\"name\":\"LLM Server Setup Part 1 - Base OS | Puget Systems\",\"isPartOf\":{\"@id\":\"https:\/\/www.pugetsystems.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/wp-cdn.pugetsystems.com\/2023\/11\/post-image.png\",\"datePublished\":\"2023-11-15T23:42:42+00:00\",\"dateModified\":\"2023-11-16T23:44:31+00:00\",\"description\":\"This post is Part 1 in a series on how to configure a system for LLM deployments and development usage. The configuration will be suitable for multi-user deployments and also useful for smaller development systems. Part 1 is about the base Linux server setup.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#primaryimage\",\"url\":\"https:\/\/wp-cdn.pugetsystems.com\/2023\/11\/post-image.png\",\"contentUrl\":\"https:\/\/wp-cdn.pugetsystems.com\/2023\/11\/post-image.png\",\"width\":768,\"height\":768},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.pugetsystems.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"HPC Posts\",\"item\":\"https:\/\/www.pugetsystems.com\/all-hpc\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"LLM Server Setup Part 1 &#8211; Base OS\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.pugetsystems.com\/#website\",\"url\":\"https:\/\/www.pugetsystems.com\/\",\"name\":\"Puget Systems\",\"description\":\"Workstations for creators.\",\"publisher\":{\"@id\":\"https:\/\/www.pugetsystems.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/www.pugetsystems.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.pugetsystems.com\/#organization\",\"name\":\"Puget Systems\",\"url\":\"https:\/\/www.pugetsystems.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.pugetsystems.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.pugetsystems.com\/wp-content\/uploads\/2022\/08\/Puget-Systems-2020-logo-color-full.png\",\"contentUrl\":\"https:\/\/www.pugetsystems.com\/wp-content\/uploads\/2022\/08\/Puget-Systems-2020-logo-color-full.png\",\"width\":2560,\"height\":363,\"caption\":\"Puget Systems\"},\"image\":{\"@id\":\"https:\/\/www.pugetsystems.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/PugetSystems\",\"https:\/\/x.com\/PugetSystems\",\"https:\/\/www.instagram.com\/pugetsystems\/\",\"https:\/\/www.linkedin.com\/company\/puget-systems\",\"https:\/\/www.youtube.com\/user\/pugetsys\",\"https:\/\/en.wikipedia.org\/wiki\/Puget_Systems\"],\"telephone\":\"(425) 458-0273\",\"legalName\":\"Puget Sound Systems, Inc.\",\"foundingDate\":\"2000-12-01\",\"duns\":\"128267585\",\"naics\":\"334111\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"LLM Server Setup Part 1 - Base OS | Puget Systems","description":"This post is Part 1 in a series on how to configure a system for LLM deployments and development usage. The configuration will be suitable for multi-user deployments and also useful for smaller development systems. Part 1 is about the base Linux server setup.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/","og_locale":"en_US","og_type":"article","og_title":"LLM Server Setup Part 1 - Base OS","og_description":"This post is Part 1 in a series on how to configure a system for LLM deployments and development usage. The configuration will be suitable for multi-user deployments and also useful for smaller development systems. Part 1 is about the base Linux server setup.","og_url":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/","og_site_name":"Puget Systems","article_publisher":"https:\/\/www.facebook.com\/PugetSystems","article_modified_time":"2023-11-16T23:44:31+00:00","og_image":[{"width":768,"height":768,"url":"https:\/\/www.pugetsystems.com\/wp-content\/uploads\/2023\/11\/post-image.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_site":"@PugetSystems","twitter_misc":{"Est. reading time":"7 minutes","Written by":"Dr. Donald Kinghorn"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/","url":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/","name":"LLM Server Setup Part 1 - Base OS | Puget Systems","isPartOf":{"@id":"https:\/\/www.pugetsystems.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#primaryimage"},"image":{"@id":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#primaryimage"},"thumbnailUrl":"https:\/\/wp-cdn.pugetsystems.com\/2023\/11\/post-image.png","datePublished":"2023-11-15T23:42:42+00:00","dateModified":"2023-11-16T23:44:31+00:00","description":"This post is Part 1 in a series on how to configure a system for LLM deployments and development usage. The configuration will be suitable for multi-user deployments and also useful for smaller development systems. Part 1 is about the base Linux server setup.","breadcrumb":{"@id":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#primaryimage","url":"https:\/\/wp-cdn.pugetsystems.com\/2023\/11\/post-image.png","contentUrl":"https:\/\/wp-cdn.pugetsystems.com\/2023\/11\/post-image.png","width":768,"height":768},{"@type":"BreadcrumbList","@id":"https:\/\/www.pugetsystems.com\/labs\/hpc\/llm-server-setup-part-1-base-os\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.pugetsystems.com\/"},{"@type":"ListItem","position":2,"name":"HPC Posts","item":"https:\/\/www.pugetsystems.com\/all-hpc\/"},{"@type":"ListItem","position":3,"name":"LLM Server Setup Part 1 &#8211; Base OS"}]},{"@type":"WebSite","@id":"https:\/\/www.pugetsystems.com\/#website","url":"https:\/\/www.pugetsystems.com\/","name":"Puget Systems","description":"Workstations for creators.","publisher":{"@id":"https:\/\/www.pugetsystems.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.pugetsystems.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.pugetsystems.com\/#organization","name":"Puget Systems","url":"https:\/\/www.pugetsystems.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.pugetsystems.com\/#\/schema\/logo\/image\/","url":"https:\/\/www.pugetsystems.com\/wp-content\/uploads\/2022\/08\/Puget-Systems-2020-logo-color-full.png","contentUrl":"https:\/\/www.pugetsystems.com\/wp-content\/uploads\/2022\/08\/Puget-Systems-2020-logo-color-full.png","width":2560,"height":363,"caption":"Puget Systems"},"image":{"@id":"https:\/\/www.pugetsystems.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/PugetSystems","https:\/\/x.com\/PugetSystems","https:\/\/www.instagram.com\/pugetsystems\/","https:\/\/www.linkedin.com\/company\/puget-systems","https:\/\/www.youtube.com\/user\/pugetsys","https:\/\/en.wikipedia.org\/wiki\/Puget_Systems"],"telephone":"(425) 458-0273","legalName":"Puget Sound Systems, Inc.","foundingDate":"2000-12-01","duns":"128267585","naics":"334111"}]}},"_links":{"self":[{"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/hpc_posts\/20669","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/hpc_posts"}],"about":[{"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/types\/hpc_post"}],"author":[{"embeddable":true,"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/users\/145"}],"replies":[{"embeddable":true,"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/comments?post=20669"}],"version-history":[{"count":0,"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/hpc_posts\/20669\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/media\/20696"}],"wp:attachment":[{"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/media?parent=20669"}],"wp:term":[{"taxonomy":"hpc_category","embeddable":true,"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/hpc_categories?post=20669"},{"taxonomy":"hpc_tag","embeddable":true,"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/hpc_tags?post=20669"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.pugetsystems.com\/wp-json\/wp\/v2\/coauthors?post=20669"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}