Canonical recently announced the release of Ubuntu 24.04 LTS, codenamed “Noble Numbat”. This update underscores Ubuntu’s ongoing commitment to enhancing performance and security, focusing on optimizing developer productivity. The latest version features an optimized Linux kernel 6.8 and significant system management upgrades as detailed in the release notes. In this blog post, we highlight the key features and improvements that Ubuntu 24.04 LTS brings to the table, specifically tailored for users of Microsoft/Azure.
Unified marketplace offering
Ubuntu 24.04 LTS introduces a consolidated Azure Marketplace experience. Easily find the official Ubuntu images created by Canonical and endorsed by Microsoft for Azure, all under a single offering: ubuntu-24_04-lts. This simplification aids your search and selection process, helping you choose the right image for your needs and ensuring optimal compatibility and performance. [Explore the Ubuntu 24.04 images on the Azure Marketplace].
Optimized for Azure
Ubuntu 24.04 LTS is finely tuned to enhance performance on Azure infrastructure, ensuring that the Ubuntu images are fully compatible and support the latest cloud features as they are released. This optimization boosts system efficiency, speed, and reliability. Integration with Azure Guest Patching and the Update Management Center facilitates streamlined and continuous system updates, thereby reinforcing the overall security and stability of Ubuntu deployments.
Enhanced developer toolchains
.NET 8 is fully compatible with Ubuntu 24.04 LTS from launch, being directly available through the official Ubuntu feeds. This synchronization with the .NET release cadence ensures developers have immediate access to the latest features and updates. Additionally, .NET 8 introduces streamlined package management and new Ubuntu container images, boosting development flexibility and deployment efficiency. (Read more in this Microsoft’s blog post).
The commitment to developer productivity also extends to other popular programming languages, including TCK-certified Java versions and the latest Rust toolchains, enhancing support and smoothing the development experience.
Confidential Computing
Ubuntu continues to lead in confidential computing with support for Confidential VMs, including capabilities for confidential AI. This is facilitated by utilizing advanced hardware security extensions such as AMD’s 4th Gen EPYC processors with SEV-SNP and NVIDIA H100 Tensor Core GPUs. These features help safeguard data at runtime from system vulnerabilities and unauthorized access, making them particularly suitable for AI training and data inference involving sensitive information.
Windows Subsystem for Linux (WSL)
Ubuntu 24.04 LTS enhances its WSL integration using the same installer technology as Ubuntu Server. This update includes support for cloud-init, standardizing developer environments across installations and ensuring consistent and streamlined workflows.
Wrapping up
As we explore the capabilities of Ubuntu 24.04 LTS, Microsoft/Azure users will experience an integration that is tailored to current technological needs and equipped for upcoming developments. This version is supported for up to 12 years, providing a stable and reliable foundation that enterprises and developers can rely on for long-term projects and innovation.
One of the biggest challenges that cloud service providers (CSPs) face these days is to deliver an extension of the public cloud they host to a small-scale piece of infrastructure that runs on customers’ premises. While the world’s tech giants, such as Amazon or Azure, have developed their own solutions for this purpose, many smaller, regional CSPs rely on open source projects like OpenStack instead. However, while OpenStack is fully suitable for powering large pools of compute and storage, shrinking it down to a small box that runs on-prem, replicating the delivery process across hundreds or thousands of customers’ sites, and operating them effectively might pose an interesting challenge.
Fortunately, there are now ways to minimise OpenStack’s footprint and accelerate its delivery. In this short blog we will showcase how you can use project Sunbeam to seamlessly deploy a small-scale OpenStack cloud and plug it in as an on-prem extension of the OpenStack public cloud you host, ensuring full API compatibility and an integration with leading Cloud Platform Management (CPM) tools.
More in this series
This blog post is part of a larger series demonstrating various use cases for project Sunbeam and OpenStack. By using practical examples, we showcase how these two technologies can be used to address real-life challenges.
Before we start, let’s briefly clarify some terms that we’ll be using in this blog.
What is Sunbeam?
Sunbeam is an upstream project under the governance of the OpenInfra Foundation (OIF) created to lower the barrier to entry for OpenStack, simplify its adoption process, and set the foundation for an autonomous private cloud. Sunbeam uses cloud-native architecture and total bottom-up automation to make OpenStack more accessible to newcomers and to help users get to grips with the platform immediately.
What is MicroStack?
MicroStack (based on Sunbeam) is an OpenStack distribution designed for small-scale cloud environments. While it is available with full commercial support from Canonical, it can also be self-deployed with no friction, effectively eliminating the need for a paid consulting engagement. MicroStack currently includes core OpenStack services only but is expected to evolve quickly to ensure full feature parity with Canonical’s Charmed OpenStack soon.
OpenStack with Sunbeam as an on-prem extension of the OpenStack public cloud
Many organisations who embrace a public-cloud-only approach face a need for an on-prem infrastructure sooner rather than later. This usually stems from cost optimisation and FinOps practices, privacy concerns, or a requirement to guarantee the desired level of performance. As a result, leading public cloud providers have been already offering their own solutions in this space for years. If you are a CSP, you’ve likely heard about AWS Outposts or Azure Stack. However, finding a reasonable equivalent to those proprietary solutions in the open source space has always been a challenge.
The challenge
Most of the CSPs who offer public cloud services run OpenStack underneath them. According to the recent report by the OpenInfra Foundation, OpenStack powers more than 300 data centres that act as a regional public cloud infrastructure. This results from its unparalleled maturity, enterprise-grade stability and versatility; fourteen years after its initial release OpenStack continues to be the open source cloud platform of choice.
However, while OpenStack is fully suitable for powering those large data centres, putting it on customers’ premises might be challenging without using proper tools. This is because OpenStack is designed to deal with big clusters, rather than running on a single machine with limited hardware resources. At the same time, customers usually don’t want to over-invest: they want to start small and grow their on-prem infrastructure as they go.
Another interesting challenge is with on-going operations. OpenStack is known to be inheritably complex. Therefore, operating it in production creates significant overhead for the cloud operations team. And this is what happens when you manage just one cloud; what about if you now have to manage hundreds or thousands of clouds? The overall cost associated with running those on-prem extensions results in an ever inflating total cost of ownership (TCO).
Fortunately, dedicated tools exist to help you bypass all of these challenges.
Take a “cloud in a box” approach
With project Sunbeam CSPs can accelerate the delivery process of those on-prem extensions by taking a “cloud in a box” approach. This means shipping hardware with a super straightforward deployment procedure, or even shipping it with OpenStack pre-installed. By using cloud-native architecture underneath and running all OpenStack services inside of containers and snaps, Sunbeam effectively shrinks down the footprint of OpenStack, making it a first-class citizen on “non-beefy” machines.
The absolute minimum is just one physical node. Just note that such configuration does not ensure high availability (HA). Anyway, simply running five commands enables you to get a fully functional OpenStack up and running. This is so easy that even your non-technical customers can do it with no friction. But this is not the end! By using full automation and rich lifecycle management capabilities, Sunbeam enables the OpenStack cloud to scale out very quickly. This way your customers can always start small and grow according to their needs.
Plugging it in to your OpenStack public cloud
One of the biggest advantages of using OpenStack everywhere is that it enables you to use exactly the same cloud management software for public cloud infrastructure as well as all those extensions running on your customers’ premises. Sunbeam-based OpenStack installations rely on the pure upstream code and ensure API compatibility. Thanks to that your customers won’t see a difference when using your public cloud services or their on-prem infrastructure you provide. Exactly in the same way as Amazon customers, for example.
What’s more, you can now easily plug all those on-prem clouds to the CMP software that you use. This way you can view all the environments you manage from a single pane of glass (SPOG) dashboard. You can monitor them, check their health status, or even provision some administrative workloads centrally. This is especially useful for ongoing performance optimisation, for example, when you have to benchmark the performance of all those distributed environments.
The final challenge to address are these on-prem environments’ ongoing operations. Let’s not forget that at the end of the day those are still OpenStack clouds. As mentioned earlier, OpenStack is inheritably complex and its operations might pose a real challenge. However, project Sunbeam uses full automation: contrary to other OpenStack distributions, all typical post-deployment operations are fully automated, not just not the initial delivery phase of the project. This extensive automation also covers procedures, such as upgrades, which were historically very complicated. All of that to make OpenStack suitable for the scale we’re talking about.
Sample CPM dashboard
Conclusions
OpenStack’s architecture has always made the challenges of mass deploying it on-prem a distant bad dream. However, with project Sunbeam CSPs can finally break down these barriers. Its minimal footprint shrinks down OpenStack to a piece of software that can run inside of a single box. By ensuring full API compatibility OpenStack clouds deployed with project Sunbeam can be seamlessly plugged into the broader cloud ecosystem of the CSP. Finally, full automation used both during the initial installation and its post-deployment phase guarantees smooth delivery and frictionless operations.
Learn more about Sunbeam
Now that you’ve got a basic understanding of project Sunbeam and how it can be used for various use cases, you might be wondering where to find more information about it. If that’s the case, we have prepared some interesting follow-up materials for you:
This blog post explores the technical and strategic benefits of deploying open-source AI models on Ubuntu. We’ll highlight why it makes sense to use Ubuntu with open-source AI models, and outline the deployment process on Azure.
Authored by Gauthier Jolly, Software Engineer, CPC, and Jehudi Castro-Sierra, Public Cloud Alliance Director, both from Canonical.
Why Ubuntu for Open-Source AI?
Open Philosophy: Ubuntu’s open-source nature aligns seamlessly with the principles of open-source AI models, fostering collaboration and accessibility.
Seamless Integration: Deploying open-source AI is smooth on Ubuntu, thanks to its robust support for AI libraries and tools.
Community: Ubuntu’s large community provides valuable resources and knowledge-sharing for AI development.
The Role of Ubuntu Pro
Ubuntu Pro elevates the security and compliance aspects of deploying AI models, offering extended security maintenance, comprehensive patching, and automated compliance features that are vital for enterprise-grade applications. Its integration with Confidential VMs on Azure enhances the protection of sensitive data and model integrity, making it an indispensable tool for tasks requiring stringent security measures like ML training, inference, and confidential multi-party data analytics.
Why use the public cloud for deploying AI models?
Using a public cloud like Azure gives straightforward access to powerful GPUs and Confidential Compute capabilities, essential for intensive AI tasks. These features significantly reduce the time and complexity involved in setting up and running AI models, without compromising on security and privacy. Although some may opt for on-prem deployment due to specific requirements, Azure’s scalable and secure environment offers a compelling argument for cloud-based deployments.
Provisioning and Configuration
We are going to explore using open models on Azure by creating an instance with Ubuntu, installing NVIDIA drivers for GPU support, and setting up Ollama for running the models. The process is technical, involving CLI commands for creating the resource group, VM, and configuring NVIDIA drivers. Ollama, the chosen tool for running models like Mixtral, is best installed using Snap for a hassle-free experience, encapsulating dependencies and simplifying updates.
Provision an Azure VM
Begin by creating a resource group and then a VM with the Ubuntu image using the Azure CLI.
az group create --location westus --resource-group ml-workload
az vm create \
--resource-group ml-workload \
--name jammy \
--image Ubuntu2204 \
--generate-ssh-keys \
--size Standard_NC4as_T4_v3 \
--admin-username ubuntu --license-type UBUNTU_PRO
Note the publicIpAddress from the output – you’ll need it to SSH into the VM.
Install Nvidia Drivers (GPU Support)
For GPU capabilities, install NVIDIA drivers using Ubuntu’s package management system. Restart the system after installation.
Important: Standard NVIDIA drivers don’t support vGPUs (fractional GPUs). See instructions on the Azure site for installing GRID drivers, which might involve building an unsigned kernel module (which may be incompatible with Secure Boot).
Deploying Ollama with Snap
Snap simplifies the installation of Ollama and its dependencies, ensuring compatibility and streamlined updates. The –beta flag allows you to access the latest features and versions, which might still be under development
sudo snap install --beta ollama
Configuration
Configure Ollama to use the ephemeral disk
sudo mkdir /mnt/models
sudo snap connect ollama:removable-media # to allow the snap to reach /mnt
sudo snap set ollama models=/mnt/models
Installing Mixtral
At this point, you can run one of the open models available out of the box, like mixtral or llama2. If you have a fine-tuned version of these models (a process that involves further training on a specific dataset), you can run those as well.
ollama run mixtral
The first run might take a while to download the model.
Now you can use the model through the console interface:
Installing a UI
This step is optional, but provides a UI via your web browser.
sudo snap install --beta open-webui
Access the web UI securely
To quickly access the UI without open ports in the Azure security group, you can create an SSH tunnel to your VM using the following command:
ssh -L 8080:localhost:8080 ubuntu@${IP_ADDR}
Go to http://localhost:8080 in your web browser on your local machine (the command above tunnels the traffic from your localhost to the instance on Azure).:
In case you want to make this service public, follow this documentation.
Verify GPU usage
sudo watch -n2 nvidia-smi
Check that the ollama process is using the GPU, you should see something like this:
+---------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|===========================================================================|
| 0 N/A N/A 1063 C /snap/ollama/13/bin/ollama 4882MiB |
+---------------------------------------------------------------------------+
Complementary and Alternative Solutions
Charmed Kubeflow: Explore this solution for end-to-end MLOps (Machine Learning Operations), providing a streamlined platform to manage every stage of the machine learning lifecycle. It’s particularly well-suited for complex or large-scale AI deployments.
Azure AI Studio: Provides ease of use for those seeking less customization.
Conclusion
Ubuntu’s open-source foundation and robust ecosystem make it a compelling choice for deploying open-source AI models. When combined with Azure’s GPU capabilities and Confidential Compute features, you gain a flexible, secure, and performant AI solution.
Connexion
