Canonical recently announced the release of Ubuntu 24.04 LTS, codenamed “Noble Numbat”. This update underscores Ubuntu’s ongoing commitment to enhancing performance and security, focusing on optimizing developer productivity. The latest version features an optimized Linux kernel 6.8 and significant system management upgrades as detailed in the release notes. In this blog post, we highlight the key features and improvements that Ubuntu 24.04 LTS brings to the table, specifically tailored for users of Microsoft/Azure.

Unified marketplace offering

Ubuntu 24.04 LTS introduces a consolidated Azure Marketplace experience. Easily find the official Ubuntu images created by Canonical and endorsed by Microsoft for Azure, all under a single offering: ubuntu-24_04-lts. This simplification aids your search and selection process, helping you choose the right image for your needs and ensuring optimal compatibility and performance. [Explore the Ubuntu 24.04 images on the Azure Marketplace].

Optimized for Azure

Ubuntu 24.04 LTS is finely tuned to enhance performance on Azure infrastructure, ensuring that the Ubuntu images are fully compatible and support the latest cloud features as they are released. This optimization boosts system efficiency, speed, and reliability. Integration with Azure Guest Patching and the Update Management Center facilitates streamlined and continuous system updates, thereby reinforcing the overall security and stability of Ubuntu deployments.

Enhanced developer toolchains

.NET 8 is fully compatible with Ubuntu 24.04 LTS from launch, being directly available through the official Ubuntu feeds. This synchronization with the .NET release cadence ensures developers have immediate access to the latest features and updates. Additionally, .NET 8 introduces streamlined package management and new Ubuntu container images, boosting development flexibility and deployment efficiency. (Read more in this Microsoft’s blog post).

The commitment to developer productivity also extends to other popular programming languages, including TCK-certified Java versions and the latest Rust toolchains, enhancing support and smoothing the development experience.

Confidential Computing

Ubuntu continues to lead in confidential computing with support for Confidential VMs, including capabilities for confidential AI. This is facilitated by utilizing advanced hardware security extensions such as AMD’s 4th Gen EPYC processors with SEV-SNP and NVIDIA H100 Tensor Core GPUs. These features help safeguard data at runtime from system vulnerabilities and unauthorized access, making them particularly suitable for AI training and data inference involving sensitive information.

Windows Subsystem for Linux (WSL)

Ubuntu 24.04 LTS enhances its WSL integration using the same installer technology as Ubuntu Server. This update includes support for cloud-init, standardizing developer environments across installations and ensuring consistent and streamlined workflows.

Wrapping up

As we explore the capabilities of Ubuntu 24.04 LTS, Microsoft/Azure users will experience an integration that is tailored to current technological needs and equipped for upcoming developments. This version is supported for up to 12 years, providing a stable and reliable foundation that enterprises and developers can rely on for long-term projects and innovation.

This blog post explores the technical and strategic benefits of deploying open-source AI models on Ubuntu. We’ll highlight why it makes sense to use Ubuntu with open-source AI models, and outline the deployment process on Azure.

Authored by Gauthier Jolly, Software Engineer, CPC, and Jehudi Castro-Sierra, Public Cloud Alliance Director, both from Canonical.

Why Ubuntu for Open-Source AI?

• Open Philosophy: Ubuntu’s open-source nature aligns seamlessly with the principles of open-source AI models, fostering collaboration and accessibility.
• Seamless Integration: Deploying open-source AI is smooth on Ubuntu, thanks to its robust support for AI libraries and tools.
• Community: Ubuntu’s large community provides valuable resources and knowledge-sharing for AI development.

The Role of Ubuntu Pro

Ubuntu Pro elevates the security and compliance aspects of deploying AI models, offering extended security maintenance, comprehensive patching, and automated compliance features that are vital for enterprise-grade applications. Its integration with Confidential VMs on Azure enhances the protection of sensitive data and model integrity, making it an indispensable tool for tasks requiring stringent security measures like ML training, inference, and confidential multi-party data analytics.

Why use the public cloud for deploying AI models?

Using a public cloud like Azure gives straightforward access to powerful GPUs and Confidential Compute capabilities, essential for intensive AI tasks. These features significantly reduce the time and complexity involved in setting up and running AI models, without compromising on security and privacy. Although some may opt for on-prem deployment due to specific requirements, Azure’s scalable and secure environment offers a compelling argument for cloud-based deployments.

Provisioning and Configuration

We are going to explore using open models on Azure by creating an instance with Ubuntu, installing NVIDIA drivers for GPU support, and setting up Ollama for running the models. The process is technical, involving CLI commands for creating the resource group, VM, and configuring NVIDIA drivers. Ollama, the chosen tool for running models like Mixtral, is best installed using Snap for a hassle-free experience, encapsulating dependencies and simplifying updates.

Provision an Azure VM

Begin by creating a resource group and then a VM with the Ubuntu image using the Azure CLI.

az group create --location westus --resource-group ml-workload
az vm create \
    --resource-group ml-workload \
    --name jammy \
    --image Ubuntu2204 \
    --generate-ssh-keys \
    --size Standard_NC4as_T4_v3 \
    --admin-username ubuntu --license-type UBUNTU_PRO

Install Nvidia Drivers (GPU Support)

For GPU capabilities, install NVIDIA drivers using Ubuntu’s package management system. Restart the system after installation.

sudo apt update -y
sudo apt full-upgrade -y
sudo apt install -y ubuntu-drivers-common
sudo ubuntu-drivers install
sudo systemctl reboot

Important: Standard NVIDIA drivers don’t support vGPUs (fractional GPUs). See instructions on the Azure site for installing GRID drivers, which might involve building an unsigned kernel module (which may be incompatible with Secure Boot).

Deploying Ollama with Snap

Snap simplifies the installation of Ollama and its dependencies, ensuring compatibility and streamlined updates. The –beta flag allows you to access the latest features and versions, which might still be under development

sudo snap install --beta ollama

Configuration

Configure Ollama to use the ephemeral disk

sudo mkdir /mnt/models
sudo snap connect ollama:removable-media # to allow the snap to reach /mnt
sudo snap set ollama models=/mnt/models

Installing Mixtral

At this point, you can run one of the open models available out of the box, like mixtral or llama2. If you have a fine-tuned version of these models (a process that involves further training on a specific dataset), you can run those as well.

ollama run mixtral

The first run might take a while to download the model.

Now you can use the model through the console interface:

Installing a UI

This step is optional, but provides a UI via your web browser.

sudo snap install --beta open-webui

Access the web UI securely

To quickly access the UI without open ports in the Azure security group, you can create an SSH tunnel to your VM using the following command:

ssh -L 8080:localhost:8080 ubuntu@${IP_ADDR}

Go to http://localhost:8080 in your web browser on your local machine (the command above tunnels the traffic from your localhost to the instance on Azure).:

In case you want to make this service public, follow this documentation.

Verify GPU usage

sudo watch -n2 nvidia-smi

Check that the ollama process is using the GPU, you should see something like this:

+---------------------------------------------------------------------------+
| Processes:                                                                |                                                                            
|  GPU   GI   CI        PID   Type   Process name                GPU Memory |
|        ID   ID                                                 Usage      |
|===========================================================================|
|    0   N/A  N/A      1063      C   /snap/ollama/13/bin/ollama     4882MiB |
+---------------------------------------------------------------------------+

Complementary and Alternative Solutions

• Charmed Kubeflow: Explore this solution for end-to-end MLOps (Machine Learning Operations), providing a streamlined platform to manage every stage of the machine learning lifecycle. It’s particularly well-suited for complex or large-scale AI deployments.
• Azure AI Studio: Provides ease of use for those seeking less customization.

Conclusion

Ubuntu’s open-source foundation and robust ecosystem make it a compelling choice for deploying open-source AI models. When combined with Azure’s GPU capabilities and Confidential Compute features, you gain a flexible, secure, and performant AI solution.

With Ubuntu confidential AI on Azure, businesses can undertake various tasks including ML training, inference, confidential multi-party data analytics, and federated learning with confidence.

The effectiveness of AI models depends heavily on having access to large amounts of good quality data. While using publicly available datasets has its place, for tasks like medical diagnosis or financial risk assessment, we need access to private data during both training and inference. 

When performing machine learning tasks in the cloud, enterprises understandably have concerns about the potential compromise of their sensitive data privacy as well as their model’s intellectual property. Additionally, stringent industry regulations often prohibit the sharing of such data. This makes it difficult, or outright impossible, to utilise large amounts of valuable private data, limiting the true potential of AI across crucial domains.

Confidential AI tackles this problem head on, providing a hardware-rooted execution environment that spans both the CPU and GPU. This environment enhances the protection of AI data and code at runtime by helping to safeguard it against privileged system software (such as the hypervisor or host OS) and privileged operators in the cloud.

To address this challenge, we are happy  to announce today  the preview of Ubuntu confidential AI on Azure, with Nvidia H100 Tensor core GPUs. This solution is built with Ubuntu 22.04 confidential VMs (CVMs), using AMD 4th Gen EPYC processors with SEV-SNP, alongside  NVIDIA H100 GPUs.  Ubuntu 22.04 is the only operating system to support this offering on Azure.

How confidential AI works

Confidential AI is made possible thanks to confidential computing, a game-changing  technology that represents a significant departure from the traditional threat model of public clouds. In the past, vulnerabilities within the extensive codebase of the cloud’s privileged system software, including the operating system, hypervisor, and firmware, posed a constant risk to the confidentiality and integrity of running code and data. Similarly, unauthorised access by a malicious cloud administrator could compromise the security of your virtual machine (VM) and its platform.

Ubuntu CVMs are here to give you back control over the security guarantees of your VMs. They enable you to run your workload within a hardware-protected Trusted Execution Environment, TEE. Such secure and isolated environments are purpose-built to prevent unauthorised access or alterations to applications and data at run-time, thereby enhancing security for organisations managing sensitive and regulated data.

As such, CVMs’ primary goal is to safeguard your guest workloads from various potential software threats, including the virtual-machine manager and other non-CVM software on the platform. CVMs also enhance your workload’s security against specific physical access attacks on platform memory, including offline dynamic random access memory (DRAM) analysis such as cold-boot attacks and active attacks on DRAM interfaces.

From confidential computing to confidential AI

While confidential computing efforts have historically focused primarily on CPUs, the advent of Nvidia H100 GPUs with confidentiality computing capabilities opens up new possibilities for extending this security paradigm to GPUs as well. The Azure solution, which integrates both CPU and GPU components, is what makes confidential AI achievable. At a high level, this solution relies on the following components:

• CPU-TEE: Ubuntu confidential VMs that run on the  AMD 4th Gen EPYC processors with SEV-SNP protect the workload’s computation while in the CPU:
  • Run-time confidentiality: the DRAM of your Ubuntu CVMs is kept encrypted thanks to the  new AES-128 hardware encryption engine that sits within the CPU memory  memory controller. This engine encrypts and decrypts memory pages whenever there is a memory read or write operation. Instead of having workload code and data in plain text in system memory, they are encrypted using a hardware-managed encryption key. This encryption and decryption process happens seamlessly within the CPU, ensuring strong memory isolation for confidential workloads.
  • Run-time Integrity: Ubuntu CVMs make use of the new AMD SEV SNP instructions and data structures that allow auditing of security-sensitive tasks typically carried out by privileged system software. These tasks encompass memory management and access to platform devices. For example, when reading memory pages mapped to confidential workloads, these new instructions also provide information about the last value written into the page. This feature helps prevent data corruption and replay attacks by detecting unauthorised modifications to memory pages.
• GPU-TEE: NVIDIA H100 Tensor Core GPUs, which protect the confidentiality and integrity of the workload’s computation within the GPU.
• Encrypted  PCIe communication between the CPUs and GPU.
• Attestation: Enables a relying party, whether it’s the owner of the workload or a user of the services provided by the workload, to cryptographically verify the security claims of both the CPU and GPU TEEs.

By integrating these components into a cohesive solution, confidential AI becomes not only feasible but also practical, allowing organisations to harness the power of AI while maintaining the highest standards of data security and confidentiality. Confidential AI can then be further augmented with cryptographic primitives, such as differential privacy, which protect the workload from further sophisticated data leakage.

Build your confidential AI workloads with Ubuntu today

Confidential AI can support numerous use cases across the entire lifecycle of building and deploying an AI application. For example,  you can use Ubuntu CVMs during the training phase to protect your data, model IP, and its weights.

Confidential AI can also be beneficial for fine-tuning large language models, whereby enterprises  need to use private data to optimize the generic models and improve their performance for their specific industries

We firmly believe that confidential AI represents a pivotal opportunity to unleash the full potential of AI, especially for industries that need to deal with security-sensitive data, such as healthcare and finance. We invite you to join us on this transformative journey with Ubuntu. Together, we can chart new horizons in AI innovation while steadfastly maintaining the highest standards of privacy and security for sensitive data.

Join us today and sign up for the Azure preview of confidential AI with Ubuntu. 

Share your questions, use cases, and feedback with us. we’re eager to hear from you and collaborate on shaping the future of AI security and innovation.

Further reading

• Introducing Confidential VMs on Ubuntu Pro for Azure
• Watch our webinar to learn more about confidential computing
• Read our blog post: “What is confidential computing? A high-level explanation for CISOs”
• Read our blog post:“Confidential computing in public clouds: isolation and remote attestation explained