Setup on-prem Jenkins with k3s

While cloud-based solutions are gaining traction, there are scenarios where maintaining an on-premises infrastructure, such as a K3s Kubernetes cluster, remains a preferred choice due to security, compliance, simplicity, or specific infrastructure requirements.

This article is designed to be a comprehensive guide for software developers, DevOps engineers, and IT professionals who are looking to set up a Jenkins server on their own on-premises K3s Kubernetes cluster. By following this three-step process, you can establish a powerful and flexible CI/CD environment that suits your organization’s unique needs.

The Problem You’re Solving In an ever-evolving technology landscape, orchestrating and managing containerized applications efficiently is paramount. Kubernetes has emerged as a leading solution for container orchestration, and K3s takes this a step further by offering a lightweight and easy-to-manage Kubernetes distribution. However, deploying and managing complex applications on Kubernetes can be challenging, and this is where K3s comes into play. Jenkins provides a robust automation platform to manage the build, test, and deployment stages of your software lifecycle. By integrating Jenkins’s ability to run agents and jobs on a K3s cluster, you can harness the power of automation while maintaining complete control over your infrastructure.

A Personal Journey: From Cloud to On-Premises

Allow me to share a personal story that highlights the importance of flexibility and adaptability in the world of DevOps. A few years ago, I was part of a DevOps team managing our Jenkins server on Amazon’s Elastic Kubernetes Service (EKS). The cloud-based EKS cluster served us well, providing scalability and ease of management. However, as our organization’s needs evolved, we faced a unique challenge.

Our corporate strategy shifted, wanted to utilize newly acquired on-premises infrastructure to meet specific security and compliance requirements. This change meant that we needed to migrate our Jenkins server, which was running seamlessly on EKS, to an on-premises environment.

Migrating to On-Premises: The Journey Begins

The journey of migrating our Jenkins server from EKS to an on-premises setup was both exciting and challenging. We knew the advantages of having complete control over our infrastructure, but the transition required careful planning and execution. We followed a similar three-step process to the one outlined in this article:

Step 1: Create Your VMs for the K3s Cluster Before diving into the technical steps, ensure that you have the necessary hardware or virtual machine resources to host your K3s cluster. Virtual Machines (VMs) will be the building blocks of your cluster. You can allocate resources according to the needs of your applications and the scale you anticipate. These VMs will serve as nodes within your K3s cluster, enabling the deployment and management of containerized applications.

Clone the k3s ansible playbook follow the ReadMe.md and configure the inventory to IPs of the provisioned VMs.

Step 2: Create the K3s Cluster Using the K3s Ansible Playbook Setting up a Kubernetes cluster manually can be complex and time-consuming. The K3s project simplifies this process by providing a lightweight Kubernetes distribution that is easy to install and manage. To automate the creation of your K3s cluster, we will use the K3s Ansible playbook. Ansible is an open-source automation tool that allows you to define infrastructure as code, making it an ideal choice for setting up and managing Kubernetes clusters.

By default the all the services that are deployed with the extra variable extra_server_args you can for example add the --disable=servicelb flag. to disable the ServiceLB controller.

This is necessary if you wish to run a different LB, such as MetalLB.

*While k3s is the Kubernetes distro used in this guide due to its simplicity to install and setup. At the end of the day Kubernetes is a container orchestrator so our containerized Jenkins Server is not limited to a specific Kubernetes distro.

Step 3: Install Jenkins Using the Jenkins Helm Chart With your K3s cluster up and running, it’s time to deploy Jenkins and start building your CI/CD pipelines. Helm, a package manager for Kubernetes, simplifies the deployment of complex applications by packaging all the necessary components and configurations into a single chart. We will use the Jenkins Helm chart to install and configure Jenkins on your K3s cluster. This will provide you with a flexible and scalable environment to automate your software delivery processes.

The official Jenkins helm chart is configured to work out-of-box to get started running your CI/CD jobs in your K3s cluster.

You can also use the My Bloody Jenkins Helm chart by odavid:

“My Bloody Jenkins is a re-distribution of the Official LTS Jenkins Docker image bundled with the most popular plugins and the ability to configure most aspects of Jenkins from a simple and single source of truth represented as YAML.”

• odavid

Some values worth considering to change for your personal uses are:

Ingress Configuration: To expose your Jenkins server securely to the internet or your internal network, consider configuring Ingress. Ingress controllers act as reverse proxies, directing traffic to the appropriate services within your cluster.

Jenkins Configuration as Code (JCasC): Managing Jenkins configurations manually can be error-prone and time-consuming. To streamline configuration management, you can utilize Jenkins Configuration as Code (JCasC). This powerful feature allows you to define Jenkins configurations using code (typically in YAML or Groovy) and apply them automatically during deployment. This ensures consistency and repeatability in your Jenkins setup.

By taking advantage of these advanced configurations, you can tailor your Jenkins deployment to meet your organization’s security, scalability, and automation needs. Whether you require a secure ingress setup, stringent access controls, or efficient configuration management, the Jenkins Helm chart offers the flexibility to make your Jenkins server deployment both robust and user-friendly.

Conclusion: In a world driven by rapid software development, combining the power of Jenkins, Kubernetes, and K3s can significantly enhance your ability to deliver high-quality applications efficiently. By following this three-step guide, you’ll have a solid foundation to build and customize your CI/CD pipelines, enabling your team to focus on innovation and delivering value to your users. However, it’s important to note that while the steps outlined here provide a strong starting point, successful implementation may require collaboration with your organization’s IT department.

For in-house DNS or webhooks from SaaS SCM sites like GitHub or Bitbucket, working with your IT team can help ensure proper configuration and security measures. Effective integration with external services requires thoughtful consideration of network setups and access controls. Furthermore, leveraging Virtual Private Networks (VPNs) can enhance security by allowing developers to access the Jenkins server remotely, ensuring seamless workflow continuity even when team members are not physically in the office.

Whether you’re a developer aiming to streamline your workflow or an IT professional seeking better control over your infrastructure, this guide equips you with the knowledge to set up an on-prem Jenkins server on a K3s cluster with ease. Remember that collaboration and adaptability are key as you tailor these steps to fit the unique needs and requirements of your organization.