Kubernetes
Kubernetes, also known as K8s, is an open-source system for automating deployment, scaling, and management of containerized applications.
It groups containers that make up an application into logical units for easy management and discovery. Kubernetes builds upon 15 years of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community.
Application Quick Links
| Application | Externally Accessible | Description |
|---|---|---|
| https://pgadmin.rizing.dev | No | Postgres pgadmin client |
| https://rizing.dev | Yes | Rizing developer guide |
| https://apps.rizing.dev | Yes | Apps launchpad |
| https://uptime.rizing.dev | Yes | App Uptime dashboard |
| https://registry.rizing.dev | Yes | Rizing's private container registry |
| https://echo-server.rizing.dev | Yes | Simple echo server |
| https://omni.rizing.dev | Yes | Omni testing site |
| https://arcgis.rizing.dev | Yes | ArcGIS Portal and server |
| https://draw.rizing.dev | Yes | Simple vector drawing canvas |
| https://identity.rizing.dev | Yes | Keycloak identity server |
If an application is NOT marked as "externally accessible" then you must be on the Rizing VPN to access it.
Tech Stack
| Logo | Name | Description |
|---|---|---|
 | Ansible | Automate bare metal provisioning and configuration |
 | FluxCD | GitOps tool built to deploy workloads and applications to Kubernetes |
 | cert-manager | Cloud native certificate management |
 | Cloudflare | DNS and Tunnel |
 | Docker | Ephermeral PXE server and convenient tools container |
 | Grafana | Operational dashboards |
 | Helm | The package manager for Kubernetes |
 | K3s | Lightweight distribution of Kubernetes |
 | Kubernetes | Container-orchestration system, the backbone of this project |
 | Loki | Log aggregation system |
 | Longhorn | Cloud native distributed block storage for Kubernetes |
 | MetalLB | Bare metal load-balancer for Kubernetes |
 | Nginx | Kubernetes Ingress Controller |
 | Prometheus | Systems monitoring and alerting toolkit |
 | Renovate | Automatically update dependencies |
 | Docker Registry | Private container registry |
Clusters
All major cluster manifests are defined and maintained in a Rizing gitops monorepository on ADO. The monorepo contains definitions for workloads (base manifests) which are shared across various clusters. Each cluster is then see the README.md in the root of the repository for more details.
Current major clusters:
| Cluster | Primary Domain | Publish Branch | Description |
|---|---|---|---|
| on-prem staging | https://rizing.xyz | staging | Staging cluster for the production on-premises cluster, used to test out changes before applying to production |
| on-prem production | https://rizing.dev | production | On-premises cluster housing the majority of internal Rizing tools |
| cloud-products production | https://rizing.app | production | Production cluster housing Rizing's SaaS products. |
| cloud-operations production | https://rizing.cloud | production | Production cluster housing utilities to support other clusters (Harbor registry, MinIO, etc.) |
Currently cluster changes are deployed when a change is made the cluster's "publishing branch" (see table above and diagram below).
GitOps branch publishing workflow
On-prem Production Cluster
IP addresses
| IP Address | Description |
|---|---|
| 23.111.141.181 | External static IP address into the on-premise environment |
| 10.3.81.160 | Internal load balanced K3s service IP (MetalLB) |
| 10.3.81.159 | Kube-vip load balanced virtual IP used for control plain HA |
| 10.3.81.160-1165 | Available (MetalLB) load balancer service IPs |
Cluster Nodes
| Node | IP | Role |
|---|---|---|
| kd3-1 | 10.3.81.151 | Control plane |
| kd3-2 | 10.3.81.152 | Control plane |
| kd3-3 | 10.3.81.153 | Control plane |
| kd3-4 | 10.3.81.154 | Worker 1 |
Cluster Overview
Domains
The on-prem Kubernetes cluster has a wildcard domain setup for *.rizing.dev. The wildcard domains are registered with Rizing's GoDaddy account but DNS is setup and managed through Cloudflare.
Dynamic External Domains
The following domains have dynamic wildcard DNS routing in place which will route all matching domains to the Kubernetes cluster.
This means that you should ONLY assign an ingress host that matches one of these patterns if you want the application to be exposed externally. Once the ingress is setup, the URI will be instantly available externally (no additional DNS setup is necessary).
- *.preview.rizing.dev ➙ on-premise PRD cluster (10.3.81.160)
- *.development.rizing.dev ➙ on-premise PRD cluster (10.3.81.160)
- *.staging.rizing.dev ➙ on-premise PRD cluster (10.3.81.160)
- *.production.rizing.dev ➙ on-premise PRD cluster (10.3.81.160)
On-prem Staging Cluster
IP addresses
| IP Address | Description |
|---|---|
| 23.111.141.181 | External static IP address into the on-premise environment |
| 10.3.81.160 | Internal load balanced K3s service IP (MetalLB) |
| 10.3.81.159 | Kube-vip load balanced virtual IP used for control plain HA |
| 10.3.81.160-1165 | Available (MetalLB) load balancer service IPs |
Cluster Nodes
| Node | IP | Role |
|---|---|---|
| kd3-dev-1 | 10.3.81.155 | Control plane |
| kd3-dev-2 | 10.3.81.156 | Control plane |
| kd3-dev-3 | 10.3.81.157 | Worker 1 |
Cluster Overview
Domains
The on-prem staging cluster has a wildcard domain setup for *.rizing.xyz. The wildcard domains are registered with Rizing's GoDaddy account but DNS is setup and managed through Cloudflare.
External Domains
This cluster does not have any external DNS routing in place. All requests to this cluster/domain need to be made while on the RIZINGIO domain (VPN).
Cloud-products Production Cluster
The cloud-products cluster is deployed in Azure Kubernetes Services (AKS) environment and houses Rizing's SaaS product offerings.
IP addresses
| IP Address | Description |
|---|---|
| 51.8.40.167 | External static IP address into the cloud-products cluster |
Cluster Nodes
The cluster node count changes based on workload.
Domains
The cloud-products Kubernetes cluster has a wildcard domain setup for *.rizing.app. The wildcard domains are registered with Rizing's GoDaddy account but DNS is setup and managed through Cloudflare.
Dynamic External Domains
The following domains have dynamic wildcard DNS routing in place which will route all matching domains to the Kubernetes cluster.
This means that you should ONLY assign an ingress host that matches one of these patterns if you want the application to be exposed externally. Once the ingress is setup, the URI will be instantly available externally (no additional DNS setup is necessary).
- *.rizing.app ➙ cloud-products cluster (51.8.40.167)
Cloud-operations Production Cluster
The cloud-operations cluster is deployed in Azure Kubernetes Services (AKS) environment and houses utility apps/services to support various production Kubernetes needs - such as storing and replicating database backups and providing a container registry (with storage and replication capability).
IP addresses
| IP Address | Description |
|---|---|
| 48.216.129.165 | External static IP address into the cloud-operations cluster |
Cluster Nodes
The cluster node count changes based on workload.
Domains
The cloud-operations Kubernetes cluster has a wildcard domain setup for *.rizing.app. The wildcard domains are registered with Rizing's GoDaddy account but DNS is setup and managed through Cloudflare.
Dynamic External Domains
The following domains have dynamic wildcard DNS routing in place which will route all matching domains to the Kubernetes cluster.
This means that you should ONLY assign an ingress host that matches one of these patterns if you want the application to be exposed externally. Once the ingress is setup, the URI will be instantly available externally (no additional DNS setup is necessary).
- *.rizing.cloud ➙ cloud-operations cluster (48.216.129.165)
Pull Requests
One major benefit of the Dev/Test Kubernetes cluster (with dynamic wildcard DNS) is the ability to create "sandboxed" preview deployments for every PR.
The Pull Request workflow is achieved by having a single/authoritative source of truth - which is the Kubernetes deployment manifests (gitops) repository. This repository represents the state of deployments within the cluster and the deployments will automatically adjust anytime the repository changes (using FluxCD).
