Homelab Networking

Exposing the home network network to the wide open internet can be a scary thing. But even with a basic home network setup, it can be done in a relatively secure manner. This post will cover setting up CloudFlare to manage your DNS and loadbalancer. Note that it will be using CloudFlare loadbalancers which requires a subscription of at least $5/month. There are other providers that can offer the same service but (imho) CloudFlare includes a lot of features as well as cloud-native integrations.

Setup

  • CloudFlare for external DNS and Loadbalancing
  • K3s Kubernetes Cluster (RPis/Intel Nucs)
  • MetalLB for baremetal network loadbalancing
  • NGINX Loadbalancing for external only network traffic

Depending on the ISP, DDNS may be required. This guide will assume it is not required and that the home network is accessible with a static IP.

Initial Setup on Cloudflare

CloudFlare requires the domain to be under its management. If it’s managed by another entity, like AWS Route53, the registrar will need to be modified to point CloudFlare’s nameservers. That can be done by clicking on “Add a site” and following the instructions to import the domain. If the domain was originally registered with AWS Route53, the nameservers can be changed in the Route53 management console under “Registered Domains.”

To ensure end-to-end encryption, it is recommended to set strict TLS encryption.

  1. Click into the “SSL/TLS” tab and set Full (Strict).
  2. On the same page, go to “Edge Certificates” and ensure “Always Use HTTPS” is enabled.
  3. Also on the same page, go to “Origin Server” and create an “Origin Certificate”. This will be the certificate installed in the homelab to securely communicate to the CloudFlare proxy. Download the certificate and keep it in a secure location.
  4. On the “Origin Server” page, it is also highly recommended to enable “Authenticated Origin Pulls”. This will make the CloudFlare proxy present a client certificate to the homelab ingress to authenticate. The .PEM file for the CA to authenticate the client certificate can be found here: https://developers.cloudflare.com/ssl/static/authenticated_origin_pull_ca.pem

The certificates will be used when installing the second IngressClass in the homelab cluster.

Preparing Homelab

The homelab K3s cluster will need to be setup before exposing external traffic for the open internet.

Setting up MetalLB

Will not mandatory, installing MetalLB will create a single loadbalanced IP to direct client traffic to the Kubernetes cluster.

To install MetalLB, use helm with the necessary values.yaml required for the homelab network environment.

helm repo add metallb https://metallb.github.io/metallb
cat <<EOF | sudo tee /tmp/metallb/values.yaml
configInline:
  address-pools:
    - name: default
      protocol: layer2
      addresses:
      - 192.168.x.x/28 # Replace this CIDR block with the desired list of IPs available on the LAN
EOF
helm upgrade -i --create-namespace -n metallb metallb/metallb -f /tmp/metallb/values.yaml

If the Kubernetes cluster is K3s based, MetalLB will render servicelb unnecessary. To disable it, simply add the following configuration to the bootstrap server’s /etc/rancher/k3s/config.yaml:

disable:
  - servicelb

Once setup, any loadbalancer type services that the cluster uses will be assigned a single IP by MetalLB, including the Traefik loadbalancer that K3s comes with. To find what IP that is, simply run kubectl -n kube-system get services -n traefik. If desired, this could be the Ingress IP for your cluster. All that would be required is setting the default certificate to CloudFlare’s and port forwarding that Ingress IP on the home router.

Setting up the Cluster’s IngressClass

An alternative to having all Ingresses being exposed to the open Internet is having another IngressClass. This way, all administrative or local services can be exposed only to the LAN while only specific services will be exposed.

For this setup, K3s comes with Traefik which will be kept as the local and default IngressClass. The following Helm Chart config will create the Traefik IngressClass as well as other custom configuration.

# Run the following as an administrator on the boostrap server node
mkdir -p /var/lib/rancher/k3s/server/manifests
cat <<EOF | sudo tee /var/lib/rancher/k3s/server/manifests/traefik-config.yaml
apiVersion: helm.cattle.io/v1
kind: HelmChartConfig
metadata:
  name: traefik
  namespace: kube-system
spec:
  valuesContent: |-
    # Specify MetalLB address pool for the loadbalancer service
    service:
      annotations:
        metallb.universe.tf/address-pool: default
    # Create default cluster IngressClass
    ingressClass:
      enabled: true
      isDefaultClass: true
    # Disable Traefik dashboard
    ingressRoute:
      dashboard:
        enabled: false
    # Ensure HTTPS traffic
    ports:
      web:
        redirectTo: websecure
EOF
systemctl restart k3s

To create the second IngressClass, NGINX is used1. This will handle all WAN traffic to and from the CloudFlare proxy. Since a specific certificate is required, the certificate downloaded from CloudFlare above will need to be created and referenced in the NGINX deployment.

# Installing the certificate.
cat <<EOF | sudo tee /tmp/cloudflare/origin-secret.yaml
apiVersion: v1
data:
  # To get the base64 of the cert/key, simply run "echo <certificate or key> | base64 -w0 && echo"
  tls.crt: <base64 of tls crt>
  tls.key: <base64 of tls key>
kind: Secret
metadata:
  name: cloudflare-origin-tls
  namespace: ingress-nginx
type: kubernetes.io/tls
EOF
kubectl apply -f /tmp/cloudflare/origin-secret.yaml

# Installing the CA
cat << EOF | sudo tee /tmp/cloudflare/ca.yaml
apiVersion: v1
data:
  ca.crt: <base64 of ca crt>
kind: Secret
metadata:
  name: cloudflare-auth
  namespace: ingress-nginx
type: Opaque
EOF
kubectl apply -f /tmp/cloudflare/ca.yaml

# Installing NGINX
helm repo add nginx https://kubernetes.github.io/ingress-nginx
cat <<EOF | sudo tee /tmp/nginx/values.yaml
controller:
  config: {}
  extraArgs:
    # Reference certificate created earlier
    default-ssl-certificate: "ingress-nginx/cloudflare-origin-tlsn"
  # Create and specify WAN IngressClass.
  ingressClassResource:
    name: nginx-public # Remember this! This will be the WAN IngressClass
    enabled: true
    default: false
  service:
    # Ensure HTTPS only
    enableHttp: false
    enableHttps: true
    # Specify MetalLB address pool for the loadbalancer service
    annotations:
      metallb.universe.tf/address-pool: default
EOF
helm upgrade -i --create-namespace ingress-nginx nginx/ingress-nginx -f /tmp/nginx/values.yaml

Once installed, MetalLB should provision an IP address for the NGINX loadbalancer which can be identified with kubectl -n ingress-nginx get services ingress-nginx-controller under EXTERNAL-IP. This IP will need to be port forwarded on the home router. Each router has different configuration methods, but it should be easy find instructions by searching the specific model of router and “Port Forwarding”.

Specifying External Ingress

Now that the cluster is ready for exposing external traffic, simply set the desired Ingress with the nginx-public ingressClassName and create annotations for client TLS authentication2. For example:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: external-service
  namespace: external
  annotations:
    # For CloudFlare Authenticated Origin Pulls
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-secret: "ingress-nginx/cloudflare-auth" # Must match created CA secret
    nginx.ingress.kubernetes.io/auth-tls-match-cn: "CN=origin-pull.cloudflare.net"
spec:
  # This is where nginx-public must be specified
  ingressClassName: nginx-public
  rules:
  - host: external.domain
    http:
      paths:
      - backend:
          service:
            name: internal-svc
            port:
              number: 80
        path: /
        pathType: ImplementationSpecific
status:
  loadBalancer:
    ingress:
    - ip: 192.168.x.x
    # This is the IP address allocated by MetalLB for the NGINX loadbalancer

Finish Configuring CloudFlare

The final step is creating the loadbalancer on Cloudflare.

  1. Click into the “Traffic” tab and then into “Load Balancing”.
  2. Create Load Balancer.
  3. Set the Hostname and ensure the orange proxy setting is enabled.
  4. Add Origin Pool and set the IP address to the home network. To identify which IP to use, simply run curl ifconfig.me. NOTE: This will not work if the ISP does not use a static IP for your network.
  5. Optionally add a Monitor which functions as a health check.
  6. Add any desired Traffic Steering and Customer Rules as needed.

Once the Loadbalancer is complete and active, anyone can connect to the homelab services securely!

Additional Notes

1 Deploying another Traefik helm release did not work as intended while testing, but should theoretically work as well. For convenience purposes, NGINX was used.

2 Personal testing has not shown a way to set client authentication by default and must be specified for every ingress.

Not covered in this guide but more secure configuration can be achieved from the NGINX ingress. It can be restricted to only CloudFlare proxies IPs found here: https://www.cloudflare.com/ips/