In my previous home hosting setup, I was running a BGP session from MetalLB within a kubernetes cluster to an EdgeRouter-X. It sounds overly complicated and silly, and it is, but it gives my cluster the ability to essentially create Elastic IPs and assign/manage them itself, while exposing them ‘publicly’.

In this setup, I can get a LoadBalancer and either allow it to be assigned dynamically or specify a static IP that is externally routable to the cluster. These IPs are announced from and routed by the nodes assigned the speaker DaemonSet — in my case, the control plane nodes.

Originally, when I configured this 2 years ago, it was my first real foray into setting up BGP in a decade or so and my first time doing it with k8s and MetalLB. I got a working config and stopped iterating. For my first attempt at migration, I took a stab at roughly converting the ERX Vyatta BGP config into the FRR format supported by the new Unifi Cloud Gateway Ultra I picked up for its 2.5G WAN port. No need to tangent on the strengths/weaknesses of Ubiquiti at this point — I’m here for BGP and equipment I have on hand.

Here’s a screenshot of the totality of the configuration details you get once you upload the FRR file and give it an arbitrary name (no hint at all that this is the single config allowed per device).

That’s it. There’s apparently no way to display the learned BGP routes, the status of any sessions, neighbors, or peer groups. But wait, surely there is more if I click on the entry? Nope. It’s a basic file upload interface with a single option, which just disables WAN monitoring:

Anyway, good thing I can Linux. I enabled SSH on the device, ignored the warranty-voiding notifications, and set to work. Through the powers of less /var/log/frr/bgpd.log (which I only found through muscle memory), I discovered that nothing was working.

Fortunately, I had the MetalLB side better understood at this point and was able to pull logs from the speakers to get their opinion of the routing session. I also have a BIRD route server configured as a separate peer, so I was able to validate that routes were still being exported from the Kubernetes cluster and were valid.

After some refinement and iteration, I arrived at a fairly concise config, with the following key identifiers:

! BGP configuration in FRR format for metallb peers
!
router bgp 64512
 bgp router-id 192.168.61.1
 no bgp default ipv4-unicast
 maximum-paths 10
 bgp graceful-restart
!
! BGP peer group configuration
 neighbor METALLB-PEERS peer-group
 neighbor METALLB-PEERS remote-as 65000
 neighbor METALLB-PEERS soft-reconfiguration inbound
 neighbor METALLB-PEERS bfd
 neighbor METALLB-PEERS timers 5 15
 neighbor METALLB-PEERS maximum-prefix 100 restart 5
 neighbor METALLB-PEERS route-map METALLB-IN in
 neighbor METALLB-PEERS route-map METALLB-OUT out
!
! BGP neighbors configuration
 neighbor 192.168.61.41 peer-group METALLB-PEERS
 neighbor 192.168.61.41 description k8s-leader-01
!
 neighbor 192.168.61.42 peer-group METALLB-PEERS
 neighbor 192.168.61.42 description k8s-leader-02
!
 neighbor 192.168.61.43 peer-group METALLB-PEERS
 neighbor 192.168.61.43 description k8s-leader-03
!
! IPv4 address family configuration
 address-family ipv4 unicast
  neighbor METALLB-PEERS activate
  redistribute connected
 exit-address-family
!
log file /var/log/frr/bgpd.log debugging
!

! Route map definitions
route-map METALLB-IN permit 10
!
route-map METALLB-OUT permit 10
!

After figuring out some unrelated VLAN/routing/ARP issues (a post in and of itself), a session came up and immediately there were valid routes on the CGU visible via CLI into my k8s cluster:

$ ip route show proto bgp
192.168.60.0 nhid 78 metric 20 
	nexthop via 192.168.61.41 dev br61 weight 1 
	nexthop via 192.168.61.42 dev br61 weight 1 
	nexthop via 192.168.61.43 dev br61 weight 1 
192.168.60.1 nhid 78 metric 20 
	nexthop via 192.168.61.41 dev br61 weight 1 
	nexthop via 192.168.61.42 dev br61 weight 1 
	nexthop via 192.168.61.43 dev br61 weight 1 
192.168.60.2 nhid 78 metric 20 
	nexthop via 192.168.61.41 dev br61 weight 1 
	nexthop via 192.168.61.42 dev br61 weight 1 
	nexthop via 192.168.61.43 dev br61 weight 1 
192.168.60.3 nhid 78 metric 20 
	nexthop via 192.168.61.41 dev br61 weight 1 
	nexthop via 192.168.61.42 dev br61 weight 1 
	nexthop via 192.168.61.43 dev br61 weight 1 
192.168.60.5 nhid 78 metric 20 
	nexthop via 192.168.61.41 dev br61 weight 1 
	nexthop via 192.168.61.42 dev br61 weight 1 
	nexthop via 192.168.61.43 dev br61 weight 1 
192.168.60.19 nhid 78 metric 20 
	nexthop via 192.168.61.41 dev br61 weight 1 
	nexthop via 192.168.61.42 dev br61 weight 1 
	nexthop via 192.168.61.43 dev br61 weight 1 
192.168.60.22 nhid 78 metric 20 
	nexthop via 192.168.61.41 dev br61 weight 1 
	nexthop via 192.168.61.42 dev br61 weight 1 
	nexthop via 192.168.61.43 dev br61 weight 1 
192.168.60.53 nhid 78 metric 20 
	nexthop via 192.168.61.41 dev br61 weight 1 
	nexthop via 192.168.61.42 dev br61 weight 1 
	nexthop via 192.168.61.43 dev br61 weight 1 

Bonus:

But wait there’s more. I had some random thoughts while throwing this togther and wanted to provide a working example for both sides of the equation here, to help the next person wandering down this path.

Route filtering

Only want to accept some prefixes? Simple enough extension to the FRR file. In this example we only will allow prefixes in the RFC1918 space in from the MetalLB peers and will only allow out the RFC6598 CGNAT ranges (contrived example for examples sake)

! Prefix list for allowed routes
ip prefix-list METALLB-PREFIXES seq 10 permit 192.168.0.0/16 le 32
!

! Prefix list for allowed outgoing routes to be shared with peers
ip prefix-list ALLOWED-OUT seq 10 permit 100.64.0.0/10 le 32

! Route map definitions
route-map METALLB-IN permit 10
 match ip address prefix-list METALLB-PREFIXES
!
route-map METALLB-IN deny 20
!
route-map METALLB-OUT permit 10
 match ip address prefix-list ALLOWED-OUT
!

MetalLB configs that work with this FRR config.

BGPPeer for my router (unchanged between ERX and CGU)

apiVersion: metallb.io/v1beta2
kind: BGPPeer
metadata:
  name: default
  namespace: metallb-system
spec:
  disableMP: false
  myASN: 65000
  peerASN: 64512
  peerAddress: 192.168.1.1
  peerPort: 179

I cleverly called this first pool first-pool (points for creativity)

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: first-pool
  namespace: metallb-system
spec:
  addresses:
  - 192.168.60.0/24
  autoAssign: true
  avoidBuggyIPs: false

Finally, a BGPAdvertisement

apiVersion: metallb.io/v1beta1
kind: BGPAdvertisement
metadata:
  name: default
  namespace: metallb-system

Questions, discussion, whatever?

Hit me up on mastodon and let me know all the ways I got this wrong or missed functionality, I’ll happily update!

Resources

I referenced a few articles now lost to time over the years while cobbling things together, but the crux of this migration was contained in this tutorial and the reference doc linked in the ‘upload an FRR file’ dialog.

Hopefully this helps anyone else looking to integrate MetalLB with a Unifi Cloud Gateway Ultra!