↓
 
Home→lacp→Rocky 9, Fedora 37, TrueNas and Juniper LACP / LAG notes

Rocky 9, Fedora 37, TrueNas and Juniper LACP / LAG notes

Software Effect Enterprises, Inc Posted on by

Purpose

1. I wanted to connect a Rocky Linux 9 and True Nas 12 to (2) juniper EX2300 switches such that either switch could be turned off and/or cables could be disconnected and the servers and the storage would be remain connected. As an additional test, I also tested Fedora 37

2. Improve overall network throughput, within the limitations of LACP*

3. Future goals was to be able to use vrrp to load balance and failover for two separate gateway routers.

* When all connections on a LACP configuration is working, all servers can use the total bandwidth available, however, a single network session can only use the maximum bandwidth available in a single network connection. Example:

We have 2 servers connected w/ (2) 1 Gig LACP connections to the switch. Assuming there is no other network traffic, when I start to copy a file from server 1 to server 2, the maximum speed of that copy will be 1 Gbps – the speed of one of the (2) 1 Gig connections. However – if while the former copy is running , I start to copy a second file – it also will be 1 Gbps – using all the bandwidth available in the 2nd 1 Gig connection.

Below are the configurations I used. The Rocky 9 box was connected to interface ge-0/0/11 on Switch1 and Switch2. The TrueNas12 box (later replaced with the Fedora 37 system) was connected to interface ge-0/0/9 on both switches. The switches were tied together using interfaces ge-0/0/1 and ge-0/0/3. Yes, we would have gotten overall better throughput if I got a bunch of fiber jumpers, some 10 gig fiber interfaces and a couple of Juniper EX4600 but:
1. I didnt have a couple of EX4600’s sitting around to play with. I had (2) EX2300-C-12P’s
2. The primary purpose for this exersize was to learn how to configure Rocky Linux, Fedora and TrueNas12 to work with LACP (after which I would be connecting them to a couple of EX4600’s which are in the production network. I will note any changes needed when  complete that action)

 

SWITCH 1 and 2

Except for the management interface and the hostname, both switches are configured the same

# we will be using 3 LACP sessions - one to the Rocky 9 server, one to the TrueNas and one connecting the two switches together
set chassis aggregated-devices ethernet device-count 3 

set interfaces ge-0/0/1 description "Connection to switch 2 via ae0"
set interfaces ge-0/0/1 ether-options 802.3ad ae0
set interfaces ge-0/0/3 description "Connection to switch 2 via ae0"
set interfaces ge-0/0/3 ether-options 802.3ad ae0

# switch 1 is connected to Rocky ifcfg-eno3
# switch 2 is connected to Rocky ifcfg-eno3
set interfaces ge-0/0/11 description "Rocky interface 3 / 4"
set interfaces ge-0/0/9 ether-options 802.3ad ae2

# switch 1 is connected to TrueNas igb0 (or fedora enp1s0f0)
# switch 2 is connected to TrueNas igb1 (or fedora enp1s0f1)
set interfaces ge-0/0/11 description "True NAS interface 0 / 1 "
set interfaces ge-0/0/11 ether-options 802.3ad ae1

# ae interface between the two switches
set interfaces ae0 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae0 unit 0 family ethernet-switching vlan members all


# ae interface to tie ge-0/0/9 on this switch to ge-0/0/9 on the second switch
# to provide lag for the TrueNas
set interfaces ae1 native-vlan-id 107
set interfaces ae1 aggregated-ether-options lacp active
set interfaces ae1 aggregated-ether-options lacp periodic fast
set interfaces ae1 aggregated-ether-options lacp system-id 00:00:00:00:00:01
set interfaces ae1 aggregated-ether-options lacp admin-key 3
set interfaces ae1 aggregated-ether-options lacp force-up
set interfaces ae1 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae1 unit 0 family ethernet-switching vlan members all


# ae interface to tie ge-0/0/11 on this switch to ge-0/0/11 on the second switch
# to provide lag for the Rocky 9 system
set interfaces ae2 native-vlan-id 107
set interfaces ae2 aggregated-ether-options lacp active
set interfaces ae2 aggregated-ether-options lacp periodic fast
set interfaces ae2 aggregated-ether-options lacp system-id 00:00:00:00:00:02
set interfaces ae2 aggregated-ether-options lacp admin-key 4
set interfaces ae2 aggregated-ether-options lacp force-up
set interfaces ae2 unit 0 family ethernet-switching interface-mode trunk
set interfaces ae2 unit 0 family ethernet-switching vlan members all

 

Rocky 9

ifcfg-eno2

 NM_CONTROLLED=yes
NAME=eno2
DEVICE=eno2
ONBOOT=yes
NETBOOT=yes
BRIDGE=brInside

TYPE=Ethernet
UUID=11840775-c75f-48ca-9847-e65fd56bc3b0

ifcfg-eno3

DEVICE=eno3
NAME=eno3
BOOTPROTO=none
ONBOOT=yes
SLAVE=yes
USERCTL=no
NM_CONTROLLED=yes
MASTER=bond1
 

ifcfg-eno4

DEVICE=eno4
NAME=eno4
BOOTPROTO=none
ONBOOT=yes
SLAVE=yes
USERCTL=no
NM_CONTROLLED=yes
MASTER=bond1
 

ifcfg-bond1

DEVICE=bond1
TYPE=Ethernet
ONBOOT=yes
USERCTL=no
NM_CONTROLLED=yes
MTU=9000
BOOTPROTO=static
IPADDR=10.254.200.21
PREFIX=24
BONDING_OPTS="mode=802.3ad miimon=100 lacp_rate=fast xmit_hash_policy=layer2+3"

 

ifcfg-brInside

NM_CONTROLLED=yes
DEVICE=brInside
NAME=brInside
TYPE=Bridge
BOOTPROTO=none
DEFROUTE=yes
DNS1=10.9.6.25
DNS2=98.159.192.2
IPV4_FAILURE_FATAL=no
IPV6_AUTOCONF=no
IPV6INIT=no
STP=yes
DELAY=2
BRIDGING_OPTS=priority=32768
HWADDR=
PROXY_METHOD=none
BROWSER_ONLY=no
IPADDR=10.9.6.64
PREFIX=24
NETMASK=255.255.255.0
GATEWAY=10.9.6.254
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
UUID=f43534f8-a3a0-4d92-a4b8-3c5c171d3827
ONBOOT=yes
 

Fedora 37

ethusb  (this chassis was older and only had 2 onbord 1 gig ports, so I augmented it with a USB-Ethernet adaptor)

[connection]
id=ethusb
uuid=c06efa51-aa1e-426b-8697-2932b5b332ef
type=ethernet
interface-name=enp0s26f7u3
master=904e45e6-6a39-41c5-ade1-8a41096194da
slave-type=bridge

[ethernet]

[bridge-port]
 

brInside

[connection]
id=brInside
uuid=904e45e6-6a39-41c5-ade1-8a41096194da
type=bridge
interface-name=brInside

[ethernet]

[bridge]

[ipv4]
address1=10.9.6.63/24,10.9.6.254
dns=10.9.6.25;
method=manual

[ipv6]
addr-gen-mode=default
method=auto

[proxy]
 

enp1s0f0

[connection]
id=enp1s0f0
uuid=00d60fb3-3ac2-4511-b01b-23e3b7b4c1db
type=ethernet
interface-name=enp1s0f0
master=bond1
slave-type=bond

[ethernet]

[bond-port]
 

enp1sof1

[connection]
id=enp1s0f1
uuid=fc6c20a2-8e76-46a4-93b5-a06ebd9ce014
type=ethernet
interface-name=enp1s0f1
master=bond1
slave-type=bond

[ethernet]

[bond-port]
 

bond1

[connection]
id=bond1
uuid=7ebfd3e3-ec82-416c-923d-e5416b9c27bd
type=bond
interface-name=bond1
timestamp=1681522128

[ethernet]

[bond]
lacp_rate=fast
miimon=100
mode=802.3ad
xmit_hash_policy=layer2+3

[ipv4]
address1=10.254.200.22/24
method=manual

[ipv6]
addr-gen-mode=default
method=auto

[proxy]
 

To configure the bond interface, I needed to use the following commands at the bash prompt

nmcli con mod bond1 bond.options mode=802.3ad,miimon=100,lacp_rate=fast,xmit_hash_policy=layer2+3
nmcli con add type bond-slave ifname enp1s0f0 con-name enp1s0f0 master bond1
nmcli con add type bond-slave ifname enp1s0f1 con-name enp1s0f1 master bond1
systemctl restart NetworkManager

 

TrueNas

 

Diagnostic tests and other notes

 

Test to show the bonding on the switch was working at 2 gig

root@SWITCH1> show interfaces ae1 | grep Speed
Link-level type: Ethernet, MTU: 1514, Speed: 2Gbps, BPDU Error: None, Ethernet-Switching Error: None, MAC-REWRITE Error: None,

Some other notes and references

https://serverfault.com/questions/569060/link-aggregation-lacp-802-3ad-max-throughput/569125#569125

https://serverfault.com/questions/805737/lacp-nfs-performance-confusion

https://serverfault.com/questions/1107811/juniper-lacp-with-linux-server-not-increase-speed

https://www.nodeum.io/howto/lacp-bonding-configuration-by-using-nmcli-centos-or-rhel-7

https://community.juniper.net/communities/community-home/digestviewer/viewthread?MID=74101

 

 

nmcli connection add type bridge ifname brServernet con-name brServernet connection.autoconnect yes
nmcli connection add type vlan con-name bond1.105 dev bond1 id 105 master brServernet connection.autoconnect yes

nmcli mod con enp7so 802-3-ethernet.mtu 9216

https://medium.com/@kbidarkar/configuring-bridges-and-vlans-using-nmcli-8cb79f45d3a6

https://www.golinuxcloud.com/nmcli-command-examples-cheatsheet-centos-rhel/

Comments are closed.

↑