Link Aggregation Control Protocol (FS Switch + Ubuntu Server)¶
Overview¶
Goal
- Create a bonded connection using LACP (IEEE 802.3ad) between an FS switch (e.g., N5860) and an Ubuntu server (
eno1+eno2), enabling higher throughput and redundancy. - Enable trunking and VLAN support
- Enable Jumbo frames
Connect via Serial Console¶
Requirements:
- Console cable (e.g., RJ45-to-USB or serial-to-USB with PL2303 chip)
- Linux PC with access to
/dev/ttyUSB0or similar picocom,minicom, orscreeninstalled
Recommended: Use picocom
sudo apt install picocom
sudo picocom -b 115200 /dev/ttyUSB0
If permissions are an issue...
Ensure your user is in the dialout group:
sudo usermod -aG dialout $USER
newgrp dialout
Or fix /run/lock permissions (for minicom):
sudo chgrp dialout /run/lock
sudo chmod g+w /run/lock
Configure FS Switch¶
STEP 1: Access FS Switch CLI¶
After connecting via picocom, press Enter. You should see:
Switch>
To enter privileged mode:
Switch> enable
To enter configuration mode:
Switch# configure terminal
STEP 2: Clean up any incorrect or residual LACP configuration (OPTIONAL)¶
Bundle eth-0-1/2 and eth-0-13/14 and eth-0-23/24 into a logical port (agg1/2/3) using LACP (dynamic)
Switch# configure terminal
Switch(config)# interface eth-0-1
Switch(config-if)# no channel-group
Switch(config-if)# exit
Switch(config)# interface eth-0-2
Switch(config-if)# no channel-group
Switch(config-if)# exit
Switch(config)# interface eth-0-13
Switch(config-if)# no channel-group
Switch(config-if)# exit
Switch(config)# interface eth-0-14
Switch(config-if)# no channel-group
Switch(config-if)# exit
Switch(config)# interface eth-0-23
Switch(config-if)# no channel-group
Switch(config-if)# exit
Switch(config)# interface eth-0-24
Switch(config-if)# no channel-group
Switch(config-if)# exit
Verify:
Switch# show channel-group summary
Expected output
port-channel group-mode: flexible
Flags: s - Suspend T - Standby
D - Down/Admin down B - In bundle
R - Layer3 S - Layer2
w - Wait U - In use
Mode: SLB - Static load balance
DLB - Dynamic load balance
RR - Round robin load balance
RLB - Resilient load balance
Aggregator Name Mode Protocol Ports
----------------+---------+--------------+--------------------------------------
-------
agg1(SU) SLB LACP(Dynamic) eth-0-1(B) eth-0-2(B)
Switch> show interface agg1
Expected output
Interface agg1
Interface current state: UP
Hardware is AGGREGATE, address is 649d.9905.8aca (bia 649d.9905.8aca)
Bandwidth 20000000 kbits
Index 2049 , Metric 1 , Encapsulation ARPA
Speed - 20.00Gb/s , Duplex - Full , Media type is Aggregation
Link type is autonegotiation
FEC config: DEFAULT
FEC status: OFF
The Maximum Frame Size is 9600 bytes
VRF binding: not bound
ARP timeout 01:00:00, ARP retry interval 1s
ARP Proxy is disabled, Local ARP Proxy is disabled
5 minute input rate 28 bits/sec, 0 packets/sec
5 minute output rate 791 bits/sec, 0 packets/sec
10 packets input, 1264 bytes
Received 0 unicast, 0 broadcast, 10 multicast
0 runts, 0 giants, 0 input errors, 0 CRC
0 frame, 0 overrun, 0 pause input
232 packets output, 34336 bytes
Transmitted 0 unicast, 0 broadcast, 232 multicast
0 underruns, 0 output errors, 0 pause output
0 output discard
STEP 3: Create proper LACP bonds¶
Switch# configure terminal
Switch(config)# interface eth-0-1
Switch(config-if)# channel-group 1 mode active
Switch(config-if)# no shutdown
Switch(config-if)# exit
Switch(config)# interface eth-0-2
Switch(config-if)# channel-group 1 mode active
Switch(config-if)# no shutdown
Switch(config-if)# exit
Switch(config)# interface eth-0-13
Switch(config-if)# channel-group 2 mode active
Switch(config-if)# no shutdown
Switch(config-if)# exit
Switch(config)# interface eth-0-14
Switch(config-if)# channel-group 2 mode active
Switch(config-if)# no shutdown
Switch(config-if)# exit
Switch(config)# interface eth-0-23
Switch(config-if)# channel-group 3 mode active
Switch(config-if)# no shutdown
Switch(config-if)# exit
Switch(config)# interface eth-0-24
Switch(config-if)# channel-group 3 mode active
Switch(config-if)# no shutdown
Switch(config-if)# exit
STEP 4: Enable VLANs and configure aggregate interfaces for trunking¶
We do not configure each member port for trunking—only the aggregated interfaces (agg1, agg2, agg3).
Note
On this platform, you do not need to specify which VLANs are allowed on each trunk interface. If you want to allow all VLANs—as in your case—you simply configure the trunk mode, and once VLANs are defined in the VLAN database, they will be accepted by default.
Switch# configure terminal
Switch(config)# vlan database
Switch(config-vlan)# vlan 10
Switch(config-vlan)# vlan 20
Switch(config-vlan)# vlan 30
Switch(config-vlan)# vlan 40
Switch(config-vlan)# exit
Switch(config)# interface agg1
Switch(config-if)# switchport mode trunk
Switch(config-if)# switchport trunk allowed vlan all
Switch(config-if)# exit
Switch(config)# interface agg2
Switch(config-if)# switchport mode trunk
Switch(config-if)# switchport trunk allowed vlan all
Switch(config-if)# exit
Switch(config)# interface agg3
Switch(config-if)# switchport mode trunk
Switch(config-if)# switchport trunk allowed vlan all
Switch(config-if)# exit
STEP 5: Verify configuration before saving¶
Important
Always verify your configuration is correct before saving to NVRAM. This prevents boot issues from incorrect configurations.
Check channel-group summary¶
Switch# show channel-group summary
Expected output should show all aggregates UP with ports bundled (B flag):
Aggregator Name Mode Protocol Ports
----------------+---------+--------------+--------------------------------------
agg1(SU) SLB LACP eth-0-1(B) eth-0-2(B)
agg2(SU) SLB LACP eth-0-13(B) eth-0-14(B)
agg3(SU) SLB LACP eth-0-23(B) eth-0-24(B)
Verify VLAN configuration¶
Switch# show vlan brief
Expected output should show VLANs 10, 20, 30, 40 with aggregates as tagged (t) members:
VLAN ID Name State Member ports
======= =============================== ======= ========================
1 default ACTIVE eth-0-3(u) ... agg1(t) agg2(t) agg3(t)
10 VLAN0010 ACTIVE agg1(t) agg2(t) agg3(t)
20 VLAN0020 ACTIVE agg1(t) agg2(t) agg3(t)
30 VLAN0030 ACTIVE agg1(t) agg2(t) agg3(t)
40 VLAN0040 ACTIVE agg1(t) agg2(t) agg3(t)
Verify trunk configuration on aggregates¶
Switch# show running-config interface agg1
Switch# show running-config interface agg2
Switch# show running-config interface agg3
Each should show:
interface aggX
switchport mode trunk
switchport trunk allowed vlan all
!
Verify LACP negotiation status¶
Switch# show lacp internal
All ports should show: - State: bndl (bundled) - Port State: 0x3d (synchronized, collecting/distributing)
Switch# show lacp neighbor
All partners should show: - Flags: FA (Fast LACP, Active) - Valid partner MAC addresses (not 00:00:00:00:00:00) - Port State: 0x3f (fully operational)
STEP 6: Save configuration to NVRAM¶
Critical Step
Without saving, all configuration will be lost on switch reboot!
Switch# write memory
Expected output:
Building configuration...
[OK]
Verify the configuration was saved:
Switch# show startup-config | include channel-group
Should show all 6 channel-group commands (2 per aggregate):
channel-group 1 mode active
channel-group 1 mode active
channel-group 2 mode active
channel-group 2 mode active
channel-group 3 mode active
channel-group 3 mode active
STEP 7: Configure Ubuntu Server for LACP¶
Warning
Jumbo frames must be configured on the host side, including on bonds, VLANs, and bridges in the Netplan configuration.
/etc/netplan/50-cloud-init.yaml
network:
version: 2
ethernets:
eno1: {}
eno2: {}
eno3: {} # Dedicated for flat external provider network (br-ex)
eno4:
critical: true
dhcp4: true
dhcp-identifier: mac
bonds:
bond0:
interfaces: [eno1, eno2]
parameters:
mode: 802.3ad
lacp-rate: fast
mii-monitor-interval: 100
transmit-hash-policy: layer3+4
mtu: 9000
dhcp4: false
vlans:
bond0.10:
id: 10
link: bond0 # Management
bond0.20:
id: 20
link: bond0 # VXLAN/Overlay
mtu: 9000
bond0.30:
id: 30
link: bond0 # Storage
mtu: 9000
bond0.40:
id: 40
link: bond0 # Octavia LBaaS
bridges:
br-mgmt:
interfaces: [bond0.10]
addresses: [172.29.236.11/22]
routes:
- to: 172.29.236.0/22
via: 172.29.236.1
nameservers:
addresses: [1.1.1.1, 8.8.8.8]
mtu: 1500
parameters:
stp: false
forward-delay: 0
br-vxlan:
interfaces: [bond0.20]
addresses: [172.29.240.11/22]
mtu: 9000
parameters:
stp: false
forward-delay: 0
br-storage:
interfaces: [bond0.30]
addresses: [172.29.244.11/22]
mtu: 9000
parameters:
stp: false
forward-delay: 0
br-lbaas:
interfaces: [bond0.40]
mtu: 1500
parameters:
stp: false
forward-delay: 0
br-ex:
interfaces: [eno3]
mtu: 1500
parameters:
stp: false
forward-delay: 0
Apply:
sudo netplan apply
End-to-End Verification¶
After configuring both the switch and Ubuntu servers, verify complete LACP functionality.
Switch-side verification¶
Switch# show channel-group summary
Switch# show interface agg1
Switch# show lacp neighbor
Server-side verification¶
Check bond status on each server:
root@epoxy-dev-1:~# cat /proc/net/bonding/bond0
Look for:
- MII Status:
up - Bonding Mode:
IEEE 802.3ad Dynamic link aggregation - Active Aggregator Info: Valid Aggregator ID and Partner MAC (not 00:00:00:00:00:00)
- Actor Churn State:
none(no churning indicates stable LACP) - Partner Churn State:
none - Port state:
63(actor) and61(partner) indicates fully synchronized
Network connectivity test¶
Test Layer 2 connectivity between nodes:
root@epoxy-dev-1:~# ping -c 3 172.29.236.12 # to epoxy-2
root@epoxy-dev-1:~# ping -c 3 172.29.236.13 # to epoxy-3
Expected: 0% packet loss, sub-millisecond latency
Jumbo frames verification¶
Verify end-to-end jumbo frame support:
root@epoxy-dev-1:~# ip link show br-vxlan # should report mtu 9000
root@epoxy-dev-1:~# ping -M do -s 8972 172.29.240.12 # test jumbo frames to epoxy-2
If jumbo frame ping succeeds, MTU 9000 is working correctly across the LACP bond and VLAN.
Troubleshooting¶
LACP not negotiating (Partner MAC: 00:00:00:00:00:00)¶
Symptoms:
- cat /proc/net/bonding/bond0 shows Partner Mac Address as 00:00:00:00:00:00
- Bond shows "churned" state
- No connectivity between nodes
Root Cause: Switch lost LACP configuration (after reset/reboot without saving)
Solution:
1. Connect to switch via serial console
2. Run show channel-group summary - if empty, channel-groups are not configured
3. Reconfigure LACP following STEP 3-6 above
4. Critical: Run write memory to save configuration
5. Verify with show startup-config | include channel-group
VLANs not passing traffic¶
Check:
Switch# show vlan brief
Switch# show running-config interface agg1
Ensure aggregates are in trunk mode with VLANs allowed.
Configuration lost after switch reboot¶
Cause: Forgot to run write memory
Prevention: Always verify with show startup-config after making changes.