[Cialug] link aggregation help requested
Hakan E. Duran
ehakanduran at gmail.com
Wed Mar 27 04:37:00 UTC 2019
Dear Jef and Tim,
Thank you very much for your very informative messages. The link below
provided by Tim was the clearest information on bonding I have seen so
far. It was very helpful to learn the differences between various
options. Based on that documentation, striping transmitted data between
two interfaces in a single switch local network situation is only
possible by balance-rr bonding mode. However, this will inevitably
create out of order packet transfers, causing TCP/IP's congestion
control system to kick in, which would then result in retransmitting of
certain segments. Therefore, the overall duration for file transfer may
or may not improve depending on the proportion of segments that need to
be retransmitted.
Tim's suggestion of using layer 3+4 is very similar to the scenario
above, and apparently is not compliant with 802.3ad standards.
Basic concepts are always the most important to know. Had I known these
before, I would have saved myself a lot of trouble by not wasting time
while trying to figure out how to create a bridge over a bond, which was
a pain in the neck to set up. Since I am not running this server for
hundreds of clients, as Jeff suggested, having a bond really doesn't
seem to provide an advantage for my personal/intended use. From the
positive side it was a learning experience, although misguided.
Thank you again so much for clarifying this for me. I think I will keep
my bond and leave it alone now that I know what to expect and not expect
from it.
Hakan
On 19/03/26 08:32, Tim McLaughlin wrote:
> Hakan,
>
> LACP rate, is just how often LACP control packets are sent, it will not
> effect bandwidth. If LACP in not coming up, try slow or 0. slow is more
> common than fast.
>
>
> take a look at
> https://www.kernel.org/doc/Documentation/networking/bonding.txt
>
> to use both interface at the same time between 2 servers, you need to do
> two things.
>
> 1. change the "xmit_hashing_policy" to "layer3+4"
>
> 2. transfer data using more than one connection.
>
> TCP/UDP port numbers are Layer4 information. A second TCP connection will
> use a different source port and the layer 4 hashing will put the data on
> the second interface.
>
> there is no standards based way to put a single data stream over 2
> interfaces. if you did, there would be the potential of out of order
> packets.
>
> With the hashing keeping a data stream on one path that issue will never
> happen.
>
>
> -Tim
>
>
>
>
>
>
> On Tue, Mar 26, 2019 at 12:36 AM Jeffrey Ollie <jeff at ocjtech.us> wrote:
>
> > A link aggregation group cannot make a single connection go faster than a
> > single link. So when you run a test using a file transfer or a basic
> > throughput tester you won't see any benefit. LAGs are really only useful
> > when one large system like big file server is talking simultaneously to
> > tens, hundreds or thousands of clients. A single file transfer won't be any
> > faster, but the file server will be able to serve many more clients at the
> > same time.
> >
> > Also, if you are hooking up a file server to a LAG you need to be sure that
> > your CPU and disks are actually capable of pushing that much data around.
> > It doesn't take all that much to build such a file server capable of
> > multi-Gb/s througput these days but you still need to pay attention to the
> > details.
> >
> >
> > On Mon, Mar 25, 2019 at 8:12 PM Hakan E. Duran <ehakanduran at gmail.com>
> > wrote:
> >
> > > Thank you Dave for your explanations. Please check my other email on
> > > this thread today. If I understand you correctly, LAG would only provide
> > > link redundancy but not stripping so to speak. Two devices that are
> > > connected to the same switch, each of which has a bond of 2 ethernet
> > > connections, would never use both of their ethernet connections
> > > simultaneously to transmit data in a faster fashion. Instead, they will
> > > keep the second connection as back up for the case the first one fails.
> > > Therefore, my original understanding of 802.3ad bonding is seriously
> > > flawed. Would you agree?
> > >
> > > Hakan
> > >
> > > On 19/03/25 11:44, Dave Weis wrote:
> > > > I'm not sure on what's causing the LACP to not come up but I can answer
> > > on
> > > > the speed question.
> > > >
> > > > Say you have two links in a group between two computers, The hashing
> > > > function looks at a specific packet and decides which of the two links
> > > that
> > > > given packet will traverse. At the lowest level the driver only knows
> > for
> > > > certain the MAC address of the receiver. The hashing function then does
> > > > some math to determine that the packet to 00:11:22:33:44:55 goes to
> > link
> > > 1.
> > > > The same thing happens on the computer on the other end of the LAG.
> > > >
> > > > This makes a transfer between two computers always stick to a single
> > > member
> > > > of the LAG. It would be different if there was a switch at each end and
> > > > multiple computers hooked to each switch all communicating to different
> > > > computers. The MAC addresses would hash differently to pick either
> > link 1
> > > > or link 2 so traffic would balance a bit better.
> > > >
> > > > Newer/more expensive gear can look further inside the packet and find
> > the
> > > > IP addresses and TCP port numbers to help encourage the hash function
> > to
> > > > spread the traffic more evenly on point to point style circuits. This
> > is
> > > > more important if you had two devices with a LAG between them such as
> > two
> > > > computers or two routers. The MAC address will never change on each end
> > > so
> > > > traffic between them will always hash to the same individual member
> > with
> > > > the basic hash. You need to incorporate IP's or something else more
> > > dynamic
> > > > into the mix.
> > > >
> > > > dave
> > > >
> > > >
> > > >
> > > >
> > > > On Sun, Mar 24, 2019 at 11:44 PM E. Hakan Duran <ehakanduran at gmail.com
> > >
> > > > wrote:
> > > >
> > > > > Thank you Dave for your reply. I actually changed the LACP rate from
> > > the
> > > > > default slow to fast today manually while troubleshooting this. I can
> > > > > easily go back to default settings since I saved the config file;
> > > however I
> > > > > know it will not be a solution to my problem. The other parameter I
> > > > > manually modified from the automatically chosen Network Manager
> > > settings
> > > > > was the transmit hash policy, which was changed to 2+3 from 0 if I
> > > recall
> > > > > correctly.
> > > > >
> > > > > Perhaps I get this wrong. I thought that by making a bond, one would
> > > > > effectively increase the bandwidth of the connection, which would
> > > translate
> > > > > to higher data transfer such as faster file transfers, etc. Is this
> > > > > perception accurate? Does layer 3 hashing enable faster data
> > transfer
> > > > > then? How can I figure whether my devices offer layer 3 hashing or
> > > not? The
> > > > > switch has a setting to modify load balance algorithm which is set at
> > > MAC
> > > > > address at the moment, but can be changed to IP/MAC address if
> > desired.
> > > > > Does this have the potential to achieve the faster data transfer
> > goal?
> > > > >
> > > > > I am in the process of upgrading the firmware and will report if it
> > > causes
> > > > > any noticable change in behavior.
> > > > >
> > > > > Thanks very much in advance and apologies for some probably
> > basic/silly
> > > > > questions. I don't have a technical background and so computing more
> > > as a
> > > > > hobbyist.
> > > > >
> > > > > Hakan
> > > > >
> > > > > On Sunday, March 24, 2019 9:03:42 PM CDT Dave Weis wrote:
> > > > > > Hello
> > > > > >
> > > > > > I looked at https://cstan.io/?p=8876&lang=en and there's a setting
> > > in
> > > > > the
> > > > > > BONDING_OPTS that might be affecting it. His/her output shows an
> > LACP
> > > > > rate
> > > > > > of slow and yours shows an LACP rate of fast.
> > > > > >
> > > > > > As far as speed, a single stream connection between two hosts will
> > > always
> > > > > > stay hashed to a single member of the LAG unless your devices offer
> > > > > layer 3
> > > > > > hashing such as port numbers. The usual method is mac address
> > > hashing and
> > > > > > that's going to be the same for every connection between the same
> > two
> > > > > hosts.
> > > > > >
> > > > > > Also make sure you have the newest firmware on your switch.
> > > > > >
> > > > > > dave
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Mar 24, 2019 at 8:37 PM Hakan E. Duran <
> > > ehakanduran at gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Dear all,
> > > > > > >
> > > > > > > This may be a stupid oversight as far as I know, but I could not
> > > solve
> > > > > > > it myself without some help for several months now and decided to
> > > ask
> > > > > > > here.
> > > > > > >
> > > > > > > I have a workstation that I am trying to set up link aggregation
> > > using
> > > > > > > the two available on-board ethernet interfaces (motherboard is
> > Asus
> > > > > > > Z10PE-D8WS if matters). I was able to set up the bond using the
> > > > > > > nm-connection-editor in linux (manjaro flavor) and also set up
> > the
> > > LAG
> > > > > > > on the Cisco managed switch I have (SG300-28, please see
> > > attached). The
> > > > > > > workstation is the LAG2 whereas LAG1 and 2 are 2 NAS units. As
> > you
> > > may
> > > > > > > see on the attached image, LAG2 seems to have a stand by member
> > on
> > > port
> > > > > > > 24, instead of having 2 active members like the NASes. I cannot
> > > figure
> > > > > > > out why one of the ethernet connections is assigned a stanby
> > member
> > > > > > > status. I pasted the output of the command "sudo cat
> > > /proc/net/bonding
> > > > > > > /bond0" below, I am unable to recognize a major problem there.
> > The
> > > > > Cisco
> > > > > > > switch documentation states that this would happen when there is
> > a
> > > > > > > hardware limitation in the connected device for building the
> > > aggregare
> > > > > > > (also see attached), but I am not sure if there would be any such
> > > > > > > limitation since this is a server main board, and also I don't
> > > want to
> > > > > > > reach that conclusion before ruling out the possibility that I am
> > > > > > > missing something. I can confirm that the current bond benchmarks
> > > are
> > > > > > > not superior to a single ethernet connection in terms of speed at
> > > this
> > > > > > > point of time (about 40-60 MBps for file transfer over Gigabit
> > > ethernet
> > > > > > > connection).
> > > > > > >
> > > > > > > Your pointers will be very much appreciated.
> > > > > > >
> > > > > > > Hakan
> > > > > > >
> > > > > > > sudo cat /proc/net/bonding/bond0
> > > > > > > ...
> > > > > > > Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
> > > > > > >
> > > > > > > Bonding Mode: IEEE 802.3ad Dynamic link aggregation
> > > > > > > Transmit Hash Policy: layer2+3 (2)
> > > > > > > MII Status: up
> > > > > > > MII Polling Interval (ms): 100
> > > > > > > Up Delay (ms): 0
> > > > > > > Down Delay (ms): 0
> > > > > > >
> > > > > > > 802.3ad info
> > > > > > > LACP rate: fast
> > > > > > > Min links: 0
> > > > > > > Aggregator selection policy (ad_select): stable
> > > > > > > System priority: 65535
> > > > > > > System MAC address: fa:aa:91:6b:bc:d2
> > > > > > > Active Aggregator Info:
> > > > > > > Aggregator ID: 1
> > > > > > > Number of ports: 2
> > > > > > > Actor Key: 9
> > > > > > > Partner Key: 1001
> > > > > > > Partner Mac Address: 00:38:df:d0:ee:49
> > > > > > >
> > > > > > > Slave Interface: enp6s0
> > > > > > > MII Status: up
> > > > > > > Speed: 1000 Mbps
> > > > > > > Duplex: full
> > > > > > > Link Failure Count: 1
> > > > > > > Permanent HW addr: 2c:fd:a1:c6:1f:19
> > > > > > > Slave queue ID: 0
> > > > > > > Aggregator ID: 1
> > > > > > > Actor Churn State: monitoring
> > > > > > > Partner Churn State: monitoring
> > > > > > > Actor Churned Count: 0
> > > > > > > Partner Churned Count: 0
> > > > > > > details actor lacp pdu:
> > > > > > > system priority: 65535
> > > > > > > system mac address: fa:aa:91:6b:bc:d2
> > > > > > > port key: 9
> > > > > > > port priority: 255
> > > > > > > port number: 1
> > > > > > > port state: 7
> > > > > > > details partner lacp pdu:
> > > > > > > system priority: 1
> > > > > > > system mac address: 00:38:df:d0:ee:49
> > > > > > > oper key: 1001
> > > > > > > port priority: 1
> > > > > > > port number: 71
> > > > > > > port state: 117
> > > > > > >
> > > > > > > Slave Interface: enp5s0
> > > > > > > MII Status: up
> > > > > > > Speed: 1000 Mbps
> > > > > > > Duplex: full
> > > > > > > Link Failure Count: 1
> > > > > > > Permanent HW addr: 2c:fd:a1:c6:1f:18
> > > > > > > Slave queue ID: 0
> > > > > > > Aggregator ID: 1
> > > > > > > Actor Churn State: monitoring
> > > > > > > Partner Churn State: monitoring
> > > > > > > Actor Churned Count: 0
> > > > > > > Partner Churned Count: 0
> > > > > > > details actor lacp pdu:
> > > > > > > system priority: 65535
> > > > > > > system mac address: fa:aa:91:6b:bc:d2
> > > > > > > port key: 9
> > > > > > > port priority: 255
> > > > > > > port number: 2
> > > > > > > port state: 7
> > > > > > > details partner lacp pdu:
> > > > > > > system priority: 1
> > > > > > > system mac address: 00:38:df:d0:ee:49
> > > > > > > oper key: 1001
> > > > > > > port priority: 1
> > > > > > > port number: 72
> > > > > > > port state: 69
> > > > > > > _______________________________________________
> > > > > > > Cialug mailing list
> > > > > > > Cialug at cialug.org
> > > > > > > https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> > > > > > >
> > > > > > _______________________________________________
> > > > > > Cialug mailing list
> > > > > > Cialug at cialug.org
> > > > > > https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> > > > > >
> > > > >
> > > > > _______________________________________________
> > > > > Cialug mailing list
> > > > > Cialug at cialug.org
> > > > > https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> > > > >
> > > > _______________________________________________
> > > > Cialug mailing list
> > > > Cialug at cialug.org
> > > > https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> > > _______________________________________________
> > > Cialug mailing list
> > > Cialug at cialug.org
> > > https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> > >
> >
> >
> > --
> > Jeff Ollie
> > The majestik møøse is one of the mäni interesting furry animals in Sweden.
> > _______________________________________________
> > Cialug mailing list
> > Cialug at cialug.org
> > https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
> >
> _______________________________________________
> Cialug mailing list
> Cialug at cialug.org
> https://www.cialug.org/cgi-bin/mailman/listinfo/cialug
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <http://www.cialug.org/pipermail/cialug/attachments/20190326/61b47b8a/attachment-0001.sig>
More information about the Cialug
mailing list