From: Ole Morten Řian (omoian@infostream.no)
Date: Wed Aug 13 2003 - 09:59:22 EDT
Hello,
We experience very bad network throughput (1/10 of expected) in a firewalled
network with to Sun Cluster 3.1 nodes running IPMP in fail-over mode. It seems
like all netwok traffic frequently stop for a secound or so, and then resumes
with lots of retransmits due to timeout, see snips from snoop below. Shuting
down all four IPMP interfaces on the switchs removes the symtom. Enableling any
of the IPMP interfaces trigges the problem again. The problem can be seen on all
nodes in the network. The forewall serves other networks (very simliar except
IPMP), but we don't see any problems there.
Any hints on what may cause this is very welcome.
The network in question is a 16 address subnet with two Sun Cluster nodes (V240
and V280, Sun Cluster 3.1) and two application servers ( 2 x V210 ) all running
Solaris 9 04/03, latest network driver and IPMP patches are installed.
The firewall is Firewall-1 from Checkpoint NG-FP2 and Stonesoft Sonebeat
Fullcluster 3.0 running on two T1 with quadcards and Solaris 8 02/02.
The cluster nodes are connecte to two 3548 Cisco switches, using IPMP in
failover mode.
The following is parts of output from "snoop -i eri0 -t r" on one of the cluster
nodes while copying a 1G file trought the firewall. Notice the gaps in relative
time. These gaps can be observed on all nodes in this net:
..
0.04353 nfs7a-prod.our.net -> db3.ft.our.net UDP IP fragment ID=20966
Offset=31080 MF=1 TOS=0x0 TTL=62
0.04358 nfs7a-prod.our.net -> db3.ft.our.net UDP IP fragment ID=20966
Offset=32560 MF=0 TOS=0x0 TTL=62
0.31012 db3-mp0.ft.our.net -> 195.225.7.161 ICMP Echo request (ID: 32514
Sequence number: 54120)
0.31044 195.225.7.161 -> db3-mp0.ft.our.net ICMP Echo reply (ID: 32514
Sequence number: 54120)
0.51015 db3-mp1.ft.our.net -> (broadcast) ARP C Who is 195.225.7.161,
195.225.7.161 ?
0.71200 ? -> (multicast) ETHER Type=0000 (LLC/802.3), size = 52 bytes
0.87255 db1-mp1.ft.our.net -> 195.225.7.161 ICMP Echo request (ID: 46339
Sequence number: 13883)
0.88992 db3.ft.our.net -> nfs7a-prod.our.net NFS C READ3 FH=6807 at 192774144
for 32768 (retransmit)
0.89518 nfs7a-prod.our.net -> db3.ft.our.net UDP IP fragment ID=21222
Offset=0 MF=1 TOS=0x0 TTL=62
0.89552 nfs7a-prod.our.net -> db3.ft.our.net UDP IP fragment ID=21222
Offset=1480 MF=1 TOS=0x0 TTL=62
0.89556 nfs7a-prod.our.net -> db3.ft.our.net UDP IP fragment
..snipp snipp..
0.94237 nfs7a-prod.our.net -> db3.ft.our.net UDP IP fragment ID=22246
Offset=31080 MF=1 TOS=0x0 TTL=62
0.94241 nfs7a-prod.our.net -> db3.ft.our.net UDP IP fragment ID=22246
Offset=32560 MF=0 TOS=0x0 TTL=62
1.63807 ? -> (multicast) ETHER Type=0111 (LLC/802.3), size = 85 bytes
1.77991 db3.ft.our.net -> nfs7a-prod.our.net NFS C READ3 FH=6807 at 192937984
for 32768 (retransmit)
1.80185 nfs7a-prod.our.net -> db3.ft.our.net UDP IP fragment ID=22502
Offset=0 MF=1 TOS=0x0 TTL=62
..snipp snipp ..
1.80613 db3.ft.our.net -> nfs7a-prod.our.net NFS C READ3 FH=6807 at 193036288
for 32768
1.83999 db3-mp0.ft.our.net -> 195.225.7.161 ICMP Echo request (ID: 32514
Sequence number: 54121)
1.84027 195.225.7.161 -> db3-mp0.ft.our.net ICMP Echo reply (ID: 32514
Sequence number: 54121)
2.42871 app2.ft.our.net -> 195.225.7.140 TCP D=7001 S=51775 Syn
Seq=2916118394 Len=0 Win=49640 Options=<mss 1460,nop,nop,sackOK>
2.43049 app2.ft.our.net -> ait2.ft.our.net TCP D=7001 S=51776 Syn
Seq=454848416 Len=0 Win=49640 Options=<mss 1460,nop,nop,sackOK>
2.71121 ? -> (multicast) ETHER Type=0000 (LLC/802.3), size = 52 bytes
2.80318 db3.ft.our.net -> ns2.c.our.net DNS C 174.7.225.195.in-addr.arpa.
Internet PTR ?
....
-- Ole Morten Xian e-mail: omoian@infostream.no Senior Systems Engineer phone: +47 2270 4715 Infostream Services mobile: +47 982 15 525 http://www.infostream.no/ _______________________________________________ sunmanagers mailing list sunmanagers@sunmanagers.org http://www.sunmanagers.org/mailman/listinfo/sunmanagers
This archive was generated by hypermail 2.1.7 : Wed Apr 09 2008 - 23:26:55 EDT