Networking-Blog

My WordPress Blog

Optimizing Long-Haul TCP Throughput

TCP, or Transmission Control Protocol, is the primary layer-4 communication protocol for file
copies, database synchronization, and related tasks over the WAN.

Many of these TCP synchronization tasks, such as database replication or rsync over ssh, are very
dependent on single-stream TCP throughput to maintain the replication consistence and latency
required to meet business objectives such as RTO and RPO.

Non-Optimized Performance

Without specific performance optimizations for high-bandwidth long-haul networks, TCP
throughput is a function of network latency.

If we assume a standard TCP window size of 64 kilobytes and cross-country latency of 80
milliseconds, the maximum possible single-stream throughput with no packet loss is:

When TCP was designed in the 1970s, the fastest long-haul networks available had far less than
1 megabit of available bandwidth for all traffic, let alone 6, so this was not a design consideration.

Optimizing Performance – Window Scaling

IETF RFC 1323 defines TCP window scaling, as an optional feature to improve performance over
“Long, Fat Networks”. These are networks with high throughput (generally in excess of 10 megabits),
but with high latency (due to geographic distance and speed-of-light delays).

TCP window scaling, if enabled, automatically scales the window to optimize the throughput of the
connection.
If we work backwards, and use the bandwidth delay product, let’s say we need 50 megabits
of throughput over a 
link with 100ms latency.

50 megabits * 100 milliseconds = 512 kilobyte TCP window

While it is possible to hand-optimize this value for specific situations, the best practice is to enable
automatic window
scaling. On Linux, this is accomplished by:

sysctl –w net.ipv4.tcp_window_scaling=1
add net.ipv4.tcp_window_scaling=1 to /etc/sysctl.conf

Optimizing Performance – Selective Acknowledgement

By default, a single lost packet can result in retransmission of an entire window worth of packets.
On long-fat networks
such as your typical WAN, this can be a substantial amount of data.

To resolve this problem, TCP Selective Acknowledgement, defined in RFC 2018, allows the receiving
system to
acknowledge all packets received even if an intermediate packet has not been received.
This results in retransmission 
of only the lost packets, instead of an entire window worth of data.

To enable selective acknowledgements on a Linux host:

sysctl –w net.ipv4.tcp_sack=1
add net.ipv4.tcp_sack=1 to /etc/sysctl.conf

Before and After

This is a real-world example, within a specific business unit of a leading Silicon Valley based software
company.
This involved replication traffic between two datacenters (one east coast, one west coast) over a
link with 90ms latency.

Throughput before optimization: 5 mb/sec
Throughput after optimization: 66 mb/sec

 

TCP window scaling can lead to increased host memory consumption due to the larger send and receive
buffers.
This is generally not an issue with modern hardware with multiple gigabytes of RAM

 

TCP window scaling and selective acknowledgements may not be supported by some older firewall
implementations.
This is generally not an issue with current products from top-tier network vendors such
as Cisco and Juniper.

 

Linux Ubuntu Disable IPV6

 

You need to disactivate IPv6, to do that, open terminal and enter the following commands:

echo “#disable ipv6″ | sudo tee -a /etc/sysctl.conf

echo “net.ipv6.conf.all.disable_ipv6 = 1″ | sudo tee -a /etc/sysctl.conf

echo “net.ipv6.conf.default.disable_ipv6 = 1″ | sudo tee -a /etc/sysctl.conf

 

Then restart your system.

How to force Network Manager to set auto negotiation on Start-Up

How to force Network Manager to set auto negotiation to false/off for a particular NIC (eth0) on start-up?

It so happens that I have a NIC that is only able to establish a connection with the current network when auto negotiation is set to false. In windows, this can be achieved rather easily by changing the NIC properties via Device Manager. Currently I am only able to do so in Ubuntu (10.04 if that matters) via the following after login:

sudo ethtool -s eth0 autoneg off speed 100 duplex full

sudo service network-manager restart

PS: I have read reports from ubuntu forums that it may be necessary to remove network-manager and only use /etc/network/interfaces for configuration, but I intend to do so only as a last resort. I would also appreciate it if someone can add auto-negotiation ethtool to the list of tags.

 

There seems to be no way to do it within Network Manager. Editing /etc/network/interfaces on the other hand is clumsy and causes delays in later editions of Ubuntu like 12.04 (see this and this). In the end the solution offered here turns out to be best (for now), by adding the ethtool command to rc.local:

sudo gedit /etc/rc.local ethtool -s eth0 autoneg off speed 100 duplex full 
# put this above 'exit 0

Cisco 887G ADSL + 3G Backup

VPN IP SLA – In order to keep vpn up and active.

IP SLA MONITOR STATIC FLOATING ROUTE SOURCE LAN :

ip sla 1
icmp-echo 10.20.0.1 source-interface vlan 1
threshold 2
timeout 1000
frequency 5
!
!
ip sla schedule 1 life forever start-time now
!
!
ip sla 10
icmp-echo 80.74.16.173 source-interface Dialer0
threshold 2
timeout 1000
frequency 5
!
ip sla schedule 10 life forever start-time now
ip sla responder
!
track 10 interface ATM0 line-protocol
delay down 15 up 15
!
!
ip route 0.0.0.0 0.0.0.0 Dialer0 name PRIMARY track 10
ip route 0.0.0.0 0.0.0.0 Cellular0 name 3G_BACKUP
!

Rest of the Configuration :

ADSL :

service internal
!
interface ATM0
no ip address
atm vc-per-vp 64
atm bandwidth dynamic
atm ilmi-keepalive 30 retry 5
pvc 0/38
encapsulation aal5mux ppp dialer
dialer pool-member 1
!
dsl sync interval 3
dsl sync mode itu
dsl operating-mode itu-dmt
dsl power-cutback 1
dsl noise-margin 3
dsl max-tone-bits 10
dsl bitswap both
!
interface Dialer0
ip address negotiated
ip verify unicast reverse-path
ip access-group INTERNET in
ip nat outside
encapsulation ppp
dialer pool 1
dialer-group 1
ppp authentication chap callin
ppp chap hostname test@securewan.co.uk
ppp chap password 0 password
ppp ipcp dns request
crypto map mapping
!

3G Interface

chat-script mobile “” “ATDT*98*1#” TIMEOUT 60 CONNECT

interface Cellular0
ip address negotiated
ip nat outside
encapsulation ppp
dialer in-band
dialer idle-timeout 0
dialer string mobile
dialer-group 1
ppp chap hostname web
ppp chap password 7 10590C1B
ppp ipcp dns request
crypto map mapping
!
!
interface Vlan1
description Corporate VLAN
ip address 10.20.4.100 255.255.255.0
ip access-group LAN in
ip tcp adjust-mss 1400
!

DNS Name-Server :

ip name-server 80.74.16.30
ip name-server 80.74.16.31
ip name-server 8.8.8.8
ip dns server
!
ip nat inside source route-map NAT_WW interface Dialer0 overload
ip nat inside source route-map NAT_3G interface Cellular0 overload
!
ip access-list extended NAT_ACL_WW
deny ip 10.20.4.0 0.0.0.255 10.20.0.0 0.0.255.255
permit ip 10.20.4.0 0.0.0.255 any
deny ip any any
!
ip access-list extended NAT_ACL_3G
deny ip 10.20.4.0 0.0.0.255 10.20.0.0 0.0.255.255
permit ip 10.20.4.0 0.0.0.255 any
deny ip any any
!
route-map NAT_WW permit 10
match ip address NAT_ACL_WW
!
route-map NAT_3G permit 20
match ip address NAT_ACL_3G

 

VPN :ADSL_3G

crypto isakmp policy 1
encr 3des
hash md5
authentication pre-share
group 2

!
crypto isakmp key cvsrtmvpn96 address 85.234.65.57crypto isakmp identity hostname
crypto isakmp keepalive 15 10

!
crypto ipsec transform-set secure esp-3des esp-md5-hmac
!
!
crypto map mapping_3g 1 ipsec-isakmp
set peer 85.234.65.57
set security-association lifetime seconds 86400
set security-association idle-time 86400
set transform-set secure
match address VPN
!
ip access-list extended VPN
permit ip 10.20.4.0 0.0.0.255 any
!

ACCESS-LIST RULE :
ip access-list extended INTERNET
remark WAVEWORKS
permit ip 80.74.16.8 0.0.0.7 any
permit ip host 80.74.17.9 any
remark IPSEC_VPN
permit esp host 85.234.65.57 any
permit udp host 85.234.65.57 any eq isakmp
permit icmp host 85.234.65.57 any
remark ICMP
permit icmp any any administratively-prohibited
permit icmp any any echo-reply
permit icmp any any packet-too-big
permit icmp any any time-exceeded
permit icmp any any traceroute
permit icmp any any unreachable
remark NTP
permit udp host 80.74.16.30 any eq ntp
permit udp host 80.74.16.31 any eq ntp
remark DNS
permit udp any eq domain any
remark DENY_ALL
deny ip any any log
!
ip access-list extended LAN
remark DENY_BROADCASTS
deny ip any host 10.20.4.0
deny ip any host 10.20.4.255
deny ip host 0.0.0.0 any
remark PERMIT_ALL
permit ip 10.20.4.0 0.0.0.255 any
remark DENY_ALL
deny ip any any log
!
ip access-list standard TELNET_SSH
permit 80.74.17.9
permit 80.74.16.8 0.0.0.7
permit 10.20.4.0 0.0.0.255
!
line vty 0 4
access-class TELNET_SSH in
!
dialer-list 1 protocol ip permit
!
line 3
exec-timeout 0 0
script dialer mobile

 

LINUX VMG VPN PROFILE :

conn cvs161_3g
left=85.234.65.57
leftsubnet=0.0.0.0/0
right=0.0.0.0
rightid=@cvs161.waves.uk.net
rightsubnet=10.20.253.40/29
type=tunnel
authby=secret
auth=esp
esp=3des-md5-96
ikelifetime=24h
keylife=24h
keyingtries=3
pfs=no
rekey=yes
auto=start
dpdaction=restart
dpddelay=15
dpdtimeout=30

ipsec.secrets :

%any 85.234.65.57 : PSK “cvsrtmvpn96”

All Traffic will be forced to be sent down the tunnel including
INTERNET BREAK-OUT terminating at the Linux VM :

Linux VPN Profile

leftsubnet=0.0.0.0/0
right=0.0.0.0

Cisco Router VPN ACL set to :

ip access-list extended VPN
permit ip 10.20.4.0 0.0.0.255 any

This will cause all traffic to be pushed down the tunnel, even internet bound traffic.

 

Changes made to Linux terminating firewall

This will locally break-out sourced traffic to the internet :
iptables -t nat -I POSTROUTING 83 -o eth1 -s 10.20.4.0/24 -j MASQUERADE
!
route add –host 85.234.66.161 gw 85.234.65.57
!

This will provide HQ connectivity and block intersite connectivity as well
as allow
internet access :

iptables -I FORWARD 46 -s 10.20.78.0/26 -d 10.20.0.0/24 -j ACCEPT
iptables -I FORWARD 47 -s 10.20.0.0/24 -d 10.20.78.0/26 -j ACCEPT
iptables -I FORWARD 48 -s 10.20.78.0/26 -d 10.20.0.0/16 -j DROP
iptables -I FORWARD 49 -s 10.20.0.0/16 -d 10.20.78.0/26 -j DROP
iptables -I FORWARD 50 -s 10.20.78.0/26 -j ACCEPT
iptables -I FORWARD 51 -d 10.20.78.0/26 -j ACCEPT

 

This is to allow HQ site to be able to establish connectivity over to remote site via
ipsec vpn tunnel

iptables -t nat -N 3g_access

sudo iptables -t nat -I POSTROUTING 85 -s 10.20.0.0/24 -j 3g_access
!
sudo iptables -t nat -I 3g_access 5 -d 10.20.78.0/26 -p icmp –icmp-type echo-request -j ACCEPT
sudo iptables -t nat -I 3g_access 6 -d 10.20.78.0/26 -j ACCEPT

LINUX – CHOWN – change the owner of a file

You can change the owner and group of a file or a directory with the chown command. Please, keep in mind you can do this only if you are the root user or the owner of the file.
Set the file’s owner:
$ chown username somefile
After giving this command, the new owner of a file called somefile will be the user username. The file’s group owner will not change. Instead of a user name, you can also give the user’s numeric ID here if you want.

You can also set the file’s group at the same time. If the user name is followed by a colon and a group name,
the file’s group will be changed as well.

$ chown username:usergroup somefile

After giving this command, somefile’s new owner would be user username and the group usergroup.
C

$ chown username somedir

Note that after giving this command, only the owner of the directory will change. The owner of the files
inside of the directory won’t change.
In order to set the ownership of a directory and all the files in that directory, you’ll need the -R option:

$ chown -R username somedir

Here, R stands for recursive because this command will recursively change the ownership of directories
and their contents. After issuing this example command, the user username will be
the owner of the
directory somedir, as well as every file in that directory.

Tell what happens:

$ chown -v username somefile

changed ownership of ‘somefile‘ to username
Here, v stands for verbose. If you use the -v option, chown will list what it did (or didn’t do) to the file.
The verbose mode is especially useful if you change the ownership of several files at once.

For example, this could happen when you do it recursively:

$ chown -Rv username somedir

changed ownership of ‘somedir/’ to username
changed ownership of ‘somedir/boringfile‘ to username
changed ownership of ‘somedir/somefile‘ to username

As you can see, chown nicely reports to you what it did to each file.

LINUX – why Linux feels slow, and how to fix that

The problem

What’s the number one cause of desktop performance problems?

Let me answer that question with another set of questions:

  • Have you noticed how slow your computer is after after unzipping a big file?
  • Perhaps you can also notice this effect overnight as well — you return to your computer,
    and every open application takes a while before they behave snappily again.
  • Do you use Amarok? Then you’ll notice this effect after Amarok has finished scanning
    your music collection.

This problem happens all the time. In fact, I would be willing to bet this is the cause most people
find
Linux to be slow.

The source of the problem

Here’s my chief finding: desktop performance has very little to do with schedulers and real
performance.
It has everything to do with disk caching and perceived performance.

Now, bear with me for a second, because what I’m about to explain will let you understand the
following.
Based on my limited understanding of the Linux kernel (please correct me if I’m
wrong, especially about the
terminology), there are two distinct cache types:

  • a block cache: caches block device contents
  • a inode/dentry cache: one layer above the block cache, caches directory entries and other
    filesystem-related things that cost more to look up than just block device contents

Problem number 1 is that the block cache competes with applications for RAM.
Problem number 2 is that the block cache competes with the inode/dentry cache.

On a normal desktop Linux computer, when you’re short of RAM, Linux will aggressively try to
free
memory to enlarge the caches. It usually does this by paging out unused portions of
applications to disk.

This is not good.

Why? Since disks are slow, the following consequences follow:

For desktop performance, where workloads are unpredictable, responsiveness needs dictate that
applications’
working sets should be in core as much as possible, because the probability of major
page faults in desktop
loads is much larger than on server loads.

All in all, how Linux uses your RAM matters a lot for desktop performance

You can see where this is going: if you have 1 GB of RAM (or less) you unzip a 300 MB archive, and the
Linux
kernel will swap out portions of your applications to disk in an effort to accomodate both the
300 MB file and
the blocks of the uncompressed files about to be written on disk. It will also shrink the
inode/dentry cache.

Linux, of course, thinks it’s doing a good thing — the archive uncompresses, in effect, rather quickly
because of this.
You, of course, want to kick Linux in the nuts because while (and after) the uncompress
took place, starting an
application or opening a new browser window suddenly takes ages, and your open
applications “blank out” for a
couple of seconds before reacting to your clicks and keystrokes.

The solution

There are two distinct and complementary measures we’ll take to solve this problem. 

Tuning swappiness to prevent impromptu RAM hijacking

Swappiness is the name Linux kernel developers gave to the preference between paging applications
out
to disk and (in practice) shrinking caches. If it’s close to 0, Linux will prefer to keep applications
in RAM
and not grow the caches. If it’s close to 100, Linux will prefer to swap applications out, and
enlarge the caches as much as possible. The default is a healthy 60.

On a desktop computer, you want swappiness to be as close to zero as possible. The reason you want
to
do this (even though it might hurt actual performance) is because this will immunize your computer
to
memory shortages caused by temporary big file manipulations (think copying a big video file to another
disk
).
The cache will still be as big as possible, but it won’t displace running applications.

Don’t believe me? If you have two gigabytes to spare on your / partition, then run the
following commands in order:

sync echo 3 > /proc/sys/vm/drop_caches 
dd if=/dev/zero of=/tmp/testfile count=1 bs=900M 
find / > /dev/null 
cp /tmp/testfile /tmp/testfile2

Don’t do anything else while they run. Yes, they will take a while. Be patient. What this does is rather
simple: creates a big 900 MB file, attempts to find all files on your file system, then copies the file one
more time.

Finished? Now do something with your computer. Un-minimize your minimized applications,
do something relating to your everyday work…

What happened? Excruciatingly slow, right? Did you notice how applications were hung for a while,
while the hard disk light blinked frantically?

OK, let’s tune swappiness. As root, run:

sysctl -w vm.swappiness=1

and retry the commands above. Big difference, right?

That’s because, with swappiness turned down, the Linux kernel no longer attempts to enlarge the
cache by paging applications out. Not unless you’re experiencing an extremely high memory shortage.

To make the change permanent, write vm.swappiness=1 on your /etc/sysctl.conf file.

Filesystem caches are more important than other caches

We’ve already established that the filesystem cache is important because, without it, file browsing goes
extremely slowly as well. Now we’ll learn how to tell Linux that we want it to prefer inode/dentry cache
to other caches.

Try this:

sync echo 3 > /proc/sys/vm/drop_caches 
dd if=/dev/zero of=/tmp/testfile count=1 bs=900M 
sysctl -w vm.vfs_cache_pressure=100 
find / > /dev/null 
cp /tmp/testfile /tmp/testfile2 
time find / > /dev/null 

sysctl -w vm.vfs_cache_pressure=50 
find / > /dev/null 
cp /tmp/testfile2 /tmp/testfile3 
time find / > /dev/null 
rm -f /tmp/testfile /tmp/testfile2 /tmp/testfile3

This series of commands does basically what the former did, only this time it attempts to
measure
the time it takes to find all files on your disk — after creating big files. It does so
twice — once with 
the default settings for cache preference, and once with settings heavily
tilted to favor
inode/dentry cache.

Check the output. Did the second run take less time? It definitely cut times in more than half
on my
desktop computer. When this knob is closer to 1, the kernel will prefer to protect
the
inode/dentry cache by freeing block cache first.

To make the change permanent, put vm.vfs_cache_pressure=50 on your
/etc/sysctl.conf file.

Please experiment with this value. Values close to 100 provide no gain. Values close to zero
can cause huge swap activity during big filesystem scans.

That’s it

Desktop Linux is quite good. However, there are certain situations in which it could improve.
Fortunately,
Linux comes right out of the box with the things you need to tune it for desktop
performance.

LINUX – SCP BACKUP FILES/DIRECTORIES

Iptables :

scp -P 22 TEST@1.1.1.1:/etc/iptables-sean.rules/home/salman/Downloads

PPTP :

Modify Access-Rights

cp /etc/ppp/chap-secrets/etc/
cp /etc/ppp/pap-secrets/etc/
chmod 777 /etc/chap-secrets
scp -P 22 TEST@1.1.1.1:/etc/chap-secrets/home/salman/Downloads
scp -P 22 TEST@1.1.1.1:/etc/pap-secrets/home/salman/Downloads

Copy directory

cp -r /etc/ppp/etc/ppp.backup
scp -P 22 -rp TEST@1.1.1.1:/etc/ppp.backup/home/salman/Downloads/BackupConfigs/CVS

 

IPSEC CVS :

Copy directory

cp -r cvs /etc/
chmod 777 cvs
scp -P 22 -rp TEST@1.1.1.1:/etc/cvs/home/salman/Downloads/BackupConfigs/CVS

 

IPSEC SECRETS :

scp -P 22 TEST@1.1.1.1:/etc/ipsec.secrets/home/salman/Downloads/BackupConfigs/CVS

 

IP ROUTE :
scp -P 22 -rp TEST@1.1.1.1:/etc/conf.d/home/salman/Downloads/BackupConfigs/CVS

SECURE COPY – SCP

What is Secure Copy?

scp allows files to be copied to, from, or between different hosts. It uses ssh for data transfer and provides
the same authentication and same level of security as ssh.

pwd = get current directory path.
!
SSH into remote machine from local computer :
ssh -p 2222 TEST@1.1.1.1

!
Copy the file “rom-0” from the remote host to a local host using port 2222

scp -P 2222 TEST@1.1.1.1:rom-0 /home/salman/Downloads
scp -P 2222 TEST@1.1.1.1:rom-0 /home/salman/Downloads

Copy the directory “rom-0” from the remote host to a local host directory “bar” using port 2222
$ scp -P 2222 -rp foo TEST@1.1.1.1:/home/salman/Downloads/bar
$ scp -P 2222 -rp /home/waveworks/foo TEST@1.1.1.1:/home/salman/Downloads/bar

Copy directory to another location :

cd /etc/temp
cp -r foo /etc/

Copy the directory  from the remote host to a local host directory :

scp -P 2222 -rp

!
Local file to root directory :
scp salman@10.255.255.134:/home/waveworks/Desktop/rom-0 rom-0

!
Copy the file “test1.txt” from the local host to a remote host using port 2222
scp -P 2222 test1.txt TEST@1.1.1.1:/home/waveworks
scp -P 2222 /home/salman/testfile TEST@1.1.1.1:/home/waveworks

!
Copy the directory “rom-0” from the local host to a remote host directory “bar”using port 2222
$ scp -P 2222 -r foo TEST@1.1.1.1:/home/waveworks
$ scp -P 2222 -r /home/salman/foo TEST@1.1.1.1:/home/waveworks

 

Examples

Copy the file “foobar.txt” from a remote host to the local host
$ scp your_username@remotehost.edu:foobar.txt /some/local/directory

!
Copy the file “foobar.txt” from the local host to a remote host
$ scp foobar.txt your_username@remotehost.edu:/some/remote/directory

!
Copy the directory “foo” from the local host to a remote host’s directory “bar”
$ scp -r foo your_username@remotehost.edu:/some/remote/directory/bar

!
Copy the file “foobar.txt” from remote host “rh1.edu” to remote host “rh2.edu”
$ scp your_username@rh1.edu:/some/remote/directory/foobar.txt your_username@rh2.edu:/some/remote/directory/

!
Copying the files “foo.txt” and “bar.txt” from the local host to your home directory on the remote host
$ scp foo.txt bar.txt your_username@remotehost.edu:~

!
Copy the file “foobar.txt” from the local host to a remote host using port 2264
$ scp -P 2264 foobar.txt your_username@remotehost.edu:/some/remote/directory

!
Copy multiple files from the remote host to your current directory on the local host
$ scp your_username@remotehost.edu:/some/remote/directory/{a,b,c} .
$ scp your_username@remotehost.edu:~/{foo.txt,bar.txt} .

 

View and Kill current processes on Ubuntu

Ubuntu has a command that lists all current processes.If you find the process ID,you can kill it by
kill command.

Type following to list all your own running processes :

ps ux

And use command to kill any process you want,the second column is PID number.

kill -9 PIDnumber

Ubuntu also has a graphical program to view current processes.Navigate to :

System–>Administrator–>System Monitor,

you will see system processes at Process tab,kill any by select it and click “End Process” button.

LINUX – Restart a Wireless Network

First thing to do is to see what wireless card you are using:

$ lspci | grep -i network

Let’s see what ipw-like modules/drivers are currently running/loaded:

$ modprobe -l | grep ipw/lib/modules/2.6.20-17-generic/kernel/ubuntu/wireless/ipw2100/ipw2100.ko/lib/modules/2.6.20-17-generic/kernel/drivers/usb/serial/ipw.ko/lib/modules/2.6.20-17-generic/kernel/drivers/net/wireless/ipw2100.ko/lib/modules/2.6.20-17-generic/kernel/drivers/net/wireless/ipw2100.ko

Here it is “ipw2100″. Let’s kill it (-r stands for “remove”):

$ sudo modprobe -r ipw2100

Let’s start it back up:

$ sudo modprobe ipw2100

Checking connectivity:
I issued the following commands:

sudo ifdown wlan0
sudo rmmod ipw2100

Waited a few seconds … and then:

sudo modprobe rtl8187
sudo ifup wlan0

Wireless networking then started working without the need of a reboot.

I had the same problem, and i should reboot the system to get wireless to work.

But now every time wireless stops working, I just enter these commands:

rmmod iwlagn
rmmod iwlcore
modprobe iwlcore
modprobe iwlagn