Linux network performance tuning

Why we need a fine tuning of network settings?

Usually the default network parameters supplied along with the OS should be able to handle the regular traffic. But if you are managing a high traffic server and if you are experiencing sluggishness in accessing your application, then it is recommended to do a linux network performance tuning of your linux operating system.

TCP Connection Establishment

As you know, web servers\application servers generally use Transmission Control Protocol(TCP) for their client-server communication. TCP is a connection oriented protocol, which means, the sender and receiver needs to establish a reliable connection between them to transmit the data. As the first step of establishing the connection, the sender will send a connection request to the receiver. If the receiver is ready to accept the data, then it will send back an acknowledgement(ACK) back with SYN bit set. Now the sender will acknowledges the receiver’s initial sequence Number and its ACK. Now the sender will start its data transfer.

performance tuningFlow Control & Window Scaling

Since the sender and receiver may not be having same network speed, the TCP uses a flow control mechanism named sliding window protocol, so that the sender and receiver will be transmitting the data at same rate. The receiver and the sender will exchange the information about the amount of data, they can accept, using a TCP segment field called receive window. The receiver updates the filed with the amount of data, that it can accept.

Upon seeing the value, the sender will adjust is data transmission, so that it will not send data above this window size, until an acknowledgement is received from the receiver. Once an acknowledgement is received and once the new receive window size is declared by the receiver, the sender can transmit the next set of data. Earlier, the maximum receive window size that can be mentioned in a TCP frame was 65,535 bytes. Now using a new feature called, Window Scaling, the limit is increased to a maximum of 1,073,725,440 bytes(1Gb)

Bandwidth Delay Product – BDP, the bits of data in transit between hosts is equal to Bandwidth * RTT

or in other words,

BDP (bytes) = total bandwidth (KBytes/sec) x round trip time (ms)

The network throughput of that network <= (TCP buffer size / RTT)

The TCP Windows size needs to be large enough to accommodate network bandwidth x maximum expected delay

or

TCP window size needs to be >= BW * RTT

On a 100 Mbps network with round trip time(RTT) value of 150 ms and with a TCP buffer size of 128 KB, the Bandwidth Delay Product will be 1.88 MB. The maximum throughput value will be <= 6.99 Mbps. To use the 100 Mbps with RTT 150ms, the TCP buffer size should be >= 1831.1 KB

Window Scaling

In our above mentioned network, we are wasting 1815 Kilo Bytes of window size(1880-65). So we need to enable the Window Scaling feature. We can modify the window scaling parameter in linux by editing the sysctl.conf file. You need to set the below parameter to 1.

net.ipv4.tcp_window_scaling = 1

You can do the same by executing the below command,

echo 'net.ipv4.tcp_window_scaling = 1' &gt;&gt; /etc/sysctl.conf

Obtain TCP Memory Values

Now obtain the TCP memory values by executing the below commands,

cat /proc/sys/net/ipv4/tcp_mem

To view receive socket memory size, please execute the below two commands,

cat /proc/sys/net/core/rmem_max
cat /proc/sys/net/core/rmem_default

To view the send socket memory size, please execute the below two commands. The first command will give its maximum value and the second command will provide you its default value.

cat /proc/sys/net/core/wmem_max
cat /proc/sys/net/core/wmem_default

To view the maximum amount of option memory buffers, please execute the below command,

cat /proc/sys/net/core/optmem_max

Performance Tuning

If the receive socket memory size is small, then sender will be able to send data equal to the receiver socket memory size. So we need to increase this value to a higher value,say 32MB. Likewise, we need the send socket memory size, also to be large, say 32MB.For a network with RTT value, 100ms and 10Gbps network, the value can be as higher as 64MB. If the RTT value is 50ms, then it can be increased to 128MB.

echo 'net.core.wmem_max=33554432' &gt;&gt; /etc/sysctl.conf
echo 'net.core.rmem_max=33554432' &gt;&gt; /etc/sysctl.conf

Next step is to increase the linux autotuning TCP buffer limit to 16MB. Here, we can set minimum amount of receive window size, which will be set to each TCP connection, even if the server is having a high load. The default value will be allocated against each TCP connection. Since we are employing the window scaling feature, the window size will grow dynamically till the maximum receive window size, set in bytes, 16777216. For a network with RTT value, 100ms and 10Gbps network, the value can be as higher as 32MB.If the RTT value is 50ms, then it can be increased to 128MB.

echo 'net.ipv4.tcp_rmem = 4096 87380 16777216' &gt;&gt; /etc/sysctl.conf
echo 'net.ipv4.tcp_wmem = 4096 65536 16777216' &gt;&gt; /etc/sysctl.conf

Also recommended to set net.ipv4.tcp_timestamps and net.ipv4.tcp_sack to 1, so that it can reduce the CPU load.

echo 'net.ipv4.tcp_timestamps = 1' &gt;&gt; /etc/sysctl.conf
echo 'net.ipv4.tcp_sack = 1' &gt;&gt; /etc/sysctl.conf

View congestion control algorithms

To view the available list of congestion control algorithms available for your machine, please execute the bwlo command. It is recommended to set htcp as the congestion control mechanism.

sysctl net.ipv4.tcp_available_congestion_control

To set htcp as your congestion control alogithm, please execute the below command,

sysctl -w net.ipv4.tcp_congestion_control=htcp

It is recommended to increase number of incoming connections backlog queue Sets the maximum number of packets, queued on the INPUT side, when the interface receives packets faster than kernel can process them.

echo 'net.core.netdev_max_backlog = 65536' &gt;&gt; /etc/sysctl.conf

View the performance tuning done

To save and reload, please execute the below command,

sysctl -p

We can use the tcpdump to view the changes on eth1, if eth1 is your NIC.

tcpdump -ni eth1

Attach a new datadrive to your Azure VM

Command to attach a datadrive in Azure Virtual Machine

Please execute the below command to attach a data drive of size 1TB to your Azure VM.

PS C:\&gt; Get-AzureVM "Your_Cloud_Service_Name" -Name "Your_VM_Name" | Add-AzureDataDisk -CreateNew -DiskSizeInGB 1024 -DiskLabel "My Data" -LUN1 | Update-AzureVM

Selecting Azure Subscription

Steps to select Azure Subscription

To select Azure subscription, as the first step,  Login to your computer and open up your Azure PowerShell and type,

Add-AzureAccount

Press Enter

Now you will see a popup, which will ask you to sign in to your Azure account. Enter your username(email address) and press ‘Continue’. Now you enter your password.

The below command will list all the subscriptions, that you have,

Get-AzureSubscription

To view the currently selected subscription, please execute the below command,

Get-AzureSubscription -Current

To select a subscription from the list of your subscriptions, please execute the below command,

Select-AzureSubscription –SubscriptionName ""

If you wish to set the selected the subscription as your default subscription, please add the switch -Default.

Select-AzureSubscription -SubscriptionName "" -Default

If you do not wish to have a default subscription, please execute the below command,

Set-AzureSubscription -NoDefaultSubscription

Reset Azure VM RDP Access

Reset Azure VM RDP Access

To reset Azure VM RDP Access, we need the VM access extension installed in an Azure VM. To check the status of the VM Access Extension installation status, please execute the below query after selecting your subscription,

$vm = Get-AzureVM -ServiceName "Cloud_Service_Name_Of_Your_VM" -Name "Your_VM_Name" write-host $vm.VM.ProvisionGuestAgent

If the write-host command returns the value as True, then the VM Access Agent is already installed. If it shows False, then we will have to install the VM Access Agent.

Here, we are assuming that, you have already installed the VM Access Agent in your VM.

To reset the RDP access to your Virtual machine, please run the below power shell query.

$cred=Get-Credential
Set-AzureVMAccessExtension -vm $vm -UserName $cred.GetNetworkCredential().Username `
-Password $cred.GetNetworkCredential().Password | Update-AzureVM
Set-AzureVMAccessExtension -vm $vm | Update-AzureVM

Commands to delete files older than X days

Delete files older than X days

To delete files older than X days, please execute the below commands. The below commands will remove files from a specific folder location in your machine\server. If you want to change the day, then just change value of 90 with your value,

find /directory/path/to/your/file -mindepth 1 -mtime +90 -delete

or

find /directory/path/to/your/file -type f -mtime +90 -exec rm {} \;

or

find /directory/path/to/your/file -mindepth 1 -type f -mtime +90 | xargs rm

Before executing the script,

delete files older than X days

After executing the script

delete files older than X days

 

Samba Slow – Oplock break failed for file

Oplock break failed – Slow copying of files in Samba

If you are facing sluggishness in copying files to your samba drive, there is a good chance that, it may be because the error “Oplock break failed for file” in your /var/log/syslog(Ubuntu) and /var/log/messages(In other linux distributions)

Jan 07 11:31:27 machinename smbd[3641]: Oplock break failed for file filename.txt

To resolve this, you may need to add the below entries to the [global] section of your smb.conf(/etc/samba/smb.conf)

[global]
.......
.......
kernel oplocks = no
nt acl support = no
strict locking = no

Then add the below entries under [your share name] section of smb.conf

[your share name]
........
........
oplocks = no
share modes = no
locking = no
acl check permissions = false
level2 oplocks = no
strict locking = no
blocking locks = no

Remote join client to WSUS – no psexec

Remote join a client to WSUS without using psexec

To remote join client  to WSUS , please execute the below two commands, without using the psexec and also without login to the client machine,

WMIC /node: process call create "cmd.exe /c GPUpdate.exe /force"
WMIC /node: process call create "cmd.exe /c wuauclt.exe /detectnow"

WSUS not showing cloned machines

WSUS not showing cloned machines

In case your WSUS not showing cloned machines, please verify the SusClientId & SusClientIdValidation keys in the cloned servers. Most probably, all the servers will be having the same key, so that only one server will be successful in establishing the connectivity towards your WSUS server. In that case, please execute the below script as a bat file in your cloned servers from an elevated command prompt.

REG Delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v SusClientId /f
REG Delete HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate /v SusClientIdValidation /f
REG Add HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate /v WUServer /t REG_SZ /d
http://:8530 /F

REG Add HKLM\SOFTWARE\Policies\Microsoft\Windows\WindowsUpdate /v WUStatusServer /t REG_SZ /d http:// /F

gpupdate
net stop wuauserv /y
net stop BITS /y
rd C:\WINDOWS\SoftwareDistribution /s /Q
del "c:\windows\windowsupdate.log"
regsvr32 WUAPI.DLL /s
regsvr32 WUAUENG.DLL /s
regsvr32 WUAUENG1.DLL /s
regsvr32 ATL.DLL /s
regsvr32 WUCLTUI.DLL /s
regsvr32 WUPS.DLL /s
regsvr32 WUPS2.DLL /s
regsvr32 WUWEB.DLL /s
regsvr32 msxml3.dll /s
net start wuauserv /y
wuauclt.exe /resetauthorization /detectnow

https://social.technet.microsoft.com/Forums/windowsserver/en-US/40b694e6-6586-47d3-8a68-dc11ecb7759b/wsus-cloned-server-2008-machines-and-registry-keys?forum=winserverwsus

Install Azure Resource Manager modules

To install Azure RM module. The first command will install the AzureRM module from the PowerShell Gallery and the second command will install the component modules of Azure Resource Manager.

PS C:\> Install-Module AzureRM
PS C:\> Install-AzureRM

To import the AzureRM.* modules, please run the below command,

PS C:\> Import-AzureRM

To import a single AzureRM module, please execute below command with module name

PS C:\> Import-Module AzureRM.Compute

To list all Azure Resource Manager modules installed,

PS C:\> Get-Module –ListAvailable AzureRM*

Remove broken packages in Ubuntu

Remove broken packages in Ubuntu

In order to remove broken packages in Ubuntu, please execute the below steps,

Step 1 : To update your package list.

sudo apt-get update

Then to clean up any partial packages, execute,

sudo apt-get autoclean

To clean up the apt cache, please execute,

sudo apt-get clean

To clean up any unneeded dependencies, please execute,

sudo apt-get autoremove

Finally to identify the broken package and to forcefully remove it, please execute the below command,

sudo dpkg –remove -force –force-remove-reinstreq package name

In case you are facing any issues, please try,

sudo dpkg –remove –force-remove-reinstreq package name

Also, remove any dependent packages, if any.

Alternatively, you may try the below as well,

sudo apt-get update -fix-missing

then execute,

sudo dpkg -configure -a

Once done, please exsecute,

sudo apt-get install -f