Traffic Control

HOWTO Maximise Download Speed via Outbound Traffic Shaping


The objective of this HOWTO is to explain the principle of and reasoning behind shaping outbound traffic (specifically TCP/IP) where by sending outbound non-payload TCP/IP acknowledgement (ACK) traffic to peers as quickly as possible with minimal delay intrinsically forces them to push inbound payload TCP/IP ACK traffic back to you at a faster rate resulting in download speeds exceeding the normal expectations for typical asymmetric broadband internet connections.

Who will find this HOWTO useful?

There has been a gradual increase in the number of hits to this page from search engines so to reduce the time you (the reader) need to spend determining if this HOWTO is going to be of any use to you, I have created a bullet point list below detailing common factors and scenarios which are relevant to this HOWTO. You should read, understand and implement the ideas presented in this HOWTO if any of the following hold true for you or your internet connection and you are looking at an elegant, simple solution to streamline your connection and improve your overall online experience.
  • You are adept in Linux and low level network configuration. If not, you are prepared to invest more of your time searching for and reading up on further literature that is relevant to the ideas and concepts described here-in.
  • Your internet connection is sluggish, laggy and experiencing symptoms of high packet latency and loss due to it being heavily congested.
  • You have an asymmetric internet connection with limited upstream capacity where the download speed far exceeds the upload speed.
  • You are experiencing consistently slow BitTorrent downloads that rarely saturate your available downstream bandwidth.
  • You are looking for a cheap, inexpensive fix to streamline your internet connection so it is running in an optimal way that meets your needs and requirements based on how you and others use it.
  • You share your internet connection with several others and typically have a couple of users who consistently abuse the connection (by continuously torrenting or leaving their online game of WOW active 24/7) thus flooding the connection and rendering it unusable, preventing you and others from performing and completing any online tasks in a respectable time frame.

Overview

There is an inherent issue with domestic broadband and that is simply the fact it is asymmetric. Commonly, such internet connections have a obscenely fast download speed with very little upload. Though, this is all good because, as a domestic (non-business) internet user, you are not supposed to be sending large quantities of outbound data to other users on the internet.

This poses a problem, however, especially when TCP/IP is concerned and I found this out recently having been upgraded by BT from a 576000/288000 bps (down/up) ADSL service to a MAX ADSL service giving me a data throughput speed of 1152000/448000 bps (down/up). I initially noticed the problem when downloading a 4 GB torrent. My upstream was saturated, and the download was not running any faster than my original 576000 bps broadband service would have done. It was at this point I decided to investigate how to introduce traffic shaping on to my connection in order to improve performance.

This short HOWTO will explain, in simplistic terms, what is required to gain maximum downstream throughput on your asymmetric broadband connection. It details why the performance of TCP/IP is horrific unless outbound traffic shaping is employed. There is also an example script included for Linux which defines some very simplistic but powerful routing policies to shape and prioritise outbound TCP/IP packets.

Theory

Why NOT shape inbound traffic?

For this entire performance boost to work, the secret, believe it or not, is NOT to shape inbound traffic. You can do this if you want and there may be instances where you need or have to do this. For example, if you have 20 PCs (in a small net café) sharing a single broadband connection, you will want to control how much bandwidth any single machine can use downstream. Or you may just want to control how much bandwidth any single particular protocol can use downstream. But if you are on a home network with only a few PCs with active users, you can safely ignore shaping downstream traffic.

The reason for ignoring downstream traffic is simple. You have absolutely no control over it and so it is best left as is. Anyone on the internet can send you anything they wish and the packets will be routed down your broadband connection and through to your network. There is NOTHING you can do to change this, except maybe obtaining your broadband service from an ISP which employs traffic shaping their side. Never the less, these downstream packets will still utilise valuable downstream bandwidth to reach your network and there is nothing that can be done to control the rate at which they reach you. We now use this fact our advantage!

Why shape outbound traffic?

Instead, we shape outbound traffic ONLY, specifically TCP/IP and this is the core reason for doing so. We observe that TCP has a number of categories of packet; packets that create connections (SYN), destroy connections (FIN) and acknowledgement packets (ACK) which determine the flow of traffic over an established TCP connection. The category of most interest to us in this HOWTO is the infamous ACK packet which also comes in two flavours; payload and non-payload ACKs. Payload ACKs are sent when transmitting useful data to your peers whilst non-payload ACKs are sent to acknowledge you received their data. So, as a peer yourself, you are interested in letting your remote peers know, as quickly as possible, that you received their data. Doing so will mean they send their next payload ACKs to you quicker, with minimal delay.

It is for this reason that if outbound traffic shaping is applied, you will not only see improved performance across all aspects of your asymmetric broadband connection but also allow the potential for complete saturation of your downstream bandwidth (especially during BitTorrent downloads) and thus intrinsically maximise your download speed!

Example Linux BASH Script employing Queue Disciplines (QDISCs)

Below is an example Linux BASH script which instructs the Linux Kernel to be especially clever when routing outbound traffic, specifically TCP/IP. It achieves this by utilising eight Linux Kernel Queue Disciplines (QDISCs) and placing traffic into an appropriate queue based on the priority it requires by filtering on packet protocol, port, flags and size. The eight queues, in order of importance, are as follows:
  1. Link-Critical Traffic
    Traffic that needs to take total and absolute priority over ANY other traffic that may be sent out over the network interface that is being shaped. This commonly applies to ARP (Address Resolution Protocol) traffic on ethernet interfaces. If the interface acquires its IP address via DHCP (common on cable broadband connections), it is best to place DHCP traffic into this class as well. In the case of PPP interfaces, this class becomes redundant and is not used.
  2. Time-Critical Traffic
    Traffic that needs to be routed out before any other type of traffic and should include TCP packets with SYN, SYN+ACK, ACK+FIN, ACK+RST and RST flags set (control packets) as well as all or some UDP traffic (depending on specific requirements). Also include non-payload TCP ACK packets (small ACK packets that contain little or no data with a size of 40 to 89 bytes).
  3. Critical Traffic
    Traffic that is critical but NOT time-critical. An example of such traffic might be an IPv6 tunnel running over IPv4. Also include TCP ACK packets that are slightly larger (90 to 159 bytes).
  4. High-Priority Interactive Traffic
    SSH and Telnet (TCP ports 22 and 23). Also include TCP ACK packets that are larger (160 to 249 bytes).
  5. Low-Priority Interactive Traffic
    HTTP and HTTPS (TCP ports 80 and 443). Also include TCP ACK packets that are larger (250 to 359 bytes).
  6. High-Priority Non-Interactive Traffic
    FTP and MySQL (TCP ports 20, 21 and 3306). Also include TCP ACK packets that are larger (360 to 489 bytes).
  7. Low-Priority Non-Interactive Traffic
    Email traffic such as SMTP and POP (TCP ports 25 and 110). Also include TCP ACK packets that are larger (490 to 639 bytes).
  8. Non-Critical Traffic
    All ICMP traffic. Also include TCP ACK packets that are larger (640 to 809 bytes).
Keep in mind that if you run a server on your connection, you may also want to give priority to outbound traffic relating to the services your server runs. You can do this by duplicating the appropriate iptables MANGLE table POSTROUTING chain rules below but using --sport rather than --dport. --sport is matching against all server related traffic (by looking at the source port number) whereas --dport is matching against all client related traffic (by looking at the destination port number). For example, if you want to shape outbound HTTP client traffic (traffic relating to websites you visit), use --dport 80. If you want to shape outbound HTTP server traffic (traffic relating to your own webserver), use --sport 80. Simple! :-)

So here is the script:
#!/bin/bash
#
# Download Maximiser (Outbound Traffic Shaper for Asymmetric Broadband)
#
# BEGIN CONFIG
# set path to IPTABLES binary
IPT=/sbin/iptables
# set path to TC binary
TC=/sbin/tc
# set the broadband network interface to be shaped
IF=ppp0
# set the IPTABLES chain for this script
chain=SHAPE
# define the maximum bits per second the broadband
# upstream operates at without queuing (IMPORTANT!)
MAX=350000
# END CONFIG
function shape {
# setup a new chain for shaping
$IPT -t mangle -N $chain
# route outbound traffic into appropriate CLASSes
#
# link-critical traffic (normally left empty unless interface acquires IP via DHCP)
#
CLA=10
# --- if interface acquires IP via DHCP, uncomment the line below
# $IPT -t mangle -I $chain -p udp --sport 68 --dport 67 -j CLASSIFY --set-class 1:$CLA
# --- if root user needs to be able to send out ping requests at highest priority, uncomment the line below
# $IPT -t mangle -I $chain -p icmp --icmp-type echo-request -m owner --uid-owner 0 -j CLASSIFY --set-class 1:$CLA
#
# time-critical traffic
#
CLA=20
# --- place DNS lookup traffic in this class (fast browsing requires fast DNS resolving)
$IPT -t mangle -I $chain -p udp --dport 53 -j CLASSIFY --set-class 1:$CLA
# --- also place all standard TCP control packets in this class
$IPT -t mangle -I $chain -p tcp --tcp-flags FIN,SYN,RST,ACK FIN,ACK -j CLASSIFY --set-class 1:$CLA
$IPT -t mangle -I $chain -p tcp --tcp-flags FIN,SYN,RST,ACK SYN,ACK -j CLASSIFY --set-class 1:$CLA
$IPT -t mangle -I $chain -p tcp --tcp-flags FIN,SYN,RST,ACK RST,ACK -j CLASSIFY --set-class 1:$CLA
$IPT -t mangle -I $chain -p tcp --tcp-flags FIN,SYN,RST,ACK RST -j CLASSIFY --set-class 1:$CLA
$IPT -t mangle -I $chain -p tcp --syn -j CLASSIFY --set-class 1:$CLA
#
# critical traffic
#
CLA=30
# --- IPv6 tunnel traffic (IP protocol 41) goes in this class
$IPT -t mangle -I $chain -p ipv6 -j CLASSIFY --set-class 1:$CLA
#
# high-priority interactive traffic
#
CLA=40
# --- inbound and outbound SSH goes in this class
$IPT -t mangle -I $chain -p tcp --dport 22 -j CLASSIFY --set-class 1:$CLA
$IPT -t mangle -I $chain -p tcp --sport 22 -j CLASSIFY --set-class 1:$CLA
#
# low-priority interactive traffic
#
CLA=50
# --- inbound and outbound HTTP and HTTPS goes in this class
$IPT -t mangle -I $chain -p tcp -m multiport --dport 80,443 -j CLASSIFY --set-class 1:$CLA
$IPT -t mangle -I $chain -p tcp -m multiport --sport 80,443 -j CLASSIFY --set-class 1:$CLA
#
# high-priority non-interactive traffic
#
CLA=60
# --- inbound FTP-data, FTP and MySQL goes in this class
$IPT -t mangle -I $chain -p tcp -m multiport --dport 20,21,3306 -j CLASSIFY --set-class 1:$CLA
#
# low-priority non-interactive traffic
#
CLA=70
# --- inbound and outbound SMTP and POP goes in this class
$IPT -t mangle -I $chain -p tcp -m multiport --dport 25,110 -j CLASSIFY --set-class 1:$CLA
$IPT -t mangle -I $chain -p tcp -m multiport --sport 25,110 -j CLASSIFY --set-class 1:$CLA
#
# non-critical traffic (default class)
#
CLA=80
# --- catch-all rule to place all other traffic in lowest priority class
$IPT -t mangle -I $chain -j CLASSIFY --set-class 1:$CLA
# setup QDISCs and CLASSes and route packets according to size
$TC qdisc add dev $IF root handle 1: htb default 10
$TC class add dev $IF parent 1: classid 1:1 htb rate ${MAX}bit
for i in $(seq 8)
do
$TC class add dev $IF parent 1:1 classid 1:$[$i*10] htb rate $[$MAX/8]bit ceil ${MAX}bit prio $[$i-1]
$TC qdisc add dev $IF parent 1:$[$i*10] handle $[$i*10]: sfq perturb 10
[ $i -gt 1 ] && $IPT -t mangle -I $chain -p tcp --tcp-flags FIN,SYN,RST,ACK ACK -m length --length $[$i*$i*10]:$[($i+1)*($i+1)*10-1] -j CLASSIFY --set-class 1:$[$i*10]
[ $i -gt 2 ] && $IPT -t mangle -I $chain -p udp -m length --length $[($i-1)*($i-1)*10]:$[$i*$i*10-1] -j CLASSIFY --set-class 1:$[$i*10]
done
# start pushing traffic into this new chain
$IPT -t mangle -I POSTROUTING -o $IF -j $chain
}
function flush {
$TC qdisc del dev $IF root && $IPT -t mangle -D POSTROUTING -o $IF -j $chain && $IPT -t mangle -F $chain && $IPT -t mangle -X $chain
}
case "$1" in
on)
echo "Enabling Outbound Traffic Shaping on $IF"
flush; shape
;;
off)
echo "Disabling Outbound Traffic Shaping on $IF"
flush
;;
show)
$TC -s class show dev $IF
;;
*)
echo "Usage: $0 {on|off|show}"
;;
esac

After running this script, it will have inserted a reference to its newly created SHAPE chain into the iptables MANGLE table POSTROUTING chain, and populated its own SHAPE chain with the following entries:
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
SHAPE      all  --  anywhere             anywhere
...
Chain SHAPE (1 references)
target     prot opt source               destination
CLASSIFY   udp  --  anywhere             anywhere            length 490:639 CLASSIFY set 1:80
CLASSIFY   tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/ACK length 640:809 CLASSIFY set 1:80
CLASSIFY   udp  --  anywhere             anywhere            length 360:489 CLASSIFY set 1:70
CLASSIFY   tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/ACK length 490:639 CLASSIFY set 1:70
CLASSIFY   udp  --  anywhere             anywhere            length 250:359 CLASSIFY set 1:60
CLASSIFY   tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/ACK length 360:489 CLASSIFY set 1:60
CLASSIFY   udp  --  anywhere             anywhere            length 160:249 CLASSIFY set 1:50
CLASSIFY   tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/ACK length 250:359 CLASSIFY set 1:50
CLASSIFY   udp  --  anywhere             anywhere            length 90:159 CLASSIFY set 1:40
CLASSIFY   tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/ACK length 160:249 CLASSIFY set 1:40
CLASSIFY   udp  --  anywhere             anywhere            length 40:89 CLASSIFY set 1:30
CLASSIFY   tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/ACK length 90:159 CLASSIFY set 1:30
CLASSIFY   tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/ACK length 40:89 CLASSIFY set 1:20
CLASSIFY   all  --  anywhere             anywhere            CLASSIFY set 1:80
CLASSIFY   tcp  --  anywhere             anywhere            multiport sports smtp,pop3 CLASSIFY set 1:70
CLASSIFY   tcp  --  anywhere             anywhere            multiport dports smtp,pop3 CLASSIFY set 1:70
CLASSIFY   tcp  --  anywhere             anywhere            multiport dports ftp-data,ftp,mysql CLASSIFY set 1:60
CLASSIFY   tcp  --  anywhere             anywhere            multiport sports http,https CLASSIFY set 1:50
CLASSIFY   tcp  --  anywhere             anywhere            multiport dports http,https CLASSIFY set 1:50
CLASSIFY   tcp  --  anywhere             anywhere            tcp spt:ssh CLASSIFY set 1:40
CLASSIFY   tcp  --  anywhere             anywhere            tcp dpt:ssh CLASSIFY set 1:40
CLASSIFY   ipv6 --  anywhere             anywhere            CLASSIFY set 1:30
CLASSIFY   tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/SYN CLASSIFY set 1:20
CLASSIFY   tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/RST CLASSIFY set 1:20
CLASSIFY   tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/RST,ACK CLASSIFY set 1:20
CLASSIFY   tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/SYN,ACK CLASSIFY set 1:20
CLASSIFY   tcp  --  anywhere             anywhere            tcp flags:FIN,SYN,RST,ACK/FIN,ACK CLASSIFY set 1:20
CLASSIFY   udp  --  anywhere             anywhere            udp dpt:domain CLASSIFY set 1:20
CLASSIFY   udp  --  anywhere             anywhere            udp spt:bootpc dpt:bootps CLASSIFY set 1:10

Summary

Going against popular intuition, the script populates its SHAPE chain before creating the qdisc and classes as this was the only way I could achieve the packet matching I wanted, in the order I wanted, whilst still keeping a default catch-all traffic rule for class 80. It gives total priority to all TCP packets with SYN, SYN+ACK, ACK+FIN, ACK+RST and RST flags set (control packets) as well as DNS lookup traffic (fast browsing requires fast DNS resolving). It then continues mangling other types of traffic into lower priority classes (modify this section to suit your requirements). Lastly, the final packet of importance is the infamous TCP ACK. This is dealt with via the same loop which creates our qdiscs and classes. The loop sets up iptables MANGLE table POSTROUTING rules so that, depending on the size or length of the TCP ACK packet, we determine its fate and what class it ends up in for postrouting, as follows:
... tcp flags:FIN,SYN,RST,ACK/ACK length 40:89 CLASSIFY set 1:20
... tcp flags:FIN,SYN,RST,ACK/ACK length 90:159 CLASSIFY set 1:30
... tcp flags:FIN,SYN,RST,ACK/ACK length 160:249 CLASSIFY set 1:40
... tcp flags:FIN,SYN,RST,ACK/ACK length 250:359 CLASSIFY set 1:50
... tcp flags:FIN,SYN,RST,ACK/ACK length 360:489 CLASSIFY set 1:60
... tcp flags:FIN,SYN,RST,ACK/ACK length 490:639 CLASSIFY set 1:70
... tcp flags:FIN,SYN,RST,ACK/ACK length 640:809 CLASSIFY set 1:80
In this same loop, we also introduce a similar setup for UDP traffic (optional):
... udp ... length 40:89 CLASSIFY set 1:30
... udp ... length 90:159 CLASSIFY set 1:40
... udp ... length 160:249 CLASSIFY set 1:50
... udp ... length 250:359 CLASSIFY set 1:60
... udp ... length 360:489 CLASSIFY set 1:70
... udp ... length 490:639 CLASSIFY set 1:80
The generalisation of the above is quite effective since we can assume that the larger the size of a TCP ACK (or UDP) packet, the more likely it is to contain payload data. The more data it has, the further it sinks in priority of being sent out our broadband connection first leaving all the real control ACK packets (with little or no payload) to exit as quickly as possible. Note the catch-all rule at the end which forces ANY unmatched packets to be placed into the lowest priority non-critical (80) class. This is required as our default class for unclassified traffic on the root QDISC is link-critical (10).

Configuration

Setting an optimal value for MAX
The most important part in the above script, before you even consider customising it, is the MAX constant in the configuration section at the beginning. This needs to be set correctly and as close to your broadband connection's upload speed as possible, but NEVER exceeding it. If this number exceeds your capable upload speed, packets will start being queued at your modem or router which completely destroys any traffic shaping Linux applied previously to your outbound traffic. I can't stress this point enough and it is imperative this value is set correctly.

It can be a difference of only 10 kbits that decides between you downloading traffic (torrents) at 50% of your capable download speed rather than 100%. For example, on my 448000 bps upload rate, I found MAX=360000 destroys the speed of torrent downloads but MAX=350000 works perfectly. You can see this if you look at the real MRTG graphics in the DM logo at the top of this page. If the outbound is saturated (thus destroying traffic shaping), the download speed is killed (left side of the graphic). If you pull the upload ceiling back slightly so it is just below your maximum upload rate, and then shape it, the downstream is maximised fully (right side of the graphic). Experimentation will be necessary to locate the optimal value of MAX and the best way of doing this is by downloading an extremely fast torrent adjusting and re-running the script with varied values of MAX until you find one which doesn't kill your download speed.
The sum of the minimum speed of each class, across all classes, must be less than or equal to MAX
This is important. In the above script, 8 classes are created each with 43750 bits of minimum outbound bandwidth allocated to them, making the total 350000 bits (or if MAX is adjusted to a different value, each class will have one eighth of MAX minimum bandwidth allocated to it). This is desirable when shaping via HTB (Hierarchical Token Bucket) since if the sum was larger than MAX, some lower priority packets may start being dropped completely which is really far from ideal. And since we are only interested in prioritising certain types of outbound packets rather than controlling the bandwidth they utilise, and the fact HTB qdiscs and classes can borrow unused bandwidth from each other as and when required, this is the simplest configuration to go with. If it is a necessity to delegate more outbound bandwidth to particular classes of traffic over others, rewrite this section of the script ensuring that the sum does not exceed the value of MAX (for example, you could have 80000, 70000, 60000, 45000, 35000, 30000, 20000 and 10000, the total of which is still 350000).
Does this script need to be tweaked so shaping will operate smoothly on an ETHERNET (eth0) network interface rather than a PPP (ppp0) network interface?
The short answer to this question is NO. If we are shaping an ethernet interface directly (that is, IF=eth0 rather than IF=ppp0 in the above script), we should be fine because our default class for all unclassified traffic (which includes ARP) is 10, due to this line:
tc qdisc add dev $IF root handle 1: htb default 10
This instructs Linux to route all traffic that has NOT been classified (ethernet ARP traffic) into the link-critical (10) class rather than the non-critical (80) class.

The reason for doing this is as follows. ARP (the Address Resolution Protocol) can be classed as link-critical traffic, which is of a higher priority than even time-critical. Link-critical traffic is traffic that sits below the IP layer, and is not normally traffic that concerns us. In terms of PPP, if you run TCPDUMP on a PPP interface, the ONLY traffic you will ever see is IP. However, if the PPP session is running over an ethernet interface, and you run TCPDUMP on that ethernet interface, you will see that the IP traffic is encapsulated inside the PPP packets which run over the ethernet interface, and you will also see link-layer PPP traffic (PING? PONG!) packets that are directly related to the PPP session. These exist in order to periodically confirm that the PPP session is responsive, active and still functioning, and that both peers are still alive.

The same PING? PONG! traffic occurs on ethernet and is called ARP. If ARP traffic is placed in a low priority class, there is then the risk of this traffic being filtered and potentially dropped by the Linux traffic shaping subsystem because the shaping policies will have been applied DIRECTLY onto the ethernet interface. This is bad news as link-critical traffic MUST NOT be dropped at all, EVER. Having investigated this further, there doesn't appear to be any way of making IPTABLES match ARP traffic, simply because IPTABLES is designed to help us only filter and manage IP traffic, not protocols residing on the network layer below this, such as ARP. But, the shaping policies that Linux uses will still see this ARP traffic as unclassifed traffic and, as such, place ARP packets into what ever class is the default one, which in the above script, is link-critical (10) and NOT non-critical (80), which is precisely what we require.

Conclusion

This is an amazingly simple and elegant traffic performance fix for asymmetric broadband connections. If you apply this script into a broadband Linux traffic shaping router, you will see increased speeds, not just in how fast things load up, but also the freedom obtained from accomplishing interactive tasks quickly, such as surfing the web, even with your upstream being completely saturated with outbound data (BitTorrent uploads). I now no longer see outbound saturated broadband connections as a bad thing! :-)

Acknowledgements

I would like to thank Shane Chen for QDISC examples located on his traffic shaping page. The ADSL Bandwidth Management HOWTO has also been of assistance.

In The Wild

I would like to mention that negge on the MikroTik forum has successfully translated the principles outlined in this HOWTO onto his MikroTek router, with astounding results. In his forum post, negge writes:

"With the current settings, if I'm seeding at full speed and somebody starts downloading something over FTP, the FTP transfer gets all upload bandwidth while the torrents almost stop. If I'm connected with SSH, there's absolutely no lag when I'm writing in the console. When surfing the web, DNS lookups happen instantly and pages load just like if I wasn't seeding at all."

Awesomeness! :-)

 

 

Thanks  Ref :  http://phix.me/dm/

Shaping Script

 


 
 
Here's a new script I've worked on. I think it's alot better than the old one so hope this is of some educational use to somebody.
			#!/bin/bash
			# htbshaper (no imq), htbshaper on (with imq), htbshaper off (shutdown shaping)
			# Internal Device
			INTDEV=eth0
			# Wireless Device
			ADMDEV=eth1
			# Internet Device
			EXTDEV=eth3
			# tc qdisc ... dev dev ( parent classid | root) [ handle major: ] htb [ default minor-id ]
			# tc class ... dev dev parent major:[minor] [ classid major:minor ] htb rate rate [ ceil rate ] burst bytes [ cburst bytes ] [ prio priority ]
			tc qdisc del dev eth0 root 2>/dev/null
			tc qdisc del dev eth1 root 2>/dev/null
			tc qdisc del dev eth3 root 2>/dev/null
			tc qdisc del dev imq0 root 2>/dev/null
			ip link set imq0 down 2>/dev/null
			rmmod imq 2>/dev/null
			iptables -F FORWARD -t mangle
			if [ "$1" = "off" ]; then 
			exit
			fi
			IMQON="$1"
			if [ "$IMQON" ]; then
			        insmod imq numdevs=1
			        ip link set imq0 up 2>/dev/null
			        echo "IMQ Devices On!"
			fi
			# ETH0 - LAN
			        tc qdisc add dev $INTDEV root handle 1: htb default 30
			        tc class add dev $INTDEV parent 1:  classid 1:1   htb rate 1900kbit burst 2500b
			        tc class add dev $INTDEV parent 1:1 classid 1:10  htb rate  100kbit ceil  300kbit burst 2500 # SSH
			        tc class add dev $INTDEV parent 1:1 classid 1:20  htb rate  200kbit ceil 1000kbit burst 2500 # SMTP
			        tc class add dev $INTDEV parent 1:1 classid 1:30  htb rate  500kbit ceil 1000kbit burst 2500 # VPN
			        tc class add dev $INTDEV parent 1:1 classid 1:40  htb rate  100kbit ceil  150kbit burst 2500 # DNS
			        #tc class add dev $INTDEV parent 1:1 classid 1:50  htb rate  200kbit ceil 1900kbit burst 2500 # HTTP(S)
			        tc class add dev $INTDEV parent 1:1 classid 1:60  htb rate  200kbit ceil  500kbit burst 2500 # FTP
			        tc class add dev $INTDEV parent 1:1 classid 1:70  htb rate  100kbit ceil 1000kbit burst 2500 # WKS
			        tc class add dev $INTDEV parent 1:1 classid 1:100 htb rate  200kbit ceil  500kbit burst 2500 # Other
			        tc qdisc add dev $INTDEV parent 1:10  handle 110: sfq perturb 10
			tc qdisc add dev $INTDEV parent 1:20  handle 120: sfq perturb 10
			        tc qdisc add dev $INTDEV parent 1:30  handle 130: sfq perturb 10
			        tc qdisc add dev $INTDEV parent 1:40  handle 140: sfq perturb 10
			        #tc qdisc add dev $INTDEV parent 1:50  handle 150: sfq perturb 10
			        tc qdisc add dev $INTDEV parent 1:60  handle 160: sfq perturb 10
			        tc qdisc add dev $INTDEV parent 1:70  handle 170: sfq perturb 10
			        tc qdisc add dev $INTDEV parent 1:100 handle 100: sfq perturb 10
			        tc filter add dev $INTDEV parent 1:0 prio 0 protocol ip handle 110 fw flowid 1:10
			        tc filter add dev $INTDEV parent 1:0 prio 0 protocol ip handle 120 fw flowid 1:20
			        tc filter add dev $INTDEV parent 1:0 prio 0 protocol ip handle 130 fw flowid 1:30
			        tc filter add dev $INTDEV parent 1:0 prio 0 protocol ip handle 140 fw flowid 1:40
			        #tc filter add dev $INTDEV parent 1:0 prio 0 protocol ip handle 150 fw flowid 1:50
			        tc filter add dev $INTDEV parent 1:0 prio 0 protocol ip handle 160 fw flowid 1:60
			        tc filter add dev $INTDEV parent 1:0 prio 0 protocol ip handle 170 fw flowid 1:70
			        tc filter add dev $INTDEV parent 1:0 prio 0 protocol ip handle 100 fw flowid 1:100
			        if [ "$IMQON" ]; then
			                tc class add dev imq0 parent 1:1 classid 1:50  htb rate  200kbit ceil 1900kbit burst 2500 # HTTP(S)
			                tc qdisc add dev imq0 parent 1:50  handle 150: sfq perturb 10
			                tc filter add dev imq0 parent 1:0 prio 0 protocol ip handle 150 fw flowid 1:50
			                #tc filter add dev imq0 parent 4:0 prio 0 protocol ip handle 160 fw flowid 4:60
			                #tc filter add dev imq0 parent 4:0 prio 0 protocol ip handle 170 fw flowid 4:70
			                #tc filter add dev imq0 parent 4:0 prio 0 protocol ip handle 100 fw flowid 4:100
			                iptables -t mangle -I PREROUTING -i $EXTDEV -j IMQ --todev 0
			                tc filter add dev imq0 parent 1:0 protocol ip prio 1 u32 match ip sport 80 0xffff flowid 1:50
			        fi
			        # LAN - [ IN ] SERVERS  # 50kbits -> 100kbit
			        iptables -A FORWARD -t mangle -i $EXTDEV -o $INTDEV -p tcp --sport 22 -j MARK --set-mark
			110 # SSH
			        iptables -A FORWARD -t mangle -i $EXTDEV -o $INTDEV -d 10.10.0.60 -p tcp --dport 25 -j MARK --set-mark 120 # SMTP
			        iptables -A FORWARD -t mangle -i $EXTDEV -o $INTDEV -d 10.10.0.60 -p tcp --dport 1723 -j MARK --set-mark 130 # VPN
			        iptables -A FORWARD -t mangle -i $EXTDEV -o $INTDEV -d 10.10.0.60 -p 47 -j MARK --set-mark 130 # VPN
			        iptables -A FORWARD -t mangle -i $EXTDEV -o $INTDEV -d 10.10.0.60 -p udp --dport 53 -j MARK --set-mark 140 # DNS
			        iptables -A PREROUTING -t nat   -t mangle -i $EXTDEV -p tcp --sport 80 -j MARK --set-mark 150 # HTTP
			        iptables -A FORWARD -t mangle -i $EXTDEV -o $INTDEV -p tcp --sport 443 -j MARK --set-mark 150 # HTTPS
			        iptables -A FORWARD -t mangle -i $EXTDEV -o $INTDEV -m state --state RELATED -j MARK --set-mark 160 # FTP
			        # LAN - [ IN ] WORKSTATIONS # 200kbit ->  500kbit
			        iptables -A FORWARD -t mangle -i $EXTDEV -o $INTDEV -d 10.10.0.64/26 -j MARK --set-mark 170
			# ETH1 - Wireless
			        tc qdisc add dev $ADMDEV root handle 2: htb default 10
			        tc class add dev $ADMDEV parent 2:  classid 2:1  htb rate 1000kbit burst 2500
			        tc class add dev $ADMDEV parent 2:1 classid 2:10 htb rate  200kbit ceil 1000kbit burst 2500
			        tc qdisc add dev $ADMDEV parent 2:10  handle 210: sfq perturb 10
			        tc filter add dev $ADMDEV parent 2:0 prio 0 protocol ip handle 210 fw flowid 2:10
			        # WIRELESS - [ IN ] WORKSTATIONS # 200kbit -> 1000kbit
			# ETH3 - Internet
			        tc qdisc add dev $EXTDEV root handle 3: htb # default 20
			        tc class add dev $EXTDEV parent 3:  classid 3:1   htb rate 1800kbit burst 2500
			        tc class add dev $EXTDEV parent 3:1 classid 3:10  htb rate  200kbit ceil 1000kbit burst 2500    # SMTP
			        tc class add dev $EXTDEV parent 3:1 classid 3:20  htb rate  500kbit ceil 1500kbit burst 2500    # VPN
			        tc class add dev $EXTDEV parent 3:1 classid 3:30  htb rate  100kbit ceil  150kbit burst 2500    # DNS
			        tc class add dev $EXTDEV parent 3:1 classid 3:40  htb rate  300kbit ceil 1500kbit burst 2500    # HTTP(S)
			        tc class add dev $EXTDEV parent 3:1 classid 3:50  htb rate  200kbit ceil  800kbit burst 2500    # FTP
			        tc class add dev $EXTDEV parent 3:1 classid 3:100 htb rate  200kbit ceil  200kbit burst 2500    # Other
			        tc qdisc add dev $EXTDEV parent 3:10  handle 210: sfq perturb 10
			        tc qdisc add dev $EXTDEV parent 3:20  handle 220: sfq perturb 10
			        tc qdisc add dev $EXTDEV parent 3:30  handle 230: sfq perturb 10
			        tc qdisc add dev $EXTDEV parent 3:40  handle 240: sfq perturb 10
			        tc qdisc add dev $EXTDEV parent 3:50  handle 250: sfq perturb 10
			        tc qdisc add dev $EXTDEV parent 3:100 handle 200: sfq perturb 10
			        tc filter add dev $EXTDEV parent 3:0 prio 0 protocol ip handle 310 fw flowid 3:10
			        tc filter add dev $EXTDEV parent 3:0 prio 0 protocol ip handle 320 fw flowid 3:20
			        tc filter add dev $EXTDEV parent 3:0 prio 0 protocol ip handle 330 fw flowid 3:30
			        tc filter add dev $EXTDEV parent 3:0 prio 0 protocol ip handle 340 fw flowid 3:40
			        tc filter add dev $EXTDEV parent 3:0 prio 0 protocol ip handle 350 fw flowid 3:50
			        tc filter add dev $EXTDEV parent 3:0 prio 0 protocol ip handle 300 fw flowid 3:100
			# LAN - [ OUT ] SERVERS  # 50kbits -> 100kbit
			        iptables -A FORWARD -t mangle -i $INTDEV -o $EXTDEV -p tcp --dport 25 -j MARK --set-mark 310 # SMTP
			        iptables -A FORWARD -t mangle -i $INTDEV -o $EXTDEV -p tcp --sport 110 -j MARK --set-mark 310 # POP3
			        iptables -A FORWARD -t mangle -i $INTDEV -o $EXTDEV -p 47 -j MARK --set-mark 320 # VPN
			        iptables -A FORWARD -t mangle -i $INTDEV -o $EXTDEV -s 10.10.0.60 -p udp --sport 53 -j MARK --set-mark 330 # DNS
			        iptables -A FORWARD -t mangle -i $INTDEV -o $EXTDEV -p tcp --sport 80 -j MARK --set-mark 340 # HTTP
			        iptables -A FORWARD -t mangle -i $INTDEV -o $EXTDEV -p tcp --sport 443 -j MARK --set-mark 340 # HTTPS
			        iptables -A FORWARD -t mangle -i $INTDEV -o $EXTDEV -m state --state RELATED -j MARK --set-mark 350 # FTP
			        # LAN - [ OUT ] WORKSTATIONS # 200kbit ->  500kbit
			        iptables -A FORWARD -t mangle -i $INTDEV -o $EXTDEV -m mark --mark 0 -j MARK --set-mark 300
			        # WIRELESS - [ OUT ] WORKSTATIONS # 200kbit ->  500kbit
			        iptables -A FORWARD -t mangle -i $ADMDEV -o $EXTDEV -j MARK --set-mark 300
			

htpwondershaper

#!/bin/bash -x
## HTBWondershaper script to do traffic shaping and policying with HTB packet scheduler. Version 1.2.1
#By Hans-Cees at hanscees<-nospammlumberjack->@hanscees.com. I already apologise for typos in documentation here.
##script can be called as:
#	HTBWondershaper :implements the defined TC classes and filters 
#	HTBWondershaper status :shows counters
#	HTBWondershaper stop   :flushes TC classes and iptables -t mangle FORWARD and OUTPUT chains
#before using it you should adjust variables DOWNLINK, CEIL EXTDEV AND LANDEV
#you can just make this script executable (chmod +x HTBWondershaper) and run it. It will make latency smaller
#and gives traffic with tos-minimum delay priority, as well as small packets (syns, acks and the like)
#This script implements traffic shaping: it filters outgoing traffic into classes and then sets limits/prioritises 
# these classes. Each class get a lower and upper bandtwith limit. Each class also has a priority, and if the prio
# is lower, the traffic goes first.
#Shaping: The process of delaying packets before they go out to make traffic confirm to a 
#configured maximum rate. Shaping is performed on egress (outgoing traffic). Colloquially, dropping packets 
#to slow traffic down is also often called Shaping.
#Policing: Delaying or dropping packets in order to make traffic stay below a configured bandwidth. 
#In Linux, policing can only drop a packet and not delay it - there is no 'ingress queue'.
# Strength of the tactics of this script is:
#	- it uses HTB filter: this guarantees minimum bandtwith per class and devides surplus bandtwith evenly
#	- it uses "iptables marks" to filter traffic into classes
#		This means you can use the rather simple iptables language to put specific traffic into TC classes
#		rather than the complex non-documented TC stuff. 
#########################################
###example (given classes setup below):
#for instance: pin network 192.168.2.0 behind this server down to 50 Kb/s UPload:
#set root class on external device
#	tc qdisc add dev $EXTDEV root handle 1: htb default 13
#	tc class add dev $EXTDEV parent 1: classid 1:1 htb rate 10000kbit ceil 10000kbit
#make class with 50kb ceiling:
#	tc class add dev $EXTDEV parent 1:1 classid 1:14 htb rate 10kbit ceil 50kbit prio 5
#	tc qdisc add dev $EXTDEV parent 1:14 handle 140: sfq perturb 10
#filter traffic mark 5 into it (set handle to 5):
#	tc filter add dev $EXTDEV parent 1:0 protocol ip prio 5 handle 5 fw classid 1:14
#give specific traffic mark 5
#	iptables -t mangle -I FORWARD -s 192.168.2.0/24 -j RETURN
#	iptables -t mangle -I FORWARD -s 192.168.2.0/24 -o $EXTDEV -j MARK --set-mark 0x5
#For download speed, for instance: pin network 192.168.2.0 behind this server down to 50 Kb/s DOWNload (=~ 6kB)
#(so use internal interface for classes):
#	tc qdisc add dev $LANDEV root handle 1: htb default 13
#	tc class add dev $LANDEV parent 1: classid 1:1 htb rate 10000kbit ceil 10000kbit
#make class with 50kb ceiling:
#	tc class add dev $LANDEV parent 1:1 classid 1:14 htb rate 10kbit ceil 50kbit prio 5
#	tc qdisc add dev $LANDEV parent 1:14 handle 140: sfq perturb 10
#filter traffic mark 9 into it (set handle to 9):
#	tc filter add dev $LANDEV parent 1:0 protocol ip prio 5 handle 9 fw classid 1:14
#give specific traffic mark 5
#	iptables -t mangle -I FORWARD -d 192.168.2.0/24 -j RETURN
#	iptables -t mangle -I FORWARD -d 192.168.2.0/24 -o $LANDEV -j MARK --set-mark 0x9
# watch it:
#tc -s qdisc show dev $LANDEV
#tc -s class show dev $LANDEV
#tc -s filter show dev $LANDEV
#delete it again:
#tc qdisc del dev $LANDEV root    2> /dev/null > /dev/null
########## end Example
####################################################
######### target for this script (which you will probably want to alter)
#this script is written for an adsl line where a local web-server generates too much 
#traffic. Therefore the rules used are aimed to prioritise lan http/other traffic from 
#behind the server above the webservers upload to internet.
#if you have a different scenario please adjust the iptables rules.
#    internet <------> server with busy webserver <--------> lan
#             <------------------priority 1 ------------------
#             <--------------- priority 5 
#                               webserver (src tcp 80)
#### Further reading ################################
#This script is based on this howto: Better bandwidth
#http://lartc.org/howto/lartc.cookbook.fullnat.intro.html
#part of linux howto bandwidth:
#http://lartc.org/howto/index.html
#see also "man tc" see http://lartc.org/manpages/
#a much more deep-going script is here: http://digriz.org.uk/jdg-qos-script/
#Explanation of htb
#http://lartc.org/howto/lartc.qdisc.classful.html#AEN1072
#htb home
#http://luxik.cdi.cz/~devik/qos/htb/
#a good other similar script http://www.linuxinfor.com/english/ADSL-Bandwidth-Management/implementation.html
# and another here: http://www.freenet.org.nz/python/pyshaper/pyshaper.conf.html
#### END Further reading ###########################
#Size of downloads. Used for igress filter all below (2mb).
DOWNLINK=1950
#Adjust CEIL to 75% of your upstream bandwith limit by now(1mb line).
#tinkering with ceil and ceil-related variables in tables is crucial for success.
#with 80% of upload, my line becomes clogged. With 70% all goes well.
CEIL=950
## On with the script:
##set devices, extdev is your internet/outside device. Lan the inside one.
EXTDEV=eth1
LANDEV=eth0
##
if [ "$1" = "status" ]
#see http://www.docum.org/docum.org/faq/cache/33.html
then
echo "[qdisc]"
tc -s qdisc show dev $EXTDEV
echo "[class]"
tc -s class show dev $EXTDEV
echo "[filter]"
tc -s filter show dev $EXTDEV
echo "[iptables]"
iptables -t mangle -L FORWARD -v 2> /dev/null
iptables -t mangle -L OUTPUT -v 2> /dev/null
exit
fi
##before either stopping or running the script, flush things first
# Reset everything to a known state (cleared)
tc qdisc del dev $EXTDEV root    2> /dev/null > /dev/null
tc qdisc del dev $EXTDEV ingress 2> /dev/null > /dev/null
/sbin/iptables -F -t mangle
if [ "$1" = "stop" ] 
then 
echo "Shaping removed on $EXTDEV."
exit
fi
# Make TC CLASSES
#First we set up some qdiscs in which we will classify the traffic. 
#We create a htb qdisc with 64classes with ascending priority. Then we have classes that will 
#always get allocated rate, but can use the unused bandwidth that other classes don't need. 
#Recall that classes with higher priority ( i.e with a lower prio number ) will get excess 
#of bandwith allocated first. Our connection is 2Mb down 1000kbits/s up Adsl. 
#I use 950kbit/s as ceil rate just because it's the higher I can set it before 
#latency starts to grow, due to buffer filling in whatever place between us and remote 
#hosts. This parameter should be timed experimentally, raising and lowering it while 
#observing latency between some near hosts.
#if you get this error: "kernel: HTB: quantum of class 10012 is small" see:
#http://www.docum.org/docum.org/faq/cache/31.html
#When a packet with size > quantum is sent, it will be sent and an error that the quantum is too small will be logged. But there is no pay back. The WRR scheduler is faster then the DRR scheduler. So make sure quantum is bigger then the default packet size. For 15 kbyte/s and default r2q, quantum is 1500 and this is exactly the maximum packet size. If you want to tune htb for rates smaller then 15 kbyte/s, you can manually set the r2q and/or quantum.
tc qdisc add dev $EXTDEV root handle 1: htb default 13
tc class add dev $EXTDEV parent 1: classid 1:1 htb rate ${CEIL}kbit ceil ${CEIL}kbit
tc class add dev $EXTDEV parent 1:1 classid 1:10 htb rate 150kbit ceil 300kbit prio 0
tc class add dev $EXTDEV parent 1:1 classid 1:11 htb rate 200kbit ceil ${CEIL}kbit prio 1
tc class add dev $EXTDEV parent 1:1 classid 1:12 htb rate 50kbit ceil ${CEIL}kbit prio 2
#$[8*$CEIL/10] this class is for upload webserver. We set the ceil not at 100% but 80%
tc class add dev $EXTDEV parent 1:1 classid 1:13 htb rate 100kbit ceil $[8*$CEIL/10]kbit prio 3
tc qdisc add dev $EXTDEV parent 1:10 handle 100: sfq perturb 10
tc qdisc add dev $EXTDEV parent 1:11 handle 110: sfq perturb 10
tc qdisc add dev $EXTDEV parent 1:12 handle 120: sfq perturb 10
tc qdisc add dev $EXTDEV parent 1:13 handle 130: sfq perturb 10
#add another class:
#tc class add dev $EXTDEV parent 1:1 classid 1:14 htb rate 100kbit ceil ${CEIL}kbit prio 4
#tc qdisc add dev $EXTDEV parent 1:14 handle 140: sfq perturb 10
########what traffic goes where?
##
##		FORWARD	to internet	| OUTPUT to internet
##level 1 acks/small packets all icmp	| dns icmp time small-packets
##level 2 rest traffic uploads &gets  	| gets etc downloads for server (dport 21|80)
##level 3 smtp upload 		 	| all but http output (smtp)
##level 4 				| -src http (output webserver)
#http://lartc.org/howto/lartc.qdisc.classless.html#LARTC.SFQ
#perturb     Reconfigure hashing once this many seconds. If unset, hash will never be reconfigured. 
#Not recommended. 10 seconds is probably a good value
#a child sfq makes sure that within a full class the queue is fairly dispersed among 
#different traffic connections.
# We have just created a htb tree with one level depth. Something like this:
#+---------+
#| root 1: |
#+---------+
#     |
#+---------------------------------------+
#| class 1:1                             |
#+---------------------------------------+
#  |      |      |      |      |      |      
#+----+ +----+ +----+ +----+ +----+ +----+
#|1:10| |1:11| |1:12| |1:13| |1:14| |1:15| 
#+----+ +----+ +----+ +----+ +----+ +----+ 
## Make a filter to direct traffic into classes
#Classifying packets
#We have created the qdisc setup but no packet classification has been made, so now all outgoing packets 
# are going out in class 1:13 ( because we used: tc qdisc add dev $LANDEV root handle 1: htb default 13 ). 
# Now we need to tell which packets go where. This is the most important part.
# Now we set the filters so we can classify the packets with iptables. I really prefer to do it with iptables, 
# because they are very flexible and you have packet count for each rule. Also with the RETURN target packets 
# don't need to traverse all rules. We execute the following commands:
#
tc filter add dev $EXTDEV parent 1:0 protocol ip prio 1 handle 1 fw classid 1:10
tc filter add dev $EXTDEV parent 1:0 protocol ip prio 2 handle 2 fw classid 1:11
tc filter add dev $EXTDEV parent 1:0 protocol ip prio 3 handle 3 fw classid 1:12
tc filter add dev $EXTDEV parent 1:0 protocol ip prio 4 handle 4 fw classid 1:13
#youcould add filter 14: tc filter add dev $EXTDEV parent 1:0 protocol ip prio 4 handle 5 fw classid 1:14
#We have just told the kernel that packets that have a specific FWMARK value ( handle x fw ) go in the 
# specified class ( classid x:x). 
# IPTABLES PART
#Next you will see how to mark packets with iptables.
#First you have to understand how packet traverse the filters with iptables:
#        +------------+                +---------+               +-------------+
#Packet -| PREROUTING |--- routing-----| FORWARD |-------+-------| POSTROUTING |- Packets
#input   +------------+    decision    +-¡-------+       |       +-------------+    out
#                             |                          |
#                        +-------+                    +--------+   
#                        | INPUT |---- Local process -| OUTPUT |
#                        +-------+                    +--------+
#this one is even better: http://l7-filter.sourceforge.net/PacketFlow.png
###iptables is already setup. so is natting
#Now check that packets are flowing through 1:15:
#tc -s class show dev $EXTDEV
#You can start marking packets adding rules to the PREROUTING chain in the mangle table
###this is about traffic coming from the internal workstations to internet
##remember we are talking about limiting upload here, so traffic from your server to internet.
#if you download http for instance the download form internet to you is not the problem.
#what is competing for bandwith is: 
#		- internet hosts downloading/visiting your website
#			the bulk of this traffic has src-port 80, and dst port some high port
#			output from your webserver (output chain)
#		- your traffic to internet. 
#			the bulk of this are syns with dst port 80 (you browsing) (prerouting)
#			and udp with destination port 53 (you doing dns)          (prerouting)
# So if you have a busy website, you better make sure src port 80 going out does not get too much prio and traffic.
# and that dst port 53 and perhaps 80 gets high prio. and icmp proto. syns out, and small packets out.
######test and use this
#iptables -t mangle -A POSTROUTING -p icmp -j MARK --set-mark 0x1
####iptables -t mangle -A POSTROUTING -p icmp -j RETURN
##### Now you should be able to see packet count increasing when pinging from machines within the private 
# network to some site on the Internet. Check packet count increasing in 1:10
#tc -s class show dev $EXTDEV
#############
##The original at #http://lartc.org/howto/lartc.cookbook.fullnat.intro.html
#uses prerouting for forwarded traffic and output for traffic out of the server.
#however, there are new chains in the mangle table (see man iptables): forward and input and postrouting.
#If you want to be efficient you use output only, for traffic from the server, and forward for forwarded
# trafic. If we want to do all outgoing traffic we can use postrouting. To keep things clear we 
# won't use postrouting however.
# remember we are doing egress filtering and are shaping outgoing traffic
########what traffic goes where?
##
##		FORWARD	to internet	| OUTPUT to internet
##level 1 acks/small packets all icmp	| dns icmp time small-packets
##level 2 urest traffic uploads &gets  	| gets etc downloads for server (dport 21|80)
##level 3 smtp upload 		 	| all but http output (smtp)
##level 4 				| -src http (output webserver)
############FORWARDING: traffic from the internal network and server
#We have done a -j RETURN so packets don't traverse all rules. Icmp packets won't match other rules below RETURN. Keep that in mind. 
#Now we can start adding more rules, lets do proper TOS handling:
iptables -t mangle -A FORWARD -m tos --tos Minimize-Delay -o $EXTDEV -j MARK --set-mark 0x1
iptables -t mangle -A FORWARD -m tos --tos Minimize-Delay -j RETURN
iptables -t mangle -A FORWARD -m tos --tos Minimize-Cost -o $EXTDEV -j MARK --set-mark 0x3
iptables -t mangle -A FORWARD -m tos --tos Minimize-Cost -j RETURN
iptables -t mangle -A FORWARD -m tos --tos Maximize-Throughput -o $EXTDEV -j MARK --set-mark 0x4
iptables -t mangle -A FORWARD -m tos --tos Maximize-Throughput -j RETURN
##level1
#insert a rule on line three to make sure output lanside does not have to go through all chains
iptables -t mangle -I FORWARD -o $LANDEV -j RETURN
##level1
## Catchs all those small TCP SYN, SYN-ack etc packets going out like this:
iptables -t mangle -I FORWARD -p tcp  -m length --length :64 -j RETURN
iptables -t mangle -I FORWARD -p tcp  -m length --length :64 -o $EXTDEV -j MARK --set-mark 0x1
#   length This module matches the length of a packet against a specific value or range of values.
#   --length length[:length]. syns/syn-ack are 62 bytes, fin-ack/rst-ack/rst are 60
#Inserts! so they go on top!
##icmp traffic 
iptables -t mangle -A FORWARD -p icmp -o $EXTDEV -j MARK --set-mark 0x1
iptables -t mangle -A FORWARD -p icmp -j RETURN
#level 3: eas smtp outgoing a bit if it is used.
iptables -t mangle -A FORWARD -p tcp --dport 25 -o $EXTDEV -j MARK --set-mark 0x3
iptables -t mangle -A FORWARD -p tcp --dport 25 -j RETURN
# Rest upload such as gets and ssh and so on 
# we terminate the  tables with:
iptables -t mangle -A FORWARD -j MARK --set-mark 0x2
iptables -t mangle -A FORWARD -j RETURN
######OUTPUT server
#I finish OUTPUT chain with -j MARK --set-mark 0x6 so forward traffic has higher priority.
iptables -t mangle -A OUTPUT -m tos --tos Minimize-Delay -o $EXTDEV -j MARK --set-mark 0x1
iptables -t mangle -A OUTPUT -m tos --tos Minimize-Delay -j RETURN
iptables -t mangle -A OUTPUT -m tos --tos Minimize-Cost -o $EXTDEV -j MARK --set-mark 0x3
iptables -t mangle -A OUTPUT -m tos --tos Minimize-Cost -j RETURN
iptables -t mangle -A OUTPUT -m tos --tos Maximize-Throughput -o $EXTDEV -j MARK --set-mark 0x4
iptables -t mangle -A OUTPUT -m tos --tos Maximize-Throughput -j RETURN	
##level1
##icmp so webserver can slow down clients with flowcontrol
iptables -t mangle -A OUTPUT -p icmp -j MARK --set-mark 0x1
#other protocols: or ports? dns certainly, and time
iptables -t mangle -A OUTPUT -p udp --dport 53 -o $EXTDEV -j MARK --set-mark 0x1
iptables -t mangle -A OUTPUT -p udp --dport 53 -j RETURN
iptables -t mangle -A OUTPUT -p tcp --dport 53 -o $EXTDEV -j MARK --set-mark 0x1
iptables -t mangle -A OUTPUT -p tcp --dport 53 -j RETURN
iptables -t mangle -A OUTPUT -p udp --dport 123 -o $EXTDEV -j MARK --set-mark 0x1
iptables -t mangle -A OUTPUT -p udp --dport 123 -j RETURN
#insert a rule on line three to make sure output lanside does not have to go through all chains
iptables -t mangle -I OUTPUT -o $LANDEV -j RETURN
#A good idea is to prioritize small packets to begin/control tcp connections, SYN,syn-ack,reset flag set:
#iptables -t mangle -I OUTPUT -p tcp  -m length --length :64 -j RETURN
#iptables -t mangle -I OUTPUT -p tcp  -m length --length :64 -o $EXTDEV -j MARK --set-mark 0x1
#Or also include dns lookups via the server: they are mostly upto 85 bytes udp
iptables -t mangle -I OUTPUT -m length --length :85 -j RETURN
iptables -t mangle -I OUTPUT -m length --length :85 -o $EXTDEV -j MARK --set-mark 0x1
#now make sure all -spt 80 is NOT prioritized: this should be before small packets!
#webserver output to internet eats this line up: level 4
iptables -t mangle -I OUTPUT -p tcp --sport 80 -j RETURN
iptables -t mangle -I OUTPUT -p tcp --sport 80 -o $EXTDEV -j MARK --set-mark 0x4
##level2
##downloads by/for the server such as anti-virus and updates: port 80 and 21
# Necessary for gets etc bigger than very small
iptables -t mangle -A OUTPUT -p tcp --dport 21 -o $EXTDEV -j MARK --set-mark 0x2
iptables -t mangle -A OUTPUT -p tcp --dport 21 -j RETURN
iptables -t mangle -A OUTPUT -p tcp --dport 80 -o $EXTDEV -j MARK --set-mark 0x2
iptables -t mangle -A OUTPUT -p tcp --dport 80 -j RETURN
#I finish OUTPUT chain with -j MARK --set-mark 0x3 so forward traffic has higher priority
iptables -t mangle -A OUTPUT -o $EXTDEV -j MARK --set-mark 0x3
iptables -t mangle -A OUTPUT -j RETURN			
########## downlink #############
#from http://lartc.org/howto/lartc.qdisc.terminology.html
#                Userspace programs
#                     ^
#                     |
#     +---------------+-----------------------------------------+
#     |               Y                                         |
#     |    -------> IP Stack                                    |
#     |   |              |                                      |
#     |   |              Y                                      |
#     |   |              Y                                      |
#     |   ^              |                                      |
#     |   |  / ----------> Forwarding ->                        |
#     |   ^ /                           |                       |
#     |   |/                            Y                       |
#     |   |                             |                       |
#     |   ^                             Y          /-qdisc1-\   |
#     |   |                            Egress     /--qdisc2--\  |
#  --->->Ingress                       Classifier ---qdisc3---- | ->
#     |   Qdisc                                   \__qdisc4__/  |
#     |                                            \-qdiscN_/   |
#     |                                                         |
#     +----------------------------------------------------------+
#Thanks to Jamal Hadi Salim for this ASCII representation.
#
#The big block represents the kernel. The leftmost arrow represents traffic entering your machine from the #network. It is then fed to the Ingress Qdisc which may apply Filters to a packet, and decide to drop it. #This is called 'Policing'.
#This happens at a very early stage, before it has seen a lot of the kernel. It is therefore a very good #place to drop traffic very early, without consuming a lot of CPU power.
########## downlink ############# (from wondershaper)
# slow downloads down to somewhat less than the real speed  to prevent 
# queuing at our ISP. Tune to see how high you can set it.
# ISPs tend to have *huge* queues to make sure big downloads are fast
#
# attach ingress policer:
tc qdisc add dev $EXTDEV handle ffff: ingress
# filter *everything* to it (0.0.0.0/0), drop everything that's
# coming in too fast:
tc filter add dev $EXTDEV parent ffff: protocol ip prio 50 u32 match ip src \
0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1
logger "htpwondershaper finished"

Traffic shaping script for maximising VoIP

#!/bin/sh
#
# Traffic shaping script for maximising VoIP quality on bursty, busy links
#
# (c) Dan Shearer 2007 dan@shearer.org
# Latest version http://shearer.org/Linux_Shaping_Template
# Under the GPL version 3.0 or higher.
#
# Version 1.1
#
#  Note: This script uses tc from iproute2. It does not use tc filters (which
#  would work) because the tc filter syntax is nasty. Instead you will need
#  some iptables commands which are given at the end of the script BUT commented
#  out. You should insert the fragment in the appropriate place in your firewall,
#  or nothing will work.
EXT=eth1          # Outbound network interface
UPLINKRATE=1300   # Total outbound bandwidth, bits per second
DOWNLINKRATE=9000 # Total inbound bandwidth -- keep this lower than actual,
# to help with inbound queue management. See TCP drop behaviour below
VOIPRATE=400      # Bandwidth guaranteed for voice, SIP and IAX Asterisk protocol
RTRATE=400        # Bandwidth guaranteed for realtime-ish packets, such as small ssh
# packets and any TCP control packets that are not piggybacked.
OTHERRATE=500     # Anything else, eg long ssh (scp/sftp) packets, mail, etc.
###############
############### Transmit controls 
###############
# This does bandwidth shaping and priority queuing, without starving
# any class of traffic of bandwidth. It also has the very beneficial
# effect of fairly sharing connections of equal priority - for example
# twenty wgets with no shaping have wildly varying download rates as first
# one and then another gets in first. With shaping all downloads continue
# at one constant rate.
#
# Documentation is widely spread, and some of it isn't accurate. The best
# thing is to build a test rig and read the developers' documentation. These
# are also helpful:
#
# http://lartc.org/howto/
# http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm
# http://www.tldp.org/HOWTO/ADSL-Bandwidth-Management-HOWTO/ (dated, good concepts)
#
# Warning! Check you are not using documentation that is specific to Linux 2.4. It is
# close to but not the same as Linux 2.6. Good for most concepts, but not exactly
# the same syntax or meaning.
# We can only shape traffic that we are sending, not receiving.
# Queues have Classes belonging to them. Filters put traffic in Classes.
# Classes have priorities.
# All notation of form a:b means queue discipline a and class b. A qdisc is
# always labelled as a:0 or a: as an abbreviation. This is how a tree can be
# specified. In an HTB tree, leaf nodes can borrow bandwidth from each other if
# it is available, but they cannot borrow bandwidth underneath another root.
# zerost - delete everything
tc qdisc del dev $EXT root
# This is the root HTB queue discipline, and defines the default priority to any
# traffic that isn't covered by a class statement below. You also need at
# least one discipline, because disciplines are how classes enforce their
# policies. The number 1 is a label to refer to this qdisc later.
# There can only be one qdisc per interface.
#
# Study the diagram at http://lartc.org/howto/lartc.qdisc.classful.html in 9.5.2.1 carefully.
tc qdisc add dev $EXT root handle 1: htb default 10
# First - prevent queues forming on uplink
ip link set dev $EXT qlen 100  
# Way lower than default of 100. So we have all queues, with none on modem
# Create root class. This has total bandwidth.
tc class add dev $EXT parent 1:  classid 1:1 htb rate \
"$UPLINKRATE"kbit ceil "$UPLINKRATE"kbit
# VoIP data - this is a nailed-up data rate. If no voicecalls, bandwidth still reserved.
tc class add dev $EXT parent 1:1 classid 1:10 htb rate \
"$VOIPRATE"kbit ceil "$UPLINKRATE"kbit prio 1 burst 3k
tc class add dev $EXT parent 1:1  classid 1:20 htb rate \
"$RTRATE"kbit  ceil "$RTRATE"kbit prio 2 burst 2k
# Other
tc class add dev $EXT parent 1:1 classid 1:30 htb rate \
"$OTHERRATE"kbit ceil "$OTHERRATE"kbit prio 3 burst 2k
# Now add queues to handle each class
# VoIP is first in first out, ie fast as you can pls
tc qdisc add dev $EXT parent 1:10 handle 10: pfifo
# The others get stochastic fairness
tc qdisc add dev $EXT parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev $EXT parent 1:30 handle 30: sfq perturb 10 
# (perturb 10 cos the docs say this is the number to use??!
###############
############### Receive controls
###############
# Following verbatim from just about all shaper scripts. This bit seems
# difficult to do in any other way. NOTE! If you make a mistake with the
# handle name, eg 1: instead of ffff: as per below, then you'll have a
# very mysteriously not-working script.
tc qdisc del dev $EXT handle ffff: ingress # delete it, with any filters
tc qdisc add dev $EXT handle ffff: ingress
# filter *TCP*, drop TCP coming in too fast:
tc filter add dev $EXT parent ffff: protocol ip prio 50 u32 match ip src \
0.0.0.0/0 police rate ${DOWNLINKRATE}kbit burst 10k drop flowid :1

Firewall Snippet

These are the iptables commands that go with the traffic shaping script above. The firewall has to:

  • stamp packets in classes so they go in correct priority queue
  • set TOS on packets

If your network has a different idea of what is priority traffic, this is what you change. For example, the length of packets is fairly crude way of determining how important they are, you might want to do a somewhat deeper packet inspection before deciding on a priority.

$EXT is the external interface, $INT is the internal interface.

#!/bin/sh
#
# iptables commands related to traffic shaping script for optimising 
# quality on bursty, busy links
#
# (c) Dan Shearer 2007 dan@shearer.org
# Latest version http://shearer.org/Linux_Shaping_Template
# Under the GPL version 3.0 or higher.
#
# Version 1.1
#
# You might want to use this in conjunction with some ideas from 
# http://shearer.org/Linux_Firewall_Template
## This is an iptables fragment to go with a tc traffic shaper script
# VoIP (IAX, SIP and related) traffic
iptables -t mangle -A POSTROUTING -o $EXT -p udp --dport 5060:5061 -j CLASSIFY --set-class 1:10
iptables -t mangle -A POSTROUTING -o $EXT -p udp --sport 5060:5061 -j CLASSIFY --set-class 1:10
iptables -t mangle -A POSTROUTING -o $EXT -p udp --sport 10000:20000 -j CLASSIFY --set-class 1:10
iptables -t mangle -A POSTROUTING -o $EXT -p udp --dport 10000:20000 -j CLASSIFY --set-class 1:10
# In case there's an Asterisk server inside the firewall
iptables -t mangle -A POSTROUTING -o $EXT -p udp --sport 4569 -j CLASSIFY --set-class 1:10
iptables -t mangle -A POSTROUTING -o $EXT -p udp --dport 4569 -j CLASSIFY --set-class 1:10
# prioritise outbound TCP control packets without a payload (ie they're short.)
# This will tend to pull inbound packets faster (especially good on an assymmetric link.)
iptables -t mangle -A POSTROUTING -o $INT -p tcp --syn -m length \
--length 40:68 -j CLASSIFY --set-class 1:20
iptables -t mangle -A POSTROUTING -o $INT -p tcp --tcp-flags ALL SYN,ACK -m length \
--length 40:68 -j CLASSIFY --set-class 1:20
iptables -t mangle -A POSTROUTING -o $INT -p tcp --tcp-flags ALL ACK -m length \
--length 40:100 -j CLASSIFY --set-class 1:20
iptables -t mangle -A POSTROUTING -o $INT -p tcp --tcp-flags ALL RST \
-j CLASSIFY --set-class 1:20
iptables -t mangle -A POSTROUTING -o $INT -p tcp --tcp-flags ALL ACK,RST \
-j CLASSIFY --set-class 1:20
iptables -t mangle -A POSTROUTING -o $INT -p tcp --tcp-flags ALL ACK,FIN \
-j CLASSIFY --set-class 1:20
# prioritise ssh, but not long packets because they are scp or sftp.
iptables -t mangle -A POSTROUTING -o $INT -p tcp --sport ssh -m length \
--length 40:100 -j CLASSIFY --set-class 1:20
iptables -t mangle -A POSTROUTING -o $INT -p tcp --dport ssh -m length \
--length 40:100 -j CLASSIFY --set-class 1:20
#ICMP. No limiting or selecting of ICMP subtypes - you should do that but
# as part of your standard firewall, not part of your traffic shaper.
iptables -t mangle -A POSTROUTING -o $INT -p icmp -j CLASSIFY --set-class 1:30
iptables -t mangle -A POSTROUTING -o $INT -p tcp --dport 80 \
-j CLASSIFY --set-class 1:30
iptables -t mangle -A POSTROUTING -o $INT -p tcp --dport 22 \
-j CLASSIFY --set-class 1:30
# Everything else (web, long ssh packets, etc) will be unclassified and so the
# tc script will put it in the default class, 1:30. Nothing more to do.
# Set TOS bits - other people's infrastructure hopefully respects them.
# On our side, we don't care much because we have a traffic shaping script, but our
# gateway will still respect TOS as well.
iptables -t mangle -A FORWARD -p udp --dport 5060:5061 \
-j TOS --set-tos Minimize-Delay
iptables -t mangle -A FORWARD -p udp --sport 5060:5061 \
-j TOS --set-tos Minimize-Delay
iptables -t mangle -A FORWARD -p udp --dport 10000:20000 \
-j TOS --set-tos Minimize-Delay
iptables -t mangle -A FORWARD -p udp --sport 10000:20000 \
-j TOS --set-tos Minimize-Delay
iptables -t mangle -A FORWARD -p udp --dport 4569 \
-j TOS --set-tos Minimize-Delay
iptables -t mangle -A FORWARD -p udp --sport 4569 \
-j TOS --set-tos Minimize-Delay
iptables -t mangle -A POSTROUTING -o eth0 -p tcp --dport ssh -m length \
--length 40:100 -j TOS --set-tos Minimize-Delay

Traffic Control with Linux Command Line tool, "tc"

Traffic Control with Linux Command Line tool, "tc"

By Scott Seong

Denial of service attacks are major nuisance for web hosts, and as a web host you'll have to take every measure to protect your resources from DoS attacks. Our APF, BFD, DDoS and RootKit article describes Linux utilities available to protect from DDoS attack, and also explains installation procedures. This article supplements above article by providing means to control traffic (bandwidth shaping) with Linux "tc" command so that no single machine can waste the entire network bandwidth.

What is Traffic Shaping?

Traffic Shaping (a.k.a Bandwidth Shaping or Packet Shaping) is an attempt to control network traffic by prioritizing network resources and guarantee certain bandwidth based on predefined policy rules. Traffic shaping uses concepts of traffic classification, policy rules, queue disciplines and quality of service (QoS).

Why implement Traffic Shaping?

Network bandwidth is an expensive resource that is being shared among many parties of an organization, and some applications require guaranteed bandwidth and priority. Traffic shaping lets you (1) control network services, (2) limit bandwidths and (3) guarantee Quality Of Service (QoS). Intelligently managed traffic shaping improves network latency, service availablity and bandwidth utilization.

What is Queue Discipline?

A queue discipline (qdisc) is rules that determine the order in which arrivals are serviced. Immagine standing in a restraurant to be seated, and waiting in an emergency room to be serviced by a physician. They both have people waiting in a queue that needs to be serviced, but have different strategies for clearing them.

Restaurants typically use first-in-first-out (FIFO) strategy with an exception when tables with number of seats do not exist for large number of customers. Customers are generally serviced in the order that they've arrived in the queue, or when the table with number of seats available. On the other hand, emergency queue requires different strategy. Regardless of order in which patients arrive, someone in a critical condition requires most attention and then someone with urgent condition. This is just examples of how queues are handled in the real life scenarios, but traffic shaping requires a lot more disciplines (rules) for clearing traffic queues.

Software Requirements

 

  • Linux RPM package called 'iproute' is required.
  • Traffic control options (including netlink support) have to enabled on the kernel build in order for certain parts of 'iproute' to function.
  • Linux kernels version 2.4 (and above) have most traffic control options turned on as a default. To explore your configuration, try running the following commands. If you can see the command responses below, you have a basic configuration setup.
# ip link list
1: lo:  mtu 16436 qdisc noqueue 
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: 
 mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:06:5b:8d:13:a0 brd ff:ff:ff:ff:ff:ff
# ip address show
1: lo:  mtu 16436 qdisc noqueue 
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
inet6 ::1/128 scope host 
valid_lft forever preferred_lft forever
2: eth0: 
 mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:06:5b:8d:13:a0 brd ff:ff:ff:ff:ff:ff
inet 216.3.128.12/24 brd 216.3.128.255 scope global eth0
inet6 fe80::206:5bff:fe8d:13a0/64 scope link 
valid_lft forever preferred_lft forever
# ip route show
216.3.128.0/24 dev eth0  proto kernel  scope link  src 
216.3.128.12 default via 216.3.128.1 dev eth0 

Bandwidth Management (Traffic Control)

Linux kernel 2.2 (and above) provides bandwidth management functionality compatible to high-end (dedicated) hardware solution. Linux does offer bandwidth management capability with tc command-line utility, with iptables and iproute2 packages.

We've written a small bash shell script to automate bandwidth shaping function on a linux machine. The downloadable source code is used to limit bandwidth of an interface, both inbound and outbound to 1mbit each. You may modify this script how ever you desire to customize your bandwidth shaping requirements.

#!/bin/bash
#
#  tc uses the following units when passed as a parameter.
#  kbps: Kilobytes per second 
#  mbps: Megabytes per second
#  kbit: Kilobits per second
#  mbit: Megabits per second
#  bps: Bytes per second 
#       Amounts of data can be specified in:
#       kb or k: Kilobytes
#       mb or m: Megabytes
#       mbit: Megabits
#       kbit: Kilobits
#  To get the byte figure from bits, divide the number by 8 bit
#
#
# Name of the traffic control command.
TC=/sbin/tc
# The network interface we're planning on limiting bandwidth.
IF=eth0             # Interface
# Download limit (in mega bits)
DNLD=1mbit          # DOWNLOAD Limit
# Upload limit (in mega bits)
UPLD=1mbit          # UPLOAD Limit
# IP address of the machine we are controlling
IP=216.3.128.12     # Host IP
# Filter options for limiting the intended interface.
U32="$TC filter add dev $IF protocol ip parent 1:0 prio 1 u32"
start() {
# We'll use Hierarchical Token Bucket (HTB) to shape bandwidth.
# For detailed configuration options, please consult Linux man
# page.
$TC qdisc add dev $IF root handle 1: htb default 30
$TC class add dev $IF parent 1: classid 1:1 htb rate $DNLD
$TC class add dev $IF parent 1: classid 1:2 htb rate $UPLD
$U32 match ip dst $IP/32 flowid 1:1
$U32 match ip src $IP/32 flowid 1:2
# The first line creates the root qdisc, and the next two lines
# create two child qdisc that are to be used to shape download 
# and upload bandwidth.
#
# The 4th and 5th line creates the filter to match the interface.
# The 'dst' IP address is used to limit download speed, and the 
# 'src' IP address is used to limit upload speed.
}
stop() {
# Stop the bandwidth shaping.
$TC qdisc del dev $IF root
}
restart() {
# Self-explanatory.
stop
sleep 1
start
}
show() {
# Display status of traffic control status.
$TC -s qdisc ls dev $IF
}
case "$1" in
start)
echo -n "Starting bandwidth shaping: "
start
echo "done"
;;
stop)
echo -n "Stopping bandwidth shaping: "
stop
echo "done"
;;
restart)
echo -n "Restarting bandwidth shaping: "
restart
echo "done"
;;
show)
echo "Bandwidth shaping status for $IF:"
show
echo ""
;;
*)
pwd=$(pwd)
echo "Usage: tc.bash {start|stop|restart|show}"
;;
esac
exit 0

The above script has been tested on Centos 4.x system and (Linux AS 2.x) versions. There is also another utility called tcng, which supposely simplify the arcane tc configuration. If you have comments or suggestions on the above script, please contact feedback@topwebhosts.org.

For detailed explanation of Linux Advanced Routing & Traffic Control HOWTO, please visit http://www.lartc.org website. The above HOWTO also describes method for preventing SYN Floods and ICMP DDoS.