Traffic Control

Wonder Shape pppN

#!/bin/bash -x
# Wonder Shaper
# please read the README before filling out these values
#
# Set the following values to somewhat less than your actual download
# and uplink speed. In kilobits. Also set the device that is to be shaped.

DOWNLINK=1600
UPLINK=1400
DEV=ppp101

# low priority OUTGOING traffic - you can leave this blank if you want
# low priority source netmasks
NOPRIOHOSTSRC=

# low priority destination netmasks
NOPRIOHOSTDST=

# low priority source ports
NOPRIOPORTSRC="20 21 80 554 5050"

# low priority destination ports
NOPRIOPORTDST="20 21 80 554 5050"


# Now remove the following two lines :-)

if [ "$1" = "status" ]
then
        tc -s qdisc ls dev $DEV
        tc -s class ls dev $DEV
        exit
fi


# clean existing down- and uplink qdiscs, hide errors
tc qdisc del dev $DEV root    2> /dev/null > /dev/null
tc qdisc del dev $DEV ingress 2> /dev/null > /dev/null

if [ "$1" = "stop" ]
then
        exit
fi


###### uplink

# install root HTB, point default traffic to 1:20:

tc qdisc add dev $DEV root handle 1: htb default 20

# shape everything at $UPLINK speed - this prevents huge queues in your
# DSL modem which destroy latency:

tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit burst 6k

# high prio class 1:10:
tc class add dev $DEV parent 1:1 classid 1:10 htb rate ${UPLINK}kbit \
   burst 6k prio 1

# bulk & default class 1:20 - gets slightly less traffic,
# and a lower priority:

tc class add dev $DEV parent 1:1 classid 1:20 htb rate $[9*$UPLINK/10]kbit \
   burst 6k prio 2

tc class add dev $DEV parent 1:1 classid 1:30 htb rate $[8*$UPLINK/10]kbit \
   burst 6k prio 2

# all get Stochastic Fairness:
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10
tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
tc qdisc add dev $DEV parent 1:30 handle 30: sfq perturb 10

# TOS Minimum Delay (ssh, NOT scp) in 1:10:

tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
      match ip tos 0x10 0xff  flowid 1:10

# ICMP (ip protocol 1) in the interactive class 1:10 so we
# can do measurements & impress our friends:
tc filter add dev $DEV parent 1:0 protocol ip prio 10 u32 \
        match ip protocol 1 0xff flowid 1:10

# To speed up downloads while an upload is going on, put ACK packets in
# the interactive class:

tc filter add dev $DEV parent 1: protocol ip prio 10 u32 \
   match ip protocol 6 0xff \
   match u8 0x05 0x0f at 0 \
   match u16 0x0000 0xffc0 at 2 \
   match u8 0x10 0xff at 33 \
   flowid 1:10

# rest is 'non-interactive' ie 'bulk' and ends up in 1:20

# some traffic however suffers a worse fate
for a in $NOPRIOPORTDST
do
        tc filter add dev $DEV parent 1: protocol ip prio 14 u32 \
           match ip dport $a 0xffff flowid 1:30
done

for a in $NOPRIOPORTSRC
do
        tc filter add dev $DEV parent 1: protocol ip prio 15 u32 \
           match ip sport $a 0xffff flowid 1:30
done

for a in $NOPRIOHOSTSRC
do
        tc filter add dev $DEV parent 1: protocol ip prio 16 u32 \
           match ip src $a flowid 1:30
done

for a in $NOPRIOHOSTDST
do
        tc filter add dev $DEV parent 1: protocol ip prio 17 u32 \
           match ip dst $a flowid 1:30
done

# rest is 'non-interactive' ie 'bulk' and ends up in 1:20

tc filter add dev $DEV parent 1: protocol ip prio 18 u32 \
   match ip dst 0.0.0.0/0 flowid 1:20


########## downlink #############
# slow downloads down to somewhat less than the real speed  to prevent
# queuing at our ISP. Tune to see how high you can set it.
# ISPs tend to have *huge* queues to make sure big downloads are fast
#
# attach ingress policer:

tc qdisc add dev $DEV handle ffff: ingress

# filter *everything* to it (0.0.0.0/0), drop everything that's
# coming in too fast:
####### ãËé Port 80 ÇÔè§à¢éÒ»ÃÐÁÒ³  440 Kbit/sec
tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip sport 80 \
   0xffff police rate 440kbit burst 10k drop flowid :1

####### ãËé Port 5050 (Port ¢Í§àÇ» pramool.com) ÇÔè§à¢éÒ»ÃÐÁÒ³  420 Kbit/sec
tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip sport 5050 \
   0xffff police rate 420kbit burst 10k drop flowid :1

####### ãËé Port 554 ÇÔè§à¢éÒ»ÃÐÁÒ³  420 Kbit/sec
tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip sport 554 \
   0xffff police rate 420kbit burst 10k drop flowid :1

tc filter add dev $DEV parent ffff: protocol ip prio 50 u32 match ip src \
   0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1

############# ¨º Êèǹ Script

ä»à¾ÔèÁ ãËéàÃÕ¡ Script µÍ¹à¹µËÅØ´ ·Õè file /etc/init.d/checklink.sh  µÃ§ºÃ÷Ѵ·éÒÂæ
 #echo Link Status = $WAN1 $WAN2 $WAN3 $WAN4

     LINK_STATUS=$WAN1$WAN2$WAN3$WAN4

     if [ $status != $oldstatus ]; then

           /etc/init.d/0route.sh $LINK_STATUS
           /etc/init.d/tablenat.sh
           /sbin/wshaper.ppp101 restart
#          /sbin/wshaper.ppp102 restart
           /etc/init.d/dhcp3-server stop

     fi

     oldstatus=$status

done

iptables + tc shaping tricks

iptables + tc shaping tricks 
ACK packets are usually very small, so putting them into a high-priority class is no problem. However, ACK packets can also cary a payload, and some indeed do so. Especially uploads in Kazaa tend to be all large ACK packets.

To counter this problem, I assign a TOS on every outgoing ACK packet. I leave ACKs which already have TOS alone.

$IPTABLES -t mangle -N chkack
$IPTABLES -t mangle -A chkack -m tos --tos ! Normal-Service -j RETURN
$IPTABLES -t mangle -A chkack -p tcp -m length --length 0:128 -j TOS --set-tos Minimize-Delay
$IPTABLES -t mangle -A chkack -p tcp -m length --length 128: -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A chkack -j RETURN
$IPTABLES -t mangle -A qos -p tcp -m tcp -tcp-flags SYN,RST,ACK ACK -j chkack
(I'm using the chain qos to mark all outgoing packets for QoS).
Another problem I encounter, is that the TOS isn't always correct. For instance, when doing rsync over ssh, the packets are marked with TOS minimize-delay.
I've tried setting the TOS of packets larger than a magic value to maximize-throughput, but occasionally legitimate large packets with TOS minimize-delay leave the network. Think: top over ssh.

So, I'm using the limit module in order to let through two large packets per second.

$IPTABLES -t mangle -N chktos
$IPTABLES -t mangle -A chktos -p tcp -m length --length 0:512 -j RETURN
$IPTABLES -t mangle -A chktos -m limit --limit 2/s --limit-burst 10 -j RETURN
$IPTABLES -t mangle -A chktos -j TOS --set-tos Maximize-Throughput
$IPTABLES -t mangle -A chktos -j RETURN
$IPTABLES -t mangle -A qos -m tos --tos Minimize-Delay -j chktos
This code isn't perfect though: all users still suffer somewhat from a rsync-over-ssh stream.

Netfilter architecture Block diagram

This is just my quick-reference for the kernel 2.4 "iptables" tool from
the netfilter framework.

Current set of default tables:

filter (default table): Starts with built-in chains:
INPUT: Arriving.
FORWARD: Being routed.
OUTPUT: Locally generated

nat (traffic that creates new connections): Starts with built-in chains:
PREROUTING: Arriving.
OUTPUT: Locally generated.
POSTROUTING: Exiting.

mangle (specialised packet alteration): Starts with built-in chains:
PREROUTING: Incoming, before routing.
OUTPUT: Locally generated.
INPUT: Arriving.
FORWARD: Being routed.
POSTROUTING: Exiting.

The admin can create/delete/rename additional chains for any target.


Each chain consists of a set of rules, consulted in order (thus the term
"chain") until one's conditions match. If none match, the default
policy applies, "-P" option. (Policies exist only for built-in chains.
Policy target may only be one of the four predefined rules.) Each rule has:
criterion: Which packets will be affected.
target: Which rule to consult next. (May optionally be one of the
predefined rules ACCEPT, DROP, QUEUE=userspace-handled, or
RETURN=policy.)
Each rule is assigned a rulenum, which can be used to refer to it in
iptables commands.


Since rulesets live in RAM, one can preserve them to disk or reload them
using iptables-save and iptables-restore, respectively.

Many of the more interesting features, such as stateful inspection, are
via dynamically-loaded helper modules (option "-m").

$IPTABLES -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT
$IPTABLES -A FORWARD -m limit --limit 3/minute --limit-burst 3 -j LOG
--log-level DEBUG --log-prefix "IPT FORWARD packet died: "

Spoofing:
$IPTABLES -t nat -A PREROUTING -i $INET_IFACE -s 192.168.0.0/16 -j DROP
$IPTABLES -t nat -A PREROUTING -i $INET_IFACE -s 10.0.0.0/8 -j DROP
$IPTABLES -t nat -A PREROUTING -i $INET_IFACE -s 172.16.0.0/12 -j DROP

## Create chain that blocks new connections, except if coming from inside.
# iptables -N block
# iptables -A block -m state --state ESTABLISHED,RELATED -j ACCEPT
# iptables -A block -m state --state NEW -i ! eth0 -j ACCEPT
# iptables -A block -j DROP
## Jump to that chain from INPUT and FORWARD chains.
# iptables -A INPUT -j block
# iptables -A FORWARD -j block

Type of Service (TOS) prioritisation: To maximize ssh response
while maintaining maximum file data transfer over HTTP connections:

# /sbin/iptables -A PREROUTING -t mangle -p tcp --sport ssh \
-j TOS --set-tos Minimize-Delay

# /sbin/iptables -A PREROUTING -t mangle -p tcp --sport http \
-j TOS --set-tos Maximize-Throughput


Netfilter architecture
Block diagram

   --->PREROUTING-->[ROUTE]--->FORWARD---------->POSTROUTING------>
Conntrack      |        Mangle      ^       Mangle
Mangle         |        Filter      |       NAT (Src)
NAT (Dst)      |                    |       Conntrack
(QDisc)        |                 [ROUTE]
v                    |
INPUT Filter      OUTPUT Conntrack
|     Conntrack      ^   Mangle
|     Mangle         |   NAT (Dst)
v                    |   Filter
>- local processes >--

my ipcop squid conf

shutdown_lifetime 5 seconds
icp_query_timeout 0
mcast_icp_query_timeout 2000
dead_peer_timeout 10 seconds
icp_port 0
tcp_outgoing_address 0.0.0.0
udp_incoming_address 0.0.0.0
udp_outgoing_address 0.0.0.0
acl Manager proto cache_object
acl All_Port port 1-65535
acl PURGE method PURGE
http_port 172.20.0.1:800 transparent no-connection-auth

acl no_cache_hosts url_regex -i "/var/ipcop/proxy/advanced/acls/dst_nocache_url.acl"
cache deny no_cache_hosts

cache_effective_user squid
cache_effective_group squid
umask 022

pid_filename /var/run/squid.pid
mime_table /etc/squid/mime.conf

cache_mem 1 MB
cache_swap_low 90
cache_swap_high 95

ipcache_size 4096
ipcache_low 95
ipcache_high 98
fqdncache_size 4096

log_fqdn off
client_netmask 255.255.255.255
ftp_passive on
ftp_user user@domain.com

quick_abort_min 0 KB
quick_abort_max 0 KB
quick_abort_pct 100

half_closed_clients off
cache_dir aufs /var/log/cache 2048 32 256
#cache_dir aufs /var/spool/squid 10240 24 256

error_directory /usr/lib/squid/advproxy/errors/English

offline_mode on

memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA


access_log /var/log/squid/access.log
cache_log /var/log/squid/cache.log
cache_store_log none

log_mime_hdrs off
forwarded_for on
via on


acl within_timeframe time MTWHFAS 00:00-24:00

acl ftp proto FTP
acl all src 0.0.0.0/0.0.0.0
acl localhost src 127.0.0.1/255.255.255.255
acl SSL_ports port 443 # https
acl SSL_ports port 563 # snews
acl SSL_ports port 5555
acl Safe_ports port 80 # http
acl Safe_ports port 21 # ftp
acl Safe_ports port 443 # https
acl Safe_ports port 563 # snews
acl Safe_ports port 70 # gopher
acl Safe_ports port 210 # wais
acl Safe_ports port 1025-65535 # unregistered ports
acl Safe_ports port 280 # http-mgmt
acl Safe_ports port 488 # gss-http
acl Safe_ports port 591 # filemaker
acl Safe_ports port 777 # multiling http
acl Safe_ports port 800 # Squids port (for icons)
http_access allow PURGE localhost
http_access deny PURGE
acl IPCop_http  port 81
acl IPCop_https port 5555
acl IPCop_ips              dst 172.20.0.1
acl IPCop_networks         src "/var/ipcop/proxy/advanced/acls/src_subnets.acl"
acl IPCop_servers          dst "/var/ipcop/proxy/advanced/acls/src_subnets.acl"
acl IPCop_green_network    src 172.20.0.0/255.255.254.0
acl IPCop_green_servers    dst 172.20.0.0/255.255.254.0
acl CONNECT method CONNECT

# Time Out
#dead_peer_timeout 10 seconds
#request_timeout 15 seconds
#forward_timeout 15 seconds
#connect_timeout 15 seconds
#peer_connect_timeout 15 seconds
#pconn_timeout 120 seconds
#read_timeout 15 minutes
#request_timeout 5 minutes
#persistent_request_timeout 2 minute
#shutdown_lifetime 5 seconds
#negative_ttl 15 seconds
#negative_ttl 15 seconds
#positive_dns_ttl 60 seconds
#negative_dns_ttl 60 seconds

dead_peer_timeout 10 seconds
request_timeout 5 minutes
forward_timeout 5 minutes
connect_timeout 5 minutes
peer_connect_timeout 1 minutes
pconn_timeout 120 seconds
read_timeout 15 minutes
request_timeout 5 minutes
persistent_request_timeout 2 minute
shutdown_lifetime 5 seconds
negative_ttl 2 minutes
negative_ttl 3 minutes
positive_dns_ttl 120 seconds
negative_dns_ttl 120 seconds


netdb_low 900
netdb_high 1000
client_db on
client_lifetime 1 day

refresh_pattern ^http://(.*?)/get_video\? 10080 90% 999999 override-expire ignore-no-cache ignore-private
refresh_pattern ^http://(.*?)/videodownload\? 10080 90% 999999 override-expire ignore-no-cache ignore-private

# mark for no cache
hierarchy_stoplist cgi-bin ? localhost .asp .aspx .php .inf .dll .Xt .xtp .ini localhost php$ inf$ dll$ Xt$ xtp$ ini$ asp$ aspx$ .exe .cfg ucg
acl QUERY urlpath_regex cgi-bin \? localhost .asp .aspx .php .inf .dll .Xt .xtp .ini localhost php$ inf$ dll$ Xt$ xtp$ ini$ asp$ aspx$ updatelist$ patch_S4 .cfg .exe ucg
cache deny QUERY


refresh_pattern ^http://(.*?)/get_video\? 1440 100% 1440 ignore-reload override-lastmod reload-into-ims override-expire ignore-no-cache ignore-private
refresh_pattern ^http://(.*?)/videodownload\? 1440 100% 1440 ignore-reload override-lastmod reload-into-ims override-expire ignore-no-cache ignore-private
refresh_pattern ^http://update.cabal.* 10080 100% 10080 ignore-reload override-lastmod reload-into-ims override-expire ignore-no-cache ignore-private
refresh_pattern ^http://download.cabal.* 10080 100% 10080 ignore-reload override-lastmod reload-into-ims override-expire ignore-no-cache ignore-private
refresh_pattern ^http://patch.sf.* 10080 100% 10080 ignore-reload override-lastmod reload-into-ims override-expire ignore-no-cache ignore-private
refresh_pattern ^http://202.43.34.11 10080 100% 10080 ignore-reload override-lastmod reload-into-ims override-expire ignore-no-cache ignore-private
refresh_pattern ^http://cbt.patch.easportsfifaonline2.in.th/.* 10080 100% 10080 ignore-reload override-lastmod reload-into-ims override-expire ignore-no-cache ignore-private
refresh_pattern ^http://patch.kr.in.th/.* 10080 100% 10080 ignore-reload override-lastmod reload-into-ims override-expire ignore-no-cache ignore-private
refresh_pattern ^http://autopatch.sdo.in.th/patch/.* 10080 100% 10080 ignore-reload override-lastmod reload-into-ims override-expire ignore-no-cache ignore-private
refresh_pattern ^http://update.hitsplay.com/.* 10080 100% 10080 ignore-reload override-lastmod reload-into-ims override-expire ignore-no-cache ignore-private
refresh_pattern ^http://61.47.57.9/.* 10080 100% 10080 ignore-reload override-lastmod reload-into-ims override-expire ignore-no-cache ignore-private
refresh_pattern ^http://www.titanonline.in.th/.* 10080 100% 10080 ignore-reload override-lastmod reload-into-ims override-expire ignore-no-cache ignore-private

refresh_pattern ^http://202.43.34.110/patch/   4320  90%  43200 override-lastmod override-expire reload-into-ims ignore-reload ignore-no-cache
refresh_pattern ^http://patch.sf.* 4320 100% 43200 override-lastmod reload-into-ims ignore-reload
refresh_pattern ^http://*.playpark.*/.* 720 90% 4320 reload-into-ims override-lastmod
refresh_pattern ^http://*.*.in.th/.* 720 90% 4320 reload-into-ims override-lastmod
refresh_pattern ^http://*.sf.*/.* 720 90% 4320 reload-into-ims override-lastmod
refresh_pattern -i .thaicybergames.com/startgame/ucg/ 0 100% 10080
refresh_pattern -i [.]dat$ 2880 100% 20160
refresh_pattern -i (html|htm|pl|cgi|/)$ 60 20% 720
refresh_pattern -i (asp\?|php\?|php3\?|php4\?) 0 20% 720
refresh_pattern -i .(raw|delta|exe)$ 1440 90% 1440 override-lastmod override-expire reload-into-ims ignore-reload
refresh_pattern -i \\.zip$ 0 100% 1440 override-lastmod override-expire reload-into-ims ignore-reload
refresh_pattern -i \\.pkg$ 0 100% 1440 override-lastmod override-expire reload-into-ims ignore-reload
refresh_pattern -i \.*$ 1440 90% 1440 override-lastmod override-expire reload-into-ims ignore-reload
refresh_pattern -i .(class|pdf|rtf|doc|wp|wp5|ps|prn)$ 1440 90% 1440 override-lastmod override-expire reload-into-ims ignore-reload
refresh_pattern -i .(mov|avi|mpg|wav|au|mid|mp3|dat)$ 1440 100% 1440 override-lastmod override-expire reload-into-ims ignore-reload
refresh_pattern -i .(zip|gz|arj|lha|lzh|rar|tgz|tar|Z)$ 1440 80% 1440 override-lastmod override-expire reload-into-ims ignore-reload
refresh_pattern -i .(jpg|gif|jpeg|png|css|js)$ 1440 19000% 1440 override-lastmod override-expire reload-into-ims ignore-reload
refresh_pattern -i .(bmp|tif|tiff|xbm)$ 1440 17000% 1440 override-lastmod override-expire reload-into-ims ignore-reload
refresh_pattern -i .(png|swf|pkg)$ 1440 18000% 1440 override-lastmod override-expire reload-into-ims ignore-reload
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern . 0 20% 4320


# ----- Options For Tuning The Cache
quick_abort_min 0 KB
quick_abort_max 0 KB
quick_abort_pct 100

half_closed_clients off


cache_effective_user squid
cache_effective_group squid
umask 022


http_access allow ftp
http_access allow Manager all
http_access allow Manager
http_access allow All_Port
http_access allow CONNECT All_Port
http_access allow localhost
http_access allow all

acl QUERYS urlpath_regex UCG.exe
acl QUERYS urlpath_regex UCGA.exe
acl QUERYS urlpath_regex UCG.dat
acl QUERYS urlpath_regex ProjectG.exe.zip
acl QUERYS urlpath_regex FantaTennis.exe
acl QUERYS urlpath_regex notice_popup
acl QUERYS urlpath_regex Court_Ad_0
acl QUERYS urlpath_regex Main_Ad



#Access to squid:
#local machine, no restriction
http_access allow         localhost

#GUI admin if local machine connects
http_access allow         IPCop_ips IPCop_networks IPCop_http
http_access allow CONNECT IPCop_ips IPCop_networks IPCop_https

#Deny not web services
http_access deny          !Safe_ports
http_access deny  CONNECT !SSL_ports

#Set custom configured ACLs
http_access allow IPCop_networks within_timeframe
http_access deny  all

#Strip HTTP Header
header_access X-Forwarded-For deny all
header_access Via deny all

memory_pools on
memory_pools_limit 256 MB
maximum_object_size 512 KB
minimum_object_size 8 KB
maximum_object_size_in_memory 32 KB

http_reply_access allow all
icp_access allow all
#request_body_max_size 0 KB

#Strip HTTP Header
header_access Allow allow all
header_access Authorization allow all
header_access Cache-Control allow all
header_access Content-Encoding allow all
header_access Content-Length allow all
header_access Content-Type allow all
header_access Date allow all
header_access Expires allow all
header_access Host allow all
header_access If-Modified-Since allow all
header_access Last-Modified allow all
header_access Location allow all
header_access Pragma allow all
header_access Accept allow all
header_access Accept-Charset allow all
header_access Accept-Encoding allow all
header_access Accept-Language allow all
header_access Content-Language allow all
header_access Mime-Version allow all
header_access Retry-After allow all
header_access Title allow all
header_access Connection allow all
header_access Proxy-Connection allow all
header_access All allow all
header_access X-Forwarded-For deny all
header_access Via deny all

header_access User-Agent deny all
header_replace User-Agent Mozilla/5.0 (compatible; MSIE 6.0)
header_access Accept deny all
header_replace Accept */*
header_access Accept-language deny all
header_replace Accept-language id, en
cache allow all

url_rewrite_children 10
acl youtube_query url_regex -i \.youtube\.com\/get_video
acl youtube_query url_regex -i \.cache[a-z0-9]?[a-z0-9]?[a-z0-9]?\.googlevideo\.com\/videoplayback
acl youtube_query url_regex -i \.cache[a-z0-9]?[a-z0-9]?[a-z0-9]?\.googlevideo\.com\/get_video
acl youtube_deny url_regex -i http:\/\/[a-z][a-z]\.youtube\.com
acl metacafe_query dstdomain v.mccont.com
acl dailymotion_query url_regex -i proxy\-[0-9][0-9]\.dailymotion\.com\/
acl google_query dstdomain vp.video.google.com
acl redtube_query dstdomain dl.redtube.com
acl xtube_query url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
acl vimeo_query url_regex -i bitcast\.vimeo\.com\/vimeo\/videos\/
acl wrzuta_query url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
url_rewrite_access deny youtube_deny
url_rewrite_access allow youtube_query
url_rewrite_access allow metacafe_query
url_rewrite_access allow dailymotion_query
url_rewrite_access allow google_query
url_rewrite_access allow redtube_query
url_rewrite_access allow xtube_query
url_rewrite_access allow vimeo_query
url_rewrite_access allow wrzuta_query
redirector_bypass on

acl store_rewrite_list url_regex ^http://(.*?)/video/x-flv\?
acl store_rewrite_list url_regex ^http://(.*?)/get_video\?
acl store_rewrite_list url_regex ^http://(.*?)/videodownload\?
acl store_rewrite_list url_regex ^http://i(.*?).photobucket.com/albums/(.*?)/(.*?)/(.*?)\?
acl store_rewrite_list url_regex ^http://vid(.*?).photobucket.com/albums/(.*?)/(.*?)\?


cache allow store_rewrite_list

storeurl_access allow store_rewrite_list
storeurl_access deny all
storeurl_rewrite_program /usr/local/bin/store_url_rewrite

# ----- Miscellaneous
collapsed_forwarding on
logfile_rotate 3
log_icp_queries off
query_icmp off
buffered_logs off
reload_into_ims on
nonhierarchical_direct off
prefer_direct on
strip_query_terms off
pipeline_prefetch on
ie_refresh on
forwarded_for on
vary_ignore_expire on
store_dir_select_algorithm round-robin
ignore_unknown_nameservers on
balance_on_multiple_ip on

reply_body_max_size 0 allow all

visible_hostname aims.wizard.net

cache_mgr biew55@hotmail.com

#url_rewrite_program /usr/sbin/updxlrator
#url_rewrite_children 10


ftp_sanitycheck off


# ACL BLOCK_http_access deny download all Access Virus
# ----------------------------------------------------
#acl bittorrent_announce url_regex -i http://.+announce.+info_hash=   #announce.php?passkey
#acl bittorrent_announce1 url_regex -i http://.+announce.php?passkey
#http_access deny bittorrent_announce
#http_access deny bittorrent_announce1

acl bittorrent_announce url_regex -i http://.+announce.+info_hash=   announce.php?passkey /announce
http_access deny bittorrent_announce

#acl download urlpath_regex  -i \.torrent  #
#http_access deny download

acl virus_1 urlpath_regex -i .*/system32/cmd\.exe.*
acl virus_2 urlpath_regex -i .*/winnt/system32/cmd.exe.*
acl virus_3 urlpath_regex -i .*/MSADC/root.exe..c.dir$
acl virus_4 urlpath_regex -i .*/scripts/root.exe..c.dir$

http_access deny virus_1
http_access deny virus_2
http_access deny virus_3
http_access deny virus_4

#acl for_throttled_urls url_regex -i "/var/ipcop/proxy/advanced/acls/dst_throttle.acl"

#Set download throttling
acl bb_download url_regex -i "/var/ipcop/proxy/advanced/acls/dst_throttle.acl"
#acl bb_download url_regex -i ftp .3gp .aac .ac3 .act .aif .aiff .amr .asf .au .avi .b5t .bin .bwt .cab .ccd .cdi .cue .dat .dct .div .divx .dss .exe .flac .fli .flv .gho .gsm .gz .ifo .img .iso .m4a .mp2 .mp3 .mp4 .mov .mpe .mpga .mpg .mpeg .mds .nrg .ogg .pdi .qt .ra .ram .rar .raw .rcd .rec .rm .rmvb .rmj .rpm .sea .shn .sri .swf .tar .tgz .vob .vox .vqf .wav .wmv .wma .zip .7z
delay_pools 1
delay_class 1 3
delay_parameters 1 256000/512000 -1/-1 256000/512000
delay_access 1 deny  IPCop_ips
delay_access 1 allow bb_download
delay_initial_bucket_level 100

# Delay pool_setup additional by Sontaya (512/256 Kbps)
# -----------------------------------------------------------------------------
# -----------------------------------------------------------------------------
#acl bcsc_local url_regex -i 172.20
#acl bcsc_admin src 172.20.1.55 172.20.0.55
#acl bcsc_download url_regex -i ftp .3gp .aac .ac3 .act .aif .aiff .amr .asf .au .avi .b5t .bin .bwt .cab .ccd .cdi .cue .dat .dct .div .divx .dss .exe .flac .fli .flv .gho .gsm .gz .ifo .img .iso .m4a .mp2 .mp3 .mp4 .mov .mpe .mpga .mpg .mpeg .mds .nrg .ogg .pdi .qt .ra .ram .rar .raw .rcd .rec .rm .rmvb .rmj .rpm .sea .shn .sri .swf .tar .tgz .vob .vox .vqf .wav .wmv .wma .zip .7z
#acl bcsc_unlimit_bandwidth time MTWHFAS 09:00-11:00
#acl bcsc_upload url_regex -i .avi .mpg .iso .wav .wma .dat .cda .wm .mid .midi .mp3
#delay_pools 3
#delay_class 1 2
#delay_parameters 1 -1/-1 -1/-1
#delay_access 1 allow bcsc_local
#delay_access 1 allow !bcsc_unlimit_bandwidth
#delay_access 1 allow bcsc_admin
#delay_class 2 2
#delay_parameters 2   131072/131072 131072/256000
#delay_access 2 allow bcsc_download
#delay_class 3 2
#delay_parameters 3   20480/20480 20480/40960
#delay_access 3 allow bcsc_upload
#delay_access 2 deny !bcsc_unlimit_bandwidth
#delay_access 2 allow bcsc_admin

# ----------------------------------------------------------------------------

Feature: Delay Pools

Feature: Delay Pools

  • Goal: To provide a way to limit the bandwidth of certain requests based on any list of criteria.

  • Status: Completed

  • Version: 2.2+

  • Developer: David Luyer

 

Delay Pools

by David Luyer.

To enable delay pools features in Squid configure with --enable-delay-pools before compilation.

 

Terminology for this FAQ entry:

pool
a collection of bucket groups as appropriate to a given class
bucket group
a group of buckets within a pool, such as the per-host bucket group, the per-network bucket group or the aggregate bucket group (the aggregate bucket group is actually a single bucket)
bucket
an individual delay bucket represents a traffic allocation which is replenished at a given rate (up to a given limit) and causes traffic to be delayed when empty
class
the class of a delay pool determines how the delay is applied, ie, whether the different client IPs are treated separately or as a group (or both)
class 1
a class 1 delay pool contains a single unified bucket which is used for all requests from hosts subject to the pool
class 2
a class 2 delay pool contains one unified bucket and 255 buckets, one for each host on an 8-bit network (IPv4 class C)
class 3
contains 255 buckets for the subnets in a 16-bit network, and individual buckets for every host on these networks (IPv4 class B )
class 4
as class 3 but in addition have per authenticated user buckets, one per user.
class 5
custom class based on tag values returned by external acl helpers in http_access. One bucket per used tag value.

Delay pools allows you to limit traffic for clients or client groups, with various features:

  • can specify peer hosts which aren't affected by delay pools, ie, local peering or other 'free' traffic (with the no-delay peer option).

  • delay behavior is selected by ACLs (low and high priority traffic, staff vs students or student vs authenticated student or so on).
  • each group of users has a number of buckets, a bucket has an amount coming into it in a second and a maximum amount it can grow to; when it reaches zero, objects reads are deferred until one of the object's clients has some traffic allowance.
  • any number of pools can be configured with a given class and any set of limits within the pools can be disabled, for example you might only want to use the aggregate and per-host bucket groups of class 3, not the per-network one.

This allows options such as creating a number of class 1 delay pools and allowing a certain amount of bandwidth to given object types (by using URL regular expressions or similar), and many other uses I'm sure I haven't even though of beyond the original fair balancing of a relatively small traffic allocation across a large number of users.

 

There are some limitations of delay pools:

  • delay pools are incompatible with slow aborts; quick abort should be set fairly low to prevent objects being retrieved at full speed once there are no clients requesting them (as the traffic allocation is based on the current clients, and when there are no clients attached to the object there is no way to determine the traffic allocation).
  • delay pools only limits the actual data transferred and is not inclusive of overheads such as TCP overheads, ICP, DNS, ICMP pings, etc.
  • it is possible for one connection or a small number of connections to take all the bandwidth from a given bucket and the other connections to be starved completely, which can be a major problem if there are a number of large objects being transferred and the parameters are set in a way that a few large objects will cause all clients to be starved (potentially fixed by a currently experimental patch).
  • in Squid 3.1 the class-based pools do not work yet with IPv6 addressed clients.
  • In squid older than 3.1 the delay pool bucket is limited to 32-bits and thus has a rather low MB cap on both bucket content and refill rate. The bucket size is now raised to 64-bit 'unlimited' values, but refill rate remains low.

 

How can I limit Squid's total bandwidth to, say, 512 Kbps?

delay_pools 1
delay_class 1 1
delay_access 1 allow all
delay_parameters 1 64000/64000          # 512 kbits == 64 kbytes per second

For an explanation of these tags please see the configuration file.

The 1 second buffer (max = restore = 64kbytes/sec) is because a limit is requested, and no responsiveness to a burst is requested. If you want it to be able to respond to a burst, increase the aggregate_max to a larger value, and traffic bursts will be handled. It is recommended that the maximum is at least twice the restore value - if there is only a single object being downloaded, sometimes the download rate will fall below the requested throughput as the bucket is not empty when it comes to be replenished.

 

How to limit a single connection to 128 Kbps?

You can not limit a single HTTP request's connection speed. You can limit individual hosts to some bandwidth rate. To limit a specific host, define an acl for that host and use the example above. To limit a group of hosts, then you must use a delay pool of class 2 or 3. For example:

acl only128kusers src 192.168.1.0/255.255.192.0
delay_pools 1
delay_class 1 3
delay_access 1 allow only128kusers
delay_access 1 deny all
delay_parameters 1 64000/64000 -1/-1 16000/64000

For an explanation of these tags please see the configuration file.

The above gives a solution where a cache is given a total of 512kbits to operate in, and each IP address gets only 128kbits out of that pool.

 

How do you personally use delay pools?

We have six local cache peers, all with the options 'proxy-only no-delay' since they are fast machines connected via a fast ethernet and microwave (ATM) network.

For our local access we use a dstdomain ACL, and for delay pool exceptions we use a dst ACL as well since the delay pool ACL processing is done using "fast lookups", which means (among other things) it won't wait for a DNS lookup if it would need one.

Our proxy has two virtual interfaces, one which requires student authentication to connect from machines where a department is not paying for traffic, and one which uses delay pools. Also, users of the main Unix system are allowed to choose slow or fast traffic, but must pay for any traffic they do using the fast cache. Ident lookups are disabled for accesses through the slow cache since they aren't needed. Slow accesses are delayed using a class 3 delay pool to give fairness between departments as well as between users. We recognize users of Lynx on the main host are grouped together in one delay bucket but they are mostly viewing text pages anyway, so this isn't considered a serious problem. If it was we could take those hosts into a class 1 delay pool and give it a larger allocation.

I prefer using a slow restore rate and a large maximum rate to give preference to people who are looking at web pages as their individual bucket fills while they are reading, and those downloading large objects are disadvantaged. This depends on which clients you believe are more important. Also, one individual 8 bit network (a residential college) have paid extra to get more bandwidth.

The relevant parts of my configuration file are (IP addresses, etc, all changed):

# ACL definitions
# Local network definitions, domains a.net, b.net
acl LOCAL-NET dstdomain a.net b.net
# Local network; nets 64 - 127.  Also nearby network class A, 10.
acl LOCAL-IP dst 192.168.64.0/255.255.192.0 10.0.0.0/8
# Virtual i/f used for slow access
acl virtual_slowcache myip 192.168.100.13
# All permitted slow access, nets 96 - 127
acl slownets src 192.168.96.0/255.255.224.0
# Special 'fast' slow access, net 123
acl fast_slow src 192.168.123.0/24
# User hosts
acl my_user_hosts src 192.168.100.2/31
# Don't need ident lookups for billing on (free) slow cache
ident_lookup_access allow my_user_hosts !virtual_slowcache
ident_lookup_access deny all
# Security access checks
http_access [...]
# These people get in for slow cache access
http_access allow virtual_slowcache slownets
http_access deny virtual_slowcache
# Access checks for main cache
http_access [...]
# Delay definitions (read config file for clarification)
delay_pools 2
delay_initial_bucket_level 50
delay_class 1 3
delay_access 1 allow virtual_slowcache !LOCAL-NET !LOCAL-IP !fast_slow
delay_access 1 deny all
delay_parameters 1 8192/131072 1024/65536 256/32768
delay_class 2 2
delay_access 2 allow virtual_slowcache !LOCAL-NET !LOCAL-IP fast_slow
delay_access 2 deny all
delay_parameters 2 2048/65536 512/32768

The same code is also used by a some of departments using class 2 delay pools to give them more flexibility in giving different performance to different labs or students.

 

Where else can I find out about delay pools?

This is also pretty well documented in the configuration file, with examples. Squid install with a squid.conf.documented or squid.conf.default file. If you no longer have a documented config file the latest version is provided on the squid-cache.org website.