How to limit bandwidth in Linux - shaperd (CBQ)

Warning
If you know english better than me, fix errors and send them to me. Maybe in the future, when I am rich, I will invest in my education, but not now.

Versions:

In the beginning I'd like to say that this software is in beta version so it means that shaperd is insufficiently tested and, in some cases, can make problems. Probably it will be upgraded more times.


Rule of working:

After appearing a new user in network and beginning data transfer from internet (local transfer don't allocate bandwidth), shaperd is testing user's connections and compare it with list of exceptions (/etc/shaper/ignore file). If daemon can't find connection in exception list then allocate bandwidth for this user. Size of bandwidth result from simple dividing maximum incoming speed of link by number of currently working users.
After specify time (default: 10 seconds) - shaperd checks utilization of user's allocated bandwidth. If user utilized less than 50% of allocated bandwidth, daemon reduce allocation about 25%. If user utilized more than 75% then shaperd increase bandwidth about minimum guaranteed bandwidth (e.g. 8000 bit/s that is 1 KB/s). If user utilized bandwidth between 50% and 75% then allocation is unchanged. Rule is to allocate bandwidth in sequence for: "new" users, low allocation users and rest of unused bandwidth is assigned to high allocated users. I hope it's clear :) Also daemon controls outgoing traffic bandwidth. Upload (outgoing - to internet) speed depends from sum of total user's download allocations and is possible to present it by simple formula: 

speed_upload = speed_max - speed_download
when:
  • speed_upload - outgoing speed (from user to internet)
  • speed_max - maximum bandwidth of link in both directions (empirical appointed)
  • speed_download - sum of all allocated bandwidths of users (from internet to users)
Thanks to it shaperd prevent taking over whole bandwidth by single maniacs of P2P software or idiot opening 30 sesions of FTP in the same moment, Shaperd I wrote in C language and funcionally match to shaper_cbq shell script (also made by me (as you see - you can't choise :)


Attention

Still exists insoluble problem in 2.4.x kernels, which cause that some old masqueraded connections are hanging on by 5 days. As a solution shaperd ignore connections with last activity time bigger than 300 seconds.


WARNING

Autor of this software will not respond for any damages, destructions or errors, caused by this software, description and anything else. Everythink you do - you do It on your own responsibility.
This software is tested on systems based on 2.4.8, 2.4.12, 2.4.18 kernels, ipchains and iptables firewalls (I'm using Linux Mandrake 8.1 for developing).



Last version: shaperd_cbq.tar.gz (1.43.5-15.11.2002)

Older versions:
shaperd_cbq_143.3.tar.gz (1.43.3-12.11.2002)
shaperd_cbq_143.tar.gz (1.43-02.11.2002)
shaperd_cbq_142.tar.gz (1.42-05.10.2002)
shaperd_cbq_140.tar.gz (1.40-26.09.2002)
shaperd_cbq_135.tar.gz (1.35-11.09.2002)
shaperd_cbq_134.tar.gz (1.34-07.09.2002)
shaperd_cbq_133.tar.gz (1.33-06.09.2002)
shaperd_cbq_132.tar.gz (1.32-05.09.2002)
shaperd_cbq_131.tar.gz (1.31-28.09.2002)
shaperd_cbq_130.tar.gz (1.30-19.08.2002)
shaperd_cbq_129.tar.gz (1.29-30.07.2002)
shaperd_cbq_128.tar.gz (1.28-29.07.2002)
shaperd_cbq_127.tar.gz (1.27-28.07.2002)
shaperd_cbq_126.tar.gz (1.26-01.07.2002)
shaperd_cbq_125.tar.gz (1.25-27.06.2002)
shaperd_cbq_124.tar.gz (1.24-27.06.2002)
shaperd_cbq_123.tar.gz (1.23-25.06.2002)
shaperd_cbq_122.tar.gz (1.22-19.06.2002)
shaperd_cbq_121.tar.gz (1.21-19.06.2002)
shaperd_cbq_120.tar.gz (1.20-19.06.2002)
shaperd_cbq_117.tar.gz (1.17-04.06.2002)
shaperd_cbq_116.tar.gz (1.16-03.06.2002)


Before you'll try to start shaperd you should check if your system accomplish following conditions:
  1. installed iproute2 package
  2. recompiled linux kernel for QoS (CBQ) support (I don't know other distributions but in Mandrake 8.1 kernel QoS (CBQ) is supported by default so I did not need to recompile kernel).
    If you don't know, how to compile kernel with QoS (CBQ) support then check this page.
  3. Daemon co-operate as well with ipchains and iptables (firewall type is recognized automatically).
  4. If you want to use squid with shaperd then you should patch squid source code
  5. Is no needed to make any rules on firewall especially for shaperd. Daemon will do everythink needed after start.
  6. You should check whether your kernel is QoS (CBQ) support. To do it write:
    tc -d qdisc
    If you'll get nothing as a result then your kernel support QoS (CBQ).
    If you'll get RTNETLINK error then this mean that your kernel don't support QoS (CBQ) (or you forgot to load modules for QoS (CBQ)).
  7. shaperd will work with few principles:
  8. Daemon has to act root on rights and to have possibility of record to directory /var/shaper. It has owner of this directory to be root. Attributes of this directory are 755
  9. In file /etc/shaper are written except for which shaperd does not assign bandwidth. Example:
    217.96.55.5 411 - IP number and port 411
    213.180.130.190 - IP number nad all ports
    22$ - all IP numers and port 22 (telnet)
    I made this therefore, in order to daemon did not assign link onto connections generating practically no's transfer (e.g. IRC, telnet). Current list taken under attention at division of numbers ip it were been possible to see writing: 
    /sbin/shaperd shownat
  10. In directory /var/shaper is gernerated bitrate_user_sh.old file - it contains IP numbers as well as assigned bandwidths. It was been possible to use it to visualization on WWW page - example of use. Content of this file can look so: 
    192.168.1.2 40000
192.168.1.9 58000
    or if speed_ext is set as Kbit: 
    192.168.1.2 40 Kbit
192.168.1.9 58 Kbit
  11. Is good to check how shaperd is working writing command:
    tc -s qdisc
    Results of his working can look so: 
     qdisc tbf d09f: dev eth0 rate 5430bps burst 10Kb lat 1.2s              
 Sent 108930 bytes 73 pkts (dropped 0, overlimits 0)                   
                                                                       
 qdisc tbf d09e: dev eth0 rate 5430bps burst 10Kb lat 1.2s             
 Sent 126618 bytes 89 pkts (dropped 0, overlimits 0)                   
                                                                       
 qdisc tbf d09d: dev eth0 rate 1638bps burst 10Kb lat 3.8s             

 Sent 54699 bytes 65 pkts (dropped 0, overlimits 321)                  
 backlog 6110b 5p                                                      
                                                                       
 qdisc tbf d09c: dev eth0 rate 10Mbit burst 10Kb lat 4.8ms             
 Sent 0 bytes 0 pkts (dropped 0, overlimits 0)                         
                                                                       
 qdisc tbf d09b: dev eth0 rate 10Mbit burst 10Kb lat 4.8ms             
 Sent 5690 bytes 53 pkts (dropped 0, overlimits 0)                     
                                                                       


 qdisc cbq 10: dev eth0 rate 10Mbit (bounded,isolated) prio no-transmit
 Sent 310829 bytes 302 pkts (dropped 0, overlimits 1555)               
 backlog 5p                                                            
  borrowed 0 overactions 0 avgidle 624 undertime 0
    Look on line from rate different from 10Mbit (or different in case of local link about different maximum speeds). In this of case of three first class illustrate transfers of three different computers in local net. Check, whether registered size of sent datas Sent are. If so this it means, that CBQ work correctly and cuts transfer.
  12. If you have problems with starting of this daemon, before you will write to me - read this description once again and check all again. You can write to me then. In view of lack of time my help can be a bit delay so I command looking to Guest Book. Maybe someone, from written admins, will want to help you earlier.
  13. If you'll configure and you'll succesfully start this programme on your server, write your datas to Guest Book. Write following data, please:
    • shaperd version
    • description of system (distribution, kernel version, remark)
    • e-mail
    • url of your server www page (if exists)
    Thanks to it you can help different administrators in installation of this daemon and you will relieve me a bit. I'll be very grateful.
  14. It will be very pleasure for me, if some happy administrator will put in somewhere onto his network WWW page this small banner:

    Powered by Shaper CBQ

    Here's a HTML code for paste: 
    <p><a href="http://sp9wun.republika.pl/">
<img src="http://sp9wun.republika.pl/linux/pics/shaper.png" border="0" 
width="89" height="32" alt="Powered by Shaper CBQ"></a></p>
See a PHP script kto.php (additional script ckwintalk) for checking is wintalk working on user's side) to generating list of active computers alongside with assigned bandwidth allocation. You have to edit the script and to write datas from your net. Start him kto.php - for PHP version.
Script demands configuration of DNS for local IP numbers. Moreover bind can step out mistakes with some versions. You have to play into altering.

Problems

To avoid itself problems, which they can result in track of normal exploitation of shaperd you have to warn some rules

  1. If shaperd crash or it does not want to start - start from compilation on your system of sources' codes. Executable file delivered in archive is compiled on fresh system and they can be problems with starting on older distributions.


  2. At the start of system script /etc/init.d/shaperd start possibly on end and in peculiarity always start him after any scripts, which can modify firewall (e.g. every scripts to configuration masquerade, firewall etc).


  3. If you executed any operations on firewall (you cleaned up meters of rules, added or deleted some rules, reloaded firewall etc) restart shaperd obligatorily. Renunciation of this can be effective stifling of whole incoming transfer, because shaperd at start creature of own rule on firewall and they have to be checked them in first order. Change of order of rules on firewall can (though it does not have) result, that registering rules will not register incoming traffic and daemon will affirm, that nobody uses assigned bandwidths so it will start to lower allotments (down to guaranteed minimum). To check whether rules for shaperd are on proper place and they work correctly it suffices to make following operation:
    • for ipchains:
      • ipchains -L output -n
        table shaper should be on the top position 
        Chain output (policy ACCEPT):                                            
target     prot opt     source                destination           ports
shaper     all  ------  0.0.0.0/0             0.0.0.0/0             n/a
ppp-out    all  ------  0.0.0.0/0             0.0.0.0/0             n/a  
eth-out    all  ------  0.0.0.0/0             0.0.0.0/0             n/a  



      • ipchains -L shaper -n
        In this table should be rules for every IP numbers from file /etc/shaper/iplist Example: 
        Chain shaper (1 references):                                               
target     prot opt     source                destination           ports   
RETURN     tcp  ------ !192.168.1.1           192.168.1.2           * ->   *
RETURN     tcp  ------ !192.168.1.1           192.168.1.3           * ->   *
RETURN     tcp  ------ !192.168.1.1           192.168.1.4           * ->   *
RETURN     tcp  ------ !192.168.1.1           192.168.1.5           * ->   *
RETURN     tcp  ------ !192.168.1.1           192.168.1.6           * ->   *
RETURN     tcp  ------ !192.168.1.1           192.168.1.7           * ->   *
RETURN     tcp  ------ !192.168.1.1           192.168.1.8           * ->   *

      • watch ipchains -vxL shaper -n
        It will permit onto observation, do rules count quantity downloaded datas for individual IP numbers. Generally - if some number at present something download from internet, his rule has to count. If it does not make this - restart shaper.

    • for iptables:
      • iptables -L FORWARD -n --line-numbers | grep shaper
        table shaper should to have No 1 position: 
        1    shaper     all  --  0.0.0.0/0            0.0.0.0/0
      • iptables -L shaper -n
        In this table should be rules for every IP numbers from file /etc/shaper/iplist Example: 
        Chain shaper (1 references)                          
target     prot opt source               destination 
RETURN     all  -- !192.168.0.0/16       192.168.1.2 
RETURN     all  -- !192.168.0.0/16       192.168.1.3 
RETURN     all  -- !192.168.0.0/16       192.168.1.4 
RETURN     all  -- !192.168.0.0/16       192.168.1.5 
RETURN     all  -- !192.168.0.0/16       192.168.1.6 




RETURN     all  -- !192.168.0.0/16       192.168.1.7 
RETURN     all  -- !192.168.0.0/16       192.168.1.8 
RETURN     all  -- !192.168.0.0/16       192.168.1.9 
RETURN     all  -- !192.168.0.0/16       192.168.1.10

      • watch iptables -vxL shaper -n
        It will permit onto observation, do rules count quantity downloaded datas for individual IP numbers. Generally - if some number at present something download from internet, his rule has to count. If it does not make this - restart shaper.

  4. Do not change times of keeping of connections on masquerade (e.g. by command: ipchains -M -S) - concerns only owners of kernel 2.2.X (because in 2.4.x this did not happen ;). shaperd changes this parameter and onto him it controls basis, which IP number is active. Arrangement of this parameter at working shaperd can cause, that shaperd will not "saw" some connections and it will not assign bandwidth for them.


  5. Dissuade inscriptions privileged (?) IP numbers to file /etc/shaper/iplist in aim of avoidance of assigning them bandwidth. Result of such work not at all is obvious and maybe can be far from intentional. In dependence from system and version installed, on some systems something such it can get down to work and choose IP number will could take over whole link for itself. However result always will be the one - shaperd will be changed allotments very violently the rest of users. It leads this doubtless to unequal burden of link and distorts idea of applying shaperd.


  6. Colleague Grzegorz Cichowski affirmed, that in case of inscription in file /etc/shaper/shaper.Cfg parameters for 100Mbit local network interface one should to write speed 8Gbit what after count onto bits would give number in result 85899343592, which is 33 bite number and overflows size of registers of processor. To write such number one should to count it on Kbit instead of to write in bit. After count number equals 8388608 (unit Kbit). I ask to remember about count onto this unit all written speed in configuration as in command line!


  7. I ask don't edit configuration files with software working under DOS or Windows from reason no standard format of end of line (CR+LF).


  8. Do not put in empty line to configuration files.
Back Linux
@
Webmaster Grzegorz Fitrzyk