drink the sweet feeling of the colour zero

Basic Linux Bandwidth Shaping

Tags: , , , ,

This post is largely for my own personal reference.

Bandwidth shaping has traditionally been very difficult.  To truly understand it you must know a fair amount about networking.  The tools are somewhat arcane.  Fortunately, some folks have given us all a leg up by greatly simplifying the process.  It is still far from “easy peasy,” but it is also no longer the black art it once was.

I have recently had cause to configure bandwidth shaping on my edge Linux routers.  For this task I have made use of the Hierarchy Token Bucket (HTB) queuing that is part and parcel of modern Linux kernels.  I set up the HTB init script on my edge Linux routers, and need to document exactly how I installed the whole thing.  This includes the HTB Webmin module, which I use as a visual reference to tell me if I have configured the HTB text files properly.

This setup involves the following:
– HTB Init Script http://sourceforge.net/projects/htbinit/
– HTB Webmin Module http://sehier.fr/webmin-htb/
– Webmin http://www.webmin.com/
– CentOS 5.5 http://centos.org/

This setup presumes the following:

– You have setup and configured a CentOS 5.5 system
– You have configured the networking on this system to serve as a network router
– You have installed Webmin
– You have properly configured the firewall to allow access to Webmin (default port: 10000)
– You have a basic working knowledge of Linux commands (wget, chmod, chown and similar.)
– You have a basic working knowledge of Webmin (how to add modules and navigate the GUI.)

Step 1:  Install the htb.init script.

Wget my modified HTB Init script (http://www.trevorpott.com/downloads/htb/htb.init) into /etc/init.d.  Chmod it 0755 and chown root:root.  This is necessary to use my modified version because of an incompatibility in the original script’s use of the “find” command.  (-maxdepth was improperly positioned and blew up under CentOS 5.5)

The original unmodified script is available on sourceforge here.

Step 2:  Install the Webmin HTB module.

Webmin –> Webmin Configuration –> Webmin Modules
Select “third party module from” and enter http://sehier.fr/webmin-htb/webmin-htb.tar.gz
(Note: also cached here: http://www.trevorpott.com/downloads/htb/webmin-htb.tar.gz)
Select “Install Module”.

This will properly create the Webmin module directory under /etc/webmin, as well as register the module with Webmin itself.  Sadly, as this is not a .wbm, there are some bugs.

Wget the .tar.gz into /etc/webmin/ and tun “tar -zxvf” against it.  This will unpack the files into the /etc/webmin/htb directory with all the proper permissions.  You can now delete the webmin.htb.tar.gz file.

Step 3:  Setting up the config files.

Create the directory /etc/sysconfig/htb.  The directory should be chmoded to 0755 and chowned root:root.  This directory houses the files that the htb.init script will use to configure htb on your system.  A sample configuration is provided here: http://www.trevorpott.com/downloads/htb/archive.tgz

An explanation of how these files work is provided below.

Step 4: Install the DAG::Tree PERL library.

Enter the following (without the quotes) into the command line: “cpan -i Tree::DAG_Node”.  When it asks if you would like to build manually, enter “no.”

Step 5: Check the configuration.

To check your config using Webmin do the following:
Webmin –> Networking –> Hierarchy Token Bucket queuing.  This module will allow you to configure HTB as well as provides alerts if there are misconfigurations.

To check the configuration using the command line do the following:
/etc/init.d/htb.init compile

Step 6:  Start the service.

If you are satisfied with the configuration, then in Webmin do the following:
System –> Bootup and Shutdown
Select “htb.init” and click “start now and on boot.”

How the config files work:

The configuration for the HTB init script is picked up both from the naming of the config files and thier content.  To really understand how HTB works you should read the user manual.  The significantly dumbed down version is as follows:

HTB provides a method to provide bandwidth shaping using your Linux box.  You set up a “root” class that contains the total bandwidth you wish to allocate for one group of IP addresses or ports.  You then create classes which are subordinate to this root class.  Each of these classes can be (and indeed should be) guaranteed a minimum amount of bandwidth.  All classes subordinate to the root can also be configured with a ceiling.  The ceiling parameter is the maximum bandwidth that class can consume.

Should the root class have extra unconsumed bandwidth available, (because one of the other subordinate classes is not consuming it’s full allotment,) then any subordinate classes requiring bandwidth above any beyond their minimum guaranteed amount will be able to “borrow” bandwidth from another class subordinate to the same root.

Subordinate classes can for example be an IP address, a subnet or a specific class of traffic such as “all traffic to port 80.”

A working real world example (as per the provided archive.tgz) is thus:

I have a small Linux box configured with two network interfaces: eth0 and eth1. My provider offers me a 100Mbit pipe, however they charge me based on throughput usage rather than total bandwidth consumption. Thanks to this, I wish to limit my total possible throughput consumption to 15Mbit symmetrical.

For the purposes of this demo file, I have changed all the IP addresses involved to be on the /27 subnet. In the real world the subnet in use has externally addressable addresses as I am using this system to shape throughput to a subnet provided me by my ISP.

eth0 is the interface going out to my ISP.
eth1 is the interface on which my ISP-delegated subnet can be found.

By shaping traffic on the eth0 interface I can control the speed of traffic flowing from my provisioned subnet to my ISP. (Upstream traffic.)

By Shaping traffic on the eth1 interface I can control the speed of traffic flowing from my ISP to my provisioned subnet. (Downstream traffic.)

The file eth0 contains one line: default=91. This tells HTB that the default class for all unclassified traffic on eth0 will be 91. The file I have setup to define this is eth0-2:91.default_up

The file eth0-2.root_up defines the root class for eth0. The root class is the number 2. The HTB init script infers this from the filename. Everything before the dash (eth0) is the interface. Everything after the dash but before the period (2) is the class. Everything after the period (root_up) is the “friendly name” of the class.

Looking at the default file we see that is has a colon. As with the root class, everything before the dash (eth0) is the interface. Everything after the dash but before the period (2) is the class. Since this class has a colon (2:91), the script will parse this as being “class 91, subordinate to class 2.” Everything after the period (default_up) is the “friendly name” of the class.

You will notice also several files with “friendly names” consisting of three numbers. These “friendly names” are simply the last octet of the IP address that rule is defining bandwidth shaping for.

The logical hierarchy defined by the filenames and their contents is as follows:

-eth0: all unclassified traffic will use class 91
–class 2 (root_up): Maximum throughput of 15Mbit.
—class 11 (137): Guaranteed 1Mbit, Ceiling of 15Mbit, SRC
—class 21 (157): Guaranteed 5Mbit, Ceiling of 15Mbit, SRC
—class 31 (133): Guaranteed 1300Kbit, Ceiling of 15Mbit, SRC
—class 41 (132): Guaranteed 600Kbit, Ceiling of 15Mbit, SRC
—class 51 (158): Guaranteed 600Kbit, Ceiling of 15Mbit, SRC
—class 61 (144): Guaranteed 4Mbit, Ceiling of 15Mbit, SRC
—class 71 (136): Guaranteed 500Kbit, Ceiling of 15Mbit, SRC
—class 91 (default_up): Guaranteed 2Mbit, Ceiling of 15Mbit, Burst 15k

-eth1: all unclassified traffic will use class 91
–class 2 (root): Maximum throughput of 15Mbit. Burst in 15k increments.
—class 10 (137): Guaranteed 4Mbit, Ceiling of 15Mbit, DEST
—class 20 (157): Guaranteed 5Mbit, Ceiling of 15Mbit, DEST
—class 30 (133): Guaranteed 1300Kbit, Ceiling of 15Mbit, DEST
—class 40 (132): Guaranteed 600Kbit, Ceiling of 15Mbit, DEST
—class 50 (158): Guaranteed 600Kbit, Ceiling of 15Mbit, DEST
—class 60 (144): Guaranteed 1Mbit, Ceiling of 15Mbit, DEST
—class 70 (136): Guaranteed 500Kbit, Ceiling of 15Mbit, DEST
—class 90 (default): Guaranteed 2Mbit, Ceiling of 15Mbit

An example based on this configuration is that of an FTP server located at It has a guaranteed 1Mbit up and 4Mbit down. (It is mostly used for other people to send files up to us.) It can however receive or send information at up to 15Mbit, should none of the other systems on the subnet be consuming their allotments.

The rate limits for each network card are set at 15Mbit total. That rate limit will affect both upstream and downstream traffic on each NIC. While I am only defining upstream caps on my eth0 NIC and downstream caps on my eth1 NIC, this configuration effectively limits my system to 15Mbit half duplex. This is by design. I want to be able to send at 15Mbit upstream or receive information at 15Mbit upstream, but I also do not want my combined upstream and downstream to surpass 15Mbit. It is a quirk of how I am billed (95th percentile of half-duplex consumed throughput.)

Additionally, of a possible usable 29 IP addresses in this subnet, only 7 are explicitly defined in the bandwidth shaping rules above. Any servers located on other IPs within the subnet would fall under the “default” rule. This allows me to do three important things:

1) Guarantee traffic to specific computers within my subnet.
2) Cap the total bandwidth consumed by all computers to 15Mbit.
3) Force all systems not explicitly defined to obtain throughput by contention.

There you have it:  a dumbed down overview of a very basic HTB shaping setup using the HTB.init script

© 2009 drink the sweet feeling of the colour zero. All Rights Reserved.

This blog is powered by the Wordpress platform and beach rentals.