drink the sweet feeling of the colour zero

Basic Linux Bandwidth Shaping

Tags: , , , ,

This post is largely for my own personal reference.

Bandwidth shaping has traditionally been very difficult.  To truly understand it you must know a fair amount about networking.  The tools are somewhat arcane.  Fortunately, some folks have given us all a leg up by greatly simplifying the process.  It is still far from “easy peasy,” but it is also no longer the black art it once was.

I have recently had cause to configure bandwidth shaping on my edge Linux routers.  For this task I have made use of the Hierarchy Token Bucket (HTB) queuing that is part and parcel of modern Linux kernels.  I set up the HTB init script on my edge Linux routers, and need to document exactly how I installed the whole thing.  This includes the HTB Webmin module, which I use as a visual reference to tell me if I have configured the HTB text files properly.

This setup involves the following:
– HTB Init Script http://sourceforge.net/projects/htbinit/
– HTB Webmin Module http://sehier.fr/webmin-htb/
– Webmin http://www.webmin.com/
– CentOS 5.5 http://centos.org/

This setup presumes the following:

– You have setup and configured a CentOS 5.5 system
– You have configured the networking on this system to serve as a network router
– You have installed Webmin
– You have properly configured the firewall to allow access to Webmin (default port: 10000)
– You have a basic working knowledge of Linux commands (wget, chmod, chown and similar.)
– You have a basic working knowledge of Webmin (how to add modules and navigate the GUI.)

Step 1:  Install the htb.init script.

Wget my modified HTB Init script (http://www.trevorpott.com/downloads/htb/htb.init) into /etc/init.d.  Chmod it 0755 and chown root:root.  This is necessary to use my modified version because of an incompatibility in the original script’s use of the “find” command.  (-maxdepth was improperly positioned and blew up under CentOS 5.5)

The original unmodified script is available on sourceforge here.

Step 2:  Install the Webmin HTB module.

Webmin –> Webmin Configuration –> Webmin Modules
Select “third party module from” and enter http://sehier.fr/webmin-htb/webmin-htb.tar.gz
(Note: also cached here: http://www.trevorpott.com/downloads/htb/webmin-htb.tar.gz)
Select “Install Module”.

This will properly create the Webmin module directory under /etc/webmin, as well as register the module with Webmin itself.  Sadly, as this is not a .wbm, there are some bugs.

Wget the .tar.gz into /etc/webmin/ and tun “tar -zxvf” against it.  This will unpack the files into the /etc/webmin/htb directory with all the proper permissions.  You can now delete the webmin.htb.tar.gz file.

Step 3:  Setting up the config files.

Create the directory /etc/sysconfig/htb.  The directory should be chmoded to 0755 and chowned root:root.  This directory houses the files that the htb.init script will use to configure htb on your system.  A sample configuration is provided here: http://www.trevorpott.com/downloads/htb/archive.tgz

An explanation of how these files work is provided below.

Step 4: Install the DAG::Tree PERL library.

Enter the following (without the quotes) into the command line: “cpan -i Tree::DAG_Node”.  When it asks if you would like to build manually, enter “no.”

Step 5: Check the configuration.

To check your config using Webmin do the following:
Webmin –> Networking –> Hierarchy Token Bucket queuing.  This module will allow you to configure HTB as well as provides alerts if there are misconfigurations.

To check the configuration using the command line do the following:
/etc/init.d/htb.init compile

Step 6:  Start the service.

If you are satisfied with the configuration, then in Webmin do the following:
System –> Bootup and Shutdown
Select “htb.init” and click “start now and on boot.”

How the config files work:

The configuration for the HTB init script is picked up both from the naming of the config files and thier content.  To really understand how HTB works you should read the user manual.  The significantly dumbed down version is as follows:

HTB provides a method to provide bandwidth shaping using your Linux box.  You set up a “root” class that contains the total bandwidth you wish to allocate for one group of IP addresses or ports.  You then create classes which are subordinate to this root class.  Each of these classes can be (and indeed should be) guaranteed a minimum amount of bandwidth.  All classes subordinate to the root can also be configured with a ceiling.  The ceiling parameter is the maximum bandwidth that class can consume.

Should the root class have extra unconsumed bandwidth available, (because one of the other subordinate classes is not consuming it’s full allotment,) then any subordinate classes requiring bandwidth above any beyond their minimum guaranteed amount will be able to “borrow” bandwidth from another class subordinate to the same root.

Subordinate classes can for example be an IP address, a subnet or a specific class of traffic such as “all traffic to port 80.”

A working real world example (as per the provided archive.tgz) is thus:

I have a small Linux box configured with two network interfaces: eth0 and eth1. My provider offers me a 100Mbit pipe, however they charge me based on throughput usage rather than total bandwidth consumption. Thanks to this, I wish to limit my total possible throughput consumption to 15Mbit symmetrical.

For the purposes of this demo file, I have changed all the IP addresses involved to be on the /27 subnet. In the real world the subnet in use has externally addressable addresses as I am using this system to shape throughput to a subnet provided me by my ISP.

eth0 is the interface going out to my ISP.
eth1 is the interface on which my ISP-delegated subnet can be found.

By shaping traffic on the eth0 interface I can control the speed of traffic flowing from my provisioned subnet to my ISP. (Upstream traffic.)

By Shaping traffic on the eth1 interface I can control the speed of traffic flowing from my ISP to my provisioned subnet. (Downstream traffic.)

The file eth0 contains one line: default=91. This tells HTB that the default class for all unclassified traffic on eth0 will be 91. The file I have setup to define this is eth0-2:91.default_up

The file eth0-2.root_up defines the root class for eth0. The root class is the number 2. The HTB init script infers this from the filename. Everything before the dash (eth0) is the interface. Everything after the dash but before the period (2) is the class. Everything after the period (root_up) is the “friendly name” of the class.

Looking at the default file we see that is has a colon. As with the root class, everything before the dash (eth0) is the interface. Everything after the dash but before the period (2) is the class. Since this class has a colon (2:91), the script will parse this as being “class 91, subordinate to class 2.” Everything after the period (default_up) is the “friendly name” of the class.

You will notice also several files with “friendly names” consisting of three numbers. These “friendly names” are simply the last octet of the IP address that rule is defining bandwidth shaping for.

The logical hierarchy defined by the filenames and their contents is as follows:

-eth0: all unclassified traffic will use class 91
–class 2 (root_up): Maximum throughput of 15Mbit.
—class 11 (137): Guaranteed 1Mbit, Ceiling of 15Mbit, SRC
—class 21 (157): Guaranteed 5Mbit, Ceiling of 15Mbit, SRC
—class 31 (133): Guaranteed 1300Kbit, Ceiling of 15Mbit, SRC
—class 41 (132): Guaranteed 600Kbit, Ceiling of 15Mbit, SRC
—class 51 (158): Guaranteed 600Kbit, Ceiling of 15Mbit, SRC
—class 61 (144): Guaranteed 4Mbit, Ceiling of 15Mbit, SRC
—class 71 (136): Guaranteed 500Kbit, Ceiling of 15Mbit, SRC
—class 91 (default_up): Guaranteed 2Mbit, Ceiling of 15Mbit, Burst 15k

-eth1: all unclassified traffic will use class 91
–class 2 (root): Maximum throughput of 15Mbit. Burst in 15k increments.
—class 10 (137): Guaranteed 4Mbit, Ceiling of 15Mbit, DEST
—class 20 (157): Guaranteed 5Mbit, Ceiling of 15Mbit, DEST
—class 30 (133): Guaranteed 1300Kbit, Ceiling of 15Mbit, DEST
—class 40 (132): Guaranteed 600Kbit, Ceiling of 15Mbit, DEST
—class 50 (158): Guaranteed 600Kbit, Ceiling of 15Mbit, DEST
—class 60 (144): Guaranteed 1Mbit, Ceiling of 15Mbit, DEST
—class 70 (136): Guaranteed 500Kbit, Ceiling of 15Mbit, DEST
—class 90 (default): Guaranteed 2Mbit, Ceiling of 15Mbit

An example based on this configuration is that of an FTP server located at It has a guaranteed 1Mbit up and 4Mbit down. (It is mostly used for other people to send files up to us.) It can however receive or send information at up to 15Mbit, should none of the other systems on the subnet be consuming their allotments.

The rate limits for each network card are set at 15Mbit total. That rate limit will affect both upstream and downstream traffic on each NIC. While I am only defining upstream caps on my eth0 NIC and downstream caps on my eth1 NIC, this configuration effectively limits my system to 15Mbit half duplex. This is by design. I want to be able to send at 15Mbit upstream or receive information at 15Mbit upstream, but I also do not want my combined upstream and downstream to surpass 15Mbit. It is a quirk of how I am billed (95th percentile of half-duplex consumed throughput.)

Additionally, of a possible usable 29 IP addresses in this subnet, only 7 are explicitly defined in the bandwidth shaping rules above. Any servers located on other IPs within the subnet would fall under the “default” rule. This allows me to do three important things:

1) Guarantee traffic to specific computers within my subnet.
2) Cap the total bandwidth consumed by all computers to 15Mbit.
3) Force all systems not explicitly defined to obtain throughput by contention.

There you have it:  a dumbed down overview of a very basic HTB shaping setup using the HTB.init script

Cascading Webmin Groups

Tags: ,

When trying to delegate modules to users in Webmin we are sadly limited by the ability to add users to only one Webmin group at a time.  Fortunately, one can cascade groups in order to work around this limitation.  Groups can function as a superset of one another as per the following example:

I require Alice, Bob and Cassandra to be able to alter SpamAssassin’s configuration.  I can create a Webmin group called SpamAssassin_Users with access to that module and add all three users to it.  Should I require Bob and Cassandra to additionally have access to the System Logs module, I can create a second group called System_Logs_Users with access to that module.  I can make System_Logs_Users a member of the SpamAssassin_Users group and then make both Bob and Cassandra a member of System_Logs_Users.  Bob and Cassandra now have access to both SpamAssassin and System Logs whilst Alice is limited to being able to Manage SpamAsassin.

Thus simple kludge can save you a lot of time when you finally sit down to delegate modules, but if you use this trick pay close attention to group permission inheritance!

A simple spam server

Tags: , , , , , ,

I can’t afford a really pricy third-party spam filtering option.  GFI, Symantec, even Microsoft offer up some pretty robust solutions.  They are pricy though, and I don’t see why I should bother fighting that particular funding war when there are some easy solutions available for free.  In my particular environment, I run an Exchange 2010 server front-ended by a CentOS box running Sendmail, SpamAssassin, ClamAV and a few others.

The first and most important thing is to of course go get the latest and greatest CentOS.  As of the time of this write-up that would be CentOS 5.5.  Toss it in a virtual machine and install it with nothing but the bare bones.  In my case, I gave it two interfaces; one directly externally accessible, and the other on my local LAN.  (I trust iptables to keep the baddies out as much as I do any other firewall, so I see little reason to hide the spam server behind a separate firewall and port forward.)  Let’s get to the build.

0) Set up your IP addressing according to your own internal schema.  Pointing the spamserver at your internal DNS (probably your domain controller) saves you having to build extensive hosts files on the spam server.  (It will be talking to your active directory, so using your AD’s DNS is a good plan.)

1) Enable the RPMforge repo.  (https://rpmrepo.org/RPMforge/Using) I use this for the simple reason that they have a tendency to keep ClamAV significantly more up-to-date than RedHat (and thus CentOS) do.  If you don’t use RPMforge, eventually ClamAV will get so out of date it will refuse to download new definitions.  Save yourself the aggravation; use RPMforge.  (I tend to wget the latest rpm, then “yum install [rpm name] –nogpgpcheck”.  This is because CentOS doesn’t natively have RPMforge’s key available, and RPMforge keeps changing the location on their site where they store the rpm installer for the key…)

2) Install the necessary software: yum install procmail sendmail sendmail-cf sendmail-milter clam* spamass* pyzor perl-Razor-Agent

3) Download and install Webmin: RPMs are available, and certainly work well enough.

4) Disable SELinux and allow ports 10000 and 25 through the firewall, as this is what centos works on.  You can usually do this from the command line via system-config-securitylevel on a base CentOS install.  Don’t forget to restart the system after disabling SELinux!  I know that there are ways around disabling SELinux, but frankly I’m too lazy to futz with the thing.  (At some point in the future I will figure out how to get SpamAssassin and ClamAV working with SELinux enabled.)

5) Create a user called Sendmail in your Active Directory under to OU “users.”

6) Save the password for this user in a file on the spam server.  I used /etc/mail/ldap.secret

7) Log into Webmin, and under servers go to “Sendmail Mail Server.”

The following is what we are going to need to modify to get Sendmail to use ClamAV and SpamAssassin.  It will also be set up to talk to your domain controller in order to look up users when a server attempts to deliver mail.  In this way the Sendmail server will be able to reject recipients who don’t exist in your organization.  (Thus avoiding a truckload of NDRs from your exchange server.)

1) Under Webmin -> Servers -> Sendmail Mail Server -> Domain Routing (mailertable)

The mailertable tells Sendmail where to send e-mail it receives for a given domain.  In the example below, domain1.com and domain2.com are being redirected to internalmailserver.company.local.  To achieve this, click on “manually edit /etc/mail/mailertable.”  Update it to suit your configuration.

Mailertable example:
domain1.com smtp:internalmailserver.company.local
domain2.com smtp:internalmailserver.company.local

2) Under Webmin -> Servers -> Sendmail Mail Server -> Spam control (access)

This file contains a list of servers allowed to use your spam server as a relay.  While e-mail relays are generally a very bad plan, in this case they are an excellent way to scan all your outbound company e-mail.  Enter the internal IP address of your exchange server (and any other e-mail sending systems) in your organizations here.  You can then configure them to treat your spam server as a “smart host,” thus providing antiviral and antispam scanning for all outbound e-mail traffic.  To achieve this, click on “manually edit /etc/mail/access.”  Update it to suit your configuration.

Acceslist example: RELAY
mail.internalmailserver.company.local RELAY

3) Under Webmin -> Servers -> Sendmail Mail Server -> Relay Domains
Enter a list (separated by carriage returns) of all domains that you will be handling internally and which you wish to pass through this spam server.

Relay Domains example:

4) Under Webmin -> Servers -> Sendmail Mail Server -> Sendmail M4 Configuration

This is the heart of configuring Sendmail.  Most of the default configuration provided by CentOS 5.5 is good, but we need to add a few goodies to get it working the way we want it.

The first and most important thing is the setting LOCAL_DOMAIN(`’).  There is a big push right now by e-mail administrators the world over to require reverse DNS.  The long story short is that the hostname of your spam server (as your incoming and outgoing mail point) absolutely must match the reverse DNS of the IP address assigned to it.  That reverse DNS also needs to contain the word “mail.”  So the hostname of your spamserver should be something akin to mail.domain.com, and the reverse DNS on your external IP address provided you by your ISP should also read mail.domain.com.

In this vein, it is a good idea to set LOCAL_DOMAIN(`’) to LOCAL_DOMAIN(`mail.domain.com’).  This means your spamserver would always accept mail for “mail.domain.com” without forwarding it to your exchange server (an odd requirement that some e-mail administrators have begun to put into place.)  It still allows you to forward mail bound for domain.com internally.
Keep an out for this command: DAEMON_OPTIONS(`Port=smtp,Addr=, Name=MTA’).  Toss a dnl #  in front of it if you want your sendmail to listen on any addresses other than!

I also tend to dnl # out EXPOSED_USER(`root’) and FEATURE(`accept_unresolvable_domains’) for sanity reasons.

The rest of the commands I won’t go into too much detail on; if you are really curious there is plenty of documentation available online as to their specific functions.  If you are reading this page, I trust you are capable of spotting where in the configuration you should be changing “domain.com” and “company.local” style commands to suit your configuration.

define(`LUSER_RELAY’,`error:5.1.1:”550 User unknown”‘)dnl
INPUT_MAIL_FILTER(`clamav-milter’, `S=/var/clamav/clmilter.socket, T=S:4m;R:4m’)dnl
INPUT_MAIL_FILTER(`spamassassin’, `S=:/var/run/spamass.sock, F=,T=C:15m;S:4m;R:4m;E:10m’)dnl
define(`confINPUT_MAIL_FILTERS’, `clamav-milter,spamassassin’)dnl
FEATURE(`ldap_routing’,, `ldap -1 -T<TMPF> -v mail -k proxyAddresses=SMTP:%0′, `bounce’)dnl
define(`confLDAP_DEFAULT_SPEC’,`-h “domaincontroller.company.local” -d “CN=sendmail,CN=Users,DC=company,DC=local” -M simple -P /etc/mail/ldap-secret -b “DC=company,DC=local”‘)dnl

Once you have finished this, go save and rebuild the Sendmail configuration.  It’s a good plan to restart Sendmail at this point to see if it blows up.  Remember that Sendmail is really grouchy if you have an extra carriage return, or forget a ` or a ‘.

For SpamAssassin configuration, first go to Webmin -> Servers -> SpamAssassin Mail Filter -> Setup Procmail For SpamAssassin and enable SpamAssassin.

Next stop is Webmin -> Servers -> SpamAssassin Mail Filter and modify to your heart’s desire.  I generally change the setting “Prepend text to Subject: header” to read [SPAM ASSASSIN DETECTED SPAM].  This then allows me to set either an Outlook rule or an Exchange -> Hub Transport -> Transport rule.

In the case of a local Outlook rule each client must be individually configured to deal with the [SPAM ASSASSIN DETECTED SPAM] in the subject line of “spam” e-mails.  (I usually have them directed to the “Junk-Email” folder.)

In the case of an Exchange -> Hub Transport -> Transport rule, I usually set exchange to assign anything with [SPAM ASSASSIN DETECTED SPAM] in the subject line to a Spam Confidence Level (SCL) of 7.  If  you want to enable SCL junk filtering and set your own SCL levels, you will need some Exchange PowerShell commands.  Google can tell you more.  http://msexchangeteam.com/archive/2009/11/13/453205.aspx is a good article to read as well.

Set-ContentFilterConfig -SCLDeleteEnabled $true -SCLDeleteThreshold 9
Set-ContentFilterConfig -SCLRejectEnabled $true -SCLRejectThreshold 8
Set-OrganizationConfig -SCLJunkEnabled $true -SCLJunkThreshold 7

Go to Exchange -> Hub Transport -> Anti-Spam -> Content Filtering.  Enable it, and uncheck any boxes except “Delete Messages that have an SCL greater than or equal to.”  The rationale behind this is that the SpamAssassin server is doing all the heavy filtering.  If you allow Exchange to reject mails, you are going to end up with a mess of rejection NDRs that will pile up and go nowhere.  Similarly, under Exchange -> Hub Transport -> Remote Domains -> Default (*) I really recommend disabling non-delivery reports.  There is a growing trend amongst email administrators to not accept mail from domains that send NDRs, as NDRs are being used by spammers as a vector to get spam into people’s e-mail boxes.

run freshclam and sa-update from the command line to get ClamAV and SpamAssassin updated to the latest definitions.

Go into Webmin -> System -> Bootup and Shutdown.  Make sure important things like ClamAV-Milter, SpamAssassin and Sendmail are all set to start on boot (and are currently running.)

That’s it!  If you’ve done it right, then you should now have a CentOS box capable of receiving e-mail from the internet, scanning it for viruses and Spam, and forwarding it on to your exchange server.  The exchange server itself can be configured with junk-filtering properties, adding a second layer of protection.  (Though in truth I’ve not needed it: SpamAssassin does the job just fine, and better than Exchange’s native capabilities.)

Linux Routers Gone Wild (Introduction)

Tags: , , , , ,

I have recently embarked upon a difficult professional journey.  The larger part of this journey is in fact an attempt to secure my network and slowly, inexorably retire as much Microsoft software from service as possible.  The reasons for this lie largely in the complexity of Microsoft licensing; I am often beset by so many IT projects that it is honestly a nightmare attempting to comprehend the plethora of licensing options and caveats.  Trying to make sure our company remains in compliance is itself almost a full time job.

The solution to this is simple: cut back on as much Microsoft software as is humanely possible.  There are naturally some fairly enormous barriers to this concept.  The first being that there is simply no way we are (ever) going to be able to ditch Microsoft on the desktop.  There is simply too much industry-specific software we are totally reliant on for this to be anything but a midsummer night’s dream.  To manage these desktops, I need a directory and something that will handle group policy like templates.  After much searching and pondering the simple reality is that Microsoft’s Active Directory is the best bang for my buck in this department, and so there is no reason to abandon it.  (I should state for the record however that Novell’s offerings an unbelievably close second.)

The second obstacle is not a hardware or software limitation, but rather one of wetware.  The wetware, (which will remain nameless,) ultimately responsible for accepting or rejecting my various schemes and proposals consists of two units.  The first unit is logical, rational and driven by nothing more than sounds business rationale.  If you can make a solid business case for something, one of the two decision making wetware units will be easily won over.  Unfortunately, this wetware unit has exceptionally limited IT knowledge; when my recommendations clash with those of the second decision making wetware unit, issues can arise.

The second decision making unit in question is rather less approachable than the first.  Though remarkably intelligent, this unit remains deeply wedded to all things Microsoft and has what I consider to be an incredibly dangerous fascination with whatever happens to be the newest technology of the day.  As a born and bred technology geek, I truly understand the “gee whiz” factor shiny new kit can bring.  As someone who goes to work a loyal company man and puts aside everything except my job, I can’t and won’t let my employer risk the business on untested or questionable gear.  (Let several someone elses walk through that minefield first.)

The wetware obstacles to reducing the corporate Microsoft overhead, (and with it the licensing burdens) are thusly formidable.  An unfortunate amount of my job has devolved into simply playing the politics necessary to be allowed to implement the right solutions for our requirements and budget.  In many cases I actually have to purchase and implement first, and then inform people of it later if I feel it corporately critical that a project be accomplished without being tied up with internal political infighting and negotiations for six to eight months.

This year I am removing Microsoft’s Internet and Security Acceleration (ISA) Server from my organisation.  Like the use of LAMP webservers, LACS e-mail sanitisation servers and the slow introduction of Linux fileserver, this is the story of nibbling at the edges of a Microsoft network with tactical implementations of Linux systems.

Redhat-based thin client tips

Tags: , ,

All of the below tips are for Redhat-based systems.  They are really minor, easily searchable items that I have found useful to remember.  I have found them of value when configuring Redhat/Fedora systems as thin clients.   We use Fedora 10 and CentOS 5, usually to turn old hardware into something that will RDP to a Windows virtual machine.  The people using them don’t want to know how it works, just that when they double click on the icon, they get a windows desktop.  The tips here make such a configuration easier to administer.

System won’t show it’s hostname in a Windows-powered DHCP:

Edit /etc/sysconfig/network-scripts/ifcfg-eth0  (Or eth1, eth2, etc.)
Add the line DHCP_HOSTNAME = [system_hotname]  (Without the brackets.)

Have gnome auto-logon a non-root user:

Edit /etc/gdm/custom.conf
Add the following lines:

TimedLoginEnable = true
TimedLogin = [username]  (Without the brackets.)
TimedLoginDelay = 0  (Salt to taste.)

Exclude a package from yum updates:

In this example, I exclude tsclient, because I prefer the version 1 client to version 2.
(This is because version 2 forces you to store an RDP password, which is terrible for our thin-client purposes.)

Edit /etc/yum.conf
Add the line exclude = tsclient

Note: You can search older RPMs on http://rpm.pbone.net/

Set up vino in gnome so that you can remote administer a system.
(This is a good configuration for auto-logged-on thin clients.)

yum install vnc vnc-server libvncserver vino

Go to System > Preferences > Internet & Network > Remote Desktop
– Allow others to view  (This enables vino.)
– Allow others to control  (So that you can manipulate the system.)
– Do not ask for confirmation  (Entirely up to you.)
– Require enter password  (Enter a password.)

© 2009 drink the sweet feeling of the colour zero. All Rights Reserved.

This blog is powered by the Wordpress platform and beach rentals.