Arming your Arm64 RockChip Gentoo against the hordes of Mindless Bots

About a week ago I was notified in #trilema that my blog was unreachable. After being convinced to resist the urge to engage in the usual shamanism, I set off to first define the problem properly.

So, in an attempt to diagnose I cut the patient back open and stuck my hands in the chest cavity to feel around. Let's look closely at the apache error logs1 :

[Tue Jun 26 05:42:31.703938 2018] [mpm_prefork:error] [pid 12260] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

Looks like apache is hitting the maximum number of workers. Let's see what netstat is saying:

pizrk003 htdocs # netstat -an | egrep ':80|:443' | grep ESTABLISHED | awk '{print $5}' | grep -o -E "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | sort -n | uniq -c | sort -nr

187 173.254.216.66

Here we see one IP address is slamming my blog with 187 unique requests (all ESTABLISHED state). A quick look at the apache config file confirms that this many requests indeed exceeds the max limit:

pizrk003 htdocs # cat /etc/apache2/modules.d/00_mpm.conf | grep 'MaxRequestWorkers'

# MaxRequestWorkers: Maximum number of child processes to serve requests

MaxRequestWorkers       150

MaxRequestWorkers       150

MaxRequestWorkers       150

MaxRequestWorkers       150

MaxRequestWorkers       150

Okay, so now we have the problem formally defined as "Apache cannot serve requests due to MaxRequestWorkers limit being exceeded".

Now to fix. Let's first try the obvious treatment of simply increasing the MaxRequestWorkers in the apache config to something like 256. Restart apache, and then netstat again:

pizrk003 htdocs # netstat -an | egrep ':80|:443' | grep ESTABLISHED | awk '{print $5}' | grep -o -E "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | sort -n | uniq -c | sort -nr

256 94.230.208.147

Sadly, no dice. Now the spamola has simply increased to hit the new max (and from a new ip, no less). Looks like we will need to do something to limit spam requests from getting through. Unfortunately for us, a firewall solution is not readily available as iptables does not function "out-of-the-box" on this thing. Nevertheless, there might be a few things we can try at the apache layer for the time-being.

Mod_evasive

The module "mod_evasive" works by giving apache the functionality to automatically deny any ip address that requests X resource more than N times per I interval. Here's how I got this up and running:

First, add "www-apache/mod_evasive-1.10.1 ~arm64" to your package.accept_keywords file (or directory, if you prefer it that way)

echo '=www-apache/mod_evasive-1.10.1 ~arm64' >> /etc/portage/package.accept_keywords

For my fellow Gentoo (and arm architecture) n00bs out there, let me take a short interlude to explain what I've come to understand about how keywords work in Gentoo, and why we need to do the above in our case2.

From https://wiki.gentoo.org/wiki/KEYWORDS:

In an ebuild the KEYWORDS variable informs in which architectures the ebuild is stable or still in testing phase.

And from https://wiki.gentoo.org/wiki/ACCEPT_KEYWORDS:

The ACCEPT_KEYWORDS variable informs the package manager which ebuilds' KEYWORDS values it is allowed to accept.

...

Stable and unstable keywords

The default value of most profiles' ACCEPT_KEYWORDS variable is the architecture itself, like amd64 or arm.

In these cases, the package manager will only accept ebuilds whose KEYWORDS variable contains this architecture.

If the user wants to be able to install and work with ebuilds that are not considered production-ready yet, they can add the same architecture but with the ~ prefix to it, like so:

FILE /etc/portage/make.conf

ACCEPT_KEYWORDS="~amd64"

This, I think, is important for us to understand because mod_evasive is definitely not listed as stable for the arm64 architecture. That being said, this Gentoo's make.conf had the value of "**" for the ACCEPT_KEYWORDS, which, I think may already tell Portage to pull in untested/unstable packages and may also override whatever you set in package.accept_keywords. In other words, this may be an exercise in redundancy, but at least we all learned something!3.

Interlude over; let's keep moving. After getting your keywords situated, let's slap the "mod_evasive" USE flag in make.conf and emerge:

pizrk003 lobbes # emerge -av www-apache/mod_evasive

[...SNIP...]

These are the packages that would be merged, in order:

Calculating dependencies... done!

[ebuild  N     ] www-apache/mod_evasive-1.10.1-r1::gentoo  20 KiB

Total: 1 package (1 new), Size of downloads: 20 KiB

Would you like to merge these packages? [Yes/No] Yes

>>> Verifying ebuild manifests

>>> Emerging (1 of 1) www-apache/mod_evasive-1.10.1-r1::gentoo

>>> Installing (1 of 1) www-apache/mod_evasive-1.10.1-r1::gentoo

>>> Recording www-apache/mod_evasive in "world" favorites file...

>>> Jobs: 1 of 1 complete                           Load avg: 0.41, 0.12, 0.04

>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.

* GNU info directory index is up-to-date.

[...SNIP...]

Seems good. And looky here, we now have a mod_evasive config file to edit:

pizrk003 lobbes # more /etc/apache2/modules.d/10_mod_evasive.conf

<IfDefine EVASIVE>

LoadModule evasive_module modules/mod_evasive.so

DOSHashTableSize 3097

DOSPageCount 5

DOSSiteCount 100

DOSPageInterval 2

DOSSiteInterval 2

DOSBlockingPeriod 10

# Set here an email to notify the DoS to someone

# (here is better to set the server administrator email)

DOSEmailNotify root

# Uncomment this line if you want to execute a specific command

# after the DoS detection

#DOSSystemCommand    "su - someuser -c '/sbin/... %s ...'"

# Specify the desired mod_evasive log location

DOSLogDir /var/log/apache2/evasive

# WHITELISTING IP ADDRESSES

# IP addresses of trusted clients can be whitelisted to insure they are never

# denied.  The purpose of whitelisting is to protect software, scripts, local

# searchbots, or other automated tools from being denied for requesting large

# amounts of data from the server.

#DOSWhitelist    127.0.0.*

#DOSWhitelist    172.16.1.*

</IfDefine>

# vim: ts=4 filetype=apache

Tweak this how you like. Here's a quick rundown on the interesting knobs:

DOSPageCount and DOSSiteCount: Amount of requests allowed per interval to unique pages and the site as a whole, respectively.

DOSPageInterval and DOSSiteInterval: Sets the interval (in seconds) for unique pages and the site as a whole, respectively.

DOSBlockingPeriod: Amount of time (in seconds) for the ip to be denied.

DOSWhitelist: Pretty self-explanatory, but allow specific IP addresses from being exempt from the above restrictions.

Let's also edit the apache config4 to load the mod_evasive module. Add  "-D EVASIVE" to the "APACHE2_OPTS" line:

APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP -D EVASIVE"

Now restart apache and do some stress-testing to confirm everything's working. Tweak your configuration as necessary.

Mod_limitipconn

While mod_evasive has utility, it didn't really address my particular problem, which was a single ip address exceeding the max amount of apache workers. Since the ips spamming my box were sending their requests at reasonable intervals5 mod_evasive wasn't much use unless I wanted to also block legit traffic. Enter mod_limitipconn. As the name suggests, this module gives apache the ability to limit unique ip addresses to a set number of max requests. Let's get it emerged and configured.

Like with mod_evasive, add the required bits to your make.conf, apache config, and package.keywords for mod_limitipconn:

echo '=www-apache/mod_limitipconn-0.24 ~arm64' >> /etc/portage/package.accept_keywords

=======================

cat /etc/portage/make.conf

[...SNIP...]

USE="mysql cgi php apache2 fpm apachetop mod_evasive mod_limitipconn -gtk3 -avahi -gnome -tls-heartbeat -gpm -X -libnotify -consolekit offensive ufw -dbus -bluetooth -systemd -wayland$

[...SNIP...]

=======================

cat /etc/conf.d/apache2

[...SNIP...]

APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP -D EVASIVE -D LIMITIPCONN"

[...SNIP...]

Now, I also had to add the following package mask to force Portage to pull in the earlier version of this thing6:

pizrk003 lobbes # more /etc/portage/package.mask/mod_limitipconn

>www-apache/mod_limitipconn-0.24

Now, emerge -av www-apache/mod_limitipconn-0.24. It should go smoothly.

Important: for this module to function, you will also need mod_status enabled with the "ExtendedStatus On". On this box, mod_status was already installed7, so we just need to define it by adding "-D STATUS" to the "APACHE2_OPTS=" line in the apache config:

"APACHE2_OPTS="-D DEFAULT_VHOST -D INFO -D SSL -D SSL_DEFAULT_VHOST -D LANGUAGE -D PHP -D EVASIVE -D LIMITIPCONN -D STATUS"

Now, time to configure mod_limitipconn by editing /etc/apache2/modules.d/27_mod_limitipconn.conf. I found this to be slightly tricky. The following worked for me, so feel free to use it as a general guide8:

<IfDefine LIMITIPCONN>

LoadModule limitipconn_module modules/mod_limitipconn.so

<Location /var/www/localhost/htdocs/>

MaxConnPerIP 10

# exempting images from the connection limit is often a good

# idea if your web page has lots of inline images, since these

# pages often generate a flurry of concurrent image requests

NoIPLimit blog/images/*

</Location>

<Location /mp3>

MaxConnPerIP 1

# In this case, all MIME types other than audio/mpeg and video*

# are exempt from the limit check

OnlyIPLimit audio/mpeg video

</Location>

<IfModule mod_limitipconn.c>

# Set a server-wide limit of 10 simultaneous downloads per IP,

# no matter what.

MaxConnPerIP 10

</IfModule>

</IfDefine>

Let's see if it works. One quick restart of apache and then netstat:

pizrk003 htdocs # netstat -an | egrep ':80|:443' | grep ESTABLISHED | awk '{print $5}' | grep -o -E "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | sort -n | uniq -c | sort -nr

25 197.231.221.211

Mwahah, 25 much better than 256, huh? The 'raw' netstat I issued right before this was showing some 300 connection attempts from this same ip, but with the status of "TIME_WAIT" and "CLOSE_WAIT" instead of "ESTABLISHED". Sure enough, a few seconds later another netstat reveals:

pizrk003 htdocs # netstat -an | egrep ':80|:443' | grep ESTABLISHED | awk '{print $5}' | grep -o -E "([0-9]{1,3}[\.]){3}[0-9]{1,3}" | sort -n | uniq -c | sort -nr

Nothing! The robots have stopped knocking.

Hopefully this will be of use to you if you are trying to carve a webserver out of the Arm64 RockChip Gentoo. Now, if I can just get iptables working...

  1. located in /var/log/apache2/error_log []
  2. Gentoo experts out there, plox to correct me here []
  3. maybe? []
  4. located at /etc/conf.d/apache2 []
  5. one per second, roughly []
  6. as the "0.24-r2" version dun seem to work []
  7. see the config in  /etc/apache2/modules.d/00_mod_status.conf []
  8. if you want more info, please see this handy README I found: http://dominia.org/djao/limitipconn2-README  []

2 Responses to “Arming your Arm64 RockChip Gentoo against the hordes of Mindless Bots”

  1. Name (required) says:

    That is a lot of entirely superfluous grep-on-grep. Sheesh.

    netstat -nt, awk has grep built in, take it from there.

  2. [...] There are now several guides to setting up MP-WP. The most recent addition by billymg offers the clearest instructions for getting MP-WP running on a Rockchip and alleviating certain persistent pains. The esthlos guide remains solid in the general case. Lobbes covers yet more interesting issues to consider. [...]

Leave a Reply