Apr 132014
 

As you should know by now CentOS uses Yum to install and update packages but sometime you need packages that are not available as standard, one way to get these packages is to use 3rd party repositories which is what i do.

A note on CentOS packages
“As with all packages in CentOS, the version numbers of released software will not change over the life time of a CentOS product, i.e. CentOS 5.0 contained PHP 5.1.6 and that is the point version PHP will stay at for the life time of CentOS 5. Security patches and bug fixes are back-ported into the shipped version. See here for details: Backporting Security Fixes

You can get a list of 3rd party repositories here.
I mainly use the following extra repositories, Remi Collet Repository which also requires Extra Packages for Enterprise Linux (EPEL).

If you are considering using a 3rd Party Repository, then you should seriously consider how to prevent unintended ‘updates’ from these side archives from over-writing some core part of CentOS. One approach is to only enable these archives from time to time, and generally leave them disabled.

The Remi repository has a page detailing how to setup and use their repository.
Scroll down the page until you find “Enterprise Linux 6 (with EPEL)” and follow the instructions
here’s the quick version but check for updates

wget http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
rpm -Uvh remi-release-6*.rpm epel-release-6*.rpm

This will install both repository’s (remi and epel). The repository’s are not enabled when install (enabled=0). You need to enable them when you need it, for example
yum --enablerepo=remi install php*

One last note, always test everything (preferable not on a live production machine) and ensure you have a backup when using 3rd party packages as sometime a simple update can have very bad side effects.

Apr 112014
 

up until now i have been manually blocking ip’s that attack my server but by the time i see them the attacks have normally finished but after the last big attack on my email server (some 35,000 attempts) i decided to find a way to automate the blocking. After a bit of research i decided to setup Fail2ban and here’s how i did it.

As i use a 3rd party repostories – EPEL i can just use yum to install it

yum install fail2ban

once installed i just needed to change the configuration to my liking, the config files can be found at “/etc/fail2ban”

first i edit “/etc/fail2ban/fail2ban.conf” and ensure the “logtarget” is set correctly

logtarget = /var/log/fail2ban.log

The default behaviour of fail2ban is configured in the file “/etc/fail2ban/jail.conf”. There’s a [DEFAULT] section that applies to all other sections unless the default options are overridden in the other sections.

I explain some of the configuration here:

ignoreip: This is a space-separated list of IP addresses that cannot be blocked by fail2ban.
bantime: Time in seconds that a host is blocked if it was caught by fail2ban (600 seconds = 10 minutes).
maxretry: Max. number of failed login attempts before a host is blocked by fail2ban.
filter: Refers to the appropriate filter file in “/etc/fail2ban/filter.d”.
logpath: The log file that fail2ban checks for failed login attempts.

so i edit “/etc/fail2ban/jail.conf” and add my ip to “ignoreip”.
then i just need to configure the jails i want to use, here’s my ssh jail

[ssh-iptables]
enabled  = true
filter   = sshd
action   = iptables-multiport[name=SSH, port="ssh, 4564"]
           sendmail-whois[name=SSH, dest=root, sender=fail2ban@server.com]
logpath  = /var/log/secure
maxretry = 3

Don’t forget to change the port to what ever port your ssh runs on and also set the “sender” and “dest” to your email.

I use a couple of other jails/filters which i’ll show you how i configured them but first i’ll show you how to start and check its running.

start fail2ban

/etc/init.d/fail2ban start

now check “/var/log/fail2ban.log” and make sure there’s no errors.
you can also check the rules are in iptables

iptables -L 

now as i said i use a couple of custom filters here’s how i did them.
Create the filter file “/etc/fail2ban/filter.d/dovecot-pop3imap.conf” and add

[Definition]
failregex = (?: dovecot: pop3-login|imap-login): (?:Authentication failure|Aborted login \(auth failed|Aborted login).*rip=(<HOST>),.*
ignoreregex =

note: the failregex may need changing to suit your system.

now add the following to “/etc/fail2ban/jail.conf”

[dovecot-pop3imap]
enabled = true
filter = dovecot-pop3imap
action = iptables-multiport[name=dovecot-pop3imap, port="110,143,995,993,25,465,587"]
    sendmail-whois[name=dovecot-pop3imap, dest=root, sender=fail2ban@server.com]
logpath = /var/log/maillog
maxretry = 5
findtime = 600
bantime = 3600

then just restart fail2ban

/etc/init.d/fail2ban restart

you can create all sorts of custom jails/filters just google for other ideas.

Apr 102014
 

Now that i have a basic server running i wanna setup Munin so i can monitor the server. Munin is a networked resource monitoring tool that can help analyse resource trends and “what just happened to kill our performance?” problems. It is designed to be plug and play. A default installation provides a lot of graphs with almost no work.

Depending on what repositories you use (i mainly use remi and el6) you maybe able to use yum to install.

yum install munin-node munin

if not grab the source and follow the instructions here.

Once installed you need to change a few settings to your liking, the config files are found in “/etc/munin”. Munin has a master/node architecture in which the master connects to all the nodes at regular intervals and asks them for data this is very useful when you got more than server.

Ok basic settings need to setup the master, edit “/etc/munin/munin.conf”

# The next three variables specifies where the location of the RRD
# databases, the HTML output, and the logs, severally.  They all
# must be writable by the user running munin-cron.
dbdir  /var/lib/munin  #RRD databases
htmldir  /var/www/munin  #HTML output (change to whatever your website uses)
logdir  /var/log/munin  #log files
rundir  /var/run/munin

# Where to look for the HTML templates
tmpldir  /etc/munin/templates

# a simple host tree
[localhost]
    address 127.0.0.1
    use_node_name yes

now basic settings for the node, edit “/etc/munin/munin-node.conf”

log_level 4
log_file /var/log/munin/munin-node.log
pid_file /var/run/munin/munin-node.pid
background 1
setseid 1

user root
group root
setsid yes

# Regexps for files to ignore
ignore_file ~$
ignore_file \.bak$
ignore_file %$
ignore_file \.dpkg-(tmp|new|old|dist)$
ignore_file \.rpm(save|new)$
ignore_file \.pod$

# A list of addresses that are allowed to connect.  This must be a
# regular expression, due to brain damage in Net::Server, which
# doesn't understand CIDR-style network notation.  You may repeat
# the allow line as many times as you'd like
allow ^127\.0\.0\.1$
allow ^192\.168\.0\.200$

# Which address to bind to;
host *
# And which port
port 4949

Now restart munin-node so it can use the changes you made
/etc/init.d/munin-node restart

Now wait 10 minutes so it can generate some data then visit the webpage to see the results, i store mine in a directory just outside the webserver directory and use “Document directory aliases” in apache so its only available on my domain and not anywhere else.
so visit “http://192.168.0.200/munin/” (change to your setting) and you should see something similar to this
munin
Now click on one of the names, I’ll pick “system”
You should see some graphs like this (obviously yours won’t be all the way across yet)
munin2
If you don’t see any graphs check the log files for any errors, there should be 5 different log files,
first check “/var/log/munin/munin-node.log” and fix any errors.
Once its all working you now have graphs monitoring your server, there is loads of extra plugins you can add to munin depending what you want to monitor.

Apr 102014
 

What is the Heartbleed Bug?

The Heartbleed Bug is a serious vulnerability in the OpenSSL software. SSL/TLS provides communication security and privacy over the Internet for applications such as websites, emails, instant messaging (IM) and some virtual private networks (VPNs).

The Heartbleed bug allows anyone on the Internet to read the memory of the systems protected by the vulnerable versions of the OpenSSL software. This compromises the secret keys used to identify the service providers and to encrypt the traffic, the names and passwords of the users and the actual content.

For a more detailed explanation go and read http://heartbleed.com/

How do i stop the leak?

You need to update to the latest version of OpenSSL and restart all services that use it.

How can i check if my server or a site i visit is still vulnerable

There are several sites where you can check, i recommend https://www.ssllabs.com/ssltest/index.html as it tests more than just the heartbleed bug.
There is also http://filippo.io/Heartbleed/

All my servers have been patched and the SSL certificates have been replaced.

Apr 072014
 

Following on from postfix blocking spam before it enters the server i setup a whitelist and SPF filtering.

The whitelist will allow me to manually allow any mail servers to bypass the spf filtering and RBL(Real-time Blacklists) lists.

What does SPF filtering do? Suppose a spammer forges a Hotmail.com address and tries to spam you. They connect from somewhere other than Hotmail. When his message is sent, you see MAIL FROM: , but you don’t have to take his word for it. You can ask Hotmail if the IP address comes from their network.
(In this example) Hotmail publishes an SPF record. That record tells you how to find out if the sending machine is allowed to send mail from Hotmail. If Hotmail says they recognize the sending machine, it passes, and you can assume the sender is who they say they are. If the message fails SPF tests, it’s a forgery. That’s how you can tell it’s probably a spammer.

Now time to start setting everything, for the spf filtering we need to install a few packages so start with (i use the epel rep http://fedoraproject.org/wiki/EPEL)

yum --enable epel install python-dns python-pydns

we also need “pyspf”. check for any updates from here
Then install it, you need to be the root user (change the version numbers if theirs an update)
wget http://sourceforge.net/projects/pymilter/files/pyspf/pyspf-2.0.8/pyspf-2.0.8.tar.gz/download
tar xvfz pyspf-2.0.8.tar.gz
cd pyspf-2.0.8/
python setup.py build
python setup.py install

Finally we need “pypolicyd-spf”. check for any updates from here
Then install it, you need to be the root user (change the version numbers if theirs an update)

wget https://launchpad.net/pypolicyd-spf/1.2/1.2/+download/pypolicyd-spf-1.2.tar.gz
tar xvfz pypolicyd-spf-1.2.tar.gz
cd pypolicyd-spf-1.2/
python setup.py build
python setup.py install

Now everything is install I need to tell postfix to use it. Since i use Webmin i just navigate to “servers”, “Posfix Mail server” then click “Edit Config Files” or manually edit “/etc/postfix/main.cf”

now find “smtpd_recipient_restrictions = “, and add “check_client_access hash:/etc/postfix/rbl_override_whitelist, check_policy_service unix:private/policyd-spf,” after “reject_unauth_destination,”
It is important that you add it AFTER reject_unauth_destination or else your system can become an open relay!
It should look like this.

smtpd_recipient_restrictions = permit_mynetworks, permit_inet_interfaces, permit_sasl_authenticated, reject_unauth_pipelining, reject_invalid_hostname, reject_non_fqdn_hostname, reject_unknown_recipient_domain, reject_unauth_destination, check_client_access hash:/etc/postfix/rbl_override_whitelist, check_policy_service unix:private/policyd-spf, reject_rbl_client dnsbl.sorbs.net, reject_rbl_client zen.spamhaus.org, reject_rbl_client bl.spamcop.net, permit 

Now I need to edit “/etc/postfix/master.cf”. Since i use Webmin i just navigate to “servers”, “Posfix Mail server” then click “Edit Config Files” and select “master.cf” from the drop box at the top.

Now i add at the end

policyd-spf  unix  -       n       n       -       0       spawn
                   user=nobody argv=/usr/bin/policyd-spf

The leading spaces before user=nobody are important so Postfix knows this line belongs to the previous one.

The last thing i need to do is create the whitelist file, so login as root

cd /etc/postfix
vi /etc/postfix/rbl_override_whitelist

Then add all ip addresses or hostname that you want whitelisted (one per line only)
here what it should look like
1.2.3.4 OK
mail.example.net OK

After you create/modify the file you need to run
postmap /etc/postfix/rbl_override_whitelist

Finally restart postfix
/etc/init.d/postfix restart 

Now send a test message from an external email account to test, if the email doesn’t arrive check the logs for any errors (something you should do regularly anyway).

Hopefully everything is working fine and you should start seeing a drop in forged emails, don’t forget to create a spf record for your domain so other servers can check your emails. There is a easy to use wizard to help create the record for you.
This is what a record looks like

v=spf1 mx a ip4:144.76.115.197/32 ip6:2a01:4f8:192:70c4::/64 include:spf.mandrillapp.com -all  

It basically lists all the ip address that are allowed to send email for my domain and says reject everything else.
If you wanna check if a particulate site has an spf record or you want to check if its working correctly there is a SPF Record Testing Tools.