Digital Ocean Articles
Posted: 2014-04-03 23:52:22 by Alasdair Keyes
Digital Ocean seem to be the new darlings of the Cloud world, and for good reason, their servers are great value for money with good performance. I'm looking to upgrade my VPS soon and I will be strongly considering one of their VMs.
One other thing that have made me respect them are their incredibly useful tech articles. I've recently been looking at MySQL clustering for a project I'm working on that could grow and need to scale. I came across MariaDB's Galera tool, which looks to be a good starting point for me.
The next question was how to set it up... never fear Digital Ocean had a clear and concise article on how to do it https://www.digitalocean.com/community/articles/how-to-configure-a-galera-cluster-with-mariadb-on-ubuntu-12-04-servers, what more could you want?
Have a read through their articles, you'll probably learn stuff you didn't even know you wanted to learn.
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
MySQL Tuner
Posted: 2014-03-28 16:18:08 by Alasdair Keyes
Sometimes an often overlooked source of poor system performance is the database. Developers and sysads can often become complacent and just continue piling data into a database without a second thought and assume that it will always carry on performing the way we'd like.
We had a mail server in our cluster at work performing poorly, and it appeared to be MySQL not keeping up with the amount of queries it was performing due to a high number of SMTP connections.
When checking the setup, I was surprised to see that no tuning had been done, even simple things like thread/table/query cache weren't enabled. Apparently these servers had slipped through the net as we'd not experienced any trouble with them before.
I broke out the trusty Mysql-Tuner http://mysqltuner.com/ tool, which should be at hand for any MySQL admin. It's an incredibly useful and easy way to diagnose simple and quick performance tweaks for your database.
It's a read-only script, so doesn't change your configuration, just alerts you to potential problems, so even if your DB is running perfectly, it's always worth running it just to see if you can improve something even further.
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
NFS v4 Callbacks in RedHat/CentOS Howto
Posted: 2014-01-03 16:05:49 by Alasdair Keyes
NFS callbacks are a feature introduced in NFS v4 to prevent deadlocks. If two clients try to access the same file, the server can recall the delegation via an asynchronous callback to one client should there be a conflicting request for access to the file from a different client.
Unlike other NFS services such as lockd or statd the callbacks are setup on the client and not the server.
This sounds good but it doesn't seem to have been implemented very well and no real documentation on exactly how to use it or the correct way to set this up. There's no default NFS callback port, it's up to the sysadmin to choose a port and set it.
There are many pages on the net explaining how you can echo a port number into a /proc path (Which doesn't work if the NFS file is auto-mounted before this port is set) Or by setting /proc values in /etc/sysctl.conf, which doesn't seem to work at all.
I did stumble across this solution, however...
Choose a port This is entirely up to you, for this example, we'll set port 10000
Open the firewall on the NFS client
iptables -A INPUT -p tcp -m tcp --dport 10000 -m iprange --src-range x.x.x.x-x.x.x.y -m comment --comment "NFS TCP callback" -j ACCEPT
Set the callback port with modprobe Create a file called /etc/modprobe.d/nfscallback.conf with the following text
options nfs callback_tcpport=10000
Restart your NFS client and then check netstat to see if it's listening
# netstat -antp | grep 10000
tcp 0 0 0.0.0.0:10000 0.0.0.0:* LISTEN -
That's it all setup!
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
PowerDNS and BIND
Posted: 2013-12-19 23:02:37 by Alasdair Keyes
Since forever (well, sometime in the 70s or 80s when IT really was a Brave New World), BIND has been the software of choice for Linux/Unix based DNS servers on the internet.
.. and why not? It does what it says on the tin and gets the job done but I feel it's time has come. Over the past few years I've made the migration across to PowerDNS on my own DNS servers.
When PowerDNS was released in 2006 it's strong point was built-in SQL database backend support, which was only available in BIND through the use of the DLZ or SDB patches. These patches often required manual compilation, which is something I try not to do too often, it adds a lot of time and stress to systems updates. BIND has now incorporated the DB backend plugins into it's core, but PowerDNS had already whetted the appetite of system administrators with it's power and easy configuration.
PowerDNS's easy config highlighted to me just how much of a pain BIND's config was, after all, why use 3 lines of config when 50 will do? With Power Admin there is also a nice third-party web-based front end to manage it.
There are many pages listed on Google on how to configure PowerDNS if you wish to check it out (and I strongly recommend you do).
It provides both an authoritative DNS server and also a Recursive caching nameserver.
If you're running the authoritative nameserver it can be setup to recurse too but you need to provide a nameserver to recurse to which you sometimes don't have available (and maybe you don't want to use openDNS).
In this instance you can run both the authoritative and recursive services on the same machine and configure the authoratative server to recurse to the recursive server with only a small config change.
The problem with running both services is that they both try and bind to port 53, to fix this, use the following configuration
In /etc/pdns-recursor/recursor.conf, bind it to another port (e.g port 54)
allow-from=127.0.0.0/8
local-port=54
In /etc/pdns/pdns.conf
lazy-recursion=yes
recursor=127.0.0.1:54
allow-recursion=127.0.0.1, 10.0.0.0/24
The restart the server and test
# service pdns-recursor restart
Stopping pdns-recursor: [ OK ]
Starting pdns-recursor: [ OK ]
# service pdns restart
Restarting PowerDNS authoritative nameserver: stopping and waiting..done
Starting PowerDNS authoritative nameserver: started
# host google.com
google.com has address 173.194.34.110
...
google.com mail is handled by 10 aspmx.l.google.com.
Easy
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
Linux Mint 16 network-manager-gnome breaks wifi?
Posted: 2013-12-19 22:51:08 by Alasdair Keyes
I'm on call at the moment, which is never fun, but to make it worse, whilst I was logged into servers at work trying to fix an issue, my wifi died.
It wasn't just my router losing connection, the network manager in Mint 16 just didn't want to know about wifi. All options to use it were greyed out and the following errors were in syslog (Please forgive the wall of text)
Dec 19 21:56:05 luka wpa_supplicant[987]: rfkill: WLAN hard blocked
Dec 19 21:56:05 luka kernel: [ 2855.552595] wlan0: deauthenticating from [wifi router mac] by local choice (reason=3)
Dec 19 21:56:05 luka kernel: [ 2855.553638] brcmsmac bcma0:0: brcmsmac: brcms_ops_bss_info_changed: disassociated
Dec 19 21:56:05 luka kernel: [ 2855.553646] brcmsmac bcma0:0: brcms_ops_bss_info_changed: arp filtering: 1 addresses (implement)
Dec 19 21:56:05 luka kernel: [ 2855.553649] brcmsmac bcma0:0: brcms_ops_bss_info_changed: qos enabled: false (implement)
Dec 19 21:56:05 luka wpa_supplicant[987]: wlan0: CTRL-EVENT-DISCONNECTED bssid=[wifi router mac] reason=3
Dec 19 21:56:05 luka NetworkManager[944]: [info] WiFi now disabled by radio killswitch
Dec 19 21:56:05 luka NetworkManager[944]: [info] (wlan0): device state change: activated -> unavailable (reason 'none') [100 20 0]
Dec 19 21:56:05 luka NetworkManager[944]: [info] (wlan0): deactivating device (reason 'none') [0]
Dec 19 21:56:05 luka kernel: [ 2855.612636] cfg80211: Calling CRDA to update world regulatory domain
Dec 19 21:56:05 luka avahi-daemon[665]: Interface wlan0.IPv6 no longer relevant for mDNS.
Dec 19 21:56:05 luka avahi-daemon[665]: Leaving mDNS multicast group on interface wlan0.IPv6 with address [wifi adapter ipv6 address].
Dec 19 21:56:05 luka avahi-daemon[665]: Interface wlan0.IPv4 no longer relevant for mDNS.
Dec 19 21:56:05 luka avahi-daemon[665]: Leaving mDNS multicast group on interface wlan0.IPv4 with address [wifi adapter ipv4 address].
Dec 19 21:56:05 luka avahi-daemon[665]: Withdrawing address record for [wifi adapter ipv6 address] on wlan0.
Dec 19 21:56:05 luka avahi-daemon[665]: Withdrawing address record for [wifi adapter ipv4 address] on wlan0.
Dec 19 21:56:05 luka avahi-daemon[665]: Joining mDNS multicast group on interface wlan0.IPv4 with address [wifi adapter ipv4 address].
Dec 19 21:56:05 luka avahi-daemon[665]: New relevant interface wlan0.IPv4 for mDNS.
Dec 19 21:56:05 luka avahi-daemon[665]: Registering new address record for [wifi adapter ipv4 address] on wlan0.IPv4.
Dec 19 21:56:05 luka wpa_supplicant[987]: rfkill: WLAN hard blocked
Dec 19 21:56:05 luka kernel: [ 2855.641875] cfg80211: World regulatory domain updated:
Dec 19 21:56:05 luka kernel: [ 2855.641879] cfg80211: (start_freq - end_freq @ bandwidth), (max_antenna_gain, max_eirp)
Dec 19 21:56:05 luka kernel: [ 2855.641881] cfg80211: (2402000 KHz - 2472000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
Dec 19 21:56:05 luka kernel: [ 2855.641882] cfg80211: (2457000 KHz - 2482000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
Dec 19 21:56:05 luka kernel: [ 2855.641883] cfg80211: (2474000 KHz - 2494000 KHz @ 20000 KHz), (300 mBi, 2000 mBm)
Dec 19 21:56:05 luka kernel: [ 2855.641885] cfg80211: (5170000 KHz - 5250000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
Dec 19 21:56:05 luka kernel: [ 2855.641886] cfg80211: (5735000 KHz - 5835000 KHz @ 40000 KHz), (300 mBi, 2000 mBm)
Dec 19 21:56:05 luka kernel: [ 2855.776410] usb 1-1.3: USB disconnect, device number 5
Dec 19 21:56:05 luka kernel: [ 2855.776419] usb 1-1.3.1: USB disconnect, device number 7
Dec 19 21:56:05 luka NetworkManager[944]: [info] (wlan0): canceled DHCP transaction, DHCP client pid 1584
Dec 19 21:56:05 luka kernel: [ 2855.786157] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
Dec 19 21:56:05 luka avahi-daemon[665]: Withdrawing address record for [wifi adapter ipv4 address] on wlan0.
Dec 19 21:56:05 luka avahi-daemon[665]: Leaving mDNS multicast group on interface wlan0.IPv4 with address [wifi adapter ipv4 address].
Dec 19 21:56:05 luka avahi-daemon[665]: Interface wlan0.IPv4 no longer relevant for mDNS.
Dec 19 21:56:05 luka NetworkManager[944]: [warn] DNS: plugin dnsmasq update failed
Dec 19 21:56:05 luka NetworkManager[944]: [info] Removing DNS information from /sbin/resolvconf
Dec 19 21:56:05 luka dnsmasq[2210]: setting upstream servers from DBus
Dec 19 21:56:05 luka dbus[584]: [system] Activating service name='org.freedesktop.nm_dispatcher' (using servicehelper)
Dec 19 21:56:06 luka kernel: [ 2855.862901] usb 1-1.3.2: USB disconnect, device number 8
Dec 19 21:56:06 luka kernel: [ 2855.864109] usb 1-1.3.3: USB disconnect, device number 9
Dec 19 21:56:06 luka bluetoothd[624]: Adapter /org/bluez/624/hci0 has been disabled
Dec 19 21:56:06 luka bluetoothd[624]: Unregister path: /org/bluez/624/hci0
Dec 19 21:56:06 luka bluetoothd[624]: Endpoint unregistered: sender=:1.55 path=/MediaEndpoint/A2DPSink
Dec 19 21:56:06 luka bluetoothd[624]: Endpoint unregistered: sender=:1.55 path=/MediaEndpoint/A2DPSource
Dec 19 21:56:06 luka bluetoothd[624]: Endpoint unregistered: sender=:1.55 path=/MediaEndpoint/HFPAG
Dec 19 21:56:06 luka bluetoothd[624]: Endpoint unregistered: sender=:1.55 path=/MediaEndpoint/HFPHS
What's interesting is rfkill detecting that my WiFi was hard blocked (disabled by a hardware switch), but my laptop doesn't have a hardware switch for disabling wireless and a reboot didn't help at all.
I had run a system update a few hours before and saw from /var/log/apt-history.log that apt had updated the network-manager-gnome package, that could well have been the cause so I rolled back....
I was currently running version 0.9.8.0-1ubuntu5.1
# dpkg -l | grep network-manager-gnome
ii network-manager-gnome 0.9.8.0-1ubuntu5.1 amd64 network management framework (GNOME frontend)
I checked what versions were available to me..
# apt-cache showpkg network-manager-gnome | grep -A 10 "Provides"
Provides:
0.9.8.0-1ubuntu5.1 -
0.9.8.0-1ubuntu5 -
Reverse Provides:
And then downgraded, it seems that libnm-gtk0 is a dependency so it had to be downgraded as well...
# apt-get install network-manager-gnome=0.9.8.0-1ubuntu5 libnm-gtk0=0.9.8.0-1ubuntu5
Reading package lists... Done
Building dependency tree
...
...
Setting up network-manager-gnome (0.9.8.0-1ubuntu5) ...
Processing triggers for libc-bin ...
I rebooted my machine and I had wireless back again, I'm not sure if the update caused the issue as wifi was working fine for a couple of hours after the update, but rolling back seemed to fix it. I don fancy trying an upgrade to the same package again to test it. I'll leave those updates to see if another one is shortly release. Worth a try if you get the same issue...
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
CentOS/Redhat PHP updates break sessions with suPHP
Posted: 2013-12-12 12:50:14 by Alasdair Keyes
With the recent PHP update for Redhat/CentOS in the past few days, it's brought to light a problem I've seen before and always forget about.
When the PHP RPM is installed it updates the permissions on the PHP session directory back to the defaults...
# stat /var/lib/php/session | grep Uid
Access: (0770/drwxrwx---) Uid: ( 0/ root) Gid: ( 48/ apache)
Like many I run suPHP, so all my sites use different users to execute. This will break sessions for all sites on my server as only root/apache can write to that folder.
As a fix, I've updated the PHP session path to be a custom location and set permissions as 1777.
mkdir /var/lib/php/mynewsessionfolder
chmod 1777 /var/lib/php/mynewsessionfolder
Then create a file called /etc/php.d/customsession.ini with the text
session.save_path = /var/lib/php/mynewsessionfolder
The permissions 1777 means that the folder is world read/writeable but when a user creates a file it is created as 600 permissions so only they can read/write to it
# ls -l /var/lib/php/mynewsessionfolder
total 4
-rw------- 1 auser auser 377 Dec 12 12:45 sess_6pjpshqnr06egukas50s0mhjk6
Next time PHP updates it will reset permissions on the standard session folder, but won't affect you
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
Noisili - Relaxing Noises
Posted: 2013-12-12 10:49:16 by Alasdair Keyes
I was forwarded this link today, it's a website that provides background and ambient noise such as rain, forest sounds, evem running water (although I'm not sure that will have a relaxing effect)
My office is often quite noisy being in the same room as people on the phone all the time and sometimes listening to music is too distracting, this could be my new favourite site at work.
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
Cryptography Introduction
Posted: 2013-12-11 12:10:54 by Alasdair Keyes
Cryptography is one of the most important branches of maths in the modern age.
Almost everything that requires security/secrecy in our life relies on it. However the basics of it aren't well understood by many, even those who work with it daily, such as System Administrators.
Cryptography can get very complex and way beyond anyone who hasn't got a doctorate, however a basic grounding in some principles behind it is very useful (and interesting, if you're that way inclined)
I came across these videos whilst browsing Reddit which I would recommend to anyone that wants to learn more.
Public key (Or Asymmetric) Cryptography: Diffie-Hellman Key Exchange
Gambling with secrets (Cryptography)
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
Elementary OS
Posted: 2013-08-22 12:56:53 by Alasdair Keyes
For the past 6 months or so I've been running Linux Mint 15 as my Desktop OS. It can be a little slow in it's operation, slight lag in response to mouseclicks etc. so I've been on the lookout for a new desktop distro when someone point out Elementary OS.
It's fairly new and based on Ubuntu 12.04 LTS. It's highly customized to look like OS X, while I'm not a huge fan of the OS X interface it does have some nice features.
I installed it on a VM for a quick play and the first thing I noticed was how fast it was. Everything seemed to open instantly.
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
PHP Session Garbage Collecting - Not great for shared hosting (Repost)
Posted: 2013-04-02 12:28:47 by Alasdair Keyes
This is a repost of an old article that I transferred across from my previous blog. I've only just noticed that it was incomplete so I've completed it and reposted it
I look after a fair sized Linux shared hosting cluster (20,000+ websites) and to provide PHP session persistence between the servers in the cluster, PHP sessions are stored on an NFS share.
I noticed that a number of processes where running for a long time on the Apache servers. At first I thought this was due to loops or bad coding on behalf of the website owners, but it didn't seem to be restricted to any particular users.
After running strace on one of these processes I saw that the processes where getting permission denied trying to delete large numbers of PHP session files.
It turns out that PHP implements it's own internal garbage collection to get rid of old sessions, however, as we run SuPHP, the PHP processes only have permissions to delete their own session files (due to the use of Linux's sticky bit permissions), but that doesn't stop the process recursing through the sessions folder and trying to delete all old sessions.
With 20,000 websites, most of which run PHP CMS systems, this is quite a drain, compounded by the fact it's on an NFS share, which also adds overhead to each filesystem request.
The solution was to turn off garbage collection in PHP config. Create a config fragment file /etc/php.d/disable_session_gc.ini on the webhead and add the following content
; Disable auto session garbage collector
session.gc_probability = 0
Obviously I didn't want the sessions building up on our NFS filer, so I just set up a cronjob to call tmpwatch and delete old files once per day. I decided 7 days would be adequate.
/etc/cron.daily/clear_php_sessions.sh
/usr/sbin/tmpwatch 168 /exports/php/sessions
If you found this useful, please feel free to donate via bitcoin to 1NT2ErDzLDBPB8CDLk6j1qUdT6FmxkMmNz
© Alasdair Keyes
IT Consultancy Services
I'm now available for IT consultancy and software development services - Cloudee LTD.
Happy user of Digital Ocean (Affiliate link)
Version:master-e10e29ed4b
