Slightly modified MySQL backup script. This one uses mysqlhotcopy. It only locks the tables for 1 or 2 seconds (every two hours in cron). Then it copies the database MyISAM files. These files then are being archived and compressed, and copied to NAS using SMB.

Advantage of this new principle is that MySQL is only occupied by this process for two seconds. It does not need to generate a .sql file with SQL statements representing the content in the database. After two seconds the MySQL database server can continue with whatever it was doing, while 1 CPU core is being used by the tar process to pack and archive it all. Script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

#!/bin/sh
SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local:/usr/local/mysql
 
#variables
DBNAME=...
DBPASS=...
DBUSER=...
SMBUSER=...
SMBPASS=...
SMBHOST=...
SMBPATH=...
backupdir=...
 
#database backup
rm -fr $backupdir/$DBNAME
/usr/local/mysql/bin/mysqlhotcopy --allowold -user=$DBUSER -password=$DBPASS $DBNAME $backupdir
 
#Compress and archive this copy of the database
tar -C $backupdir/ -cjf $backupdir/$DBNAME-1.tar.bz2 $DBNAME
 
#mount backup directory
mkdir $backupdir/backup
mount -t cifs -o username=$SMBUSER,password=$SMBPASS //$SMBHOST/$SMBPATH $backupdir/backup
 
#recycle files
for ((i=99;i>=1;i-=1)); do
echo $i
j=$(( $i + 1 ))
mv -f $backupdir/backup/$DBNAME-$i.tar.bz2 $backupdir/backup/$DBNAME-$j.tar.bz2
done
 
#copy backup
mv -f $backupdir/$DBNAME-1.tar.bz2 $backupdir/backup/$DBNAME-1.tar.bz2
 
#unmount and clean
umount $backupdir/backup
rm -fr $backupdir/$DBNAME

To stress test our servers we’re benchmarking them. By monitoring both the webservers and database we can tune our setup and see where the performance bottleneck is located.

To properly test the website we need to take a “snapshot” from the requests that are performed on the webservers. To do that, we make a log on the frontend server with the logformat:

1

log_format  request '$request'

Now log all requests for a certain timespam, until we have an accurate set of requests. The logfile needs to be parsed with a small perl, bash or php script. See below. This file will output all request URL’s to url.txt. Since we have two frontends the script creates two lines for every url, pointing to two different domains.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

<?php
        $logfile = file_get_contents("./myp2p.access.log");
        $logfile = explode("\n",$logfile);
        $i = 0;
        $servers = array("fe1.myp2p.eu","fe2.myp2p.eu");
        foreach($logfile as $line){
                if(preg_match("/GET (.*\.php.*) HTTP/",$line,$regs)){
                        foreach($servers as $server){
                                $output .= "http://" . $server . $regs[1] . "\n";
                        }
                        $i++;
                }
        }
        #echo $i . " items gevonden\n";
        file_put_contents("url.txt",$output);
?>

With the http_load benchmark tool we can now load this url file, and randomly visit them, using a couple of different methods. For example, we can benchmark for 30 seconds with 100 requests/second, and see what that asks from our servers:

1

http_load -rate 100 -seconds 30 url.txt

On 16mbit traffic we are experiencing 130 hits served by one frontend every second (php files, images, css files, etc). On peaks we have about 1000mbit traffic, meaning that on such points we would have around 8000 hits per second on one frontend server. To simulate this behaviour we need to perform 16000 hits per second on both two frontends.

Make sure you have a fresh default installation of debian 505 running. Also make sure you have ssh running.

New kernel

1
2
3
4
5
6
7

cd /usr/src/
wget http://.../linux-headers-2.6.34.1-myp2p_2.6.34.1-myp2p-10.00.Custom_amd64.deb
wget http://.../linux-image-2.6.34.1-myp2p_2.6.34.1-myp2p-10.00.Custom_amd64.deb
dpkg -i linux-image-*.deb
dpkg -i linux-headers-*.deb
update-initramfs -k 2.6.34.1-myp2p -c
pico /boot/grub/menu.list

Now make sure that the file system is correct and that there is a initrd line in the new kernel entries.

Time and date

1
2
3
4
5
6
7
8

dpkg-reconfigure tzdata
apt-get install ntpdate
cd /etc/init.d/
wget http://.../ntpdate
chmod +x ntpdate
./ntpdate
crontab -e
add: 25 6   * * *     /etc/init.d/ntpdate

Lighttpd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

cd /usr/src/
wget http://download.lighttpd.net/lighttpd/releases-1.4.x/lighttpd-<version>.tar.gz
tar xzf lighttpd-<version>.tar.gz
cd lighttpd-<version>
apt-get install zlib1g zlib1g-dev libpcre3 libpcre3-dev make build-essential
./configure --without-bzip2
make
make install
mkdir /etc/lighttpd
cd /etc/lighttpd
wget http://.../lighttpd.conf
pico lighttpd.conf (edit if needed)
cd /etc/init.d/
wget http://.../lighttpd
chmod +x lighttpd

PHP & APC

1
2

apt-get install php5-cgi php5-memcache php5-dev php5-gd php5-mysql
apt-get install memcached

configure php to use memcache as session file host with:

1
2

session.save_handler = memcache
session.save_path="tcp://10.0.0.67:11211?persistent=1&weight=1&timeout=1&retry_interval=15"

in

1

pico /etc/php5/cgi/php.ini

install APC:

1
2
3

apt-get install php-pear
pecl install apc
pico /etc/php5/cgi/conf.d/apc.ini

and add to this file something like:

1
2
3

extension=apc.so
apc.enabled=1
apc.shm_size=30

Now start lighttpd:

1
2
3
4
5
6
7

mkdir /var/www/
cd /var/www/
wget http://.../index.html
wget http://.../phpinfo.txt
mv phpinfo.txt phpinfo.php
mkdir /home/myp2p/public_html
/etc/init.d/lighttpd

Misc

1
2
3
4
5
6
7
8

apt-get install postfix (install with "no configuration")
cd /etc/postfix/
wget http://.../main.cf
pico main.cf
pico /etc/hosts (add the line "77.247.179.66   myp2p.eu")
rm -fr sysctl.conf
wget http://.../sysctl.conf
sysctl -p

Website sync

1
2
3
4
5
6
7

cd /home/myp2p/
wget http://.../sync.sh
chmod +x sync.sh
apt-get install rsync
su myp2p -c "ssh-keygen -t rsa" (Copy this key (/home/myp2p/.ssh/id_rsa.pub) to the core server)
./sync.sh
crontab -e (add the line "*/5 *  * * *     /home/myp2p/sync.sh" and the line "* *    * * *     find /home/myp2p/public_html/unicache/* -mmin +10 -exec rm -fr {} \;")

Since Februari 12 2010, Etch has become oldstable. Packages are no longer updated. So time to use the archive apt mirror:

1
2

deb http://archive.debian.org/debian/ etch main
deb-src http://archive.debian.org/debian/ etch main

Photoshop drawing of Nelson:

For the mPossible project we had a photoshoot for promotional goals. Some pretty good results. Big thanks to Rebecca.

Quick note on upgrading Alsa on the mPossible. This solved our issues with the relatively new VIA sound card chip on our new Asus motherboard.

Download the following 3 source packages:

ftp://ftp.alsa-project.org/pub/driver/alsa-driver-1.0.23.tar.bz2
ftp://ftp.alsa-project.org/pub/lib/alsa-lib-1.0.23.tar.bz2
ftp://ftp.alsa-project.org/pub/utils/alsa-utils-1.0.23.tar.bz2

Extract, configure, compile, install with:

1
2
3

./configure
make
make install

You might need debian packages:

1

apt-get install build-essential ncurses-dev gettext xmlto libasound2-dev linux-headers-`uname -r` libncursesw5-dev

Reboot, and set all output levels with a program like alsamixer.

One of our servers has been dealing with huge peaks of iowait. After some investigation i noticed that this is (obviously) caused by a disk writing. The iostat program gave me huge Blk_wrtn/s numbers for my first partition device. I wrote a very poor but usefull php command line application that detects processes that are in the D-state, and then checks all the opened file descriptors of this process. This php script showed a lot of write actions to the php session directory.

So to avoid this, I implemented memcached into php. You can configure the php processes to store their session files in memcached. Install memcache, give it some memory to work with (512M here, but depending on the size of your site) and edit the php.ini with:

1
2

session.save_handler = memcache
session.save_path="tcp://127.0.0.1:11211?persistent=1&weight=1&timeout=1&retry_interval=15"

On high load this will reduce the iowait CPU time significantly.

Iowait debug script

Creating a box with rounded corners using css always is quite a hassle, so i decided to figure it out properly once and for good. Using absolutely positioned elements you can create corners, and all elements between those corners. The tricky part is to let the border parts between the corners have a 100% width or height, but letting them not overlap with the corners, using a margin.

To be cross-browser friendly, it’s important to declare the doctype. IE7 will give wrong output if no doctype declared.

1

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

Check out the eBox that i made at http://www.edoxy.net/ebox/ebox.html. Download the pack, also for PSD’s on the imaged border and overlay gradient.

Be aware of the fact that the overlay covers the complete box, including the radius corners. If the box is not large enough, you will see the overlay at those corners. Make sure that the overlay is completely transparant at the top and at the bottom.