Installing apache solr with tomcat6 on debian in /usr/local (based on http://xdeb.org/node/1213)

Change /etc/apt/sources.list so that non-free packages can be used:

1

deb http://ftp.nl.debian.org/debian/ lenny main contrib non-free

Install Java:

1

apt-get install sun-java6-jdk

Install Tomcat6 to /usr/local:

1
2
3
4

cd /usr/src
wget http://apache.osuosl.org/tomcat/tomcat-6/v6.0.24/bin/apache-tomcat-6.0.24.tar.gz
tar xzf apache-tomcat-6.0.24.tar.gz
mv apache-tomcat-6.0.24 /usr/local/tomcat6

Install Solr:

1
2
3
4
5
6

cd /usr/src
wget http://apache.osuosl.org/lucene/solr/1.4.0/apache-solr-1.4.0.tgz
cp ./apache-solr-1.4.0/dist/apache-solr-1.4.0.war /usr/local/tomcat6/webapps/solr.war
cp -r ./apache-solr-1.4.0/example/solr /usr/local/solr
mkdir -p /usr/local/tomcat6/conf/Catalina/localhost/
pico /usr/local/tomcat6/conf/Catalina/localhost/solr.xml

Now add the configuration here:

1
2
3

<Context docBase="/usr/local/tomcat6/webapps/solr.war" debug="0" crossContext="true" >  
  <Environment name="solr/home" type="java.lang.String" value="/usr/local/solr" override="true" />
</Context>

Edit pico /usr/local/tomcat6/conf/server.xml and change the Connector tag to:

1
2
3
4
5

    <Connector port="8080" protocol="HTTP/1.1"
               connectionTimeout="20000"
               redirectPort="8443"
               address="127.0.0.1"
               URIEncoding="UTF-8" />

Add the tomcat service to the system:

1

echo "ENABLED=1" > /etc/default/tomcat

Add the init.d startup script to /etc/init.d/tomcat:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

# Tomcat auto-start
## description: Auto-starts tomcat
# processname: tomcat
# pidfile: /var/run/tomcat.pid
ENABLED=0
if [ -f /etc/default/tomcat ]; then  
  . /etc/default/tomcat
fi
if [ "$ENABLED" = "0" ]; then  
  exit 0
fi
export JAVA_HOME=/usr/lib/jvm/java-6-sun/jre
export JAVA_OPTS="$JAVA_OPTS -Dsolr.solr.home=/usr/local/solr"
export CATALINA_OPTS="-Xms64m -Xmx64m"
case $1 in
start)  
  sh /usr/local/tomcat6/bin/startup.sh  
  ;;
stop)  
  sh /usr/local/tomcat6/bin/shutdown.sh  
  ;;
restart)  
  sh /usr/local/tomcat6/bin/shutdown.sh  
  sh /usr/local/tomcat6/bin/startup.sh  
  ;;
esac
exit 0

Add tomcat user and give ownership:

1
2
3

adduser --shell /bin/false tomcat tomcat
chown -R tomcat:tomcat /usr/local/tomcat6
chown -R tomcat:tomcat /usr/local/sol

Information extraxted from from CBlue (original arcticle http://www.cblue.be/blog/debian-lenny-netinst-new-igb).

On our new proliant servers we have igb network cards. These network cards are not (yet) supported in the default debian initrd netinstall. We need to create an own initrd with compiled igb drivers in it. This is quite a hassle.

Compile drivers for debian netinstall image. Make sure you do this on a system with desired cpu architecture.

1
2
3
4
5
6
7
8

apt-get install linux-headers-(uname -r)
cd /usr/src/
# get the stable igb drivers:
wget http://sourceforge.net/projects/e1000/files/igb%20stable/...
tar zfvx igb-version.tar.gz
cd igb-version
make
make install

Then the compiled version of the igb module is in /lib/modules/(uname -r)/kernel/drivers/net/igb/igb.ko, and dca.ko (also required) is in /lib/modules/(uname -r)/kernel/drivers/dca/dca.ko.

Then extract the current initrd:

1
2
3

mkdir /usr/src/temp
cd /usr/src/temp
zcat initrd.gz | cpio -i

Create needed directory and copy your modules:

1
2
3
4

mkdir lib/modules/2.6.26-2-486/kernel/drivers/net/igb
mkdir lib/modules/2.6.26-486/kernel/drivers/dca
cp (...)/igb.ko lib/modules/2.6.26-2-486/kernel/drivers/net/igb
cp (...)/dca.ko lib/modules/2.6.26-2-486/kernel/drivers/dca

Then re-compress the initrd.gz:

1

find . -print | cpio -o -H newc | gzip > ../initrd.gz

Then replace the official initrd.gz by the new one in the netinstall iso. Still in linux:

mount -o loop -t iso9660 debian-505-amd64-netinst.iso ./iso
cp -r iso iso2
cp initrd.gz ./iso2/install.amd/
cd iso2
md5sum `find -follow -type f` > md5sum.txt
mkisofs -o debian-505-amd64-netinst-with-igb.iso -r -J -no-emul-boot -boot-load-size 4 -boot-info-table -b isolinux/isolinux.bin -c isolinux/boot.cat ./iso2

Burn the iso and boot your server. At the end of the netinstall installation, before rebooting, you need to install the driver for the kernel you just installed.

First, in the console, chroot the new installation:

1
2
3
4
5
6
7
8
9

chroot /target
echo "http://backports.debian.skynet.be/ debian-lenny main non-free contrib" >> /etc/apt/sources.list
gpg -a --keyserver pgpkeys.mit.edu --recv-key EA8E8B2116BA136C
gpg -a --export EA8E8B2116BA136C | apt-key add -
apt-get update
cd /root
wget http://blog.waja.info/downloads/igb-dkms_2.0.6_all.deb
dpkg -i igb-dkms_2.0.6_all.deb
apt-get install -f

The last command will fail. Dkms tries to discover the running kernel but does not find it on the current system. This is correct since we boot on Debian netinst. However, now we downloaded everything we needed while the network was working.

Now, finish the installation, reboot the server.

Once your new system is up, just log in and:

1
2
3

apt-get install -f
modprobe igb
/etc/init.d/networking restart

Since we’re running our vbulletin software for myp2p on multiple servers, an nginx frontend balances the traffic. We have a memcached cookie sync for session persistance with the following php.ini configuration lines:

1
2
3

[Session]
session.save_handler = memcache
session.save_path="tcp://<memcache host>:11211?persistent=1&amp;weight=1&amp;timeout=1&amp;retry_interval=15"

Make sure to remove the old session.save_handler/path lines.

The includes/class_core.php file needs te be edited, in order to obtain the proper remote ip address. Replace all instances of

1

$_SERVER['REMOTE_ADDR']

with

1

$_SERVER['HTTP_X_FORWARDED_FOR']

. In order to do this, your proxy must allow some kind of X_FORWARDED_FOR header rewrite thing.

If your backends are running on a different port than the frontend reverse proxy, you also need to manually adjust the

1

$_SERVER['SERVER_PORT']

value in includes/class_core.php.

Also make sure that there is one master server, and that slave servers sync files with rsync from the master. See this post.

Total lighttpd configuration after the break.
Read the rest of this entry »

In the lighttpd config:

1
2
3
4
5
6
7
8
9

$HTTP["host"] == "host" {
  server.document-root = "/home/exitable/public_html/"
  url.rewrite-final = (
    "^/([^.?]*)\?(.*)$" => "/index.php?q=$1&$2",
    "^/([^.?]*)$" => "/index.php?q=$1",
    "^/([^.?]*\.html)$" => "/index.php?q=$1"
  )
  server.error-handler-404 = "/index.php"
}

and then add to the /sites/default/settings.php file some code to catch 404 pages:

1
2
3
4
5
6
7
8
9
10
11
12

/* modification for lighttpd and imagecache*/
$_lighty_url = urldecode($base_url . $_SERVER['REQUEST_URI']);
$_lighty_url = @parse_url($_lighty_url);
 
if ($_lighty_url['path'] != '/index.php' && $_lighty_url['path'] != '/update.php' && $_lighty_url['path'] != '/') {
	$_SERVER['QUERY_STRING'] = $_lighty_url['query'];
	parse_str($_lighty_url['query'], $_lighty_query);
	foreach ($_lighty_query as $key => $val){
		$_GET[$key] = $_REQUEST[$key] = $val;
	}
	$_GET['q'] = $_REQUEST['q'] = substr($_lighty_url['path'], 1);
}

Supported languages:

Actionscript, ADA, Apache Log, AppleScript, ASM, ASP, AutoIT, Backus-Naur form, Bash, BlitzBasic, C, C for Macs, C#, C++, CAD DCL, CadLisp, CFDG, CFDG, ColdFusion, CSS, Delphi, DIV, DOS, Eiffel, Fortran, Fortran, FreeBasic, GML, Groovy, HTML, Inno, IO, Java, Java 5, Javascript, LaTeX, Lisp, Lua, Microprocessor ASM, mIRC, MySQL, NSIS, Objective C, OCaml, OpenOffice BASIC, Oracle 8 SQL, Pascal, Perl, PHP, PL/SQL, Python, Q(uick)BASIC, robots.txt, Ruby, SAS, Scheme, SDLBasic, Smalltalk, Smarty, SQL, T-SQL, TCL, thinBasic, Uno IDL, VB.NET, Visual BASIC, Visual Fox Pro, Winbatch, X++, XML, Z80 ASM

This command helps you detect the number of connections that one ip has to a (linux) server. Useful for detected ddos attack:

1

netstat -anp |grep 'tcp\|udp' | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n

We used to have a samba share sync on the myp2p backends. But this is pretty instable. Rsync, through ssh is much more convenient.. if you know how it works.

Ok, so we have our main server where every backend mirros some files from. Every backend executes the command:

1

rsync -avzR -e ssh myp2p@77.247.179.67:/home/myp2p/public_html/feeds /

All feeds get properly copied to /home/myp2p/public_html/feeds, on the local machine (backend). In order not to get a password prompt, we’re going to put an authorization on the main server. Perform on the backend:

1
2

su myp2p
ssh-keygen -t rsa

Now copy the content of the created file (~/.ssh/id_rsa.pub) into the ~/.ssh/authorized_keys, for the desired user on the main server, onto a new line. Executing the rsync command will now go without password prompt.

With a new kernel compile, make sure to enable the broadcom netxtream II drivers in the menuconfig setup. If you don’t do this, you’re gonna have to copy some files to /lib/firmware/bnx2/..

Backup batch file for backing up the cycle once in a while to an external hard drive. Windows scheduled task manager runs this file. Fairly easy.

1
2
3
4
5
6
7
8
9
10

@echo off
 
set hour=%time:~0,2%
if "%hour:~0,1%"==" " set hour=0%time:~1,1%
set filename=%date:~9,4%-%date:~6,2%-%date:~3,2%_%hour%%time:~3,2%
echo %filename%
 
copy /Y F:\myp2p_database\myp2p_user_backup1.sql.gz E:\myp2p_database\myp2p_user_%filename%.sql.gz
copy /Y F:\myp2p_database\myp2p_backup1.sql.gz E:\myp2p_database\myp2p_%filename%.sql.gz
copy /Y F:\myp2p_forum_database\myp2p_backup1.sql.gz E:\myp2p_forum_database\myp2p_%filename%.sql.gz