Good beginner book for learning Python

If you are completely new to programming take a look at How to Think Like a Computer Scientist. It seems quite good, and starts at the very beginning. It is also available on-line, so you don’t have to run to the bookstore and fork out $$$.

It teaches Python, which I think is a good beginner language. It also teaches programming in general, and you will find that what you learn will help you later if you want to learn other languages. Many people will tell you a lot about which programming language is the best first programming language. I don’t think it matters much really. What is more important is to learn one language and to learn it well.

Of course, there are a lot of other good resources as well. It’s good to have access to as many as possible. Each one will have its strengths and drawbacks.

Once you have got a good understanding of the basics, I recommend Dive Into Python which is also available in the Debian repositories (aptitude install diveintopython).

What Debian needs

Debian needs to be stable.
Oh wait, it already is.

Debian needs a sane release policy that is dictated by “ready for release” not “rush it out”.
Oh wait, it already has.

Debian needs to be able to run on old hardware.
Oh wait, it already does.

Debian needs to have a large repository of software.
Oh wait, it already has.

Debian needs to provide sane defaults.
Oh wait, it already does.

Debian needs to include “An Anarchist FAQ” in its repositories.
Oh wait, it already does.

Debian needs to be superior to its 1001 (decimal) derivates.
Oh wait, it already is.

Debian needs to use upstart.
No, it doesn’t.

Debian needs to..
Oh wait, this is getting silly.

Copying mysql databases the simple way

Some people seem to be bent on making their own lives as difficult as possible. I’ve seen friends got through some very complex procedures (complete with heavy breathing and all) just to copy a mysql database from one host to another. I do admit that some solutions I have seen have been really creative.

I lack imagination, so I just do it the simple way instead. First of all, I have a server called, amazingly enough, db. This server happens to be running Debian, but in this case that is not important whatsoever (apart from the fact that it never crashes). I have set up a special account for myself that gives me global access from any host:

mysql> GRANT ALL ON *.* TO led@'%' IDENTIFIED BY 'somethingthatyoucanrem3mber';

No, that is not my actual password. Now let’s suppose that I have been toying around with a database called volcanoes, that holds loads of data relating to volcanoes. I have been developing this on my laptop and now I want to copy it over to the database server. On the laptop, which only I use, the mysql root user doesn’t even have a password set (bad, bad me).

In a terminal (on the laptop):

% mysqldump -u root -B volcanoes | mysql -h db -u led -p
Enter password:
%

Anything more complex than this is just pure torture. The -B volcanoes part will ensure that the dump contains the CREATE DATABASE statement. Without this you would have to ensure that there is already an (empty) database on the database server before issuing the command.

Of course, you have lots and lots of options. For example, if you want only want to copy the tables structures and not the data, add -d to mysqldump.

% mysqldump -u root -d -B volcanoes | mysql -h db -u led -p
Enter password:
%

If I instead wanted to copy a database from the server to my laptop, perhaps to work on it at home over the weekend, that is equally simple. Let’s say I have another database kanji located on the server.

% mysqldump -h db -u led -p -B kanji | mysql -u root
Enter password: 
%

You can do a lot more as well. The mysql and mysqldump man pages will give you all the available options.

Note: If you are running mysql on a public server, such as a webhost, it is a good idea to at least think about it for a minute before you issue a GRANT ALL ON *.* TO myuser@'%'. If you are aware of the implications and don’t care cool. Your database, your free will.

Very simple NAT set-up on Debian

Many people ask me how to set up network address translation (NAT), aka. IP masquerading on a Debian (Etch) box. There are different ways of doing this, but this just works(tm).

You need to know (1) which interface is connected to your ISP, (2) which interface is connected to your LAN and (3) the IP address of your ISPs gateway. Edit /etc/network/if-up.d/iptables (the file probably does not exist), and enter the following:

 1 #! /bin/sh
 2
 3 # Firewall rules.
 4 #
 5 # Assumptions:
 6 #   eth0: connected to ISP
 7 #   eth1: connected to LAN
 8 #   ISP’s gw: 44.219.1.114
 9
10 # Flush all rules.
11 iptables -F
12 iptables -t nat -F
13 iptables -t mangle -F
14
15 # Set up NAT.
16 iptables -t nat -A POSTROUTING -o eth0 -j SNAT –to 44.219.1.114
17
18 # Enable IP forwarding
19 echo 1 > /proc/sys/net/ipv4/ip_forward
20
21 # Secure.
22 iptables -A INPUT -m state –state ESTABLISHED,RELATED -j ACCEPT
23 iptables -A INPUT -m state –state NEW -i ! eth0 -j ACCEPT
24 iptables -P INPUT DROP
25 iptables -A FORWARD -i eth0 -o eth0 -j REJECT

You will need to modify to use the correct gateway address (no, that is *not* my ISPs gateway address, I made it up randomly). You may also need to swap eth0 and eth1 unless you have eth0 connected to your ISP.

This script will only set up what is necessary to enable NAT and to provide some rudimentary security. You will want to modify this script to provide other rules as well, to suit your own requirements.

You are done. Your Debian box can now act as a gateway to the Internet for other computers on your LAN, at least once they are configured to use the Debian box as their default gateway.

Oh Derby! Dumb, dumb, dumb Derby.

I was playing around with Apache Derby today. You know, that database server that started life as Cloudscape, became Derby and is now included in the JDK. Yeah, that one. For Debian (Sid) users, it is available via a aptitude install sun-java6-javadb.

I had a little file I was trying to import, called cartoons.sql.

 1 /*
 2  * Here there be ‘toons!
 3  */
 4
 5 CONNECT ‘jdbc:derby:toonsdb;create=true’;
 6
 7 CREATE TABLE cartoons(
 8     cartoon_id INT NOT NULL PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
 9     cartoon VARCHAR(255) NOT NULL,
10     UNIQUE(cartoon)
11 );
12
13 INSERT INTO cartoons(cartoon) VALUES(‘Bugs Bunny’);
14 INSERT INTO cartoons(cartoon) VALUES(‘Daffy Duck’);
15 INSERT INTO cartoons(cartoon) VALUES(‘Tom the Cat’);
16 INSERT INTO cartoons(cartoon) VALUES(‘Jerry Mouse’);

So, I tried to import:

% java -cp /usr/lib/jvm/java-6-sun/db/lib/derby.jar:/usr/lib/jvm/java-6-sun/db/lib/derbytools.jar:. org.apache.derby.tools.ij  IJ ERROR: Unable to establish connection
ij> IJ ERROR: Unable to establish connection
ij>

Hm, weird. I started ij tried running all the commands manually and that worked fine. Then I found out that could call ij with the name of the file directly (that is without the redirection). This would echo all the commands.

% java -cp /usr/lib/jvm/java-6-sun/db/lib/derby.jar:/usr/lib/jvm/java-6-sun/db/lib/derbytools.jar:. org.apache.derby.tools.ij cartoons.sql                   
ij version 10.3
ij> /*
 * Here there be 'toons!
 */

CONNECT 'jdbc:derby:toonsdb;
IJ ERROR: Unable to establish connection
ij> create=true';

CREATE TABLE cartoons(
    cartoon_id INT NOT NULL PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
    cartoon VARCHAR(255) NOT NULL,
    UNIQUE(cartoon)
);

INSERT INTO cartoons(cartoon) VALUES('Bugs Bunny');
INSERT INTO cartoons(cartoon) VALUES('Daffy Duck');
INSERT INTO cartoons(cartoon) VALUES('Tom the Cat');
INSERT INTO cartoons(cartoon) VALUES('Jerry Mouse');
;
IJ ERROR: Unable to establish connection
ij>

Something is not right. I thought the comment should be discarded? So why do I see it in the output above? It doesn’t give any errors per se, but its presence there is disturbing. What happens if I remove the comment?

% java -cp /usr/lib/jvm/java-6-sun/db/lib/derby.jar:/usr/lib/jvm/java-6-sun/db/lib/derbytools.jar:. org.apache.derby.tools.ij cartoons.sql
ij version 10.3
ij> CONNECT 'jdbc:derby:toonsdb;create=true';
ij> CREATE TABLE cartoons(
    cartoon_id INT NOT NULL PRIMARY KEY GENERATED ALWAYS AS IDENTITY,
    cartoon VARCHAR(255) NOT NULL,
    UNIQUE(cartoon)
);
0 rows inserted/updated/deleted
ij> INSERT INTO cartoons(cartoon) VALUES('Bugs Bunny');
1 row inserted/updated/deleted
ij> INSERT INTO cartoons(cartoon) VALUES('Daffy Duck');
1 row inserted/updated/deleted
ij> INSERT INTO cartoons(cartoon) VALUES('Tom the Cat');
1 row inserted/updated/deleted
ij> INSERT INTO cartoons(cartoon) VALUES('Jerry Mouse');
1 row inserted/updated/deleted
ij>

Go figure! The comment is not being parsed, but does not end in a ; so it causes errors when I try to CONNECT. How fucking stupid is that?

For those who want an embeddable database written in Java, look at HSQLDB.

Upgraded to Firefox 3beta4

I had a little bit of time over so I was able to upgrade to the latest beta of Firefox 3. No issues so far. It hasn’t crashed yet, my ScrapBook extension seems to work and the flash, java and mplayer plug-ins as work fine.

But it *still* cannot keep my bookmarks sorted alphabetically.

JBoss on Debian quickstart

I frequently get asked about setting up and using JBoss, so I thought I’d write a bit about it, just in case it helps anybody out there. In this first post I’ll just show you how to get JBoss AS up and running. In later posts I’ll get into more details, and show you how to set up things like JBoss Portal. I’m doing this on a Debian Etch system, but the instructions should more or less apply across Linux/UNIX systems.

What about JBoss in the Debian repositories?

If you feel like giving it a try do:

# aptitude install jbossas4

But, it won’t do much (at least they didn’t last time I tried). The JBoss packages in Debian need some work before they become usable. (I’m not sure about the status in Sid.)

Downloading

Before you do anything else, you need to have Java installed. I’m using Sun’s JDK 6, and that seems to work fine. JDK5 should also work.

Go here and grab a copy of JBoss AS. Latest stable at this time of writing is 4.2.2 GA. (I’ll probably do a post on JBoss 5 at some point.) When you get to the Sourceforge page, make sure you grab the correct package. The packages with src in their names are source packages. Only download them if you want to compile JBoss yourself. I will assume you don’t.

Installing

Once you have downloaded JBoss AS, you need to extract it. I chose to do so into /opt. The following needs to be done as root:

# cd /opt
# unzip /home/lorenzod/dl/jboss-4.2.2.GA.zip

After a few seconds it should be done, and ls will show you that there is a new directory called jboss-4.2.2.GA. If you want you can change the name of the directory to jboss, jboss4 or whatever makes you happy. I’ll keep the name as it is.

Testing the server

To make sure everything is fine up to this point, let’s try to start the server.

# cd /opt/jboss-4.2.2.GA
# bin/run.sh

You should see loads of log messages flying by in your terminal. That is normal. Eventually you will see a message like this:

22:31:21,073 INFO  [Server] JBoss (MX MicroKernel) [4.2.2.GA (build: SVNTag=JBoss_4_2_2_GA date=200710221139)] Started in 14s:615ms

Note that by default JBoss will only listen on 127.0.0.1 (localhost), which means it’s only accessible from the computer you are running JBoss on. To make it listen to other interfaces as well restart the server like this (you can press CTRL+C to stop JBoss if it s running):

# cd /opt/jboss-4.2.2.GA
# bin/run.sh -b 0.0.0.0

This will make JBoss listen on all the network interfaces. If you only want it to listen on a specific interface, use that IP address after the -b, for example bin/run.sh -b 192.168.0.5 will make JBoss only listen on the IP address 192.168.0.5.

Next, open a browser. If your browser is on the same machine as JBoss AS, you can browse to http://localhost:8080. Otherwise you’ll have to use either the name of the machine or its IP address. I’ve got JBoss running on 192.168.0.5 and that machine is known as devel in our local DNS, so I can access JBoss as by either http://192.168.0.5:8080 or http://devel:8080.

If you get a page server by JBoss then everything is working. Otherwise, you will have to go back and make sure you didn’t leave something out.

Other

Eventually, you will want to add something like this into your ~/.zshrc (or whichever start-up files your $SHELL uses):

export JBOSS_HOME=/opt/jboss-4.2.2.GA

IIRC, the JBoss documentation also suggests you add $JBOSS_HOME/bin to your PATH. I don’t personally do that because the names of the scripts in that directory are too generic (run.sh, shutdown.sh etc). But if you like you can do this somewhere (~/.zshenv is a good idea):

path=($path $JBOSS_HOME/bin)

Obviously you would have to set JBOSS_HOME before this is executed, otherwise you’ll have problems. You might want to rename the scripts to jboss-run.sh, jboss-shutdown.sh, etc.

Conclusion

I have shown how to install JBoss and get it up an running. It won’t start automatically when you boot the server yet. You will need an init script for that. I will get to that soon.

Note that there isn’t all that much interesting you can do with the application server yet. In order to make it more interesting you will need to develop some applications for it, or install some existing ones. Next time, I’ll look at how to set up JBoss Portal.

Alfresco on Debian: using MySQL

Yesterday I blogged about how to perform the most basic possible installation of Alfresco on a Debian system. That setup included using Hypersonic SQL as the database backend. Eventually you will want to migrate to using MySQL instead. Let us look at how to set it up.

I am assuming that you have grabbed the Alfresco Tomcat bundle (the community version). I am also assuming that you are running Debian Etch or beyond (if you are running experimental and things don’t work, that’s your mess to sort out). You should also have MySQL installed and up and running. I have MySQL 5.0.45 here and that works fine.

Setting it up

Before you start, make sure the Alfresco server is not running. Shut it down if it is. Then you need to run an SQL script that will create the MySQL database for Alfresco to use. Do this (as root):

# cp /opt/alfresco/extras/databases/mysql
# mysql --user root -p < db_setup.sql

You should be prompted for the password of the MySQL root user. If you don’t have a password for the root user, you should leave out the -p option. Of course, you are already aware of the security implications of not having a password set, so I won’t go over that here. Your server, your choice.

Next, you need to edit two files located in /opt/alfresco/tomcat/shared/classes/alfresco/extension. The first one is custom-repository.properties where you need to comment out the lines relating to HSQL and uncomment the MySQL ones.
Here is the part of the file that needs to be modified with the changes made.

#
# HSQL connection
#
#db.driver=org.hsqldb.jdbcDriver
#db.url=jdbc:hsqldb:file:alf_data/hsql_data/alfresco;ifexists=true;shutdown=true
;

#
# MySQL connection (This is default and requires mysql-connector-java-5.0.3-bin.
jar, which ships with the Alfresco server)
#
db.driver=org.gjt.mm.mysql.Driver
db.url=jdbc:mysql://localhost/alfresco

The other file that needs to be edited is custom-hibernate-dialect.properties. Again, un-comment the line that relates to MySQL and comment out the one relating to HSQL. Look at the very top of the file and you will find the lines to modify.

#
# HSQL dialect
#
#hibernate.dialect=org.hibernate.dialect.HSQLDialect

#
# MySQL dialect (default)
#
hibernate.dialect=org.hibernate.dialect.MySQLInnoDBDialect

If this was a completely new Alfresco installation and you have never run it before you should be good to go. Start the Alfresco server and wait. It does take a little longer to start Alfresco when you are using MySQL. You could follow along by doing:

# tail -f /opt/alfresco/alfresco.log

That will give you Alfresco’s log output in the terminal.

If it doesn’t work

If the Alfresco server has been started at least once before migrating to MySQL, Alfresco will probably not start. You will be able to start Tomcat, but the Alfresco application will not be available. If this is the case, look inside the log file and you should find a few lines like this:

Caused by: org.alfresco.repo.search.SearcherException: More than one root node i
n index: 2

This fix for that, which is available here is:

1. # cd /opt/alfresco
2. # ./alfresco stop
3. # rm -rf alf_data/lucene_indexes
4. # cd tomcat/webapps/alfresco/WEB-INF/classes/alfresco
5. # vim repository.properties

Make the following changes to the file:

# The index recovery mode (NONE, VALIDATE, AUTO, FULL)
#index.recovery.mode=VALIDATE
index.recovery.mode=FULL

# Change the failure behaviour of the configuration checker
#system.bootstrap.config_check.strict=true
system.bootstrap.config_check.strict=false

Now you can start Alfresco again. Hopefully it will work. If it doesn’t, stop Alfresco again, drop the database and re-create it, remove the /opt/alfresco/alf_data/lucene-indexes directory again and start. Then it *should* work.

Once you have been able to start Alfresco successfully and gone to http://localhost:8080/alfresco to confirm that it is indeed working, you can revert the changes that you made in /opt/alfresco/tomcat/webapps/alfresco/WEB-INF/classes/alfresco/respository.properties. If you don’t, Alfresco will attempt a full index recovery every time you start Alfresco.

The best option is probably that you configure Alfresco to use MySQL instead of HSQL before you start it the first time. That way you should not have any issues (at least I never have).

Other thing that could go wrong

Occasionally you see error messages like this one in your log file:

java.io.IOException: Cannot bind to URL [rmi://localhost:50500/alfresco/jmxrmi]:
 javax.naming.NameAlreadyBoundException: alfresco/jmxrmi [Root exception is java
.rmi.AlreadyBoundException: alfresco/jmxrmi]

They will show up if you try to start Alfresco when its already running. But there are a few other circumstances as well. One is if you shut it down and restart it without giving it enough time to fully shut down. You can do:

# ps ax | grep java

to see if there are any java processes related to Tomcat and/or Alfresco still alive. If you find any, give them a minute or two to terminate.

While you are trying to migrate to MySQL, it sometimes happens that some processes refuse to close (I have waited for over twenty minutes without them croaking). In that case, just use ps to find the PID of the process and manually kill it.

For example, suppose I find this:

# ps ax | grep java
11180 pts/0    Sl     0:40 /usr/lib/jvm/java-6-sun/bin/java -Xms128m -Xmx512m -server -XX:CompileCommand=exclude,org/apache/lucene/index/IndexReader$1,doBody -XX:CompileCommand=exclude,org/alfresco/repo/search/impl/lucene/index/IndexInfo$Merger,mergeIndexes -XX:CompileCommand=exclude,org/alfresco/repo/search/impl/lucene/index/IndexInfo$Merger,mergeDeletions -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=/opt/alfresco/tomcat/conf/logging.properties -Djava.endorsed.dirs=/opt/alfresco/tomcat/common/endorsed -classpath :/opt/alfresco/tomcat/bin/bootstrap.jar:/opt/alfresco/tomcat/bin/commons-logging-api.jar -Dcatalina.base=/opt/alfresco/tomcat -Dcatalina.home=/opt/alfresco/tomcat -Djava.io.tmpdir=/opt/alfresco/tomcat/temp org.apache.catalina.startup.Bootstrap start
11362 pts/0    R+     0:00 grep java

The process with PID 11180 is an Alfresco process. If I have waited long enough for this to terminate, but it isn’t, step in and be the executioner:

# kill 11180

Conclusion

Migrating Alfresco to MySQL involves a little more work than just using HSQL, but not much more. It really isn’t that complex, just that it can be a bit frustrating when you keep trying to get it up and running and it keeps refusing. But just monitor you log file and you should be okay.

Next time I’ll look at how to set Alfresco using only the WAR file, into your own Tomcat installation. This is a good idea if you already have Tomcat installed. There are a few additional things you will have to do if you use your own Tomcat installation instead of the bundled one. But, on the other hand, using our own gives us the opportunity to experiment with Alfresco on Tomcat 6.

Setting up Alfresco on Debian (Tomcat bundle)

I have been playing around a lot with Alfresco of late. I’ll write some posts about what I have learned up to now, just in case it helps somebody out there.

Before we begin, here is what I’m using:

• Debian Etch. For most part, these instructions should work on any Linux system.
• Sun Java 6 (packages from Sid repository)

Installation

In this post I am going to use the Alfresco Tomcat bundle, which is available here. I will start by using all the defaults, which will use Hyperonic SQL as a backend database. This may not be a good idea in the “real world”, but is acceptable for testing. Most importantly, we should make sure that works before moving on.

Once you have downloaded Alfresco, you need to extract it. Perform the following (as root):

# mkdir /opt/alfresco
# cd /opt/alfresco
# tar zxf /home/lorenzod/dl/alfresco-community-tomcat-2.1.0.tar.gz

Of course, you will need to change the path in the last line to wherever you downloaded Alfresco to. If everything went well, ls should give you the following files (and directories):

alf_data
amps
extras
README_osx.txt
zstart_oo.sh
alfresco.log
apply_amps.sh
licenses
README.txt
alfresco.sh
bin
README_mysql.txt
tomcat

Starting the server

Before you can start the Alfresco server, you need to set JAVA_HOME if you have not already done so. On my system, I did the following:

# export JAVA_HOME=/usr/lib/jvm/java-6-sun

(Check to make sure where your JDK is installed.) To save yourself from having to type this over and over again, add that line to file such as /etc/profile. That way JAVA_HOME will automatically be set for you.

Now you should be able to start Alfresco. Type (still as root):

# ./alfresco.sh start

Hopefully you will see this message:

Using CATALINA_BASE:   /opt/alfresco/tomcat
Using CATALINA_HOME:   /opt/alfresco/tomcat
Using CATALINA_TMPDIR: /opt/alfresco/tomcat/temp
Using JRE_HOME:       /usr/lib/jvm/java-6-sun

If, on the other hand, you get this message:

Neither the JAVA_HOME nor the JRE_HOME environment variable is defined
At least one of these environment variable is needed to run this program

Then JAVA_HOME hasn’t been set properly. Go back and follow the instructions, properly this time. Once Alfresco is up and running, open a browser and go to http://localhost:8080/alfresco if you are browsing from the same machine that you installed Alfresco to, or http://your-server:8080/alfresco if you are browsing from a different machine.

If that doesn’t work, try http://localhost:8080 to make sure Tomcat itself is up. Depending on your hardware, it make take a while for Alfresco to fully pull itself up, so give it a little time and try again. Otherwise look at the log file (/opt/alfresco/alfresco.log), something you should do even if Alfresco does run properly.

Once you see the Alfresco dashboard in your browser, you can click around a little to familiarize yourself with the interface. At the top you will see a login link. Click there and you can log in. The only user that exists by default is admin with password admin.

Peeking at the log file

When you look at the log file there are some errors and warnings that you might see. I will try to explain them as we go along.

With the current set-up the only error you are likely to see is:

21:31:17,079 ERROR [org.alfresco.smb.protocol.netbios] NetBIOSNameServer setup e
rror:
java.net.BindException: Address already in use

This indicates that you already have Samba running. This will prevent Alfresco’s own CIFS server from starting. To prevent this, stop the Samba server before running Alfresco. Note that this error will not prevent Alfresco itself from running, but you will not be able use its CIFS server.

Shutting down the server

To shut down Alfresco, go to /opt/alfresco and type ./alfresco stop. If anything goes wrong during shutdown, there may be some running Java processes on the system. Use ps to locate them, and kill them manually. Normally you shouldn’t have any problems shutting Alfresco down.

Conclusion

As you have seen, getting Alfresco up and running on Debian is not difficult. Next time, I will show you how to migrate the database to MySQL, and some of the common issues you are likely to encounter along the way.

SCIM + Lenny = Ouch!

Just a little warning if you’ve got Lenny in your sources.list and considering upgrading. If SCIM is something you cannot live without, you may want to postpone that aptitude upgrade. Otherwise your computer will soon be as fast as a ZX Spectrum.

The SCIM I upgraded to was 1.4.7-3. There is Debian bug #434180 which contains a possible work-around. You could try it, but it doesn’t work for me. My problem is not sporadic lock-up, but that SCIM itself takes forever to start and consumes everything in its sight while doing so.

I’m going to run a few more searches and play around a bit. If the bug doesn’t seem to be in Debian BTS, I’ll have to add it. But I’m still hoping that there’s something I can do to at least (1) get rid of the problem, or (2) be a duplicate of an existing bug.

Of course, I’m on a etch/lenny/sid hybrid with possibly a sprinkle of experimental as well (although this was my first upgrade in ages). But still, think about it for at least 0.02 seconds before starting that upgrade.