Clustering programs

Lately, I have been working on refining my clustering program so that it functions without the need for sockets. After messing with the dirty things, I have found that file IO works better and more predictable than sockets. This does not imply that sockets never have any use. I am dealing with a machine that has restricted abilities to be connected to at any ports. This machine also has restricted abilities to send sockets out at any ports. My only link to these machines is through SSH.

The current plan has the central machine not only sending its input to all these machines, but a request-file. This file tells the servers, (1) what the program is. (2) the program's input, and (3) where the result should go to. The result isnt the output of the program but a low-bandwidth notification to the central machine that it has completed. The central machine at a later point probes for notifications. If all the notifications have arrived, and aggregation process occurs. As an added side-feature, a request-file may allow for sabotage. This is describes a situation where as soon as one server finishes it is obligated to kill other distributed processes. The sabotaging program then sends fake notifications from all the other processes.

The client is being written in Python and Common Lisp. For flexibility the server components were written in perl. The assumption being these machines should have perl installed. The most interesting aspect of coding this has thus far been effectively managing enormous file streams. Mismanaging this one crucial process makes the rest of the program pointless. It doesnt matter how fast your server programs parse the pieces of the input. A function that splits the original input in a horribly inefficient way is all it takes to kill a program.

New Projects

Now that I am a CL gardener this blog will serve a slightly new purpose. Instead of capturing minor insights into the Singularity and Artificial General Intelligence, I am just going to document what I do to help improve the Common Lisp community. Currently, I am working on getting CLOCC to be installable with ASDF-Install.

As it stands the CLOCC being "Common Lisp Open Code Collection" is old. It was written before nicer things really existed. One of these nice things is ASDF. This program which is tangentally linked to cliki.net is a way to do various things with libraries in Common Lisp. It works somewhat like make in Unix. One of the more interesting things that caught my eye was ASDF-install. ASDF-install lets you install libraries straight from the REPL. The problem arrives when one understands that CLOCC was written before ASDF-install existed. One of the things CLOCC impliments is MK-Defsystem, asdf's predeccesor.

So the program is made to use mk-defsystem. It should be fun to make it work with asdf-install. The creator of the project does not mind the change so I feel it will be a nice way to formally enter the community.

I was going to make about half the entries in this blog links but I always hate when others do that.