SAGA Shell Tutorial

Introduction

SAGA provides a high-level programming interface to various grid systems. SAGA stands for the Simple API for Grid Applications. The API is available for C++, Python, and Java.

In this tutorial, we will use Java SAGA to create a simple shell program. Our shell will mimic a very basic UNIX prompt in which we can issue commands to browse file systems and execute jobs.

For this tutorial, you will need:

1. A Java compiler (Sun Java SDK, version 1.6 or higher)
2. Ant

Installing Java SAGA

1. Go to the files on the website and get a file called saga-impl-1.1-src.zip Unzip’ing it results in a directory saga-impl-1.1.
2. Compile Java SAGA:

1. cd saga-impl-1.1
2. ant

4. Set the environment variable JAVA_SAGA_LOCATION to the directory where you have put it:

1. using bash: export JAVA_SAGA_LOCATION=/absolute/path/to/saga-impl-1.1
2. using csh (or Windows): please do the right thing for your environment

Getting the Shell Tutorial Code

1. Get complexHPC2011-SAGA-HandsOn.zip from the files on the website

Unzip the file, resulting in a directory saga-shell

2. Compile the shell framework:

1. cd saga-shell
2. ant
   (if this gives errors, you likely have problems with JAVA_SAGA_LOCATION)

Running the Framework

1. Start the shell with ./saga-shell.sh

1. The prompt shows the URLs of the current resource manager (local://localhost) and the current working directory (file://localhost/home/you/saga-shell).

2. get help using ‘help’

1. This lists the commands understood by the SAGA shell.
2. The current working directory can be modified with 'cd'. The other file-related commands (pwd, ls, cp, mv, mkdir, and rm) mimic the common UNIX commands, but with URLs instead of normal paths.
3. The job-related commands (prm, crm, discover, run, jobs, and kill) use the current resource manager to discover resources and execute jobs.
4. The context-related commands addc, lsc, and rmc respectively add, list, and remove security contexts from the default SAGA session.
5. These contexts are used to authenticate with certain filesystems and resource managers, and typically contain username/password combinations, paths to certificates etc.

The 'saga-shell.sh' script sets the correct classpath, configures the logging system and starts the main class 'org.ogf.saga.apps.shell.SagaShell'.

Each command in the shell has an associated object in the 'org.ogf.saga.apps.shell.command' package that implements the interface 'org.ogf.saga.apps.shell.command.Command'. The main class creates a map of command names (e.g. 'ls') and the corresponding Command object (e.g. 'org.ogf.saga.apps.shell.command.ListDirectory'). The shell then repeatedly reads a command line, looks up the associated command object, and invokes its 'execute()' method.

Three commands are not implemented yet: 'ls', 'kill', and 'cat'. You task in this tutorial is to implement the 'execute()' methods of their command objects.

Enabling logging output

A great help during debugging are the log messages of the SAGA shell and (possibly) SAGA Java. Both use SLF4J plus log4j for logging statements. You can configure the logging output of the SAGA shell reference implementation and the tutorial code by editing, the configuration file 'log4j.properties'.

Example: to make the SAGA shell more verbose, edit log4j.properties and uncomment this line:

#log4j.logger.org.ogf.saga.apps.shell=DEBUG

Useful references

1. Javadoc of the Java SAGA API (http://www.cs.vu.nl/ibis/javadoc/saga-api-1.1/)

1. It can also be found as part of the implementation, at doc/saga-api-1.1/index.html

2. Javadoc of the Java 1.6 API (http://download.oracle.com/javase/6/docs/api/)
3. OGF SAGA specification (GFD.90) (online at http://www.ogf.org/documents/GFD.90.pdf)

Task 1: implement 'ls'

Your first task is to implement the 'ls' command using SAGA. You will have to edit the 'execute()' method of the class 'org.ogf.saga.apps.shell.command.ListDirectory'. Currently, this method only prints the message "You will have to implement 'ls'!".

Invoking 'ls' should print a list of entries in the current working directory. Java experts may try to print the entries in alphabetical order. Note that for simplicity, the 'ls' command does not take any parameters.

Before you begin: trying to implement a full-fledged “ls -l”, including file sizes, owner, group, and permissions is rather tricky, due to limitations of the Java language. It is fine if you list just the file names. If you want it fancy, you can show whether an entry is a directory or the size of a file.

Implementation hints:

1. Read the Javadoc comments in the class 'org.ogf.saga.apps.shell.command.Command' to understand the parameters to the 'execute()' method.
2. Pseudo code of the 'ls' implementation:

1. Get the current working directory
2. Get a list of entries
3. Possibly sort the list
4. Print each entry in the list

3. Possible SAGA exceptions will have to be caught and shown to the user

Useful methods:

1. org.ogf.saga.apps.shell.Environment.getCwd()
2. org.ogf.saga.file.Directory.list()
3. java.util.Arrays.sort()
4. org.ogf.saga.apps.shell.Util.printSagaException()

Task 2: implement 'kill'

Your second task is to implement the 'kill' command, which terminates a background job. The 'kill' code must be added to the method org.ogf.saga.apps.shell.command.KillJob.execute().

Background jobs can be started in the SAGA shell with a 'run' command that ends with a '&'. Example:

[local://localhost+file://localhost/home/you] run /bin/sleep 200 &

All background jobs are added to the task container in the Environment object of the SAGA shell. The 'jobs' command lists all background jobs:

[local://localhost+file://localhost/home/you] jobs
[7615bb11-3bf5-4f3a-a9f5-c9ae177fe58c] RUNNING        [JavaGAT]-[0]        /bin/sleep 200

The first string between square brackets is the 'cookie' of the job: a string that identifies the job in the task container of background jobs. The remainder of the line contains more information about the background job, e.g. the job adaptor used (in this case, 'JavaGAT') and the job's startup command. The 'cookie' of the job is the single argument that should be given to a 'kill' command. In this example, we could kill the background job with:

[local://localhost+file://localhost/home/you] kill 7615bb11-3bf5-4f3a-a9f5-c9ae177fe58c

Jobs are run with the current resource manager (in this case 'local://localhost'). The resource manager can be changed with the command 'crm' command. Note that running remote jobs (e.g. via ssh) may require an extra security context. The 'addc' command can be used to add a context, and 'lsc' lists all contexts. Example of running a remote job via ssh:

[local://localhost+file://localhost/home/you] addc ssh UserID=john UserPass=ask
Value of 'UserPass': ******
[local://localhost+file://localhost/home/you] crm ssh://host.example.com
[ssh://host.example.com+file://localhost/home/you] run /bin/sleep 300 &
[ssh://host.example.com+file://localhost/home/you] jobs
[530162fc-4f84-434c-87a7-79c674b4d81d] RUNNING        [JavaGAT]-[1]        /bin/sleep 300
[ssh://host.example.com+file://localhost/home/you] kill 530162fc-4f84-434c-87a7-79c674b4d81d
[ssh://host.example.com+file://localhost/home/you] jobs
[530162fc-4f84-434c-87a7-79c674b4d81d] CANCELED        [JavaGAT]-[1]        /bin/sleep 300

Now implement and test the 'kill' command yourself!

Useful methods:

1. org.ogf.saga.apps.shell.Environment.getBackgroundJobs()
2. java.lang.Integer.parseInt()
3. org.ogf.saga.task.TaskContainer.getTask()
4. org.ogf.saga.job.Job.cancel()

Task 3: implement 'cat'

Your third task is to implement the 'cat' command in the method org.ogf.saga.apps.shell.command.PrintFile.execute().

The 'cat' command prints the contents of a (possibly remote) file. Its argument consists of a single URL that denotes to file to print. The URL can either be relative to the current working directory (e.g. "cat foo.txt"), absolute in the same filesystem (e.g. "cat /etc/passwd") or absolute in a different filesystem (e.g. "cat ssh://example.com/etc/passwd").

Implementation hints:

1. Pseudo code:

1. Open the file
2. Create a SAGA buffer
3. While something has been read from the file:
4. Print what has been read
5. Finally, close the file

2. Accessing remote files with your 'cat' may require an extra security context. Example using the default 'anonymous' FTP context:
3. [local://localhost+file://localhost/home/you] addc ftp
   [local://localhost+file://localhost/home/you] lsc
   [0] ftp UserID=anonymous UserPass=<secret>
   [local://localhost+file://localhost/home/you] cat ftp://kernel.org/welcome.msg
                               Welcome to the
   
                           LINUX KERNEL ARCHIVES
                               ftp.kernel.org

Useful methods:

1. org.ogf.saga.file.Directory.openFile()
2. org.ogf.saga.file.File.read(org.ogf.saga.buffer.Buffer, int offset, int len)
3. java.lang.String(byte[] bytes)

Solutions

The solutions to the SAGA shell tutorial are available are available inside the Java SAGA distribution, well, under apps/shell/...