LFC is an efficient, user-level communication system for Myrinet.
LFC provides:

• Reliable point-to-point and multicast communication
• A simple and flexible packet-based communication abstraction
• Interrupt-driven and polling-based packet delivery
• Delayed interrupts to reduce interrupt overhead

• Raoul Bhoedjang
• Tim Ruhl
• Kees Verstoep
• Henri Bal

Using LFC, we have implemented

• The Panda communication library
• The Orca parallel programming language
• The TreadMarks shared virtual memory system
• The CRL distributed shared memory system
• The MPI message passing system

LFC is Open Software and runs on Linux. The configurations we tried are RedHat 5.2 (kernel version 2.0.36) and RedHat 6.0 (kernel version 2.2.5-15).

LFC is available under a very liberal XFree86-style copyright (i.e., do what you want with it; just leave the copyright notices intact).

To build and use it, you will need the following packages:

• The LFC-2.0 (alpha 1) source code , which includes documentation and installation instructions.
• A number of patches that should be applied to the Myricom-supplied device driver (version 3.22c)
• Optionally, you can download the following RedHat 6.0 based LFC binary distribution that should be installed in /usr/local. In addition to a precompiled version of LFC itself, it contains all required Myricom libraries, binaries and tools. Note that the LFC source distribution does contain instructions for building this all yourself, if you prefer.

Also provided for illustrative purposes only is the following package, containing the code of the experimental designs described in our TOCS publication (see below):

• LFC-2.9-alpha source code

For the programming interface that LFC offers, have a look at the on-line LFC User Guide in Postscript or PDF format. Note that this documentation is already contained in the LFC source and binary distributions. Performance

Here are a few performance numbers for LFC, measured on the Distributed ASCI Supercomputer (PentiumPro's 200 MHz, LANai-4.1). The multicast measurements were done on 64 nodes.

• unicast null latency: 10.4 usec
• unicast throughput: 68.2 MByte/sec (76.2 MB/s with 2K pkts)
• multicast null latency (binary tree): 51.8 usec avg, 72.4 usec max
• multicast null latency (binomial tree): 48.8 usec avg, 57.5 usec max
• multicast throughput (binary tree): 24.5 MByte/sec (29.6 MB/s with 2K pkts)
• multicast throughput (binomial tree): 12.1 MByte/sec (14.8 MB/s with 2K pkts)

On more modern hardware (PIII 800 MHz, LANai-9.0, PCI 64bit/66MHz) we have measured the following with a new experimental version of LFC:

• unicast null latency: 3.9 usec
• unicast throughput: 127.2 MByte/sec (143.2 MB/s with 2K pkts)
• unicast throughput using a recent zerocopy extension: 236.2 MByte/sec

We are still working on software distributions for the following packages:

• An amazingly fast implementation of Java/RMI (Remote Method Invocation) on LFC using the Manta system.
• The latest version of the Panda communication library, which supports both LFC and traditional IP sockets.
• An extended version of the well-known MPI implementation MPICH, which uses the Panda communication interface.

Communication Architectures for Parallel-Programming Systems
R.A.F. Bhoedjang, June 2000

Page formats:

• US letter format: [PDF] [PS] [PS.gz]
• A4 format: [PDF] [PS] [PS.gz]

Papers

Cluster Communication Protocols for Parallel-Programming Systems (PDF, PS, PS.gz)
K. Verstoep, R.A.F. Bhoedjang, T. Ruhl, H.E. Bal, R.F.H. Hofman
ACM Transactions On Computer Systems (TOCS), Vol. 22, No. 3, August 2004.

Evaluating Design Alternatives for Reliable Communication on High-Speed Networks (PDF, PS, PS.gz)
R.A.F. Bhoedjang, K. Verstoep, T. Ruhl, H.E. Bal, R.F.H. Hofman
Nineth Int. Conf. on Architectural Support for Programming Languages and Operating Systems (ASPLOS-IX), Cambridge, MA, November 2000.

Reducing Data and Control Transfer Overhead through Network-Interface Support (PDF, PS, PS.gz)
R.A.F. Bhoedjang, K. Verstoep, T. Ruhl, H.E. Bal
First Myrinet User Group Conference, September 2000, Lyon, France

User-Level Network Interface Protocols
R.A.F. Bhoedjang, T. Ruhl, H.E. Bal
IEEE Computer, Vol. 31, No. 11, pp. 53-60, November 1998 (special issue on network interfaces).
For download: see the IEEE Computer website (may require subscription).
Note: this paper was based on technical report below.

Design Issues for User-Level Network Interface Protocols on Myrinet (PDF, PS, PS.gz)
R.A.F. Bhoedjang, T. Ruhl, H.E. Bal
Technical Report IR-455, Vrije Universiteit, Amsterdam, November 1998.

Efficient Multicast On Myrinet Using Link-Level Flow Control (PDF, PS, PS.gz)
R.A.F. Bhoedjang, T. Ruhl, H.E. Bal
International Conference on Parallel Processing, pp. 381-390, August 1998, Minneapolis, MN
(best paper award)

LFC: A Communication Substrate for Myrinet (PDF, PS, PS.gz)
R.A.F. Bhoedjang, T. Ruhl, H.E. Bal
Fourth Annual Conference of the Advanced School for Computing and Imaging,
June 1998, Lommel, Belgium.