DIPUS - Distributed IPC by Proxying UNIX Sockets================================================Sockets with user authentication, file permissions, encryption,service discovery and network transparent addressing.
Using services:---host1$ dcat lame-svc foo.wav | dcat madplay.host3host1$ dcat sep://example.com/image/jpeg/to/image/png foo.jpeg | dcatssh://user / another.com/image/png/to/text/plain | less
Creating services:---host3$ dipus-local-pipe 'madplay -' madplayhost2$ dipus-export-pipe 'lame - -' lame-svc
Some performance numbers:---Local throughput around 50MBps (or 100MBps, depending whether you usesocat as a client or not), unencrypted network throughput 9.8MBps in a100Mbps LAN, CPU-limited when using SSH (my other computer is slow.)
In what languages can I use and create services?---Anything that supports a) UNIX domain sockets, and b) listingfilesystem directories. So, just about every language available. Ifyou install the helper scripts in bin, you don't even need those, justthe ability to read from stdin and write to stdout.
Download: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2.tar.gzRDocs: http://www.cs.helsinki.fi/u/irkheikk/dipus-rdoc/Darcs repo: http://dark.fhtr.org/repos/ruby-dipus
Readme: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/readme.txtDesign: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/doc/dipus-design.txtProtocol spec: http://www.cs.helsinki.fi/u/irkheikk/ruby-dipus-0.1.2/doc/dipus-specification.txt
^^^^^^^^^^^^^^^^^^^^^^^^^^
Hi, designed and implemented a distributed IPC system that takeslessons from UNIX (keep it simple) and HTTP (keep it simple.) Opinionson the design or anything else? I'm dying to get some feedback,writing stuff in a bubble can take you only so far.
And, now, with the first version of this December project out of theway, I'm off to librend land~--Ilmari Heikkinen
vvvvvvvvvvvvvvvvvvvvvvvvvv
What is this for?-----------------You have a program. You want to access it remotely with the minimumhassle. It'd be nice to access it efficiently locally too. Withoutrewriting your connection code.
And it'd be nice to put some connections over SSH. And be notifiedwhen a service dies or a new one is created. And not expose anyof the individual services to the network, or even to other userson the machine running them. And it'd be cool to be able to have adifferent name and protocol for each service without having to shoehornFTP over XML-RPC. Oh, also, display a real-time list of servicesrunning on the local network and on explicitly defined machines thataren't on the local network. And it'd be especially cool if all thesoftware could just use existing libraries like sockets and files.
So, take UNIX sockets and sprinkle some sugar to make them usable fordistributed IPC.

Supported systems-----------------Should work on Linux and *BSD, likely on OS X too.No Windows support, although it's theoretically possible.

Required--------Ruby version 1.8.2+ (one released in 2005)cat

Recommended-----------OpenSSH version > 3.9 (for connection sharing support (ControlPath))ssh-agent setup (for maximum convenience)socatMore than one computer.

How does it work?-----------------To create a service, make a listening socket in a subdir of/tmp/dipus, so that the path is like:/tmp/dipus/exported/audio/x-wav/to/audio/mpeg/lame.mybox..0
To access a service, open its socket.
To make a service only accessible to members of group 'audio':chgrp audio socket_namechmod 770 socket_name
Listing services that you can see:tree /tmp/dipus
DIPUS provides helper methods and daemons to manipulate thesockets in /tmp/dipus, e.g. create proxies to remote sockets,delete dead sockets, provide network access to exportedsockets, find sockets by their service name, hostname, orprotocol.

Quick usage example-------------------# (optional) start dipusd to automatically discover and import services,# export services, and delete dead sockets.#[kig / desktop] dipusd[kig / encode] dipusd
# Create local madplay service for playing MP3s on# the computer connected to speakers.# The first argument is the command to run for each incoming connection.# The second argument is service_name/protocol, e.g. my_www/http.## The following command creates a socket with the path# /tmp/dipus/local/audio/mpeg/to/audio/out/madplay.stereo..0# and runs 'madplay -' for each incoming connection, setting the# connection to madplay's stdin and stdout.#[kig / stereo] dipus-local-pipe 'madplay -' madplay/audio/mpeg/to/audio/out
# Create an exported lame service for encoding wavs to MP3s.#[kig / encode] dipus-export-pipe 'lame - -' lame/audio/x-wav/to/audio/mpeg
# Manually import the services from stereo.# Import over SSH to gain access to all services and not just exported ones.#[kig / desktop] dipus-import ssh://stereo
# Dipusd should've discovered and imported the lame service from encode,# so we don't need to manually import that. But if that didn't happen, we# can import it by doing `dipus-import sep://encode`, sep being# Socket Export Protocol (see doc/dipus-specification.txt)
# Now, let's pipe some wavs to the stereo.# First encode the wav to MP3, using any service with a protocol that says# that the service converts audio/x-wav to audio/mpeg.# Play using a service named madplay running on stereo.#[kig / desktop] dcat audio/x-wav/to/audio/mpeg *.wav | dcat madplay.stereo
# We could also access the services without importing them by giving the# service URI as the service name.## Here's the above piping example using service URIs.# Dcat lists the services matching the URI and uses the first match.#[kig / desktop] dcat sep://encode/audio/x-wav/to/audio/mpeg *.wav | \              dcat ssh://stereo/audio/mpeg/to/audio/out/madplay.stereo..0

List of DIPUS commands---------------------- Pass --help as an argument to get the full USAGE.
 * dls -- list services   dls sep://foobar/audio   dls ssh://barfoo
 * dcat -- pipe stdin and/or given files to a DIPUS socket   dcat lame foo.wav bar.wav > foobar.mp3   dcat ssh://foobar/image/png/to/image/jpeg cat.png > cat.jpg
 * dipus-import -- imported all found service from an URI   dipus-import ssh://foobar/audio   dipus-import sep://barfoo
 * dipusd -- start exporter, importer, advertiser, discoverer,             dead socket monitor, and default services
 * dipus-exporter -- start exporter
 * dipus-discoverer -- discover exporters on the LAN and print                       their addresses
 * dipus-service-monitor -- monitor for service changes on localhost,                            used by SSH importers
 * dipus-ssh-importer -- import services over SSH to user's DIPUS                         subdirectory (/tmp/dipus/username), can use                         discovery to find exporter-running hosts on                         localnet
 * dipus-local-pipe -- create a local DIPUS service that forks the                       given command for the connection   dipus-local-pipe 'head' head/text
 * dipus-export-pipe -- create an exported DIPUS service like above   dipus-export-pipe 'madplay -' madplay/audio/mpeg/to/audio/out

Using socat to create and connect to DIPUS services---------------------------------------------------# proxy a web servermkdir -p /tmp/dipus/exported/httpsocat UNIX-LISTEN:/tmp/dipus/exported/http/foo.foobar..0,fork \      TCP4-CONNECT:foo.example.com:80rm /tmp/dipus/exported/http/foo.foobar..0
# using a lame service to encode a wavsocat foo.wav \      UNIX-CONNECT:/tmp/dipus/local/audio/x-wav/to/audio/mpeg/lame.foobar..0 \      > foo.mp3
# or by using dls to find the filenamesocat foo.wav UNIX-CONNECT:`dls -1 -f lame` > foo.mp3
# proxy a DIPUS service to a TCP socketsocat TCP4-LISTEN:1234,fork EXEC:'dcat lame'

Possibilities-------------Carry around a mobile phone with an Internet connection. Use it torun a media player. The media player sees files on your home computerand the files on the local network. Then it pipes the files in itsplaylist to the closest encode server, and from there to the wantedoutput (headphones, speakers, file, streaming server...)
Send your 3D animation file over the Internet to a thousand idlemachines to render.
Walk into an auditorium and display slides on the overhead projector,streamed from a network mount, controlled from your mobile.
Run a search engine, giving URIs to remote services, making automatablethe task of finding and utilizing a network service.