This is part 2 of a 2 part tutorial on Prosody, which will show you how to install and configure extra modules to extend prosody's functionality. Part 1, which deals with basic installation and testing of prosody, is here. Here's an overview of the important additional functionality we will add to prosody:

• Multi user chats (XEP-0045)
• Server side message storage (XEP-0313 Message Archive Management)
• File and image transfer (SI File Transfer and Jingle signalling; in-band and out-of-band methods)
• Stream Management for surviving connection interruptions and switches - particularly useful on mobile (XEP-0198 Stream Management)
• Synchronising messages between multiple devices (XEP-0280 Message Carbons)
• Battery Saving techniques for mobile clients (XEP-00352 Client State Indication)
• User Blocking (XEP-0191 Blocking Command)

Not all clients support all of these XEPs. I consider the Android client Conversations (F-Droid | Google Play) to be pretty cutting edge in this respect - it's pushing forward a lot of XEPs that are required if we are going to see XMPP compete with the features of proprietary chat clients, particularly on mobile.

Installing new modules

Prosody is bundled with about 55 modules, many of which are enabled by default. Default modules are stored at /usr/lib/prosody/modules; you can list the files in this directory to see which modules have been packaged with your version of prosody:

$ ls /usr/lib/prosody/modules/
adhoc                         mod_http_files.lua     mod_s2s
mod_admin_adhoc.lua           mod_http.lua           mod_s2s_auth_certs.lua
mod_admin_telnet.lua          mod_iq.lua             mod_saslauth.lua
mod_announce.lua              mod_lastactivity.lua   mod_storage_internal.lua
mod_auth_anonymous.lua        mod_legacyauth.lua     mod_storage_none.lua
mod_auth_cyrus.lua            mod_message.lua        mod_storage_sql1.lua
mod_auth_internal_hashed.lua  mod_motd.lua           mod_storage_sql.lua
mod_auth_internal_plain.lua   mod_net_multiplex.lua  mod_storage_xep0227.lua
mod_blocklist.lua             mod_offline.lua        mod_time.lua
mod_bosh.lua                  mod_pep.lua            mod_tls.lua
mod_c2s.lua                   mod_ping.lua           mod_unknown.lua
mod_carbons.lua               mod_posix.lua          mod_uptime.lua
mod_component.lua             mod_presence.lua       mod_vcard.lua
mod_compression.lua           mod_privacy.lua        mod_version.lua
mod_debug_sql.lua             mod_private.lua        mod_watchregistrations.lua
mod_dialback.lua              mod_proxy65.lua        mod_websocket.lua
mod_disco.lua                 mod_pubsub             mod_welcome.lua
mod_groups.lua                mod_register.lua       mod_windows.lua
mod_http_errors.lua           mod_roster.lua         muc

This modules in this list are all stable modules, which is why they have been bundled with prosody. However, they are by no means the only modules available. Some of the important modules are missing like mam (message archive management) which allows clients to receive a history of previous conversations from the server. There is also a package in the repos called prosody-modules, but it doesn't provide the modules we need:

$ apt-file list prosody-modules
prosody-modules: /usr/lib/prosody/modules/ldap.lib.lua
prosody-modules: /usr/lib/prosody/modules/mod_auth_ha1/mod_auth_ha1.lua
prosody-modules: /usr/lib/prosody/modules/mod_auth_ldap2/mod_auth_ldap2.lua
prosody-modules: /usr/lib/prosody/modules/mod_carbons/mod_carbons.lua
prosody-modules: /usr/lib/prosody/modules/mod_carbons_adhoc/mod_carbons_adhoc.lua
prosody-modules: /usr/lib/prosody/modules/mod_carbons_copies/mod_carbons_copies.lua
prosody-modules: /usr/lib/prosody/modules/mod_log_auth/mod_log_auth.lua
prosody-modules: /usr/lib/prosody/modules/mod_smacks/mod_smacks.lua
prosody-modules: /usr/lib/prosody/modules/mod_storage_ldap/ldap/vcard.lib.lua
prosody-modules: /usr/lib/prosody/modules/mod_storage_ldap/mod_storage_ldap.lua
prosody-modules: /usr/share/doc/prosody-modules/README.Debian
prosody-modules: /usr/share/doc/prosody-modules/README_mod_lib_ldap.md.gz
prosody-modules: /usr/share/doc/prosody-modules/changelog.Debian.gz
prosody-modules: /usr/share/doc/prosody-modules/copyright

Luckily, there's an easy way to get all of the most up to date modules, using a Mercurial repository hosted at the prosody website. Since this software is not packaged as part of the distro, we should put it in /opt. First, install mercurial, the revision control system:

sudo apt-get update
sudo apt-get install mercurial

Now change directory to /opt:

cd /opt

And then initialise the repository:

sudo hg clone https://hg.prosody.im/prosody-modules/ prosody-modules

This will create /opt/prosody-modules containing over 200 interesting modules. In future, you will have to update the repository manually, which you can do using Mercurial:

cd /opt/prosody-modules/
sudo hg update
sudo hg pull

If you want to see all of the available modules, list the directories in the repo:

ls /opt/prosody-modules/

Warning! If you're going to experiment with these modules, be aware that some of them are pretty bleeding edge and may be a bit buggy. Most of the directories contain a README.markdown file that explains the compatibility with different prosody versions - we're using 0.10, I've found that this means most modules labelled as compatible with version 0.9 (current stable) will work unless noted otherwise. To use these modules, we need to tell prosody where to look. Open /etc/prosody/prosody.cfg.lua and add this line above the modules_enabled section:

plugin_paths = { "/opt/prosody-modules" }

This will add /opt/prosody-modules to the default path /usr/lib/prosody/modules, making all of the modules in both directories available. Restart prosody:

sudo service prosody restart

Now we can add some modules!

Switching to sqlite storage

A few of the modules we are about to configure will need a way to store lots of data. The default method for storing all data in prosody is plain text files at /var/lib/prosody - in fact, if you're using the plain authentication mechanism you can read the passwords for users if you have root privileges, e.g.:

$ sudo cat /var/lib/prosody/samhobbs%2eco%2euk/accounts/sam.dat
return {
        ["password"] = "secret";
};

To install the driver required for prosody to use an sqlite database, we need to install the lua-dbi-sqlite3 package:

sudo apt-get update
sudo apt-get install lua-dbi-sqlite3

To make prosody use sql storage for data in all modules unless told otherwise, you can add this line to the config file:

default_storage = "sql"

We also need to add information about the sql database. If this was a mysql database, the brackets would contain a hostname, username and password. Since it's an sqlite database, it's just a database file that prosody will write to, and none of that is necessary.

sql = {
	driver = "SQLite3";
	database = "prosody.sqlite";
}

Be aware that after doing this, you will need to re-add your existing users, and all data will be reset.

Server side message storage (XEP-0313 Message Archive Management)

This module is really handy when you connect a new client, and you want to be able to retrieve old conversations. It's also useful in situations where you have two devices and one is offline for a bit while you are using the other - using a basic XMPP server, you'd only have a record of the conversation on whichever device you were using at the time, which is confusing if you switch devices often. In /etc/prosody/prosody.cfg.lua, add this inside the modules_enabled brackets. I like to put all of the contributed modules installed from the prosody-modules repository underneath a heading to remind me where they come from:

modules_enabled = {
...
	-- Contrib modules installed in /opt/prosody-modules
		"mam"; -- message archive management
}

We also need to add a way for prosody to store messages offline server-side. For performance reasons, it's important that mod_mam can store messages in a database instead of plain text files. By default, everything is stored in text files, unless you changed the default storage method earlier. The following configuration will enable messages to be stored in an sqlite3 database using mod_mam (not required if you specifed sql storage for everything by default, but it doesn't harm to specify it twice):

storage = {
	-- this makes mod_mam use the sql storage backend (others will use the default)
	archive2 = "sql";
}

Again, restart prosody to make the changes take effect:

sudo service prosody restart

Message Carbons (XEP-0280)

Message carbons allow clients to request carbon copies of all messages that are sent. If you use two clients at the same time, this means messages sent on one will appear on the other, and messages sent to you will appear on both clients instead of the one with the highest priority. mod_carbons is included in prosody in version 0.10, but you still need to enable it:

modules_enabled = {
    ...
    "carbons";
}

sudo service prosody restart

Multi user chats (XEP-0045)

If you want to create "chat rooms" where you can chat with more than one person at a time, you need mod_muc. At the bottom of /etc/prosody/prosody.cfg.lua, you will find a section labelled "Components". You can add a multi-user chat as a subdomain in this section, which must be unique in the prosody configuration (not one of the domains defined as a VirtualHost, i.e. if you have a virtualhost called domain1.com, then you can have a chat room called conference.domain1.com or rooms.domain1.com or domain2.com, but if you used domain1.com in the definition it would fail). Here's an example using conference.example.com:

Component "conference.example.com" "muc"
        name = "The example.com chatrooms server"
        restrict_room_creation = admin

The name part of the configuration is optional, and the restrict_room_creation part means that only admins can add new rooms (this is one of the reasons why we made your user an admin in part 1). The prosody server must be restarted before this change takes effect:

sudo service prosody restart

If you want users on other servers to access these chat rooms, you need to add a DNS A record for whichever subdomain you chose (e.g. conference.example.com); if you only want to talk to users on your server in these rooms then you don't need to make any changes. The prosody docs for this module are here.

Client State Indication (XEP-0352)

Client State Indication is useful for mobile clients to reduce battery consumption and data usage. It enables the client to opt out of some types of message that aren't useful if the XMPP client is running but not open at the time (e.g. if another app is on top, or the phone is in your pocket). Client State Indication is provided in prosody by mod_csi. Two additional XEPs enable clients to opt out of specific types of content:

• mod_throttle_presence for filtering presence updates (away/busy/online etc.)
• mod_filter_chatstates to suppress messages about users typing

To enable all three, just load the modules. No extra configuration is necessary:

modules_enabled = {
    ...
    "csi" -- client state indication
    "throttle_presence" -- presence throttling in CSI
    "filter_chatstates" -- disable "X is typing" type messages
}

sudo service prosody restart

File and image transfer

File and image transfer is a bit of a pain in XMPP. The problem is that there are multiple methods of signalling to other clients that a transfer should start, and also multiple methods of transferring files. If you dive straight in and try to send files between different clients, you may get lucky and find that it works. You might also find that file transfer is extremely slow, or that it doesn't work at all. Understanding the many ways this can go wrong is important when you're trying to find a client + server combination that works!

Transfer Methods

There are two different ways of sending files between XMPP clients:

• SOCKS5 Bytestreams (XEP-0065) defines a method of sending data out of band (i.e. not within XML). Two clients can connect directly to each other if there are no firewalls, or they can use a proxy.
• In-Band Bytestreams (XEP-0047) defines a method for sending files inside XMPP, by breaking the file down into chunks and base64 encoding it.

SOCKS5 bytestreams are much faster than in band bytestreams, but in most situations a proxy is required to get around firewalls.

Signalling

The following methods can be used to negotiate data channels between XMPP clients and servers.

• Jingle (XEP-0116) is a method for enabling clients to negotiate media sessions (voice chat, video chat, file transfer) using XMPP for the signalling part, and then use a different channel to transfer data.
  • Jingle File Transfer (XEP-0234) is the XEP that deals with transferring files.
  • Jingle SOCKS5 Bytestreams Transport Method (XEP-0260) defines a Jingle transport method that sends data via a SOCKS5 Bytestream as defined in XEP-0065.
  • Jingle In-Band Bytestreams Transport Method (XEP-0261) defines a Jingle transport method that sends data via an in-band bytestream as defined in XEP-0047
• SI File Transfer (XEP-0096) is a method of setting up a type of Stream Initiation (XEP-0095)
• Out of Band Data (XEP-0066), which provides a "least common denominator" method for sharing files via URIs - it's really basic and generally not what we are looking for.

Modules

All of the signalling methods described above are features that are implemented in XMPP clients - they require no additional server modules. However, we need to add a proxy module to enable file transfers using SOCKS5 bytestreams. This is provided by mod_proxy65. Since it's a component that you want to enable for a specific server, add it as a specific component at the end of the file:

Component "proxy.yourdomain.com" "proxy65"
        proxy65_address = "yourdomain.com"
        proxy65_acl = { "yourdomain.com" }

Change the domains to match your domain name. The proxy65_address part is yourdomain.com not proxy.yourdomain.com so that the common name on your TLS certificate matches the domain name for secure connections. This assumes that the DNS A record for youdomain.com resolves to the IP address of your XMPP server. The access control list (proxy65_acl) is a list of users allowed to use the proxy. The list can contain domains (yourdomain.com), bare Jabber IDs like you@yourdomain.com or full Jabber IDs including a resource / client, e.g. you@yourdomain.com/work. If you leave that line out, anyone can use the proxy.

sudo service prosody restart

You will also need to set up port forwarding on your router so that port 5000 on your server is accessible from the internet.

Client Support

A lot of desktop clients support legacy signalling methods only, whereas mobile clients tend to have been written more recently and therefore support Jingle signalling methods - but not always! Clients that use the Telepathy framework (KDE Telepathy and the built-in SailfishOS client) are slow to add new features because the telepathy framework is quite complicated. Because of this, at the time of writing Telepathy based clients don't support Jingle file transfer. Some clients attempt Jingle signalling before falling back to SI file transfer. Similarly, most clients will attempt to send messages using SOCKS5 bytestreams and fall back to in-band file transfer as a last resort (it's much slower). Conversations only supports Jingle for signalling - the developer made a conscious decision not to implement SI file transfer as a fallback, to keep the codebase clean and encourage developers of desktop clients support the more modern Jingle signalling. If you want file transfer to work, you need both clients to support the same signalling method and the same transfer method. If file transfer is important to you, then I suggest you use Gajim on the desktop in combination with Conversations, because both support Jingle file transfer.

Stream Management (XEP-0198 Stream Management)

Stream management helps to prevent message loss when your connection to the XMPP server is intermittent. When it's enabled, both the client and server keep a list of recently sent messages, which are only cleared when the other side acknowledges that the messages have been received. If your client disconnects, it has a configurable amount of time left to connect and retrieve the messages before it's considered to be offline. Stream management is provided by mod_smacks.

modules_enabled = {
    ...
    "smacks" -- stream managment
}

sudo service prosody restart

User Blocking (XEP-0191 Blocking Command)

Blocking Command replaces more complicated legacy methods of blocking users (XEP-0016 Privacy Lists), and enables easy blocking and unblocking of specific Jabber IDs. In Prosody 0.10, the module is implemented as mod_blocklist:

modules_enabled = {
    ...
    "blocklist" -- blocking command
}

sudo service prosody restart

Sharing components between virtualhosts

In XMPP, clients can query their server to find out which services are supported. This is called "service discovery" and is handled by mod_disco, which is a module enabled by default so you don't have to enable it. If you add modules for a specific virtualhost, then they should be automatically discoverable on that virtualhost. However, if for example you have a component like a proxy or a multi-user chat room on one domain and you want to use it on your other domain, you need to make it discoverable with mod_disco. In the following example, the chat server for yourdomain.com is made discoverable on seconddomain.com:

----------- Virtual hosts -----------
VirtualHost "yourdomain.com"
        enabled = true
        ssl = {
                key = "/etc/prosody/certs/yourdomain.com.key";
                certificate = "/etc/prosody/certs/yourdomain.com.ca-bundle";
        }

VirtualHost "seconddomain.com"
        enabled = true
        ssl = {
                key = "/etc/prosody/certs/seconddomain.com.key";
                certificate = "/etc/prosody/certs/seconddomain.com.ca-bundle";
        }
        disco_items = {
                { "rooms.yourdomain.com", "The yourdomain.com chatrooms server" };
        }

------ Components ------
Component "rooms.yourdomain.com" "muc"
        name = "yourdomain.com chatrooms server"
        restrict_room_creation = admin
        modules_enabled = {
                "mam_muc",
        }