Deployment

This guide serves as reference deployment for setting up a FEDn network consisting of:

  • One host/VM for the object storage and database services (MinIO, MongoDB)

  • One host/VM for the controller / reducer

  • One host/VM for the combiner

Note

In this guide we will deploy using the provived docker-compose templates. Please note that additional configurations would be needed for a production-grade network.

Prerequisites

Hosts / VMs

We assume that you have root access to 3 Ubuntu 20.04 LTS or 22.04 LTS Server hosts / VMs. We recommend at least 4 CPUs and 8GB RAM for the base services and the reducer, and 4 CPUs and 16BG RAM for the combiner host. Each host needs the following:

Networking

You will need to configure security groups / ingress settings for each host matching the port settings in the docker-compose templates. The reducer and clients need to be able to resolve the hostname for the combiners. In this example we show how this can be achieved if no external DNS resolution is available, by setting “extra host” in the Docker containers for the Reducer and client. Note that there are many other possible ways to achieve this, depending on your setup.

1. Deploy storage and database services (MinIO, MongoDB and MongoExpress)

First, deploy MinIO and Mongo services on one of the hosts. Edit the docker-compose.yaml file to change the default passwords and ports.

sudo docker-compose up -d minio mongo mongo-express

Remember to open ports on the host so that the API endpoints (the exported port in the ‘ports’ property for each of the services) can be reached.

Warning

The deployment of MinIO and MongoDB above is insecure. For a production network, please ensure production deployments of the base services.

2. Deploy the reducer

Copy the file “config/settings-reducer.yaml.template” to “config/settings-reducer.yaml”, then

  1. Edit ‘settings-reducer.yaml’ to provide the connection settings for MongoDB and Minio from Step 1.

  2. Copy ‘config/extra-hosts-reducer.yaml.template’ to ‘config/extra-hosts-reducer.yaml’ and edit it, adding a host:IP mapping for each combiner you plan to deploy.

Then start the reducer:

sudo docker-compose \
   -f docker-compose.yaml \
   -f config/reducer-settings.override.yaml \
   -f config/extra-hosts-reducer.yaml \
   up -d reducer

Note

the use of ‘extra-hosts-reducer.yaml’ is a way to add the host:IP mapping to /etc/hosts in the Docker container in docker-compose. It can be skipped if you handle DNS resolution in some other way.

3. Deploy combiners

Copy ‘config/settings-combiner.yaml.template’ to ‘config/settings-combiner.yaml’ and edit it to provide a name for the combiner (used as a unique identifier for the combiner in the FEDn network), a hostname (which is used by reducer and clients to connect to the combiner RPC server), and the port (default is 12080, make sure to allow access to this port in your security group/firewall settings). Also, provide the IP and port for the reducer under the ‘controller’ tag. Then deploy the combiner:

sudo docker-compose \
   -f docker-compose.yaml \
   -f config/combiner-settings.override.yaml \
   up -d combiner

Optional: Repeat this step for any number of additional combiner nodes. Make sure to provide an unique name for each combiner, and update extra_hosts for the reducer (you need to restart the reducer to do so).

Warning

Note that it is not possible to use the IP address as ‘host’. gRPC does not support certificates based on IP addresses.

4. Attach clients to the FEDn network

You can now choose an example, upload a compute package and an initial model, and attach clients.

Note

The clients will also need to be able to resolve each combiner node usign the ‘host’ argument in the combiner settings file. There is a template in ‘config/extra-hosts-client.yaml.template’ that can be modified for this purpose.