Configuring multi-server Crowdsec

A Crowdsec multi-server configuration has been on my to do list ever since I consolidated my proxy containers and hosted websites onto a single dedicated server. This left the applications running on my other server without Crowdsec coverage.

This guide simplifies the process of creating and running Crowdsec in a multi-server environment.

My motivation for this article is to correct the issues I encountered along the way where no single documentation page, guide, or video painted a complete picture on how to configure a Crowdsec multi-server environment.

The process of splitting my application hosting across multiple machines left my original server without a Crowdsec instance.

In order to retain Crowdsec’s ability to parse logs I ended up building a hasty solution using network shares to write logs to the hosting server running the Traefik proxy and Crowdsec stack. This configuration proved particularly unreliable, particularly around file permissions and remounting the share after restarts.

Prerequisites

This guide assumes that you already have a working single instance of Crowdsec as well as any bouncers.

While not mandatory, I recommend leveraging virtual networks when exposing services to the Internet. Using a flat network is risky.

Optional Database Migration

By default, Crowdsec uses a local SQLite instance and is more than adequate for smaller environments.

Migrating databases is something I will cover at a later date after I get a chance to test switching databases. I am undecided on whether I will create another MariaDB isntance or take the plunge into PostgreSQL, both are supported.

In the meantime here is the official Crowdsec database configuration documentation and list of supported databases.

Configuration

This is the high level configuration.

1. Spin up a second instance of Crowdsec.
2. Register the new secondary instance with the primary.
3. Validate the new instance.
4. Disable the local API on the secondary instance.

Crowdsec Agent / Secondary Instance

Perform these steps in the secondary agent.

1. Exec into the secondary Crowdsec container.
   1. docker exec -it CROWDSEC-SECONDARY sh
2. Run the registration command to point Crowdsec local API at the primary server
   1. cscli lapi register --machine "SECONDARY-INSTANCE-NAME" -u http://IP-ADDRESS-OF-PRIMARY-CS-INSTANCE:8080*
   2. Using the --machine flag assigns a name to the machine, making it easier to read than a random guid.

*This can be the IP address of the primary Crowdsec container if you are using virtual networks, or the container name if using a Docker bridge network.

Primary Crowdsec Instance

The next steps are performed in the primary Crowdsec container.

1. Exec into the primary Crowdsec container.
   1. docker exec -it CROWDSEC-SECONDARY sh
2. List any registered machines, note the new unverified instance.
   1. cscli machines list
3. Validate the recently added secondary agent.
   1. cscli machines validate "MACHINE-NAME" in this case I went with CS-SECONDARY.

Disabling the local API on the secondary agent(s)

Once the secondary Crowdsec agent has been registered and validated there is one step left to complete the multi-sever configuration. We need to disable the Crowdsec API running on the secondary agent, because we are now delegating the API functionality to the primary agent.

This last step is crucial and something that is easily missed, and not up to date in the official documentation, although it is present on the Docker Hub page. The documentation and video instead instructs you comment out the server block in the config.yaml file, which causes the container to fail to start.

1. Declare the DISABLE_LOCAL_API variable in your Docker configuration and set it to true.
   1. DISABLE_LOCAL_API=true

Conclusion

With the above configuration I am now able to run two separate instances of Crowdsec on two servers.

The primary agent runs alongside my Traefik proxy and Traefik bouncer. This has now been happily running for the last two months. It scans the log files local to that server using the WordPress and Nginx collections.

Meanwhile, the secondary Crowdsec agent on the other server parses the local log files using the Nextcloud and Emby collections and forwards the results to the primary agent.

I confirmed the setup is working by trying to brute force bad credentials while connected to a mobile network and then checking with cscli decisions list which returns any recent banned IP addresses.