has_many :codes
Tips and walkthroughs on web technologies and digital life
social

Setting up a Mastodon-Twitter crossposter

Published  

In the previous post I described how I set up a personal Mastodon instance to be in control of my destiny on the "fediverse". In this follow up, I will share how to set up a couple of "cross posters" so that you can have your tweets posted to Mastodon and your toots posted to Twitter, automatically.

Why would you want to do that? If you, like me, are not 100% ready to let go of Twitter yet and are waiting to see how the situation evolves, you might want to use both at the same time for now. In that case, it's nice to be able to cross post automatically so when you for example post a toot to Mastodon, your Twitter followers can also see the same content. Same thing for Twitter to Mastodon crossposting.

Now there are two options for crossposters that I am aware of, and both are also available as "public" services:

- https://crossposter.masto.donte.com.br/
- https://moa.party/

Both are very easy to configure and use. All you need to do is link your Mastodon and Twitter accounts and configure some options (such as whether you want to crosspost retweets and things like that) and that's it. From that moment on your content will be crossposted automatically.

The problem with these public services is that they are very slow. There's a huge delay between when you post content on one platform and when the same content is crossposted to the other platform. The reason is that lots of people are using these apps, so they are likely being heavily rate-limited by Twitter (as we'll see, you need to set up a Twitter application in order to interact with the Twitter platform in an automated way).

For this reason, and since I am already self hosting Mastodon anyway, I decided to also self host a cross poster so that my content is crossposted right away without any delays. Both the options I mentioned above are available as open source and therefore we can self host any of them. In this post I will show you how to self host either of them so you can choose which one you prefer.


First option: @[email protected]'s cross poster

This crossposter, whose source code is available on Github, is a Ruby on Rails app just like Mastodon itself. Like Mastodon, it's pretty easy to self host using Docker. The instructions here assume you have set up Mastodon as described in the previous post, so if you haven't you will need to adapt these steps accordingly.

First things first, edit mastodon/docker-compose.yml and add the following:

  crossposter-db:
    restart: always
    image: postgres:14-alpine
    container_name: "crossposter-db"
    healthcheck:
      test: pg_isready -U postgres
    environment:
      POSTGRES_HOST_AUTH_METHOD: trust
    volumes:
      - /home/vito/apps/mastodon/crossposter/postgres:/var/lib/postgresql/data

  crossposter-redis:
    restart: always
    image: redis:6.0-alpine
    container_name: "crossposter-redis"
    healthcheck:
      test: redis-cli ping
    volumes:
      - /home/vito/apps/mastodon/crossposter/redis:/data

  crossposter-web:
    restart: always
    build: https://github.com/renatolond/mastodon-twitter-poster.git#main
    image: mastodon-twitter-poster
    container_name: "crossposter-web"
    env_file:
      - /home/vito/compose/mastodon/crossposter.env
    environment:
      ALLOWED_DOMAIN: "botta.social"
      DB_HOST: crossposter-db
      REDIS_URL: "redis://crossposter-redis"
    depends_on:
      - crossposter-db
      - crossposter-redis

  crossposter-sidekiq:
    restart: always
    build: https://github.com/renatolond/mastodon-twitter-poster.git#main
    image: mastodon-twitter-poster
    container_name: "crossposter-sidekiq"
    env_file:
      - /home/vito/compose/mastodon/crossposter.env
    environment:
      ALLOWED_DOMAIN: "botta.social"
      REDIS_URL: "redis://crossposter-redis"
      DB_HOST: crossposter-db
    command: bundle exec sidekiq -c 5 -q default -q high
    healthcheck:
      test: ps aux | grep '[s]idekiq\ 6' || false
    depends_on:
      - crossposter-db
      - crossposter-redis

You may also share the Postgres and Redis containers between Mastodon and the crossposter, but here for simplicity I am using separate containers. Like Mastodon, the cross poster requires both a Postgres database and Redis, and consists of two processes: one web process that allows us to configure the "bridge" between Mastodon and Twitter, and a background worker that actually handles the crossposting using Sidekiq jobs.  Be sure to replace the username "vito" with yours in the YAML above. Also set the ALLOWED_DOMAIN env variable for the web container to the domain you use for your Mastodon instance, to prevent other people from using your cross poster (unless you are fine with that).

We also need to create the file /home/vito/compose/mastodon/crossposter.env with some environment variables:

LIBRATO_EMAIL=
LIBRATO_TOKEN=
TWITTER_CLIENT_ID=...
TWITTER_CLIENT_SECRET=...
SECRET_KEY_BASE=...
RAILS_ENV=production
RACK_ENV=production
RAILS_LOG_TO_STDOUT=enabled
# The name that's going to be displayed in the page, but also in the bottom of toots as the app name (optional)
CROSSPOSTER_APP_NAME=Mastodon Twitter Crossposter
# The domain where the app can be accessed
CROSSPOSTER_DOMAIN=https://crossposter.domain.com
# The repo, if it's a fork (optional)
CROSSPOSTER_REPO=https://github.com/repo-to/crossposter
# Optional: URL to a custom privacy policy
# CROSSPOSTER_PRIVACY_URL=https://example.com/privacy
# The stats page, if exists (optional)
CROSSPOSTER_STATS=https://grafana.example.com

# The fediverse account where announcements about this instance of the crossposter can be found
CROSSPOSTER_FEDI_ACCOUNT_ADDRESS=https://instance.example.com/@crossposter
[email protected]

# Twitter account of the admin of this instance, if exists (optional)
CROSSPOSTER_ADMIN_TWITTER=...

# Fediverse account that users can use to reach the admin in case of issues
CROSSPOSTER_ADMIN_FEDI_ADDRESS=https://your.mastodon/@adminuser
CROSSPOSTER_ADMIN_FEDI_AT[email protected]

# If you are using statsd to save metrics, you need the three following variables
STATSD_ENABLED=false
STATSD_HOST=127.0.0.1
STATSD_PORT=8125

# You should only enable one of the following lines at a time
ALLOWED_DOMAIN=your-mastodon-domain
BLOCKED_DOMAINS=evil.corp,bad.instance

# Database settings
# Postgres
DB_HOST=crossposter-db
DB_PORT=5432
DB_NAME=crossposter
DB_USER=postgres
DB_PASS=

# Redis only for Sidekiq
REDIS_URL=redis://crossposter-redis:6379/0

You can use any random string for SECRET_KEY_BASE but for the Twitter Client ID and the Client Secret you need to create a Twitter application. So head to https://developer.twitter.com and sign up with your Twitter account, then create an application of type web; you will be asked a few questions such as whether your app needs to pull people's data from Twitter and things like that. Answer no to those since you don't need to do those things. All you need is to be able to post tweets on behalf of the user, and nothing else. The only critical configuration options you'll need to specify are the callback URL in the format "https://crossposter.domain.com/users/auth/twitter/callback" and the cross poster website URL. You will need to wait for Twitter to approve your app before you can do anything with it (perhaps it's better you don't mention Mastodon in the description of the app). Be sure to give the app both read and write permissions. Once the app is ready, you will need the consumer key (which you will set as TWITTER_CLIENT_ID in the env vars file above) and the consumer secret (TWITTER_CLIENT_SECRET). 

Finally, to create the crossposter's containers run

compose/start.sh

Then run the following command to prepare the database:

docker compose run --rm crossposter-web bundle exec rake db:setup

Next, you'll need to configure the DNS for a subdomain like crossposter.yourdomain.com or something like that so that it points to the server. Then open the Nginx Proxy Manager control panel and add a new proxy host for that domain pointing to the crossposter-web container at port 3000. Be sure to request a new SSL certificate for it. 

Once ready, visit the crossposter domain in the browser and proceed with linking your Mastodon/Twitter accounts, as well as configuring the available options to your linking. That's it! Now tweets should be crossposted to Mastodon and toots to Twitter, automatically.

Second option: moa.party

Unlike the first crossposter, moa.party is a Python app rather than Rails. I am not familiar with Python so I must admit it took me a while to get it working but this is actually what I am using at the moment since I like the configuration options more.

For starters, clone the source code from https://gitlab.com/fedstoa/moa, then create a Dockerfile in the root with the following content:

FROM python:3.7

RUN apt-get update -qq \
  && DEBIAN_FRONTEND=noninteractive apt-get install -yq --no-install-recommends build-essential git-core \
  && rm -rf /var/lib/apt/lists/*

COPY requirements.txt requirements.txt

RUN pip3 install -r requirements.txt

COPY . .

CMD ["python3", "app.py"]

Also create the file worker.sh which will run the worker. The worker doesn't seem to be a persistent process; instead, it checks whether there are bridges to process and when it's done crossposting it exits. For this reason we are using a simple loop to repeat this every 30 seconds:

#!/bin/bash

while true; do
  python3 -m moa.worker
  sleep 30
done

Make the script executable:

chmod +x worker.sh

Next, build the image:

docker build --platform linux/amd64 -t moa .

If you are building directly on the server then you don't really need to push the image to a registry so you can name/tag it as you wish.

Next, create the compose/mastodon/moa/config.py file as follows:

from defaults import DefaultConfig

class DevelopmentConfig(DefaultConfig):
    DEBUG = False
    DEVELOPMENT = True
    TESTING = False
    CSRF_ENABLED = True
    # secret key for flask sessions http://flask.pocoo.org/docs/1.0/quickstart/#sessions
    SECRET_KEY = '...'
    SQLALCHEMY_TRACK_MODIFICATIONS = False
    TWITTER_CONSUMER_KEY = '...'
    TWITTER_CONSUMER_SECRET = '...'
    INSTAGRAM_CLIENT_ID = ''
    INSTAGRAM_SECRET = ''
    # define in config.py
    # SQLALCHEMY_DATABASE_URI = 'mysql+pymysql://moa:[email protected]/moa'
    SQLALCHEMY_DATABASE_URI = 'mysql+pymysql://root:[email protected]/moa?charset=utf8mb4'
    SEND = False

    # This option prevents Twitter replies and mentions from occuring when a toot contains @[email protected] This
    # behavior is against Twitter's rules.
    SANITIZE_TWITTER_HANDLES = True

    SEND_DEFERRED_EMAIL = False
    SEND_DEFER_FAILED_EMAIL = False
    MAINTENANCE_MODE = False

    STATS_POSTER_BASE_URL = None
    STATS_POSTER_ACCESS_TOKEN = None

    TRUST_PROXY_HEADERS = False
    WORKER_JOBS = 1
    SERVER_NAME = 'crossposter.domain.com'
    PREFERRED_URL_SCHEME = 'https'

You already know what the Twitter consumer key and secret are. Choose a random string for the SECRET_KEY, AND IN SQLALCHEMY_DATABASE_URI write the correct password for MariaDB as per the previous post.

Open a MySQL shell:

mysql -uroot -p<mariadb password> -h 127.0.0.1

And create the database for the crossposter:

CREATE DATABASE moa;

Next, add the following to the Mastodon docker-compose.yml file:

  moa-web:
    restart: always
    image: moa
    container_name: "moa-web"
    environment:
      MOA_CONFIG: config.DevelopmentConfig
      FLASK_DEBUG: "0"
    volumes:
      - /home/vito/apps/mastodon/moa:/data
      - /home/vito/compose/mastodon/moa/config.py:/config.py

  moa-worker:
    restart: always
    image: moa
    container_name: "moa-worker"
    environment:
      MOA_CONFIG: DevelopmentConfig
    command: "/worker.sh"
    volumes:
      - /home/vito/apps/mastodon/moa:/data
      - /home/vito/compose/mastodon/moa/config.py:/config.py

As usual replace the username etc.

Start the containers:

compose/start.sh

And run the following command to prepare the database:

docker run --rm -it -e MOA_CONFIG=DevelopmentConfig -v /home/vito/compose/mastodon/moa/config.py:/config.py --network apps vitobotta/moa-crossposter:v8 python3 -m moa.models

Then create a proxy host in Nginx Proxy Manager using the moa-web container and the port 5000 this time. Also set the callback URL in the Twitter app to https://crossposter.domain.com/twitter_oauthorized.

Open the crossposter domain in the browser, connect the Mastodon and Twitter accounts, and configure the crossposting. Done.


Wrapping up

If you have set up Mastodon as per the previous post, the instructions above should be straightforward, but you don't need to change them much otherwise. Let me know in the comments if you run into any issues and I will try to help.

Previous: Setting up a personal Mastodon instance
Next: Storing Grafana Loki log data in a Google Cloud Storage bucket

Comments

© Vito Botta