Using multiple PostgreSQL databases in your Gitlab CI

How to configure gitlab-ci.yml to use multiple databases with your multi-db Rails setup

Multi database support for Rails was introduced back in January 2019, but I didn’t get a chance to try it out until recently.

Part of the reason was I was afraid to migrate existing apps to it, for fear it might break something in production. The other was that none of the applications I am involved with had such scaling needs.

Nonetheless, my wish finally came true last week when I found the excuse to use multiple databases.

Ever since I’ve been running my code against edge Rails (highly recommended, by the way), I’ve taken more initiative to try out newer Rails features. It also helps to be writing a lot of tests to make sure that nothing breaks. In fact, I’ve grown so confident of my set up that I sometimes forget to let tests finish running before I start working on something else. So you can guess my surprise when I started noticing none of my builds were getting pushed out to the production servers.

I use a custom Gitlab runner to run the tests, and lo-and-behold, everything started failing after I updated the database configuration (in the database.yml). In the old days, my method here would have broken the production servers, but luckily CI saved my butt (albeit, on a Saturday only).

My Gitlab CI wasn’t configured to run two databases, which my configuration now demanded. Previously, I set up my gitlab-ci.yml to create the database itself, but this was not working anymore.

So, I fixed it with something like this…

test:
  before_script:
    - apt-get update -qq && apt-get install -y -qq nodejs
    - curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
    - echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
    - apt update
    - apt install yarn
    - gem install bundler --no-document
    - bundle install --jobs $(nproc)  "${FLAGS[@]}"
    - echo "$RAILS_MASTER_KEY" > config/credentials/test.key
    - yarn install --check-files
    - bundle exec rake db:reset RAILS_ENV=test
    - bundle exec rake db:test:prepare RAILS_ENV=test
    - bundle exec rails webpacker:compile RAILS_ENV=test

services:
  - postgres:12.1
  - redis:latest

variables:
  POSTGRES_USER: runner
  POSTGRES_PASSWORD: ""
  REDIS_URL: redis://redis:6379/0
  PARALLEL_WORKERS: 2

What you see here is that gitlab-ci.yml instructs the runner to set up the PostgreSQL user, but does not create the database. We use Rails itself (with the correct credentials) to create the databases with the command bundle exec rake db:test:prepare.

Under database.yml, we have:

default: &default
  adapter: postgresql
  encoding: unicode
  pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>

development:
  primary:
    <<: *default
    database: app_development
    migrations_paths: db/migrate
  secondary:
    <<: *default
    database: app_secondary_development
    migrations_paths: db/secondary_migrate

test:
  primary:
    <<: *default
    database: app_primary_test
    username: runner
    host: postgres
    migrations_paths: db/migrate
  secondary:
    <<: *default
    database: app_secondary_test
    username: runner
    host: postgres
    migrations_paths: db/secondary_migrate

And that’s it. Now your CI should be able to handle having multiple databases for Rails.