Using multiple PostgreSQL databases in your Gitlab CI
How to configure gitlab-ci.yml to use multiple databases with your multi-db Rails setup
Multi database support for Rails was introduced back in January 2019, but I didn’t get a chance to try it out until recently.
Part of the reason was I was afraid to migrate existing apps to it, for fear it might break something in production. The other was that none of the applications I am involved with had such scaling needs.
Nonetheless, my wish finally came true last week when I found the excuse to use multiple databases.
Ever since I’ve been running my code against edge Rails (highly recommended, by the way), I’ve taken more initiative to try out newer Rails features. It also helps to be writing a lot of tests to make sure that nothing breaks. In fact, I’ve grown so confident of my set up that I sometimes forget to let tests finish running before I start working on something else. So you can guess my surprise when I started noticing none of my builds were getting pushed out to the production servers.
I use a custom Gitlab runner to run the tests, and lo-and-behold, everything started failing after I updated the database configuration (in the database.yml). In the old days, my method here would have broken the production servers, but luckily CI saved my butt (albeit, on a Saturday only).
My Gitlab CI wasn’t configured to run two databases, which my configuration now demanded. Previously, I set up my gitlab-ci.yml to create the database itself, but this was not working anymore.
So, I fixed it with something like this…
test:
before_script:
- apt-get update -qq && apt-get install -y -qq nodejs
- curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
- echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
- apt update
- apt install yarn
- gem install bundler --no-document
- bundle install --jobs $(nproc) "${FLAGS[@]}"
- echo "$RAILS_MASTER_KEY" > config/credentials/test.key
- yarn install --check-files
- bundle exec rake db:reset RAILS_ENV=test
- bundle exec rake db:test:prepare RAILS_ENV=test
- bundle exec rails webpacker:compile RAILS_ENV=test
services:
- postgres:12.1
- redis:latest
variables:
POSTGRES_USER: runner
POSTGRES_PASSWORD: ""
REDIS_URL: redis://redis:6379/0
PARALLEL_WORKERS: 2
What you see here is that gitlab-ci.yml instructs the runner to set up the PostgreSQL user, but does not create the database. We use Rails itself (with the correct credentials) to create the databases with the command bundle exec rake db:test:prepare.
Under database.yml, we have:
default: &default
adapter: postgresql
encoding: unicode
pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
development:
primary:
<<: *default
database: app_development
migrations_paths: db/migrate
secondary:
<<: *default
database: app_secondary_development
migrations_paths: db/secondary_migrate
test:
primary:
<<: *default
database: app_primary_test
username: runner
host: postgres
migrations_paths: db/migrate
secondary:
<<: *default
database: app_secondary_test
username: runner
host: postgres
migrations_paths: db/secondary_migrate
And that’s it. Now your CI should be able to handle having multiple databases for Rails.