Setup Solid Queue/Cable/Cache in Rails 8 to share a single database
An alternative to Sidekiq and Redis
Rails 8 ships with Solid Queue, Solid Cache, and Solid Cable.
The following setup will configure the app to run the app's primary database, along with Solid queue / cache / cable, all in a single Postgresql database.
For most small apps, this setup should be fine. As apps grow, we can split them out to separate databases easily.
Note: As usual, I set up both production and staging environments.
Create config/environments/staging.rbif you haven't already.
Update database.yml
I set up all 3 databases (development, production, staging) exactly the same way.
production and staging each specify a url for the database, which points to <%= ENV["DATABASE_URL"] %>. that ENV variable must be set in the staging and production app on whichever hosting service we're using.
Replace projectname with the actual project name.
# PostgreSQL. Versions 9.3 and up are supported.
#
# Install the pg driver:
# gem install pg
# On macOS with Homebrew:
# gem install pg -- --with-pg-config=/usr/local/bin/pg_config
# On Windows:
# gem install pg
# Choose the win32 build.
# Install PostgreSQL and put its /bin directory on your path.
#
# Configure Using Gemfile
# gem "pg"
#
default: &default
adapter: postgresql
encoding: unicode
# For details on connection pooling, see Rails configuration guide
# https://guides.rubyonrails.org/configuring.html#database-pooling
pool: <%= ENV.fetch("RAILS_MAX_THREADS") { 5 } %>
development:
primary: &primary_development
<<: *default
database: projectname_development
cache:
<<: *primary_development
queue:
<<: *primary_development
cable:
<<: *primary_development
# The specified database role being used to connect to PostgreSQL.
# To create additional roles in PostgreSQL see `$ createuser --help`.
# When left blank, PostgreSQL will use the default role. This is
# the same name as the operating system user running Rails.
#username: projectname
# The password associated with the PostgreSQL role (username).
#password:
# Connect on a TCP socket. Omitted by default since the client uses a
# domain socket that doesn't need configuration. Windows does not have
# domain sockets, so uncomment these lines.
#host: localhost
# The TCP port the server listens on. Defaults to 5432.
# If your server runs on a different port number, change accordingly.
#port: 5432
# Schema search path. The server defaults to $user,public
#schema_search_path: myapp,sharedapp,public
# Minimum log levels, in increasing order:
# debug5, debug4, debug3, debug2, debug1,
# log, notice, warning, error, fatal, and panic
# Defaults to warning.
#min_messages: notice
# Warning: The database defined as "test" will be erased and
# re-generated from your development database when you run "rake".
# Do not set this db to the same as development or production.
test:
<<: *default
database: projectname_test
# As with config/credentials.yml, you never want to store sensitive information,
# like your database password, in your source code. If your source code is
# ever seen by anyone, they now have access to your database.
#
# Instead, provide the password or a full connection URL as an environment
# variable when you boot the app. For example:
#
# DATABASE_URL="postgres://myuser:mypass@localhost/somedatabase"
#
# If the connection URL is provided in the special DATABASE_URL environment
# variable, Rails will automatically merge its configuration values on top of
# the values provided in this file. Alternatively, you can specify a connection
# URL environment variable explicitly:
#
# production:
# url: <%= ENV["MY_APP_DATABASE_URL"] %>
#
# Read https://guides.rubyonrails.org/configuring.html#configuring-a-database
# for a full overview on how database connection configuration can be specified.
#
production:
primary: &primary_production
<<: *default
url: <%= ENV["DATABASE_URL"] %>
cache:
<<: *primary_production
queue:
<<: *primary_production
cable:
<<: *primary_production
staging:
primary: &primary_staging
<<: *default
url: <%= ENV["DATABASE_URL"] %>
cache:
<<: *primary_staging
queue:
<<: *primary_staging
cable:
<<: *primary_staging
Staging, Production, Development environment configs
The following are the config files I used on instrumental.dev. You can use these as a starting point and tweak as needed.
config/environments/staging.rb:
require "active_support/core_ext/integer/time"
Rails.application.configure do
# Settings specified here will take precedence over those in config/application.rb.
# Code is not reloaded between requests.
config.enable_reloading = false
# Eager load code on boot for better performance and memory savings (ignored by Rake tasks).
config.eager_load = true
# Full error reports are disabled.
config.consider_all_requests_local = false
# Turn on fragment caching in view templates.
config.action_controller.perform_caching = true
# Cache assets for far-future expiry since they are all digest stamped.
config.public_file_server.headers = { "cache-control" => "public, max-age=#{1.year.to_i}" }
# Enable serving of images, stylesheets, and JavaScripts from an asset server.
# config.asset_host = "http://assets.example.com"
# Store uploaded files on the local file system (see config/storage.yml for options).
config.active_storage.service = :local
# Assume all access to the app is happening through a SSL-terminating reverse proxy.
config.assume_ssl = true
# Force all access to the app over SSL, use Strict-Transport-Security, and use secure cookies.
config.force_ssl = true
# Skip http-to-https redirect for the default health check endpoint.
# config.ssl_options = { redirect: { exclude: ->(request) { request.path == "/up" } } }
# Log to STDOUT with the current request id as a default log tag.
config.log_tags = [ :request_id ]
config.logger = ActiveSupport::TaggedLogging.logger(STDOUT)
# Change to "debug" to log everything (including potentially personally-identifiable information!)
config.log_level = ENV.fetch("RAILS_LOG_LEVEL", "info")
# Prevent health checks from clogging up the logs.
config.silence_healthcheck_path = "/up"
# Don't log any deprecations.
config.active_support.report_deprecations = false
# Replace the default in-process memory cache store with a durable alternative.
config.cache_store = :solid_cache_store
# Replace the default in-process and non-durable queuing backend for Active Job.
config.active_job.queue_adapter = :solid_queue
config.solid_queue.connects_to = { database: { writing: :queue } }
# Ignore bad email addresses and do not raise email delivery errors.
# Set this to true and configure the email server for immediate delivery to raise delivery errors.
# config.action_mailer.raise_delivery_errors = false
# Set host to be used by links generated in mailer templates.
config.action_mailer.default_url_options = { host: "staging.instrumental.dev" }
# Specify outgoing SMTP server. Remember to add smtp/* credentials via rails credentials:edit.
# config.action_mailer.smtp_settings = {
# user_name: Rails.application.credentials.dig(:smtp, :user_name),
# password: Rails.application.credentials.dig(:smtp, :password),
# address: "smtp.example.com",
# port: 587,
# authentication: :plain
# }
# Enable locale fallbacks for I18n (makes lookups for any locale fall back to
# the I18n.default_locale when a translation cannot be found).
config.i18n.fallbacks = true
# Do not dump schema after migrations.
config.active_record.dump_schema_after_migration = false
# Only use :id for inspections in production.
config.active_record.attributes_for_inspect = [ :id ]
# Enable DNS rebinding protection and other `Host` header attacks.
# config.hosts = [
# "example.com", # Allow requests from example.com
# /.*\.example\.com/ # Allow requests from subdomains like `www.example.com`
# ]
#
# Skip DNS rebinding protection for the default health check endpoint.
# config.host_authorization = { exclude: ->(request) { request.path == "/up" } }
end
config/environments/production.rb:
require "active_support/core_ext/integer/time"
Rails.application.configure do
# Settings specified here will take precedence over those in config/application.rb.
# Code is not reloaded between requests.
config.enable_reloading = false
# Eager load code on boot for better performance and memory savings (ignored by Rake tasks).
config.eager_load = true
# Full error reports are disabled.
config.consider_all_requests_local = false
# Turn on fragment caching in view templates.
config.action_controller.perform_caching = true
# Cache assets for far-future expiry since they are all digest stamped.
config.public_file_server.headers = { "cache-control" => "public, max-age=#{1.year.to_i}" }
# Enable serving of images, stylesheets, and JavaScripts from an asset server.
# config.asset_host = "http://assets.example.com"
# Store uploaded files on the local file system (see config/storage.yml for options).
config.active_storage.service = :local
# Assume all access to the app is happening through a SSL-terminating reverse proxy.
config.assume_ssl = true
# Force all access to the app over SSL, use Strict-Transport-Security, and use secure cookies.
config.force_ssl = true
# Skip http-to-https redirect for the default health check endpoint.
# config.ssl_options = { redirect: { exclude: ->(request) { request.path == "/up" } } }
# Log to STDOUT with the current request id as a default log tag.
config.log_tags = [ :request_id ]
config.logger = ActiveSupport::TaggedLogging.logger(STDOUT)
# Change to "debug" to log everything (including potentially personally-identifiable information!)
config.log_level = ENV.fetch("RAILS_LOG_LEVEL", "info")
# Prevent health checks from clogging up the logs.
config.silence_healthcheck_path = "/up"
# Don't log any deprecations.
config.active_support.report_deprecations = false
# Replace the default in-process memory cache store with a durable alternative.
config.cache_store = :solid_cache_store
# Replace the default in-process and non-durable queuing backend for Active Job.
config.active_job.queue_adapter = :solid_queue
config.solid_queue.connects_to = { database: { writing: :queue } }
# Ignore bad email addresses and do not raise email delivery errors.
# Set this to true and configure the email server for immediate delivery to raise delivery errors.
# config.action_mailer.raise_delivery_errors = false
# Set host to be used by links generated in mailer templates.
config.action_mailer.default_url_options = { host: "instrumental.dev" }
# Specify outgoing SMTP server. Remember to add smtp/* credentials via rails credentials:edit.
# config.action_mailer.smtp_settings = {
# user_name: Rails.application.credentials.dig(:smtp, :user_name),
# password: Rails.application.credentials.dig(:smtp, :password),
# address: "smtp.example.com",
# port: 587,
# authentication: :plain
# }
# Enable locale fallbacks for I18n (makes lookups for any locale fall back to
# the I18n.default_locale when a translation cannot be found).
config.i18n.fallbacks = true
# Do not dump schema after migrations.
config.active_record.dump_schema_after_migration = false
# Only use :id for inspections in production.
config.active_record.attributes_for_inspect = [ :id ]
# Enable DNS rebinding protection and other `Host` header attacks.
# config.hosts = [
# "example.com", # Allow requests from example.com
# /.*\.example\.com/ # Allow requests from subdomains like `www.example.com`
# ]
#
# Skip DNS rebinding protection for the default health check endpoint.
# config.host_authorization = { exclude: ->(request) { request.path == "/up" } }
end
config/environments/development.rb:
require "active_support/core_ext/integer/time"
Rails.application.configure do
# Settings specified here will take precedence over those in config/application.rb.
# Make code changes take effect immediately without server restart.
config.enable_reloading = true
# Do not eager load code on boot.
config.eager_load = false
# Show full error reports.
config.consider_all_requests_local = true
# Enable server timing.
config.server_timing = true
# Enable/disable Action Controller caching. By default Action Controller caching is disabled.
# Run rails dev:cache to toggle Action Controller caching.
if Rails.root.join("tmp/caching-dev.txt").exist?
config.action_controller.perform_caching = true
config.action_controller.enable_fragment_cache_logging = true
config.public_file_server.headers = { "cache-control" => "public, max-age=#{2.days.to_i}" }
else
config.action_controller.perform_caching = false
end
# Change to :null_store to avoid any caching.
# config.cache_store = :memory_store
# Replace the default in-process memory cache store with a durable alternative.
config.cache_store = :solid_cache_store
# Replace the default in-process and non-durable queuing backend for Active Job.
config.active_job.queue_adapter = :solid_queue
config.solid_queue.connects_to = { database: { writing: :queue } }
# Store uploaded files on the local file system (see config/storage.yml for options).
config.active_storage.service = :local
# Don't care if the mailer can't send.
config.action_mailer.raise_delivery_errors = false
# Make template changes take effect immediately.
config.action_mailer.perform_caching = false
# Set localhost to be used by links generated in mailer templates.
config.action_mailer.default_url_options = { host: "localhost", port: 3000 }
# Print deprecation notices to the Rails logger.
config.active_support.deprecation = :log
# Raise an error on page load if there are pending migrations.
config.active_record.migration_error = :page_load
# Highlight code that triggered database queries in logs.
config.active_record.verbose_query_logs = true
# Append comments with runtime information tags to SQL queries in logs.
config.active_record.query_log_tags_enabled = true
# Highlight code that enqueued background job in logs.
config.active_job.verbose_enqueue_logs = true
# Raises error for missing translations.
# config.i18n.raise_on_missing_translations = true
# Annotate rendered view with file names.
config.action_view.annotate_rendered_view_with_filenames = true
# Uncomment if you wish to allow Action Cable access from any origin.
# config.action_cable.disable_request_forgery_protection = true
# Raise error when a before_action's only/except options reference missing actions.
config.action_controller.raise_on_missing_callback_actions = true
# Apply autocorrection by RuboCop to files generated by `bin/rails generate`.
# config.generators.apply_rubocop_autocorrect_after_generate!
end
Setup Puma.rb
Find the solid_queue plugin line and replace it with this:
This will ensure that solid queue runs inside the main app server.
# Run the Solid Queue supervisor inside of Puma for single-server deployments
plugin :solid_queue if ENV["SOLID_QUEUE_IN_PUMA"] || Rails.env.development?
config/cable.yml
# Async adapter only works within the same process, so for manually triggering cable updates from a console,
# and seeing results in the browser, you must do so from the web console (running inside the dev process),
# not a terminal started via bin/rails console! Add "console" to any action or any ERB template view
# to make the web console appear.
development:
adapter: solid_cable
connects_to:
database:
writing: cable
polling_interval: 0.1.seconds
message_retention: 1.day
test:
adapter: test
production:
adapter: solid_cable
connects_to:
database:
writing: cable
polling_interval: 0.1.seconds
message_retention: 1.day
staging:
adapter: solid_cable
connects_to:
database:
writing: cable
polling_interval: 0.1.seconds
message_retention: 1.day
Move Solid Queue, Solid Cache, and Solid Cable database tables into regular migrations for the main (sole) database
By default, Rails 8 assumes we will add Solid Queue / Cable / Cache to its own separate databases. Since we're running it on the primary database, we need to move the table migrations into regular migrations.
Generate 3 migrations:
rails g migration addSolidQueueTables
rails g migration addSolidCacheTables
rails g migration addSolidCableTables
In each of those migration files, inside of the change block, copy/paste all of the contents of the db/queue_schema.rb , db/cache_schema.rb , and db/cable_schema.rb respectively — not including the wrapping ActiveRecord::Schema[7.1].define(version: 1) do ...
Then run
rails db:migrate
Now all of the Solid tables should be in the primary database.
Install Mission Control
(as of this writing, Mission Control isn't yet part of Rails Core, but it's from the same people who built Solid Queue)
Mission Control gives you an admin web interface for viewing jobs info.
As of this writing, I found that you won't actually see any output after running this.
In fact, I think that with that puma plugin :solid_queue active, the bin/dev isn't even needed.
Create a background job
Here's an example of a simple job that updates the title of of a post.
jobs/update_post_title_job.rb
class UpdatePostTitleJob < ApplicationJob
queue_as :default
def perform(post_id)
Rails.logger.info "Starting UpdatePostTitleJob for Post ID: #{post_id}"
post = Post.find(post_id)
current_time = Time.current.strftime("%Y-%m-%d %H:%M:%S")
post.update(title: "#{post.title} - updated at #{current_time}")
post.save!
Rails.logger.info "Completed UpdatePostTitleJob for Post ID: #{post_id}"
end
end
And the post.rb for this:
class Post < ApplicationRecord
after_create :enqueue_update_title_job
private
def enqueue_update_title_job
UpdatePostTitleJob.perform_later(self.id)
end
end