Fast UIs need to feel like a lot is happening, even if that’s in the background and out of site of your user. On a mobile app, usually that means moving network calls outside the “main thread” and into a background thread that calls back. On websites, it means making every request back to the server as short as possible, and then waiting for some update back from the server. This could be accomplished by polling the server, but with ActionCable and Turbo Streams, you can let the server do whatever background work is needed and have it push those updates to you.
One example is processing CSV files from your users. Sometimes APIs are great for getting data into your app, and you can’t beat comma separated values for updating records. The general idea is updating several records based off rows in a spreadsheet. This can be broken down into two steps: upload the file, and process the file.
Uploading the file can be done with JavaScript and a loading bar, so we know when the file is done uploading, and how much more progress there is. Processing the file happens on the backend, and that’s where getting visibility into the process can be tough. Do we create a database record that we use to keep track of the progress? Does that record get purged after the upload and processing are complete? Those are business problems that can be different depending on the situation. But given that the server that renders a webpage might be different from the server that processes the file, putting the file into a cloud bucket makes the most sense, at least until we’re done with the processing.
On the server side, using web sockets to communicate progress back to the front end can provide a sense of progress, and relief that something is actually happening, whether it’s 10 or 10,000 rows in the spreadsheet.
You can probably come up with other processes that need to display background progress, such sending out batches of notifications, but let’s work through processing books like in our previous Book example.
Uploading a CSV file
We can use ActiveStorage to help get the file into our system, and to help keep track of progress throughout the upload and processing. Each Blob in ActiveStorage has a database record, and can be purged once the whole process is complete. ActiveStorage also has a mechanism to put the file directly into the cloud or file storage we’re using, and it emits off progress events that we can use to show a loading indicator on the page. Once the file is uploaded, then we send the key back to the server and start processing the file.
Start with a clean app:
$ rails new BookImports -c tailwind
$ cd BookImports
$ rails g scaffold Book title:string author:string publisher:string category:string isbn:string dewey_decimal_number:string binding:integer
$ bin/rails active_storage:install
$ rails db:migrate
We’re going to make binding an enum since there are only a few options, so change models/book.rb to this:
class Book < ApplicationRecord
enum binding: [:hardcover, :paperback]
end
To test our system with plenty of books, We will use the Faker gem to create about 100 fake books in a CSV file that we can upload.
In our Gemfile:
gem 'faker'
Then run
$ bundle install
Then we’ll create the books in our db/seeds.rb file.
require 'csv'
path = 'public/books.csv'
CSV.open(path, 'w', headers: ['title', 'author', 'publisher', 'category', 'isbn', 'dewey_decimal_number', 'binding'], write_headers: true) do |csv|
100.times do |i|
csv << [Faker::Book.title, Faker::Book.author, Faker::Book.publisher, Faker::Book.genre, "#{Faker::Number.number(digits: 3)}-#{Faker::Number.number(digits: 1)}-#{Faker::Number.number(digits: 2)}-#{Faker::Number.number(digits: 6)}-#{Faker::Number.number(digits: 1)}", "#{Faker::Number.number(digits: 3)}.#{Faker::Number.number(digits: 3)}", Faker::Number.between(from: 0, to: 1)]
end
end
Working on the import pages, generate the controller:
$ rails g controller Import show create
Change the route for create into a post in routes.rb:
post 'import/create'
In the form, we’re going to set it as multipart, so we can upload a file, but first, let’s create the Stimulus controller that will show the upload progress.
$ rails g stimulus uploads
Add the ActiveStorage direct upload library with your current JavaScript management tool, for example with importmap:
$ ./bin/importmap pin @rails/activestorage
Add ActiveStorage to your javascript/application.js:
import * as ActiveStorage from "@rails/activestorage";
ActiveStorage.start();
The uploads_controller.js is going to listen for two events: direct-upload:initialize@window which will trigger showing the progress indicator, and direct-upload:progress@window, which will give us the current upload progress. The controller has a wrapper around the progress indicator and the progress indicator itself, which are updated by these events.
import { Controller } from "@hotwired/stimulus"
// Connects to data-controller="uploads"
export default class extends Controller {
static targets = ["progress", "progressWrapper"];
initializeDirectUpload() {
this.progressWrapperTarget.classList.remove("hidden");
}
progressDirectUpload(event) {
const { id, progress } = event.detail;
this.progressTarget.style.width = `${progress}%`;
}
}
The form, views/import/show.html.erb, is as follows:
<div class="max-w-3xl mx-auto">
<h1 class="font-bold text-4xl">Import Books</h1>
<%= form_with url: import_create_path, multipart: true do %>
<div class="divide-y divide-gray-200 space-y-5">
<div class="mt-5 space-y-5">
<div class="grid grid-cols-3 gap-4 items-start border-t border-gray-200 pt-5">
<label class="block text-sm font-medium text-gray-700 mt-px pt-2">Data File (CSV):</label>
<div class="mt-0 col-span-2"
data-controller='uploads'
data-action='direct-upload:initialize@window->uploads#initializeDirectUpload direct-upload:progress@window->uploads#progressDirectUpload' >
<div class="max-w-lg flex justify-center px-6 pt-5 pb-6 border-2 border-gray-300 border-dashed rounded-md">
<div class="space-y-1 text-center">
<div class="w-full flex text-sm text-gray-600">
<label for="file" class="relative cursor-pointer bg-white rounded-md font-medium text-blue-700 hover:text-blue-600 focus-within:outline-none focus-within:ring-2 focus-within:ring-offset-2 focus-within:ring-blue-500">
<%= file_field_tag :file, class: 'input', direct_upload: true %>
</label>
</div>
<div class="w-full pt-1 hidden"
data-uploads-target="progressWrapper">
<span>Uploading file</span>
<div class="overflow-hidden w-60 h-4 text-xs flex rounded bg-blue-200">
<div data-uploads-target="progress"
style="width: 0%"
class=" shadow-none flex flex-col text-center whitespace-nowrap text-white justify-center bg-blue-800" ></div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="pt-5">
<div class="flex justify-end">
<%= submit_tag "Upload File", class: "inline-flex justify-center py-2 px-4 border border-transparent shadow-sm text-sm font-medium rounded-md text-white bg-blue-700 hover:bg-blue-800 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-red-500" %>
</div>
</div>
</div>
</div>
<% end %>
</div>
At this point, when you upload the file, you get a nice progress indicator, but the controller doesn’t respond appropriately, so nothing happens on the page. We need to handle the uploaded file. Let’s start by creating a background job that will process the CSV.
$ bin/rails generate job ImportBooks
The create action on the import controller will queue the import job by passing in the blob, and then pass back the HTML that will display the progress indicator, and create the ActionCable channel that will listen for progress updates. Let’s wrap the form from the show.html.erb page with a turbo_frame_tag so that the response can replace that part of the response.
<div>
<h1 class="font-bold text-4xl">Import Books</h1>
<%= turbo_frame_tag "import" do %>
<%= form_with url: import_create_path, multipart: true do %>
<!-- the rest of the form is ommitted for brevity -->
<% end %>
<% end %>
</div>
The create template should be renamed to create.turbo_stream.erb and will have:
<%= turbo_stream.replace "import" do %>
<%= turbo_stream_from @blob.key %>
<div class="divide-y divide-gray-200 space-y-5">
<div class="mt-5 space-y-5">
<div class="grid grid-cols-3 gap-4 items-start border-t border-gray-200 pt-5">
<label class="block text-sm font-medium text-gray-700 mt-px pt-2">Data File (CSV):</label>
<div class="mt-0 col-span-2"
data-controller='uploads'
data-action='direct-upload:initialize@window->uploads#initializeDirectUpload direct-upload:progress@window->uploads#progressDirectUpload' >
<div class="max-w-lg flex justify-center px-6 pt-5 pb-6 border-2 border-gray-300 border-dashed rounded-md">
<div class="space-y-1 text-center">
<div class="w-full flex text-sm text-gray-600">
<label for="file" class="relative cursor-pointer bg-white rounded-md font-medium text-blue-700 hover:text-blue-600 focus-within:outline-none focus-within:ring-2 focus-within:ring-offset-2 focus-within:ring-blue-500">
<%= @blob.filename %>
</label>
</div>
<div class="w-full pt-1">
<span>Processing file</span>
<div class="overflow-hidden w-60 h-4 text-xs flex rounded bg-blue-200">
<%= render partial: "blob_progress", locals: { blob: @blob, progress: 0 } %>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="pt-5">
<div class="flex justify-end">
<%= link_to "View Results", books_path, class: "inline-flex justify-center py-2 px-4 border border-transparent shadow-sm text-sm font-medium rounded-md text-white bg-blue-700 hover:bg-blue-800 focus:outline-none focus:ring-2 focus:ring-offset-2 focus:ring-red-500" %>
</div>
</div>
</div>
</div>
<% end %>
We’re removing the form, but keeping the layout similar to what’s replaced. It uses the turbo_stream_from tag to listen on the blob’s key. This gets around the fact that ActiveStorage::Blob has a different result for the identify method that the TurboStream class uses to create the channel.
Finally, the ImportBooksJob is queued in the create action:
class ImportController < ApplicationController
def show
end
def create
@blob = ActiveStorage::Blob.find_signed params[:file]
ImportBooksJob.perform_later @blob
respond_to do |format|
format.turbo_stream
end
end
end
Processing a file
In this sample, we’re using a CSV and the processing will be in the background job ImportBooks. The job will download the file from file storage, use Ruby’s CSV library to read each row, and then update or insert records depending on what the file contains. One technique I’ve used is to farm out each record into its job, so that if any errors occur, they don’t stop the entire import, but We’ll keep this simple for now. The progress will be reported back to the front end through the TurboStream connection.
Let’s create a little template that we’ll use to render the progress, app/views/application/_blob_progress.html.erb:
<div id="<%= dom_id blob %>" class="rounded animate-pulse shadow-none flex flex-col text-center whitespace-nowrap text-white justify-center bg-blue-800" style="width: <%= progress %>%" ></div>
This takes in the blob and the amount of progress as local variables.
The ImportBooksJob is passed the blob, downloads the file, and parses the CSV. After each iteration, it sends the progress to the front end, and we get a nice loading bar on the page.
require 'csv'
class ImportBooksJob < ApplicationJob
queue_as :default
def perform(blob)
blob.open do |file|
csv = CSV.read(file, headers: true)
row_count = CSV.read(file, headers: true).size
csv.each_with_index do |row, progress|
Turbo::StreamsChannel.broadcast_replace_to blob.key,
target: blob, partial: "blob_progress",
locals: { blob: blob, progress: ((progress.to_f / row_count) * 100) }
end
end
end
end
The Turbo::StreamsChannel can broadcast to the front end. The first parameter, blob.key which matches what we are listening for on the page. The target matches the DOM id on the page, which is the progress indicator, and the partial and locals named parameters are considered the rendering block. This all gets wrapped into a template, and replaced on the front page.
Passive yet Interactive
The user wanted to load up a lot of records into the app. We provided a way to use a CSV to upload all the records, so they wouldn’t have to add each one manually. They get a progress bar for the upload, which is delightful if we have a big file, but it still feels luxurious for our website to tell us what’s going on. Once the files been uploaded, a new progress indicator shows how the record import is going. The whole action by the user was a few clicks, but they feel in control because they know what’s going on the whole time. This is the kind of interactivity uses crave in a web app, and a lot of it comes for free in Rails.
2 comments on “Displaying Progress in a Long Running Background Job using HOTWire”
If you have an attachment which is referenced in a view that’s broadcasted from the model. In your view you use the url to display the attachment(video), how can you make sure that rails has the necessary model information?
Sorry, I don’t quite follow what you’re asking.
I’ve used this technique in my apps, and I pass along extra information to the job processing, so the correct work gets done, such as processing a zip of pictures or importing models into the database. Does that answer what you asked?