As we continue in building out the HOTWire Hacker News Progressive Web App, it’s time to look at loading the comments. This is an interesting data problem, because each item has comments, and each comment has comments, and you can go recursively until you run out. The original HNWPA pushed all the comment loading into a background worker, and then used ActionCable to send the results back to the front end. This was okay, but there was a race condition where the background job might finish loading all the comments before the WebSocket connection was running. The fix was to send the load comments command over a web socket, and then wait for the response. If there were only a few comments, it felt pretty quick. But when it was a popular item, with a large discussion, it could take a while.

I think the comment threads are a good candidate to experiment with recursive lazy loading. First, when you visit the comment page for an item, it displays the item’s information, and then it loads the comments lazily. This means going to the Hacker News API to load each individual comment. Once the comment is displayed, each of its sub comments are lazily loaded. The repeats until there are no more descendant comments.

This tutorial introduces a Comment model. It’s not very different from the Item model in terms of the data it stores, but it’s used differently, so I decided to break it into a separate table. This is different from the original HNPWA app, which used items throughout, and made caching partials interesting.

Creating the Comment model

Here is the comment model generation:

rails g model Comment by:string hn_id:bigint:index parent_id:bigint:index text:text dead:boolean time:datetime location:integer

It uses the parent’s hn_id and it’s location to determine where it falls in order on the page. It has a populate helper method that pulls out the relevant data from the JSON response. It has_one parent through the parent_id, and it has_many kids through those sub comment’s parent_id.

Here is the model, app/models/comment.rb:

class Comment < ApplicationRecord
  has_one :hn_parent, class_name: 'Comment', primary_key: 'parent_id', foreign_key: 'hn_id'
  has_many :kids, class_name: "Comment", primary_key: 'hn_id', foreign_key: 'parent_id'

  def populate(json) 
    if json.nil?
      return
    end
    self.hn_id = json['id'] if json['id']
    self.by = json['by'] if json['by']
    self.parent_id = json['parent'] if json['parent']
    self.text = json['text'] if json['text']
    self.dead = json['dead'] if json['dead']
    self.time = DateTime.strptime("#{json['time']}",'%s') if json['time']
  end
end

The Item class now gets a has_many relationship based on the it’s children’s parent_id. Here is the first three lines of app/models/item.rb:

class Item < ApplicationRecord
  has_one :top_item
  has_many :kids, class_name: "Comment", primary_key: 'hn_id', foreign_key: 'parent_id'

Routes

The routes.rb file will be updated to add Comment as a sub resource of Item:

  resources :items do
    resources :comments
  end

Comments Controller

The item/show.html.erb will lazy load the comments of an item at comments#index. Then, any child comments will be directly loaded at comments#show. Each of these methods is similar, loading the requested resource, either an Item or a Comment, and then loading any descendant comments. There is a helper method called load_kids which attempts to load in any children comments, should they exist. Since there are many HTTP requests to the same server, this controller uses the HTTP.persistent option to keep up a connection to the server while the request is progressing.

class CommentsController < ApplicationController 

  def index
    @item = Item.find_by_hn_id params[:item_id]

    http = HTTP.persistent "https://hacker-news.firebaseio.com"
    item_json = JSON.parse http.get("/v0/item/#{@item.hn_id}.json").to_s

    if item_json.nil?
      return
    end
    @item.populate(item_json)
    @item.save
    load_kids(http, @item.hn_id, item_json)
  end

  def show 
    http = HTTP.persistent "https://hacker-news.firebaseio.com"
    @item = Item.find_by_hn_id params[:item_id]
    @comment = Comment.find_by_hn_id params[:id]
    item_json = JSON.parse http.get("/v0/item/#{@comment.hn_id}.json").to_s
    if item_json.nil?
      return
    end
    @comment.populate(item_json)
    @comment.save
    load_kids(http, @comment.hn_id, item_json)
  end

  private 

  def load_kids(http, parent_id, item_json)
    if item_json and item_json.has_key? 'kids'
      item_json['kids'].each_with_index do |kid_hn_id, kid_location|

        kid_json = JSON.parse http.get("/v0/item/#{kid_hn_id}.json").to_s
        if kid_json.nil?
          next
        end

        kid = Comment.where(hn_id: kid_hn_id).first_or_create
        kid.location = kid_location
        kid.parent_id = parent_id
        kid.populate(kid_json)
        kid.save
      end
    end
  end
end

Views

The app/views/items/show.html is going to use turbo-frame to replace the comments:

<header class="bg-red-600 py-1 px-3 mb-11 text-white flex">
  <p class="text-2xl flex-grow">
    <%= link_to "Top Stories", top_path %></p>
</header>
<%= render @item %>
<%= turbo_frame_tag "comments", src: item_comments_path(@item.hn_id) %>

The comments page will use a turbo-frame to tell turbo where the comments are on the HTML page. However, if this page is loaded outside of the items#index frame, it will show the Item object. Here is app/view/comments/index.html.erb:

<header class="bg-red-600 py-1 px-3 text-white flex">
  <p class="text-2xl flex-grow">
    <%= link_to "Top Stories", top_path %></p>
</header>
<%= render @item %>
<%= turbo_frame_tag "comments" do %> 
  <%= render partial: 'comments', locals: { item: @item, comments: @item.kids.order(:location) } %>
<% end %>

The difference between these two pages comes down to the src directive on items/show.html.erb and actually rendering the comments in comments/index.html.erb.

The individual comment page is simpler. It renders the comment, inside a turbo-frame that has the id of the comment. Here is app/views/comments/show.html.erb:

<%= turbo_frame_tag dom_id(@comment) do %>
  <%= render @comment %>
<% end %>

The app/views/comments/_comments.html.erb partial loops through the list of comments, and sets up a turbo-frame with a src tag that will call out to the comment.

<div class="flex flex-col">
  <% comments.each do |comment| %>
    <%= turbo_frame_tag dom_id(comment), src: item_comment_path(item.hn_id, comment.hn_id) %>
  <% end %>
</div>

Finally, the app/views/comments/_comment.html.erb will display the text, the author username, the published date, and any children in a sub-rendering of the _comments.html.erb partial.

<% unless comment.dead %>
  <div class="w-full">
    <div class="border-l border-b pl-4 mb-4">
      <%= sanitize comment.text %>
      <footer class="border-t flex justify-between">
        <p class="py-1 px-3">
          <a href="/user/<%= comment.by %>"><strong><%= comment.by %></strong></a>
        </p>
        <p class="py-1 px-3">
          <%= comment.time.strftime('%c') %>
        </p>
      </footer>
    </div>
    <div class="w-11/12 float-right">
      <%= render partial: 'comments', locals: { item: @item, comments: comment.kids.order(:location) } %>
    </div>
  </div>
<% end %>

HHNPWA vs. HNPWA

This is a bigger improvement over the previous comment loading process in the HNPWA. The previous implementation used a fragile WebSocket connection, and sometimes the comments wouldn’t load. The newer version uses fetch() when the page is loaded, so there won’t be a delay such as when the WebSocket connection was setup. Another advantage is that the comments are able to load piecemeal. The WebSocket method was only able to load the all the comments at once. It had a progress indicator, but I think loading the comments sooner rather than later feels faster and more interactive.

Next Steps

Caching optimizations are the obvious next steps. Stay tuned for tutorial #6!

Subscribe for updates as this project takes shape. You can see all the code at Github: https://github.com/OnRailsBlog/hhnpwa.

Previous Tutorials

• HOTWire HNPWA #1: Setting up for Top Stories
• Turbo Streaming Top Items – HOTWire HNPWA #2
• Paginating Top Items – HOTWire HNPWA #3
• Russian Doll Caching – Building HOTWire HNPWA #4