Rails 4 Application Development HOTSHOT (2014)

Chapter 9. Video Streaming Website using Rails and HTML5

Video as a medium is quite appealing to a lot of users. It is a very effective way of communication, and the effect can be very long lasting. YouTube (http://www.youtube.com), Vimeo (http://www.vimeo.com), Dailymotion (http://www.dailymotion.com), and Khan Academy (http://www.khanacademy.org) are some of the most popular sites where a variety of content exists. Advertising, raising awareness, organizing campaigns, distributing films, and providing education are some of the most common uses of these. This has increased the accessibility of the content and allowed content creators of various languages to reach a very wide and diverse set of audiences.

Mission briefing

This project is a video-streaming website where a user uploads the video and the video is encoded to a HTML5 friendly format. We will also take screenshots of the video post their upload so that we can make thumbnails out of it. We will work on caching and performance improvement with videos. We will also take a look at process queues in Rails.

Why is it awesome?

A lot of ideas have been tried around video. With the advent of HTML5, the video standards are becoming much more flexible and device friendly. HTML5 reduces external dependencies and plugins in order to display and run videos. This will make the video and audio protocols more standardized and open. We will use some standards that work seamlessly with HTML5 video and make sure it works on different devices. Once this is in place, we will cache the video and text. We will use Russian Doll caching, a technique introduced in Rails 3.2 but carried forward in Rails 4. We will also see queues in Rails. We will allow our application to simultaneously process videos as jobs. The final project screen with a list of videos will appear as shown in the following screenshot:

Why is it awesome?

At the end of this project, we will have a basic video-streaming web application.

Your Hotshot objectives

While building this application, we will have to go through the following tasks:

·        Uploading the video

·        Encoding the video

·        Displaying the video panel and playing the video

·        Caching the content – text and video

·        Queuing the job

Mission checklist

We need the following installed on the system and also need to sign up for the API keys before we start with our mission:

·        Ruby 1.9.3 / Ruby 2.0.0

·        Rails 4.0+

·        MySQL

·        FFmpeg

·        Devise

·        Git

·        Redis

·        Sidekiq

·        jQuery

·        Video.js

·        Bootstrap 3.0

Uploading the video

We will begin our project with video-uploading methods. We have already seen file uploading with the carrierwave gem in our previous projects (Project 3Creating an Online Social Pinboard). In this project, we will take it one step forward by uploading videos.

We will also add the friendly_id gem to our application in order to create slugs:


gem 'carrierwave', :github => "jnicklas/carrierwave"

gem 'friendly_id', '5.0.3'

gem 'anjlab-bootstrap-rails', :require => 'bootstrap-rails',

                              :github => 'anjlab/bootstrap-rails',

                              :branch => '3.0.0'

Only Version 5.0.3 friendly_id is compatible with Rails 4.1. Also, at this step, make sure you have devise installed and have generated a user model to handle user authentication.

Engage thrusters

We will start by installing Rails API and generating our skeleton application:

1.    We will first generate a video model and controller. Be sure to write tests before that, as follows:

2.  mutube$ rails g scaffold video title:string description:string

3.        invoke  active_record

4.        create    db/migrate/20140105125840_create_videos.rb

5.        create    app/models/video.rb

6.        invoke    test_unit

7.        create      test/unit/video_test.rb

8.        create      test/fixtures/videos.yml

9.        invoke  resource_route

10.       route    resources :videos

11.      invoke  scaffold_controller

12.      create    app/controllers/videos_controller.rb

13.      invoke    erb

14.      create      app/views/videos

15.      create      app/views/videos/index.html.erb

16.      create      app/views/videos/edit.html.erb

17.      create      app/views/videos/show.html.erb

18.      create      app/views/videos/new.html.erb

19.      create      app/views/videos/_form.html.erb

20.      invoke    test_unit

21.      create      test/functional/videos_controller_test.rb

22.      invoke    helper

23.      create      app/helpers/videos_helper.rb

24.      invoke      test_unit

25.      create        test/unit/helpers/videos_helper_test.rb

26.      invoke  assets

27.      invoke    coffee

28.      create      app/assets/javascripts/videos.js.coffee

29.      invoke    scss

30.      create      app/assets/stylesheets/videos.css.scss

31.      invoke  scss

32.   identical    app/assets/stylesheets/scaffolds.css.scss

33.          Once we have the skeleton for the video, we will add the carrierwave gem to Gemfile and run bundle install:


35.gem 'carrierwave', :github => "jnicklas/carrierwave"

36.          We will then generate the video uploader using the carrierwave generator:

37. mutube$rails g uploader Video

38.      create  app/uploaders/video_uploader.rb

39.          Mount the video uploader on the video model:


41.class Video < ActiveRecord::Base

42.  mount_uploader :video, VideoUploader

43.  extend FriendlyId

44.  friendly_id :title, use: :slugged


45.          Now we will add a column for video file parameters to our videos table:

46.class AddVideoToVideos < ActiveRecord::Migration

47.  def change

48.    add_column :videos, :media, :string

49.  end


50.          We also need to pass the parameters for the video as a whitelist in our controller and add the friendly_id association to our set_video action:


52. private

53.    # Use callbacks to share common setup or constraints between actions.

54.    def set_video

55.      @video = Video.friendly.find(params[:id])

56.    end

57.    # Never trust parameters from the scary internet, only allow the white list through.

58.    def video_params

59.      params.require(:video).permit(:title, :description, :media, :media_cache)


60.          We will edit the form to add the upload field for the video:


62.<%= form_for(@video, :html => {:multipart => true}) do |f| %>

63.  <% if @video.errors.any? %>

64.    <div id="error_explanation">

65.      <h2><%= pluralize(@video.errors.count, "error") %> prohibited this video from being saved:</h2>

66.      <ul>

67.      <% @video.errors.full_messages.each do |msg| %>

68.        <li><%= msg %></li>

69.      <% end %>

70.      </ul>

71.    </div>

72.  <% end %>

73.  <div class="form-group">

74.    <label >Title</label>

75.    <%= f.text_field :title, :class=>"form-control", :placeholder => "title" %>

76.  </div>

77.  <div class="form-group">

78.    <label >Description</label>

79.    <%= f.text_area :description, :class => "form-control" %>

80.  </div>

81.  <div class="form-group">

82.    <label for="InputFile">Upload Video</label>

83.     <%= f.file_field :media %>

84.     <%= f.hidden_field :media_cache %>

85.    <p class="help-block"></p>

86.  </div>

87.  <%= f.submit "Save", :class => "btn btn-default" %> <%= link_to 'Cancel', videos_path, :class => "btn btn-danger" %>

<% end %>

88.          Lastly, we will restrict our video formats to MP4, OGV, and AVI. This will only allow the whitelisted file formats to be uploaded:


90.# encoding: utf-8

91.class VideoUploader < CarrierWave::Uploader::Base

92.  include CarrierWave::MimeTypes

93.  def store_dir

94.    "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"

95.  end

96.  def extension_white_list

97.     %w(mp4 ogv avi)

98.  end


Objective complete – mini debriefing

This task was a recap of things we have already done in our past projects. We created model and controllers views for the video. We added the carrierwave uploader and restricted the formats for a video upload in order to avoid malicious uploads as shown in the following code:

def extension_white_list

  %w(mp4 ogv avi)


Please keep in mind that we have chosen the storage mechanism as file only for the sake of convenience. Video files are generally big in size, hence, we do not want to select them again and again. Therefore, we added media_cache to the video_params so that the form retains the selected video, even if the validation fails and the form reloads afterwards. There are several other mechanisms such as Amazon S3 and Rackspace Block Storage to store the files.

The form for video upload with the upload video file field looks like the following screenshot:

Objective complete – mini debriefing

Encoding the video

Video encoding should be part of the upload process. We need to encode the uploaded video files to an HTML5-friendly format, basically the MP4 format, which is fully implemented in the new HTML standard. We will use the ffmpeg on the system side andcarrierwave-video extensions on our application side to do so. During the implementation of this process, we will also update the library for carrierwave-video to ensure it suits our needs.

Prepare for lift off

We will first install the dependencies for ffmpeg. We'll also need to install the Theora and Vorbis protocols for audio and video respectively.

1.    The best way to install ffmpeg on Mac OS X is through the use of homebrew:

2.  $ brew install ffmpeg --with-fdk-aac --with-ffplay --with-freetype --with-frei0r --with-libass --with-libvo-aacenc --with-libvorbis --with-libvpx --with-opencore-amr --with-openjpeg --with-opus --with-rtmpdump --with-schroedinger --with-speex --with-theora –with-tools

3.  ==> Installing dependencies for ffmpeg: texi2html, yasm, x264, faac, lame, xvid, libpng, freetype, libogg, xz, libvorbis, theora, libvpx, rtmp

4.  ==> Installing ffmpeg dependency: texi2html

5.    Following are the instructions for building ffmpeg on Ubuntu:

6.  mutube$ sudo apt-get -y install autoconf automake build-essential git libass-dev libgpac-dev \

7.    libsdl1.2-dev libtheora-dev libtool libva-dev libvdpau-dev libvorbis-dev libx11-dev \

8.    libxext-dev libxfixes-dev pkg-config texi2html zlib1g-dev

9.    It is most preferable to install all dependencies by compiling them from the source. The first dependency is yasm, an assembler used by video and audio encoders:

10.$wget http://www.tortall.net/projects/yasm/releases/yasm-1.2.0.tar.gz

11.yasm$ tar xvzf yasm.tar.gz

12.yasm$ sed -i 's#) ytasm.*#)#' Makefile.in &&

13../configure --prefix=/usr &&


15.yasm$ make install

16.          Check whether yasm is installed or not:

17.yasm $ sudo which yasm


19.          Now, we will install x264, the video encoder:


21.$tar xvjf last_x264.tar.bz2

22.x264$ ./configure --prefix="$HOME/ffmpeg_build" --bindir="$HOME/bin" --enable-static

23.x246$ make

24.x264$ sudo make install

25.          After the installation of the video encoder, we will install the audio encoder, aac. The newer version of ffmpeg uses aac instead of the earlier library, libfaac:

26.$ git clone git@github.com:mstorsjo/fdk-aac.git

27.$cd fdk-aac

28.fdk-aac$autoreconf -fiv

29.fdk-aac$ ./configure --prefix="$HOME/ffmpeg_build" --disable-shared

30.fdk-aac$ make

31.fdk-aac$ make install

32.          Next, we will add support for .mp3 audio:

33.$ sudo apt-get install libmp3lame-dev

34.          We also need to add Opus's encoder and decoder support:

35.$ sudo apt-get install libopus-dev

36.          We need support for V8/V9 video formats, so we will compile the libvpx project extracted from Android:

37.$git clone http://git.chromium.org/webm/libvpx.git

38.cd libvpx

39./configure --prefix="$HOME/ffmpeg_build" --disable-examples


41.make install

42.          After all the dependencies are installed, we will compile ffmpeg using different protocol supports such as aac, x264, and x11 compatibilities:

43.$ git clone --depth 1 git://source.ffmpeg.org/ffmpeg

44.$ cd ffmpeg

45.ffmpeg$ PKG_CONFIG_PATH="$HOME/ffmpeg_build/lib/pkgconfig"

46.ffmpeg$ export PKG_CONFIG_PATH

47.ffmpeg$ ./configure --prefix="$HOME/ffmpeg_build" \

48.  --extra-cflags="-I$HOME/ffmpeg_build/include" --extra-ldflags="-L$HOME/ffmpeg_build/lib" \

49.  --bindir="$HOME/bin" --extra-libs="-ldl" --enable-gpl --enable-libass --enable-libfdk-aac \

50.  --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx \

51.  --enable-libx264 --enable-nonfree --enable-x11grab

52.ffmpeg$ make

53.ffmpeg$ make install

54.          We will test our ffmpeg installation with a simple command to check if aac support is installed or not:

55.$ ffmpeg -formats 2>&1 | grep aac

56.  configuration: --prefix=/home/rwub/ffmpeg_build --extra-cflags=-I/home/rwub/ffmpeg_build/include --extra-ldflags=-L/home/rwub/ffmpeg_build/lib --bindir=/home/rwub/bin --extra-libs=-ldl --enable-gpl --enable-libass --enable-libfdk-aac --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree --enable-x11grab --enable-libfaac

57. D  aac             raw ADTS AAC (Advanced Audio Coding)

58.          The command will return the configuration details of AAC and hence we know that ffmpeg is installed properly.

59.          For installation on Windows, the builds are available at http://ffmpeg.zeranoe.com/builds/. For using the archive, we need 7-zip installed on our machine. In order to install it, we need to download and unzip the archive first. From the bin folder inside our unzipped archive, we will see a file called ffmpeg.exe. We need to copy it to the path C:/Tools/bin in our filesystem.

Engage thrusters

In this task, we will encode our video during our upload process:

1.    We will use a plugin called carrierwave-video that in turn uses the streamio-ffmpeg gem to connect to and subsequently use ffmpeg features. Before we proceed with its installation, we will customize it a bit. I have forked the original gem to my GitHub ID and cloned the repository on my local machine as follows:

2.  $ git clone https://github.com/saurabhbhatia/carrierwave-video.git

3.    We will change the custom option under default options, and remove qscale from the options:

4.  carrierwave_video/lib/carrierwave/video/ffmpeg_options.rb

5.  h[:custom] = '-qscale 0 -preset slow -g 30'

+            h[:custom] = "-strict experimental -preset slow -g 30"

6.    We will also remove the default audio codec from MP4 and let ffmpeg autodetect the audio codec by itself:

7.  carrierwave_video/lib/carrierwave/video/ffmpeg_options.rb

-              h[:audio_codec] = 'aac'

8.    So our method now looks like the following:

9.  lib/carrierwave/video/ffmpeg_options.rb


11.  def defaults

12.    @defaults ||= { resolution: '640x360', watermark: {} }.tap do |h|

13.      case format

14.      when 'mp4'

15.        h[:video_codec] = 'libx264'

16.        h[:custom] = "-strict experimental -preset slow -g 30"

17.      when 'ogv'

18.        h[:video_codec] = 'libtheora'

19.        h[:audio_codec] = 'libvorbis'

20.        h[:custom] = '-b 1500k -ab 160000 -g 30'

21.      when 'webm'

22.        h[:video_codec] = 'libvpx'

23.        h[:audio_codec] = 'libvorbis'

24.        h[:custom] = '-b 1500k -ab 160000 -f webm -g 30'

25.      end

26.    end


27.          Once it is ready, we can commit and push these changes to the repository. We will now bundle directly from our forked repository to pick up the changes we just did:


29.gem "streamio-ffmpeg"

30.gem 'carrierwave-video', :github => 'saurabhbhatia/carrierwave-video'

31.          We will add an encode process to encode our video to MP4:


33.def encode

34.  process encode_video: [:mp4, callbacks: { after_transcode: :set_success } ]


35.          In our video_uploader.rb file, we will have to include the CarrierWave video module. We will also have to add our encode method as a process to generate the MP4 version of the video:


37.class VideoUploader < CarrierWave::Uploader::Base

38.  include CarrierWave::MimeTypes

39.  include CarrierWave::Video

40.  storage :file


42.  def store_dir

43.    "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"

44.  end


46.  version :mp4 do

47.    process :encode

48.  end


50. def encode

51.   process encode_video: [:mp4, callbacks: { after_transcode: :set_success } ]

52. end


54.  def extension_white_list

55.     %w(mp4 ogv avi)

56.  end


This will encode the video and convert it into MP4 once it is uploaded.

57.          We would also like to add a watermark to our video after it is uploaded. This is to avoid plagiarism as much as possible. We will first set the path for the watermark image:



60.    watermark: {

61.      path: Rails.root.join('mutube.png')

62.    }


63.          Now we will modify the encode method we previously wrote to add a watermark to the video:


65.def encode

66.  encode_video(:mp4, DEFAULTS) do |movie, params|

67.   if movie.height < 720

68.      params[:watermark][:path] = Rails.root.join('mutube.png')

69.   end

70.  end


71.          So finally, our uploader looks like the following:


73.# encoding: utf-8

74.class VideoUploader < CarrierWave::Uploader::Base

75.  include CarrierWave::MimeTypes

76.  include CarrierWave::Video

77.  storage :file


79.  def store_dir

80.    "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"

81.  end


83.  DEFAULTS = {

84.    watermark: {

85.      path: Rails.root.join('mutube.png')

86.    }

87.  }

88.  version :mp4 do

89.    process :encode

90.  end


92.     ]

93.  def encode

94.    encode_video(:mp4, DEFAULTS) do |movie, params|

95.      if movie.height < 720

96.        params[:watermark][:path] = Rails.root.join('mutube.png')

97.      end

98.    end

99.  end


101.     def extension_white_list

102.        %w(mp4 ogv avi)

103.     end


104.     We will generate a screenshot for our video now. We will directly access and use the streamio-ffmpeg library in order to generate a screenshot and save it to the specified path:

105.   app/models/video.rb

106.   def video_screenshot

107.       screenshot_path = Rails.root+"/app/assets/images/screenshots/#{self.slug}_#{self.id}.jpg"

108.       if FileTest.exists?(screenshot_path)

109.         @screenshot = screenshot_path

110.       else

111.         video_file = FFMPEG::Movie.new("#{Rails.root}/public"+self.video.url(:mp4))

112.         @screenshot = video_file.screenshot("#{screenshot_path}")

113.       end


114.     When we upload the video, we can see the following parameters:

Engage thrusters

Objective complete – mini debriefing

In this task, we began with the installation of ffmpeg. Their website (http://ffmpeg.org) defines it as:

FFmpeg is a complete, cross-platform solution to record, convert and stream audio and video.

We installed all the dependencies required to run ffmpeg and compiled it from the source. Be sure to check more command-line options at the following links:

·        Generic options: http://ffmpeg.org/ffmpeg.html#Generic-options

·        Customizing videos: http://ffmpeg.org/ffmpeg.html#Video-Options

·        Advanced options: http://ffmpeg.org/ffmpeg.html#Advanced-Video-Options

We then customized the carrierwave-video (https://github.com/rheaton/carrierwave-video) plugin. We updated the protocols for video and audio encoding to aac. The earlier version of ffmpeg used a libfaac to connect to the audio protocol. It has been deprecated in the newer version. We allowed ffmpeg to autodetect the audio protocol and encode accordingly. Next, we added the ability to watermark our videos in our application. The uploader will call the watermark image and send it along with the encoding command. You can see the following command, which is being fired to encode our video to MP4:

ffmpeg -y -i /home/user/mutube/public/uploads/tmp/1389051112-8075-3957/mp4_scroll_index.mp4 -vcodec libx264 -s 640x358  -strict experimental -preset slow -g 30 -vf "movie=/home/user/mutube/mutube.png [logo]; [in][logo] overlay= [out]" -aspect 1.7877094972067038 /home/user/mutube/public/uploads/tmp/1389051112-8075-3957/tmpfile.mp4

We also generated a screenshot. We first checked whether the file already exists or not. We used FileTest and pointed it to the exact path:

    screenshot_path = "#{Rails.root}/app/assets/images/screenshots/"+"#{self.slug}_#{self.id}.jpg"

    if FileTest.exists?(screenshot_path)

      @screenshot = screenshot_path

To generate the screenshot, we used the streamio-ffmpeg library:

      video_file = FFMPEG::Movie.new("#{Rails.root}/public/"+self.video.url(:mp4))

      @screenshot = video_file.screenshot("#{Rails.root}/app/assets/images/screenshots/"+"#{self.slug}_#{self.id}.jpg")



Displaying the video panel and playing the video

Displaying and playing videos has been a challenge for a long time. Flash has dominated the game all along and still powers most of the major websites. However, flash has not been good at optimization for mobile devices. HTML5 then released a fresh set of standards including MP4 and OGV for video, AAC, and OGG for audio. We will use these standards to display the video along with the video.js library.

Engage thrusters

We will display the uploaded videos in this task:

1.    We will first download the latest version of video.js (download it from http://www.videojs.com/downloads/video-js-4.4.2.zip) and unzip it. We will place it under ourjavascripts folder under app/assets/. We will load it to our manifest file:

2.  app/assets/javascripts/application.js

3.  //= require jquery

4.  //= require jquery_ujs

5.  //= require twitter/bootstrap

6.  //= require video

7.  //= require turbolinks

//= require_tree

8.    We will copy the video-js.css file to the stylesheets folder under app/assets/ and add it's reference to the CSS manifest file:

9.  app/assets/stylesheets/application.css

10.*= require_self

11. *= require twitter/bootstrap

12. *= require video-js

13. *= require sticky-footer-navbar

14. *= require font-awesome

15. *= require_tree .


16.          We will also have to place video-js.swf in our Rails application root.

17.          We will start by adding the initialization code to the show.html.erb file:



20.  videojs.options.flash.swf = "#{Rails.root}/video-js.swf";


21.          We will need to create a video element in our show.html.erb and load video.js default skin. We will also load the MP4 version of the video:

22.app/views/videos/ show.html.erb

23.<div class="row">

24.    <div class="col-lg-8">

25.      <h3><%= @video.title %></h3>

26.      <video id="video_1" class="video-js vjs-default-skin" controls preload="none" width="640" height="264"

27.      data-setup="{}">

28.  <source src="<%=@video.video.url(:mp4)%>" type='video/mp4' />

29.      </video>

30.      <br/>

31.      <p><%= @video.description %></p>

32.    </div>


We will now reload our page to see the video:

Engage thrusters

33.          In order to increase the engagement of our website, we will display all the videos other than the current video in our show page.

34.          We will first write a class method to find all videos other than the current video being viewed:

35. app/models/video.rb

36. def self.get_other_videos(video_id)

37.    videos = Video.where.not(id: video_id) rescue []

38.    return videos


39.          We will make a call on this method in the videos_controller.rb file:


41.def show

42.  @videos = Video.get_other_videos(@video.id)


43.          Then, we will loop through these videos and display them as a list on the right-hand side of the page. We will also create a helper to display the screenshot for each video:


45.module VideosHelper

46.  def display_screenshot(video_slug,video_id)

47.    "screenshots/#{video_slug}_#{video_id}.jpg"

48.  end



51.   <div class="row">

52.    <div class="col-lg-8">

53.      <h3><%= @video.title %></h3>

54.      <video id="example_video_1" class="video-js vjs-default-skin" controls preload="none" width="640" height="264"

55.      data-setup="{}">

56.    <source src="<%=@video.video.url(:mp4)%>" type='video/mp4' />

57.      </video>

58.      <br/>

59.      <p><%= @video.description %></p>

60.    </div>

61.    <div class="col-lg-4">

62.      <h3>Other Videos</h3>

63.      <% @videos.each do |video| %>

64.          <h3><%=link_to video.title, video %></h3>

65.          <% video.video_screenshot%>

66.          <p><%= image_tag display_screenshot(video.slug,video.id) , :width => 200, :height => 150 %></p>

67.          <p><%= link_to 'See this Video »'.html_safe ,video, :class=>"btn btn-success"%></p>

68.     <%end%>

69.    </div>


70.          The following screenshot displays the videos on the right-hand side of the screen:

Engage thrusters

Objective complete – mini debriefing

In this task, we used the video.js library to display the uploaded and encoded video. We added the appropriate JavaScript and CSS to the application.js and application.css files. After that, we initiated videojs with options. We kept video-js.swf in order to keep a fallback for browsers that do not support the HTML5 video as yet:


  videojs.options.flash.swf = "#{Rails.root}/video-js.swf";


Then we created a video element to load the video on it:

  <video id="video_1" class="video-js vjs-default-skin" controls preload="none" width="640" height="264"


    <source src="<%=@video.video.url(:mp4)%>" type='video/mp4' />


We also created a helper method to create our screenshot path and called it in our view.

The video element has been introduced in HTML5 as a native HTML tag. By default, it supports MP4 and OGV formats. The work on video support is still in progress but most modern browsers such as Chrome and Firefox support it fully. Video.js forms a layer on top of the HTML5 element and modifies it to fall back to flash for older browsers and provide more advanced options for video control. It also gives a lot of flexibility to change the skins of the video player.

As part of HTML5 video standard, multiple device compatibility comes inbuilt. We will test our application on devices other than the desktop. The first test is done on iPad Retina, running iOS 7. The video works flawlessly on it as shown in the following screenshot:

Objective complete – mini debriefing

We will do the second test on an Android phone, running the Android 4.2.2 Jelly Bean as shown in the following screenshot:

Objective complete – mini debriefing

We are using the friendly_id gem to generate slugs, so the params[:id] will pass the name of slug instead of id as the gem modifies the to_param method in Rails to call the name attribute instead of id. In order to access the id attribute in the show method, we haveused the video object directly:

def show

    @videos = Video.get_other_videos(@video.id)


We queried our database to get videos other than the current video. Our query omits the video_id and finds all videos except the current one. We will use the catch a nil exception using rescue.nil in order to avoid failures caused due to nil records:

videos = Video.where.not(id: video_id) rescue nil

Also, we can see the watermark now, the one we created in our previous task, on the top-left side of the screen in the following screenshot:

Objective complete – mini debriefing

Caching the content – text and video

In this task, we will look at some of the newer techniques of caching. Russian Doll Caching was introduced in Rails 3.2 and is now used in Rails 4 as the main mechanism of page and fragment caching. It also implements the usage of cache digest. This will lead to effective versioning of cached items even if someone misses out on adding the right version of cache while writing the code.

Engage thrusters

We will now take steps to add specific caching mechanisms to our application:

1.    In our video model, we will add ActiveRecord's touch method to keep the served content fresh:

2.  class Video < ActiveRecord::Base

3.    belongs_to :user, touch: true

4.    mount_uploader :video, VideoUploader

5.    extend FriendlyId

6.    friendly_id :title, use: :slugged


8.    def video_screenshot

9.      screenshot_path = "#{Rails.root}/app/assets/images/screenshots/"+"#{self.slug}_#{self.id}.jpg"

10.    if FileTest.exists?(screenshot_path.to_s)

11.       @screenshot = screenshot_path.to_s

12.    else

13.       video_file = FFMPEG::Movie.new("#{Rails.root}/public"+self.video.url(:mp4))

14.       @screenshot = video_file.screenshot(screenshot_path.to_s)

15.    end

16.  end


18.  def self.get_other_videos(video_id)

19.   videos = Video.where.not(id: video_id) rescue nil

20.   return videos

21.  end


22.          We will then cache the various fragments of our view. Our home page has two parts as shown in the following screenshot:

Engage thrusters

23.          In order to keep up with the template changes, we will create versions for our cache. The first segment is the video itself. We will cache video.title, video.description, and video object:


25. <% cache ["v1",@video] do %>

26.   <div class="row">

27.    <div class="col-lg-8">

28.      <h3><%= @video.title %></h3>

29.      <video id="example_video_1" class="video-js vjs-default-skin" controls preload="none" width="640" height="264"

30.      data-setup="{}">

31.  <source src="<%=@video.video.url(:mp4)%>" type='video/mp4' />

32.      </video>

33.      <br/>

34.      <p><%= @video.description %></p>

35.    </div>

 <% end %>

36.          The second segment for caching is the right-hand side bar where we display the videos. We will cache the @videos object, video title, and also the screenshot for all the videos:


38.<% cache ["v1",@videos] do%>

39.    <div class="col-lg-4">

40.      <h3>Other Videos</h3>

41.      <% @videos.each do |video| %>

42.          <h3><%=link_to video.title, video %></h3>

43.          <% video.video_screenshot%>

44.          <p><%= image_tag "screenshots/#{video.slug}_#{video.id}.jpg", :width => 200, :height => 150 %></p>

45.          <p><%= link_to 'See this Video »'.html_safe ,video, :class=>"btn btn-success"%></p>

46.     <%end%>

47.    </div>

48.   </div>

 <% end %>

49.          We will precompile our assets, boot them into production, and reload our page:

50.$ bundle exec rake assets:precompile

51.          We can see the command-line output, as shown in the following screenshot, after booting into production:

Engage thrusters

52.          In case someone misses the version number in the cache definition, it will lead to an error. Hence, we will make our application version free by removing the versions that are not required:


54.<% cache @video do %>

55.   <div class="row">

56.    <div class="col-lg-8">

57.      <h3><%= @video.title %></h3>

58.      <video id="example_video_1" class="video-js vjs-default-skin" controls preload="none" width="640" height="264"

59.      data-setup="{}">

60.<source src="<%=@video.video.url(:mp4)%>" type='video/mp4' />

61.      </video>

62.      <br/>

63.      <p><%= @video.description %></p>

64.    </div>

65. <% end %>

66. <% cache @videos do%>

67.    <div class="col-lg-4">

68.      <h3>Other Videos</h3>

69.      <% @videos.each do |video| %>

70.          <h3><%=link_to video.title, video %></h3>

71.          <% video.video_screenshot%>

72.          <p><%= image_tag "screenshots/#{video.slug}_#{video.id}.jpg", :width => 200, :height => 150 %></p>

73.          <p><%= link_to 'See this Video »'.html_safe ,video, :class=>"btn btn-success"%></p>

74.     <%end%>

75.    </div>

76.   </div>

 <% end %>

77.          We need to prepare our production configuration to load our asset pipeline first:


79.# Precompile additional assets.

80.  # application.js, application.css, and all non-JS/CSS in app/assets folder are already added.

  config.assets.precompile += ['*.js', '*.css', '*.css.erb']

81.          After we have prepared our production environment for the cache and asset pipeline, we will boot our production server and render our show page, where we enabled the cache. We can see the cache digest for our videos in the page that will appear in the following screenshot:

Engage thrusters

Objective complete – mini debriefing

In this task, we used the Russian Doll caching technique to divide the page into fragments and cache them separately. In order to keep our caching objects free from the complexity of logged in objects, we have cached only those sections that do not need logging in and ones that don't depend on session objects whatsoever. We started by adding the touch method to our video-model association:

 belongs_to :user, touch: true

The touch method is used to keep the data in the cache fresh. It is particularly useful for associations. So in our use case, whenever a new video is created by a particular user, the cache expires and loads with the details of the new video. Also, when a video has been updated or deleted, the touch method automatically resets the cache and updates it.

Then, we added versioned caching to our page fragments:

<% cache ["v1",@videos] do%>

<% end %>

The cache generates fragments with version numbers as follows:

I, [2014-01-07T07:49:18.018543 #10073]  INFO -- : Cache digest for videos/show.html: 17d26e8a6e5adce61cf3a6d90e1eabd5

I, [2014-01-07T07:49:18.020463 #10073]  INFO -- : Read fragment views/v1/videos/6-20140106233228000000000/17d26e8a6e5adce61cf3a6d90e1eabd5 (0.3ms)

I, [2014-01-07T07:49:18.026462 #10073]  INFO -- : Write fragment views/v1/videos/6-20140106233228000000000/17d26e8a6e5adce61cf3a6d90e1eabd5 (4.2ms)

I, [2014-01-07T07:49:18.030010 #10073]  INFO -- : Read fragment views/v1/videos/5-20140106233152000000000/17d26e8a6e5adce61cf3a6d90e1eabd5 (0.3ms)

Then we removed the version numbers in order to avoid pitfalls due to versioning. By default, Rails generates a unique ID and version for cache digests with every render:

<% cache @videos do%>

<% end %>

The cache now generates fragments without version numbers:

I, [2014-01-07T07:33:13.491723 #8193]  INFO -- : Read fragment views/videos/6-20140106233228000000000/8d8424568d7560f7eb85717b3b6e8a71 (0.4ms)

I, [2014-01-07T07:33:13.494620 #8193]  INFO -- : Read fragment views/videos/5-20140106233152000000000/8d8424568d7560f7eb85717b3b6e8a71 (0.3ms)

In order to expire or invalidate the cache, we can simply add the expires_in option:

 <% cache [@videos, expires_in: 30.minutes] do%>

It should also be noted that we can add race_condition_ttl along with our expires_in option to avoid something called the dogpile effect. This happens when there are two simultaneous requests and there is a possibility that the cache ID generated for both requests could be the same.

Queuing the job

Video uploading and encoding might sometimes take a lot of time while processing, depending on the size of the video, the kind of Internet connection, and upload bandwidth of the user. In this case, waiting for the video to upload in order to go to other pages could be a very annoying experience for the user. In order to enhance the user experience, we can run our encoding as a background job and make an asynchronous queue of videos in order to schedule and encode them as we do other tasks. We will use Sidekiq to generate queues and manage the background processing; we will connect it to the carrierwave gem.

Prepare for lift off

1.    We will start by installing Redis on our machine. We will download the latest copy of the Redis source and build it from its source:

2.  $ wget http://download.redis.io/redis-stable.tar.gz

3.  $ tar xvzf redis-stable.tar.gz

4.  $ cd redis-stable

5.  redis-stable$ make

We will then start the Redis server:

$ redis-server

6.    This installation works for Linux- and Mac OS X-based systems. For further details on this, you can visit the Redis download page at http://redis.io/download.

7.    We will test the Redis configuration by pinging the redis-cli command:

8.  $ redis-cli ping

9.  PONG

10.          Add redis to our application's Gemfile and run bundle install:

11.'redis', '>= 3.0.6'

12.'redis-namespace', '>= 1.3.1'

13.          Create an initializer to connect to the local redis instance:


15.$redis = Redis.new(:host => 'localhost', :port => 6379)

16.          We will test the connection to Redis from our Rails console:

17.mutube$ rails c

18.Loading development environment (Rails 4.0.2) :001 > $redis

20. => #<Redis client v3.0.6 for redis://localhost:6379/0

21.          Redis is now working within our application, and we can now go ahead with our Sidekiq installation.

Engage thrusters

In the following steps, we will add a queuing mechanism to our application:

1.    Add sidekiq to the Gemfile and run bundle install:

2.  Gemfile

3.  gem 'sidekiq'

4.    Once the bundle is successful, we will also need to add a carrierwave extension to run the background job:

5.  Gemfile

6.  gem 'carrierwave_backgrounder'

7.    We will generate the initializer once the gem is successfully installed:

8.   mutube$ rails g carrierwave_backgrounder:install

9.        create  config/initializers/carrierwave_backgrounder.rb

By default, delayed_job is used as the backend for carrierwave_backgrounder with :carrierwave as the queue name.

To change this, edit config/initializers/carrierwave_backgrounder.rb.

10.          In the initializer for carrierwave_backgrounder.rb, we will define sidekiq as the queuing methodology and carrierwave as the default queue name:


12.CarrierWave::Backgrounder.configure do |c|

13.  #c.backend :delayed_job, queue: :carrierwave

14.  # c.backend :resque, queue: :carrierwave

15.   c.backend :sidekiq, queue: :carrierwave

16.  # c.backend :girl_friday, queue: :carrierwave

17.  # c.backend :sucker_punch, queue: :carrierwave

18.  # c.backend :qu, queue: :carrierwave

19.  # c.backend :qc


20.          We will include the CarrierWave::Backgrounder module in our uploader:


22.class VideoUploader < CarrierWave::Uploader::Base

23.  include CarrierWave::MimeTypes

24.  include CarrierWave::Video

25.  include ::CarrierWave::Backgrounder::Delay

26.  storage :file

27.  def store_dir

28.    "uploads/#{model.class.to_s.underscore}/#{mounted_as}/#{model.id}"

29.  end

30.  DEFAULTS = {

31.    watermark: {

32.      path: Rails.root.join('mutube.png')

33.    }

34.  }

35.  version :mp4 do

36.    process :encode

37.  end

38.  version :screenshot do

39.    process :screenshot

40.  end

41.  def encode

42.    encode_video(:mp4, DEFAULTS) do |movie, params|

43.      if movie.height < 720

44.        params[:watermark][:path] = Rails.root.join('mutube.png')

45.      end

46.    end

47.  end

48.  def extension_white_list

49.     %w(mp4 ogv avi)

50.  end


51.          We will load this module onto our model using a method called process_in_background and call our uploader in it:


53.class Video < ActiveRecord::Base

54.  belongs_to :user, touch: true

55.  mount_uploader :video, VideoUploader

56.  extend FriendlyId

57.  friendly_id :title, use: :slugged

58.  process_in_background :video

59.def video_screenshot

60.  screenshot_path = "#{Rails.root}/app/assets/images/screenshots/"+"#{self.slug}_#{self.id}.jpg"

61.  if FileTest.exists?(screenshot_path.to_s)

62.    @screenshot = screenshot_path.to_s

63.  else

64.    video_file = FFMPEG::Movie.new("#{Rails.root}/public"+self.video.url(:mp4))

65.    @screenshot = video_file.screenshot(screenshot_path.to_s)

66.  end


68.def self.get_other_videos(video_id)

69.  videos = Video.where.not(id: video_id) rescue nil

70.  return videos

71.  end


72.          We will now load our console and make a test call on our Sidekiq method to check whether everything is fine or not:

73.Loading development environment (Rails 4.0.2) :001 > Sidekiq::Client.registered_workers

75.2014-01-05T10:56:01Z 20119 TID-2dbz4 INFO: Sidekiq client using redis://localhost:6379/0 with options {}

76. => []

77.          Looks good! So we can now fire up our Sidekiq server:

78.$ bundle exec sidekiq -q carrierwave,5 default

79.2014-01-06T23:10:41Z 4735 TID-5rkyo INFO: Booting Sidekiq 2.14.0 using redis://localhost:6379/0 with options {}

80.2014-01-06T23:10:41Z 4735 TID-5rkyo INFO: Running in ruby 1.9.3p327 (2012-11-10 revision 37606) [x86_64-linux]

81.2014-01-06T23:10:41Z 4735 TID-5rkyo INFO: See LICENSE and the LGPL-3.0 for licensing details.

82.2014-01-06T23:10:41Z 4735 TID-5rkyo INFO: Starting processing, hit Ctrl-C to stop

83.2014-01-06T23:10:43Z 4735 TID-c6ffw CarrierWave::Workers::ProcessAsset JID-c565e44604416585b8a01a8e INFO: start

84.2014-01-06T23:10:43Z 4735 TID-c8hgc CarrierWave::Workers::ProcessAsset JID-7491d6a930d19b5f593197d8 INFO: start

85.2014-01-06T23:10:44Z 4735 TID-c6ffw CarrierWave::Workers::ProcessAsset JID-c565e44604416585b8a01a8e INFO: done: 1.044 sec

86.2014-01-06T23:10:44Z 4735 TID-c8hgc CarrierWave::Workers::ProcessAsset JID-7491d6a930d19b5f593197d8 INFO: done: 1.04 sec

87.          We will now try to upload a video and see how the process looks in our console:

Engage thrusters

88.          We will also enable the web interface for Sidekiq. For this, we need sinatra. In our Gemfile, add sinatra and run bundle install:


90.gem 'sinatra', '>= 1.3.0', :require => nil

91.          Mount the Sidekiq route in our application routes:


93.require 'sidekiq/web'

94.MuTube::Application.routes.draw do

95.  devise_for :users

96.  get "home/index"

97.  resources :videos

98.  mount Sidekiq::Web => '/sidekiq'

99.  root 'videos#index'


100.     Reboot the server and browse to http://localhost:3000/sidekiq/. We can now see the Sidekiq job management dashboard in the following screenshot:

Engage thrusters

Objective complete – mini debriefing

Before we began the task, we prepared our system to run Redis. This is a dependency to run Sidekiq. It is worth noting that most of the job queues use Redis as a choice of persistence. This is because Redis is fast, easy to manage, and is document-oriented. Sidekiq has other popular alternatives as well, for example, resque and delayed job. The reasons we chose Sidekiq over the other two were as follows:

·        Delayed job has no management dashboard

·        Resque is known to be more memory inefficient than Sidekiq

Though we need to make sure our code is threadsafe with Sidekiq, as it inherently does not protect itself from non-threadsafe objects, it still is good enough for most tasks as it occupies much less memory than its peers. The following is an example of the Sidekiq dashboard, which even provides reports for jobs:

Objective complete – mini debriefing

We first enabled sidekiq with the carrierwave queue.

c.backend :sidekiq, queue: :carrierwave

Then, we loaded the module for queues in our uploader:

include ::CarrierWave::Backgrounder::Delay

Furthermore, we enabled a background job in the model. So now when we upload a video, it is immediately sent to a queue. Untill then, we can proceed to do other tasks as the video is being encoded and worked on in the backend. We do have to make sure, however, that the server for Sidekiq needs to be up all the time. The other purposes of using queues could be mailing newsletters or reindexing solr.

Mission accomplished

We have successfully created an app where we can upload and encode a video, then display it. Let's have a quick recap of what we did.

Some of the areas we covered in this project are as follows:

·        We created a Rails app, video model, and uploader. We also added methods to create slugs using the friendly_id gem.

·        We restricted the formats of videos to be uploaded.

·        We installed ffmpeg and its dependencies.

·        We used customized carrierwave-video to suit our needs for encoding the video. We transcoded the videos to the MP4 format.

·        In order to display the video, we used video.js.

·        We made sure it works on multiple devices and platforms.

·        We used Russian Doll caching to cache our videos, screenshots, and text.

·        We used Sidekiq to create and manage queues for multiple video uploads.

We are using ffmpeg that is compiled from the source. If we are using it for production, we need to be very sure about fixing a version for a long time. This is because command-line flags and encoding filenames in ffmpeg change very often, and there are several deprecations between versions. To check the current configuration of ffmpeg, we will run the following code:

$ ffmpeg

ffmpeg version git-2014-01-01-07728a1 Copyright (c) 2000-2013 the FFmpeg developers

  built on Jan  1 2014 20:16:31 with gcc 4.8 (Ubuntu/Linaro 4.8.1-10ubuntu9)

  configuration: --prefix=/home/rwub/ffmpeg_build --extra-cflags=-I/home/rwub/ffmpeg_build/include --extra-ldflags=-L/home/rwub/ffmpeg_build/lib --bindir=/home/rwub/bin --extra-libs=-ldl --enable-gpl --enable-libass --enable-libfdk-aac --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-nonfree --enable-x11grab --enable-libfaac

  libavutil      52. 59.100 / 52. 59.100

  libavcodec     55. 47.100 / 55. 47.100

  libavformat    55. 22.102 / 55. 22.102

  libavdevice    55.  5.102 / 55.  5.102

  libavfilter     4.  0.103 /  4.  0.103

  libswscale      2.  5.101 /  2.  5.101

  libswresample   0. 17.104 /  0. 17.104

  libpostproc    52.  3.100 / 52.  3.100

Hyper fast Audio and Video encoder

usage: ffmpeg [options] [[infile options] -i infile]... {[outfile options] outfile}...

Use -h to get full help or, even better, run 'man ffmpeg'

MP4 encoding is a tricky thing using ffmpeg. By default, MP4 hinting is not set (hinting allows the video to be streamed as soon as it completes to load by setting a flag). This disables the autoplay completely, as it looks to download the entire file before it starts playing. MP4Box or Nginx MP4 streaming can be used for this purpose. We need to make sure of the following factors:

·        Make sure the file size limit is defined and is enough for videos to be uploaded

·        Allow incoming files while uploading

Hotshot challenges

We created a fully functional video platform that can be used to upload and manage videos. We can extend it further to make it even cooler:

·        Allow encoding to the OGV and Theora format of the uploaded videos

·        Expire the cache key after 5 minutes

·        Add custom skins to video.js

·        Enable automatic retry in case the job fails

·        If the file size of video is very small, then bypass the background job