Rails 4 Application Development HOTSHOT (2014)

Chapter 6. Creating an Analytics Dashboard using Rails and Mongoid

We rely a lot on various analytics tools in our web applications. Google Analytics, Mixpanel, Kissmetrics, and Crazy Egg are some of the most popular web-analytics tools that give a deep insight into who's visiting the website, from where, and what pages are getting the most hits. These analytics help in addressing demographic-based issues, improving the user experience on the site.

Mission briefing

In this project, we will create an analytics dashboard, which will give the user an insight on which kind of content is getting what kind of traffic. There are three types of behavior that we will track with our application:

·        Clicks

·        Views

·        Visits

Clicks and views will be tracked for the users who have logged in. Visits are for the users who unknown and are are not logged in. We will use MongoDB to track and store this data. Also, we will create charts of different types in order to visualize our data. MongoDB is scalable and is meant to be fault tolerant.

We will name our application Authorly and the following is a glimpse of what we are going to achieve:

Mission briefing

Why is it awesome?

Sometimes analytics and visibility for our data needs to be part of our system. Also, if this dashboard is easy to roll out and manage, you can build an entire highly customized system in the long term. This data is valuable for the administrators of the system. In our application, we will create articles and give users the flexibility to track clicks and views to their articles through a dashboard.

Analytics comprises the following three tasks:

·        Collecting the data

·        Analyzing the collected data

·        Reporting the data

At the end of this project, we will be able to build a fully functional analytics dashboard.

Your Hotshot objectives

While building this application, we will go through the following tasks:

·        Creating a MongoDB database

·        Creating a click-tracking mechanism

·        Creating a visit-tracking mechanism

·        Writing map-reduce and aggregation to fetch and analyze the data

·        Creating a dashboard to display clicks and impression values

·        Creating a line graph of the daily clicking activity

·        Creating a bar graph of the daily visit activity

·        Creating a demographic-based donut chart

Mission checklist

We need the following software installed on the system before we start with our mission:

·        Ruby 1.9.3 / Ruby 2.0.0

·        Rails 4.0.0

·        MongoDB

·        Bootstrap 3.0

·        Sass

·        Devise

·        morris.js for charts

·        Git

·        A tool for mockups

·        jQuery

·        ImageMagick and RMagick

Creating a MongoDB database

In this task, we will work towards setting up the base for our application. This includes setting up mongoid, rolify, and creating articles. This task is more like a revision of some of the concepts that we have covered in the book already. The new thing here is that we are doing it all with Mongoid.

Prepare for lift off

In order to start working on this project, we will first have to add the mongoid gem to the Gemfile:

Gemfile

gem 'mongoid', github: 'mongoid/mongoid'

Bundle the application and run the mongoid generator:

$ rails g mongoid:config

At the time of writing this book, the master branch of rolify is compatible only with the master branch of mongoid. So, in order to ensure that both work well together, we need to keep both our Mongoid and rofily on the master branch.

Engage thrusters

The steps for creating a MongoDB database are as follows:

1.    We will take the first step in this task by setting up the skeleton of the application.

2.    We will install rolify from the master branch by adding it to the Gemfile and run bundle:

3.  Gemfile

4.  gem 'rolify', :github => 'EppO/rolify'

5.   

authorly $bundle install

6.    We will then generate the configuration file for rolify:

7.   authorly$rails g rolify Role User -o mongoid

8.        invoke  mongoid

9.        create    app/models/role.rb

10.      invoke    test_unit

11.      create      test/models/role_test.rb

12.      create      test/fixtures/roles.yml

13.      insert    app/models/role.rb

14.      insert  app/models/user.rb

      create  config/initializers/rolify.rb

15.          The initializers generated in order to access mongoid instead of ActiveRecord looks like the following code:

16.config/initializers/rolify.rb

17. 

18.Rolify.configure do |config|

19.  config.use_mongoid

end

20.          We will generate an article's model, view, and controller. This will allow the users to create articles:

authorly$ rails g scaffold article title:string body:text

21.          MongoDB generates pretty ugly URLs, with 12-byte long Binary JSON (BSON) type IDs trailing them. We need to create good looking URLs with MongoDB. For this, we will use the mongoid_slug gem with our application. Again, here we are using the master branch of GitHub to maintain the compatibility with Rails 4 and mongoid 4 beta versions:

22.Gemfile

23. 

gem 'mongoid_slug', github: 'digitalplaywright/mongoid-slug'

24.          After adding it to Gemfile, run bundle install.

25.          In order to set up the slugging mechanism, we will first include the Mongoid::Slug module in our article model:

26.app/models/article.rb

27. 

28.class Article

29.  include Mongoid::Document

30.  include Mongoid::Slug

31. 

32.  field :title, type: String

33.  field :body, type: String

34.  field :user_id, type: String

35.  belongs_to :user

end

36.          Also, we need to store the history of our URL slugs to avoid 404 errors in case the slug changes. This will be stored in an array inside the _slug field in the article model:

37.app/models/article.rb

38. 

39.class Article

40.  include Mongoid::Document

41.  include Mongoid::Slug

42. 

43.  field :title, type: String

44.  field :body, type: String

45.  field :_slugs, type: Array, default: []

46.  field :user_id, type: String

47.  slug  :title, :history => true

end

48.          We will set up an article list such that it can be viewed by anyone without logging in as well as by people who are logged in. Before this step, please make sure devise is installed on your system:

49.app/controllers/articles_controller.rb

50.

51.before_filter :authenticate_user!, except: [:show, :index]

52.def index

53.   

54.      @articles = Article.all

55.   

  end

56.          Lastly, do not forget to add a slug and user ID to the permitted parameters in your articles_controller file:

57.app/controllers/articles_controller.rb

58.   

59. private

60.    # Use callbacks to share common setup or constraints between actions.

61.    def set_article

62.      @article ||= Article.find(params[:id])

63.    end

64.   

65.    # Never trust parameters from the scary internet, only allow the white list through.

66.    def article_params

67.      params.require(:article).permit(:title, :body, :_slugs, :user_id)

    end

Objective complete – mini debriefing

In this task, we started by assuming that devise and cancan have already been installed, as there was no change needed for any of them to work with mongoid. We directly proceeded to the step where we installed rolify with mongoid. We created a model for articles and restricted the access for the show and index pages being accessed by anyone. We then saw the use of a mongoid slug, a library that is used to create pretty and search-friendly URLs. A good solution for slugs not only makes the URL pretty and search friendly, but also maintains a history of the changes done to the URLs. There are chances that the slug might change as it is dependent on the article's title. If a user edits the title, the slug is bound to change. However, if the article is popular and is used by several people, they might have bookmarked it. We used the history feature to maintain both old as well as new URLs, thus avoiding the 404 (URL not found) errors. We also added _slugs to the parameter's whitelist.

Creating a click-tracking mechanism

There is a difference between tracking clicks and tracking impressions. Clicks can be the traffic that is received through an organic search via search engines such as Google, or via searching the website, or whenever a click action is performed. Impression, on the other hand, is how many times the page has been viewed. It is possible that someone has bookmarked the page and repeatedly read an article. In this case, the act will be the counting of impressions. In our application, both clicks and impressions will be bound to the show method because that's what is mainly required to render the page.

Engage thrusters

We will now go ahead and create a click-tracking mechanism for our articles:

1.    We will first create a model for clicks and associate it with the article:

2.  app/models/click.rb

3.   

4.  class Click

5.    include Mongoid::Document

6.    field :ip, type: String

7.    field :url, type: String

8.    field :article_id, type: String

9.    field :user_id, type: String

10. 

11.  belongs_to :article

end

12.          In our article, we will associate our article model with the clicks too:

13.app/models/article.rb

14. 

has_many :clicks

15.          We will first add methods to get the full path of the URL and get the IP address of the user clicking in our show method, inside our articles_controller file:

16.app/controllers/articles_controller.rb

17.def show

18.@url = request.fullpath.to_s

19.    @ip = request.remote_ip

end

20.          Now, we will track the click action whenever it is performed and the show method is fired. Also, we will save article_id with our click. We will do this with the following code:

21.app/models/concerns/record_data.rb

22.module RecordData

23. 

24.  extend ActiveSupport::Concern

25. 

26.  included do

27. 

28.    def self.record(url, ip, article_id, user_id)

29. 

30.      self.create!(url: url, ip: ip, article_id: article_id, user_id: user_id)

31. 

32.     end

33. 

34.  end

35. 

36.end

37. 

38.app/controllers/articles_controller.rb

39. def show

40. 

41.    @clicks = @article.track_clicks_per_article

42. 

43. 

44.    url = request.fullpath.to_s

45. 

46.    ip = request.remote_ip

47. 

48. 

49.    if user_signed_in? && (current_user.id != @article.user_id)

50. 

51.      Click.record(url, ip, @article.id, current_user.id.to_s)

52. 

53.    elsif !user_signed_in?

54. 

55.      Click.record(url, ip, country, city, @article.id, "anonymous")

56. 

57.    end

58. 

  end

59.          Now, we will have the click recorded every time a user clicks on the show method. For an anonymous user, the query looks like the following code:

  MOPED: 127.0.0.1:27017 INSERT database=project6_development collection=clicks documents=[{"_id"=>BSON::ObjectId('528243f37277750cd90a0000'), "url"=>"/articles/the-body-of-lies", "ip"=>"127.0.0.1", "article_id"=>BSON::ObjectId('528011687277750d4a000000'), "user_id"=>"anonymous"}] flags=[]

60.          For a logged-in user, the query looks like the following code:

  MOPED: 127.0.0.1:27017 INSERT database=project6_development collection=clicks documents=[{"_id"=>BSON::ObjectId('5283648d7277750b6a050000'), "url"=>"/articles/the-body-of-lies", "ip"=>"127.0.0.1", "article_id"=>BSON::ObjectId('528011687277750d4a000000'), "user_id"=>"527ce7927277750d00000000"}] flags=[]

Objective complete - mini debriefing

In the preceding task, we created a simple click-tracking mechanism that executes and saves every time the show link is clicked in the frontend. We saved the ID of the article along with our click in order to see which article gets how many clicks:

@url = request.fullpath.to_s

 @ip = request.remote_ip

We created a model concern to create a new click record every time the user clicks on the show action. In our previous project (Project 4Creating a Restaurant Menu Builder), we created a controller concern for the subdomain. Here, we created a reusable classmethod that we can call on different models if we have to create a scorecard with those attributes. In order to include the class method in our model, we just included the module in our model and called the class method on our Click model. We also took measures to track the ID of the user if they are logged in. If the user is anonymous, we will know that the traffic is from a source where the user is not logged in. This will give us wholesome statistics on the clicks received on the article.

Creating a visit-tracking mechanism

In order to track visits and impressions, we will take a slightly different approach. We will use a gem called impressionist to track the page impressions. At the end of the task, we will also debate whether the solution is scalable or not. The difference between impressions and clicks lies in how the article is accessed. So, for example, if a user writes an article that is linked in another website and someone clicks on the link, this would count as a click. However, if a link is bookmarked and the user tries to access it from the bookmarks, it would count as an impression. Hence, we have tied both impressions and clicks to the show method.

Engage thrusters

We will now create view tracking for our articles:

1.    We will first add the impressionist gem to our Gemfile and run bundle. Even here, we will keep our gem to master head so that we grab the latest version that is compatible with Rails 4 and mongoid 4:

gem 'impressionist', github: 'charlotte-ruby/impressionist'

2.    We will now generate the impressionist initializer:

3.  :~/authorly$ rails g impressionist --orm mongoid

4.        invoke  mongoid

      create  config/initializers/impression.rb

5.    The is_impressionable method in the article model will allow impressionist to access the article mode:

6.  app/models/article.rb

7.  class Article

8.    include Mongoid::Document

9.    include Mongoid::Slug

10. 

11.  field :title, type: String

12.  field :body, type: String

13.  field :_slugs, type: Array, default: []

14.  field :user_id, type: String

15. 

16.  is_impressionable

17. 

18.  slug  :title, :history => true

19.  belongs_to :user

20.  has_many :clicks

end

21.          After associating with the model, we will have to pass the article object to impressionist:

22.app/controllers/articles_controller.rb

23.  def show

24.    impressionist(@article,message:"A User has viewed your article")

25. 

26.    url = request.fullpath.to_s

27. 

28.    ip = request.remote_ip

29. 

30.    if user_signed_in? && (current_user.id != @article.user_id)

31. 

32.      Click.record(url, ip, @article.id, current_user.id.to_s)

33. 

34.    elsif !user_signed_in?

35. 

36.      Click.record(url, ip, @article.id, "anonymous")

37. 

38.    end

39. 

  end

40.          Also, we can set a filter to run impressionist for specific actions:

41.app/controllers/articles_controller.rb

42.class ArticlesController < ApplicationController

43.  before_action :set_article, only: [:show, :edit, :update, :destroy]

44.  before_filter :authenticate_user!, except: [:show, :index]

45.  impressionist :actions=>[:show]

end

46.          We are now ready to track the page views. We, however, do not have a collection for the impressions yet. So, we will generate a model for impression:

authorly$ rails g model page_impression impressionable_type:string impressionable_id:string user_id:string controller_name:string action_name:string view_name:string request_hash:string ip_address:string session_hash:string message:string referrer:string

47.          The impression model should also include the timestamps with it:

48.app/models/page_impression.rb

49.class PageImpression

50.  include Mongoid::Document

51.  include Mongoid::Timestamps::Created

52. 

53.  field :impressionable_type, :type => String

54.  field :impressionable_id, :type => String

55.  field :user_id, :type => String

56.  field :controller_name, :type => String

57.  field :action_name, :type => String

58.  field :view_name, :type => String

59.  field :request_hash, :type => String

60.  field :ip_address, :type => String

61.  field :session_hash, :type => String

62.  field :message, :type => String

63.  field :referrer, :type => String

end

64.          We just need to ensure that the model is being saved properly. So, we will navigate to the show method to see the queries:

65.Processing by ArticlesController#show as HTML

66.  Parameters: {"id"=>"the-body-of-lies"}

67.  MOPED: 127.0.0.1:27017 QUERY        database=project6_development collection=articles selector={"_slugs"=>{"$in"=>["the-body-of-lies"]}} flags=[] limit=1 skip=0 batch_size=nil fields=nil runtime: 0.8295ms

68.  MOPED: 127.0.0.1:27017 QUERY        database=project6_development collection=users selector={"$query"=>{"_id"=>BSON::ObjectId('527ce7927277750d00000000')}, "$orderby"=>{:_id=>1}} flags=[] limit=-1 skip=0 batch_size=nil fields=nil runtime: 0.5881ms

69.  MOPED: 127.0.0.1:27017 INSERT       database=project6_development collection=impressions documents=[{"_id"=>BSON::ObjectId('5283648d7277750b6a030000'), "impressionable_type"=>"Article", "impressionable_id"=>"the-body-of-lies", "controller_name"=>"articles", "action_name"=>"show", "user_id"=>BSON::ObjectId('527ce7927277750d00000000'), "request_hash"=>"871961ef69818fd7f9e0be0f510f583fd387144ef4e919ed132982144e930f8a", "session_hash"=>"457126f191ff2b6da6d92c9f6ceaa62f", "ip_address"=>"127.0.0.1", "referrer"=>"http://localhost:3000/articles", "updated_at"=>2013-11-13 11:37:49 UTC, "created_at"=>2013-11-13 11:37:49 UTC}] flags=[]

                         COMMAND      database=project6_development command={:getlasterror=>1, :w=>1} runtime: 0.7574ms

70.          In order to display the impressions, we just need to make a call to the impressionist_count method on the article object:

71.app/views/articles/show.html.erb

<%= "#{@article.impressionist_count} views so far!" %>

Objective complete – mini debriefing

This task included setting up the impressionist gem and associating it with the model and object. We generated an initializer to associate it with mongoid. In our controller, we added the impressionist method to record the impressions. We also added apage_impression model in order to save the impression-related data. The impressionist method, however, is not the best and the most scalable solution. The reason for this is every time the method counts, it starts counting from the beginning. With a large recordset of 8 to 10 million records to count, it would take more than 10 seconds just to fetch the count. A good way to save and count from our previous saved values is to use the ensureIndex option in MongoDB:

db.collection.ensureIndex

In order to add this to our model, we used the index method in mongoid. This method fires ensureIndex in MongoDB:

  index ({impressionable_type: 1, impressionable_id: 1 ,user_id: 1, controller_name: 1, action_name: 1, view_name: 1, request_hash: 1, ip_address: 1, session_hash: 1, referrer: 1, message: 1}, { name: "page_impression_index" })

If there are multiple fields to index, make sure to add a name to the index. This will keep the last counted value indexed and run the impressionist query after the value is indexed. This will also bring up the performance and decrease the count query time to less than 1 second.

Also, the default model generated does not have dates in it by default. In order to add created_at and updated_at, we added the following code to our model:

  include Mongoid::Timestamps::Created

The following screenshot shows how the impressions will be displayed on the show page:

Objective complete – mini debriefing

Writing map-reduce and aggregation to fetch and analyze data

The data is in the database now. However, we still need to read and analyze it. We will query our database in different ways and get the data based on this. We will track the total number of clicks on an article, the total number of impressions on an article, and the total number of unique impressions per day. We will use MongoDB queries and the map-reduce function to achieve this.

The map-reduce function is a combination of two procedures:

·        Map: This is a procedure that filters and sorts the records

·        Reduce: This is an operation that performs the remaining function, for example, counting

Clicks and impressions increase really quickly in huge volumes, and normal queries can be too slow; the performance could take a beating because of this. In case we need to collect our data in different ways, we can use the map-reduce function.

Engage thrusters

Let us first work with getting the data for the number of clicks and then for the number of impressions in this task:

1.    In order to get the number of clicks, we will get all the clicks associated with a particular article and count them. This is an instance method:

2.  app/models/article.rb

3.   

4.   def track_clicks_per_article

5.      clicks =  Click.where(article_id: "#{self.id}")

6.      click_count = clicks.count

7.      

  end

8.    For a logged-in user, we can display the click count on the article's show page; however, this will be displayed only to the logged-in user. The following code describes how we do this inside the show method:

9.  app/controllers/articles_controller.rb

10. 

11.def show

12. 

13.    impressionist(@article,message:"A User has viewed your article")

14. 

15.    url = request.fullpath.to_s

16. 

17.    ip = request.remote_ip

18. 

19.    if user_signed_in? && (current_user.id != @article.user_id)

20. 

21.            @clicks = @article.track_clicks_per_article

22.      Click.record(url, ip, @article.id, current_user.id.to_s)

23. 

24.    elsif !user_signed_in?

25. 

26.      Click.record(url, ip, @article.id, "anonymous")

27. 

28.    end

29. 

  end

30.          In show.html.erb, @clicks displays the number of clicks:

<% if user_signed_in? %><%= @clicks %> clicks so far!<% end %>

31.          In order to count the daily clicks, we will use the map-reduce function of MongoDB. We will first write the map function. The this.created_at and this.article_id methods will basically select these fields from the click collection. They will also initiate a count:

32.app/models/click.rb

33. 

34.def self.clicks_per_article_per_day

35.  map = %Q{

36.  function() {

37.    emit({created_at: this.created_at, article_id: this.article_id}, {count: 1});

38.   } 

39.  }

end

40.          Our reduce function will count the number of times article_id has occurred on a created_at date. This will generate an array with a daily count of clicks:

41.app/models/click.rb

42.

43.reduce = %Q{

44.   function(key, values) {

45.    var count = 0;

46.    values.forEach(function(v) {

47.      count += v['count'];

48.    });

49.    return {count: count};

50.   }

  }

51.          Finally, we will run map-reduce and return the value in a variable form:

52.app/models/click.rb

53.

54.def self.clicks_per_article_per_day

55.  map = %Q{

56.  function() {

57.    emit({created_at: this.created_at, article_id: this.article_id}, {count: 1});

58.   } 

59.  }

60. 

61. reduce = %Q{

62.   function(key, values) {

63.    var count = 0;

64.    values.forEach(function(v) {

65.      count += v['count'];

66.    });

67.    return {count: count};

68.   }

69.  }

70.  click_count = self.map_reduce(map, reduce).out(inline: true)

71.  return click_count

 end    

72.          We will fire up the console now and try to run map-reduce:

73.1.9.3-p327 :004 > @daily_clicks =  Click.clicks_per_article_per_day

74. => #<Mongoid::Contextual::MapReduce

75.  selector: {}

76.  class:    Click

77.  map:     

78.  function() {

79.    emit({created_at: this.created_at, article_id: this.article_id}, {count: 1});

80.   } 

81.      reduce:  

82.   function(key, values) {

83.    var count = 0;

84.    values.forEach(function(v) {

85.      count += v['count'];

86.    });

87.    return {count: count};

88.   }

89.  finalize:

  out:      {:inline=>true}>

90.          Since daily_clicks is an array, we will use the each method to loop over it and print the clicks on our command line:

91.  MOPED: 127.0.0.1:27017 COMMAND      database=project6_development command={:mapreduce=>"clicks", :map=>"\n  function() {\n    emit({created_at: this.created_at, article_id: this.article_id}, {count: 1}); \n   }  \n  ", :reduce=>"\n   function(key, values) {\n    var count = 0;\n    values.forEach(function(v) {\n      count += v['count'];\n    });\n    return {count: count};\n   }\n  ", :query=>{}, :out=>{:inline=>true}} runtime: 112.6887ms

92.{"_id"=>{"created_at"=>#<BSON::Undefined:0x0000000399fd88>, "article_id"=>BSON::ObjectId('528011687277750d4a000000')}, "value"=>{"count"=>22.0}}

93.{"_id"=>{"created_at"=>2013-11-13 15:18:00 UTC, "article_id"=>BSON::ObjectId('528011687277750d4a000000')}, "value"=>{"count"=>1.0}}

94.{"_id"=>{"created_at"=>2013-11-13 15:18:00 UTC, "article_id"=>BSON::ObjectId('528011687277750d4a000000')}, "value"=>{"count"=>1.0}}

95.{"_id"=>{"created_at"=>2013-11-13 23:14:43 UTC, "article_id"=>BSON::ObjectId('528011687277750d4a000000')}, "value"=>{"count"=>1.0}}

96.{"_id"=>{"created_at"=>2013-11-13 23:56:30 UTC, "article_id"=>BSON::ObjectId('528011687277750d4a000000')}, "value"=>{"count"=>1.0}}

{"_id"=>{"created_at"=>2013-11-13 23:56:30 UTC, "article_id"=>BSON::ObjectId('528011687277750d4a000000')}, "value"=>{"count"=>1.0}} 

97.          In order to track the daily impressions, we will essentially use the same functions. The only difference here is that we will define it in the page_impression model, as we have already included the impressionist models in it:

98.app/models/page_impression.rb

99. 

100.   def self.unique_impressions_per_day

101.    

102.      map = %Q{

103.    

104.      function() {

105.    

106.       emit(this['_id']['created_at'], {count: 1});

107.    

108.      }

109.    

110.     }

111.    

112.    reduce = %Q{

113.    

114.      function(key, values) {

115.    

116.       var count = 0;

117.    

118.       values.forEach(function(v) {

119.    

120.         count += v['count'];

121.    

122.       });

123.    

124.       return {count: count};

125.    

126.      }

127.    

128.     }

129.    

130.     unique_impressions = self.map_reduce(map, reduce).out(inline: true)

131.    

132.     return unique_impressions

133.    

 end

Objective complete – mini debriefing

In this task, we started by counting the number of clicks on a particular article. We created a map-reduce function to count the number of unique impressions created on a daily basis. The first part of the map-reduce function is map. It is basically a function that creates an association between a key and a value and emits the key-value pair subsequently:

map = %Q{

  function() {

    emit({created_at: this.created_at, article_id: this.article_id}, {count: 1});

   } 

  }

The map function shown in the preceding example emits the key-value pairs and its value using created_at and similar other pairs. The this attribute in this.created refers to the document on which map-reduce is supposed to run; in this case, PageImpression. So, we see that the this function is exactly the same as the self function. After that, the reduce function basically reads the key and value and counts the occurrences of the key-value pairs to return the count. We then initialize the count at zero (0) and increment it as and when we hit the identical values:

 reduce = %Q{

   function(key, values) {

    var count = 0;

    values.forEach(function(v) {

      count += v['count'];

    });

    return {count: count};

   }

  }

The map-reduce function is a practice used very specifically for extremely large datasets. For relatively smaller datasets, it might be an overkill. Also, map-reduce generates an array of objects as a result of this. We have to loop over this array and extract the value of the click attributes from it. We also use map-reduce to find the unique impressions per day. Some of the other use cases include a data clustering, distributed data processing, and search based on specific patters in the use cases.

The following screenshot displays the count of both clicks and impressions on the article's show page:

Objective complete – mini debriefing

Creating a dashboard to display clicks and impression values

Until now, we have created various ways in the previous tasks to record, calculate, and analyze the data. As a result, we now have the data and also the count of clicks as well as impressions, and we need a dashboard to display these values. In this task, we will create a dashboard for this purpose. We have to create a dashboard controller and an admin namespace similar to the one we created in our previous project.

Engage thrusters

In the following steps, we will add an admin dashboard to the application:

1.    In dashboard_controller, we will call all the articles:

2.  app/controllers/admin/dashboard_controller.rb

3.  class Admin::DashboardController < ApplicationController

4.    before_filter :authenticate_user!

5.   

6.    def index

7.      @articles = Article.all

8.    end

end

9.    We will now loop over these articles and call on the methods to calculate clicks on each article:

10.app/views/admin/dashboard/index.html.erb

11.<h3>Clicks and Impressions Per article</h3>

12.<table class="table"><thead><tr><th>Article</th><th>Cicks</th></tr></thead>

13. 

14.<tbody><% @articles.each do |article| %><tr><td><%=link_to article.title, article %></td><td><%= article.track_clicks_per_article %></td></tr><% end %></tbody>

</table>

15.          We will also count the number of impressions and display them in the table:

16.app/views/admin/dashboard/index.html.erb

17.<h3>Clicks and Impressions Per article</h3>

18.<table class="table"><thead><tr><th>Article</th><th>Cicks</th><th>Impressions</th></tr></thead>

19. 

20.<tbody><% @articles.each do |article| %><tr><td><%=link_to article.title, article %></td><td><%= article.track_clicks_per_article %></td><td><%= article.impressionist_count %></td></tr><% end %></tbody>

</table>

Objective complete – mini debriefing

We have created a table to display all the articles and the corresponding values of clicks and impressions on them. This is one part of the reporting structure that we're going to provide to the content creators. In the next tasks, we're going to plot our data and make better looking reports for our system.

Objective complete – mini debriefing

Creating a line graph of the daily click activity

For content creators, "clicks per day" is a very important metric. They love to see the interaction and engagement happening on a day-to-day basis. We can plot the click data for the authors of the articles using the morris.js charts where morris.js is a library for plotting the data as line charts, bar charts, and donut charts. This is the reporting part of our analytics dashboard.

Engage thrusters

We will now plot the data that we have collected and analyzed in our previous tasks:

1.    The morris.js library comes packaged as a gem. It also depends on an SVG that renders a canvas library called raphael js.

2.  Gemfile

3.  gem 'morrisjs-rails'

gem 'raphael-rails'

4.    We will add this to the Gemfile and run bundle.

5.    We will then define the JavaScript in our application.js file. We have to ensure that these lines are placed before require turbolinks and require_tree:

6.  app/assets/javascripts/application.js

7.  //= require raphael

8.  //= require morris

9.  //= require turbolinks

//= require_tree .

10.          Also, we will add the morris.js style sheet to our asset pipeline:

11.app/assets/stylesheets/application.css

12.    *= require morris

    *= require_tree .

13.          In order to feed data to the JavaScript charts, we will have to prepare our data in the JSON format. To do this, first call the clicks_per_article_per_day method. As you can see, we have created a new method called clicks for this:

14.app/controllers/admin/dashboard_controller.rb

15.  def clicks

16.    @daily_clicks = Click.clicks_per_article_per_day

  end

17.          We need the count of clicks and the date in order to plot this graph. Hence, we will get the results of the clicks_per_article_per_day method and generate a json hash for morris.js to read. For this, we will first create a model class method that loops over the data to generate a hash:

18.app/models/click.rb

19.def self.get_click_data

20. 

21.   daily_clicks = self.clicks_per_article_per_day

22. 

23.   click_data = []

24. 

25.   daily_clicks.each do |d|

26. 

27.    id = d["_id"]

28. 

29.    daily_clicks = d["value"]

30. 

31.    date = d["_id"]["created_at"]

32. 

33.    clicks = daily_clicks["count"]

34. 

35.    click_data <<  {:date => date.to_i, :clicks => clicks.to_i}

36. 

37.   end

38. 

39.   return click_data

40. 

41. end

42. 

43.app/controllers/admin/dashboard_controller.rb

44.  def clicks

45. 

46.    @click_data=  Click.get_click_data

47. 

48.    respond_to do |format|

49. 

50.     format.json { render json: @click_data }

51. 

52.     end

53. 

  end

54.          Despite the availability of data in JSON, we need a way to access it. So, we will write a route to access the data using the this method.

55.config/routes.rb

56.  namespace :admin do

57.   get '', to: 'dashboard#index', as: '/'

58.   get "dashboard/clicks"

  end

59.          In our app/views/admin/dashboard/index.html.erb file, we will initiate the script for clicks. The Morris.Line function is a function to create a line graph. We will keep the date as the key for the x axis and clicks as the key for y axis:

60.app/views/admin/dashboard/index.html.erb

61.<script>

62.var url = "/admin/dashboard/clicks.json"

63.var click_json= $.getJSON(url, null, function(data) {

64.var get_click_data = click_json.responseText;

65. 

66.new Morris.Line({

67.  element: 'click_chart',

68.  data: $.parseJSON((get_click_data)),

69.  xkey: 'date',

70.  ykeys: ['clicks'],

71.  labels: ['Clicks']

72.});

73.done();

74. 

75.});

</script>

76.          Lastly, we will render this in a div tag. Make sure <div id> and the element name in the Morris.Line definition are the same:

77.app/views/admin/dashboard/index.html.erb

78.<h3>Clicks Per Day</h3>

<div id="click_chart" style="height: 250px;"></div>

Objective complete – mini debriefing

The previous task included the creation of JSON from the data we already have, and morris.js accepts this data in a particular format. We had to extract the data from our map-reduce function and format it according to the format accepted by morris.js. Please see the following screenshot for the acceptable format:

Objective complete – mini debriefing

You will notice that date is in the integer format because or to get the date, we did the following in our map function:

  map = %Q{

  function() {

    emit({created_at: this.created_at.getDate()}, {count: 1});

   } 

  }

The getDate() function will return the date in the float format. In order to render it on the frontend, we will convert the float datatype to the integer datatype:

@click_data <<  {:date => date.to_i, :clicks => clicks.to_i}

This method generates json, which can be directly read by visiting the /admin/dashboard/clicks.json URL. To display the clicks, we made a call on the clicks.json data by directly calling this URL:

var url = "/admin/dashboard/clicks.json"

To extract the data from json, we will use the function(data) jQuery method and store the data text in get_click_data:

var click_json= $.getJSON(url, null, function(data) {

var get_click_data = click_json.responseText;

click_json.responseText;

Finally, we passed the data to the Morris.Line method to generate the line graph. The morris.js line graph accepts xkey and ykeys as axes and labels to represent data at each data point. You can set colors, customize the text, set the line width, and set data formats and units for each datapoint:

new Morris.Line({

  element: 'click_chart',

  data: $.parseJSON((get_click_data)),

  xkey: 'date',

  ykeys: ['clicks'],

  labels: ['Clicks']

});

});

});

Insert

Creating a bar graph of the daily visit activity

In the previous task, we already learned how to display the daily click data on a line graph. In this task, we will use bar charts to display the daily visit activity of the impression data. We will also create json from the impression data we have and feed it to themorris.js method to generate our graph.

Engage thrusters

We will now use the following steps to create a bar chart of the impression data:

1.    In dashboard_controller, we will create a method called impressions to construct the impressions JSON:

2.  app/controllers/admin/dashboard_controller.rb

3.    def impressions

4.      @daily_impressions =  Article.impressions_per_article_per_day

  end

5.    In the article model, we will edit our map method and change the format of created_at to getDate():

6.  app/models/article.rb

7.  def self.impressions_per_article_per_day

8.    map = %Q{

9.    function() {

10.   emit({created_at: this.created_at.getDate()}, {count: 1});

11.   } 

12.  }

end

13.          In the impressions method, we will construct JSON and render it:

14.app/controllers/admin/dashboard_controller.rb 

15.def impressions

16.    daily_impressions =  Article.impressions_per_article_per_day

17.    @impressions_data = []

18. 

19.    daily_impressions.each do |d|

20.      id = d["_id"]

21.      daily_impressions = d["value"]

22.      date = d["id"]["created_at"]

23.      impressions = daily_impressions["count"]

24.      @impressions_data <<  {:date => date.to_i, :impressions => impressions.to_i}

25.    end

26.    respond_to do |format|  format.json { render json: @impressions_data }

27.    end

  end

28.          We will tie this to a route in order to generate the URL:

29.config/routes.rb

30.  namespace :admin do

31.   get '', to: 'dashboard#index', as: '/'

32.   get "dashboard/clicks"

33.   get "dashboard/impressions"

  end

34.          The function to generate a bar graph is quite similar to the one for a line graph. The axis key definitions are also the same:

35.app/views/admin/dashboard/index.html.erb

36.<script>

37.var url = "/admin/dashboard/impressions.json"

38.var json=json= $.getJSON(url, null, function(data) {

39.var get_impression_data = json.responseText;json.responseText;

40. 

41.new Morris.Bar({

42.  element: 'impressions_chart',

43.  data: $.parseJSON((get_impression_data)),

44.  xkey: 'date',

45.  ykeys: ['impressions'],

46.  labels: ['Impressions']

47.});

48.});

</script>

49.          With the JavaScript method ready to create a bar chart, we just need to render our graph:

50.<h3>Impressions Per Day</h3>

<div id="impressions_chart" style="height: 250px;"></div>

Objective complete – mini debriefing

In this task, we used a bar chart to represent the impression data that we collected in our previous tasks. We used the same method as clicks to generate JSON. We used Morris.Bar to generate the bar graph. We used xkey and ykeys to represent the x and y axes. We used labels to represent data against each bar. Some of the other options provided in the morris.js bar graph are as follows:

·        You can enable or disable grid lines by setting the grid option to true or false

·        You can enable or disable the display of axes by setting the axes option to true or false

·        To manipulate the text properties of the grid you have gridTextColor, gridTextSize, and gridTextWeight

·        You have a stacked option—a Boolean value—to allow bars to be vertically stacked

·        You have a BarColors option, which is an array to set the colors of the bars

·        You have a HideHower option to show or hide the data on Hower

·        You also have a HowerCallback option that allows additional functions to generate custom howers

The following screenshot shows a bar graph:

Objective complete – mini debriefing

Creating a demographic-based donut chart

We have already plotted our click and impression data as both a line and bar graph. However, we also have to track the demographics of our user visits. One of the parameters for demographics is the location of the visitor. As a part of our requests, we can easily track the country and city of the user based on the user's IP address. We will add these methods for our tracking mechanisms and generate a donut chart to visualize our visitor's locations.

Prepare for lift off

In order to proceed with this section, we will be using a Geocoder to track the location of the visitor. A Geocoder is a very comprehensive library to not only locate the user and get the coordinates, but also run the Geospatial queries; for example, to find nearby users. For this, we will add the geocoder gem to Gemfile and run bundle install:

Gemfile

gem 'geocoder'

Engage thrusters

The following steps include the methods to generate and represent the demographic data of our visitors:

1.    In order to get the demographics, we need to get the country data. In order to record the country data, we will add two fields to our click collection:

2.  app/models/click.rb

3.   field :country, type: String

4.   

  field :city, type:String

5.    We will add request.location.country and request.location.city to our show method inside articles_controller.rb. We will also save these as part of our click objects:

6.  app/controllers/articles_controller.rb

7.   def show

8.      @country = request.location.country

9.      @city = request.location.city

10. 

11.    click.country = @country

12.    click.city = @city

  end

13.          So, our final show method will look like the following code:

14.app/controllers/articles_controller.rb

15.def show

16.    @clicks = @article.track_clicks_per_article

17.    impressionist(@article,message:"A User has viewed your article")

18.    @url = request.fullpath.to_s

19.    @ip = request.remote_ip

20.    @country = request.location.country

21.    @city = request.location.city

22. 

23.    url = request.fullpath.to_s

24. 

25.    ip = request.remote_ip

26. 

27.    country = request.location.country

28. 

29.    city = request.location.city

30. 

31.    if user_signed_in? && (current_user.id != @article.user_id)

32. 

33.      Click.record(url, ip, country, city, @article.id, current_user.id.to_s)

34. 

35.    elsif !user_signed_in?

36. 

37.      Click.record(url, ip, country, city, @article.id, "anonymous")

38. 

39.    end

40. 

  end

41.          Also, we will modify our record_data concern to save these values to the database:

42.app/models/concerns/record_data.rb

43. 

44.module RecordData

45. 

46.  extend ActiveSupport::Concern

47. 

48.  included do

49. 

50.    def self.record(url, ip, country, city, article_id, user_id)

51. 

52.      self.create!(url: url, ip: ip, country: country, article_id: article_id, user_id: user_id)

53. 

54.      end

55. 

56.  end

57. 

end

58.          Once we have the mechanism set up to record the data, we will write a map-reduce function to count the number of visits from a particular country:

59.app/models/click.rb

60. def self.clicks_per_country

61.  map = %Q{

62.  function() {

63.    emit({country: this.country}, {count: 1});

64.   } 

65.  }

66. 

67. reduce = %Q{

68.   function(key, values) {

69.    var count = 0;

70.    values.forEach(function(v) {

71.      count += v['count'];

72.    });

73.    return {count: count};

74.   }

75.  }

76.  unique_clicks = self.map_reduce(map, reduce).out(inline: true)

77.  return unique_clicks

 end

78.          In dashboard_controller, we will add a demographics method to generate JSON for our recorded data:

79.  def demographics

80.    demographics =  Click.clicks_per_country

81.    @impressions_data = []

82. 

83.    demographics.each do |d|

84.      id = d["_id"]

85.      demographics = d["value"]

86.      country = id["country"]

87.      visits = demographics["count"]

88.      @impressions_data <<  {:country => country, :visits => visits.to_i}

89.     

90.    end

91.    respond_to do |format|  format.json { render json: @impressions_data }

92.    end

  end

93.          We will add a route for this to generate the URL:

94.config/routes.rb

95.  namespace :admin do

96.   get '', to: 'dashboard#index', as: '/'

97.   get "dashboard/clicks"

98.   get "dashboard/impressions"

99.   get "dashboard/demographics"

  end

100.     We will initialize a donut chart to display this data:

101.   app/views/admin/dashboard/index.html.erb

102.   var url = "/admin/dashboard/demographics.json"

103.   var demographic_json=demographic_json= $.getJSON(url, null, function(data) {

104.   var get_demographic_data = demographic_json.responseText;

105.    

106.   Morris.Donut({

107.     element: 'demographic',

108.     data: get_demographic_data

});

109.     Lastly, display the chart in div:

110.   app/views/admin/dashboard/index.html.erb

111.   <h3>Demographics</h3>

<div id="demographic" style="height: 250px;"></div>

Objective complete – mini debriefing

We first used the request.location.country and request.location.city methods to look for the country and city based on the IP address of the visitor. These methods were available as soon as we bundled the geocoder gem in our applications. We wrote a map-reducefunction to count the number of visits from a particular country. The map function aggregated all the impressions based on the country and the reduce function in this case counted the size of each aggregation.

A donut chart is very similar to a pie chart. In our case, it represents the break up of visits from a particular country. We created a method called demographic in our dashboard controller. We generated a json hash that included demographics that were consumed by the morris.js donut chart method. Donut charts do not have axes. The data here is represented as a label and a value. It also accepts the colors and formatter parameters. Colors contain the HTML color code for the donut segment.

The following is how a donut chart looks when generated using morris.js:

Objective complete – mini debriefing

Mission accomplished

We have created a fully functional analytics dashboard in this project. As mentioned earlier, the analytics dashboard has three main parts:

·        Recording: We created a mechanism to track clicks, visits or impressions, and demographics of the user

·        Analyzing: We wrote various queries and map-reduce methods to count visits, clicks, unique visits, and visits from each country

·        Reporting: We created tables and charts of different types in order to represent and visualize the data we recorded and analyzed

Hotshot challenges

In an analytics dashboard, the possibilities are endless as to how you can imagine the data. We can improve our dashboard with some exercises:

·        Write map-reduce to make a leaderboard for articles and display the top 10 articles

·        Create localized slugs for our articles

·        Use ensureindex to create an index and improve the performance of the impressionist query

·        Display the article names on the line and bar charts

·        Create an area chart to compare the activities of the top three articles by a particular user