Serving Up OpenNTI “Deconstructed”

One modern culinary trend that often elicits a colourful spectrum of opinions, ranging from admiration of artistic expression on one end, to opinions of borderline laziness on the other end, is “deconstructed cuisine”.  A deconstructed dish contains the same classical ingredients found in the original dish, but differs in its execution, with individual ingredients getting prepared separately, and reliance on plating and presentation to bring all the individual components together.

In response to the my blog posts on setting up telemetry streaming from Juniper devices and collecting it using an open-source tool called OpenNTI (Part 1Part 2 and Part 3), I was recently asked how to deploy the constituent components of OpenNTI (ie. Grafana, InfluxDB, FluentD and Telegraf) separately, in “deconstructed” mode and in the absence of Docker containers.  This blog post aims to describe, step-by-step, how to deploy OpenNTI “Deconstructed” and still use it to collect telemetry data from Juniper devices.

For the examples shown in this blog post, the various components were installed on a Ubuntu 16.04 server, but note that they can be installed on any Linux or Mac machine.

Installing InfluxDB

Installing InfluxDB is a fairly straightforward process.  The first step is to retrieve the Debian software package file using “wget”:

root@ubuntu:~# wget https://dl.influxdata.com/influxdb/releases/influxdb_1.3.6_amd64.deb
--2017-10-29 15:32:02--  https://dl.influxdata.com/influxdb/releases/influxdb_1.3.6_amd64.deb
Resolving dl.influxdata.com (dl.influxdata.com)... 54.192.142.192, 54.192.142.208, 54.192.142.227, ...
Connecting to dl.influxdata.com (dl.influxdata.com)|54.192.142.192|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 18742318 (18M) [application/octet-stream]
Saving to: ‘influxdb_1.3.6_amd64.deb’

influxdb_1.3.6_amd64.deb      100%[=================================================>]  17.87M  5.86MB/s    in 3.0s    

2017-10-29 15:32:05 (5.86 MB/s) - ‘influxdb_1.3.6_amd64.deb’ saved [18742318/18742318]

Once the package is retrieved, install InfluxDB using the package manager:

root@ubuntu:~# sudo dpkg -i influxdb_1.3.6_amd64.deb
Selecting previously unselected package influxdb.
(Reading database ... 97932 files and directories currently installed.)
Preparing to unpack influxdb_1.3.6_amd64.deb ...
Unpacking influxdb (1.3.6-1) ...
Setting up influxdb (1.3.6-1) ...
Created symlink from /etc/systemd/system/influxd.service to /lib/systemd/system/influxdb.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/influxdb.service to /lib/systemd/system/influxdb.service.

Once installed, start the InfluxDB service and verify that it is up and running:

root@ubuntu:~# sudo service influxdb start
root@ubuntu:~# service influxdb status
 influxdb.service - InfluxDB is an open-source, distributed, time series database
   Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2017-10-29 15:50:58 PDT; 5s ago
     Docs: https://docs.influxdata.com/influxdb/
 Main PID: 29508 (influxd)
    Tasks: 6
   Memory: 4.8M
      CPU: 60ms
   CGroup: /system.slice/influxdb.service
           └─29508 /usr/bin/influxd -config /etc/influxdb/influxdb.conf

The FluentD plugin, which we will install later in this blog post, expects that a database named “juniper” is already created in InfluxDB.  This database can be created using the InfluxDB CLI client:

root@ubuntu:~# influx
Connected to http://localhost:8086 version 1.3.6
InfluxDB shell version: 1.3.6
> create database juniper
> show databases
name: databases
name
----
_internal
juniper
> quit
root@ubuntu:~# 

Installing FluentD

Within OpenNTI, FluentD is used as the open-source collector to ingest Juniper Native Streaming telemetry data.  To get FluentD installed outside the umbrella of OpenNTI, there are some preliminary steps to be followed as well some prerequisite packages that need to be installed.

First, we need to check the number of maximum of file descriptors currently configured for the system, and increase them if they are insufficient.  To do this, we first issue the “ulimit -n” command and see the output.  If the console shows an output of 1024, this is insufficient, and we have to increase the number (typically to 65536 for large deployments) by editing the “/etc/security/limits.conf” file, adding the following lines to the end of the file and rebooting your machine:

root@ubuntu:~# more /etc/security/limits.conf | grep 65536
root soft nofile 65536
root hard nofile 65536
* soft nofile 65536
* hard nofile 65536
root@ubuntu:~# 

Next, install some preliminary packages required by FluentD, such as gcc, make and ruby:

sudo apt-get install gcc
sudo apt-get install make
sudo apt-get install ruby
sudo apt-get install ruby-dev

Use the Ruby package manager to fetch and install the FluentD Ruby gem:

root@ubuntu:~# gem install fluentd -v "~> 0.12.0" --no-ri --no-rdoc
Fetching: msgpack-1.1.0.gem (100%)
Building native extensions. This could take a while...

[... CONTENT OMITTED FOR BREVITY ...]

Fetching: fluentd-0.12.40.gem (100%)
Successfully installed fluentd-0.12.40
10 gems installed
root@ubuntu:~#

After installing the Ruby gem, auto-create the FluentD configuration file (“fluent.conf“) in directory “/etc“:

root@ubuntu:~# cd /etc
root@ubuntu:/etc# fluentd --setup ./fluent
Installed ./fluent/fluent.conf.
root@ubuntu:/etc#

Replace the entire contents of the “fluent.conf” configuration file with the following:

#############
## INPUT ##
#############

<source>
   @type udp
   tag jnpr.jvision
   format juniper_jti
   port 40000
   bind 0.0.0.0
</source>


##############
## OUTPUT ##
##############

<match jnpr.**>
   type copy

   <store>
      type influxdb
      host localhost
      port 8086
      dbname juniper
      user juniper
      password juniper
      value_keys ["value"]
      buffer_type memory
      # buffer_chunk_limit 524288 # 512 * 1024
      # buffer_queue_limit 1024
      flush_interval 2
      # retry_limit 17
      # retry_wait 1.0
      # num_threads 1
   </store>
</match>

NOTE: In the configuration above, note that the port number (eg. 40000) must align with the port number used in the “set services analytics streaming-server [SERVER] remote-port [PORT]” command on the router.

At this point, we are now ready to install the “juniper-telemetry” FluentD plugin using the Ruby package manager:

root@ubuntu:/etc/fluent# gem install fluent-plugin-juniper-telemetry
Fetching: concurrent-ruby-1.0.5.gem (100%)
Successfully installed concurrent-ruby-1.0.5

[... CONTENT OMITTED FOR BREVITY ...]

Fetching: protobuf-3.8.1.gem (100%)
Successfully installed protobuf-3.8.1
Fetching: fluent-plugin-juniper-telemetry-0.3.0.gem (100%)
Successfully installed fluent-plugin-juniper-telemetry-0.3.0
Parsing documentation for concurrent-ruby-1.0.5
Installing ri documentation for concurrent-ruby-1.0.5

[... CONTENT OMITTED FOR BREVITY ...]

Parsing documentation for protobuf-3.8.1
Installing ri documentation for protobuf-3.8.1
Parsing documentation for fluent-plugin-juniper-telemetry-0.3.0
Installing ri documentation for fluent-plugin-juniper-telemetry-0.3.0
Done installing documentation for concurrent-ruby, i18n, activesupport, middleware, thor, protobuf, fluent-plugin-juniper-telemetry after 15 seconds
7 gems installed
root@ubuntu:/etc/fluent#

By default, outside of the OpenNTI umbrella, the “juniper-telemetry” plugin outputs its data to stdout (ie. to the console).  In order to get this plugin to output and persist its data to InfluxDB, we need to install a special InfluxDB plugin.  To get this installed, copy the contents of the following ruby script: “https://raw.githubusercontent.com/Juniper/open-nti/master/plugins/input-jti/plugins/out_influxdb.rb“, and paste it into a file called “out_influxdb.rb” in directory “/etc/fluent/plugin“.

Finally, the last step is to run FluentD as a background process:

root@ubuntu:/etc# fluentd -c /etc/fluent/fluent.conf -vv &
[1] 3780
root@ubuntu:/etc# 2017-10-30 10:42:30 -0700 [info]: fluent/supervisor.rb:471:read_config: reading config file path="./fluent/fluent.conf"
[... CONTENT OMITTED FOR BREVITY ...]
2017-10-30 10:42:31 -0700 [trace]: fluent/plugin.rb:122:register_impl: registered output plugin 'influxdb'
2017-10-30 10:42:31 -0700 [info]: fluent/engine.rb:126:block in configure: gem 'fluent-plugin-juniper-telemetry' version '0.3.0'
2017-10-30 10:42:31 -0700 [info]: fluent/engine.rb:126:block in configure: gem 'fluentd' version '0.12.40'
2017-10-30 10:42:31 -0700 [info]: fluent/agent.rb:129:add_match: adding match pattern="jnpr.**" type="copy"
2017-10-30 10:42:31 -0700 [debug]: plugin/out_copy.rb:44:block in configure: adding store type="influxdb"
2017-10-30 10:42:31 -0700 [info]: fluent/root_agent.rb:152:add_source: adding source type="udp"
2017-10-30 10:42:31 -0700 [info]: fluent/engine.rb:133:configure: using configuration file: <ROOT>
  <source>
    @type udp
    tag jnpr.jvision
    format juniper_jti
    port 40000
    bind 0.0.0.0
  </source>
  <match jnpr.**>
    type copy
    <store>
      type influxdb
      host localhost
      port 8086
      dbname juniper
      user juniper
      password xxxxxx
      value_keys ["value"]
      buffer_type memory
      flush_interval 2
    </store>
  </match>
</ROOT>
2017-10-30 10:42:31 -0700 [info]: plugin/in_udp.rb:37:listen: listening udp socket on 0.0.0.0:40000

NOTE: To stop FluentD, do a “ps -ef | grep fluent“, find the associated PID for the process and kill it with “kill -HUP <PID>“.

Installing Telegraf

Within OpenNTI, Telegraf is used as the open-source collector to ingest Juniper gRPC/OpenConfig Streaming telemetry data.  Telegraf is built using the Go programming language, and because we are going to download and install Telegraf from source, we need to first install Go as a prerequisite.

To install Go, we first download the latest stable version (1.10 as of this writing) using “wget”:

sudo wget https://dl.google.com/go/go1.10.linux-amd64.tar.gz

Next, we extract the archive into “/usr/local“, creating a Go tree in “/usr/local/go“:

root@ubuntu:~# tar -C /usr/local -xzf go1.10.linux-amd64.tar.gz
root@ubuntu:~# ls
go  go1.10.linux-amd64.tar.gz

Next, we need to add Go’s root location to the PATH environment variable, and test out the Go installation with a quick version test:

root@ubuntu:~# export PATH=$PATH:/usr/local/go/bin
root@ubuntu:~# go version
go version go1.10 linux/amd64
root@ubuntu:~#

After installing Go, we have to setup a Go workspace.  From your $HOME directory, setup the following subdirectories, and setup the GOPATH environment variable:

mkdir go
mkdir go/src
mkdir go/bin
mkdir go/pkg
mkdir go/src/github.com
mkdir go/src/github.com/influxdata
export GOPATH=$HOME/go

Next, download and install Google gRPC:

root@ubuntu:/etc# go get -u -v google.golang.org/grpc
Fetching https://google.golang.org/grpc?go-get=1
Parsing meta tags from https://google.golang.org/grpc?go-get=1 (status code 200)
get "google.golang.org/grpc": found meta tag main.metaImport{Prefix:"google.golang.org/grpc", VCS:"git", RepoRoot:"https://github.com/grpc/grpc-go"} at https://google.golang.org/grpc?go-get=1
google.golang.org/grpc (download)
github.com/golang/protobuf (download)
Fetching https://golang.org/x/net/context?go-get=1

[... CONTENT OMITTED FOR BREVITY ...]

google.golang.org/grpc/transport
google.golang.org/grpc
root@ubuntu:/etc#

At this point, we are now ready to install the “openconfig_telemetry” plugin for Telegraf from source.  The first step is to ‘cd’ to the “$HOME/go/src/github.com/influxdata” directory and grab the “openconfig_telemetry” Telegraf branch from GitHub:

root@ubuntu:~# cd $HOME/go/src/github.com/influxdata
root@ubuntu:~/go/src/github.com/influxdata# git clone https://github.com/ajhai/telegraf.git
Cloning into 'telegraf'...
remote: Counting objects: 16667, done.
remote: Compressing objects: 100% (2/2), done.
remote: Total 16667 (delta 0), reused 0 (delta 0), pack-reused 16665
Receiving objects: 100% (16667/16667), 12.04 MiB | 4.66 MiB/s, done.
Resolving deltas: 100% (8602/8602), done.
Checking connectivity... done.
root@ubuntu:~/go/src/github.com/influxdata# 

After downloading the Telegraf source, we then build the source using ‘make‘:

root@ubuntu:~/go/src/github.com/influxdata# cd $HOME/go/src/github.com/influxdata/telegraf
root@ubuntu:~/go/src/github.com/influxdata/telegraf# make
go get github.com/sparrc/gdm
gdm restore
======= Go Dependency Manager =======
= working dir:  /root/go/src/github.com/influxdata/telegraf
= checkout dir: /root/go/src
= GOPATH:       /root/go
=====================================

[... CONTENT OMITTED FOR BREVITY ...]

> Restoring /root/go/src/gopkg.in/yaml.v2 to a83829b6f1293c91addabc89d0571c246397bbf4
go install -ldflags \
 "-X main.version=1.0.0-beta1-410-g7eb8b85 -X main.commit=7eb8b85 -X main.branch=master" ./...
root@ubuntu:~/go/src/github.com/influxdata/telegraf# 

The Telegraf executable will be in “$HOME/go/src/github.com/influxdata/telegraf“. Add this path to the PATH environment variable:

export PATH=$PATH:$HOME/go/src/github.com/influxdata/telegraf

Before we startup Telegraf, we first have to generate a Telegraf configuration file and configure the required input and output plugins we will be using.  To generate a generic telegraf config file containing all of the available input and output plugins, issue the following command from within the “$HOME/go/src/github.com/influxdata/telegraf” directory:

telegraf config > telegraf.conf

To generate an openconfig-telemetry-specific Telegraf configuration file, specify the required input and output plugins, as shown in the following command:

telegraf --input-filter jti_openconfig_telemetry --output-filter influxdb config > telegraf.conf

Once the Telegraf configuration file has been created, we need to edit the “OUTPUT PLUGINS” and “SERVICE INPUT PLUGINS” sections with the pertinent InfluxDB and gRPC telemetry parameters, respectively.

The “OUTPUT PLUGINS” sections should look something like this:

###############################################################################
#                               OUTPUT PLUGINS                                #
###############################################################################

# Configuration for influxdb server to send metrics to
[[outputs.influxdb]]
 ## The full HTTP or UDP endpoint URL for your InfluxDB instance.
 ## Multiple urls can be specified as part of the same cluster,
 ## this means that only ONE of the urls will be written to each interval.
 # urls = ["udp://localhost:8089"] # UDP endpoint example
 urls = ["http://localhost:8086"] # required
 ## The target database for metrics (telegraf will create it if not exists).
 database = "juniper" # required

## Retention policy to write to. Empty string writes to the default rp.
 retention_policy = ""
 ## Write consistency (clusters only), can be: "any", "one", "quorum", "all"
 write_consistency = "any"

## Write timeout (for the InfluxDB client), formatted as a string.
 ## If not provided, will default to 5s. 0s means no timeout (not recommended).
 timeout = "5s"
 username = "juniper"
 password = "juniper""
 ## Set the user agent for HTTP POSTs (can be useful for log differentiation)
 # user_agent = "telegraf"
 ## Set UDP payload size, defaults to InfluxDB UDP Client default (512 bytes)
 # udp_payload = 512

The “SERVICE INPUT PLUGINS” sections should look something like the sample below.  Recall that in the case of gRPC/OpenConfig streaming, the router functions as the gRPC server that listens on port 50051, so the “server” parameter uses the management IP address of the router from which telemetry data is being streamed.

###############################################################################
#                            INPUT PLUGINS                                    #
###############################################################################

# Read OpenConfig Telemetry from listed sensors

[[inputs.jti_openconfig_telemetry]]

  ## List of device addresses to collect telemetry from.
  servers = ["10.102.183.59:50051"]

  ## Frequency to get data, eg. "5s" or "5000ms".
  sample_frequency = "5000ms"

  ## Sensors to subscribe for
  ## A identifier for each sensor can be provided in path by separating with space
  ## Else sensor path will be used as identifier
  ## When identifier is used, we can provide a list of space separated sensors.
  ## A single subscription will be created with all these sensors and data will
  ## be saved to measurement with this identifier name
  ## We allow specifying sensor group level reporting rate. To do this, specify the
  ## reporting rate in Duration at the beginning of sensor paths / collection
  ## name. For entries without reporting rate, we use configured sample frequency
  sensors = [
   "/junos/system/linecard/interface/",
   "10000ms /junos/system/linecard/cpu/memory/"
  ]

  ## Each data format has it's own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  ##data_format = "influx"

Finally, start Telegraf (using the freshly modified configuration file) as a background process using the following command:

telegraf --config $HOME/go/src/github.com/influxdata/telegraf/telegraf.conf &

NOTE: To stop Telegraf, do a “ps -ef | grep telegraf“, find the associated PID for the process and kill it with “kill -HUP <PID>“.

Installing Grafana

Within OpenNTI, Grafana is used as a data visualization or dashboarding tool for displaying the telemetry data collected from Juniper devices.  Installing Grafana is a fairly straightforward process that first requires the retrieval of the Debian software package file using “wget”:

root@ubuntu:~# wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_4.5.2_amd64.deb
--2017-10-30 18:58:03--  https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_4.5.2_amd64.deb
Resolving s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)... 52.218.160.36
Connecting to s3-us-west-2.amazonaws.com (s3-us-west-2.amazonaws.com)|52.218.160.36|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 50586862 (48M) [application/x-debian-package]
Saving to: ‘grafana_4.5.2_amd64.deb’
grafana_4.5.2_amd64.deb         100%[======================================================>]  48.24M  14.8MB/s    in 3.7s    
2017-10-30 18:58:08 (13.0 MB/s) - ‘grafana_4.5.2_amd64.deb’ saved [50586862/50586862]

Once the package is retrieved, install Grafana using the package manager:

root@ubuntu:~# sudo dpkg -i grafana_4.5.2_amd64.deb
Selecting previously unselected package grafana.
(Reading database ... 142426 files and directories currently installed.)
Preparing to unpack grafana_4.5.2_amd64.deb ...
Unpacking grafana (4.5.2) ...
Setting up grafana (4.5.2) ...
Adding system user `grafana' (UID 112) ...
Adding new user `grafana' (UID 112) with group `grafana' ...
Not creating home directory `/usr/share/grafana'.
### NOT starting on installation, please execute the following statements to configure grafana to start automatically using systemd
 sudo /bin/systemctl daemon-reload
 sudo /bin/systemctl enable grafana-server
### You can start grafana-server by executing
 sudo /bin/systemctl start grafana-server
Processing triggers for systemd (229-4ubuntu10) ...
Processing triggers for ureadahead (0.100.0-19) ...
root@ubuntu:~# 

Finally, start the Grafana server using the “systemctl” service manager:

sudo /bin/systemctl start grafana-server

Verify that the Grafana web UI is up and running by pointing your browser to “http://<server_ip>:3000” and logging in with the default login/password of “admin”/”admin”.

In Conclusion

At this point, all the necessary ingredients (InfluxDB, FluentD, Telegraf and Grafana) have been prepared and served independently of OpenNTI and in the absence of Docker containers … ie. “OpenNTI Deconstructed“.  Bon Appetit!

Leave a Reply