DELL iDRAC9 Power Edge Servers User Guide

: June 15, 2024
: Dell

Table of Contents

DELL iDRAC9 Power Edge Servers
Product Information
Information
Executive summary
Prerequisites
Reference Tools Architecture
Workflows
Troubleshooting Tips
Technical support and resources
References
Read User Manual Online (PDF format)
Download This Manual (PDF format)

DELL iDRAC9 Power Edge Servers

DELL-iDRAC9-Power-Edge-Servers-product

Product Information

Specifications

iDRAC9 version: 4.00
Datacenter license
Telemetry Reference Tools
Integration with external big databases: Elasticsearch, Prometheus, Timescale, and InfluxDB
Visualization tools: Kibana, Grafana, Splunk Analytics
Prerequisites:
- iDRAC firmware: 4.00.00.00 or later
- Datacenter license
- Windows or Linux system (e.g., Ubuntu OS version 20.04.1 LTS or
  later) with at least 16 GB RAM and 250GB available disk space
- Docker and Docker-Compose tools 2.2.0 or higher
- Remote Desktop connection to the Linux system
- Basic knowledge of Splunk and Open-Source tools: ELK stack (Elastic Search and Kibana), InfluxDB, Prometheus, and Grafana

FAQ

Q: What are the prerequisites for using iDRAC9 Telemetry Reference Tools?

A: The prerequisites for using iDRAC9 Telemetry Reference Tools are:
- iDRAC firmware: 4.00.00.00 or later
- Datacenter license
- Windows or Linux system (e.g., Ubuntu OS version 20.04.1 LTS or later) with at least 16 GB RAM and 250GB available disk space
- Docker and Docker-Compose tools 2.2.0 or higher
- Remote Desktop connection to the Linux system
- Basic knowledge of Splunk and Open-Source tools: ELK stack (Elastic Search and Kibana), InfluxDB, Prometheus, and Grafana

Q: How do I start the Influx pump with Grafana?

A: To start the Influx pump with Grafana:
- Make sure the ports 3000 and 8086 are not blocked by the firewall.
- Check the status of the docker containers to ensure that everything is Up and running: docker ps -a

Information

Abstract

Dell PowerEdge Servers with iDRAC9 version 4.00 and Datacenter license, stream data to help IT administrators better understand the inner workings of their server environment. This white paper explains using Telemetry Reference tools to collect the health and performance data and integration with the visualization solutions.
June 2022

Revisions

Date	Description
June 2022	Update for the Reference Tools improvements

Acknowledgments

Authors: Sankara Gara, Sailaja Mahendrakar

Executive summary

With iDRAC9 firmware version 4.00.00.00 or later and Datacenter license, IT managers can integrate advanced server telemetry data such as health, statistics, and performance into their existing analytics and visualization solutions. Telemetry data is provided as granular, time-series data that is streamed, or pushed from iDRAC to the subscribing clients in the form of reports as JSON content in DMTF-compliant format. Ingesting this data into big databases for analytics or visualizations would require some learning and may require significant effort to get started. The Telemetry Reference Tool is a set of opensource tools published at GitHub, that runs outside iDRAC and provide docker containers to facilitate few easy steps to ingest iDRAC telemetry into the timeseries databases like Elasticsearch and InfluxDB. This eventually gets integrated with visualization solutions like Kibana, Grafana and Splunk Analytics. Overview

The iDRAC9Telemetry Reference Tools is a set of docker container applications that facilitate the collection of telemetry data from Dell PowerEdge servers, integrates with external big databases for downstream analytics and visualization. Basic workflow includes the Telemetry data collection from one or more iDRACs.
Ingest the data into big databases like Elasticsearch, Prometheus, Timescale and InfluxDB.
Visualize this data using the visualization tools like Kibana, Grafana, or Splunk Analytics.

The following diagram shows the basic workflow of the Reference Tools set.

DELL-iDRAC9-Power-Edge-Servers-fig-1

Terms and definitions

Telemetry Report: A telemetry report is a DMTF telemetry specification-compliant JSON document that consists of metric names, metric values, and timestamps.
FQDD – Fully Qualified Device Descriptor
SSE – Server-sent events allows a client to open a web service connection which can continuously push data to the client as needed.
ServiceTag – Service tag of the PowerEdge Server.

Prerequisites

iDRAC with firmware 4.00.00.00 or later versions and with Datacenter license
Windows or Linux system like Ubuntu OS version used 20.04.1 LTS or later with at least 16 GB RAM and 250GB available disk space
Docker and Docker-Compose tools 2.2.0 or higher
Remote Desktop connection to the Linux system
Basic knowledge of Splunk and the Open-Source tools: ELK stack (Elastic Search and Kibana), InfluxDB, Prometheus, and Grafana

Reference Tools Architecture

DELL-iDRAC9-Power-Edge-Servers-fig-2

Reference application architecture is based on a plug-in model. Message broker facilitates the communication between the applications. Reference applications are listed under the cmd directory. Ingress telemetry reader application establishes SSE connection with configured iDRAC sources and reads the stream of telemetry metrics. Ingest pump applications ingest metrics into big data endpoints. For example: elkpump- ingests the metrics into elastic search database, influxpump- ingests the data into InfluxDB, prometheuspump- exposes the metrics in Prometheus endpoint.

Configui, dbdiscauth, simpledisc, and simpleauth applications allow mechanisms to configure sources. Configui is a frontend user interface to configure iDRAC sources. DBdiscauth application is responsible for servicing all the requests to onboard the iDRACs and write to the MySQL database. Simpledisc/simpleauth provides a simple file-based source identification.

The toolset is hosted on GitHub at: https://github.com/dell/iDRAC-Telemetry- Reference-Tools.git

Default Ports Used by the Framework are

3000 – Grafana
8080 – configgui port (external). Internally it uses 8082
8088 – Splunk HTTP Event Listener (if using Splunk)
8000 – Splunk Management UI (if using Splunk)
5601 – Kibana
8161 – ActiveMQ Administrative Interface (default credentials are admin/admin)
61613 – ActiveMQ messaging port. Redfish read will send to this port

Workflows

The basic steps involved in visualizing telemetry data from one or more PowerEdge Servers (iDRACs) are:

Setting up the Reference Toolset
If using Splunk, setup HTTP Collector on Splunk web interface
Starting the docker containers for the desired metric pump – splunk, influx, timescale or prometheus etc.
Enabling Telemetry streaming service and reports in the data sources(iDRACs)
Adding Telemetry data sources (iDRACs)
Visualize using Splunk Analytics, Grafana, or Kibana dashboards

Setting up the Reference Toolset

Clone the GitHub repository that contains the iDRAC9 Telemetry reference toolset:
- git clone https://github.com/dell/iDRAC-Telemetry-Reference-Tools.git

Starting Influx Pump

Starting the Influx pump with Grafana is a 2 step process. It is one time setup to generate the Influx and Grafana tokens and starting the Influx pump.
- cd iDRAC-Telemetry-Reference-Tools/docker-compose-files/
- compose. sh setup –influx-test-db
- compose.sh start –influx-test-db
Make sure the ports 3000 and 8086 are not blocked by the firewall.
Check the status of the docker containers to ensure that everything is “Up” and running.
- docker ps -a

Starting Prometheus Pump

Start the Prometheus pump with Grafana and other required applications as docker containers:
- cd iDRAC-Telemetry-Reference-Tools
- docker-compose-files/compose. sh –prometheus-test-db start
Check the status of the docker containers to ensure that everything is “Up” and running.
- docker ps -a

Splunk HTTP Collector Configuration and Start Splunk Pump

Make sure the Splunk service is already running.
Open web GUI interface for Splunk.
Configure Metrics Index: Settings -> Indexes
In the top right of the screen click “New Index”
Create a name, set Index Data Type to Metrics, and Timestamp Resolution to Seconds.
Configure HTTP Event Collector: Settings -> Data Inputs
On the following screen click “+Add new” next to HTTP Event Collector.
Select any name you like for the collector and click “Next” at the top of the screen.
Select “Automatic” for Source type and for Index select the metrics index you created previously.
Click Review at the top, make sure everything is correct and then click “Submit” (again at the top) View HTTP Event Collector: Settings -> Data Inputs -> HTTP Event Collector
Make note of the Token Value assigned to use in the next step (SPLUNK_KEY).
Click Global Settings and make sure All Tokens option is Enabled, check or uncheck Enable SSL option as per your need and ensure HTTP Port Number is specified (default 8088). Click Save after the default settings are modified.
Start the Splunk (metric) Pump:
- Set the following environment variables as per the “HTTP Event Collector” configuration in the Splunk.
- SPLUNK_HEC_KEY=
- SPLUNK_HEC_URL=http://:
- SPLUNK_HEC_INDEX=
- Start the Splunk pump and other required applications as docker containers:
- cd iDRAC-Telemetry-Reference-Tools/docker-compose-files
- compose.sh –splunk-pump start
Check the status of the docker containers to ensure that everything is “Up” and running.
- docker ps -a
Enable Telemetry Streaming Service in Data Sources (iDRACs)
- Enable the Telemetry Streaming service in iDRAC9 with the relevant metric reports. Below sample scripts on
- GitHub repository can be used to enable the metrics reports:
- wget https://github.com/dell/iDRAC-Telemetry-Scripting/archive/master.zip-0iDRACTelemetryScripting-master.zip
- unzip iDRAC-TelemetryScripting-master.zip
- cd iDRAC-TelemetryScripting-master/
- python3 ./ConfigurationScripts/EnableOrDisableAllTelemetryReports.py -ip -u -P -s Enabled
Adding Telemetry Data Sources (iDRACs)
ConfigUl application allows the web user interface-based configuration of the Telemetry data sources.
In a browser address bar, enter the loca/host:8082 to access the “Redfish Telemetry Source Services” dashboard.
Click “Add New Service” button and enter iDRAC IP and credentials in dialog box and click “Add Service”
Refresh the browser page to see that the IDRAC connection state is “Running”.
Enable the desired metric reports on the iDRACs to stream telemetry data (see section 3.1).
Multiple iDRACs can also be added by uploading CVS file with list of iDRACs as below.

  * <iDRAC1 IP>,<user>,<password>
  * <iDRAC2 IP>,<user>,<password>
  * <iDRACn IP>,<user>,<password>

Visualize using Splunk Pump and Splunk Analytics

DELL-iDRAC9-Power-Edge-Servers-fig-10

In the Splunk web interface, click Search & Reporting – Analytics to see metrics in the left panel and charts in the right panel, then click Open Charts button.

Visualize using InfluxDB and Grafana

The Grafana opensource tool can be used to visualize the timeseries metrics of the telemetry streams from various iDRACs.

DELL-iDRAC9-Power-Edge-Servers-fig-11

Login to the Grafana UI (http://localhost:3000) using default credentials: admin/admin)
Make sure the data source under the Configuration tab is auto populated with the Datasource name specified in the configuration, for example: “InfluxDBDataSource”.
Navigate to Home->General to see available dashboards, e.g., FanRPMAutoDashboard and PowerReadingsAutomatedDashboard
Now data can be visualized by clicking on the above dashboards. Following view is with the dashboard FanRPMHexagon View which uses the Polystat plugin installed in the Grafana.

Visualize using Influx UI

Open Influx (http://localhost:8086) using the admin/DOCKER_INFLUXDB_INITPASSWORD (see the env file to find the DOCKER INFLUXDB3 INIT_PASSWORD).
Navigate to Data > Load Data > Buckets and select my-org-bucket.
Select required telemetry metrics from the “my-org-bucket” database to visualize the metrics.

Visualize using Prometheus and Grafana

DELL-iDRAC9-Power-Edge-Servers-fig-15

Login to the Grafana UI (http://localhost:3000) using default credentials (admin/admin).
Create Prometheus data source and test.
Navigate to Configuration > Data Sources, click Add data source and select Prometheus.
In Settings, enter a name for the data source in the Name field, for example: Prometheus_datasource.
Enter http://prometheus:9090 for the source URL.
Click Save & test to test the data source. After testing, a green check mark is displayed that indicates the data source is functional.
Ensure that the metrics are available by entering http://localhost:2112/metrics in a browser address bar.
Create a dashboard to visualize the metrics from the telemetry streams.
- Navigate to Create > Dashboard, click Add an empty panel and select the above created data source from the Query > Data source drop-down.
- In the Query, click Metrics browser to see the list of metrics or enter metric keyword to get the shortlisted metrics.
- Select the desired metric and click the panel above to show the list of metric instances metric along with the FQDD and ServiceTag).
- Select the time and desired metric instance in the panel to visualize the time series metric values.
- Selecting all metrics in the Metrics browser shows the graph as below:

Visualize using Prometheus UI

DELL-iDRAC9-Power-Edge-Servers-fig-18

Browse to http://localhost:9090 and click on the Graph.
Open the Metrics Explorer, select metrics in the drop down and click on the “Execute”

Troubleshooting Tips

Issues	Possible Causes	Solution

No telemetry data streaming from iDRAC

|

Datacenter license not added

|

Add a Datacenter license.

Global telemetry/metric reports disabled

| Enable global telemetry and the metric reports.

Incorrect System time

| Ensure that the system time on iDRAC is set correctly to stream the reports in current time interval and match with the

visualization time-period on the Grafana/Kibana.

InfluxDB data source not working| Incorrect database name provided during data source creation| Database name to be provided under the data source because ingest pumps creates database with this name.
Kibana Index pattern creation not having timescale| ‘poweredge_telemetry_metrics’ database not created properly|

Restart the containers.

Web UI container dashboards not accessible| Invalid IP and port specified| The IP address should be “localhost”. Find the ports numbers to access the containers in web UI using ‘docker ps – a’ command.

Make sure the ports are not blocked by firewall.

docker image prune -f

---|---|---

Cannot select one server data

| The input data not filtered on server ServiceTag| Select the filter with iDRAC Service Tag while creating visualization.

Metrics not seen in Prometheus database

|

Possible database population or configuration issue

| Ensure that the metrics stored in Prometheus container are present by accessing the container IP with port 2112/metrics, for example, localhost:2112/metrics
Metrics not seen in InfluxDB| Possible database population or configuration issue| Check the container status:

docker ps -a

Ensure influx token, Grafana token, api keys are there and not NULL by checking the log files:

docker-compose-files/container-info- influx-pump.txt

docker-compose-files/container-info- grafana.txt

Historical data is getting lost on docker compose restarts| Missing file system mappings for persistence.| Update the docker compose reference files to include file system mappings for persistence.
Telemetry-receiver and ingester containers not starting up| Firewall or proxy rules may prevent downloading of Golang dependency packages when the go applications start.| Define the proxy setting in the compose file as elaborated here. Configure Docker to use a proxy server | Docker Documentation
Blank Visualization Charts| Possible incorrect time interval selected.| In both Kibana and Grafana the time interval to visualize the metrics (interval ranging from 5m to 7d or absolute time) and the type of graph (bar, line, plot etc.) can be selected.
Cannot pull Telemetry Reference tools from GitHub| Docker account does not exist or is inactive.| Create a docker account at https://hub.docker.com/ and login to the docker in the VM using command docker login.
Reached max pull rate limit.| You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate- limits/ . You must authenticate your pull requests.

Technical support and resources

Telemetry reference toolset from Dell:
- https://github.com/dell/iDRAC-Telemetry-Reference-Tools
Telemetry Reference Tools Setup guide:
- https://github.com/dell/iDRAC-Telemetry-Reference-Tools/blob/master/docker-compose-files/setup_guide.pdf
Whitepapers: “Telemetry Custom Reports Getting Started”
- https://www.dell.com/support/home/en-us/product-support/product/enterprise-solution-resources/docs
iDRAC Telemetry Workflow Examples
- https://github.com/dell/iDRAC-Telemetry-Scripting/
Open source iDRAC REST API with Redfish Python and PowerShell examples.
- https://github.com/dell/iDRAC-Redfish-Scripting
The iDRAC support home page provides access to product documents, technical white papers, how-to videos, and more.
- www.dell.com/support/idrac
iDRAC User Guides and other manuals
- www.dell.com/idracmanuals
Dell Technical Support
- Dell.com/support

The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose. Use, copying, and distribution of any software that is described in this publication requires an applicable software license.

Copyright © 2022 Dell Inc. or its subsidiaries All Rights Reserved. Dell and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be trademarks of their respective owners. [8/22/2022] [White Paper] [Document 543]