HDD Monitoring

Monitoring hard disk performance is essential to keep networks up and running at all times. This will ensure that service level agreements are maintained and prompt the swift and smooth delivery of business-critical solutions. Proactive monitoring involves diagnosing and highlighting hard disk issues before they are evident to the end user and actively eliminating threats.

Hard Disk Drive (HDD) Monitoring is fundamentally concerned with observing how a server (be it bare-metal or a virtualized instance) reacts to the operating load placed upon it.

Standard Monitoring Process

Figure: SUMMIT Platform components

In the SummitAI IT Operations Management, two types of technologies are used to collect and monitor the availability and performance of devices. 

• Agentless Monitoring: In the Agentless model, a Proxy Server is installed to monitor other devices. The Proxy Server remotely accesses and pulls information from other devices. This type of monitoring requires no separate installation and is easy to deploy. In the Agentless model for Standard HDD monitoring process, data flows from Proxy Servers to Data Collectors to Database
  
  
• Agent-based Monitoring: In Agent-based model, the Agents should be installed on all the devices and information about the device is collected and sent to the centralized Server. Every Agent communicates with the Proxy Server and the Proxy Server consolidates the data, posts the information to the centralized Server. This requires more installations. The http/s protocols should be open for communications to occur. There are two types of Agents, Wintel (computers that run Windows on an Intel processor) and Non-Wintel Agents. In the Agent-based model for Standard HDD monitoring process, data flows from Agent to Proxy Servers to data collectors to Database.

SummitAI Agent

Agent-based Monitoring allows you to collect, analyse and process significantly more data from a given system. This is usually a given, considering that software and hardware vendors provide agents that are specifically designed for their respective platform and have programmed the agent to interface with it specifically.

A brief overview of the functionalities of SummitAI Agent are as listed below:

• SummitAI Agents (Asset agents and Server Agents) are deployed on the endpoint devices to aid various information collection and other activities.
• Asset Agents are typically used for Asset Management functions and Server Agents are typically installed on the devices for monitoring functions.
• Agents are configured against a Data Collector, where they post all the collected information. However, in a branch office like setup, such data may be sent to a proxy server, which can communicate and upload the data on a periodic basis.

SummitAI Data Collector

Collector is an application that runs on a Linux or Windows server within your infrastructure and uses standard monitoring protocols to intelligently monitor devices within your infrastructure.

By default, Collectors are not configured to use proxies. To change this, you will need to make updates to several proxy settings located in one of the Collector’s configuration files (i.e., the agent.conf file).

A brief overview of the functionalities of SummitAI Data Collector are as listed below:

• It aggregates data collected from various devices, agents, and services.
• The data is collated and updated to the database.
• It needs direct connectivity to the database. Hence, it has to be made part of the data centre infrastructure as it needs consistent network connectivity.
• Interacts with the application hosted on IIS server.
• Provides a mechanism to rationalize on database connections such that database updates are controlled from a selected set of servers.

SummitAI Proxy Server

A brief overview of the functionalities of SummitAI proxy server are as listed below:

• When SummitAI application is hosted in Cloud environment, a proxy server discovers, monitors, and collects asset data from all the enterprise assets. Such data is collated at the proxy and sent to the data collector server for subsequent update to the SummitAI database.
• It is deployed in the branch office environment that provides connectivity and feasibility to the available devices in the branch office.
• Usage of one or more proxy servers are mandatory for following activities:
  • Discovery of devices.
  • Service Automation.
  • Active Directory based Authentication for SummitAI – when the SummitAI server is hosted on the cloud.
  • Active Directory Synchronization Service – when the SummitAI server is hosted on the cloud.
  • Collection of data from all the devices in a remote location and transferring to the remote data collector, on a periodic schedule.

Data Handling Issues

In SummitAI database, data is stored in two forms – XML or table type. The latter is a SQL object that can hold records and be passed as is, to the stored procedure where the data is processed. The issue was that at times, the data output for utilization would be one of the following:

• Non-numeric value in capacity and utilization data causing errors during processing.
• Exceeding the max value of capacity & utilization possible.
• Null value or no value is passed in the mandatory non-numeric columns such as the HDD volume. For example, in a windows system HDD volume must be D: Drive or E: Drive but instead a blank or null value is recorded.
• Incorrect server ID

In these cases, both Standard Monitoring & Store and Forward HDD Monitoring functions receive all data from XML and table type and then process only the correct data outputs.

Audit Table

In case of a customer highlighting an incorrect value on the UI. An intermediate Audit table can be configured to save the pre-processed data. This allows the tracking of pre-processed XML & Table-type data (irrespective of it being correct) and the corresponding outputs. This Applies to on only SF HDD Monitoring.

Store and Forward method for HDD Monitoring

In the Agentless model for Standard HDD monitoring process, data flows from Proxy Servers to Data Collectors to Database while in the Agent-based model for Standard HDD monitoring process, data flows from Agent to Proxy Servers to data collectors to Database. Irrespective of Agent-based or Agentless HDD monitoring, all the data processed is processed simultaneously. This could lead to issues in utilization.

For example, A network has N number of proxies, and all data is being processed simultaneously which in turn means that it is occupying utilization of CPU and memory resources. It could also lead to utilization crossing the threshold limit causing a process timeout.

Store and Forward mechanism solves this issue by grouping the data received from the Data Collector and processing it as chunks of records at a time. This is actually two processes carried by enabling two jobs – Submit Scheduler Job and Database Job (SQL Agent).

Submit scheduler job will push the data into intermediate table known as the Staging Table. In the second SQL job, the data from the Staging Table is processed chunk-wise.

To enable Store and Forward mechanism, use the following key in web.config: