Free Storage, SAN and LAN Performance and Capacity Monitoring

Dell storage installation

Back to storage installation home.


Source of performance data are not storages themself but REST API of one of these:
  • Dell Storage Enterprise Manager
  • Dell Storage Manager Data Collector
Dell Compellent monitoring example

User rights options

  • Create user stor2rrd under admin role or use already existing admin account.
    • Admin rights give you visibility of the whole storage include items not being under any folder and system subsystems like:
    • Node (controller): CPU, IO, data, response time
    • Port: IO, data, response time

  • Create user stor2rrd under Reporter role and assign him rights to all or specific:
    • volume folders
    • server folders
    • disk folders
    In this case will be reported utilization only for assigned folders and their items.
    • You can whenever switch from Reporter access to admin one and vice versa.

    In both cases you have to create a user on all storages and on Enterprise Manager/Data collector where you have to attach those storages under that Reporter or admin account.

Storage connectivity

  • Install the Enterprise Manager or DSM Data Collector and attach there your Compellent boxes.

  • Create user stor2rrd on the Enterprise Manager or DSM Data Collector under admin role.
    Unfortunatelly admin role is only the role which can get all performance data from the storage.

  • If you use web tool Unisphere central for SC series then registration there is not enough.
    You have to register storage for user in Enterprise Manager or DSM Data Collector.

  • Allow access from the STOR2RRD host to the Enterprise Manager on port 3033 (HTTPS).
    $ perl /home/stor2rrd/stor2rrd/bin/conntest.pl 192.168.1.1 3033
  Connection to "192.168.1.1" on port "3033" is ok
  • Set "Report gathering settings" of IO usage to 5 minutes.
    Dell Compellent Report gathering settings

STOR2RRD storage configuration

  • All actions below under stor2rrd user (lpar2rrd on Virtual Appliance)

  • On the picture you can see 3 compellent storages in the Enterprise Manager.
    Dell Compellent monitoring schema


    Create config entry in etc/storage-list.cfg per each storage.
    Uncomment (remove the hash) example line and adjust it.
    Use 1 line per each storage you want to monitor, use same storage name as you see in the Enterprise Manager.
    $ vi /home/stor2rrd/stor2rrd/etc/storage-list.cfg

#
# Dell SC Series (Compellent)
#
# Storage name from Enterprise Manager:COMPELLENT:IP or hostname of the Enterprise Manager:port:storage user:encrypted password:VOLUME_AGG_DATA_LIM:VOLUME_AGG_IO_LIM:SAMPLE_RATE_MINS
# to encrypt password use: perl ./bin/spasswd.pl
#Dell-Compellent01:COMPELLENT:ent-manager_host.example.com:3033:stor2rrd:KT4mXVI9N0BUPjZdVQo=:1024:10:
#Dell-Compellent01:COMPELLENT:ent-manager_host.example.com:3033:stor2rrd:KT4mXVI9N0BUPjZdVQo=

compellent01:COMPELLENT:192.168.1.1:3033:stor2rrd:KT4mXVI9N0BUPjZdVQo=
compellent02:COMPELLENT:192.168.1.1:3033:stor2rrd:KT4mXVI9N0BUPjZdVQo=
compellent03:COMPELLENT:192.168.1.1:3033:stor2rrd:KT4mXVI9N0BUPjZdVQo=
    Above configures 3 storages attached to the Enterprise Manager or DSM Data Collector with IP 192.168.1.1 which will be visible in the UI as compellent01, compellent02 and compellent03

    Use encrypted password in storage line above: 
    $ cd /home/stor2rrd/stor2rrd
$ perl bin/spasswd.pl

  Encode password for storage authentication:
  -------------------------------------------
  Enter password:
  Re-enter password:

  Copy the following string to the password field of the corresponding line in etc/storage-list.cfg:

  KT4mXVI9N0BUPjZdVQo=
  • Assure you have enough of disk space on the filesystem where is STOR2RRD installed
    Roughly you might count 2 - 30 GB per a storage (it depends on number of volumes, 30GB for 5000 volumes) 
    $ df -g /home   # AIX
$ df -h /home   # Linux
  • Check storage connectivity 
    $ cd /home/stor2rrd/stor2rrd
$ ./bin/config_check.sh
  =====================
  STORAGE: compellent01 : COMPELLENT
  =====================
  connection ok
    If you get an errors like "Missing LWP::Protocol::https", "Connection reset by peer" or "500 SSL negotiation failed" then follow this.

  • Schedule to run storage agent from stor2rrd crontab (lpar2rrd on Virtual Appliance, it might already exist there) 
    $ crontab -l | grep load_compellentperf.sh
$
    Add if it does not exist as above 
    $ crontab -e

# Dell SC Series (Compellent)
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /home/stor2rrd/stor2rrd/load_compellentperf.sh > /home/stor2rrd/stor2rrd/load_compellentperf.out 2>&1
    Assure there is already an entry with the UI creation running once an hour in crontab 
    $ crontab -e

# STOR2RRD UI (just ONE entry of load.sh must be there)
5 * * * * /home/stor2rrd/stor2rrd/load.sh > /home/stor2rrd/stor2rrd/load.out 2>&1
  • Let run the storage agent for 15 - 20 minutes to get data, then: 
    $ cd /home/stor2rrd/stor2rrd
$ ./load.sh
  • Go to the web UI: http://<your web server>/stor2rrd/
    Use Ctrl-F5 to refresh the web browser cache.

It is a LSI rebranded storage. Configure it like IBM DS5k storage in the product.

In case of usage of Virtual Appliance
  • Use local account lpar2rrd for hosting of STOR2RRD on the virtual appliance
  • Use /home/stor2rrd/stor2rrd as the product home
  • Replace lpar2rrd by stor2rrd user in all examples below.

SMcli installation

  • You can find SMcli install package inside Resourse DVD
    Download DELL_MDSS_Consolidated_RDVD_6_5_0_1.iso
    Dell MD3000 storage
  • You can download SMcli package even from the IBM site (you need to know storage serial and have IBM ID free account)
    Go to IBM fix central and follow this:
    Product Group [System Storage] ➡
      Select from System Storage [Disk systems] ➡
        Select from Entry level disk systems [DS3000] ➡
            Release [All] ➡
              Platform [AIX/Linux ...]

    Download only *SMIA* package like:
    SM10.86_Linux_64bit_x86-64_SMIA-10.86.x5.43.tgz
    SM10.86_AIX_SMIA-10.86.x5.43.tgz

    Note actual IBM page does not seem to contain SMIA package, use Storage manager which should contain SMCli: Disk-SM-AIX-Dec-2013-version-10.86.x5.43
  • AIX prerequisites:
    Install X11 packages from your AIX install media:
    • X11.adt
    • X11.base
    • X11.motif
    # lslpp -L |egrep "X11.adt|X11.base|X11.motif"
  X11.adt.bitmaps            6.1.0.0    C     F    AIXwindows Application
  X11.adt.ext                6.1.6.0    C     F    AIXwindows Application
  X11.adt.imake              6.1.6.0    C     F    AIXwindows Application
  X11.adt.include            6.1.4.0    C     F    AIXwindows Application
  X11.adt.lib                6.1.2.0    C     F    AIXwindows Application
  X11.adt.motif              6.1.6.0    C     F    AIXwindows Application
  X11.base.common            6.1.0.0    C     F    AIXwindows Runtime Common
  X11.base.lib               6.1.6.0    C     F    AIXwindows Runtime Libraries 
  X11.base.rte               6.1.6.0    C     F    AIXwindows Runtime Environment
  X11.base.smt               6.1.6.0    C     F    AIXwindows Runtime Shared
  X11.base.xpconfig          6.1.0.0    C     F    Xprint Configuration Files 
  X11.motif.lib              6.1.6.0    C     F    AIXwindows Motif Libraries 
  X11.motif.mwm              6.1.6.0    C     F    AIXwindows Motif Window
  • Install SMcli under root user
    # umask 022
# gunzip SM10.86_AIX_SMIA-10.86.x5.43.tgz  
# tar xvf SM10.86_AIX_SMIA-10.86.x5.43.tar   
# cd AIX10p86/AIX 
# sh SMIA-AIX-10.86.0605.0043.bin -i console
  (accept the license agreement, type "enter" many times )

  Please choose the Install Set to be installed by this installer.

  1- Typical (Full Installation)
  --> 2- Management Station
  3- Host

  4- Customize...

  ENTER THE NUMBER FOR THE INSTALL SET, OR PRESS <ENTER> TO ACCEPT THE DEFAULT
  : 2
  ......
  Congratulations. IBM System Storage DS Storage Manager 10 has been successfully installed to:
  /opt/IBM_DS

# lslpp -L|grep SMclient
  SMclient.aix.rte      10.86.6543.0    C     F    IBM DS Storage Manager Client
# ls -l /usr/SMclient/SMcli
  -r-x------    1 root     system         1008 Oct 10 2013  /usr/SMclient/SMcli
    Check install log below for errors if you do not see "Congratulations ..." message after the installation.
    /opt/IBM_DS/IBM_System_Storage_DS_Storage_Manager_10_InstallLog.log

  • Note: you might hit into the IBM installer bug during rpm install of SMclient-10.86.0G05.0043-1.noarch. Below is the workaround. Then try installation again. 
    # ln -s /opt/IBM_DS/ia/jre/ /opt/IBM_DS/jre
  • When you have Debian based Linux (Ubuntu etc) which does not support rpm packaging then you will find uninstalled packages in /opt/IBM_DS
    Convert them in debian packages and install them as root: 
    # cd /opt/IBM_DS
# alien SMclient.rpm 
# alien SMesm.rpm 
# alien SMruntime.rpm 
# dpkg -i smclient_10.86.0G05.0043-2_all.deb 
# dpkg -i smesm_10.86.0G05.0007-2_all.deb 
# dpkg -i smruntime_10.86.0A05.0001-2_amd64.deb 
# ls -l /usr/SMclient/SMcli
  -r-x------    1 root     system         1008 Oct 10 2013  /usr/SMclient/SMcli
  • Stop SMmonitor service
    AIX: 
    # SMmonitor stop
# rmitab smmonitor
    Linux: 
    # /opt/SMgr/client/SMmonitor stop
# chkconfig --list SMmonitor
# chkconfig --level 2345 SMmonitor off
alternatively:
  # rm -f /etc/rc2.d/S99SMmonitor /etc/rc.d/init.d/SMmonitor
    Linux systemd: 
    # systemctl stop smmonitor.service
# systemctl disable smmonitor.service
  • Allow user stor2rrd run SMcli API (lpar2rrd user on Virtual Appliance)
    Under root execute following: 
    # rm -f /var/opt/SM/emwdata_v04_bin.lock /var/opt/SM/pref_01_bin.lock
# chown stor2rrd /opt/IBM_DS/client/SMcli /var/opt/SM/prefbackup_01.bin 
# chown stor2rrd /var/opt/SM/emwback_v04.bin /var/opt/SM /var/opt/SM/emwdata_v04.bin
# chown stor2rrd /var/opt/SM /var/opt/SM/pref_01.bin
# chmod -f 644 /usr/SMruntime/jre/lib/security/java.security
# chmod -f 755 /opt/IBM_DS/client/lib/*
    If you do not do it then you get such error by running it under someone else than root:
    (Do not forget to deactivate SMmonitor as above as it can change file rights back to root itself) 
    # su - stor2rrd
$ SMcli -d
  An internal error, error code 12, has occurred.  This is possibly due to
  initialization problems when loading the necessary internal files.  Please check
  your installation of the storage management client. If problems continue 
  to persist, re-install the storage management client.

Storage configuration

  • All actions below under stor2rrd user (lpar2rrd on Virtual Appliance) 
    # su - stor2rrd
  • Allow access from the STOR2RRD host to the storage on both controllers on port 2463.
    Test if port is open: 
    $ perl /home/stor2rrd/stor2rrd/bin/conntest.pl 192.168.1.1 2463
  Connection to "192.168.1.1" on port "2463" is ok
  • Connect new storage, you must connect both storage controllers 
    $ SMcli -d
  There are currently no storage subsystems listed in the configuration file. Add
  storage subsystems using the Add Storage Subsystem option in the storage management
  software or by command line.
$ SMcli -A 192.168.1.1 192.168.1.2
  New storage subsystem was discovered at address 192.168.1.1
  New storage subsystem was discovered at address 192.168.1.2
  SMcli completed successfully
$ SMcli -d
  MD3800_01 ControlerA_hostname  ControlerB_hostname
    Use "MD3800_01" further for etc/storage-list.cfg configuration

  • Check if the storage needs user authorization (new firmwares allows to specify password for monitor user role): 
    $ SMcli -n MD3800_01 -e -c "show hostTopology;"
    When it fails then use "monitor" role credetials (password can be set in the storage UI): 
    $ SMcli -n MD3800_01 -e -c "show hostTopology;"            
  The operation needs either the 'Administrator' or 'Monitor' password to be set.
$ SMcli -n MD3800_01 -p <password> -R monitor -e -c "show hostTopology;"
    Use that credentials in future etc/storage-list.cfg configuration

STOR2RRD storage configuration

  • All actions below under stor2rrd user (lpar2rrd on Virtual Appliance)

  • Configure storages in etc/storage-list.cfg
    Under stor2rrd account create config entry. Uncomment (remove the hash) example line and adjust it: 
    $ vi /home/stor2rrd/stor2rrd/etc/storage-list.cfg  

#
# Dell PowerVault MD3000 (use DS5K as storage type)
# NetApp E-series (use DS5K as storage type)
# IBM DS3000/4000/5000 & DSC3700
#
#storage_alias:DS5K:storage user:user_password:VOLUME_AGG_DATA_LIM:VOLUME_AGG_IO_LIM:SAMPLE_RATE_MINS
#new storages can be configured to use username/password (use "monitoring" account), old ones do not have this option
#
#MD3800:DS5K:monitor:password:256:10:5
#MD3800:DS5K:
MD3800_01:DS5K:
  • Assure you have enough of disk space on the filesystem where is STOR2RRD installed
    Roughly you might count 2 - 30 GB per a storage (it depends on number of volumes, 30GB for 5000 volumes) 
    $ df -g /home   # AIX
$ df -h /home   # Linux
  • Test the storage connectivity: 
    $ cd /home/stor2rrd/stor2rrd
$ ./bin/config_check.sh 
  =========================
  STORAGE: MD3800_01: DS5K
  =========================
  /usr/SMclient/SMcli -n MD3800_01 -e -c "storageSubsystem summary;"
  connection ok
    When connection failed and direct command says this: 
    $ SMcli -n MD3800_01 -e -c "show hostTopology;"
  Warning! No Monitor password is set for the storage subsystem.
    Then it looks like that the latest firmwares require enabling of password authorization.
    Set password for monitor account in the storage UI and then place same password into etc/storage-list.cfg
    Do not use shell special characters like #!?|$*[]\{}`"'& in the password, use ;:.+-%@ instead. 
    MD3800_01:DS5K:monitor:password01:
    Then verify connection via: 
    $ ./bin/config_check.sh MD3800_01
...
  • Schedule to run storage agent from stor2rrd crontab (lpar2rrd on Virtual Appliance, it might already exist there) 
    $ crontab -l | grep load_ds5perf.sh
$
    Add if it does not exist as above 
    $ crontab -e  

# LSI rebrands like Dell MD3000, IBM DS3000/4000/5000 and NetApp E-series
0,5,10,15,20,25,30,35,40,45,50,55 * * * * /home/stor2rrd/stor2rrd/load_ds5perf.sh > /home/stor2rrd/stor2rrd/load_ds5perf.out 2>&1
    Assure there is already an entry with the UI creation running once an hour in crontab 
    $ crontab -e

# STOR2RRD UI (just ONE entry of load.sh must be there)
5 * * * * /home/stor2rrd/stor2rrd/load.sh > /home/stor2rrd/stor2rrd/load.out 2>&1
  • Let run the storage agent for 15 - 20 minutes to get data, then: 
    $ cd /home/stor2rrd/stor2rrd
$ ./load.sh
  • Go to the web UI: http://<your web server>/stor2rrd/
    Use Ctrl-F5 to refresh the web browser cache.