Free Storage, SAN and LAN Performance and Capacity Monitoring

Hitachi CCI & Export Tool troubleshooting

Hitachi CCI (HORCM/raidcom)

  • Hitachi official docu: How To Debug CCI Issues

  • This is most comon error after testing stor2rrd connection to the storage:
    $ ./bin/config_check.sh <Storage Name>
    
      STORAGE: SVPG01: VSPG
      TCP connection to "<SVP IP>" on port "1099" is ok
      TCP connection to "<SVP IP>" on port "51099" is ok
      TCP connection to "<SVP IP>" on port "51100" is ok
      WARNING : TCP connection to "<SVP IP>" on port "5110" has failed! It might need to be open on the firewall.
      VSPG Error: command failed: raidcom -login lpar2rrd  -I
      connection failed!!
    
    1. Assure you use proper user credential (user & password)
    2. Assure you do not use shell special characters like #!?|$*[]\{}`"'& in the password, use ;:.+-%@ instead

  • Error during starting of HORCM under stor2rrd user (lpar2rrd on the virtual appliance)
    raidcom: [EX_ATTHOR] Can't be attached to HORC manager
    
    Problem is that it is running under root! Stopt it under root and start it up under stor2rrd (lpar2rrd)
    When that happen set user permission for /HORCM again as per adivse below, starting under root screw user rights.
    # /HORCM/usr/bin/horcmshutdown.sh 1
    # su - lpar2rrd
    $ /HORCM/usr/bin/horcmstart.sh 1 
    
    Check if they are running
    $ ps -ef | grep horcm
      lpar2rrd 19660912 1 0 Feb 26 - 0:03 horcmd_01
    
  • Assure you are able to start HORCM instance under stor2rrd user (lpar2rrd on the virstuall appliance)
    $ /HORCM/usr/bin/horcmstart.sh 1
    $ ps -ef |grep horcmd_01
      lpar2rrd 14124     1  0 17:21 ?        00:00:00 horcmd_01
    
  • When it does not start then check file rights in /HORCM dir
    Assure that everything is executable and writeable by stor2rrd user (lpar2rrd user on the Virtual Appliance)
    Under root identification execute this:
    # touch /HORCM/etc/USE_OLD_IOCT
    # chown stor2rrd /HORCM
    # chown -R stor2rrd /HORCM/* /HORCM/.uds
    # chmod 755 /HORCM /HORCM/usr/bin /HORCM/usr/bin/* /HORCM/log* /HORCM/etc/horcmgr /HORCM/etc/*conf /HORCM/.uds/
    
  • If you us commande device for storage communication in /etc/horcm1.conf then assure that stoar2rrd user owns it:
    $ ls /dev/sd* | /HORCM/usr/bin/inqraid -CLI -fx
      DEVICE_FILE     PORT    SERIAL  LDEV CTG  H/M/12  SSID R:Group PRODUCT_ID
      sda             -            -     -   -       -     -       - Virtual disk
      sdb             -            -     -   -       -     -       - Virtual disk
      sdc             CL4-A   411711   64B   -       -  000A A:00001 OPEN-V-CM
    $ ls -l /dev/sdc
      brw-rw---- 1 lpar2rrd disk 8, 0 Oct 30 10:23 /dev/sdc
    
  • Try to log on manually, this example uses /etc/horcm1.conf conf file
    $ raidcom -login stor2rrd  -I1
    $ raidcom get pool -I1
      PID  POLS U(%) SSCNT Available(MB) Capacity(MB)     Seq# Num LDEV# H(%)   FMT_CAP(MB)
      000  POLN  69    212     21376320     70428792    310878  48 12211  90             -
      001  POLN  15     13    199501344    234772440    310878 160 12514  90             -
    $ raidcom -logout -I1
      raidcom : User on Serial#(310444) was removed.
    
  • Use the latest CCI, older CCI do not have to work with newer storage firmware
    $ raidqry -h
      Model  : RAID-Manager/Linux/x64
      Ver&Rev: 01-49-03/00
    
  • restart HORCM instance under proper user stor2rrd or lpar2rrd (on the virtual appliance):
    $ /HORCM/usr/bin/horcmshutdown.sh 1
    $ /HORCM/usr/bin/horcmstart.sh 1
    
  • meta_resource was missing on the array for stor2rrd user group in this case:
    $ ./bin/config_check.sh <Storage Name>
    
      STORAGE: SVPG01: VSPG
      TCP connection to "<SVP IP>" on port "1099" is ok
      TCP connection to "<SVP IP>" on port "51099" is ok
      TCP connection to "<SVP IP>" on port "51100" is ok
      VSPG Error: command failed: raidcom get resource -key opt -I440983
      connection failed!!
      raidcom : User on Serial#(310444) was removed.
    

Hitachi Export Tool

  • Hitachi official docu:
    Troubleshooting Export Tool for Performance Monitor
    Device Manager - Storage Navigator Messages

  • This is most comon error after testing stor2rrd connection to the storage:
    $ ./bin/config_check.sh <Storage Name>
    
      STORAGE: SVPG01: VSPG
      TCP connection to "<SVP IP>" on port "1099" is ok
      TCP connection to "<SVP IP>" on port "51099" is ok
      TCP connection to "<SVP IP>" on port "51100" is ok
      WARNING : TCP connection to "" on port "5110" has failed! It might need to be open on the firewall.
      Loading ExportTool...
      Missing ip
      2019/01/02 15:46:48 start Export Loader Version : 88-00-00
      2019/01/02 15:46:48  command file path=/opt/hds/hds_test/command2.txt
      2019/01/02 15:46:48  log file path    =/opt/hds/88-03-02/log
      2019/01/02 15:46:48  log file name    =loader0102154648.log
      2019/01/02 15:46:48  ExportLoader mode=ALL
      2019/01/02 15:46:48  download port    =51099
      2019/01/02 15:46:48 read target address from command file.
      2019/01/02 15:46:48 Missing ip
      2019/01/02 15:46:48 ExportLoader fail.
      2019/01/02 15:46:48 ExportLoader finish. exitcode=101
      Loading ExportTool...
      Export tool start [Version 88-03-01-FF/00]
      command file = /opt/hds/hds_test/command.txt
      [  1] ip <SVP IP>      ; Specifies IP adress of SVP
      [  2] dkcsn 415002           ; Specifies Serial Number of SVP
      [  3] login User = ["user"], Passwd = [****************]
      Login failed [line = 3]
      Execution stops.
      Export tool end
      ...
    
      The most comon reason why Export Tool does not work is different version of the storage firmware and Export Tool.
      Export Tool version MUST be at least same as the storage firmware, it could be even higher as per our experience.

    • This could mean wrong user or password:
        Login failed [line = 2] 
      
    • Assure you have installed Java version 8. It might not work with Java 6 especiall when you this in error output:
        [  3] login User = ["user"], Passwd = [****************]
      
      Java version:
      $ java -version
        openjdk version "1.8.0_131"
      
    • Assure that /opt/hds and all objects inside are owned by stor2rrd user (lpar2rrd on the virtual appliance)

  • In the log you see the error:
    "2019/02/04 16:03:29 . Find Exception = java.net.MalformedURLException: invalid authority: //eka_hus-vm_1.eka.tmk.group/RMIControl".
    
    Export tool uses RMI and the server hostname for communication with HUS VM.
    The Internet standards for protocols mandate that hostname labels may contain only the ASCII letters ‘a’ through ‘z’ (in a case-insensitive manner), the digits ‘0’ through ‘9’, and the hyphen(‘-‘).
    Change the underscore (_) to hyphen(‘-‘) in hostname and issue is resolved.

  • In the log you see the error:
    The error "sanproject.serverux.data.SANRmiException : RMI server error (1, 5400)" 
    
    Hitachi description: A different user might open one of the following windows or run the export tool. In this case, wait until one of the users moves to a different window or exits the export tool.

    Someone else is using Export Tool at the same time.