Free Server Performance Monitoring
STOR2RRD

IBM Power Systems installation

To install LPAR2RRD follow all tabs from the left to the right.
Skip Prerequisites and Web tabs in case of configuring Virtual Appliance

  • LPAR2RRD can be hosted on any Unix or Linux Operating system
  • Rough disk space requirements:
    • IBM Power edition: 6GB for 100 LPARs with OS agents
    • IBM i OS agent: 170MB per each IBM i OS agent
  • Perl: arbitrary version coming with the OS with some additional modules, more below
  • Apache or any other Web server
Skip it in case of configuring Virtual Appliance

User creation

  • create lpar2rrd user on a server which owns the tool and will have access to HMC/IVM
  • add a user on Unix under root account
    # useradd -c "LPAR2RRD user" -m lpar2rrd
    
  • Increase memory limits for user lpar2rrd and the WEB server user (under root)
    • AIX
      # chuser  data=-1  lpar2rrd
      # chuser  stack=-1 lpar2rrd
      # chuser  rss=-1   lpar2rrd
      # chuser  data=-1  apache
      # chuser  stack=-1 apache
      # chuser  rss=-1   apache
      
    • Linux
      # vi /etc/security/limits.conf
      
      @lpar2rrd        hard    stack           -1
      @lpar2rrd        soft    stack           -1
      @lpar2rrd        hard    data            -1
      @lpar2rrd        soft    data            -1
      @apache          hard    stack           -1
      @apache          soft    stack           -1
      @apache          hard    data            -1
      @apache          soft    data            -1
      
If you have already running STOR2RRD on the same host then skip to Web server configuration.

Software prerequisites

  • Apache
  • RRDTool with Perl support (rrdtool & rrdtool-perl)
  • Perl TimeDate module
  • Perl XML::Simple module
  • Perl PDF::API2 for PDF reporting

Linux

  • Linux RedHat, CentOS
    # yum list | egrep "httpd|rrdtool|perl\.|perl-TimeDate|perl-XML-Simple|perl-XML-SAX|CGI\.|perl-Env\.|sharutils"
        httpd.i386                             2.2.15-15.el6
        perl.i386                              4:5.10.1-119.el6_1.1
        rrdtool.i386                           1.2.30-1.wrl
        rrdtool-perl.i386                      1.2.30-1.wrl
        rrdcached.i386                         1.2.30-1.wrl
        perl-TimeDate.noarch                   1:1.16-11.1.el6
        perl-XML-SAX.noarch                    0.96-7.el6
        perl-XML-Simple.noarch                 2.14-8.el6
        perl-CGI.i386                          3.51-119.el6_1.1
        perl-Env.noarch                        1.04-2.el7
    # yum install rrdtool rrdtool-perl httpd
    # yum install epel-release
    # yum install perl perl-TimeDate perl-XML-Simple perl-XML-SAX perl-XML-LibXML perl-Env perl-CGI perl-Data-Dumper perl-LWP-Protocol-https perl-PDF-API2 perl-libwww-perl
    # yum install ed bc libxml2 sharutils
    
    Note that rrdtool-perl and epel-release (this is necessary only for PDF reporting) might not be in your RedHat base repository especially for RHEL 6.x and olders.
    Install CentOS package instead rrdtool-perl-1.4.8-9.el7.x86_64.rpm and epel-release-7-9.noarch.rpm

  • Linux Debian, Ubuntu
    # apt-get install apache2 perl rrdtool
    # apt-get install librrdp-perl libxml-sax-perl libxml-simple-perl libtimedate-perl libenv-sanctify-perl libcgi-pm-perl libdata-dumper-simple-perl libpdf-api2-perl liblwp-protocol-https-perl
    # apt-get install ed bc libxml2-utils
    
    When you have done that then skip to Web server configuration
    yum or apt-get then go for manual RPM installation how is described in AIX section -->

AIX

Solaris

    # /opt/csw/bin/pkgutil -y -i pm_xml_simple pm_libxml_perl pm_xml_sax rrdtool
    
You have 2 options how to connect the HMC.
  • HMC REST API: it brings you physical adapter stats, enhanced configuration in compare to CLI access,
    it uses read-only user on the HMC with a password authentification
  • HMC CLI: it uses ssh-key based authorisation on HMC, read-only account

HMC CLI

Assure you can connect to the HMC via ssh. If not then:
  • Enable remote cmd execution:
    HMC UI (classic) ➡ HMC management ➡ Remote Command Execution ➡ Enable
  • Assure that HMC firewall allows incoming ssh conections :
    HMC UI (classic) ➡ HMC management ➡ Change net settings ➡ LAN adapters ➡ Select open interface ➡ Firewall ➡ Select SSH ➡ Allow incoming

Allow data collection on the HMC

    Allow utilization data collection (either via the HMC UI or globally via following cmd as hscroot)
    $ ssh hmc1 -l hscroot 
    hscroot@hmc1:~> chlparutil -r config -s 60
    hscroot@hmc1:~> lslparutil -r config  -F name,sample_rate
    PWR6A-9117-MMA-SN103AXXX,60
    PWR6B-9117-MMA-SN103BXXX,60
    
    Use padmin account for IVM, or sysadmin for SDMC/FLEX
    Repeat that for all your HMCs/IVMs/SDMCs.

Allow password less access to the HMC

    Create ssh keys (type enter when you are prompted for passphrase) under lpar2rrd user on LPAR2RRD server
    $ who am I
      lpar2rrd    pts/0       Jan 10 10:13     (192.168.1.240)
    $ ssh-keygen -t rsa
      Generating public/private rsa key pair.
      Enter file in which to save the key (/home/lpar2rrd/.ssh/id_rsa):
      Enter passphrase (empty for no passphrase):
      Enter same passphrase again:
      ...
    
    $ cat /home/lpar2rrd/.ssh/id_rsa.pub 
    'ssh-dss AAAAB3NzaC1kc3MAAACBALvbupPLnqy6UfJjgMG5SRnnFDfD013OtBxFv8P7qoHfGKCG0Vu2
    IhNCYttpYMmsMR+BWADo4c9oT7r92raLfVDjNW8uO5C5fatK305+sNqazbT91HDDNDKQnbpoKUqybVRC2
    BQbPR8ESh+ws3uHdxXSQKwOSyrHO6Nwkkx8/h4TAAAAFQDZgT9MxrEUQg4uOwxhciwwdoOO8QAAAIAH37
    x06Ia4FiCdlk3U9vuOI0QEvCKMXL+ZPFNoRcgiqiCnR2WeiaG5qM+odYWk/F/owV0Y/DmkmrrjPrgDADj
    S1uKoSy+NMg803+4cS1B06EMtia+RmMbWkr+kQZ90WvB8C5cxofzcllnQjztQxVrZnZPmMI73/SQdS7QB
    U9a9WwAAAIA3VAdMvP+ZO3zs57FaznMySoszxgNWxuINlcP61TQuyuCp5Bq3mFXwxFwJWPpqpbW5yHj9N
    1+3pscSsFxEQb9YVvb1oL9c5QdPDnmXZK8BHHmNZMn+ftuzJKHgR2lynaxQ4hxTnz+xT0ywGwokeuuCnr
    4G3Bftc70Yn9vCNjuzGQ== lpar2rrd@p550-2'
    
  • HMC


    create a HMC account with role hmcviewer on the HMC server
    $ ssh -l hscroot hmc1
    hscroot@hmc1:~> mkhmcusr -u lpar2rrd -a hmcviewer --passwd abc1234
    
    Paste the pub key created above to mkauthkeys command on the HMC
    (as hscroot, all in one line, without carriage returns):
    hscroot@hmc1:~> mkauthkeys -u lpar2rrd -a 'ssh-dss AAAAB3NzaC1kc3MAAACBALvbupPL
    nqy6UfJjgMG5SRnnFDfD013OtBxFv8P7qoHfGKCG0Vu2IhNCYttpYMmsMR+BWADo4c9oT7r92raLfVDjNW
    8uO5C5fatK305+sNqazbT91HDDNDKQnbpoKUqybVRC2BQbPR8ESh+ws3uHdxXSQKwOSyrHO6Nwkkx8/h4T
    AAAAFQDZgT9MxrEUQg4uOwxhciwwdoOO8QAAAIAH37x06Ia4FiCdlk3U9vuOI0QEvCKMXL+ZPFNoRcgiqi
    CnR2WeiaG5qM+odYWk/F/owV0Y/DmkmrrjPrgDADjS1uKoSy+NMg803+4cS1B06EMtia+RmMbWkr+kQZ90
    WvB8C5cxofzcllnQjztQxVrZnZPmMI73/SQdS7QBU9a9WwAAAIA3VAdMvP+ZO3zs57FaznMySoszxgNWxu
    INlcP61TQuyuCp5Bq3mFXwxFwJWPpqpbW5yHj9N1+3pscSsFxEQb9YVvb1oL9c5QdPDnmXZK8BHHmNZ{{M
    n+ftuzJKHgR2lynaxQ4hxTnz+xT0ywGwokeuuCnr4G3Bftc70Yn9vCNjuzGQ== lpar2rrd@p550-2'
    
    Note a problem on HMC V8.8.6+ where user hscroot is not able to set user's SSH keys with no printing any error message. As a workaround log as lpar2rrd and run mkauthkeys:
    $ ssh -l lpar2rrd hmc1
      password:
    
    lpar2rrd@hmc1:~> ls -l .ssh/authorized_keys2
      ls: cannot access .ssh/authorized_keys2: No such file or directory
    
    lpar2rrd@hmc1:~> mkauthkeys -a 'ssh-dss AAAAB3NzaC1kc3MAAACBALvbupPL
    nqy6UfJjgMG5SRnnFDfD013OtBxFv8P7qoHfGKCG0Vu2IhNCYttpYMmsMR+BWADo4c9oT7r92raLfVDjNW
    8uO5C5fatK305+sNqazbT91HDDNDKQnbpoKUqybVRC2BQbPR8ESh+ws3uHdxXSQKwOSyrHO6Nwkkx8/h4T
    AAAAFQDZgT9MxrEUQg4uOwxhciwwdoOO8QAAAIAH37x06Ia4FiCdlk3U9vuOI0QEvCKMXL+ZPFNoRcgiqi
    CnR2WeiaG5qM+odYWk/F/owV0Y/DmkmrrjPrgDADjS1uKoSy+NMg803+4cS1B06EMtia+RmMbWkr+kQZ90
    WvB8C5cxofzcllnQjztQxVrZnZPmMI73/SQdS7QBU9a9WwAAAIA3VAdMvP+ZO3zs57FaznMySoszxgNWxu
    INlcP61TQuyuCp5Bq3mFXwxFwJWPpqpbW5yHj9N1+3pscSsFxEQb9YVvb1oL9c5QdPDnmXZK8BHHmNZ{{M
    n+ftuzJKHgR2lynaxQ4hxTnz+xT0ywGwokeuuCnr4G3Bftc70Yn9vCNjuzGQ== lpar2rrd@p550-2'
    
  • SDMC/FLEX


    create lpar2rrd user under sysadmin account with role SMMonitor:
    $ ssh -l sysadmin smdc1
    sysadmin@sdmc:~> smcli mkuser -u lpar2rrd -g smmon -p abc1234
    sysadmin@sdmc:~> smcli chuser -u lpar2rrd -e expire 9999 # do not let it expire
    sysadmin@sdmc:~> mkauthkeys -u lpar2rrd -a 'ssh-dss AAAAB3NzaC1kc3MAAACBALvbupPL
    nqy6UfJjgMG5SRnnFDfD013OtBxFv8P7qoHfGKCG0Vu2IhNCYttpYMmsMR+BWADo4c9oT7r92raLfVDjNW
    8uO5C5fatK305+sNqazbT91HDDNDKQnbpoKUqybVRC2BQbPR8ESh+ws3uHdxXSQKwOSyrHO6Nwkkx8/h4T
    AAAAFQDZgT9MxrEUQg4uOwxhciwwdoOO8QAAAIAH37x06Ia4FiCdlk3U9vuOI0QEvCKMXL+ZPFNoRcgiqi
    CnR2WeiaG5qM+odYWk/F/owV0Y/DmkmrrjPrgDADjS1uKoSy+NMg803+4cS1B06EMtia+RmMbWkr+kQZ90
    WvB8C5cxofzcllnQjztQxVrZnZPmMI73/SQdS7QBU9a9WwAAAIA3VAdMvP+ZO3zs57FaznMySoszxgNWxu
    INlcP61TQuyuCp5Bq3mFXwxFwJWPpqpbW5yHj9N1+3pscSsFxEQb9YVvb1oL9c5QdPDnmXZK8BHHmNZ{{M
    n+ftuzJKHgR2lynaxQ4hxTnz+xT0ywGwokeuuCnr4G3Bftc70Yn9vCNjuzGQ== lpar2rrd@p550-2'
    
  • IVM


    Create lpar2rrd user under padmin account with role view:
    $ ssh -l padmin ivm1
    padmin@ivm1:~> mkuser -attr pgrp=view lpar2rrd
    padmin@ivm1:~> passwd lpar2rrd
      Changing password for "lpar2rrd"
      lpar2rrd's New password:
      Enter the new password again:
    padmin@ivm1:~> su - lpar2rrd    # it must be done under the user identification!
    lpar2rrd@ivm1:~> mkauthkeys -a 'ssh-dss AAAAB3NzaC1kc3MAAACBALvbupPLnqy6UfJjgMG5SRnn
    FDfD013OtBxFv8P7qoHfGKCG0Vu2IhNCYttpYMmsMR+BWADo4c9oT7r92raLfVDjNW8uO5C5fatK305+sNq
    azbT91HDDNDKQnbpoKUqybVRC2BQbPR8ESh+ws3uHdxXSQKwOSyrHO6Nwkkx8/h4TAAAAFQDZgT9MxrEUQg
    4uOwxhciwwdoOO8QAAAIAH37x06Ia4FiCdlk3U9vuOI0QEvCKMXL+ZPFNoRcgiqiCnR2WeiaG5qM+odYWk/
    F/owV0Y/DmkmrrjPrgDADjS1uKoSy+NMg803+4cS1B06EMtia+RmMbWkr+kQZ90WvB8C5cxofzcllnQjztQ
    xVrZnZPmMI73/SQdS7QBU9a9WwAAAIA3VAdMvP+ZO3zs57FaznMySoszxgNWxuINlcP61TQuyuCp5Bq3mFX
    wxFwJWPpqpbW5yHj9N1+3pscSsFxEQb9YVvb1oL9c5QdPDnmXZK8BHHmNZ{{Mn+ftuzJKHgR2lynaxQ4hxT
    nz+xT0ywGwokeuuCnr4G3Bftc70Yn9vCNjuzGQ== lpar2rrd@p550-2'
    

Connectivity test

  • test the connection between the LPAR2RRD hosted server and HMC
    (should not request password any more) as lpar2rrd user:
    $ ssh -l lpar2rrd hmc1
      The authenticity of host 'hmc1 (9.138.236.97)' can′t be established.
      RSA key fingerprint is 97:e1:a5:c2:58:3c:c6:0c:d1:6f:4d:6e:62:9c:68:4b.
      Are you sure you want to continue connecting (yes/no)? yes
    lpar2rrd@hmc1:~> 
    
  • if you use a commercial SSH distribution (like F-Secure ssh) which supports only SSH2 format (above example was for OpenSSH format), then you need to convert public keys onto OpenSSH format before uploading the keys into HMC. Convert SSH2 to OpenSSH format and after that use it in mkauthkeys:
    $ ssh-keygen -i -f your-ssh2.pub > your-openssh.pub
    
  • Configure HMC from LPAR2RRD UI (v6.00+)

  • Allow "real-time" refresh on demand, WEB user (apache, nobody) needs to have access to SSH keys
    • find out WEB user
      $ ps -ef|egrep "apache|httpd"|grep -v grep|awk '{print $1}'|grep -v "root"|head -1 
        nobody
      
    • under root user copy SSH keys and change ownership to the WEB user
      # cp /home/lpar2rrd/.ssh/id_rsa /home/lpar2rrd/.ssh/realt_rsa  
      # chown nobody /home/lpar2rrd/.ssh/realt_rsa  
      # chmod 600 /home/lpar2rrd/.ssh/realt_rsa
      
  • Assure you have enough of disk space on the fs where is LPAR2RRD installed
    Roughly you might count 5MB/lpar (10MB/lpar in dual HMC setup) == 1GB/100lpars (without the OS agent)
    $ df -g /home # AIX
    $ df -h /home # linux
    
  • Schedule alerting
    $ crontab -e  
    
    0,10,20,30,40,50 * * * * /home/lpar2rrd/lpar2rrd/load_alert.sh > /home/lpar2rrd/lpar2rrd/load_alert.out 2>&1
    
  • Schedule job for getting network IVE (HEA) statistics if you use it in your IBM Power servers
    $ crontab -e
    
    0,5,10,15,20,25,30,35,40,45,50,55 * * * * /home/lpar2rrd/lpar2rrd/load_hea.sh > /home/lpar2rrd/lpar2rrd/load_hea.out 2>&1
    
Skip it in case of configuring Virtual Appliance

Apache download

  • AIX
    Apache 2.4.18 from IBM AIX Toolbox for Linux Applications
    Apache 2.4.4 from M. Perlz's archive.

  • Linux CentOS, RedHat
    # yum install httpd
    
  • Linux Debian, Ubuntu
    # apt-get install httpd
    

Apache configuration

Apache start / restart

    Command depends on Apache version, one of these should work:
    # apachectl restart
    # /opt/freeware/apache/sbin/apachectl restart
    # /etc/init.d/httpd restart
    # service apache2 restart
    

Test web pages

    Point your web browser to http://<your lpar2rrd host>/lpar2rrd/ as soon as you configure LPAR2RRD
    Do not forget to ensure that Apache starts after the OS reboot (there must be a starting script in /etc/rc.... )

Apache configuration per platform and version

  • Linux CentOS, RedHat


    Append at the end of /etc/httpd/conf/httpd.conf following
    # vi /etc/httpd/conf/httpd.conf
    
    AddHandler cgi-script .sh
    # DocumentRoot  "/home/lpar2rrd/lpar2rrd/www/"
    Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
    <Directory "/home/lpar2rrd/lpar2rrd/www/">
        AllowOverride AuthConfig
        Options Indexes FollowSymLinks
        Require all granted
    </Directory>
    # CGI-BIN
    ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
    <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
        AllowOverride AuthConfig
        SetHandler cgi-script
        Options ExecCGI FollowSymLinks
        Require all granted
    </Directory>
    
  • Linux Debian, Ubuntu : Apache2 (2.4+)


    Enable CGI-BIN
    # a2enmod cgi
    
    Append this at the end of /etc/apache2/apache2.conf
    # vi /etc/apache2/apache2.conf
    
    AddHandler cgi-script .sh
    Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
    <Directory "/home/lpar2rrd/lpar2rrd/www/">
        Options Indexes FollowSymLinks 
        Require all granted
    </Directory>
    # CGI-BIN
    ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
    <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
        AllowOverride None
        Options +ExecCGI +SymLinksIfOwnerMatch
        Require all granted
        SetHandler cgi-script
    </Directory>
    
  • Apache 2.4+ generally


    Configure httpd.conf, it might be in different locations (depends of distribution)
    • /etc/httpd/httpd.conf
    • /etc/httpd/conf/httpd.conf
    • /etc/opt/freeware/apache/httpd.conf (AIX)
    • /opt/freeware/etc/httpd/httpd.conf (AIX)
    • /opt/freeware/etc/httpd/conf/httpd.conf (AIX)

    1. Verify alias_module and cgi_module is loaded (uncommented it, the path can be different)
      LoadModule alias_module /opt/freeware/lib/httpd/modules/mod_alias.so
      LoadModule cgi_module /opt/freeware/lib/httpd/modules/mod_cgi.so
      
    2. cgi_module could depend on mpm_prefork_module like here.
      Allow it as well (uncomment it) and disable mpm_worker_module, mpm_event_module
      LoadModule mpm_prefork_module /opt/freeware/lib/httpd/modules/mod_mpm_prefork.so
      # LoadModule mpm_worker_module /opt/freeware/lib/httpd/modules/mod_mpm_worker.so
      # LoadModule mpm_event_module modules/mod_mpm_event.so
      <IfModule mpm_prefork_module>
           LoadModule cgi_module /opt/freeware/lib/httpd/modules/mod_cgi.so
      </IfModule>
      
    3. Append this at the end of httpd.conf
      AddHandler cgi-script .sh
      Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
      <Directory "/home/lpar2rrd/lpar2rrd/www/">
          Options Indexes FollowSymLinks 
          Require all granted
      </Directory>
      # CGI-BIN
      ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
      <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
          AllowOverride None
          SetHandler cgi-script
          Options ExecCGI FollowSymLinks
          Require all granted
      </Directory>
      
      Sometimes appears necessary to remove "Require all granted" lines in both directives.

  • Apache 1.3 - 2.3


    Configure httpd.conf, it might be in different locations (depends of distribution)
    • /etc/httpd/httpd.conf
    • /etc/httpd/conf/httpd.conf
    • /etc/opt/freeware/apache/httpd.conf (AIX)
    • /opt/freeware/etc/httpd/httpd.conf (AIX)
    • /opt/freeware/etc/httpd/conf/httpd.conf (AIX)

    # vi /etc/opt/freeware/apache/httpd.conf
    
    Alias /lpar2rrd  "/home/lpar2rrd/lpar2rrd/www/"
    <Directory "/home/lpar2rrd/lpar2rrd/www/">
        Options Indexes FollowSymLinks 
        Order allow,deny
        Allow from all
    </Directory>
    # CGI-BIN
    ScriptAlias /lpar2rrd-cgi/ "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi/"
    <Directory "/home/lpar2rrd/lpar2rrd/lpar2rrd-cgi">
        AllowOverride None
        Options ExecCGI FollowSymLinks
        Order allow,deny
        Allow from all
    </Directory>
    
  • SELinux changes


    You need to disable the SELinux service protection to be able to utilize basic web services.
    You must do it in case you get "(13)Permission denied:" in Apache error_log.
    You have 2 options:

    • Disable SELinux only for Apache
      Step 1. Query for the Boolean value you need to change:
      # getsebool -a | grep httpd_dis
        httpd_disable_trans --> off
      
      Step 2. Disable the SELinux protection:
      # setsebool -P httpd_disable_trans=1
      
      Step 3. Verify that the Boolean has changed:
      # getsebool -a | grep httpd_dis
        httpd_disable_trans --> on
      
    • Globally disable SELinux
      # setenforce Permissive
      # vi /etc/sysconfig/selinux
        SELINUX=disabled
      

OS agent is add-on feature for monitoring from operating system level.
It is monitoring CPU, memory utilization, paging, LAN and SAN traffic on all adapters.
It requires the OS agent deployment to every monitored LPAR.
The agent is written in Perl and calls basic OS commands to obtain required statistics like vmstat, lparstat and svmon.

OS agent architecture

Additional information about the OS agent: NMON support via the OS agent
HMC monitoring via the OS agent
IBM i (AS/400) OS agent

Prerequisites

  • Perl on each LPAR (each OS already contains it, no special modules necessary)
  • Opened TCP communication between each LPAR and LPAR2RRD server on port 8162.
    Connections are initiated from LPARs.
  • Additional disk space on LPAR2RRD server (about 40MB per each monitored LPAR)
  • Create preferable dedicated user lpar2rrd on each host with minimum rights
    # useradd -s /usr/bin/ksh  -c "LPAR2RRD agent user" -m lpar2rrd
    
  • In case of VIO server do not use padmin account and mkuser cmd, that would not work!
    Create the account under root in the same way as on usual AIX (like above useradd).

OS agent installation (client)

  • Get the latest OS agent from download page

  • AIX & Linux installation under root
    # rpm -Uvh lpar2rrd-agent-6.00-0.ppc.rpm
    # rpm -qa|grep lpar2rrd-agent
      lpar2rrd-agent-6.00-0
    
    AIX 7.1+ note: you might need to specify option "--ignoreos" if you get an error:
    "package lpar2rrd-agent-5.00-0 is for a different operating system"
    # rpm -Uvh --ignoreos lpar2rrd-agent-6.00-0.ppc.rpm
    
  • Linux Debian
    # apt-get install lpar2rrd-agent_6.00-0_all.deb
      lpar2rrd-agent-6.00-0
    
  • Oracle Solaris installation under root:
    # gunzip lpar2rrd-agent-5.00-0.solaris-i86pc.tar.gz
    # tar xf lpar2rrd-agent-5.00-0.solaris-i86pc.tar
    # pkgadd -d .
      The following packages are available:
      1  lpar2rrd-agent     LPAR2RRD OS agent 5.00
                            (i86pc) 5.00
     ...
    
    Oracle Solaris upgrade under root:
    # pkgrm lpar2rrd-agent
    # pkgadd -d .
    
  • Schedule its run every minute from the crontab on every LPAR.
    This line must be placed into lpar2rrd crontab:
    # su - lpar2rrd
    $ crontab -e 
    * * * * * /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl <LPAR2RRD-SERVER> > /var/tmp/lpar2rrd-agent.out 2>&1
    
    Replace <LPAR2RRD-SERVER> by hostname of your LPAR2RRD server.

  • You might need to add lpar2rrd user into /var/adm/cron/cron.allow under root user if above "crontab -e" fails.
    # echo "lpar2rrd" >> /var/adm/cron/cron.allow
    

LPAR2RRD server (daemon)

  • Edit etc/lpar2rrd.cfg and set following (if it is not already set):
    $ vi /home/lpar2rrd/lpar2rrd/etc/lpar2rrd.cfg
    
    LPAR2RRD_AGENT_DAEMON=1
    
  • The daemon is started when load.sh starts
    $ ./load.sh
      Starting LPAR2RRD daemon on port:8162
      ...
    
  • Assure it is running and listening on port 8162:
    $ ps -ef|grep lpar2rrd-daemon
      lpar2rrd 10617010 1 0 Mar 16 - 0:00 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/lpar2rrd-daemon.pl
    $ netstat -an| grep 8162
      tcp4  0  0  *.8162   *.*   LISTEN
    
  • OS agent data graphs will appear in the UI, use Ctrl-F5 to refresh your web browser

Troubleshooting

  • Client (agent) side:
    • Test if communication through the LAN is allowed.
      $ telnet  <LPAR2RRD-SERVER> 8162
        Connected to 192.168.1.1   .
        Escape character is '^]'.
      
      This is ok, exit either Ctrl-C or ^].

    • Check following agent files:
      data store: /var/tmp/lpar2rrd-agent-*.txt
      error log: /var/tmp/lpar2rrd-agent-*.err
      output log: /var/tmp/lpar2rrd-agent.out

    • run the agent from cmd line:
      $ /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl -d <LPAR2RRD-SERVER>
        ...
        Agent send     : yes : forced by -d 
        Agent send slp: sending wait: 4
        OS/HMC agent working for server: <LPAR2RRD-SERVER>
        store file for sending is /var/tmp/lpar2rrd-agent-<LPAR2RRD-SERVER>-lpar2rrd.txt
      
      It means that data has been sent to the server, all is fine
      Here is example when the agent is not able to sent data :
      $ /usr/bin/perl /opt/lpar2rrd-agent/lpar2rrd-agent.pl -d <LPAR2RRD-SERVER>
        ...
        Agent send     : yes : forced by -d 
        Agent send slp: sending wait: 1
        OS/HMC agent working for server: <LPAR2RRD-SERVER>
        store file for sending is /var/tmp/lpar2rrd-agent-<LPAR2RRD-SERVER>-lpar2rrd.txt
        Agent timed out after : 50 seconds /opt/lpar2rrd-agent/lpar2rrd-agent.pl:265
      
      It means that the agent could not contact the server.
      Check communication, port, above telnet example, DNS resolution of the server etc.

  • Server side:
    • test if the daemon on LPAR2RRD server is running, and checking the logs
      $ ps -ef|grep lpar2rrd-daemon
        lpar2rrd 10617010 1 0 Mar 16 - 0:00 /usr/bin/perl -w /home/lpar2rrd/lpar2rrd/bin/lpar2rrd-daemon.pl
      $ cd /home/lpar2rrd/lpar2rrd
      $ tail logs/error.log-daemon
      $ tail logs/daemon.out
        new server has been found and registered: Linux (lpar=linuxhost01)
        mkdir : /lpar2rrd/data/Linux/no_hmc/linuxhost01/
      
      It means that new OS agent has been registered from linuxhost01 (Linux stand-alone example)

    • Test if OS agent data is being stored on the LPAR2RRD server and have actual timestamp:
      $ cd /home/lpar2rrd/lpar2rrd
      $ ls -l data/<server name>/*/<lpar name>/*mmm
        -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/<hmc name>/<lpar name>/cpu.mmm
        -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/<hmc name>/<lpar name>/lan-en1.mmm
        -rw-r--r-- 2 lpar2rrd staff 10790264 Mar 17 16:16 data/<server name>/<hmc name>/<lpar name>/mem.mmm
        -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/<hmc name>/<lpar name>/pgs.mmm
        -rw-r--r-- 2 lpar2rrd staff  7193736 Mar 17 16:16 data/<server name>/<hmc name>/<lpar name>/san-vscsi0.mmm
        -rw-r--r-- 2 lpar2rrd staff  3597208 Mar 17 16:16 data/<server name>/<hmc name>/<lpar name>/san_resp-vscsi0.mmm
      $ find data -name mem.mmm -exec ls -l {} \;
        ...
      
  • In case of a problem check our forum or contact us via support@lpar2rrd.com.
    We would need this data for start of troubleshooting.

Notes

    LPAR2RRD agent is available since 3.56 release.
    You will not need to upgrade LPAR2RRD agents regularly with each LPAR2RRD upgrade.
    Read release notes if that is necessary.
    Check OS agent upgrade steps.

You have 2 options how to connect the HMC.
  • HMC REST API: it brings you physical adapter stats, enhanced configuration in compare to CLI access,
    it uses read-only user on the HMC with a password authentification
  • HMC CLI: it uses ssh-key based authorisation on HMC, read-only account

HMC REST API

It supports HMC v8+, use HMC CLI in case of older HMCs
It is supported since LPAR2RRD v6.00.

Configuration

  • Allow access from the LPAR2RRD host to all HMCs on port 12443 TCP
    You might either verify it manually from command line or run "test" in the UI after HMC addition.
    $ perl /home/lpar2rrd/lpar2rrd/bin/conntest.pl 192.168.1.1 12443
      Connection to "192.168.1.1" on port "12443" is ok
    
  • AIX: These 2 packages are mandatory on AIX, assure they are installed (on LPAR2RRD server only):
    # rpm -qa | egrep -i "Crypt-SSLeay|Net_SSLeay"
      perl-Crypt-SSLeay-0.57-2
      perl-Net_SSLeay.pm-1.55-3
    
    Download it from sourceforge.net or lpar2rrd.com
    # rpm -Uvh perl-Crypt-SSLeay-0.57-2.aix6.1.ppc.rpm
    
    Download it from sourceforge.net or lpar2rrd.com
    # rpm -Uvh --nodeps --replacefiles perl-Net_SSLeay.pm-1.55-3.aix6.1.ppc.rpm
    
  • Create lpar2rrd user on each HMC with read-only role, with password.

  • Allow remote access via the web:
    HMC ➡ Manage User Profiles and Access ➡ select lpar2rrd ➡ modify ➡ user properties ➡ Allow remote access via the web
    IBMPower REST API remote access


  • LPAR2RRD requires HMC LongTermMonitor enabled. It is part of Performance Monitoring in HMC Workplace.
    HMC Management ➡ Operations ➡ Change Performance Monitoring Settings ➡ All On
    This performance monitoring can store up to 366 days of data. LPAR2RRD uses performance data only 30 minutes old so you can set the minimum number of days to store performance data (1 day).

    IBM Power Systems REST API cfg


  • Configure HMC access in the LPAR2RRD UI, start at the settings button on the top right.
    IBMPower REST API cfg
     
    IBMPower REST API cfg

  • If you use IVM (Integrated Virtualization Manager) as data source then they must excluded as they have no REST API
    It must be configured for HMC CLI access (lslparutil).
    Edit etc/lpar2rrd.cfg and place all IVMs into new entry:
    IVM_LIST="ivm1 ivm2 ..."
    
  • Assure you have a cron job for HMC REST API in place (upgrade script might do it automatically)
    Skip it on the Virtual Appliance, it is already there.
    # su - lpar2rrd
    $ crontab -l | egrep "load_hmc_rest_api.sh"
    $
    
    Add if they do not exist like above
    $ crontab -e
    
    # IBM Power Systems - REST API
    0,20,40 * * * * /home/lpar2rrd/lpar2rrd/load_hmc_rest_api.sh > /home/lpar2rrd/lpar2rrd/load_hmc_rest_api.out 2>&1
    
  • Wait about 30 minutes, then refresh the web browser cache by Ctrl-F5
    http://<your web server>/stor2rrd/

Migration from older HMC CLI

If you already use LPAR2RRD with usage CLI (lslparutil via SSH) then you can simply migrate to REST API access.
Just follow same steps as above.
Once you connect the first HMC then all communications will be switched to REST API.
Original HMC CLI (lslparutil) will be disabled.
Connect all your HMCs then to do not lose data.

Examples

Check our live demo site where all IBM Power servers data came through REST API.

Issues

Some older IBM Power Systems models (firmwares) might report warnings when is enabled "Data Collection" on the HMC.
It is not connected to LPAR2RRD!
Both issues we have seen were confirmed by IBM support as informative only, no real problem.
  • This might appear on a LPAR having physical FC adapters which are not connected in its errpt log
    LABEL:          FCA_ERR4 
    IDENTIFIER:     7BFEEA1F 
    ...
    Class:          H 
    Type:           TEMP 
    WPAR:           Global 
    Resource Name:  fcs3 
    Resource Class: adapter 
    Resource Type:  7710322514101e0 
    Location:       UXXCB.001.WXXXXXX-P1-C3-T4 
    Description 
      LINK ERROR 
    
  • This appeared on an old server with recommendation to upgrade firmware
    LABEL:          SCAN_ERROR_CHRP 
    IDENTIFIER:     BFE4C025 
    ...
    Class:          H 
    Type:           PERM 
    WPAR:           Global 
    Resource Name:  sysplanar0 
    Resource Class: planar 
    Resource Type:  sysplanar_rspc 
    Description 
      UNDETERMINED ERROR 
    Failure Causes 
      UNDETERMINED 
            Recommended Actions 
            RUN SYSTEM DIAGNOSTICS. 
    
    To avoid that you can disable "Data Collection" on the HMC for that particular server.
    This server can be further disabled even in LPAR2RRD REST API processing and it will continue via HMC CLI (lslparutil)
    Let us know, UI form for that is not implemented yet, however manually it is possible ...



Type

Source

Sample rate

Data download

HMC
HMC REST API1 minute20 mins
HMC CLI1 hour
IVM CLI
OS agent
AIX1 minute~20 mins
VIOS
Linux on POWER
Linux x86
IBM i (AS/400)
NMON
AIX1 minute10 mins or
offline batch
VIOS
Linux on POWER
Linux x86


Skip it in case of configuring Virtual Appliance

Install LPAR2RRD server

  • Download the latest LPAR2RRD server
    Upgrade your already running LPAR2RRD instance.

  • Install it:
    # su - lpar2rrd
    $ tar xvf lpar2rrd-6.XX.tar
    $ cd lpar2rrd-6.XX
    $ ./install.sh
    $ cd /home/lpar2rrd/lpar2rrd
    
  • Schedule to run it from lpar2rrd crontab (it might already exist there)
    $ crontab -l | grep load.sh
    $
    
    Add if it does not exist as above
    $ crontab -e
    
    # LPAR2RRD UI
    0 * * * * /home/lpar2rrd/lpar2rrd/load.sh > /home/lpar2rrd/lpar2rrd/load.out 2>&1 
    
    Assure there is just one such entry in crontab.

  • You might need to add lpar2rrd user into /var/adm/cron/cron.allow if crontab command fails
    Allow it for lpar2rrd user as root user.
    # echo "lpar2rrd" >> /var/adm/cron/cron.allow
    
  • Assure you have a cron job for HMC REST API in place (upgrade script might do it automatically)
    Skip it on the Virtual Appliance, it is already there.
    $ crontab -l | grep "load_hmc_rest_api.sh"
    $
    
    Add it if it does not exist like above
    $ crontab -e
    
    # HMC REST API support
    0,20,40 * * * *  /home/lpar2rrd/lpar2rrd/load_hmc_rest_api.sh > /home/lpar2rrd/lpar2rrd/load_hmc_rest_api.out 2>&1 
    
  • Initial start from cmd line:
    $ cd /home/lpar2rrd/lpar2rrd
    $ ./load.sh
    
  • Go to the web UI: http://<your web server>/stor2rrd/
    Use Ctrl-F5 to refresh the web browser cache.