[Ipmitool-devel] IPMI on Penguin Computing Relion 1600

[Michael Carmack]

Hi all. I'm having a bit of a hard time getting remote ipmitool commands
to work on a Relion 1600 from Penguin Computing. Local ipmitool commands
work fine, but when I try to connect from another machine, I start
running into problems...

This is a rather verbose email, as I'm going to go through everything I
do step by step with lots of output. Hopefully someone will be able to
see where I am going wrong.

First of all, let's look at the default settings that the machien
shipped with:

====================================================

***@penguin$ ipmitool lan print 2
Set in Progress : Set Complete
Auth Type Support : NONE MD5 PASSWORD
Auth Type Enable : Callback : NONE MD5 PASSWORD
: User : NONE MD5 PASSWORD
: Operator : NONE MD5 PASSWORD
: Admin : NONE MD5 PASSWORD
: OEM : NONE MD5 PASSWORD
IP Address Source : Static Address
IP Address : 192.168.2.100
Subnet Mask : 255.255.255.0
MAC Address : 00:a0:d1:e5:d1:8b
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
Default Gateway IP : 192.168.2.203
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 0,1,2,3
Cipher Suite Priv Max : Not Available

=======================================================

Now I'm going to use ipmitool to set the IP address to something I can
access from the Internet. First, we'll look at my network settings for
eth0 on the server:

=======================================================

***@penguin$ ifconfig eth0
eth0 Link encap:Ethernet HWaddr 00:A0:D1:E5:D1:88
inet addr:XXX.XXX.XXX.206
Bcast:XXX.XXX.XXX.255
Mask:255.255.255.0


***@penguin$ route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
XXX.XXX.XXX.0 0.0.0.0 255.255.255.0 U 0 0
0 eth0
0.0.0.0 XXX.XXX.XXX.1 0.0.0.0 UG 0 0
0 eth0

=================================================================

The server is configured for XXX.XXX.XXX.206, so to avoid any potential
conflicts, I'm going to set the BMC for XXX.XXX.XXX.205 using ipmitool:

==================================================================


***@penguin$ ipmitool lan set 2 ipaddr XXX.XXX.XXX.205
Setting LAN IP Address to XXX.XXX.XXX.205

***@penguin$ ipmitool lan set 2 defgw ipaddr XXX.XXX.XXX.1
Setting LAN Default Gateway IP to XXX.XXX.XXX.1

***@penguin$ ipmitool lan print 2
Set in Progress : Set Complete
Auth Type Support : NONE MD5 PASSWORD
Auth Type Enable : Callback : NONE MD5 PASSWORD
: User : NONE MD5 PASSWORD
: Operator : NONE MD5 PASSWORD
: Admin : NONE MD5 PASSWORD
: OEM : NONE MD5 PASSWORD
IP Address Source : Static Address
IP Address : XXX.XXX.XXX.205
Subnet Mask : 255.255.255.0
MAC Address : 00:a0:d1:e5:d1:8b
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
Default Gateway IP : XXX.XXX.XXX.1
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 0,1,2,3
Cipher Suite Priv Max : Not Available

=======================================================

The public IP address is now configured. Let's check the channel access:

========================================================

***@penguin$ ipmitool channel getaccess 2 1
Maximum User IDs : 16
Enabled User IDs : 14

User ID : 1
User Name :
Fixed Name : Yes
Access Available : call-in / callback
Link Authentication : disabled
IPMI Messaging : enabled
Privilege Level : ADMINISTRATOR

========================================================

And the general information:

========================================================

***@penguin$ ipmitool channel info 2
Channel 0x2 info:
Channel Medium Type : 802.3 LAN
Channel Protocol Type : IPMB-1.0
Session Support : multi-session
Active Session Count : 0
Protocol Vendor ID : 7154
Volatile(active) Settings
Alerting : enabled
Per-message Auth : enabled
User Level Auth : enabled
Access Mode : always available
Non-Volatile Settings
Alerting : enabled
Per-message Auth : enabled
User Level Auth : enabled
Access Mode : always available

=========================================================

And now I'm going to set the password to 'admin':

==========================================================

***@penguin$ ipmitool lan set 2 password admin
Password set for user 1

==========================================================

And now, finally, I go to my other computer and try to use ipmitool with
both lan and lanplus. Here is the failure:

===========================================================

***@a0$ /pkg/ipmitool/1.8.9/bin/ipmitool -I lanplus -H XXX.XXX.XXX.205
chassis status
Password:
Error: Unable to establish IPMI v2 / RMCP+ session
Error sending Chassis Status command
***@a0$ /pkg/ipmitool/1.8.9/bin/ipmitool -I lan -H XXX.XXX.XXX.205
chassis status
Password:
Error: Unable to establish LAN session
Error sending Chassis Status command

=============================================================

I feel like there is some simple, obvious thing that I am overlooking.
Any suggestions?

[David A. Ranch]

A few things stick out in my mind:

1. Since the MAC addresses are different for both the IPMI and eth0,
this means that, potentially, you should be able to ping the IPMI card
and get responses. Does that work?

2. When you did a "lan print 2", there weren't any Cipher suites shown
for a given privledge level. Maybe your machine's IPMI BMC only
supports IPMI 1.0 or "-I lan". For example, this is what I see:
--
RMCP+ Cipher Suites : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14
Cipher Suite Priv Max : Xaaaaaaaaaaaaaa
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
--

3. Finally, as I understand it, you CANNOT use the local machine to
communicate to the BMC via the Network. The previous connections you
were making was using the internal Open or "kcs" interface. If you want
to do IPMI testing via the network interface, you need to use some other
machine.

--David

Post by Michael Carmack
Hi all. I'm having a bit of a hard time getting remote ipmitool commands
to work on a Relion 1600 from Penguin Computing. Local ipmitool commands
work fine, but when I try to connect from another machine, I start
running into problems...
This is a rather verbose email, as I'm going to go through everything I
do step by step with lots of output. Hopefully someone will be able to
see where I am going wrong.
First of all, let's look at the default settings that the machien
====================================================
Set in Progress : Set Complete
Auth Type Support : NONE MD5 PASSWORD
Auth Type Enable : Callback : NONE MD5 PASSWORD
: User : NONE MD5 PASSWORD
: Operator : NONE MD5 PASSWORD
: Admin : NONE MD5 PASSWORD
: OEM : NONE MD5 PASSWORD
IP Address Source : Static Address
IP Address : 192.168.2.100
Subnet Mask : 255.255.255.0
MAC Address : 00:a0:d1:e5:d1:8b
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
Default Gateway IP : 192.168.2.203
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 0,1,2,3
Cipher Suite Priv Max : Not Available
=======================================================
Now I'm going to use ipmitool to set the IP address to something I can
access from the Internet. First, we'll look at my network settings for
=======================================================
eth0 Link encap:Ethernet HWaddr 00:A0:D1:E5:D1:88
inet addr:XXX.XXX.XXX.206
Bcast:XXX.XXX.XXX.255
Mask:255.255.255.0
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use
Iface
XXX.XXX.XXX.0 0.0.0.0 255.255.255.0 U 0 0
0 eth0
0.0.0.0 XXX.XXX.XXX.1 0.0.0.0 UG 0 0
0 eth0
=================================================================
The server is configured for XXX.XXX.XXX.206, so to avoid any potential
==================================================================
Setting LAN IP Address to XXX.XXX.XXX.205
Setting LAN Default Gateway IP to XXX.XXX.XXX.1
Set in Progress : Set Complete
Auth Type Support : NONE MD5 PASSWORD
Auth Type Enable : Callback : NONE MD5 PASSWORD
: User : NONE MD5 PASSWORD
: Operator : NONE MD5 PASSWORD
: Admin : NONE MD5 PASSWORD
: OEM : NONE MD5 PASSWORD
IP Address Source : Static Address
IP Address : XXX.XXX.XXX.205
Subnet Mask : 255.255.255.0
MAC Address : 00:a0:d1:e5:d1:8b
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
Default Gateway IP : XXX.XXX.XXX.1
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 0,1,2,3
Cipher Suite Priv Max : Not Available
=======================================================
========================================================
Maximum User IDs : 16
Enabled User IDs : 14
User ID : 1
Fixed Name : Yes
Access Available : call-in / callback
Link Authentication : disabled
IPMI Messaging : enabled
Privilege Level : ADMINISTRATOR
========================================================
========================================================
Channel Medium Type : 802.3 LAN
Channel Protocol Type : IPMB-1.0
Session Support : multi-session
Active Session Count : 0
Protocol Vendor ID : 7154
Volatile(active) Settings
Alerting : enabled
Per-message Auth : enabled
User Level Auth : enabled
Access Mode : always available
Non-Volatile Settings
Alerting : enabled
Per-message Auth : enabled
User Level Auth : enabled
Access Mode : always available
=========================================================
==========================================================
Password set for user 1
==========================================================
And now, finally, I go to my other computer and try to use ipmitool with
===========================================================
chassis status
Error: Unable to establish IPMI v2 / RMCP+ session
Error sending Chassis Status command
chassis status
Error: Unable to establish LAN session
Error sending Chassis Status command
=============================================================
I feel like there is some simple, obvious thing that I am overlooking.
Any suggestions?
------------------------------------------------------------------------
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDE
V
_______________________________________________
Ipmitool-devel mailing list
https://lists.sourceforge.net/lists/listinfo/ipmitool-devel

[Michael Carmack]

Post by David A. Ranch
1. Since the MAC addresses are different for both the IPMI and eth0,
this means that, potentially, you should be able to ping the IPMI card
and get responses. Does that work?

Nope. Nothing comes back when I try to ping the card.

Post by David A. Ranch
2. When you did a "lan print 2", there weren't any Cipher suites shown
for a given privledge level. Maybe your machine's IPMI BMC only

Yes, I noticed that also, but I wasn't sure what it meant or if it
actually implied something was wrong. The machine is certainly
advertised as supporting IPMI 2.0 though, as you can see here:

http://www.penguincomputing.com/index.php?option=com_content&task=view&id=326&Itemid=474

I've tried the ipmitool command with both "-I lan" and "-I lanplus", and
both end up with the same result: Several seconds of hanging, followed
by "Unable to establish LAN session".

Post by David A. Ranch
3. Finally, as I understand it, you CANNOT use the local machine to
communicate to the BMC via the Network. The previous connections you
were making was using the internal Open or "kcs" interface. If you want
to do IPMI testing via the network interface, you need to use some other
machine.

Yes, that is my understanding also. All of the remote ipmitool commands
were attempted from a different machine.

[Al Chu]

Post by Michael Carmack
The server is configured for XXX.XXX.XXX.206, so to avoid any
potential conflicts, I'm going to set the BMC for XXX.XXX.XXX.205

This is just a guess. Since you use a different IP address than the
node, maybe XXX.XXX.XXX.205 hasn't been resolved to the right MAC
address. You can check your ARP cache via /sbin/arp. If it isn't, you
might want to see if gratuitous arps are turned on in your BMC
configuration.

Al

[Michael Carmack]

Post by Al Chu

Post by Michael Carmack
The server is configured for XXX.XXX.XXX.206, so to avoid any
potential conflicts, I'm going to set the BMC for XXX.XXX.XXX.205

This is just a guess. Since you use a different IP address than the
node, maybe XXX.XXX.XXX.205 hasn't been resolved to the right MAC
address. You can check your ARP cache via /sbin/arp. If it isn't, you
might want to see if gratuitous arps are turned on in your BMC
configuration.

It doesn't show up when I run arp, but it also doesn't look like turning
on gratuitous arps has any effect on this card. At least if it does,
it's not reflected in the "print" command:

===========================================

***@penguin$ ipmitool lan set 2 arp generate on

***@penguin$ ipmitool lan print 2
Set in Progress : Set Complete
Auth Type Support : NONE MD5 PASSWORD
Auth Type Enable : Callback : NONE MD5 PASSWORD
: User : NONE MD5 PASSWORD
: Operator : NONE MD5 PASSWORD
: Admin : NONE MD5 PASSWORD
: OEM : NONE MD5 PASSWORD
IP Address Source : Static Address
IP Address : XXX.XXX.XXX.205
Subnet Mask : 255.255.255.0
MAC Address : 00:a0:d1:e5:d1:8b
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
Default Gateway IP : XXX.XXX.XXX.1
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 0,1,2,3
Cipher Suite Priv Max : Not Available

=========================================

Also, I probably should have mentioned that I originally tried setting
this up using the same IP address as eth0 (XXX.XXX.XXX.206). It was only
after that failed that I started trying a separate IP address (.205).
The errors were the same whether I used .206 or .205.

[David A. Ranch]

In your original email, you posted the MAC address of your eth0 NIC.
Looking in this URL, that's a Inventec MAC

http://www.coffer.com/mac_find/?string=00%3AA0%3AD1%3AE5%3AD1%3A88


Yet.. looking at Penguin Computing's page, it says:
--
On-Board LAN 2 x Intel 10/100/1000 ethernet, dedicated 10/100 management
interface
--

So.. the question is.. WHICH Ethernet port are you trying to use here?
I've seen IPMI setups like this where you MUST use the "management"
Ethernet for IPMI traffic. Sometimes you can also use this Ethernet
port for general traffic but this isn't always the case.

If you are indeed using this management interface (not the Intel GE
ports), do you know if the BMC really get's it's own MAC address (like
how Broadcom and very new Intel chips do it) or does it share the MAC
with this Inventec MAC (how older Intel chipsets did things. On my
Supermicro boards, they label one MAC for eth0 and then another MAC on
the IPMI daughter card.

This is a critical difference you need to know.

--David

Post by Michael Carmack

Post by Al Chu

Post by Michael Carmack
The server is configured for XXX.XXX.XXX.206, so to avoid any
potential conflicts, I'm going to set the BMC for XXX.XXX.XXX.205

This is just a guess. Since you use a different IP address than the
node, maybe XXX.XXX.XXX.205 hasn't been resolved to the right MAC
address. You can check your ARP cache via /sbin/arp. If it isn't,

you

Post by Al Chu
might want to see if gratuitous arps are turned on in your BMC
configuration.

It doesn't show up when I run arp, but it also doesn't look like turning
on gratuitous arps has any effect on this card. At least if it does,
===========================================
Set in Progress : Set Complete
Auth Type Support : NONE MD5 PASSWORD
Auth Type Enable : Callback : NONE MD5 PASSWORD
: User : NONE MD5 PASSWORD
: Operator : NONE MD5 PASSWORD
: Admin : NONE MD5 PASSWORD
: OEM : NONE MD5 PASSWORD
IP Address Source : Static Address
IP Address : XXX.XXX.XXX.205
Subnet Mask : 255.255.255.0
MAC Address : 00:a0:d1:e5:d1:8b
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
Default Gateway IP : XXX.XXX.XXX.1
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 0,1,2,3
Cipher Suite Priv Max : Not Available
=========================================
Also, I probably should have mentioned that I originally tried setting
this up using the same IP address as eth0 (XXX.XXX.XXX.206). It was only
after that failed that I started trying a separate IP address (.205).
The errors were the same whether I used .206 or .205.
------------------------------------------------------------------------
-
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share
your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDE
V
_______________________________________________
Ipmitool-devel mailing list
https://lists.sourceforge.net/lists/listinfo/ipmitool-devel

[Michael Carmack]

Post by David A. Ranch
So.. the question is.. WHICH Ethernet port are you trying to use here?
I've seen IPMI setups like this where you MUST use the "management"
Ethernet for IPMI traffic. Sometimes you can also use this Ethernet
port for general traffic but this isn't always the case.

Wow, that was it! All this time I was trying to go over eth0. As soon as
I strung a separate cable to the Management Port everything started working.

It actually did occur to me to try the Management Port at one point, but
something else must have been misconfigured at the time and it didn't
work, so I wrote it off and went back to the shared eth0 trials.

Thanks a million. If we ever run into each other, drinks are on me :-)

[Michael Carmack]

For the sake of anyone else who may be struggling with getting IPMI
running on the Relion, I've typed up some instructions:

http://karmak.org/2007/relion-1600-ipmi

This should work for both the Relion 1600 and 2600 (as far as I know
they're exactly the same except one is 2u).

[Al Chu]

Post by Michael Carmack
It doesn't show up when I run arp, but it also doesn't look like turning
on gratuitous arps has any effect on this card. At least if it does,

As a test, perhaps you could manually put the ip -> mac mapping into
your arp cache?

If your machine can't resolve the IP -> MAC address, then IPMI will
surely not work.

Al

Post by Michael Carmack
===========================================
Set in Progress : Set Complete
Auth Type Support : NONE MD5 PASSWORD
Auth Type Enable : Callback : NONE MD5 PASSWORD
: User : NONE MD5 PASSWORD
: Operator : NONE MD5 PASSWORD
: Admin : NONE MD5 PASSWORD
: OEM : NONE MD5 PASSWORD
IP Address Source : Static Address
IP Address : XXX.XXX.XXX.205
Subnet Mask : 255.255.255.0
MAC Address : 00:a0:d1:e5:d1:8b
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
Default Gateway IP : XXX.XXX.XXX.1
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 0,1,2,3
Cipher Suite Priv Max : Not Available
=========================================
Also, I probably should have mentioned that I originally tried setting
this up using the same IP address as eth0 (XXX.XXX.XXX.206). It was only
after that failed that I started trying a separate IP address (.205).
The errors were the same whether I used .206 or .205.
-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys-and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV
_______________________________________________
Ipmitool-devel mailing list
https://lists.sourceforge.net/lists/listinfo/ipmitool-devel

[Michael Carmack]

Post by Al Chu

Post by Michael Carmack
It doesn't show up when I run arp, but it also doesn't look like turning
on gratuitous arps has any effect on this card. At least if it does,

As a test, perhaps you could manually put the ip -> mac mapping into
your arp cache?
If your machine can't resolve the IP -> MAC address, then IPMI will
surely not work.

It's ok, the problem has been solved. There is a separate Management
Port on the server that was apparently supposed to be used instead of
sharing a line with eth0 (see my reply to David for details).