Posted by mike on Jul 26, 2010 in VMware | View Comments
The current issue I have come across in our HP storage environment is an issue with the storage controller cache battery modules. We had a module fail recently on one of our 8100 series EVAs. There can be up to four modules per controller. In our environment, we are using two modules per controller.
A healthy set of modules looks like this:
Now, for the EVA we have problems with, it looks like this:
This problem occurred after we had this particular module fail. We received a replacement from HP and swapped it out. However, after a few days, it was marked as failed again. Again we received a replacement from HP, and swapped it out. A few days later, same result. In contacting HP a third time, I explained what had occurred. In response, I received this notification:
This is just another somewhat oddball error that we deal with on a regular basis. Now, on to the fix! To restart the controller in question, first note as per Command View which controller is in question. In my case, it is Controller A (just follow the bang indicators)
A restart of the controller should be done during your change / maintenance window (all those years of ITIL ingrained in me!). To do so, you have a few choices.
The first is via Command View:
On the controller’s page, hit shutdown, then restart and the controller (A/B).
The second is via the SSSU utility (installs as part of the Command View install):
Restart controller A, but not its peer controller:
RESTART “\Hardware\Rack 1\Enclosure 7\Controller A” NOALL_PEERS
Note that when restarting the controller, if it is the master controller the vdisks will transfer to the other controller without any downtime. In my experience with the EVAs, they are a touchy lot. I prefer using the SSSU utility for a halfway decent command line interface. Pretty powerful too. I’ll be writing up a blog posting discussing good uses for SSSU in the future.
read more
Posted by mike on Jul 16, 2010 in VMware | View Comments
So this is an error / issue we’ve had to live with for some time. It’s a bit strange to be sure. Essentially in Command View, the Disk Group occupancy is completely incorrect, and continues to grow. Yes, that’s right, I said grow. For some photo-visual enjoyment, here is what a NORMAL Disk Group looks like.
So as you can see, we have a total Capacity (which is correct and accurate), and the Occupancy. The information listed is correct, and it’s refreshing. Now, for the problematic Disk Group (and please ignore the name as it was before my time).
Here we see the correctly calculated Capacity, but lo! What’s this? What on earth is going on with that Occupancy? It seemingly continues to grow a bit here and there as we take snapshots of various. HP doesn’t know what’s wrong either. When we upgrade from Command View 9.0 to 9.1, it did not fix the issue either. The big problem here is that we have to figure out how much space is actually used since we’ll never get alerted. It is a pain. I can turn to Replication Solutions Manager to obtain the correct size by simply adding all the luns (easy way to view them, as opposed to Command View), so that’s the workaround. Not very satisfying if you consider how much money was spent on these beasts.
read more
Posted by mike on Jul 16, 2010 in VMware | View Comments
This is a walkthrough for HP Command View for EVA. Part of my daily routine is to take a jaunt through CV, to check things over and look for alerts that I may not already be aware of.
Launched from the shortcut on the Windows host, or is accessible via a web browser at https://servername:2372
You are presented with the login screen:
It is here that you will also see the version number (down below). We have not yet upgraded this server to version 9.2 as we are building a VM to run Command View from and retire this physical box.
After login, you are greeted by the overview page. Here you will see all the EVAs listed, as well as the stats for your overall environment. The first thing of note is that two of my EVAs have bang lights on them, indicating something is amiss. I’ll investigate both as part of my next posting.
On the top right, there are some hyperlinks, your login id, and the ip of the Command View server. Most are self explanatory. Server Options provides for a place to enter license codes, setup RSM relationships, and a few other features. I seldom visit this page, so like once a year?
Moving right along, let’s take a look at a healthy EVA, in this case DS-SAN-2:
As you can see from the above, a healthy EVA has a number of folders beneath it. It breaks out into many subfolders. On the right hand side are the numbers. It would be here that you can get the logs for HP support (I’ll cover that in a separate blog post). You can see the current capacity level, view the Version level (this is the XCS code release running on the EVA. 6220 is the latest for the 8100 series). The left column is broken down like such:
Virtual Disks – this is where the luns (vdisks in EVA-speak) are listed. The folder structure is entirely man-made. That is to say, it’s for human organizational purposes (and plays an important role for setting up RSM jobs. More on that will be covered on a separate posting).
Hosts – here is where hosts are setup in Command View. You will provide the host’s OS, and it’s WWN’s for fiber cards. Hosts MUST be setup on every single EVA that you want to present disks from. Annoying, I know.
Disk Groups – these are comprised of physical disks. Best practice says to build these in multiples of 8, and of all the same speed and size. You can choose to not follow these and your performance will suck majorly. I’ve worked on rebuilding two of the 4 EVAs that had improperly constructed Disk Groups. It is PAINFUL to correct, but I’m glad I did. I will cover that also as a separate blog post. FYI: The ungrouped disks folder is for disks that have failed or been ungrouped on purpose. Ungrouping takes time as the EVA moves data from the drive to free it up to be removed or replaced.
Data Replication – it is here that you can create DR groups. This allows you to replicate (synchronous or asynchronously) between EVA arrays. A replication group is comprised of 1 or more vdisks. Sounding like a broken record, I will have a separate posting on replication.
Hardware – it is here that you can check out the status of the hardware. Both controllers are listed, as are all the disks. If there are hardware issues (per a bang light) then you can come here to find out why. The status of failed items is usually fairly straightforward and understandable as to what happened and what should be done.
read more
Posted by mike on Jul 12, 2010 in VMware | View Comments
I’d like to start discussing the storage I have at my work. I’ve had some good experiences in learning and working with it, and I’d like to share! I haven’t found alot specifically regarding working with the EVAs like I do daily, so I hope some folks find this useful!
So, as an overview for the environment, we currently have 3 8100 series and one 4400 series. EVAs arrays use two controllers, and our models are active/active. Our VMware environment at work is currently a VI3 schmorgesborg and we are planning an update (well migration) to vSphere. For SAN switches, we have older Mcdata’s that are workhorses! We manage them using EFCM software (also java, sigh).
HP provides Command View for EVA as the management tool for the EVAs. This installs and runs on Windows, and is now supported running in a VM as of the current release (9.2). Replication is handled with Business Copy and Continuous Access (separately licensed features). Automation of replication is handled using Replication Solutions Manager, which is a java-based App. :-/
For my next post, I’ll be giving a walkthrough of Command View and discuss how the EVAs we have are configured. Lots of screenshots too!
read more
Posted by mike on May 23, 2010 in VMware | View Comments
Welp, thankfully I have a rack and things are coming together nicely. I picked up a UPS also, and two Linksys/Cisco SLM2008s. Also grabbed a third DL360 G4. Once things are cabled up and running today, I plan to do a fresh install of the entire environment to VI3. First things first though, I need to build a win2k3 vm for a template, then deploy a SQL2005 box and virtualcenter server. Then I need to setup openfiler for iscsi storage (the DL580 G2 has a drive cabinet attached with 800~GB raid 5 storage (with a hot spare too!)). I’m also thinking about setting up and playing with vyatta to see how well it can handle routing. All in good time though. Here’s some pics of the setup. I’ll be blogging more regularly with my machinations. And I fully realize doing this means I’ll be totally showing off for the ladies 
Also, the nice thing about having the WRT54G running tomato down in the basement in wireless bridge mode means I can sit upstairs with a lemonaide in the sunshine and VM to my hearts content without having to run cat5 everywhere!
read more
