Virtual Hints Things I've learned managing an ESX environment.

7Sep/110

vmworld 2011 Monday General Session

At vmworld this year, during the Monday general session, we all learned about vmware’s vision of virtualization and cloud computing during the Monday general session. The session started with some obligatory ads for platinum and diamond conference sponsors, then jumped right in to announcements and vision of where vmware (and virtualization in general) is headed. Paul proved that vmware has a fantastic understanding of virtualization and an ambitious and exciting vision for the future.

During his welcome, Rick Jackson talked about the environment powering the hands on labs for vmworld 2011. vmware decided to eat their own dog food this year, so they hosted all of the labs on public clouds. There was no on-site safety net like in years past. The 3 public providers used were switch, terramark and colt.

Paul Maritz came on and talked about the progression of computing and end user interactions. Starting with basic automated book keeping in the 1970’s and moving on to ERP systems, CRM, e-commerce, etc. Today, the next step in this progression, is to move to a more user centric approach. No longer will the user’s interaction with the data be dictated by IT staff. In 3 years, PC’s will account for less than 20% of all client devices. Users are beginning to consume their data and applications wherever they are and via many different devices. Cloud computing is the only way to provide the elastic resources necessary to facilitate this end user experience. It is our job as IT administrators to create new data fabrics and computing resources to enable real-time, highly scalable analytics and commerce.

vmware has laid the foundation for this next stage of IT by providing ways to leverage current infrastructure (compute, storage and networking) to bridge the gap to the next generation of computing. Up to now, the reasons to virtualize have primarily been capital savings and efficiencies. As services become more accessible to end users, these savings will begin to expand into operational savings as well. Employees in the organization will become more efficient as we allow users to get to their data and applications whenever and wherever they want in the fastest, most efficient way possible.

Every year, vmware has announced a major version of ESX/vSphere. 2011 is no exception. vSphere 5.0 just went GA, and View 5.0 was announced. One of the reasons vmware pushes forward on regular update schedules like this is because they believe that the hypervisor layer needs to be like hardware: stable, with consistent updates and refreshes The regular update schedule will begin to be applied to other products such as view, vCenter Operations Manager, etc. New versions of products will be released on a common schedule. These new versions will all be tested together prior to release and version numbers will be brought into sync to minimize confusion.

To see where the IT industry is heading, vmware looked to young programmers. What are they doing? They rely heavily on frameworks to reduce complexity and aren’t concerned with the nuts and bolts under the hood. They want services that are fast, reliable and easy to use. vFabric is a suite of tools that vmware is releasing to provide a stable base for these new frameworks to live on. vFabric Data Director was announced as a simplified database platform, built on PostgreSQL, optimized for virtualization.

Cloud Foundry is the tool that will allow you to plug into vFabric to create and scale applications. The goal is a portable and language independent framework for developers to work with. To help adoption of Cloud Foundry, vmware is releasing MicroCloud Foundry. It is a local, fully functional Cloud Foundry installation to allow developers to work when and where they want.

View 5.0 is continuing to build on the success of previous versions. There are more clients; every platform will be supported. VoIP and UM integration are included and there has been a lot of work to ensure improvements in bandwith usage.

Project Horizon was introduced as a way for IT increase the end user experiences while maintaining a secure environment. A way to associate applications and permissions with people, not devices and programs. One of the pieces of Project Horizon includes partnerships with Samsung and LG, offering phone based virtualization. Multiple versions of Android can run on a single phone enabling seamless separation of work and personal data and applications. This increases security without affecting usability.

At vmworld in 2009, Paul Maritz announced that a major virtualization tipping point had been reached. 50% of all new server deployments were virtual. Virtualization success has continued and now more than 50% of all servers worldwide are virtualized. What about the next 50%? The remaining workloads are largely mission critical apps, and vSphere 5 was designed to address concerns over virtualizing these workloads.

vSphere Essentials was released as a virtualization on-ramp for small and medium businesses to get started in virtualization. There are some tools to remove some of the operational and cost barriers for virtualization. The VSA is included to remove the shared storage dependencies and reduce cost. It allows an ESX host to share local storage with other nodes, enabling features such as vMotion and fault tolerance.

Random Factoids:
60,000 vmug members
19,000+ attendees
800,000 vm admins
New vm provisioned every 6 seconds
Over 20 Million virtual machines
5.5 vmotions per second - more virtual machines in flight than airplanes.
68,000 VCP’s
1,650 ISV partners
6,000 certified apps
48 vCloud service providers
vSphere 5.0 involved 2M man hours of QA and 1M engineering hours
Next year vmworld will be August 27-30, 2012 in San Francisco

The vision of the future that was presented is amazing. Several of the technologies and facts presented were mentioned and hinted at in past years, but it always seems to take me 2-3 times hearing it to wrap my head around the concepts. At the end of the keynote, there were no doubts left that vmware far and away the leader and innovator in the virtualization world. They see where IT is heading and are providing the technologies to get us all there.

Be the first to like.
Tagged as: , No Comments
17Sep/100

Top vmware blogs

Eric Siebert at vsphere land has a lot of fantastic resources on his website. One of my favorite features is the vpad; a list of the different vmware blogs. Every year he polls his readers to find the top vmware virtualizations blogs. Voting is open now. Go choose your 10 favorite virtualization blogs. 2 random voters will be selected to win a copy of his book, Maximum vSphere, and a copy of TrainSignal's vSphere Pro Series 2 trainings.

Thanks for the great resources Eric!

Be the first to like.
17Sep/100

Streak*

I have been wanting to replace my old phone for a while now, but because of my service contract, I haven't. Happily I won a Dell Streak at vmworld this year (thanks again Dell!). 2 weeks ago, I plugged my SIM card in and I haven't looked back since. I am really impressed with the Android OS (my first Android device) and also with the build quality of the Streak.

Pros:
Both . and , on the initial keyboard - Speeds up typing
Customization - This is one of the big reasons I run linux on my desktop and laptop computers. Finally I have a phone as customizable as my computers.
Huge - I can fit an amazing amount of information on the screen. I love it.
Text messaging screen shows character/messages count.
Camera - For the first time, my phone takes usable photos, and I don't need to always have a digital camera on me.
Screen - I was worried that with such a huge screen, it would be really pixelated. It looks really good. Even when you zoom out and have very small text.
QR codes - Other phones can use QR codes, but I never "got it" until Android. Contact sharing is done via QR code, Android websites embed QR codes to make it easier to download direct to the phone.

Cons:
Android 1.6? Honestly Dell? - 2.2 has been out for a while now. I ran Android 1.6 for about 10 minutes before flashing it to 2.1 (O2 8105 ROM). 2.1 is a huge improvement over 1.6, but I want 2.2.
Huge - I got used to the size really fast (less than a day). Everybody that sees it comments on the size. A lot.
Camera - Autofocus cameras are slower than old cell cameras. The autofocus on the streak is slower than most.

I love the Streak and Android. This is the best phone I have ever owned.

*This topic is pretty far off the typical topic of the blog. My justification is that I won the phone at vmworld, and it is a cool new toy I am excited to use. :)

Be the first to like.
Filed under: OffTopic, random No Comments
30Aug/100

Disappearing vmfs volume

When we upgraded to ESX 4.0, we ended up purchasing Enterprise licenses. The host profiles of Enterprise Plus appealed to us, but we couldn't justify the additional cost. We use kickstart and postinstall scripts to configure our ESX hosts. This gives us most of the benefits of host profiles, without the costs. To maintain this consistency over time, we have elected to reinstall all of our hosts for all .x releases. Incremental patches are done with update manager.

Reinstalling to upgrade to ESX 4.1 finally made it to the top of the todo list on Friday. At some point while doing these upgrades, I lost one of my datastores. In vCenter it became grey and italicised. On the service console, the disks were not listed by UUID or name in /vmfs/volumes. After a little googling, I found the folling two pages. It turns out the partition tables were somehow corrupted for the vmfs volume. Recreating the table was fairly straight forward. If you are familiar with linux partitioning, the process will be very familiar to you.

All of these steps are performed from the service console.
To verify that the partition table is gone/corrupt, use fdisk -l.

A healthy vmfs partition table should look like this:

[root@vm01 ~]# fdisk -l /dev/sdh

Disk /dev/sdh: 1714.7 GB, 1714730565632 bytes
255 heads, 63 sectors/track, 208470 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdh1               1      208470  1674535211   fb  VMware VMFS
[root@vm01 ~]#

A broken partition table will look simlar to this:

[root@vm01 ~]# fdisk -l /dev/sdh

Disk /dev/sdh: 1714.7 GB, 1714730565632 bytes
255 heads, 63 sectors/track, 208470 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

      Device Boot      Start         End      Blocks   Id  System
[root@vm01 ~]#

If you have a corrupted partition table, you need to create a new one. This does not edit the portion of the disk that contains your vmfs fileystem holding your virtual guest files. Here are the steps to do this with fdisk:

  1. Press n and press Enter to create a new partition.
  2. Press p and press Enter to select that this is a primary partition.
  3. Press 1 and press Enter to make the first partition.
  4. Press Enter to retain the default value.
  5. Press Enter again to retain the default value.
  6. Change the partition to type fb (VMFS):
    1. Press t and press Enter.
    2. Press 1 and press Enter.
    3. Enter fb and press Enter.
  7. Align the partition to sector 128.
    1. Type x and press Enter to move to expert mode.
    2. Type b and press Enter to change the beginning of the partition.
    3. Type 1 and press Enter to select the first partition.
    4. Type 128 and press Enter to move to the block 128 the beginning of the partition.
    5. Type w and press Enter to save.

This is what it looks like on the console:

[root@vm01 ~]# fdisk /dev/sdh

The number of cylinders for this disk is set to 208470.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-208470, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-208470, default 208470):
Using default value 208470

Command (m for help): t
Selected partition 1
Hex code (type L to list codes): fb
Changed system type of partition 1 to fb (VMware VMFS)

Command (m for help): x

Expert command (m for help): b
Partition number (1-4): 1
New beginning of data (63-3349070549, default 63): 128

Expert command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.
[root@vm01 ~]#

After you have done this, you need to rescan for disks. You can do this within the vCenter client, or from the CLI by running `vmkfstools -V`. If all goes well (and it was only your partition table that was corrupted), you should now see your vmfs and everything should be happy.

Be the first to like.
27Aug/100

VMworld: I’m most excited about…

I am really excited to be attending VMworld again this year. I was amazed last year by the quality of materials presented and the firehose of information. There is no way you can take in all of the information presented. As I prepare to go again, I have a few things I will be doing the same, and a few things I will do differently.

Things I will do the same:

  • The vmworld Fun run. You can register up until Saturday Aug. 28 at 8:00 PM. This was one of the highlights of last years trip.
  • Wear identifiable clothing. Last year, it was a green Adidas bucket hat. Several people recognized me based on the hat alone. Sadly I lost that hat on a trip to Denver this year. This year, I will stick with the hat theme, but use my white ReAL Salt Lake hat. If you see someone wearing one, it is probably me, come say hi!
  • Enjoy San Francisco while I am there. Last year my flight out wasn't until Friday evening. On Friday I walked all over San Francisco and enjoyed the sights. I spent some time on the wharf, went to China Town, walked down Lombard Street, visited the Cable Car Museum, went to the top of Coit Tower and stopped in at Ghirardelli square. All of these were definitely worth the walk.
  • Schedule more things than there are hours in the day. I will likely miss several scheduled items for the solutions exchange or other events. This year there are a handful of customer events and tweetups that are scheduled at the same time as the sessions I am most interested in going to. I may go to the offsite events anyways (I'm looking at you #creampuffup).

Things I will do differently:

  • Remember to take my running shoes for the fun run. I did alright in Tevas last year, but it would have been a lot happier with the right shoes.
  • Talk to people. Last year, I largely stuck to myself. I didn't go to any customer appreciation parties, I rarely talked to people in sessions/solutions exchange. This year I am going to talk to everybody I can and try to associate faces and real names with twitter names.
  • Bring more people from work. Last year, only 1 person from work was able to go (me). This year There will be 2 of us going.
  • Attend customer events, parties and tweetups.
  • Win one of the prizes (thinking positive helps, doesn't it?).
  • Use social media. I will be blogging and tweeting while at vmworld to share my thoughts and observations.

Here is my current schedule.

Sunday
5:00 PM - vmworld fun run
7:00 PM - Warm up Party as a Service (WuPaaS)

Monday
9:00 AM - SP9760 - Dell’s Next Generation Datacenter and VMware
10:30 - TA6720 - Troubleshooting using ESXTOP for Advanced Users
12:00 PM - TA9420 - vSphere 4.1 Overview
1:30 PM - V18268 - Deep Dive on Virtualization - How Stuff Works in Virtualization
3:00 PM - TA8129 - A Beginner’s Guide to Performance Troubleshooting with vSphere
4:00 PM - Welcome Reception
6:00 PM - One of the customer appreciation events - Still deciding which one

Tuesday
9:00 AM - General session
11:00 AM - MA7140 - vCloud Architecture Design Strategies and Design Considerations
12:30 PM - PC9920 - Compliance in the Cloud: Managing Risks and Addressing Concerns
2:00 PM - MA9800 - How To Get From A VMware Cluster To Cloud?
3:30 PM - ALT3004 - VMware vSphere Troubleshooting
5:00 PM - SE8195 - Managing Security Compliance for VMware Virtualization

Wednesday
9:00 AM - TA8595 - Virtual Networking Concepts and Best Practices
10:30 AM - EA8660 - Virtualizing Tier 1 Applications: Leverage the vSphere Private Cloud as a Better Platform for Apps
12:00 PM - PC6940 - Networking Best Practices for vCloud
1:30 PM - TA8233 - Prioritizing Storage Resource Allocation in ESX Based Virtual Environments Using Storage I/O Control
3:00 PM - TA8623 - Storage Super-Heavyweight Challenge
4:00 PM - #creampuffup
4:30 PM - TA8245 - ESXi Internals: Better Understanding for Better Management and Troubleshooting
7:00 PM - vmworld party

Thursday
9:00 AM - General Session
10:30 AM - TA8270 - Get the Best VM Density From Your Virtualization Platform
12:00 PM - MA9800 - How To Get From A VMware Cluster To Cloud?
1:30 PM - TA7171 - Performance Best Practices for vSphere
3:00 PM - SS8222 - Transitioning to ESXi, Architecture for Next Generation vSphere
8:00 PM - #v0dgeball

Friday
8:00 AM - Play happy tourist

Be the first to like.
Filed under: Uncategorized No Comments
31Jul/100

Vmworld 2010 fun run

One of the highlights of VMworld last year was the fun run. It was a 5K run across the Golden Gate Bridge and back. I met some great people, got some great photo opportunities, and accomplished one of my goals while in San Francisco (run the Golden Gate Bridge). After I got approval to attend VMworld this year, one of the first things I did was to look for information on the fun run.

Yesterday on the VMworld blog they had a post about this year's fun run. Apparently it was so successful that VMware decided to make it an official event. It won't be going across the Golden Gate Bridge this year, but close to it. Additional information will be released next week, so watch the VMworld blog for more information.

The important information:

What: 5k out-and-back course
When: Sunday, August 29th at 5:00 p.m.
Where: Crissy Field, San Francisco
Who: VMworld 2010 attendees and spouses
How: Registration and further information to come out next week.

I am pretty excited for another fun run. I will definitely be there.

1 person likes this post.
Tagged as: No Comments
2Jun/100

Deploy portgroup to entire cluster

Quick and dirty bash one liner to deploy a port group to all of your ESX nodes in a cluster.

for X in `seq -w 1 16`
  do
    ssh root@vm$X.cluster1.company.com "/usr/sbin/esxcfg-vswitch -A 'One Thousand Port Group' vMachineSwitch0 ; /usr/sbin/esxcfg-vswitch -p 'One Thousand Port Group' -v 1000 vMachineSwitch0"
    ssh root@vm$X.cluster2.company.com "/usr/sbin/esxcfg-vswitch -A 'One Thousand Port Group' vMachineSwitch0 ; /usr/sbin/esxcfg-vswitch -p 'One Thousand Port Group' -v 1000 vMachineSwitch0"
done

You notice that this assumes some uniform in host names. Ours are vm01-16.clustername.company.com. If you chose to use names rather than numbers, you can still do this, just replace `seq -w 1 16` with a space delimited list of host names.

You will note that the commands above first create the port group, then add the tagged vlan to that port group. Also, I am updating 2 clusters.

Be the first to like.
1Jun/100

vmxnet3 and Red Hat

We have been having weird network issues with some of our RedHat guests, and at the suggestion of VMware support, I am converting the NICs from e1000 to vmxnet3. There are a couple gotchas to be aware of when doing this.

  1. Out of date tools mean no NIC until you update them. It is in your best interest to fix this before halting to add hardware.
  2. In RedHat, if you change the MAC, you need to update HWADDR line in /etc/sysconfig/network-scritps/ifcfg-ethX
  3. When editing that file, don't just comment out the HWADDR. I tried this first and it worked on 3/4 guests. The 4th guest created a NIC named __tmp440647619 with no network configuration. After updating the HWADDR line to reflect the new MAC, everything worked great.
  4. Be the first to like.
1Jun/100

Security conference topics

One of the groups I am involved with puts on a security conference every year. This year I have been asked to help plan and implement the conference, but I need some help. If you were going to a security conference, what topics would you like to see covered? It is not a VMware conference, so any topic on any area of personal/computer security is fair game.

Leave your suggestions in the comments and/or post them via twitter @virtualhints.

Be the first to like.
29Apr/100

Great tools out there…

We are currently looking at replacing our EMC Clariion CX3-80 SAN. As a part of the process, we had a performance audit of our SAN and found that our datastores are not being hit evenly. Not even close. I decided I needed to look into things more to figure out what was causing the unbalanced load.

I ran through a few of the scripts included in the vmware and VMware Community (formerly virtu-al.net) powerpacks for PowerGUI and . The one that was the most help is called "Datastore with more than X VMs" from the Community PowerPack.

With a quick glance at this report, I could tell that a lot of storage vMotion was in my future. VMs had been balanced across LUNs purely based on free space. Some datastores had over 20 VMs (thin provisioning is a double edged sword) and some had under 5. We decided to migrate VMs around so that all of the LUNs had a comparable number of guests to see how that helped. Things are not perfect now, but they are significantly better than they were before.

The moral of the story is: there are outstanding tools out there. Use them. To everyone out there that contributes your skills to the community and helps to create these PowerPacks and supporting infrastructure: Thank you for helping me to work smarter, not harder.

Be the first to like.