Note to self: If you make a snapshot on a production vm in your virtual infrastructure, don't keep it much longer than a day or two at the most.

Not sure how else to say this... but, oops. We have this mail server that we use for our ISP customers. It is running in our Vi. It seemed to perform quite well until we moved a couple thousand pop accounts to it. We could not figure out where the slow-down was. We added more memory, more priority (I bet you didn't know that priority was a resource!)... The memory helped a bit, but it was still sluggish.

It seemed like this could have been a case of one of those kind of servers that is not meant for consolidation. But, as a last ditch effort, we decided to add a second virtual processor. Before doing the deed we made a snapshot of the VM just in case things went badly. Everything went fine just as one would expect. Performance did improve, but not to the extent that made us change our minds about rephysicalizing (another new word). We thought we'd give it a month to settle down and look at some long term trends before taking the plunge back to physical from virtual.

After getting back from VMWorld 2006 we thought it would be a good idea to get our Vi up-to-date. Seems we were a little early in adopting Vi3. The newest patch (3.0.1 for ESX and 2.0.1 for VC) contained over 500 bug fixes, so I was told... and that this patch would greatly improve the overall performance of our virtual infrastructure. When it came time to VMotion this mail server off a host so we could upgrade the host, it gave an error stating something about there being an active snapshot... yeah, kinda forgot about that.

This is where the "Note-to-self" from above comes in. Apparently it is a bad idea to leave a snapshot in place for much longer than a day or two. We were running it for about two months. After a little discussion, we decided to delete the snapshot since it seemed that running on vSMP was ok and after all this time we were not going to revert back. Easy, right? sure.... till the task times out. The vmdk snapshot file for the mail-store drive had grown to about 35GB. When we deleted the snapshot, the 35GB file was locked and a new snapshot file was created and used until the 35GB of changes were incorporated back into the original 150GB vmdk. I guess on a very disk-busy drive, that takes a while. I paniced and called VMware. They said that it could take as long as 8 hours to finish. So we waited and hoped nothing crazy happened in the interim.

Two hours later, it was done and it finished without a hitch. The mail server was then VMotioned off and the host got its update applied.

Now that we've learned our lesson, the mail server is performing perfectly. The second processor was the answer but we did not realize the difference in performance because of the overhead of the too-long-lived-snapshot. So, in the end we learned that snapshots are short-term friends and we will not have to put our mail server back in the physical world. That leaves just a few servers to go before we've totally virtualized all our servers. Woo Hoo!

Syndicated via RSS From: http://www.vmwarez.com/

VMware's yearly virtualization conference day two...

Well, first, last night my wife and I ate dinner at "The Pizza Next Door" a NY style pizza joint just around the corner from our hotel. It seems that they just opened, but they are an extension of the greek bakery next door (hence the name). Wow, for about five bucks each we had 2 huge slices and a coke. Can't beat that with a stick!

This morning we had a "Birds of Feather" breakfast with the VMTN folks. By the time I left, not too many had arrived. I'm not sure if any did after all. Oh well.

The general session was way more interesting today than yesterday. Most impressive is Mendel Rossenblum. With him on VMware's side, there's no way MS could have a chance. The guy has vision. He is one of those super smart people that when you see him before starts talking, you think, "oh, here's one of those really smart people who's too smart for their own good - can't communicate, etc" but then you realize he's an awesome public speaker, just as smart as expected and wow... insight and vision out the wazoo.



One of the things that was highlighted in the general session was the capability of recording a VMs instructions as they happen... then being able to play them back.... for more on that read this entry in Scott Lowe's blog: VMworld 2006 Day 2 Keynote

Sessions in the morning were good as usual, not much to report.

Today I was at the Core Customer Lunch. The food was really good. I got to sit next to Sindy from VMware (their webmaster) and Melinda from the Core Customer team (also a member of the VMware band). I received one of eight or so honorable mentions for participation in the Core Customer program. Woo Hoo! They took pictures, if I had one I'd post it. Then there was the presentation of the VMware Vanguard Awards... and other awards for the core customers and the top VMTN forum posters were recognized.

After the PM sessions we went out to dinner with the Lowes. (his blog is here) We ate at The Original Pantry... Yummo. Then went to fivebucks for some liquid refreshments. Thanks Lowes... it was fun. We'll be in touch after we get our feet back on the ground.

Here's another article I found that is pretty interesting. It is an interview with Mendel about the direction that virtualization is going...

More later....

Syndicated via RSS From: http://www.vmwarez.com/

People started to arrive at VMworld 2006 for registration yesterday afternoon/evening (except for VMware Partners who had sessions Monday, so they were already there).



The morning general session was good as usual. One of the most interesting things was VMware's emphasis on saving the world from global warming… I think it was said a little tongue-in-cheek, but I could be wrong, they are from California! Diane Greene introduced a guy from PG&E who announced their new rebate program that will pay companies for reducing their server count using virtualization. Besides the power savings realized by server consolidation, you can also get credit on your account per server eliminated. That’s pretty cool.

There was then a demo about Virtual Infrastructure. It was somewhat interesting but nothing new. Next came a panel about virtual appliances. Lots of good stuff there. Last year the buzz seemed to be about the hosted desktop, this year, it seems that the virtual appliance is getting a lot of attention.

The last bit of the general session was an industry perspective from a few big dogs… Marc Andreessen (wrote mosaic), Andy Bechtolsheim (a chief architect at Sun) and David Cheriton (a professor of computer science at Stanford). It was mediated by Lee Gomes of the WSJ. This may have been the most entertaining and insightful part of this session. Cheriton, from academia was ranting (and mostly rightfully so) about how little attention is paid to quality in today’s programming. Andreessen likened PERL to crack and Bechtolsheim impressed everyone with his ability to wear a suit and tie WITH berks – and get away with it! He was the only one of the panelists who did not want to say anything specific about the topics… seemed he was worried about being misquoted or something. Anyway, it was all good stuff.

Next stop, the solutions exchange.

Syndicated via RSS From: http://www.vmwarez.com/

Wouldn't you know that our first evening out I forgot my camera. We arrived in LA at around noon and took the metro to our hotel (the Ritz-Milner). When we tried to check in, they told us we were too early... so we went to get a bite to eat at a little sandwich shop around the corner - Paninio. They make Paninis. Good ones. Then we checked in. A shower was in order followed by a trip the the LA Convention Center to register for VMworld.

After getting back, Tina and I went to Cicada - a really nice place - for a little get-together of the top VMTN forum posters and VMware Bloggers. I got to put faces to names... talk a little about everything from blogging in general to virtualization.

This meeting was upstairs at the bar and after it ended we went downstairs for dinner. It was out of this world. Carpaccio, kobe beef raviolis, gnocchi... Mmm Mmm Mmmmmmm.

Tomorrow AM I will remember to bring the camera and will talk more about our favorite subject - virtualization... maybe a little more about my other favorite subject - eating really good food :)

Syndicated via RSS From: http://www.vmwarez.com/

As a means of promoting virtualization - and this site - and its sponsors, we've started doing some contests.

Here's a list of them:

• The Keep in Touch Contest (Running till 12/1/2006): This is the easiest. Just signup for our daily update of blog posts and you'll be entered to win one of three VMware Store Gifts certs. First place will get one for $300, second and third place will get ones for $100 ea. Winners here
• The Contest Contest (Ended) - I was at a loss for what to do for a contest so our first on was to email in ideas for what to do for future contests. The winner was BrendanK and he got a $100 Gift Certificate to the VMware Store.

If you are a vendor and would like to get some attention, let me know what product or service you'd like to donate for use in a future contest. You may even want the object of the contest to be something that highlights the use of your warez... email me at jim (at) vmwarez dot com if you're interested in setting something up.

Syndicated via RSS From: http://www.vmwarez.com/

How many ways can virtualization improve your life as an IT guy? First, let's just consider its use in server consolidation. Just the bottom part of the rack pictured below is our virtual infrastructure. In the top of the rack (unseen) we have 4 servers running their own OS (how last year :) and in our c.o., we have another 4 stand alone servers. The three servers in this picture are running VMware ESX 3. This cluster of ESX boxes is, at the time of this writing, running 21 virtual servers.

Just think of the implications of running these 21 boxes and physical ones:

• KVM Switches
• Power considerations
• Cooling
• Network ports
• etc...

A lot of time could be spent on each of these topics, but just try to picture it. This simple half rack of a san and three servers vs. 21 servers. They say a picture is worth a thousand words... how much do you think virtualization/server consolidation is worth?

Email me (jim (at) vmwarez.com) a picture of your virtual infrastructure and tell me how many VMs your running... and I'll post it here for all to see!

Syndicated via RSS From: http://www.vmwarez.com/