Brook vs Primary Data

Brook attempts the obstacle course at Primary Data's booth at VMworld 2015

Favorite line: You got to love Rick's Fusion-io pride when he says, "Ladies and gentlemen thank you for coming by Fusion-io".  I miss Fuison-io too.  :-)

VMworld 2014 - Sights to See

San Francisco sights to see, within walking distance of the Moscone Center.

San Francisco Bay

Golden Gate Bridge

Alcatraz

Pier 39 and Fisherman's Warf

San Francisco City Hall

Chinatown

Grace Cathedral

Cable Car Museum

Coit Tower

Lombard Street

Ghirardelli Square

VMworld 2014 - Getting the Most out of vMotion - Architecture, Features, Debugging

Getting the Most out of vMotion - Architecture, Features, Debugging

"Getting the Most out of vMotion - Architecture, Features, Debugging vMotion is a key, widely adopted technology which enables the live migration of virtual machines on the vSphere platform. It enables critical datacenter workflows, including automated load-balancing with DRS and DPS, hardware maintenance, and the permanent migration of workloads. Each vSphere release introduces new vMotion functionality, and significant performance improvements to address key customer requests and enable new use cases. In this session, join engineers from the development and performance teams to get an insiders' view of vMotion architecture, cutting edge features, best practices, and tools for performance troubleshooting. Performance studies will be presented for some of the hot topics including Monster VM migrations, migrations over IPv6, and Metro migrations over Distributed/Federated storage deployments. Finally, take a sneak-peek into the future and performance directions for vMotion including long distance migrations and migration to public clouds".

• Gabriel Tarasuk-Levin - Staff Engineer 2, VMware
• Sreekanth Setty - Staff Engineer, VMware

vMotion

Transperent move of VM to another host.

vMotion requires shared storage

vMotion enables features like DRS and FT

vMotion Workflow

1. skeleton vm on destination
2. copy vm memory state - most complex portion of workflow
3. quiesce vm on source
4. transfer device state and transfer remaining memory changes
5. resume vm on destination
6. poweroff source vm

How does the memory copy work? Pretty complex. Uses iterative memory pre-copy. Cycle through memory pages, copy and monitor for changes (dirty pages). Repeat until converges on equality.

Storage vMotion

Flip side of vMotion, only care about disk state.

VM remains on same host.

Storage vMotion has similar workflow as vMotion

1. skeleton vm on destination
2. copy vm cold data, such as snapshots
3. copy vm hot data content
4. quiesce vm on source
5. transfer device state and hand off memory state
6. resume vm on destination
7. free vm resources on source

vMotion without Shared Storage

Available since ESXi 5.1

Only available from the Web Client

Moving VM atomically to another host without shared storage!

Workflow looks like Storage vMotion

Current transfers cold data across management network - architectural decision that VMware is trying to fix

Works with any storage type (NFS, SAS, etc)

Technology written by presenter

Features

History:

• vMotion (2003)
• v5.0: Multi-NIC vMotion, Sun during Page send (DPS)
• v5.1: vMotion without shared storage
• v5.5: Metro VPLEX support, IPv6 improvements

Performance

Performance metrics:

• migration time (memory/disk)
• switch over time
• application impact (throughput and latency)

Monster VM Migration Performance

• 2 NICs shows significant benefit. 3rd NIC not so much (due to vMotion helper threads limitation).

vMotion Across Metro Distances

Metro has up to 10 ms round trip time

EMC VPLEX optimizes vMotion duration

VPLEX uses some caching features for syncing across Metro

What's Next for vMotion

• vNIC improvements - such as 3rd NIC performance
• support Array Replication with VVOLs
• Long Distance vMotion
• vMotion within/to the Hybrid Cloud (vCHS)

VMworld 2014 - Virtual SAN Best Practices for Monitoring and Troubleshooting

Presentation Notes

Virtual SAN

Virtual SAN - software-based storage built into ESXi

• Aggregates local Flash and HDDs
• Shared datastore for VM consumption
• Distributed architecutre
• Deeply integrated with VMware stack

VSAN GA with ESXi 5.5 Update 1

RVC

RVC - started as a VMware Labs "Fling"

• Interactive command line, with lots of VSAN commands
• Included in VC since 5.5 (windows and appliance)
• Presents inventory as a file structure

HCL

Verify Hardware against VMware Compatibility Guide (VCG)

HCL Guides:

• vSphere general compatibility guide (Servers, NICs, etc)
• Virtual SAN compatilibyt guide - adpaters, Flash and HDDs

show adapters using RVC:

vsan.disk_info --show-adapters <cluster>/hosts/*

Virtual SAN HCL - http://vmware.re/vsanhcl

HCL steps:

• Step 1 - collect hardware information
• Step 2 - check HCL - http://vmware.re/vsanhcl
• Step 3 - verify your drivers

When viewing HCL entry, also check the "Class" and performance is important

Network

Network - Misconfiguration Detected

• VSAN requires 10GBe (or 1G dedicated)
• Single L2 network among ESX hosts
• IP Multicast

Show ESX configuration:

esxcli vsan cluster get
RVC: vsan.cluster_info <cluster>

Ensure all hosts have VSAN vmknic configured

WebClient: host -> manage -> networking -> vmkernel adapters
esxcli vsan network list
RVC: vsan.cluster_info <cluster>

Ensure VSAN vmknics are on right subnet

WebClient: host -> manage -> networking -> vmkernel adapters
esxcli ?
RVC: vsan.cluster_info <cluster>

Ensure Multicast is configured

tcpdump-uw -i <vmknic> udp port 23451
tcpdump-uw -i <vmknic> igmp

Issues

VM shows as non-compliant / inaccessible / orphaned

• non-compliant - maybe one mirror down
• inaccessible - really bad
• orphaned - VC has forgotten about the VM

VSAN object accessible:

• at least one RAID mirror is fully intact
• quorum: more than 50% of components need to be available (witnesses count here)

RVC Reports

VSAN RVC state reports:

vsan.vm_object_info <vm>
vsan.disks_stats <cluster>
vsan.obj_status_report <cluster>
vsan.obj_status_report --filter-table 2/3 -print uuids <cluster>
vsan.cluster_info <cluster>
vsan.resync_dashboard <cluster>
vsan.check_state --refresh-state <cluster>
vsan.disks_stats <cluster>
vsan.check_limits <cluster>

Diagnostics

Use the vSphere Web Client - the C# desktop client doesn't show VSAN or VSAN errors

VM Provisioning Started Failing -

• don't use: Cluster - Manage - Settings - Disk Management (where dissk were setup, it is not the right place to check disk health)
• Use: monitor - virtual SAN - physical disk

Proactive approach, try creating vm on every host on the cluster:

web client: standard method
rvc: diagnostics.vm_create -d <statstore> -v <vmfolder> <cluster>

VMware believes in "Dog Fooding" - have many internal VSAN clusters running

Benchmarking

VSAN Observer (vsan.observer in RVC)

• collects stats every 60 seconds
• web interface
• HOL Plug: check out VSAN Observer Hands On Labs

Outstanding IO chart in Observer is a good indicator that SSD speed is not sufficient (affects latency)

VSAN implements a priority traffic scheduler

Good References

Webinars on Monitoring/Troubleshooting:

• How To Monitor Virtual SAN (VSAN) - YouTube - https://www.youtube.com/watch?v=rHofTkK6K40
• How to Troubleshoot Your Virtual SAN (VSAN) - YouTube - https://www.youtube.com/watch?v=ASL3WVqy65o

VMware Blogs:

• RVC series:
  • Managing Virtual SAN with RVC: Part 1 - Introduction to the Ruby vSphere Console | VMware vSphere Blog - VMware Blogs - https://blogs.vmware.com/vsphere/2014/07/managing-vsan-ruby-vsphere-console.html
• VSAN blog:
  • Official VMware Virtual SAN Blog Index | VMware vSphere Blog - VMware Blogs - https://blogs.vmware.com/vsphere/2014/07/official-vmware-virtual-san-blog-index.html
  • VMware Virtual SAN Quick Monitoring & Troubleshooting Reference Guide - https://communities.vmware.com/servlet/JiveServlet/previewBody/25934-102-2-34323/VMware_Virtual_SAN_Quick_Monitoring_Reference_Guide.pdf

Community Blogs:

• Troubleshooting, Automation, Nested ESX: http://virtuallyghetto.com/category/vsan
• All kinds of things VSAN: http://vmwa.re/vsan

Useful References

VMworld 2014 - Virtualizing Databases and Doing It Right

Presentation Notes

Book:

• Virtualizing SQL Server with VMware: Doing IT Right - by Michael Corey, Jeff Szastak, Michael Webster

Presentation covers Microsoft SQL and Oracle

Microsoft SQL people don't want to talk to Oracle people, and vice a versa - we are going to do it anyway

DBAs shouldn't care about the infrastructure. You don't care about the Cell Phone towers, you just expect them to work.

Number #1 issue that causes BSOD - drivers. With Virtualization, the driver depth is minimized

Virtualization is an incredible return on investment

Able to adjust resources with a click of the button (assuming there are additional shared virtual resources available to add)

If you adjust memory, you will need to restart the database to take advantage

"Any Resource, Any Server, At Any Time" in the (Pool)

Is your database too "Big" to virtualize? - doubt it

Virtualization has about a 5% overhead, but if your setup doesn't have at least 5% wiggle room, you are doing something wrong anyway

Management expectations - need to set correctly, and explain the costs and what it actually will take

If asked to meet an SLA, make sure it can be meet, and set the expectations. If you can't meet it, let management know as soon as possible.

Optimize optimize optimize - the defaults on servers, applications, etc are not tuned for performance

Read the documentation from all vendors!

Professional Association of SQL Server - join it if you are doing SQL Server - http://virtualization.sqlpass.org/

Oracle VMware users group - http://ioug.org/VMware/

http://longwhiteclouds.com/ - good blog

SLAs:

• two nines - 99%
• three nines - 99.9%
• four nines - 99.99%
• five nines - 99.999%

If it doesn't perform will in physical, don't expect it to perform will in virtualized - garbage in, garbage out

Baseline, baseline, baseline - "there are no silver bullets"

slow storage array equates to slow database

When baselining, make sure your sample set is reasonable (seconds). A lot can happen in minutes.

SLOB (Silly Little Oracle Benchmark) - good free tool to look at

"Check It Before You Wreck It" -- Jeff Szastak

"Build New" - when migrating from Physical to Virtual, take the opportunity to "Build New"

VMware http://tsanet.org/ - hardware or software - VMware can setup support calls that can include Oracle

If your OS and database don't know they are virtualized, don't tell them

Understand your workload types - if you don't know, how can you tune and configure??

• Also allows you to combine VMs that may have offset time workloads (gaining more for my investment)

Seperate development from test from production environments

Trivia:

• First use of the word "Nerd" - Doctor Seusse (If I ran to the zoo)
• Americans eat the most food on Super Bowl Sunday

Have more VMs than less. Giant VMs with everything in it are harder to manage then smaller VMs. Better resource management and tuning options.

Storage - Spindle count and RAID configurations still rule

Know where your bottlenecks are at

VMFS vs RDM - perform about the same, valid reasons for using either

• VMware recommends VMFS unless you have a really good reason

Thin Provision - first write penalty - use Thick Eager Zeroed for performance

Microosft recommends 1 datafile per CPU

PVSCSI adapters are high-performance - use them

80% of issues are performance or storage misconfiguration

1. 1 issue is not enough spindles to support the app

vCPUs - count hyper threading as only .2 of a CPU (nearly nothing) when doing your calculations

Recommendation: 1-1 Ratio physical cores to vCPUs

Ntirety Rule - for SQL server

NUMA - Non-Uniform memory Access - size VMs to fit within a socket realm (ex. 128GB with 4 sockets would be <32GB optimal performance)

vNUMA (exposed to OS) is better than yNUMA (interleaving)

Swapping:

• Guest VM
• ESXi host level

Ballooning abd memory compression slows things down

• Ballooning is good, don't shut it off, but there is a performance hit when it kicks in

How many VMs can fit on a host? As many as fit within active memory

Memory Reservations can lock out memory for critical VMs

Jumbo Frames are good, if you use them correctly

• Have to configure on ESX, network switch, and application
• If there is a constriction point, breaks down. Don't set Jumbo Frames larger than smallest bottleneck in the network path

• Use VMXNET3 - reduces physical CPU overhead)

WSFC - Cluster Validation Wizard - should run before you call Microsoft Support

blog:

• http://michaelcorey.ntirety.com/
• http://www.dbtablog.com

VMworld 2014 - Backpack SWAG

Backpack SWAG

As silly as this sounds, the VMworld backpack SWAG is significantly better quality than last year's, and that makes me happy.  My old laptop bag was getting pretty old, so now I have a decent replacement.

VMworld 2014

VMworld 2014

Arrived at VMworld 2014 - excited to see what new things await us...

How To VMware VMDirectPath (PCI Passthrough)

How To setup VMware ESXi VMDirectPath (aka PCI Passthrough)

VMware's PCI Passthrough solution is by far the best I have used.  The other virtualization platforms (eg. Microsoft Hyper-V, Xen, Citrix XenServer, Oracle VM, KVM, etc) provide little, or no, PCI Passthrough support.  I have found that ESXi 5.0 had the best PCI Passthrough support, so stick with that version.  (ESXi 5.1 was not quite as stable in this area)

VMDirectPath I/O Summary

VMDirectPath I/O, aka PCI Passthrough, is a method of giving a VM direct access to a PCI device.
This VMware KB article describes configuring a device for PCI Passthrough:

• Configuring VMDirectPath I/O pass-through devices on an ESX host - http://kb.vmware.com/kb/1010789

As there is direct access to the PCI device, the only setup required is to create a VM and then assign the desired PCI device to the VM, using the direction in the KB article. No driver need be installed on the ESX host, only within the VM's guest OS.  To configure the VM follow the standard procedures for the target operating system, as though it were a physical system.

Requirements

The server's CPU must support and and BIOS must have enabled: "Intel® Virtualization Technology for Directed I/O (VT-d)" or "AMD I/O Virtualization Technology (IOMMU)"

Limitations

• Not all ESX servers have full support for VMDirectPath, and on some servers not all PCIe slots do either. (especially the desktop class systems)
• The server CPU/BIOS must support Intel VT-d or AMD IOMMU.
• Limited to a maximum of 8 pass through devices powered on simultaneously, per host
• Limited to passing through a maximum of 6 devices per VM

WARNING: When updating firmware through VMDirectPath, when fio-update-iodrive requests a reboot, this means a reboot of the physical hardware, not just the VM.

VMDirectPath Procedure

Note: These same procedures work for ESX/ESXi 4.x and ESXi 5.x.

1. Export out PCI devices through "Advanced Configuration".  The names may not be what you would recognize, so it is best to match on "Device ID" and "Vendor ID".

2. Reboot ESX/ESXi server.

3. Add PCI Device to VM's configuration.

4. Set Memory Reservation.  PCI Passthrough requires all allocated memory to be reserved for the VM.  Set the VM's Memory Reservation to match the VM's Memory Allocation.  Simply drag the Memory Reservation slider all the way to the right to the orange triangle.  This will also need to be done anytime the memory, for the VM, is adjusted in the future.

5. Treat device, in VM, as you would a physical server (install the OS specific driver within the target VM).

VMworld 2013 Conference Highlights

CEO General Session

Speaker: VMware’s CEO Patt Galsinger

“The Software Defined Data Center”

22,000 attendees this year to VMworld 2013!

VMware’s 3 Imperatives:

1. Virtualize all the things
2. IT management gives way to automation
3. Hybrid cloud will be ubiquitous

4 Pillars of a Software Defined Data Center:

1. Compute
2. Storage
3. Network
4. Management and Automation

Product Announcements:

• Compute:
  • ESXi 5.5 (vSphere 5.5)
• Storage:
  
  • Virtual SAN (vSAN)
  • Virtual Volumes
  • vFlash Read Cache
  • Virsto
• Network:
  
  • VMware NSX (major topic for VMworld this year)
• Management/Automation:
  
  • vCloud Hybrid Services

New in ESXi 5.5:

• 2x (cores and other limits raised)
• App Aware H.A.
• Big Data Extensions
• Hadoop

Software Defined Storage:

• Policy driven control plane
• Virtual data plane
• Application centric data services

New with vSAN:

• Simplifies Storage
• distributed software datastore
• Resilient
• Policy driven
• Elastic
• High performance
• * Release in 1H 2014 *

New in Networking with NSX:

• Software Defined Networking (SDN)
• What ESX was to hardware, NSX is to networking
• Network virtualization platform

Summary: VMware is hell bent on removing all IT admins (Storage/Server/Network/Management) from the data center and replacing them with Virtualization Admins!

VMworld 2014 – Save the Date – August 24^th - 28^th, 2014

COO General Session

Speaker: VMware COO, Carl Eschenbach

ITaaS (IT as a Service)

• Self service
• Transparent Pricing
• Governance
• Automation

All about “having the perception that you have control”

Focus: East and West Fabric

• There is a major shift of traffic volume in the data center from North-South (client-server) to East-West (server-server).

New Product Summary:

• vSAN – Simplifies storage, distributed software datastore
• Virtual Vol – VVOLS – VM centric, snapshot VMDK from storage
• Virstro – Appliance that aggregates storage pool as NFS data store
• Horizon View – Desktop Virtualization Product (renamed from “VMware View”)

War Room – designated for critical issues

Virtual San

Presenter: Kiran Madnani

• On Beta forms, for vSAN, have had conversations with him before

Virtual Data Plane – pooling hardware like a hypervisor does – “hypervisor based pooling of heterogeneous hardware”

Virtual SAN (vSAN):

• Single datastore called “vsanDatastore” by default, can be renamed.
• Optimized for resources that are a mix of expensive and inexpensive
• Scale out
• Software solution
• Cluster disks and cache
• High performance – flash based distributed read / write cache
• VM centric management
• Automated SLA management
• High resiliency – distributed RAID
• Dynamic capacity scaling
• Rebuilds happen in parallel
• Different sizes of servers – including magnetic disks
• Storage policy specification
  • Capacity, availability, performance
  • Distribution, RAID and SLA are on a per VM level
  • Works with vMotion, storage vMotion, DRS, SRM, etc

Requirements:

• 3 x ESXi 5.5 Hosts
• vCenter 5.5
• 1 x SSD empty
• 1 x HDD empty
• Network: 1 Gbps, 10 Gbps (preferred)
• SAS/SATA controller working in passthru/JBOD/HBA mode

Performance: 80 – 100K IOPS ???  (I have not seen this level of performance from my tests)

Use Cases:

• VDI
• Test/Dev
• Data Center Consolidation
• Big Data
• Private Cloud
• Disaster Recovery

Join the vSAN beta: http://vsanbeta.com

Power CLI Best Practices

Strong PowerCLI community.  Google “PowerCLI Community”

Based off of PowerShell

Works off of objects (can pipe objects too!)

Uses verb-noun construct in plain English

Easy help: get-help

New in 5.5:

• Same: core, image, auto deploy, license, vds, cloud, tenants
• Tagging
• Open-vmconsole - awesome

Best Practices:

• Store results in variables
• Help and Examples
• -ShowWindow switch
• Functions breakdown
• Splatting (put parameters into hash table)
• Make it simple
• Try, catch, throw
• Filter on the left
• Steal from the best
• Don’t use write-host

Check out VMware Flings – free projects that are very useful!

Exchange Messages Between Guest and Host using RPC

Cool stuff, might be useful for vCenter Plugin?

Uses a combination of Guest API and VMware Tools

Fling Guest API Library – http://labs.vmware.com

VAM (Virtual Machine Monitor)

Good for monitoring an application and posting to vSphere Client

vSAN Panel

Discussion:

• Be able to track trends (request from audience)
• Provision without having to pester IT
• CapEx driven
• Direct Attach – host pinning – get around
• Moves responsibility from Storage Team to virtualization team, but can they handle the extra load?

Join the vSAN beta: http://vsanbeta.com

Ask the Experts

Experts:

• Duncan Epping, VMware – Yellow Bricks blog
• William Lam, VMware – Virtually Ghetto blog
• Scott Lowe, VMware – scottlowe.org
• Vaughn Steward, NetApp – Virtual Storage Guy.com

Awesome blogs, add to favorites!

What about Linux side Automation – PowerCLI seems to be the focus now

• pySphere – soap suds (recommended by William Lam)

vSAN Best Practices

Use Cases:

• Best used for VDI
• Tier 2 & Tier 3 Test / Dev Private Cloud
• DR Target (SRM)
• Branch Office making use of replication

vSAN is just a cluster object/property

vSAN Workflow:

• Setup Network and vmk interfaces
• Enable vSAN
• Set vSAN Auto Mode or Manual Mode
  • Manual Mode: Pick disks

Best practice: have each host similarly configured for best results

Storage policies are like Storage Profiles v2

Policy Capabilities:

• Stripes per object
• Failures to tolerate
• Space reservation
• Flash read cache reservation
• Force provisioning

Performance Monitoring:

• Esxtop
• Performance manager UI
• RVC & Observer (where does one get these from??)

Performance Group Discussion

NUMA:

• Trust VMware NUMA default settings, the experts know better.  Modify at your own peril!
• App must be NUMA aware
• NUMA allows scheduling RAM on same CPU

Virtual CPUs:

• Virtual Cores only added for licensing reasons.  Using Virtual Sockets vs Virtual Cores makes no difference to VM.
• Match Hardware and Virtual CPU to actual need for best results
• Max, don’t allocate more Virtual CPUs than system has of cores (ignore HT and the like)

Latency:

• If latency sensitive – don’t over subscribe!  Simple as that.
• Monitor with vCenter Operations
• No silver bullet
• New “latency” property in 5.5 for Network latency related issues (reserves CPU time)

Storage:

• Split out disks (luns) is still a good practice for database
• Qdepth, queues, and paths

PCIe Hotplug

Supported with Native Driver – another good reason to switch to native driver model

VMware Ready

VMware: Joe Taylor, Pat Richards, Matt Stander

Matt Stander – runs Dev Center (mstandar@vmware.com, or is that mstander@vmware.com?)

All in the Ecosystems Engineering department

Dev Center & VMware Ready – One stop development shop for partners

VMware has 20,000 partners, and 3,000 technical partners.

Contact: tapalliance@vmware.com

Resources: Dev Center, Partner Central, Dev Community, vmdev.net

Workbench IS – for integrating directly into existing eclipse installs

SDK UI Integration docs and guides

Questions:

• Community forums for developers?
  • Coming
• How to get better dev support?
  • Sales Force pay style coming
• How to get in as inbox driver?
  
  • One off requests
• TAM or SDK support (SLA)?
  
  • Ecosystems Engineering
• IOVP Recert 5.5?
  • Will better communicate
• vSAN Cert on Dev Center?
  
  • Coming
• Native Driver?
  
  • Slowly migrate and deprecate Linux
• Workbench – why SLES?
  
  • Legal reasons, getting unified with other departments
• Need packaging repo
  
  • Conference for Dev or more technical dev sessions at VMworld?

Have we tried the new Test Manager that can help automate IOVP process?

• Workbench – top left tutorials
• CLI / Log browser

DDV – Dynamic Driver Verifier on Dev Center

ROSA – will migrate to Dev Center eventually

Dev Support SLA – can hire a guy for a week through Dev Consulting at about $11K a week.  Managed through “Sandy”.  Contact Martin Lister / Joe Taylor for more information (Ecosystem Engineering)

Recommend using Ecosystems Engineering over TAM.  (TAM account manager-> GSS -> Product Management)

vCenter Deep Dive

Web Client now in Simple Install

Custom install: for distributed setup

New in Web Client 5.5:

• Now includes Mac OSX Support
• Filter by tags
• Recent Objects Tab
• Missing: VUM, Host Client, SRM
• Performance still sucks
• Database “shattered” into smaller ones to help with performance

SSO:

• Dropped database
• Dropped master password