Back in 1999, we hired a guy from Veritas to install our first Netbackup software, version 3.5 it was (I think). Little I knew that it would be 2016 and I would still be working with this. It outlived other things that look more promising at that time.
I wanted to create this quick post about how to gain root access to a Netbackup appliance because it changed a little bit from version 2.6 to version 2.7. See my input in red.
login as: admin
admin@<ip>'s password: <password>
Last login: Wed May 4 09:52:28 2016 from <my mac>
Appliance Manage master and media appliances.
Exit Log out and exit from the current shell.
Manage Manage NetBackup appliance.
Monitor Monitor NetBackup appliance activities.
Network Network Administration.
Reports Examine the running and historical state of the host.
Settings Change NetBackup appliance settings.
Shell Shell operations.
Support NetBackup Support.
<NBU-Appliance-Hostname>.Main_Menu> Support
Entering NetBackup support view...
Checkpoint Appliance Checkpoint Management.
DataCollect Gather device logs.
Disk Gather disk information.
Errors Display NetBackup errors.
Exit Log out and exit from the current shell.
FactoryReset Reset this system to factory install image
InfraServices Show/control infrastructure services.
iostat Execute the iostat command.
IPMI Remote management port configuration.
KillRemoteCall Terminate running or hung remote calls on peer compute nodes.
Logs Set level, share, unshare, view, and upload debug logs.
Maintenance Launch maintenance shell.
Messages Display the messages file.
NBDNA Network traces with NBDNA tool (Select Create or Remove).
Nbperfchk Execute the disk IO or network bandwidth check.
NBSU Manage NetBackup support data files (Select Create or Remove).
Processes Show/control NetBackup and Admin Console processes.
Reboot Reboot the system.
RecoverStorage View or delete directories backed up during appliance re-image.
Return Return to the previous menu.
Service Service management.
Shell Shell operations.
Show Display information.
Shutdown Power off the system.
Storage Storage data and configuration
Test Test the current status of various appliance components.
<NBU-Appliance-Hostname>.Support> Maintenance
<!-- Maintenance Mode --!>
maintenance's password: <password>(this script was for 2.6)
maintenance-!> /opt/Symantec/scspagent/IPS/sisipsoverride.sh
bash: /opt/Symantec/scspagent/IPS/sisipsoverride.sh: No such file or directory
(this is the right script for 2.7)
maintenance-!> /opt/Symantec/sdcssagent/IPS/sisipsoverride.sh
Symantec Data Center Security Server Policy Override
Agent Version: 6.5.0 (build 355)
Current Policy: NetBackup Appliance Prevention Policy, r38
Policy Prevention: Enabled
Policy Override: Allowed
Override State: Not overridden
To override the policy and disable protection, enter your login password.
Password: <password>
Choose the type of override that you wish to perform:
1. Override Prevention except for Self-Protection
2. Override Prevention Completely
Choice? [1] 2
Choose the amount of time after which to automatically re-enable:
1. 15 minutes
2. 30 minutes
3. 1 hour
4. 2 hours
5. 4 hours
6. 8 hours
Choice? [1] 4
Enter a comment. Press Enter to continue.
<comment or just Press Enter>
Please wait while the policy is being overridden.
.................
The policy was successfully overridden.
maintenance-!> elevate
<NBU-Appliance-Hostname>:/home/maintenance #
After the initial setup is done, I recommend to go ahead and install the latest available firmware to get the most out of your environment. First, go to http://www.cisco.com, and from the Support Menu, click “All downloads.”
Image 1. Cisco Support Site – > All Downloads
Note: just hover over Support, don’t click
Image 2. Select The UCS B-Series Blade Server Software
Image 3. Select the Server Software Bundle
Image 4. Select the UCS Manager version and download the respective bundles.
Number one (1) is the UCS Manager release version, number two (2) is the Blade Firmware, number three (3) is the bundle for the C-series. The C-Series bundle is not necessary, but it’s recommended to upload it too. Number four (4) is the UCS Manager and the FI firmware. With this, now your are ready to start the software update.
Phase 1: Pre-Work
Check that the NTP is configured and working (view Figure 6 on Part 1)
Take note of the management interfaces IP addresses and check that the admin status is enabled.
Backup the UCS configuration
Verify the status of each FI, the status should be ok and green before proceeding.
Verify that the IO modules are up and operable
If the environment has been previously configured and is in production, check that all the servers (and all the rest of the equipment in general) are working correctly.
Verify that there is available capacity for the upgrade. If space is low delete any packages that are not in use (from the same place we will upload the packages later).
Before upgrading to Cisco UCS Manager Release 3.1, ensure that the key ring in use has a modulus size of 2048 bits or more by doing the following:
Connect to the UCS manager via ssh
Verify the modulus size of the key ring in use by using the following commands:
If the default key ring is in use and has a modulus size less than 2048 bits, reconfigure the modulus size to 2048 bit or more, and regenerate the certificate by using the following commands:
UCS-A# scope security
UCS-A /security # scope keyring default
UCS-A /security/keyring # set modulus mod2048
UCS-A /security/keyring # set regenerate yes
UCS-A /security/keyring # commit-buffer
UCS-A /security/keyring # show detail
Upload the packages, click on the plus sign to upload the files.
If the UCS Manager cluster uses the Fiber Interconnect 6296, please verify the Hardware version. SSH into the cluster and issue the “connect nxos” command, then “show module”
If the system has version 1.0 installed, then open a support case with Cisco to update it to version 1.1.
Click on the drop-down menu to select the new version
In case this error message pops:
Go ahead and clear the start up version and try again:
Re-login after a few minutes
Phase 3: FI Firmware
Equipment->Equipment->Firmware Management->Firmware Auto Install and then Click Install Infrastructure Firmware
Select Ignore All and click Next.
Select the version from the drop-down menu, check the Upgrade Now button and Click Finish.
Click OK
Verify the pending activities on top of the UCS manager to acknowledge the reboot of the FI’s. Verify that the hosts are working properly before the acknowledge.
Click Yes
Click OK
Monitor the Progress, this could take 40 minutes per FI
When the primary FI reboots, the UCS manager will disconnect, and you will have to re-login. Check the progress and reboot the secondary FI when the option appears in pending activities on top.
Phase 4: Blade Servers
Now is the blade server turn. If the system is in production, check that all Service Profiles have the Maintenance Policy set to User Ack to avoid an immediate reboot of the servers. Before continuing, it is advisable to check that all components are up and operable.
Click on the Install Server Firmware link
Follow the wizard clicking next, at the end click Install and click Confirm Install to the pop-up windows.
Monitor the progress and check the Pending Activities on the top for the Acknowledge, if the system is in production make sure you schedule a maintenance window for this.
An alternative would be to create a new Firmware Policy with the new version and then apply the policy to the Service Profile or the Service Profile Template.
With the system at the latest software version, we are ready to continue with the configuration of the Pools, Policies, and Templates.
I always wanted to write about the UCS B-Series installation process, but I was always in a hurry and could not take snapshots of a real installation. Here is a summary of that I did on my last installation:
Part I: Initial Setup
First, you need to rack and cable the equipment. For the correct way to install the equipment in the rack please check the Cisco UCS 5108 Server Chassis Installation Guide. In this guide, you can use pages 41-49 for the Chassis rack instructions and page 81 for the power cables needed. Then check the Cisco UCS 6200 Series Fabric Interconnect Hardware Installation Guide on pages 26 to 28. The next step would be to connect to the console port in the Fiber Interconnect (FI).
From the Guide (pp. 55-56):
Plug the RJ-45 end of the serial management cable into the Console port on the fabric interconnect, and connect the DB-9 male end into the serial port on a laptop or other computer. If the computer you will use does not have a serial port, you will need to use the Serial to USB adapter. Be sure to install the drivers for your adapter.
Start your terminal software.
Configure the terminal software as follows:
The COM port for the connection you are about to establish is the connection to the fabric interconnect. You may need to look in the computer’s device manager to confirm this. Example COM1 or COM5.
The other connection parameters are 9600 baud, 8 data bits, no parity, 1 stop bit.
Use the terminal software’s command to open the connection to the Fabric Interconnects. A session window will start, let’s take a look at the next to screenshots:
Now you are ready to connect to your UCS manager using your browser (Using the Cluster IPv4 address from the first screenshot).
Configure the Fiber Interconnect ports used to connect the Chassis as Server Ports. This will acknowledge the chassis and let you configure the rest.
Figure 4. Set the Chassis ports to Server Ports
I would like to show a few basic initial steps before we are ready to upgrade the firmware. After login into the UCS manager using your browser, change the Power Policy to “grid”. Equipment->Policies->Global Policies->Grid
Figure 4. Power Policy
The grid redundant configuration is sometimes used when you have two power sources to power a chassis or you require greater than N+1 redundancy. If one source fails (which causes a loss of power to one or two power supplies), the surviving power supplies on the other power circuit continue to provide power to the chassis. A common reason for using grid redundancy is if the rack power distribution is such that power is provided by two PDUs and you want the grid redundancy protection in the case of a PDU failure.
Another set of basic configuration parameters: the Call Home, the NTP and the Timezone.
Figure 5. Call Home Setting (leave it off until everything has been configured)
Figure 6. NTP and Timezone Settings
In the next new blogs, I will discuss the rest of the steps for configuration.
Recently I joined this community: https://www.cloudcredibility.com, and I feel that is not fair. I mean for people like me that love leveling in a game or score points or climb the positions in a list is really difficult. I consider myself to have some kind of obsessive compulsive disorder and joining is going to give me a few long nights for a few days, until I lose interest and find the next shining thing.
The topic of resource reservation and allocation is very delicate. I am of the school of thought that is better to have enough resources to make everyone happy and still be at 75% utilization rather than having to manage resource shares. Well, in an ideal world everyone gets its own share and never takes more, right? For the real world in which the budget is never enough, and VM’s roam the hosts consuming resources unscrupulously we have the vSphere feature of resource allocation/management.
Resource Allocation Shares
Shares specify the relative importance of a virtual machine (or resource pool). If a virtual machine has twice as many shares of a resource as another virtual machine, it is entitled to consume twice as much of that resource when these two virtual machines are competing for resources.
Shares are typically specified as High, Normal, or Low and these values specify share values with a 4:2:1 ratio, respectively. You can also select Custom to assign a specific number of shares (which expresses a proportional weight) to each virtual machine.
These values depend on the number of vCPUs and RAM on each VM. Take a look at Table 2-1 on page 12 in the vSphere Resource Management pdf (link at the bottom).
Figure 1. Right click on a VM and select Edit Resource Settings:
Figure 2. Edit Resource Settings Window:
Here you will find the values I mentioned before, and a couple more. The Reservation is the guaranteed allocation of resources and the Limit is the upper limit.
We can also configure Resource Pools to simplify the administration of resources, instead of configuring resources on a per VM basis, we can manage those VMs in groups.
One last concept you should be familiar with is Admission Control (before we dive into the blueprint for this topic).
When you power on a virtual machine, the system checks the amount of CPU and memory resources that have not yet been reserved. Based on the available unreserved resources, the system determines whether it can guarantee the reservation for which the virtual machine is configured (if any). This process is called admission control. If enough unreserved CPU and memory are available, or if there is no reservation, the virtual machine is powered on. Otherwise, an Insufficient Resources warning appears.
Check this post about Resource Pools by Duncan Epping on Yellow-Bricks, it is a little bit old, but I think is still relevant.
After this brief introduction, I am going back to the exam blueprint and try to explain each concept. In this section we are going to cover the following topics:
Create/Remove a Resource Pool
Add/Remove virtual machines from a Resource Pool
Configure custom resource pool attributes
Determine the effect of the Expandable Reservation parameter on resource allocation
Create a resource pool hierarchical structure
Determine how resource pools apply to vApps
Describe vFlash architecture
Create/Delete vFlash Resource Pool
Assign vFlash resources to VMDKs
Given a scenario, determine appropriate shares, reservations and limits for hierarchical Resource Pools
Create/Remove a Resource Pool
From you Home view in the vSphere Web Client, click Hosts and Clusters in the middle pane, then right click on a host to find the New Resource Pool link.
Figure 1. New Resource Pool
Figure 2. New Resource Pool Configuration Window
After you create a Resource Pool you ca delete it by going to the same view from which you created it and right cl1ck on the Resource Pool name.
Fogire 3. Remove Resource Pool
Click Yes on the popup confirmation dialog window and done.
Add/Remove virtual machines from a Resource Pool
You can drag and drop the VM in and out of at Resource Pool or you can migrate the VM into a Resource Pool.
Video 1. Migrate a VM to a new Resource Pool
Configure custom resource pool attributes
Going back to Resource Allocation Shares:
Shares are typically specified as High, Normal, or Low and these values specify share values with a 4:2:1 ratio, respectively. You can also select Custom to assign a specific number of shares (which expresses a proportional weight) to each virtual machine.
Figure 4. Custom Resource Pool Attributes
If you click on the Resource Pool (RP-Parent) then you can click on Edit resource pool settings and the configuration dialog will appear.
Determine the effect of the Expandable Reservation parameter on resource allocation
When the check box is selected (default), expandable reservations are considered during admission control. If you power on a virtual machine in this resource pool, and the combined reservations of the virtual machines are larger than the reservation of the resource pool, the resource pool can use resources from its parent or ancestors.
In other words, if my son wants to watch a movie and he only has $3 he will come to me and ask for money, if I have some cash and I can give it to him, then he can go to the theater, if not he stays home.
There is a great example on page 57 of the Managing Resource Pools pdf (link at the bottom).
Create a resource pool hierarchical structure
Figure 5. Parents, Children, and Siblings in Resource Pool Hierarchy
The root resource pool is the cluster in this case.
Determine how resource pools apply to vApps
A vSphere vApp allows packaging of multiple interoperating virtual machines and software applications that you can manage as a unit and distribute in OVF format.
A vApp can contain one or more virtual machines, but any operation carried out on the vApp, such as clone or power off, affects all virtual machines in the vApp container
Reservations on vApps and all their child resource pools, child vApps, and child virtual machines count against the parent resources only if those objects are powered on.
Procedure
1. Navigate to a vApp in the inventory and click Edit vApp Settings.
2. In the Deployment section, click CPU resources to allocate CPU resources to this vApp.
3. In the Deployment section, click Memory resources to allocate memory resources to this vApp.
4. Click Ok.
Describe vFlash architecture
VMware vSphere Flash Read Cache virtualizes server side flash providing a high performance read cache layer that dramatically lowers application latency. The caching is fully transparent to the VM without requiring any guest agents. Flash Read Cache enables allocation of flash resources at a per-VMDK granularity providing VMware vSphere vMotion consistent read caching and integration with VMware vSphere Distributed Resource Scheduler for initial placement.
vFRC is enabled on a per-VMDK basis. Each VMDK can be configured with a certain size of flash cache with a certain cache block size. Once vFRC is enabled for a virtual disk, the cache is created when the virtual machine boots. vFRC is a writethrough cache. This means that even though write I/O requests are cached by vFRC, I/O request completion status is sent to the guest virtual machine only after the data is written to physical storage. Because of this design, there is no change in the existing data reliability and availability guarantees.
Create/Delete vFlash Resource Pool
Figure 6. Navigate to Host and Clusters -> Click a Host -> Click Manage -> Click Settings -> Click Virtual Flash Resource Management and then click on Add Capacity
Figure 7. Selection Dialog Window
After you select the capacity in figure 7, you will be able to see the available amount in figure 6.
Assign vFlash resources to VMDKs
Figure 8. Edit settings on a VM to add vFlash to a vmdk. Click Advance to enable the cache
Given a scenario, determine appropriate shares, reservations and limits for hierarchical Resource Pools
For individual VMs take into consideration the number of vCPUs and the amount of memory:
For example, an SMP virtual machine with two virtual CPUs and 1GB RAM with CPU and memory shares set to Normal has 2×1000=2000 shares of CPU and 10×1024=10240 shares of memory.
I love the Brocade GUI, but I hate Java. I usually find my way with the workarounds out there to be able to display the Java plugins, but this time I wasted too much time on something that can be done in the command line. I was at a customer’s site and we added a few servers to the Fabric. These are the commands I used to update the zoning.
switchshow: Displays switch and port status.
portloginshow <port #>: display WWPN on NPIV ports
alishow: Displays zone alias information.
cfgshow: Use this command to display zone configuration information.
zoneshow: Displays zone information
zonehelp: To help find the commands you will need.
With the information I gathered from the commands above I was able to add aliases, zones and then add the zone to the current configuration. The save and enable the config.
And there you have it, three servers with two HBA’s each, added to a VNX with two FC connections to each Fabric. In this case the original configuration doesn’t make any distinctions between servers on each switch. I always try to add HBA1 or HBA2, or SPA_P0 and SP_P1, but the approach used here makes it very simple. Check the complete guide:
Two weeks ago I was fortunate to be at VMworld 2015. I think it was of great value for me as a professional to be able to attend because by participating in the different sessions and Hands on Labs I learned and validated a lot of the knowledge I use on a day to day basis. I wrote some of this in my post VMworld PEX, but I would like to go over one of the sessions that was about Best Practices for vSphere across different scenarios. The session started with the presenter talking about virtualization all in. There is no reason or, in other words, there is not a single application that could not be virtualized and work for production. He made the disclaimer that the design might call for a change in hardware to accommodate for the performance needed, but in the end it could be done. Here is a summary:
VMware all in!
vSphere is achieving low network latency, into the microseconds.
Due the characteristics of virtualization and its optimizations, VMware was able to score better performance running a hadoop cluster than the physical counterpart.
The vCenter Web Client runs faster on Chrome
Install the vCenter DB close to the vCenter, avoid having the database across the network.
Place the vCenter and its database on Tier one storage.
Don’t change statistics levels, the ones that come with the vCenter are practically useless, use Operations Manager instead.
Check VMware KB 2021302 (previous to 6) for JVM correct sizing.
Use Latency Sensitivity cautiously, it reserves a core.
You can adjust the latency sensitivity of a virtual machine to optimize the scheduling delay for latency sensitive applications.
ESXi is optimized to deliver high throughput. You can optimize your virtual machine to meet the low latency requirement of latency sensitive applications. Examples of latency sensitive applications are VOIP or media player applications, or applications that require frequent access to the mouse or keyboard devices.
Use multiple vSCSI adapters
Don’t user RDM’s. And if you need them for Microsoft Cluster then stop using MS cluster :).
Follow you OEM’s integration guides, all vendors have white papers on how to configure this or that specifically for their equipment.
In Windows set the Power Policy to High Performance to avoid core parking.
The use of network receive side scaling (RSS) must be enabled at the guest NIC and Windows OS.
Rightsize is better that oversize, configure the VM’s with what they need. I would add to oversize a bit on the host side.
Oracle doesn’t leverage NUMA well and often benefits from disabling it at the database level
Disable Interrupt Coalescing
Messaging Recommendations
Disable Interrupt Coalescing
Fast Storage!
In general, single threaded applications will benefit from higher frequency CPU’s, a high multicore CPU will work better for applications with a higher degree of parallelism.
Don’t over commit the memory, design for approximately 75% RAM use. Take into consideration your admission control.
Last night I experienced what happens after hours around VMworld, and it was a lot of fun. First I headed to the vExpert reception and there were vExperts from around the world literally, I met people from Costa Rica, Australia and of course, the US. After a couple of speakers, Pat Gelsinger came down to the floor to share a few minutes with us, it was really exciting. After that we walked to the VMware Latin America appreciation event, it is always refreshing to share some time talking about whatever we can think about, but in Spanish. While at this event we heard that VEEAM was throwing a party and apparently it had a great reputation, so we headed that way. We finish our night at the VEEAM party, it lived to the hype, and we had a lot of fun.
I am going to skip day one, we saw the vMotion across clouds, something that we have been waiting for a while. For day two we were promised a surprise guest, as I write this I am waiting to see who it is (live blogging). Twice now they mentioned something about the projection to the screens, maybe Apple paid for them to pave the road ahead? I am not sure, but let me go back to the main subject of this session, and that is mobility and end use computing. This is the ecosystem from VMware, SDDC, EUC and the App, being used from any device.
The apps will come from any device, in a secure and flexible way. IDC named VMware a market leader in EUC for 2015, ahead of Citrix, but they assured to the general session that they would continue to optimize everything for Xen.
The secret guest is from Microsoft is Jim Alkove VP of security. The vision for Windows 10 is IT simplified. “VMware loves Windows 10” The conversation revolves around security and identity management to be able to achieve the app mobility. In my opinion seeing Microsoft and VMware together on stage is very reassuring.
VMware Workspace Suite joins the application delivery management and the device. VMware is combining Airwatch and App volumes to bridge the gap between cloud and legacy apps, Project A2 (square, not two). One thing I can see from yesterday’s session and today’s is that NSX is an integral part on all the new VMWare capabilities.
Sesame Street simple! Now Martin Casado joins the stage to talk about security. BTW this is just an encore of the general session at PEX back on Sunday. What is going to be the VMware strategy on security, is called micro-segmentation using NSX (network virtualization) no to be confused with Software Defined Networking (SDN).
The surface attack is becoming too wide, and the challenge is to deliver trusted applications to devices that may not be trusted or secure. VMware is presenting Distributed Network Encryption to protect the data. Also encryption of Data at Rest, like the vmdks. I wonder if the NSA already has some master keys for all the apps ;).
Pat Gelsinger joins the stage now for the closing. The online connectivity estimates for 2025 is 5 billion. The size of the internet economy is 21% of the World GDP. To win or join the ecosystem, the start-ups need to play with a different set of rules and create an asymmetric battle. We are all expecting Flo to join Pat now on stage, but seriously Pat is giving us a view of how innovation can disrupt incumbent technologies and how it can change the way we live today.
The challenge is still on the bridge from the private to the public cloud. That is where VMware with its Unified Hybrid Cloud enters the game giving apps the capability to span across both clouds, becoming the Hybrid Applications with common networking, management, and security. Is about enabling a global point of view across all clouds. The apps and your data will be within you “borders” (under you rules) no matter where they reside. The Unified Hybrid Cloud is the future.
Virtualization provides the best platform to architect for security. Virtualization has the perfect alignment to be able to deliver the app or service to where it should be. Ubiquity -> the capacity to be present everywhere at the same time. I had to look that up, sorry. We are in the renaissance of Security. The building blocks of proactive technology are analytics and Big Data. Rule one of building the cloud is to automate everything. Of the top 100 of the IT companies, 50 of them will disappear in the next decade. These were the last remarks from Pat, now back to VMworld!.
P.S. As always pardon my English, more so with this post on the run.
This week I am at San Francisco for VMworld 2015. This is my first time at VMworld, and I am really enjoying the event. This past Sunday we had a really busy and long day. The day started with a nice breakfast at Mel’s, thanks to Simplivity for picking up the tab. Then I continued my day with some Best Practices, I try to stay away from marketing sessions as much as possible. My first stop was at session #PAR6421, Best Practices for Deploying IaaS with vCloud Suite and vRealize Automation. The session discussion was around Process, Architecture, Backup, and Upgrading. The basic idea is not too different than any other consulting engagement. The first and most important advice is to ask the customer what is it that they want, then design for that aligning your solution and products with those needs. For automation, is important to understand the day to day tasks of the administrator, and try to automate those first. Pay attention to repetitive tasks, automating those will lower the errors due to a manual process. The basic idea here is to understand the processes and make sure that these processes are good, it is a huge mistake to automate a bad process (garbage in, garbage out).
For the architecture part, make sure that you are using the reference architecture documents and identify the kind of use or expected functionality from the system. If the client is expecting to use HA and a system that can’t go down or have downtime, then choose the appropriate reference architecture for this (medium). If the client can handle an RTO of 24 hours, then you can be confident that the small core reference architecture will be enough.
I am not ging to discuss backup in depth because it follows common sense, like backing up everything after the installation, but before the actual customization begins. Also make sure that a backup is consistent across the platform, don’t backup one part now, and wait 20 minutes to backup another.
In terms of upgrading, if you are in version 5.x there is not a direct path upgrade, it would require a new installation and then a migration. Keep in mind that customizations might not carry on with the upgrade.
Moving on to the general session, we find the theme of magic phrase for VMworld, One Cloud, any application, any device. The concept of Hybrid Applications comes up and we go back to old catch phrases like The network is the computer (how I miss Sun Micro), but now the application is the network. The software is the wine and the hardware is the bottle.
Next I go across the street for the VMundergriund panels, I realize that is much more interesting to sit down and listen to a panel of experts talk about relevant topics than sit down and watch powerpoints for an hour. Anyway, it was nice to see Duncan in a 30-minute talk about VSAN use cases, then also later at the Solution Exchange see him talking about it again.
The real deal came at the Mark “A” session, on maximizing vSphere performance, here are some bullet points that might help:
Use Chrome as your web browser to connect to the vSphere Web client.
Install your vCenter close to its DB and in a TIER 1 storage.
Don’t change statistics levels in the vCenter, they are useless anyway,
Check you Java Virtual Machine size (previous to 6) KB2021302
Rightsize, not oversize and never undersize.
Don’t Use vCPU hot add, it will disable vNUMA
Select High performance in the BIOS
Always enable Hyper threading
Use the latest Virtual Machine hardware.
Keep VM tools updated (this one is mine)
Use vmxnet3
Disable Interrupt Coalescing
Use Jumbo Frames for thing like the iSCSI.
Use multiple vSCSI adapters
Don’t use RDM’s
At this point, I started being lazy and started using my phone as my documentation tool. Here is a picture of the performance for virtualized DB’s best practices
Finally, the best advice when troubleshooting performance is to know the key performance indicators and define the acceptable values, in other words, don’t accept a performance problem with a vague or subjective description. At the end of the day, we crashed into the VMunderground party for some more networking and socializing.