Articoli

Pubblicato il

NUC upgrade to ESXi 7.0.1 – Part 2

Phase 2: Upgrading ESXi Host

In the previous article, I described how to prepare a customized ISO. If you lost it please click on this link.

Let’s continue!

Requirements: the task I performed before starting the procedure here described has been the vCenter upgrade to 7.0.1-c. I remind you that my starting point was Esxi 7.0.0.b that is fully supported by vCenter 7.0.1-c.

There are two main ways to upgrade the ESXi Hosts.

The first is related to the use of a VMware feature. It’s the vSphere Lifecycle Manager (vLCM) and you can taste part of its potential by watching this video guide.

Instead, I preferred to use an old approach working with a bootable USB pen with the custom ISO prepared in the previous article. The steps are:

1. Shutdown ESXi Hosts
2. Remove the USB-NIC
3. Insert the Pen Drive with custom ISO
4. Upgrade the host following the wizard (The main point is shown in Picture 1)

Picture 1

5. Reboot the host.

6. Now it’s time to install the USB-Nic Driver. It is available from “USB Network Native Driver for ESXi” web site and this article allows me to say thx to the excellent job of  Songtao Zheng and William Lam

Before proceeding please read the instruction carefully and DO NOT insert all the USB cards together. (I have got three USB-NIC cards)

Why? Because during the procedure, I have had more than one purple screen and after a deep analysis I discovered that it depended on the USB-NIC cards.

To get over this issue I created the following procedure.

Picture 2

7. Switch off the ESXi NUC and insert the first USB-NIC card in port-1 and the second in port-4 (please refer to picture 2 to know the nomenclature of the port)

8. Now switch on the NUC and check if it boots correctly.

9. Switch off the NUC and insert the third USB-NUC on port-2.

10. Reboot NUC and check if it works as aspected.

Before ending this article I suggest creating a map between vmks and the physical MAC Address of the USB NIC. The main advantage is that it allows maintaining the same vmkusb address after a reboot also.

Some useful commands:

To Identify the Mac Address:
# esxcli network nic list |grep vusb |awk ‘{print $1, $8}’

To Check persisting binding:
# esxcli system module parameters list -m vmkusb_nic_fling

Pubblicato il

NUC upgrade to ESXi 7.0.1- Part 1

Last weekend I upgraded all my Servers to the last VMware ESXi release (7.0.1 C) and this article is meant to describe all steps I performed.

Just a recommendation before starting. I worked in my lab that it’s not a production environment.

MyLAB before upgrade:

  1. NUC8i7beh2
  2. VMware ESXi 7.0.0 (Build 16324942)
  3. Every NUC has three more network cards added to the embedded standard NIC. They have been obtained through the USB ports leveraging three adapter USB/Ethernet and the flings driver.  Please refer to the FLINGS website to get all info.

The procedure is composed of two main phases and this article will cover the first part.

Phase 1: Creating a customize ISO

Is this step required?

Oh well, it depends if the Standard ESXi VMware ISO has already the driver of your embedded network card inside. The standard ISO, unfortunately, does not contain the NUC8i7BEH network drive (it is named ne1000)

If you upgrade the ESXi through the standard ISO, the process fails with the error shown in picture 1.

Picture 1

How to get over it?

Before upgrading it, it’s necessary to know the driver used by the embedded NIC Card. If you don’t know it, please read the next instructions carefully (they are command launched on host ESXi you are going to upgrade):

1.   lspci -v | grep -A1 -i ethernet
take a note of the string composed of 4:4 values ( xxxx:yyyy)

2.   lspci -n | grep xxx:yyy
take a note of how the nic is named (in my case [vmnic0])

The next step is getting the name of the driver directly from the VMware website (Matrix compatibility).

From that web page, filling up the empty field with the value yyyy and filtering the result by IO Devices, it’s possible to get the device driver name.

For my LAB the result is shown in picture 2 where I highlighted the device driver name in yellow.

Picture 2

The last command to check if the driver is already installed (It should be present) is:

3.   vmkload_mod -l | grep “Device Driver”

In my case: vmkload_mod -l | grep ne1000
                          ne1000          1          352

Optional: if you use the USB ports to add more NIC, please uninstall the fling drivers before proceeding.

4.   esxcli software vib remove  –vibname=vmkusb-nic-fling (before vibname two scores –   –  )

It’s time to create our custom ISO

a- Download the offline bundle from VMware Site, for example:

VMware- ESXi-7.0U1c-17325551-depot.zip

b- Download the NUC ethernet driver for your device (ne1000 in my case).

I found an useful PowerShell script to get it:

#add the software repository
Add-EsxSoftwareDepot https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml
#define as variable the name of the driver
$vib = Get-EsxSoftwarePackage ne1000
$vib | select Name,Version | ft -AutoSize
$vib[4] | fl
#Get the driver
Invoke-WebRequest $vib[4].SourceUrls[0] -OutFile d:\pkg\ne1000_0.8.4-10vmw.700.1.0.15843807.vib

c- The PowerShell script to create a custom ISO is available on VMware Front Experience Site.

This great script has a lot of options; please refer to the official documentation to see how to create the ISO.

In my case I just launched the following command:

.\ESXi-Customizer-PS-v2.6.0.ps1 -v701 -izip D:\ISO\ESXi-7-0-1\VMware-ESXi-7.0U1c-17325551-depot.zip -pkgDir D:\pkg\ -OutDir D:\ISO\ESXi-7-0-1\ -nsc

d- The last step is creating a bootable USB pen using the just created custom ISO as a source.

I have chosen Rufus to perform this task.

In the next article, we are going to see the final step to upgrade the NUC

Pubblicato il

Ransomware defense part 4: Deep Dive

In this last article about Security and Ransomware, I’m going to add new features and deep dive those you’ve already read in my first article about Veeam Backup & Replication.

The starting point is creating copies of your data (remember the 3-2-1 rule) and from these copies perform automatic tests of availability and security.

Which technology should be deployed to sleep safer?
In this article, I’m going to show you some Veeam technologies that address the threats explaining why they are a must to be used.

Sure-Backup

It is the best way to be certain that a backup is really usable.

What does it mean exactly?

In my public speeches, I often use a pen to explain the Sure-Backup concept correctly.

How can I be sure that the pen is usable? The answer is very easy. If it writes on paper it works and it is usable.

Going back to our scenario,  the only way to know if your backup is usable is to perform a restore in an environment logically separate from the production.

How Sure-Backup works?

It is composed of two parts:
The application group identifies the backup VMs that have to be verified.
DataLab is the way through which VMs are switched-on into a specific network that can’t communicate with the production network.

This great technology allows you to add the number zero meaning no errors (from 3-2-1 to 3-2-1-0) to the above rule.

One more important thing to add here is that you can create scripts to random test your backup.

In the following article by Luca Dell’Oca, you can have an excellent example of how scripts can help you.

How can you test 1000 VMs with Veeam SureBackup?

Sure-Replica:

VBR is a solution that allows customers to perform Backup and Replicas of your VMs.
Replicas are commonly deployed to create a Disaster Recovery Site.

Veeam has “integrated” the Sure-Backup technology for Replica too. It is named Sure-Replica and it gives all advantages shown for Sure-Backup

For managing the automation of Replicas I suggest customers use a new Veeam product, the Veeam Availability Orchestrator (VAO).

On this site, you can find a very detailed guide to use set-up VAO.

Veeam Availability Orchestrator – v.3.0 – VAO-Baseline-1

Secure Restore

It scans the saved  VM Windows  (VM that has at least one valid restore point), with your antivirus software before restoring the VM to the production environment.

Secure Restore is available for the following restore tasks:

  • Instant VM Recovery
  • Entire VM Restore
  • Virtual Disks Restore

The only requirement is that your antivirus has to be installed on the mount server and supports CLI.

Data Integration API

Veeam Data Integration API is a set of Veeam PowerShell cmdlets that allow you to represent data of backup files as a mounted Windows folder.

This feature allows you to access backed-up data in read-only mode.

It has a lot of possible usages to example data mining and data warehouse.

In the field of security, it can be adopted to check if a virus is already present in your VM checking via backup files the guest OS files through your antivirus software (It scans the files of the VMs and not the backup file). A good example of use is in the following blog article by Niels Engelen

V10: Reuse your data with the new Data Integration API

Storage Integration

VBR has two different storage integration.

The primary storage integration allows customers to perform backup more frequently because this technology allows creating backup without impacting the VMware environment.
The integration allows creating a test environment because it is integrated with Datalab and Sure technologies.

The secondary storage integration is commonly used with deduplication appliances that can allow you another layer of security for ransomware.

When the Veeam Data Mover Service is installed directly on the appliance, it reduces the risk of ransomware encrypting and deleting data.

In addition, if for some reason the first data is compromised, copies can be replicated to a second data center or in the cloud.

Before leaving you my two last cents:
1) The starting point is always to perform copies of your data. Veeam gives you powerful tools to manage them and to perform more check to verify that your data are safe from threats.

2) Some customers still think that the backup asset is just a cost because they are used just to restore. With Veeam you can use backup data to perform different actions relieving the production of not business core tasks (for example think the great use of Data API Integration for data mining, data warehouse etc)

Take care guys and see you at soon

Pubblicato il

Ransomware defense part 3: Monitoring and more

In the previous articles, I described some good ideas to design your architecture to keep it safer as much as possible.

One of the greatest challenges the IT guys have to face is finding the right balance among design, deployment and budget.

It’s very important to have the right tools to measure architectural behaviour. In this way you can easily:

  1. Watch from a privileged point of view the architecture. Let’s image to be on the top of a mountain watching people and goods moving at the bottom of the valley”
  2. Launch the defending actions when an attack is on-going. Referring to my previous example, it’s like blocking some passages to people and goods.
  3. When the attack is over remove any possible threat left (cleaning the passages).
  4. Do a thorough workup understanding of the weak points of your architecture and create a plan to reinforce it.

Monitor tools are your sentinels, but they need to be trained to trigger also the first defense lines. Imagine the new sentinel as a lieutenant warrior with a varied arsenal of weapons.
To be clearer: the required features monitor and respond to actions in function of the severity of the alarm.

But why is measuring so important? The reason is that you can define the KPI (Key performance indicator) for your environment and periodically check if the measures are respected.
In other words, it is possible to measure the service level and understand if the budget and skill invested in the company are enough to address the backup security challenges or if more tunings actions or some great changes are needed.

Let’s see how to use Veeam One to address this common request:

The Possible ransomware activity alarm keeps tracking of the Operating system of the VM.

As shown in picture 1 the monitored counters are by default CPU, Datastore write Rates and networking transmit rate (the case of copy offsite of sensible data for future blackmail).
The value counters can be changed to adapt to your own needs (Tuning phases) and more counters can be added to monitor more objects as shown in picture 2.

Picture 1

Picture 2

Another alarm already present in Veeam One is “Suspicious increment backup size“.

It checks if the restore point size is significantly different from the previously created ones.

The two main reasons I like  Veeam ONE are:

  1. Very easy to use
  2. Customizing the action after an alarm has been triggered

Thx to “customizing action” it’s possible to launch your antivirus/antimalware on the VMs belonging to the backup job that has triggered the suspicious alarm, or disconnect the repository from the network, or what else you wrote on your incident and rescue procedure.

The main point here is that you can manually click on it or automatically execute the action as shown in picture 3

Picture 3

Veeam One has furthermore an exhaustive technology of reporting.

If an alarm is a good way to intercept an error or a misconfiguration because  it works in real-time, through the reporting it is possible to check the status of your protection (KPI, SLA….), understanding the exercise and security cost of your production environment and forecasting the new investment to implement in the next years.

Which are the reports to use?

All of them are important and an all report list is available from the following link: Reports 

Just as an example please check the use of the following

The next article will talk about which are the automatic procedures you can adopt to check your backup infrastructure.

Take care and see you soon

Pubblicato il

Ransomware defense part 2: Hardening

There are many documents on the internet that describe how to address this common request.

In this article, I’ll give you a track to move easier around this topic pointing out the most interesting articles.

Before starting let me thank Edwin Weijdema who created an  exhaustive guide to answer the common question (please click here to get it)

Are you ready? Let’s start

1- The first magic point for starting is Wikipedia where I got a good definition:

In computing, hardening is usually the process of securing a system by reducing its surface of vulnerability, which is larger when a system performs more functions; in principle, a single-function system is more secure than a multipurpose one. Reducing available ways of attack typically includes changing default passwords, the removal of unnecessary software, unnecessary usernames or logins, and the disabling or removal of unnecessary services.

2- The second point is to understand the concept of Perimeter security:

It is natural barriers or artificially built fortifications that have the goal of keeping intruders out of the area . The strategies can be listed as:

  • Use rack-mount servers
  • Keep intruders from opening the case
  • Disable the drives
  • Lock up the server room
  • Set up surveillance

A complete article is available by clicking here

3- The third point is  Network segmentation:

It is the division of an organization network into smaller and, consequently, a more manageable grouping of interfaces called zones. These zones consist of IP ranges, subnets, or security groups designed typically to boost performance and security.

In the event of a cyberattack, effective network segmentation will confine the attack to a specific network zone and contain its impact by blocking lateral movement across the network via logical isolation through access controls.

Designating zones allows organizations to consistently track the location of sensitive data and assess the relevance of an access request based on the nature of that data.  Designating where sensitive data reside permits network and security operations to assign resources for more aggressive patch management and proactive system hardening.

A complete article is available by clicking here

4- Hardening your Backup Repositories

The next good rules involve your backup architecture and in specific the Backup Repositories:

Windows:

a. Use the built-in local administrator account

b. Set permissions on the repository directory

c. Modify the Firewall

d. Disable remote RDP services

Linux:

e. Create a Dedicated Repository Account

f. Set Permissions on the Repository Directory

g. Configure the Linux Repository in VeeamModify the Firewall

h. Use Veeam Encryption

Do you want to know more about security? If so the Veeam Best Practices are for sure the answer.

The next article will cover monitoring and automatic actions using Veeam-ONE.

5- Prevent injection of shady boot code​

Code injection, also called Remote Code Execution (RCE), occurs when an attacker exploits an input validation flaw in software to introduce and execute malicious code.

To prevent the attack please follow the following rules:

a. Run with UEFI Native Mode​
b. Use UEFI with Secure Boot Standard Mode​
c. Combine Secure Boot with TPM
d. Equip critical servers with a TPM 2.0

Stay tuned and see you soon

Pubblicato il

Site Restyling

In the next few days, the site will have some changes. The main reason is helping the internet surfer to find faster the article they want reading.

The First 5 improvements are:

1. The “who am I?” page has been included in the “all articles & about me” page
2. All the articles present on the site can be easily consulted from the “all articles & about me” page.
3. The backup menu name has been changed to Veeam.
4. The Veeam menu has three new categories. It allows identifying the technology scope
5. BBO bridge tournament results have been added to the bridge menu.

 

Nei prossimi giorni il sito subirà alcune modifiche. Lo scopo è quello di aiutare i frequentatori del sito a trovare più velocemente gli articoli che vogliono leggere.

I primi 5 miglioramenti sono:

1. La pagina “who am I?” è stata inserita nella  pagina “all articles & about me”
2. Tutti gli articoli presenti nel sito sono facilmente consultabili dalla pagina “all articles & about me“.
3. Il nome del menu backup viene modificato in Veeam.
4. Al menu Veeam sono state aggiunte 3 nuove sotto categorie per meglio individuare in quale ambito utilizzare le tecnologie.
5. Al menu del bridge sono stati aggiunti i risultati dei tornei di bridge BBO.