Check Replica Status – Before deleting it – Part 1

Last month, a partner had to face-up a strange VBR behavior.

From VBR console he deleted a VM’s replica (Picture 1), and suddenly the production VM has been also erased (don’t worry, before doing any activity he tested the backups using sure backup technology).

Picture 1

The reason why it happened become clear to me once I read the logs.

To do it shortly, some weeks before someone started a Failover directly from vCENTER console without doing any communication to the internal IT team.

This article wants to explain how to avoid this common mistake.

The first step is understanding some basic concepts:

a) VMware identify any single VM with a number named MorefID and a UUID.

b) Any single operating system has an identifier named Instance UUID (Universal Unique IDentifier); in my lab, I set-up  more than one replica job for a single VM

Table 1. row 2. shows the name of production VM (Ubuntu-02), its morefID  (vm2270), where it is running (Milan), the UUID (…bcc12) and its VM UUID … f58b.

Table 1. row 3-4 shows the name of  VMs replicas,  morefID, instance UUID and its UUID.

All tables shown in these articles have been created using Veeam One

Name PCName morefID DataCenter Instance UUID UUID
Ubuntu-02 ubuntu-02 vm-2270 Milan 502d2405-cc8f-de73-1a19-57e8496bcc12 564d013a-7835-9d1b-841e-32855790f58b
Ubuntu-02_Rep_VC01 ubuntu-02 vm-2694 Milan 502d2d90-d08f-08aa-efcf-d9feaa1d13f8 564d013a-7835-9d1b-841e-32855790f58b
Ubuntu-02_Rep_VCDR ubuntu-02 vm-399 Venice 501d517b-672e-30e0-665a-fd4b4af7dcb6 564d013a-7835-9d1b-841e-32855790f58b


Picture 2 shows the VM  source (highlighted in yellow) from vCENTER console.

Picture 2

After checking-up that VM source is switched off, it’s possible to start a Failover (Picture 3).

Picture 3

Next five pictures show the step by step wizard to complete the procedure correctly.  As you can see from picture 4 the VM that has been replicated with two different jobs (Picture 5) is always Ubuntu-02.

Picture 4

Picture 5

Pictures 6-8 show the result of the failover.

Picture 6

Picture 7

Picture 8

What happens when you complete the task with the Permanent failover? (Picture 9/10/11)

Picture 9

Picture 10

Picture 11

First of all, comparing picture 3 with 12 it is possible to see that one of the Replica Ready VM, and precisely the VM in permanent failover, has been deleted 

Picture 12

Picture 13 shows that now the replica job contains 0 objects. The right behavior is confirmed by pictures 14,15,16 and 17  where it is shown that the replica is not available anymore.

Picture 13

Picture 14

Picture 15

Picture 16

Picture 17

The cloning job option didn’t change the correct behavior (Pictures 18 and 19)

Picture 18

Picture 19

Let’s sum up. Following the right procedure, the Failover works as aspected

Now …. why the VM has been deleted? The next article will explain it in detail.

Check Replica Status – Before deleting it – Part 2

My previous article explained the procedure to perform a failover from VBR console explaining why it is safe (Click here to read it)

In this second article, I’ll show you what can happen if you try a failover in a different way, answering the behavior that a partner had during a cleanup procedure.

In my lab, I created a new Replica Job where the original VM is still Ubuntu-02 (Picture 1) and the replica VM has the suffix _Rep_vc01-1-1 (Picture 2).

Picture 1

Picture 2

After the VM has been created (Picture 3) It is switched it on directly from the vCENTER console (Picture 4). To be sure it works as expected, it is possible to connect remotely.

Picture 3

Picture 4

Picture 5

Going back to VBR console it’s possible to see that nothing has changed (Picture 6) just because the power-on action has been performed directly from the  VMware console

Picture 6

Attention point: If you try to perform a replica task it fails because the VM is running (picture 7)

Picture 7

Now the main point of the two articles:

It’s a bad choice deleting VMs from VBR “Ready Replica VM”  menu (picture 8/9) without knowing if the VMs have been started from vCENTER console. Why? Because also the production VM gets deleted as shown in picture 10.

Picture 8

Picture 9

Picture 10

Let’s get a little wild with some supposing:

1. If you see the “active” status icon switched on ( from VBR console) it means the failover is started (picture 11)

Picture 11

2. If a permanent failover was performed, the VM disappears from “Replica Ready” menu and replica job results to be empty (Picture 12)

Picture 12

3. If the replica job works fine it means that no permanent failover has been performed

4. If the replica Job works fine but when clicking on the delete button (from “Replica ready menu” of VBR console) (picture 13) the production VM disappears, it means that a new replica job has been re-created after the manual failover has been launched (picture 14).

Picture 13

Picture 14

Knowing that deleting a VM replicated from VBR console needs a little attention, especially if you do not have the continuous and complete control of the VMware architecture, the question is: is it possible to think an easy checking-up before deleting VM?

The answer is Yes and Veeam One can easily help just creating them.

a. From vCENTER: setting up a report that checks if the VM to be deleted is running (power state status) (Table 1)

b. From VBR: if a Replication job is setted-up for that VM (Table 2)

Name Computer name MoRef Datacenter Instance UUID Boot  Power-State
Ubuntu-Master ubuntu-10 vm-1243 Milan 502da7c3-ae30-0448-58f8-5bb220af7b2d 28/05/2020 19:56:42 powered-Off
Ubuntu-Proxy ubuntu-03 vm-126 Venice 501d20dd-afc8-84ae-a9d4-d8cfa8e91fca 31/08/2020 11:38:31 powered-On
Ubuntu-02 ubuntu-02 vm-2270 Milan 502d2405-cc8f-de73-1a19-57e8496bcc12 powered-Off
Ubuntu-02_Rep_VCDR Not set vm-399 Venice 501d517b-672e-30e0-665a-fd4b4af7dcb6 powered-Off
Ubuntu-02_replica_VAO Not set vm-413 Venice 502d2405-cc8f-de73-1a19-57e8496bcc12 powered-Off
Ubuntu-02_Rep_vc01-1 ubuntu-02 vm-2760 Venice 502d7549-a3cd-d58d-92bd-3741f6705478 02/09/2020 19:04:41 poweredOn
Ubuntu-02_Rep_VC01_3 Not set vm-2780 Milan 502d37f9-18f7-8cf2-858d-068850884463 poweredOff

Table 1

Name Has Replica Host Guest OS Has Restore Point Is Backed Up by Multiple Jobs Platform
Ubuntu-02 True esx-01.thegable.internal Ubuntu Linux (64-bit) True True VMware


Is there another way to check it up?

Yes. using Powershell scripts.

The example you can find here below is just the first idea that can be polished with a little bit of your effort.

NB1: I’m not a PowerShell expert, I just love writing scripts easy to read from anyone.

NB2: Before trying it please ask your PowerShell expert a consultant!

NB3: It is meant to be launched from VBR.

NB4: If you think that it can be a feature request write to me!

#snapin Veeam
Add-PSSnapin VeeamPSSnapin -ErrorAction SilentlyContinue
#accessing to vCenter
$credentials=Get-Credential -UserName administrator@yoursite.local -Message “Enter your vCenter password”
Connect-VIServer -server yourvCenter -Credential $credentials
#listing of all job configurated in VBR
#selecting replica job
$replicajobs = $alljobs | Where-Object {$_.isReplica -eq ‘true’}
#getting just Name
#getting VM powered on
$VMsPoweredOn = get-vm | where {$_.Powerstate -eq “PoweredOn”} | Where-Object {$_.Name -like “*Rep*”}
#Showing Replica jobs name & VMs switched on
Disconnect-VIServer -Force -confirm:$false
Read-Host “Replicated VMs = $replicajob_name “,”VMsPoweredOn = $VMsPoweredOn” “Click enter to Exit”

Veeam Availability Orchestrator – v.3.0 – DR from Netapp – Storage Replica

The fifth article will show how to use VAO version 3.0 to automatize the Disaster Recovery using the Netapp snap-mirror technology as an engine.


Netapp – SnapMirror

The article (quite long) is composed of 6 parts:

  1. Setting up Netapp Snap-mirror Protection
  2. Setting up the Recovery Location
  3. Setting up the Scope
  4. Creation of Orchestration Plan
  5. Starting the Plan
  6. Checking the Orchestration Plan status

1 Setting up Netapp Snap-mirror Protection

In my personal lab, I added two NetApp simulators 9.6 setted-up in peer relationships.

Pictures 1 to 5,  show how I set-up  a protection rule for a single Volume (named Vol_iScsi_N01)

Picture 1

Picture 2

Picture 3

Picture 4

Picture 5

Picture 6 shows the VMware console view of DR site; the replicated volume is presented as read-only volume.

Picture 6

Tips: My personal suggestion is asking your storage expert the right procedure to set-up a snap-mirror relation between two Netapp storages.

2. Setting up the Recovery Location

The recovery location wizard is shown in pictures 7, 8 and 9.

Picture 7

Picture 8

Picture 9

3. Setting up the Scope

The Scope wizard is shown in picture 10.

Picture 10

4. Creation of Orchestration Plan

Let’s go back to the main steps.

The wizard drives the users to select the right voices as shown in pictures 11, 12, 13 and 14.

Picture 11

Picture 12

Picture 13

Picture 14

The test scenarios are available in this configuration too

The Readiness Test is a low-impact and fast method one to confirm that configuration of an orchestration plan matches the DR environment.

The Data-lab test verifies the DR plan starting VM in a separate network.

 As shown in pictures 15,16,17 and 18. the steps are:

– Assigning  Datalab to VM Group (it is available from the admin menu)

– Setting up the Lab Group

Picture 15

Tips: for testing the Netapp integration select the option Restore.

It means the Replica option is just available for VBR Replica jobs.

Picture 16

Picture 17

Picture 18

5. Starting the Plan

Next pictures show how to run the just created Orchestration plan.

Picture 19

Picture 20

Picture 21

Picture 22 shows the restore points available.

Tips:  the shown Restore points are the replicated snapshot  (created by snap-mirror).

Picture 22

Picture 23 shows my favorite option available on VAO 3.0 in the Storage replica scenario.

Let’s imagine you run the orchestration plan.
While failover is running, the primary site is back normal operativity.

Your IT manager asks to revert the failover. Is it possible? Well, it can be. If you selected “reprotect storage volumes after failover” during the steps you can easily do it!

Picture 23

Picture 24

Picture 25

6. Checking the Orchestration Plan status

Time to run the orchestration plan and watch the steps performed.

I would like you to give your attention to the following three pre-plan steps:

a. Breaking the snapshot relationship

b. Putting the destination volumes on-line

c. Mounting the volumes

(Clicking on the picture you can enlarge the images)

Picture 26

From picture 27 you can see  the main steps:

d. VM registration

e. Network founding and connection

f. Booting VM

Picture 27

Pictures 28 and 29 show the result of post plan steps:

g. heart-beating test

h. Unmounting source Datastore

Picture 28

Picture 29

Tips: how to check in 5 points if everything is correctly working

1. VM in the DR site is running (from DR vCenter console connect remotely to VM as shown in picture 30 and 31).

Picture 30

Picture 31

2. Source VM has been deleted (from Production vCenter console check if VM is disappeared) (Picture 32).

Picture 32

3. The Orchestration plan launch button is grey and the plan has to be reset (picture 33)

Picture 33

4. Destination volumes on Netapp console have been set as read/write (Picture 34)

Picture 34

5. The Netapp relationship between source and destination volumes is broken (Picture 35)

Picture 35

– Readiness check report example dowload

That’s all folks for now.