Hi guys and thanks for replying and sharing your experiences...
I correct my self saying that there are 3 vm (maybe is not so important but for specifying): "LinuxServer", "WindowsServer" and "CentOS6", but the error appears always during the cloning of "WindowsServer" vm.
I checked the "VM_BACKUP_ROTATION_COUNT" and is set to 2 in every vm backup config file. I can't figure out why it doesn't happens every night: if it would be a disk free space problem it should do so, but it happens every 3 schedules.. another weird whing is that it past all gone ok: since a certain moment, the backup starts to fail (every 3 days).
@Scowse: what do you mean with "Are you quiescing by any chance?"?
There is another strange thing, maybe related with this: the "VM_BACKUP_ROTATION_COUNT", as I said is 2, but in the WindowsServer destination backup directory there are 3 subdirs (the other vm backup dirs are ok):
/vmfs/volumes/6fbca334-cebfcc77/WindowsServer # ls -l
drwxr-xr-x 1 root root 4096 May 28 03:23 WindowsServer-2013-05-25_22-00-01
drwxr-xr-x 1 root root 4096 May 27 02:49 WindowsServer-2013-05-26_22-00-01
drwxr-xr-x 1 root root 4096 May 28 03:23 WindowsServer-2013-05-27_22-00-02
/vmfs/volumes/6fbca334-cebfcc77/WindowsServer # ls -l WindowsServer-2013-05-25_22-00-01/
-rw-r----- 1 root root 30 May 26 03:22 STATUS.ok
It seems that ghetto couldn't delete the WindowsServer-2013-05-25_22-00-01/ directory... Here is the log of the good schedule (in debug mode) and it said "Removing /vmfs/volumes/backup_esxi//WindowsServer/WindowsServer-2013-05-25_22-00-01" but actually it didn't do it. I manually deleted it, I'll see if this helps (what ingenuous... :-) )
2013-05-27 23:15:09 -- info: Initiate backup for WindowsServer
2013-05-27 23:15:09 -- info: Creating Snapshot "ghettoVCB-snapshot-2013-05-27" for WindowsServer
2013-05-27 23:15:14 -- debug: Waiting for snapshot "ghettoVCB-snapshot-2013-05-27" to be created
2013-05-27 23:15:14 -- debug: Snapshot timeout set to: 900 seconds
2013-05-27 23:15:15 -- debug: findVMDK() - Searching for VMDK: "WindowsServer.vmdk" to backup
2013-05-27 23:15:15 -- debug: /sbin/vmkfstools -i "/vmfs/volumes/datastore1/WindowsServer/WindowsServer.vmdk" -a "lsilogic" -d "thin" "/vmfs/volumes/backup_esxi//WindowsServer/WindowsServer-2013-05-27_22-00-02/WindowsServer.vmdk"
Destination disk format: VMFS thin-provisioned
Cloning disk '/vmfs/volumes/datastore1/WindowsServer/WindowsServer.vmdk'...
^MClone: 9% done.^MClone: 10% done.^MClone: 11% done.^MClone: 12% done.^MClone: 13% done.^MClone: 14% done.^MClone: 15% done.^MClone: 16% done.^MClone: 17% done.^MClone: 18% done.^MClone: 19% done.^MClone: 20% done.^MClone: 21% done.^MClone: 22% done.^MClone: 23% done.^MClone: 24% done.^MClone: 25% done.^MClone: 26% done.^MClone: 27% done.^MClone: 28% done.^MClone: 29% done.^MClone: 30% done.^MClone: 31% done.^MClone: 32% done.^MClone: 33% done.^MClone: 34% done.^MClone: 35% done.^MClone: 36% done.^MClone: 37% done.^MClone: 38% done.^MClone: 39% done.^MClone: 40% done.^MClone: 41% done.^MClone: 42% done.^MClone: 43% done.^MClone: 44% done.^MClone: 45% done.^MClone: 46% done.^MClone: 47% done.^MClone: 48% done.^MClone: 49% done.^MClone: 50% done.^MClone: 51% done.^MClone: 52% done.^MClone: 53% done.^MClone: 54% done.^MClone: 55% done.^MClone: 56% done.^MClone: 57% done.^MClone: 58% done.^MClone: 59% done.^MClone: 60% done.^MClone: 61% done.^MClone: 62% done.^MClone: 63% done.^MClone: 64% done.^MClone: 65% done.^MClone: 66% done.^MClone: 67% done.^MClone: 68% done.^MClone: 69% done.^MClone: 70% done.^MClone: 71% done.^MClone: 72% done.^MClone: 73% done.^MClone: 74% done.^MClone: 75% done.^MClone: 76% done.^MClone: 77% done.^MClone: 78% done.^MClone: 79% done.^MClone: 80% done.^MClone: 81% done.^MClone: 82% done.^MClone: 83% done.^MClone: 84% done.^MClone: 85% done.^MClone: 86% done.^MClone: 87% done.^MClone: 88% done.^MClone: 89% done.^MClone: 90% done.^MClone: 91% done.^MClone: 92% done.^MClone: 93% done.^MClone: 94% done.^MClone: 95% done.^MClone: 96% done.^MClone: 97% done.^MClone: 98% done.^MClone: 99% done.^MClone: 100% done.
2013-05-28 03:21:08 -- info: Removing snapshot from WindowsServer ...
2013-05-28 03:21:08 -- debug: Removing /vmfs/volumes/backup_esxi//WindowsServer/WindowsServer-2013-05-24_22-00-02
2013-05-28 03:21:09 -- debug: Removing /vmfs/volumes/backup_esxi//WindowsServer/WindowsServer-2013-05-25_22-00-01
2013-05-28 03:23:00 -- info: Slept 1 seconds to work around NFS I/O error
2013-05-28 03:23:00 -- info: Backup Duration: 247.85 Minutes
2013-05-28 03:23:00 -- info: Successfully completed backup for WindowsServer!
2013-05-28 03:23:03 -- debug: Storage Information after backup:
2013-05-28 03:23:03 -- debug: SRC_DATASTORE: datastore1
2013-05-28 03:23:04 -- debug: SRC_DATASTORE_CAPACITY: 926.5 GB
2013-05-28 03:23:04 -- debug: SRC_DATASTORE_FREE: 161.1 GB
2013-05-28 03:23:04 -- debug: SRC_DATASTORE_BLOCKSIZE: 1
2013-05-28 03:23:04 -- debug: SRC_DATASTORE_MAX_FILE_SIZE: 256 GB
2013-05-28 03:23:04 -- debug:
2013-05-28 03:23:04 -- debug: DST_DATASTORE: backup_esxi
2013-05-28 03:23:04 -- debug: DST_DATASTORE_CAPACITY: 1832.3 GB
2013-05-28 03:23:04 -- debug: DST_DATASTORE_FREE: 839.1 GB
2013-05-28 03:23:05 -- debug: DST_DATASTORE_BLOCKSIZE: NA
2013-05-28 03:23:05 -- debug: DST_DATASTORE_MAX_FILE_SIZE: NA
2013-05-28 03:23:05 -- debug:
2013-05-28 03:23:05 -- info: ###### Final status: All VMs backed up OK! ######
Thanks a lot
Luca