Disk to Disk to Tape backup in Backup Exec

Notice: It seems that D2D2T feature in Backup Exec 11d is buggy. D2D2T duplicate jobs (which transfer data from disk to tape) are insanely slow and nobody has yet solved this problem. You can try to implement backup of raw Backup to Disk Folder, but it is associated with number of  difficulties when restoring. Files from Backup to Disk Folders are media and they conflict with media which is currently used for backup.

Typical backup solution in most organizations consists of backup server and tape drive/autoloader/tape library connected directly to backup server. Every night backups are pushed to tape through backup server. But sometimes it is more complicated. We have NetApp filer with StorageTek tape library connected to the filer. Backup server sends NDMP commands to the filer and filer in its turn performs actual data transfer to tapes from disk shelves. Most of our hosts are VMware virtual machines. We backup whole .vmdk files, but we also want to perform file-level backups from some of virtual machines. To accomplish that we set up backup agents on all virtual machines, but we can’t backup files directly to tapes, because they do not originate from NetApp filer volumes. So we decided to implement D2D2T multistage backup. The idea here is to create a CIFS share on the filer, backup data there and then transfer data from CIFS share to tapes.

First step here is to configure disk to disk backup. Backup Exec stores disk to disk backups in binary files. Folder where files are stored is listed on Devices tab and files are listed on Media tab. Initially, you need to create a Backup to Disk Folder in Devices tab. There you choose size for backup-to-disk files and maximum number of backups per backup-to-disk file. If backup is larger than file size, it is splitted in several files. If file size is smaller than backup, several backups will be written to one file. I use defaults with 16GB file size. Then you create backup jobs as usual (by configuring selection list and policy) using Backup to Disk Folder as target device.

As a second step you need to instruct Backup Exec to transfer backed up files to tape, upon disk to disk job completion. Backup Exec has “duplicate jobs” to implement that. Go to your backup policy properties, click “New Template”, choose “Duplicate Backup Sets Template”, pick template for which you want to create duplicate, in “Devices and Media” choose your tape library, in “Schedule” choose “Run only according to rules for this template”. This will create duplicate template and rule which will start duplicate job after main job completes. As a result you will have duplicate data on disk and on tape.

Advertisement

Disconnect stalled NDMP sessions

Once, I started installation of Symantec Backup Exec service pack update when tape library inventory job was running. After installation has been completed I ended up with library offline and not available. It happened because of hanged NDMP sessions. To list your media changer and tape drives information run:

storage show mc
storage show tape

or

sysconfig -m
sysconfig -t

To list and kill particular NDMP sessions run:

ndmpd status
ndmpd kill job_id

Then restart Backup Exec service.

GFS backup scheme in Symantec Backup Exec

Grandfather-Father-Son is an industry standard backup scheme, where you have 5 daily backups, 5 weekly backups and as many monthly as you need. Symantec Backup Exec has prebuilt policy for GFS, but before going into configuring backup scheme itself, lets talk a little bit about general backup job configuration in Backup Exec.

Basic Terminology

Inside user interface you see Jobs, Policies, Selection Lists and Media Sets. First of all you need to create Selection List, which describes what you want to backup. There you select files and folders from your Windows, Unix or NDMP servers. Then you create Media Set, which is a collection of tapes with particular append and retention periods. Append period specifies how long data can be added to the same tape and retention period tells for how long data cannot be overwritten. Retention period starts form the time of last append to the tape. Then you create Policy. Policy, by means of templates, defines when backup jobs are run, where backups are stored and what is the type of backup – incremental, differential or full. One policy can consist of several templates. In template you specify backup date and time, as well as target tape library.

GFS Implementation

Backup Exec has a template for GFS backup rotation scheme. Click “New policy using wizard”, choose GFS scheme and then select schedule, target backup device and media sets for daily, weekly and monthly backups. By default Backup Exec suggests the following configuration.

Three tape media sets:

• Daily Media Set – 1 week overwrite, 1 week append
• Weekly Media Set – 5 weeks overwrite, 5 weeks append
• Monthly Media Set – 1 year overwrite, 1 year append

Policy with three templates:

• Daily Backup – Monday to Friday, Incremental
• Weekly Backup – every Friday, Full
• Monthly Backup – first Saturday of each month, Full

Also Backup Exec automatically creates rules to resolve conflicts. For example when both Daily and Weekly backups try to run on Friday, jobs do not conflict, because weekly backups always supersede daily. Same for monthly.

I personally prefer another schedule. First of all, if you run your jobs after midnight, you will need to shift your schedules from Mon – Fri to Tue – Sat. Additionally, I run monthly backup on the first Saturday of the month. Backup Exec by default (taking into consideration my one day shift) would suggest first Sunday for the monthly backup. However, it doesn’t make much sense to have weekly on Saturday and then monthly next day on Sunday. You would just consume more space without any benefit. Also, you can schedule monthly on the last Saturday of the month, but if the last day is Thursday, for example, then you will loose four business days from your monthly backup.

After the policy is created, you need to create backup jobs using this policy by clicking on New jobs using policy. All three jobs will be created automatically according to Selection List, as well as Policy Schedule, Target, and Backup Type parameters.

I’d also recommend everyone to configure notifications. There are general Alerts properties as well as inside each job.

NetApp NDMP with Symantec BackupExec

Some time ago I uploaded a bunch of photos from the data center, where you can find our backup setup. We connect Sun StorageTek SL500 tape library directly to NetApp filer to perform backups of the virtual infrastructure using NDMP protocol. As opposed to LAN backup, NDMP allows you to offload LAN from backup traffic. Look at the following picture:

Here BackupExec only sends NDMP control commands to NDMP host, which in its turn send data to directly attached tape library. We use slightly more complicated 3-way backup architecture:

We have two filers in high availability cluster. And each of the filers has its own hard drive shelves and data. Filer under number 3 on the picture is the primary source of backup data and data from filer 2 is backed up occasionally. Since filer 2 has no connection to the library, when backup is initiated it is send via LAN from filer 2 to filer 3 and then to the tape library.

NetApp configuration

NDMP configuration involves several steps. First of all enable ndmpd on NetApp and set version 4, which Symantec BackupExec works with:

ndmpd on
ndmpd version 4

Then it’s a generally good idea to restrict NDMP access only to particular hosts and interface, because by default access is allowed from anywhere. In our setup NDMP traffic goes through completely isolated management network. We added two IP addresses to allowed hosts. First is the backup server and second is the partner filer:

options ndmpd.access hosts=ip_1,ip_2
options ndmpd.access if=manage_if

Then I’d recommend to create separate user for NMDP backups, change its group to Backup Operators and create special ndmp password which you will use to connect from BackupExec:

useradmin useradd backup
useradmin user modify backup -g “Backup Operators”
ndmpd password backup

As a last recommendation I suggest changing preferred network interface for data connections. By default for data traffic filer uses the same network interface from which it receives control commands. But if you have separate network for filer to filer communications its preferable to use it. In our configuration it’s the same management interface so for us it doesn’t make any difference:

options ndmpd.preferred_interface manage_if

Additionally you can use the following command to list your tape library robots:

storage show mc

Do the same configuration for all filers, if you have more than one.

BackupExec configuration

For NDMP to work in BackupExec you should obtain a licence key and install NDMP Option module. Then go to Devices section, click Add NDMP Server. In Add NDMP Server dialog box specify server name and logon account. If you have more than one filer, do it for each one.

That’s it. Now you have filer volumes in backup selection lists, tapes in Media section and you are ready to do backups.

Sun StorageTek SL500

I made several pictures of tape library which serves as primary backup facility in our data centre. Here what you can achieve with this library in maximum configuration:

• 18 drives with more than 9TB/hour throughput.
• Up to 575 cartridges and 862TB of uncompressed storage.

Our configuration is rather small, 2 drives and 45TB of storage.

Click pictures to enlarge.

Here you can see how robo-arm performs library inventorization by reading barcodes with infrared scanner:

Both tape drives and robot are connected to NetApp filer with SCSI cables. All data is backed up from disk shelves directly to tape library via NDMP protocol. There is no need to feed data through backup server which eliminates any LAN congestion.

Here are connections to NetApp:

Security on NetApp Filer

Storage systems usually store data critical for organization like databases, mailboxes, employee files, etc. Typically you don’t provide access to NAS from Internet. If Filer has real IP address to provide CIFS or NFS access inside organization you can just close all incoming connections from outside world on frontier firewall. But what if networking engineer mess up firewall configuration? If you don’t take even simple security measures then all your organization data is at risk.

Here I’d like to describe basic means to secure NetApp Filer:

• Disable rsh:

options rsh.enable off

• Disable telnet:

options telnet.enable off

• Restrict SSH access to particular IP addresses. Take into consideration that if you enabled AD authentication Administrator user and Administrators group will implicitly have access to ssh.

options ssh.access host=ip_address_1,ip_address_2

• You can configure Filer to allow files access via HTTP protocol. If you don’t have HTTP license or you don’t use HTTP then disable it:

options http.enable off

• Even if you don’t have HTTP license you can access NetApp FilerView web interface to manage Filer. You can access it via SSL or plain connection, apparently SSL is more secure:

options http.admin.enable off

options http.admin.ssl.enable on

• Restrict access to FilerView:

options httpd.admin.access host=ip_address_1,ip_address_2

• If you don’t use SNMP then disable it:

options snmp.enable off

• I’m using NDMP to backup Filer’s data. It’s done through virtual network. I restrict NDMP to work only between Filers (we have two of them) and backup server and only through particular virtual interface:

On Filer1:

options ndmpd.access “host=backup_server_ip,filer2_ip_address AND if=interface_name”

options ndmpd.preferred_interface interface_name

On Filer2:

options ndmpd.access “host=backup_server_ip,filer1_ip_address AND if=interface_name”

options ndmpd.preferred_interface interface_name

• Disable other services you don’t use:

options snapmirror.enable off

options snapvault.enable off

• Module which is responsible for SSH and FilerView SSL connections is called SecureAdmin. You probably won’t need to configure it since it’s enabled by default. You can verify if ssh2 and ssl connections are enabled by:

secureadmin status

• Make sure all built-in users have strong passwords. You can list built-in users by:

 useradmin user list

• By default Filer has home directory CIFS shares for all users. If you don’t use them, disable them by deleting:

/etc/cifs_homedir.cfg

• Filer also has ETC$ and C$ default shares. I’d highly recommend to restrict access to these shares only to local Filer Administrator user. In fact, if you enabled AD authentication then also domain Administrator user and Administrators group will implicitly have access to these shares, even if you don’t  specify them in ACL. Delete all existing permissions and add:

cifs access share etc$ filer_system_name\Administrator Full Control
cifs access share c$ filer_system_name\Administrator Full Control

Basically this is it. Now you can say that you know hot to configure simple NetApp security.