Joining ESXi to AD in Disjoint Namespace

What is Disjoint Namespace?

Typically, when using Microsoft Active Directory you use AD-integrated DNS and your AD domain name matches you DNS domain name, but you don’t have to. This is quite rare, but I’ve seen cases where the two don’t match. For example, you might have a Linux-based DNS, where you register an esx01.example.com DNS record for your ESXi host and then you join it to an Active Directory domain called corp.local.

That’s called a disjoint namespace. You can read this Microsoft article if you want to know more details: Disjoint Namespace.

In my personal opinion, using a disjoint namespace is asking for trouble, but it will still work if you really want to use it.

Problem

If you end up going down that route, there’s one caveat you should be aware of. When you joining a machine to AD, among other things, it needs to populate DNS name field property of the AD computer object. This is an example of ESXi computer object in Active Directory Users and Computers snap-in:

If you configure example.com domain in your ESXi Default TCP/IP stack, like so:

And then you try to, for example, join your ESXi host to corp.local AD domain, it will attempt to use esx-01a.example.com for computer object DNS name field. If you’re using a domain account with privileges restricted only to domain join, this operation will fail.

This is how the problem manifested itself in my case in ESXi host logs:

Failed to run provider specific request (request code = 8, provider = ‘lsa-activedirectory-provider’) -> error = 40315, symbol = LW_ERROR_LDAP_CONSTRAINT_VIOLATION, client pid = 2099303

If you’re using host profiles to join ESXi host to the domain, remediation will fail and you will see the following in /var/log/syslog.log:

WARNING: Domain join failed; retry count 1.

WARNING: Domain join failed; retry count 2.

Likewise (ActiveDirectory) Domain Join operation failed while joining new domain via username and password..

Note: this problem is specific to joining domain using a restricted service account. If you use domain administrator account, it will force the controller to add the computer object with a DNS name, which doesn’t match the AD name.

Solution

Make sure ESXi domain name setting matches the Active Directory domain name, not DNS domain name. You can still use the esx-01a.example.com record to add the ESXi hosts to vCenter, but you have to specify corp.local domain in DNS settings (or leave it blank), because this is what is going to be used to add the host to AD, like so:

This way your domain controller will be happy and ESXi host will successfully join the domain.

Additional Notes

While troubleshooting this issue I saw a few errors in ESXi host logs, which were a distraction, ignore them, as they don’t constitute an error.

This just means that the ESXi host Active Directory service is running, but host is not joined to a domain yet:

lsass: Failed to run provider specific request (request code = 12, provider = ‘lsa-activedirectory-provider’) -> error = 2692, symbol = NERR_SetupNotJoined, client pid = 2111366

IPC is inter-process communication. Likewise consists of multiple services that talk to each other. They open and close connections, this is normal:

lsass-ipc: (assoc:0x8ed7e40) Dropping: Connection closed by peer

I also found this command to be useful for deeper packet inspection between an ESXi host and AD domain controllers:

tcpdump-uw -i vmk0 not port 22 and not arp

References

• Joining vCenter Server Appliance or ESXi host into Active Directory domain fails with error: LW_ERROR_LDAP_CONSTRAINT_VIOLATION or LW_ERROR_LDAP_INSUFFICIENT_ACCESS (52929)
• Enabling logging for Likewise agents on ESXi/ESX (1026554)
• ESXI AND LIKEWISE – TROUBLESHOOTING GUIDE – PART 1
• ESXI AND LIKEWISE – TROUBLESHOOTING GUIDE – PART 2

Advertisement

Troubleshooting Cisco UCS LDAP

If you ever configured LDAP integration on a blade chassis or a storage array, you know that troubleshooting authentication is painful on these things. It will accept all your configuration settings and if you’ve made a mistake somewhere all you get when you try to log in is “Authentication Error” message with no clue of what the actual error is.

Committing configuration changes

There three common places where you can make a mistake when setting up LDAP authentication on UCS. Number one is committing configuration changes to the Fabric Interconnects in UCS Manager.

There are four configuration options which you need to set to enable Active Directory authentication to your domain:

• LDAP Providers – these are your domain controllers
• LDAP Provider Groups – are used to group multiple domain controllers of the same domain
• LDAP Group Maps – where you give permissions to your AD groups and users
• Authentication Domains – final configuration step where you enable authentication via the domain

Now if you decide to delete a LDAP Provider Group which is configured under an Authentication Domain in attempt to change the settings, this may become an issue.

What is confusing here is UCS Manager will let you delete the LDAP Provider Group, save the changes and LDAP Provider Group will disappear from the list. And you may legitimately conclude that it’s deleted from UCS, but it’s actually not. This is what you’ll see in UCS Manager logs:

[FSM:STAGE:STALE-FAIL]: external aaa server configuration to primary(FSM-STAGE:sam:dme:AaaEpUpdateEp:SetEpLocal)
[FSM:STAGE:REMOTE-ERROR]: Result: resource-unavailable Code: ERR-ep-set-error Message: Re-ordering/Deletion of Providers cannot be applied while ldap is used for yourdomain.com(Domain) authentication(sam:dme:AaaEpUpdateEp:SetEpLocal)

The record will stay on the UCS and you may encounter very confusing issues where you change your LDAP Provider settings but changes are not reflected on UCS. So make sure to delete the object from the higher level entity first.

Distinguished Name typos

There are two ways to group Active Directory entities on a domain controller – Security Groups and Organizational Units. When configuring your AD bind account in LDAP Providers section and setting up permissions in LDAP Group Maps, make sure to not confuse the two. The best advice I can give – always use ADSI Edit tool to find the exact DN. Why? As an example let’s say you want to give permissions to the builtin administrator group and you use the following DN:

CN=Administrators,OU=Builtin,DC=yourdomain,DC=com

This won’t work, because even though Builtin container may look like a OU, it’s actually a CN in AD, as well as Users and Computers containers.

ADSI Edit will give you the exact Distinguished Name. Make sure to use it to save yourself the hassle.

Group Authorization settings

Last but not least are the following two LDAP Provider configuration settings:

• Group Authorization – whether UCS searches within groups when authenticating
• Group Recursion – whether UCS searches groups recursively

If you add an AD group which the user is a part of in LDAP Group Maps and do not enable Group Authorization, UCS simply won’t search within the group. Enable this option unless you give permissions only on a per user basis.

Second option enables recursive search within AD groups. If you have nested groups in AD (which most people have) enable recursive search or UCS won’t look deeper than 1 level.

If you get really stuck

If you’ve set all the settings up and are certain they the are correct, but authentication still doesn’t work, then there is a relatively easy way to localize the issue.

First step is to check whether UCS can bind to your LDAP Providers and authenticate users. Pick a user (LDAP Group Maps don’t matter at this point), SSH to a Fabric Interconnect and type the following:

ucs # connect nxos
ucs(nxos)# test aaa server ldap yourdc.yourdomain.com john password123

yourdc.yourdomain.com – is the domain controller you’ve configured in LDAP Providers section. If authentication doesn’t work, then the issue is in LDAP Provider settings.

If you can authenticate, then the next step is to make sure that UCS searches through the right AD groups. To check that you will need to enable LDAP authentication logging on a Fabric Interconnect:

ucs # connect nxos
ucs(nxos)# debug ldap aaa-request-lowlevel

Now try to authenticate and look through the list of groups which UCS is searching through. If you can’t see the group which your user is a part of, then you most likely using a wrong DN in LDAP Group Maps.

In my case the settings are configured correctly and I can see that UCS is searching in the Builtin Administrators group:

2015 Dec 1 14:12:19.581737 ldap: value: CN=Enterprise Admins,CN=Users,DC=yourdomain,DC=com
2015 Dec 1 14:12:19.581747 ldap: ldap_add_to_groups: Discarding. group map not configured for CN=Enterprise Admins,CN=Users,DC=yourdomain,DC=com
2015 Dec 1 14:12:19.581756 ldap: value: CN=Administrators,CN=Builtin,DC=yourdomain,DC=com
2015 Dec 1 14:12:19.581767 ldap: ldap_add_to_groups: successfully added group:CN=Administrators,CN=Builtin,DC=yourdomain,DC=com
2015 Dec 1 14:12:19.581777 ldap: value: CN=Exchange Organization Administrators,OU=Microsoft Exchange Security Groups,DC=yourdomain,DC=com

Make sure to disable logging when you’re done:

ucs(nxos)# undebug all

References:

• Sample Configuration: Cisco UCS, LDAP and Active Directory
• UCSM LDAP Troubleshooting guide

VNX LDAP Integration: AD Nested Groups

Have you ever stumbled upon AD authentication issues on VNX, even though it all looked configured properly? LDAP integration has always been a PITA on storage arrays and blade chassis as usually there is no way to troubleshoot what the actual error is.

If VNX cannot lookup the user or group that you’re trying to authenticate against in AD, you’ll see just this. Now go figure why it’s getting upset about it. Even though you can clearly see the group configured in “Role Mapping” and there doesn’t seem to be any typos.

Common problem is Nested Groups. By default VNX only checks if your account is under the specified AD group and doesn’t traverse the hierarchy. So for example, if your account is under the group called IT_Admins in AD, IT_Admins is added to Domain Admins and Domain Admins is in “Role Mapping” – it’s not gonna work.

To make it work change “Nested Group Level” to something appropriate for you and this’d resolve the issue and make your life happier.

How to reset Active Directory DRSM password

Login to the domain controller, run ntdsutil from the command line. Then enter:

set dsrm password
reset password on server null

After that you will be asked for the new password.

Highly available Windows network infrastructure

When number of computers in company starts to grow, IT services become critical for company operation, every IT department starts to think how to make their network infrastructure highly available. If it’s a Windows environment, then the first step is usually an additional domain controller. Bringing second DC up and running is rather simple. The only thing you need to do is to run dcpromo and follow the instructions given by the wizard. Then make additional DC a Global Catalog, so that it will serve authentication requests, by going to Active Directory Sites and Services and in NTDS settings on General tab check Global Catalog option. Windows File Replication Services (FRS) will do the rest.

However, it’s usually not enough. Computers rely on DNS service to resolve servers names and in case of primary DC failure your network will be paralyzed. Dcpromo don’t automatically install and configure additional DNS server. You need to do that manually. Moreover, if you use DHCP service to provide network settings to client computers and it’s located on the same server you will also have major issues. The problem here is that you can’t have two active DHCP servers giving out same addresses. But this problem also have its solution.

In case of DNS you should go to Add or Remove Windows Components and find DNS in Networking Services. Install it as AD integrated. Then on the primary DNS, for all your forward and reverse lookup zones, in properties add secondary DNS IP on Name Servers tab. After that DNS will automatically replicate all data. Don’t also forget to add your secondary DNS to DHCP configuration, otherwise clients won’t know about it.

When it comes to DHCP you have an option to use so called 80/20 rule to divide scope between DHCP servers (if you work on Windows server 2008 platform you can build HA DHCP cluster). Simply configure your first DHCP server to lease first 80% of network IP addresses and leave 20% to the second DHCP server. Then in case of first server failure most of computers will already have their IP addresses and you will still have 20% to distribute. In my case network is quite small and I split scope in 50/50. Just make equal configurations for two servers (reservations, exclusions, scope options, etc), but configure scope to have non-overlapping ranges. Then if you use 80/20 rule, you want your primary server to lease IP address in normal circumstances. If both servers will lease addresses with equal rights then you will quickly run out of addresses on 20% server and in case of primary server failure you won’t have enough addresses to lease. To solve that, tweak Conflict detection attempts option.

Basically, this is it. Of course, you will still have many points of failure, like network switch, UPS, etc. But this topic goes beyond this post.

Moving Active Directory roaming profiles to another server

Relocating Active Directory roaming profiles can be a tricky task. You have many shared folders with particular permissions which won’t move unlike NTFS permissons to another server if you simply copy these folders. On top of that, you need to change profile paths in Active Directory Users and Computers snap-in. And if you have hundreds of users it’s not what you will happy to do. Given these two objectives lets move on to implementation.

Moving shares preserving permissions

I ran into several suggestions how to do that, like using robocopy, xcopy, permcopy or other tools. I don’t know to what extent they might help. I’d like to suggest simpler solution. Microsoft has  File Server Migration Toolkit (FSMT). It’s very basic and limited tool. It means you will probably need to do some hand work. But it solves the primary problem which is copying shares along with their permissions.

FSMT has additional feature of creating DFS links for you but I didn’t use it. GUI is rather intuitive, so there is not much to explain. The particular problem with FSMT is that it changes target share and folder paths. Say you have share with the name ~UNAME$ which is located on server CONTOSO_PDC. As a result of movement you will have share with the name ~UNAME$_contoso_pdc$. Which is not what we expect to have in our case. Same thing for target folder. For example, if the source folder for the share is D:\Profiles\UNAME, after migration you’ll get D:\Profiles\contoso_pdc\~UNAME$. Apart from additional folder in between, as you can see last part of source path is changed to share name in the target path (~UNAME$ instead of UNAME).

In my case I had to revert all these changes back to what it originally was. The trick here is to create FSMT project, add server and shares to it and then exit without performing the actual move. Then open project .xml file and correct all paths by search/replace. Since I had complicated share names I had to use replace with substitution feature in MS Office Word. For example to change target path from D:\Shared\~PROF\~UNAME$ to D:\Shared\~PROF\UNAME I used following masks for search and replace:

D:\\Shared\\~USER\\\~(*)\$

D:/Shared/~USER/\1

Here word processor searches for the first string and use word from parenthesis as the substitution for the special sequence \1.

Changing profile paths in Active Directory

Here you also have several ways to accomplish that. You can use ADModify tool. But I simply wrote a Powershell script which I share with you as it is. I believe it’s mostly self-explanatory. For convenience I also uploaded this script to FileDen. Download it from here.

# Bind to the root of the current domain
$ldapPath = "LDAP://ou=Users and Computers,dc=contoso,dc=com"
$objDomain = New-Object System.DirectoryServices.DirectoryEntry($ldapPath)

$objSearcher = New-Object System.DirectoryServices.DirectorySearcher
$objSearcher.SearchRoot = $objDomain
$objsearcher.Filter = ("(objectCategory=User)")
$colResult = $objSearcher.FindAll()

foreach($objResult in $colResult) {
	$user = $objResult.GetDirectoryEntry()
	write-host "For user" $user.cn ":"

	$profilePath = $user.ProfilePath
	$parts = $profilePath.ToString().Split("\")

	# Identifying profile type (XP = 0 or Windows 7 = 1)
	if($parts.Length -eq 4) { $profType = 0 }
	elseif($parts.Length -eq 5) { $profType = 1 }

	# Constructing new profile paths
	if($profilePath) {
		if($profType -eq 0) {
			$newProfPath = "\\SERVERNAME\" + $parts[3];
			$newProfDirPath = "D:\~PROF\" + $user.sAMAccountName
			$newProfShareName = $parts[3]
		}
		# Windows 7 profiles do not have individual shares. There is
		# one share for all roaming profiles.
		elseif($profType -eq 1) {
			$newProfPath = "\\SERVERNAME\Profiles\" + $parts[4]
		}
	}

	# Constructing new home directory paths
	$homeDirectory = $user.homeDirectory
	$parts = $homeDirectory.ToString().Split("\")
	if($homeDirectory) {
		$newHomePath = "\\SERVERNAME\" + $parts[3];
		$newHomeDirPath = "D:\~USER\" + $user.sAMAccountName
		$newHomeShareName = $parts[3]

	}

	if($profilePath) {
		# Changing profile path
		write-host "Changing profile path from" `
			$user.ProfilePath "to" $newProfPath
		$user.ProfilePath = $newProfPath
	}
	if($homeDirectory) {
		# Changing home directory path
		write-host "Changing home directory path from" `
			$user.homeDirectory "to" $newHomePath
		$user.homeDirectory = $newHomePath
	}
	# Commit changes
	$user.setinfo()
}

DPM not sending alerts

I had funny experience today configuring notifications in Microsoft Data Protection Manager 2010. We have Postfix mail server instead of Exchange. Also we use SMTP authentication for outgoing mail from our users. DPM notifications configuration page has Username and Password fields for this. But simply using username and password of account registered in Postfix doesn’t work. It just says:

ID 2013

Details: Logon failure: unknown username or password

DPM needs this user to be registered in Active Directory. I have no idea why. So the simple solution to this was adding this account to AD with same password as in Postfix users database.

One more thing I’ve discovered is that for some reason Windows Server 2008 sends computer name without domain part in HELO command. Even though domain part is configured at System Properties. It wasn’t allowed in our mail server configuration. Solution to this was shifting of reject_non_fqdn_hostname option to the end of smtpd_recipient_restrictions (but before permit option) rule and addition of permit_mynetworks before it.

AD snap-in in Windows XP

Active Directory snap-in is not accessible by default in Windows XP MMC. Firstly you need to install Windows Server 2003 Administration Tools Pack which is installable on Windows XP SP1. Here is the link.

Time Synchronization with AD

I had weird issue with Active Directory service where there were no Windows Time service at all. w32time.dll and w32tm.exe files were in place. So I just registered w32time.dll by:

w32tm /register

After that Windows Time service appeared under ‘services’. Start service by:

net start w32time

Then to check that time service work on DC run from any client:

w32tm /stripchart /computer:dc_ip /samples:5 /dataonly

To synchronize with DC right away on client run:

w32tm /resync

In case client synchronizes with outside NTP instead of DC edit HKEY_LOCAL_MACHINE\ SYSTEM\ CurrentControlSet\ Services\ W32Time\ Parameters. If it has ‘NTP’ in ‘Type’ parameter then change it to ‘NT5DS’ and run:

w32tm /config /update

 

Security on NetApp Filer

Storage systems usually store data critical for organization like databases, mailboxes, employee files, etc. Typically you don’t provide access to NAS from Internet. If Filer has real IP address to provide CIFS or NFS access inside organization you can just close all incoming connections from outside world on frontier firewall. But what if networking engineer mess up firewall configuration? If you don’t take even simple security measures then all your organization data is at risk.

Here I’d like to describe basic means to secure NetApp Filer:

• Disable rsh:

options rsh.enable off

• Disable telnet:

options telnet.enable off

• Restrict SSH access to particular IP addresses. Take into consideration that if you enabled AD authentication Administrator user and Administrators group will implicitly have access to ssh.

options ssh.access host=ip_address_1,ip_address_2

• You can configure Filer to allow files access via HTTP protocol. If you don’t have HTTP license or you don’t use HTTP then disable it:

options http.enable off

• Even if you don’t have HTTP license you can access NetApp FilerView web interface to manage Filer. You can access it via SSL or plain connection, apparently SSL is more secure:

options http.admin.enable off

options http.admin.ssl.enable on

• Restrict access to FilerView:

options httpd.admin.access host=ip_address_1,ip_address_2

• If you don’t use SNMP then disable it:

options snmp.enable off

• I’m using NDMP to backup Filer’s data. It’s done through virtual network. I restrict NDMP to work only between Filers (we have two of them) and backup server and only through particular virtual interface:

On Filer1:

options ndmpd.access “host=backup_server_ip,filer2_ip_address AND if=interface_name”

options ndmpd.preferred_interface interface_name

On Filer2:

options ndmpd.access “host=backup_server_ip,filer1_ip_address AND if=interface_name”

options ndmpd.preferred_interface interface_name

• Disable other services you don’t use:

options snapmirror.enable off

options snapvault.enable off

• Module which is responsible for SSH and FilerView SSL connections is called SecureAdmin. You probably won’t need to configure it since it’s enabled by default. You can verify if ssh2 and ssl connections are enabled by:

secureadmin status

• Make sure all built-in users have strong passwords. You can list built-in users by:

 useradmin user list

• By default Filer has home directory CIFS shares for all users. If you don’t use them, disable them by deleting:

/etc/cifs_homedir.cfg

• Filer also has ETC$ and C$ default shares. I’d highly recommend to restrict access to these shares only to local Filer Administrator user. In fact, if you enabled AD authentication then also domain Administrator user and Administrators group will implicitly have access to these shares, even if you don’t  specify them in ACL. Delete all existing permissions and add:

cifs access share etc$ filer_system_name\Administrator Full Control
cifs access share c$ filer_system_name\Administrator Full Control

Basically this is it. Now you can say that you know hot to configure simple NetApp security.