Unable to Delete vCenter Endpoint in vRealize Automation

vRealize Automation Error

More than once in my experience I’ve had a need to delete an endpoint in vRealize Automation. Maybe configuration has changed or you simply made a typo in vCenter hostname or credentials. Once you’ve specified vCenter address and saved the endpoint you can no longer change it (only delete and re-add).

But even when you try to delete it, you will get an error something along the lines of:

You cannot delete this endpoint because 1 compute resources and 0 storage paths use it.

CloudClient Error

There is a KB article that walks you through the process of how to do that using a special tool called CloudClient: Error “This endpoint is being used by # compute resources and # storage paths and cannot be deleted” when you attempt to delete an endpoint in vRA 7.x (2150548)

But even that approach not always work. When you run this command from the KB article “vra computeresource inactive list” you may get the following error:

Error: Something went wrong while processing your request. Please check the application logs for details.

Solution

There is almost no mention of this second error on the Internet and I can see how someone can keep banging his head trying to solve it, so I thought I’d share a solution here. And it’s simple – open a GSS ticket. They can delete the endpoint for you. If you see this error, there’s no other way that I know of to solve this problem without involving GSS.

Clean-up

You can see an error similar to the following in vRA logs if you didn’t stop proxy agents before deleting the endpoint:

Error processing ping response
Error occurred while executing stored proc usp_InsertUpdateHost The INSERT statement conflicted with the FOREIGN KEY constraint “FK_ManagementEndpoint_Host”. The conflict occurred in database “vRa_IaaS”, table “dbo.ManagementEndpoints”, column ‘ManagementEndpointID’.
The statement has been terminated.
Inner Exception: The INSERT statement conflicted with the FOREIGN KEY constraint “FK_ManagementEndpoint_Host”. The conflict occurred in database “vRa_IaaS”, table “dbo.ManagementEndpoints”, column ‘ManagementEndpointID’.
The statement has been terminated.

All you need to do to get rid of it is restart your proxy agents.

Conclusion

Hope this post saves someone the hassle of hours searching for the answer in blogs and forums.

Advertisement

VMware ESXi Core Processes

There are not much information on VMware ESXi architecture out there. There is an old whitepaper “The Architecture of VMware ESXi” which dates back to 2007. In fact, from the management perspective there are only two OS processes critical two ESXi management. These are: hostd (vmware-hostd in ESXi 4) and vpxa (vmware-vpxa in ESXi 4) which are called “Management Agents”.

hostd process is a communication layer between vmkernel and the outside world. It invokes all management operations on VMs, storage, network, etc by directly talking to the OS kernel. If hostd dies you won’t be able to connect to the host neither from vCenter nor VI client. But you will still have an option to connect directly to the host console via SSH.

vpxa is an intermediate layer between vCenter and hostd and is called “vCenter Server Agent”. If you have communication failure between vCenter and ESXi host, it’s the first thing to blame.

Say you have a storage LUN failure and hostd can’t release the datastore. You can restart hostd or both of the processes using these scripts:

# vmware-hostd restart
# vmware-vpxa restart

Installing Symantec Backup Exec Agent for Linux

Symantec Backup Exec Linux/Unix agent is called RALUS which stands for Remote Agent for Linux and Unix Servers. I obtained my RALUS installation from official Symantec CDs. If you don’t have them you probably can download them from Symantec web site. Here is the sequence:

1. Mount CD or iso image to your Linux host.
2. Run ./installralus script and follow instructions. I use defaults. The only thing you should enter is Media Server IP address. Installation script add itself to rc*.d levels automatically.
3. After installations is completed create backup user, add it to beoper group and set its password: # useradd backup -c “User for Symantec Backup Exec”;  # usermod -G beoper backup; # passwd backup.
4. Start BE agent manually for the first time: # /etc/init.d/VRTSralus.init start

That’s it. Now you can see your server under Linux/Unix Servers section when creating backup job.

Add #1: If agent doesn’t start and you get an error with libstdc++.so.5 missing in /var/VRTSralus/beremote.service.log then install compat-libstdc++-33.

Add #2: If you have active firewall then you need to open additional ports. For me it was tcp 10000-10200. It’s 10000 plus port range you can find on media server in Tools->Options->Network and Security tab. For CentOS firewall rule would be:

-A RH-Firewall-1-INPUT -m tcp -p tcp -s media_server_ip –dport 10000:10200 -j ACCEPT

Add #3: In case you also write firewall rules to OUTPUT chain then open output tcp 10000:

-A RH-Firewall-1-OUTPUT -m tcp -p tcp -d media_server_ip –dport 10000 -j ACCEPT

If you don’t have RH-Firewall-1-OUTPUT add also:

:RH-Firewall-1-OUTPUT – [0:0]
-A OUTPUT -j RH-Firewall-1-OUTPUT

I leave possibility of me being wrong, but SBE documentation says:

Symantec recommends having port 10000 open and available on the Backup Exec media
server as well as on the remote systems.

Additional connections from the media server to the Remote Agent will be initiated on any available port.

I understand that as both agent and media server may connect to each other’s 10000 port and additional 10001:10200 connections are initiated from medias server.