How to backup Cisco ISE correctly!
In this post I will explain how to correctly backup ISE. There are a few instances I have seen, where ISE is not being backed up using the supported (GUI (WEB APPLICATION) ) method as dictated by Cisco. This is especially true in vSphere environments, where VCB backups of the ISE nodes are used as the sole backup. Whenever one of your ISE nodes fails and your backup solution is to use VCB backups (snapshots or alike) and choose to restore from one, it may actually create LOTS of issues, not to mention at the moment the snapshot is processing, ISE may become unresponsive and drop RADIUS requests.
Ready for a brief story ????
Whatever backup solution you use to backup your virtual infrastructure with, you may sometimes end up with VM snapshots that need to be cleaned up. After a backup failure alert, I use the following PowerCLI one-liners to quickly identify and remove snapshots left behind (by say Netapp SMVI). Get-VM | Get-Snapshot | Where-Object {$_.Name -like 'smvi*'} | ft VM,Name,Created -AutoSize Get-VM | Get-Snapshot | Where-Object {$_.Name -like 'smvi*'} | Remove-Snapshot -RunAsync -Confirm:$false
Recently I had an instance where post a backup failure the snapshot failed to remove with the error Unable to communicate with the remote host, since it is disconnected.
From what I could ascertain, following the backup failure the VM had been knocked offline and marked as (Invalid) in vCenter registration.
So step 1 was to remove the VM from vCenter and re-add back to the inventory via the right-clicking on the VMX file. Once back in vCenter I was able to remove the snapshot. However, the VM was then marked as Virtual machine disks consolidation is needed.
Taking the Consolidate option from the Snapshot menu however failed with the error Unable to access file since it is locked.
There are loads of posts out there on how this can happen if you are using a backup solution that mounts vmdks into the backup appliance and that removing the disk from the backup appliance and retrying should resolve the issue.
I’m not sure if Netapp SMVI even works that way, but I didn’t have any additional disks on the VM used for that purpose. I powered off the SMVI VM anyway, tried the consolidate again, but still no luck.
The VM was now registered on a different host than it had been during the backup failure, but the file lock was still present. I decided to place the original host in maintenance mode in case that was still maintaining a lock on the file. The host failed to get into maintenance mode and hung at 83% for ages after migrating all VMs off. A restart of the ESXi management agents resolved this and I was then able to place the host and maintenance mode and restarted it for good measure.
Lesson I have learned from this story:
IF YOU ENABLE SNAPSHOTS OF YOUR CISCO ISE VM , YOU WILL HAVE A HORRIBLE TIME ! Therefore to setup the BACKUP correctly and to make it part of your automated backup schedule, please follow the steps below:
First things first !
Create a Repository Log into the ISE web UI and navigate to Administration > System > Maintenance > Repository and click Add. You will need to select the Protocol to use, I generally recommend using SFTP so that your backup is secure over the wire and it’s off box. The figure below shows an example of my DEMO SFTP Backup Server.
After you hit submit, you are presented with the following warning: Add SSH Host Key *** This only applies for Repositories created using SFTP, skip for FTP etc. Once you have created the SFTP repository, you need to add the SSH key into your Primary Administration Node and if Applicable your Secondary Administration Node.
To add the host key, the steps are:
ISE/admin# crypto host_key add host https://www.linkedin.com/redir/invalid-link-page?url=10%2e21%2e211%2e22
The output you should receive on each node (PAN/SAN) is as follows:
host key fingerprint added
# Host https://www.linkedin.com/redir/invalid-link-page?url=10%2e21%2e211%2e22 found: line 1 type RSA 2048 e6:44:9c:c3:50:99:ab:40:4c:35:39:7f:4a:7D:8f:1d https://www.linkedin.com/redir/invalid-link-page?url=10%2e21%2e211%2e22 (RSA)
Create Backup Schedules/ Start On-Demand Backup Now, back in the ISE UI (Web Page) navigate to Administration > System > Backup & Restore, you will need to create a schedule for:
• Configuration: Contains both application-specific and Cisco ADE operating system configuration data
• Operational: Contains monitoring and troubleshooting data Select create under the appropriate heading and fill in the details of your scheduled backup, you will need to enter in the Encryption Key, which is extremely important that you record this in a safe place as it will be used for restoration of your ISE environment in the event of a disaster.
When creating schedules, I recommend doing a configuration backup once a week in a fairly static environment and operational backups once a day. Let’s start a On-Demand Backup by selecting the Backup Now button, as mentioned before you will need to enter and record the encryption key used.
Backup your certificates: ISE 1.3 and above!
If your are using ISE 1.3 and above, you must backup your certificates and keys manually and in a secure manner so you can restore them back onto your Secondary Administration Node (SAN) in the event the Primary Administration Node (PAN) fails and you promote the SAN to a PAN.
The main point to remember is, this backup must be done once your certificates are all in place for your ISE nodes and is not included as part of your configuration backup, therefore it’s imperative that you do it. The certificate backup contains the following certificates for your reference:
1. ISE Root CA Cert
2. ISE Sub CA Cert
3. ISE Endpoint RA Cert
4. ISE OSCP Responder Cert
The backup needs to be performed from the Primary Administration Node (PAN). The following sub-section will detail how to create a repository on the CLI and perform a backup. Creating a repository from global configuration mode:
repository REPO_NAME url sftp://IP_ADDRESS user USER password plain PASS crypto host_key add IP_ADDRESS
Once you have created the backup repository perform the backup by issuing the following commands:
application configure ise
select option 7
[7] Export Internal CA Store
Export Repository Name: Mike_SFTP_BACKUP
Enter encryption-key for export: PLAIN_PASS
After you have backed up the keys up, make sure you keep them in a safe place (NOT in the CLOUD !!!!!!!!!!!!!!)
Writer : Mike Ghahremani
Editor : Jeremy smith
All rights reserved ©
Picture source : cisco.com
TRADEMARK LEGAL NOTICE All product names, logos, and brands are property of their respective owners in the Austria or other countries.All company, product and service names used on this website are for identification purposes only. Pheniix is notaffiliated with or an official partner of Cisco, CompTIA,Dimension Data, VMware, Amazon, Microsoft, Certified Ethical Hacker, (ISC)², Juniper, Wireshark, Offensive Security,Google, GNS3, F5, Python, Linux, Java, Openstack, Vagrant, Ansible, Docker, GIT, , Blockchain or other companies.Use of these names, logos, and brands does not imply endorsement.The opinions expressed on pheniix are personal perspectives and not those of Cisco , Dimension Data or any other comany. Pheniix runs as an independent blog.
#Cisco #CISCO_ISE #ISE #Security #Cyber_Security #Pheniix #MikeGhahremani #Mike_Ghahremani #VMWare #Backup
#CISCOISE #CyberSecurity #ISESupport #ISE #MikeGhahremani #security #Backup #Howtobackup