Welcome to my first post of 2018! And managing certificates is still a pain! What’s new?
I’ve been very busy since my last post, I became a Sophos certified Security Architect (Woo!), and because they’re so great they gave me some licenses to set up a virtual XG Firewall in my lab.
One of the cool things it does is SSL Inspection, whilst this isn’t a new thing, it is easier to configure than a FirePower module on a Cisco 5506-X. It’s a lot quicker to setup as well (I’ll just leave that there).
So – we all know how this works, you break the certificate chain between the browser and the server and you insert your intermediary, i this case the XG. The XG then forms an encrypted tunnel between itself and the remote server, and then between itself and the client. This last stage is the tricky one as the XG signs the certificate between itself and the client, you will get security warnings and some SSL websites (especially ones that use certificate pinning) will fail to load entirely.
The solution is simple – just install the root certificate from the XG and push that to all your machines through group policy.
I don’t like this solution. I don’t want to push another root CA key to all my clients. I already have a CA my clients trust. Importantly – my mobile clients already trust it.
Managing mobile device certificates without MDM is not great fun, and some MDM vendors don’t allow you to do this at all (Meraki make this very easy, if not immediately obvious). If you have already put your CA root key on those devices, it makes a lot more sense to have any certificate you issue to those clients, be trusted by the certificate they already have.
This is achieved by installing a subordinate CA certificate on the XG. Sophos have a great knowledge base article about it here. It has screen shots and everything, but it depends on using the web interface to the CA, and because I don’t have IIS installed on that server (and have no intention of doing).
You’ll need the following command to generate the certificate from the command line:
Once you’ve done this, upload the certificate to the XG firewall and follow the rest of the KB article to change the Web CA Settings to use this new certificate authority and your internal pages will now show as trusted through their existing domain root CA key.
You probably now want to generate a certificate for the XG itself so that the management and user portals also lose their security warnings. The process is nearly exactly the same as before except this time you use the following command to generate an SSL certificate for the device itself:
Avaya One-X Portal, it’s a fantastic application, but it’s a nightmare for large deployments. If you have 200 users all wanting to use the client integration features, you’re going to have a problem when you install a new service pack on your IP Office. You will of course recognise the message:
The first thing you would normally do is jump onto the one-x web portal, and download the new client bundle, whilst this is fine for small deployments and a mild inconvenience, it becomes a big problem, because the IP Office no longer sits in the sub 500 seat space, and can now be used as a centralised phone system, with all the benefits that that brings, it has some down sides, such as this.
So, what do you do then? You need to remotely deploy the client to your enterprise workstations. How? Enter in your favourite management agent – System Centre (Other solutions are available, such as Meraki MDM, and I hope to cover that in another post).
First, go ahead and download the new client from the One-X web portal as you normally would, you’ll find it under Desktop Integration on the Configure tab. Download the unnecessarily long filename executable and save it somewhere.
Anyways, once you’ve got the file in a safe place (I should add – not on a network folder, unless you have a drive mapping set up) drop to command line in that folder you need to run the executable with the /a parameter. Select a Network location to deploy the image, I had real issues actually selecting a network location so just use a local path. Note, this will not update the client on your machine if you have it installed, it just creates the msi files and two folders (program files and Win).
Next, we need to customise the .msi file and inject your company data so that the one-x server, ports, and ssl settings are already in the client when it launches for the first time. This is important, just deploying the client isn’t enough, the goal here is to have as little end user disruption as possible.
Avaya recommend the ORCA tool for this, it’s available from here. Avaya’s implementation manual has the following to say on the matter:
After downloading and installing ORCA, right-click on the MSI file and select Edit with Orca.
From the left-hand Table item select Property Table.
From the list of tables select Property. Scroll to the bottom of the table, right-click and select Add Row.
Add the following Properties with corresponding values to specify one-X Portal server address, port and secure communication mode.
ONEX_PORTAL_SERVER = IP or FQDN, this must be FQDN if using secure mode.
PORT_NUMBER = server port number – 9443 if using secure mode
SECUREMODE = 1 for enabled or 0 for disabled
After adding all the properties, save the MSI file and close the ORCA tool
You can download the full document here, it gives quite clear guidance on doing this simply using group policy objects. However if you’ve invested in System Centre you want to use it, so lets crack on..
Your Orca-edited file should look like this when you’ve finished:Create a Share for this deployment package on your SCCM server, copy the files to it.
Now we need to create a new application in SCCM, so goto Software Library -> Application Management -> Applications, right click in the main window and Select Create Application. Brose to your network share containing the client and hit next, accepting the warning about not being able to verify the publisher.. Proceed to General Information and put in some information like version number. I also set an administrative category of Productivity. I also chose to install for system to accommodate multi user systems. Click next through the rest of the process.
Now that we’ve told SCCM about our package we need to deploy it. Right click on it and select Deploy. You will need to have a collection of devices defined to deploy your software to, select this under collection, if this doesn’t exist you will need to create it. Exchange servers don’t need this client installing on them, and you’re likely to piss somebody off if you try. Click next and select a distribution point.
On the next screen select action of install and purpose required. This is a mandatory install of software for this device collection, although I don’t think a deadline is necessary – so just click next on the next page.
For user experience, I’m going to leave these on for my environment, although you can turn them off if you wish. I’m also not going to configure alerts but of course you can if you want. Press next some more till you come to the end.
After the client machines run their cycle to detect new software, they will automatically and silently if selected download and install the client. This is great for machines that did not have the software before, but not so great for machines that have the out dated software on, as system centre’s default detection rules will identify it as already installed, so won’t do anything. Go back to Software Library -> Application Management -> Applications, and right click on the Avaya Clients App, go to Properties and Select the Deployment Types Tab, Select the entry that is there and click Edit. You then need to go to the Detection method tab and edit the default rule, click browse and find your msi file again, then select the bottom radio button and select Version from the MSI Property drop menu, the Value should be automatically populated, like this:
Existing machines will now get a message instructing them that the IT Department needs to make software changes to their PC, and attempt to deploy the new client on top of the existing one, unfortunately this will fail. We must first remove the existing client from the system, and what we’re about to do is a bit of a bodge because I didn’t have the old install msi kicking around, so I created a new folder on the distribution share and copied the files there, then I added a new application to SCCM as we did with the latest client, and ran through the settings again. Then I went into the detect the detection settings to match the old client that was installed unmanaged. After doing this I deployed the application however this time selected uninstall from action dialog of wizard.
After the client ran its Machine Policy and Application deployment cycles, Configuration Manager removed the old client, which in turn allowed the new client to be deployed. This was successful but despite the old client being removed it remained running, closing it prompted a notification from CM that the installation was successful, implying that the installer was pending the application being closed, or a shutdown/logoff event.
Thankfully updating the application this way requires zero user interaction, unless the application is running, but even then its a simpler process for then end user than manually downloading and installing, and allows you to leverage SCCM to ensure compliance with applications your business runs on.
I recently updates System Center and noticed one of the new features seemed to relate to force closing a running application during deployment, however I have no idea where I put the screenshot. Avaya are due to release a new version of their IP Office platform very soon so, I will optimize my time and investigate this new and exciting feature then. I’ll also post an update about how to manage the life-cycle of the One-X application more thoroughly.
Update: April 2018
Avaya have now moved the goalposts with this, and after upgrading your system to IP Office 10.1 SP2, clients are offered the software update within the application, however installation will require local admin access, so if you don’t allow this to your users, you may still be best deploying by SCCM.
So, you thought you’d buy into the next generation of firewall technology early? #FAIL
After ongoing reporting from The Register, Cisco have finally admitted some of their products are affected by the intel C2000 series chip problem. This affects most (all?) ASA5506’s, 5508’s, and 5516’s, along with some other equipment, for a full list – look at Cisco’s product notices here.
The solution – fill out their spreadsheet with your devices details and get it sent over to Cisco, they promise to replace the equipment as soon as possible. I will keep this topic updated with my experience, but so far the only replies I’ve had to my email with the sheet was a mailbox full message (!) and a confirmation saying that I should get a response within the next two to three weeks, so don’t wait to get your request in.
This problem is not isolated to Cisco equipment, with Synology, Netgear, HP, Dell and Supermicro also having products affected by the faulty chip. For more information head over to The Register for the latest.
So Far the list of chips suspected to be vulnerable to sudden death are the C2308, C2338, C2350, C2358, C2508, C2518, C2530, C2538, C2550, C2558, C2718, C2730, C2738, C2750, and C2758. Although it seems vendors are reluctant to admit to problems being caused by the processors it seems that the writing is on the wall, go out and check what your new network device is actually running, and contact your vendor to make sure they have a plan to get you back up and running when it fails!
Azure, wow isn’t it amazing? IAAS has arrived, and you can do some pretty cool stuff with azure, without worrying about licensing or hardware, beyond the size and spec of your virtual machines in their Data Centres. But that’s just the bloody problem isn’t it? it’s in their network, not yours! So, you need to set up a site to site link between Azure and your local supported device.
Click here to view a list of supported devices and their configuration instructions. This will tell you what type of VPN your equipment will support, and what you should deploy in Azure portal.
My local supported device is a Cisco ASA, which you will notice is only supported for Policy Based, not route Based VPN, so you need to start out right or you’re going to waste a lot of time deploying VPN gateways in Azure, which takes about 45 minutes. The instructions on GitHub are easy to follow and available here [.docx].
If you follow these instructions, you will get going fairly quickly, but if you have multiple subnets you want to allow to communicate with Azure you will have to make some modifications. For example, my protected wireless network should have access to Azure, because I’m going to put a domain controller on a virtual machine there, to handle DHCP (not supported!), DNS, and 802.1x authentication in a Disaster Recovery scenario.
You will need to modify your NAT like this:
first create names for the networks you want to access your azure networks:
Ok, so adding a Nano server to your on premise active directory is not as straight forward as you might like, but of course there’s no GUI so what did you expect? The process is not the same as adding server core to your domain, as it can only be managed remotely, you must first connect to the machine using winRM / PowerShell, and perform a three stage operation to join the Azure Nano machine to your domain.
Of course, you’ve already set up your site to site VPN with Azure? No – I haven’t blogged about it yet! Of course, because you’ve read my blog post about that too! It is straight forward enough though, follow the instructions provided, and check your NAT, but other than that it’s a breeze.
First you must deploy your Nano from the azure portal, I’m not going to go through this part here, most of the blogs I’ve seen on the azure portal are out of date or relate to the classic portal, and with a major server OS release having just happened I expect there will probably be more changes to the azure portal, so I’m going to leave that but for now.
Stage one: Connect to the Azure Nano Server using powershell
Open PowerShell locally and start winRM
PS C:\WINDOWS\system32> net start winrm
The Windows Remote Management (WS-Management) service is starting.
The Windows Remote Management (WS-Management) service was started successfully.
You then need to configure your trusted hosts list for winRM (more info on this here).
So you now need to change the DNS server to be one of your own domains:
netsh interface ip set dnsservers name="Ethernet" static 10.87.0.10 primary
Microsoft have turned the firewall on the Nano on by default, and you will need to enable the firewall ports for file and print services to be able to transfer the file you will create in the next stage: adding it to your domain.
netsh advfirewall firewall set rule group=”File and Printer Sharing” new enable=yes
Stage thee: Joining to the domain
Ok, we’re done here for now. Over to your domain joined server, you need to use djoin , which will put the machine into AD and spit out a file for you to import into the Nano, run this command from your desktop to make your life easy, as it the file is generated in the directory it is run from! it looks a little something like this:
Loading provisioning data from the following file: [c:\temp\WAN02ADC01PV-DOMAIN].
The provisioning request completed successfully.
A reboot is required for changes to be applied.
The operation completed successfully.
You will then need to seal the deal with a remote reboot:
You’re done, it is added, you can now manage it using your domain credentials, and it can be added to server manager, although I don’t believe you will be able to add role and features to it this way, but this may change, who knows.
OK, so I thought I was pretty lucky to install Windows 10, and the only thing it broke was itself. I turns out I was not so lucky and it has broken my Cisco VPN Client aswell, I am now greeted with the pretty serious looking message shown below, after what looks like the client trying to install itself again after trying to run it:
Error 27850. Unable to manage networking component. Operating system corruption may be preventing installation.
So, Windows 10 is a corrupted operating system? nice one – I just installed it and now it’s corrupt. Really??! No, but it was fun to procastinate.
So first reaction that something in the Windows 10 upgrade has changed the networking configuration, so we should allow the client to try to repair the install through programs and features control panel, this (sort of expectedly) failed with the same error. But at least we tried it, right?
So lets propperly remove the client and do a full reinstall. Uninstalling the client was unnerving, which is not something I have ever felt when uninstalling an application! The windows installer loading bar, loaded and exited without any prompts, and the client was removed from the list of installed applications. You can see an example of the windows installer progress bar above.
The software failed to install with the same error message we’ve seen twice now. Frustrating? Just a little.
Lets try DNE! Ok so some rebooting invloved here unfortunatly. And also an enlightening experience thinking about my Windows 10 issues. When installng the client after performing the DNE update as per my earlier post, I got some installation problems relating to folder redirection, so I had to install the client using a local admin account. This sort of confirms my suspicious that folder redirection is causing issues with my machine more investigation required on that one I think, I’ll let you know.
Now, rather alarmingly, it seems that the client is on an offical Windows 10 unsupported list, in that if you try to run the .exe to install the VPN client, you get directed to a windows 10 incompatible app page. Run the .msi, everything will be OK..
Back on track, and having now installed the software using the .msi and a local admin account (you will not need to do this if your machine is not domain joined), I have logged back on using my domain account, and hey presto, the client seems to be working. It loads without warning, anyway, it won’t connect though. The log reads Failed to download keys. Error 433, Bitch.
So, apparantly installing another vpn client, the Sonic Wall 32bit or 64 bit client first then installing the cisco client prevents the problem from occouring. So here we go… again… uninstall the cisco VPN client… and reboot.. then uninstall the DNE Update. Then reboot and run winfix again. Reboot, then install sonic wall then cisco VPN client, make sure you apply the registry fix if you get the cannot enable virtual adapter error, and away you go. In business.. 2 hours later…
There you go, eight simple steps to get it working again. Simple when you know how.
And the cause? Well Microsoft will update their operating systems from time to time, and Cisco want you to use anyconnect now so havent supported this client for some time, this will get you running for now, but maybe it’s time to start thinking about a new platform?
The Cisco VPN Client used was vpnclient-winx64-msi-5.0.07.0440-k9
Let me know if this works for you, or if it doesn’t!
When installing Avaya IP Office Voicemail Pro 9.13.0, I came across this error, 1327 Invalid Drive:<driver letter of problem drive> and was unable to install voicemail pro.
Searching google suggested that a change is made to the registry to resolve this issue for office installs, and some suggested disconnecting the drive and disabling automatic reconnect.
There is, however, an easier solution:
Step 1: Create the local voicemail user
Step 2: Log in as that user,
Step 3: Run your installation application again. It should now complete without error. If you are asked to restart the machine, you must log in using the local account to continue installation.
Step 4: Once the install is completed, you can continue to use your domain credentials.
I hope this saves you some time.
This problem is not really Avaya related, but an issue with Microsoft Windows, so this procedure may or may not work for you if you are trying to install some other application, but it’s worth a shot. Let me know if it worked for you…
OK, lets start this off with a fairly uncommon problem that allows a VPN to connect but does not pass any traffic. The VPN profile and account are tested on another machine and found to be working, so the problem is (gasp!)… the client machine!
You can verify this problem by noticing an ‘error with call to iphlpapi.dll’ in the cisco client logs (you may need to enable them on the logs tab of the client).
I hope this saves you time, and stress, in the future.
I have seen this resolve problems in similar circumstances with WatchGaurd as well as Cisco ASA’s (5505 or 5510) VPNs aswell, only they actually don’t seem to connect at all, I haven’t seen the error message but if I find it I’ll put it here. Thanks to “The other Martin” for the tip.