PowerShell, HTTPS/SSL, and Self-Signed Certificates

I ran into this gem the other day and figured it was worth bitching about documenting here.

TLDR at the bottom.

For reasons that aren’t relevant to this blog I spend a fair amount of time writing PowerShell scripts to automate interactions with some networking equipement.

As you saw in one of my previous posts, I covered how to use PowerShell and POSH-SSH to log into a wad of boxes at the same time here.

Now that might work in a pinch, but lets face it- that is a hack only suitable for instances where a vendor hasn’t implemented a robust API for interacting with their devices.

Now, suppose you have a piece of network gear that does have a robust (REST) API, and you want to use that for automation like a good engineer.

Obviously you’re going to use it, right?  Right?  RIGHT?!

Okay, so you’re using the API, and you’ve written some scripts that interact with it, and you’ve automated the shit out of everything and you’re feeling good, but an auditor/PM/PO takes a look and the following conversations occurs:

(T=them, Y=you)

T: “How do you know who is hitting the API and sending commands to these devices with your script?”
Y: “Oh we ask the user for authentication as part of the script launch, then we use that data to get an authorization token from the REST API, all interactions that use a given token are tied to the credentials used to get the token.”
**(side note: you’re not hard-coding creds are you?  If you are you’re bad and you should feel bad)**
T: “That’s good, but what protocol is the API using?”
Y: “Oh, well we are just using the Invoke-RestMethod cmdlets from PowerShell, so I guess it defaults to HTTP now that I think about it.”
**(queue alarms and screaming and general chaos in your head)**
T: “Well HTTP is insecure, and if you’re using it for authentication somebody could steal your credentials!”
Y: “You’re right, but I noticed the API supports HTTPS, so I’ll just update the script to use HTTPS instead, should be a two minute fix.”
T: “Perfect, but before you do that we’ll want to make sure that we get it assigned to a sprint at our next syngery session tomorrow.”
Y: “Uh, it will only take a minute.”
T: “Great, so the number of effort points will be low.”
Y: “But I…”
T: “Can’t wait cover this tomorrow!”
Y: “::sigh:: Sure thing.”

Of course you start looking at it right away!
All we need to do is change  Invoke-RestMethod -URI http://myserver.myorg.foo/apistuff(...)  to Invoke-RestMethod -URI https://myserver.myorg.foo/apistuff(...)

Then we do a quick test run and…**explosion**

What the hell do you mean “The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.” ?!

Then you think.

Oh, PowerShell must want me to use valid certificates, and my appliance just uses a self-signed certificate.

Coolio, we’ll just flip the -insecure flag and be done with it.

What the hell do you mean there is no -insecure flag for Invoke-RestMethod?!   Curl has one!

Okay, well now I have two options:

  1. Install valid signed certificated on all of my appliances running HTTPS APIs
  2. Find a way to get PowerShell to ignore certificate errors

Well, number is certainly a possibility, but many vendors don’t support changing the self-signed admin cert on their HTTPS pages, and also, you have hundreds of devices that would need this (you could automate it with the API, but oh. damnit.  Serious chicken/egg situation here.)

Okay, so now we’re at, “How do I disable the certificate validation?”.  Well you probably did some googling and found this little one-liner (or similar):

You run your script again, and now you don’t have TLS errors any more and your authentication is happening over HTTPS!  YES!

Unfortunately the fun stops there, any subsequent calls you make over HTTPS fail with the cryptic message: “The underlying connection was closed: An unexpected error occurred on a send.

Great, what the hell is this?

Well it turns out the hack above only works for the first call.

So now we do more Googling and find there are some better options, like adding this to the top of your script:

So you run it and get the same failure.  What the hell?  They said it works though!

Well as it turns out, this little diddy:

has some lasting effects for the length your PowerShell ISE is open.  So remove that line, add the function above, save your file, and restart your PowerShell ISE/powershell terminal, then re-run the script.

POW!  Everything works now!  Time for Scotch!

TLDR;

This has semi-permanent effects if your initial flailing/googling had you try this.

Put the code below at the top of your code, save the file and re-launch your ISE/powershell session.

Now you’re good to go!

IPSEC, SCEP, NDES and other stuff you probably wish you hadn’t heard of

 

I’m in an environment now where I have to proof-of-concept complicated/large ideas for environments to prove their feasibility.

The latest project:

  • High performance IPSEC for site to site encryption
  • BGP over the IPSEC tunnels
  • Use of certificates for authentication
  • Automatic enrollment/renewal of said certificates
  • Verification of certificates via OCSP

 

Well then, lets get started.

IPSEC and BGP aren’t much new and are pretty straight forward.  However, most places I’ve ever been have used Pre-Shared Keys for their IPSEC as using certificates means that you have some sort of CA infrastructure and can require a lot of overhead.  Getting static certificates to work is one thing, but getting them automatically created and renewed is another thing entirely.  Then there’s OCSP, again typically not used because of the infrastructure that is required to support it.

First, the topology:

IPSEC-Topology

 

We have the following:

  • a vRouter (vyOS, which deserves its own post for how awesome it is)
  • a Windows 2012 server running AD and a compliment of Certificate services (which we will cover)
  • two virtual appliances running IPSEC that are running eBGP between the vRouter (and each other)

Phase 0 – What is in (and out) of scope for this post

This post isn’t intended as an all inclusive guide to setting up the underlying environment, but rather configuring the pieces that exist on top of said environment.

A quick summary of the underlying environment:

  • 2 Dell R810 ESXi 6 servers managed by VCSA 6
  • Virtual router provided by vyOS
  • The ‘Internet’ as depicted in the diagram is an upstream router that has access to other lab networks (as well as the upstream Internet)
  • All of the point to point networks are vSwitches (actually a dvSwitch) with a corresponding backend VLAN on my lab switch

Phase 1 – Setting up the underlying network architecture (and making sure it works)

First lets make sure that we get the vRouter properly configured, here is what it should look like when finished.

Now that we have that portion, lets go ahead and configure the A10 devices (IPSEC-A and IPSEC-B)

 

IPSEC-A

IPSEC-B

Now that we have that configured, lets make sure that IPSEC is working, and that BGP is working.

From here we can see that both A10 devices have peered with the Lab6 vRouter and are both advertising two networks.

And if we take a deeper look:

Based on this, we can determine the following:

  1. BGP is working between the Lab6 vRouter and both A10 Devices
  2. IPSEC is working between both A10 devices
    1. We know this because of the path of the learned networks from the A10 peer, which could only be learned over the IPSEC tunnel

But still, its probably worth confirming IPSEC on the A10 devices anyway:

On IPSEC-A

And on IPSEC-B

Furthermore, lets verify that we are learning routes via BGP over the IPSEC tunnel:

IPSEC-A

and on IPSEC-B

So, IPSEC is up, and BGP is working perfectly. Lets move on to Phase 2.

Phase 2 – Deploying Windows and Active Directory

I’m going to give you the cliff-notes version here, because you either already have AD installed and running, or you can go to some MSFT blog to figure it out.

So…  Install your Windows 2012 server, deploy Active Directory, and make sure that it is reachable by your lab environment (I attached mine to the ‘management’ network.

Phase 3 – Configuring Certificate Services for Windows 2012

This is where it gets a bit hairy.  Windows Server makes it really easy to deploy services by just choosing to add a feature.  Unfortunately sometimes it can be really confusing.

The good folks at Microsoft have published a pretty good guide for configuring a CA, and OCSP responder and NDES (SCEP) server.

You can find all of that hotness here:

https://technet.microsoft.com/en-us/library/cc772393(v=ws.10).aspx

Unfortunately NDES seems to have been a bit of an after-thought for the Windows Server environment, so once you have the role set up, the only way you can make meaningful configuration changes is via registry keys (and then restarting the IIS service).

However, once again, the Microsoft folks come through with more solid documentation and have a posting that shares everything you ever wanted to know about NDES (including appropriate registry keys!)

More hotness here:

http://social.technet.microsoft.com/wiki/contents/articles/9063.network-device-enrollment-service-ndes-in-active-directory-certificate-services-ad-cs.aspx

 

Here are some of the common problems that I found when setting this up:

  1. OCSP
    1. Did you remember to do the post deployment tasks available from Server Manager?
    2. Have you created your Revocation configuration? (available from the Online Responder MMC snap-in)
    3. Did you remember to create a copy of the Online Responder Template that allowed auto-enroll and put it into the list of available templates for your Certificate Authority? (this can be done using the Certificate Templates and Certificate Authority MMC snap-ins)
    4. Did you verify that it is actually working? (you can do this by creating a certificate and then running the following command against that cert ‘certutil -URL <fullcertpathhere>’ then checking the revocation information)
  2. NDES
    1. Some devices (like the A10’s) don’t yet use the old certificates for the renewal process and will require re-use of the old password
      1. You need to enable password re-use
      2. You probably want to make that password much longer, since it can be re-used
      3. You probably need to increase the password cache to the number of devices you can support so that you don’t have to re-use passwords
      4. You probably want a certificate that doesn’t last two years for your IPSEC devices (plus we’re auto-rotating so its not like we personally have to do it (do this by creating a copy of the IPSEC (offline request) template and making appropriate changes
      5. See the registry configuration values from the NDES link above to change the first 3 items and to update NDES to point to your new certificate template for the fourth

 

 

Phase 4 – Configuring the A10 devices to use certificates

Now we have a few things that need to be done:

  1. Install the trusted Root CA on the A10 Devices
  2. Configure SCEP on the A10 devices so that they can automatically get and renew certificates
  3. Configure OCSP on the A10 devices so that they can actually verify that the certificates haven’t been revoked
  4. Update the VPN config to use the new certs

Install the trusted root CA (you know from the CA you created earlier) on both of your A10 devices.

There are a couple of ways to do this (FTP, TFTP,SCP,SFTP or GUI).

Since all of the command line ways require you have another server from which to upload, I suggest using the GUI, where you can upload the file directly.

Log into the GUI on your A10 devices and then go to ADC–>SSL Management–>Import

Choose the name that the A10 will reference the CA by (I choose IPSEC-CA), choose the appropriate options (don’t forget to click the radio button for CA certificate) choose the file that is your CA Cert (that you downloaded from your CA) and click import.

 

Configuring SCEP on IPSEC-A
The password will be the password that you received from the URL on the http://<your NDES server here>/mscep_admin/
You will should get a new one for each device you enroll

Configure SCEP on IPSEC-B

Now let’s verify that we actually got our SCEP certs:

Verification on IPSEC-A:

Verification on IPSEC-B:

Woo-hoo- we got our SCEP certs!

Now let’s configure OCSP for checking certificate revocation:
On IPSEC-A

and on IPSEC-B

Finally, lets update our VPN to use all this new auto-certificate hotness!

On IPSEC-A

and on IPSEC-B

Now, once you’ve done this, you’ll want to verify that it is working with the same VPN commands that were shown earlier.
If you are having trouble getting the tunnels to come up you can troubleshoot by using the following commands:

These will set the debugging to the highest level possible for VPN (which will show you keys so you can verify they are working correctly)
The log command will follow your vpn debug packet by packet and will provide helpful information as to whether or not you are failing auth (or something else like OCSP)

 

I hope this was somewhat helpful.

Time for…

Phase 6 – Go get a beer.

Or a Scotch.  Or both.  Yeah, probably both- better play it safe.