Alert Channels - Solution for Splunk with Self-Signed Certificates

How to set up an AWS Application Load Balancer (ALB) in front of Splunk

Why would you need to do this?
If using HTTPS on your Splunk listener, Lacework Alert Channel integrations will reject self-signed certificates. If you use a self-deployed Splunk environment on EC2 in AWS and are using self-signed certs, your Lacework Alerts integration is not going to work without some additional effort.

If you’re using Splunk Cloud, this isn’t necessary.

How can I tell if we’re using self-signed certs?
Curl the Splunk URL with the -vvv option in the command line.


Note line 17 above. If you see “SSL certificate problem: self signed certificate in certificate chain”, this article is for you.

Solutions
There are 2. (Technically there are 3, but disabling SSL is a terrible idea)

  1. Deploy a real certificate on their Splunk servers, or…
  2. Set up an AWS ALB with a real certificate from Amazon Certificate Manager (ACM) to terminate the HTTPS request from Lacework and then make the connection to Splunk. ALBs don’t care if you have a real cert on your server(s) or not.

We’ll be doing the second one.

Prerequisites
AWS Console access with permissions to administer:

  • Application Load Balancers
  • Amazon Certificate Management
  • DNS
    • could be Route53, or an external provider
    • if external provider, 2 records will ultimately be created
      • cname record specified by AWS to prove domain ownership
      • cname for your Splunk instance pointing to the ALB DNS name generated by AWS

Procedure
Create Target Group (AWS Console - Target Groups)

  • Target type will be Instances
  • Give it a name
  • Protocol is HTTPS, port is 8088 (usually for Splunk)
  • Pick the VPC where your Splunk listener sits
  • Protocol version HTTP1

The ALB will only route requests to ‘healthy’ instances. We’re only going to have 1 Splunk instance, but we still need to show it’s up and healthy. We need a HTTP 200 status code to come back, so we’re going to instead check the Splunk login page on port 8000. This is under Advanced health check settings.

Health check path will be /en_US/account/login

Override the port and use 8000 instead of the listener port of 8088.

  • Choose HTTPS
  • Health check path is /en_US/account/login
  • Open Advanced health check settings and override the port to use 8000
  • Everything else can be left as default.


Add tags if desired (it’s always good practice to tag), then click Next

  • Select the EC2 instance running your Splunk listener and click “include as pending below”
    Obtain a valid SSL certificate (AWS ACM Console)

We need a certificate from ACM to put on the Load Balancer we’ll be creating. If you have an existing wildcard cert (you might) you can skip this part.

  • Request a public cert, click Next.


  • NOTE: The Fully qualified domain name MUST match what you intend to use for the public DNS name AND you must be able to prove ownership of this domain through a domain verification procedure

  • Here we’ve clicked in to the certificate and see it’s pending and we need to validate. If external DNS, create that CNAME record shown in your DNS control panel. If Route53 hosted, click on the “Create Records in Route 53”

  • This is what it looks like to create in Route53.

Once validated, we can now create the Load Balancer
Create Application Load Balancer (AWS Console - Application Load Balancers)

  • Name it and make it Internet-facing

  • Pick the VPC where your Splunk instance sits and at least 2 AZs.
  • Select a security group. Make sure this security group allows TCP/8088 (or whatever the Splunk listener port is if not default) from the Lacework IPs

Here is where we select the Target Group and SSL Cert that we created in previous sections.

  • Choose HTTPS for the protocol.
  • Use port 8088
  • Forward traffic to the Target Group we created earlier.
  • Select the Default SSL certificate from the one we created via ACM

It’s created, but now we have one final step. We need to head back to DNS and create a CNAME for our external Splunk URL and point it to the DNS A record that AWS assigned to this ALB.

Route53


Create the CNAME

If you can’t connect to the Splunk listener, validate your Security Group settings for the Load Balancer.


Here’s the setting in Lacework, with SSL enabled, pointing to the listener CNAME for the AWS Load Balancer.

Hope this helps!

2 Likes