Complete the steps to configure CloudWatch Log inputs for the SplunkAdd-on for Amazon Web Services (AWS):
- You must manage accounts for the add-on as a prerequisite. See Manage accounts for the Splunk Add-on for AWS.
- Configure AWS services for the CloudWatch Log input.
- Configure AWS permissions for the CloudWatch Log input.
- (Optional) Configure VPC Interface Endpoints for STS and logs services from your AWS Console if you want to use private endpoints for data collection and authentication. For more information, see the Interface VPC endpoints (AWS PrivateLink) topic in the Amazon Virtual Private Cloud documentation.
- Configure CloudWatch Log inputs either through Splunk Web or configuration files.
Due to rate limitations, don’t use pull-based (API) input configurationsto collect CloudWatch Log data which has the source typeaws:cloudwatchlogs:*. Instead, use push-based (Amazon KinesisFirehose) input configurations to collect CloudWatch Log and VPC FlowLogs. The push-based (Amazon Kinesis Firehose) input configurations forthe Splunk Add-on for AWS include index-time logic to perform thecorrect knowledge extraction for these events through the Kinesis inputas well.
Configure AWS permissions for the CloudWatch Log input¶
Required permissions for Logs:
DescribeLogGroupsDescribeLogStreamsGetLogEventss3:GetBucketLocation
See the following sample inline policy to configure CloudWatch Log inputpermissions:
{ "Version": "2012-10-17", "Statement": [ { "Action": [ "logs:DescribeLogGroups", "logs:DescribeLogStreams", "logs:GetLogEvents", "s3:GetBucketLocation" ], "Effect": "Allow", "Resource": "*" } ]}You must also ensure that your role has a trust relationship that allowsthe flow logs service to assume the role. While viewing the IAM role,choose Edit Trust Relationship and replace that policy with thisone:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": "vpc-flow-logs.amazonaws.com" }, "Action": "sts:AssumeRole" } ]}Configure a CloudWatch Logs input using Splunk Web¶
To configure inputs using Splunk Web, click Splunk Add-on for AWS inthe navigation bar on Splunk Web home, then choose one of the followingmenu paths depending on the data type you want to collect:
- Create New Input > VPC Flow Logs > CloudWatch Logs
- Create New Input > Custom Data Type > CloudWatch Logs
Fill out the fields as described in the table:
Argument in configuration file | Field in Splunk Web | Description |
|---|---|---|
| AWS Account | The AWS account or EC2 IAM role the Splunk platform uses toaccess your CloudWatch Logs data. In Splunk Web, select an account fromthe drop-down list. In |
| AWS Region | The AWS region that contains the data. In |
| Use Private Endpoints | Check the checkbox to use private endpoints of AWS Security TokenService (STS) and AWS Simple Cloud Storage (S3) services forauthentication and data collection. In inputs.conf, enter |
| Private Endpoint (Logs) | Private Endpoint (Interface VPC Endpoint) of your logs service,which can be configured from your AWS console. |
| Private Endpoint (STS) | Private Endpoint (Interface VPC Endpoint) of your STS service,which can be configured from your AWS console. |
| Log group | A comma-separated list of log group names. |
| Only After | GMT time string in '%Y-%m-%dT%H:%M:%S' format. If set, onlyevents after this time are queried and indexed. Defaults to1970-01-01T00:00:00. |
| Stream Matching Regex | REGEX to strictly match stream names. Defaults to |
| Interval | The number of seconds to wait before the Splunk platform runs thecommand again. The default is 600 seconds. |
| Use Metric Index? | Whether to use metric index or event index. The default value is No (use event index). This field is only visible when creating VPC Flow Logs -> CloudWatch Logs inputs. |
| Source type | A source type for the events.
aws:cloudwatchlogs if you arecollecting any other types of CloudWatch Logs data. |
| Index | The index name where the Splunk platform puts the CloudWatch Logsdata. The default is main. |
| Query Window Size (minutes) | Specify the interval of data to be collected in each request(in minutes). Default=10, Min=1 & Max=43200(30days). For example, if the calculated start date is 2024-01-01T00:00:00 (midnight on January 1, 2024) and query window size is 60 minutes, then end date for the request will be 2024-01-01T00:01:00 (one hour after midnight). The time period will continue sliding by 60 minutes until no more recent logs are available.. |
Configure a CloudWatch Logs input using configuration files¶
To configure the input using configuration files, create$SPLUNK_HOME/etc/apps/Splunk_TA_aws/local/aws_cloudwatch_logs_tasks.confusing the following template:
[<name>]account = <value>groups = <value>index = <value>interval = <value>only_after = <value>region = <value>private_endpoint_enabled = <value>logs_private_endpoint_url = <value>sts_private_endpoint_url = <value>sourcetype = <value>stream_matcher = <value>metric_index_flag = <value>query_window_size = <value>Here is an example stanza that collects VPC Flow Log data from two loggroups:
[splunkapp2:us-west-2]account = splunkapp2groups = SomeName/DefaultLogGroup, SomeOtherName/SomeOtherLogGroupindex = defaultinterval = 600only_after = 1970-01-01T00:00:00region = us-west-2sourcetype = aws:cloudwatchlogs:vpcflowstream_matcher = eni.*metric_index_flag = 0query_window_size = 10)
