Serverless Framework Unlimited: Resource Extensions

Serverless Framework is great tooling, however, when it comes to building more complex systems, especially an enterprise one with strict security requirements, we soon hit the wall with the first restrictions of that magically working black box.

With strict security requirements, anything could apply, from having Cloud Security checkers with alerting rules for not conforming to best practices, such as secured API Gateways, CORS configuration for S3 buckets or simply having your stored data encrypted with some Customer-managed keys.

For the sake of comprehensibility, we will take on the latter to provide a working example of how using Serverless Framework may introduce more overhead than working with more powerful, yet complex, tools, such as Terraform.

The Blessing

Serverless Framework hands off a lot of configuration overhead from the developers when it comes to creating and deploying functions or applications utilizing the Faas paradigm of various Cloud Providers. For instance, deploying and creating the infrastructure around an AWS Lambda function would produce much more IaC overhead when doing it using Terraform than is the case with Serverless Framework. Let's compare those two with an example.

Case Study

We want a basic Lambda function, which gets some input through an AWS API Gateway and provides an HTTP response to the client according to the request payload. Further, we want to have an AWS Cloudwatch Log Group for this Lambda function to have better insights into issues or debugging concerns. We will ignore the part about creating an API Gateway for this example to prevent exhaustive overhead for the reader.

Terraform

First, let's define the Lambda function.

resource "aws_lambda_function" "lambda" {
  function_name    = "basic-http-api-lambda"
  role             = aws_iam_role.lambda_role.arn
  runtime          = "nodejs16.x"
  handler          = "handler.handler"
  filename         = data.archive_file.lambda_package.output_path
  source_code_hash = data.archive_file.lambda_package.output_base64sha256
}

data "archive_file" "lambda_package" {
  type        = "zip"
  source_file = "<path-to-handler>/handler.js"
  output_path = "<location-of-deployment-package>/handler.zip"
}

We need some Cloudwatch Log Group for it, too.

resource "aws_cloudwatch_log_group" "lambda" {
  name = "/aws/lambda/${aws_lambda_function.lambda.function_name}"
}

Now, we need permission to access this Cloudwatch Log Group from the Lambda function.

resource "aws_iam_role" "lambda" {
  name               = "basic-http-api-lambda-role"
  assume_role_policy = data.aws_iam_policy_document.assume_role.json
}

data "aws_iam_policy_document" "assume_role" {
  statement {
    effect  = "Allow"
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

resource "aws_iam_policy" "logs" {
  name        = "lambda-log-group-access"
  description = "IAM policy for creating logs by Lambda function"
  policy      = data.aws_iam_policy_document.lambda_log_group.json
}

resource "aws_iam_role_policy_attachment" "logs" {
  role       = aws_iam_role.lambda.name
  policy_arn = aws_iam_policy.logs.arn
}


data "aws_iam_policy_document" "lambda_log_group" {
  statement {
    effect = "Allow"

    actions = [
      "logs:CreateLogGroup",
      "logs:CreateLogStream",
      "logs:PutLogEvents",
      "logs:DescribeLogStreams",
    ]

    resources = [
      "arn:aws:logs:*:<account-id>:log-group:*:*",
    ]
  }
}

Serverless Framework

service: my-http-service

provider:
  name: aws
  runtime: nodejs16.x

functions:
  basic-http-api-lambda:
    package:
      artifact: <location-of-deployment-package>/handler.zip
    handler: <path-to-handler>/handler.handler
    name: basic-http-api-lambda
    events:
      - http:
          path: salutations
          method: get

That's it. This Serverless model will create a Lambda function similar to the Terraform example above. In addition to that, by default, the AWS Lambda function created with the Serverless Framework will have logging activated with CloudWatch and the necessary IAM permissions to use it. The Log Group will implicitly be named /aws/lambda/basic-http-api-lambda

The Curse

Let's assume that this time, this is an enterprise application in which the Lambdas need to have Logging enabled but this functionality comes with some requirements, such as log retention times and encryption with Customer-managed KMS keys.

Terraform

For our existing Terraform declarations, we can simply achieve this by modifying the existing resources as follows:

resource "aws_cloudwatch_log_group" "lambda" {
  name              = "/aws/lambda/${aws_lambda_function.lambda.function_name}"
  retention_in_days = 30
  kms_key_id        = data.aws_kms_alias.kms_alias_for_cloudwatch.id
}

data "aws_kms_alias" "kms_alias_for_cloudwatch" {
  name = "alias/<cloudwatch-kms-key-id>"
}

Assuming that we have created the KMS key and its alias to be accessible from all Lambda functions, we only need to attach the resources to our already existing Cloudwatch Log Group resource and apply the changes.

Serverless Framework

Doing the same thing when having Serverless Framework as the IaC tooling of choice, things are not that easy because

We don't have any resources declared for those Cloudwatch Log Groups, thus, no declaration configuration to modify to the newest requirements.
Serverless Framework only allows the implicit creation of further infrastructure around your Lambda functions, so there is no way to overwrite it with clear instructions on how to create those Log Groups.

At this point, our options may be

Prevent Serverless from creating Log Groups automatically for our Lambda functions and spin them up later with some other IaC tooling, e.g. Terraform
Let Serverless create the Log Groups for us and configure the additional requirements afterward using some other IaC tooling, e.g. Terraform
Let Serverless create the Log Groups for us and configure the additional requirements afterward using the AWS Console (GUI)

The first option seems to be somehow feasible in a manner that neglects the complexity of the codebase in favor of the ability to automate the provisioning and deployment of our infrastructure. However, typically we don't want to augment our used framework with another when we could simply use the latter (here: Terraform) to do everything from start to end. It's unlikely that Serverless Framework have been chosen if we were ready to commit ourselves to maintaining and updating yet another framework in our project.

The second option seems to be even less considerable as we have to figure out what the resource identifier (ARN) of the already existing Log Groups is. This will potentially incorporate heavy workarounds in which one would need to keep the state of those resources and continuously update the information which will then get grabbed by our secondary tooling to refer to the correct Log Group resource.

Lastly, fixing the Log Groups manually over the AWS Console by clicking the right fields after our CI/CD pipeline has finished is not a desirable goal in any DevOps process and only be considered if there is no other way to solve a problem, which will rarely be the case.

Concluding, we see that these options are not considerable for solving such a problem.

Serverless Resource Extensions

Logs have a retention period of 30 days
Logs are encrypted using a Customer-managed KMS key

Our Serverless model needs to be modified like follows:

service: my-http-service

custom:
    cloudwatchKmsAliasArn: arn:aws:kms:<region>:<account>:alias/<cloudwatch-kms-key-id>

provider:
  name: aws
  runtime: nodejs16.x

functions:
  basic-http-api-lambda:
    package:
      artifact: <location-of-deployment-package>/handler.zip
    handler: <path-to-handler>/handler.handler
    name: basic-http-api-lambda
    events:
      - http:
          path: salutations
          method: get

resources:
  extensions:
    BasicDashhttpDashapiDashlambdaLogGroup:
      Properties:
        RetentionInDays: 30
        KmsKeyId: ${self:custom.cloudwatchKmsAliasArn}

The custom section defines a variable called cloudwatchKmsAliasArn that contains the Amazon Resource Name (ARN) of a KMS Key used to encrypt the logs generated by the basic-http-api-lambda function.

The resources section defines a CloudWatch Log Group extension called BaiscDashhttpDashapiDashlambdaLogGroup. It configures the log group to retain logs for 30 days and to use the the Customer-managed KMS key specified in the custom section to encrypt the logs.

Specifics of Resource Extensions

Note that we have not used resource extensions to create new resources on which our Lambda function can depend but rather referenced existing ones, the Log Group of our basic-http-api-lambda function to extend it with additional functionality.

Referencing existing resources

When referencing existing resources that were automatically generated through a function definition, we can access it with the following pattern:

The function id in the functions section above starting with a capital letter
Replace dashes with the string "Dash" if there are any in the function name
Add the resource type in Pascal case

Thus, the Log Group of our Lambda function basic-http-api-lambda can be referenced by BasicDashhttpDashapiDashlambdaLogGroup.

Extension of existing resource declarations

For extending existing resources with further functionality, we can use resources.extensions to declare the resources that should be extended and provide the configuration variables compliant with the respective CloudFormation attributes.

The Cloudformation template for our Log Group would look like this

Resources:
  MyLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: /basic-http-api-lambda
      RetentionInDays: 30
      KmsKeyId: arn:aws:kms:<region>:<account>:alias/<cloudwatch-kms-key-id>

We can use this to form our equivalent resource extension properties

resources:
  extensions:
    BasicDashhttpDashapiDashlambdaLogGroup:
      Properties:
        RetentionInDays: 30
        KmsKeyId: ${self:custom.cloudwatchKmsAliasArn}

Using Resource Extensions for more complex solutions

Resource extensions represent the Swiss knife of Serverless Frameworks toolset. It is guaranteed to be up to date with anything that can be defined using CloudFormation templates (Serverless - AWS Infrastructure Resources). We have demonstrated how to extend existing auto-generated resources around our Lambda functions in this article, however, you can use resource extensions for many more tweaking and fine-tuning of your Serverless deployed Lambda applications. For more details on this, take a look at Serverless - Override AWS Cloudformation Resource.

Conclusion

Serverless resource extensions are a powerful feature of the Serverless Framework that allows you to extend and customize the resources created by the framework for your serverless application. With resource extensions, you can define and provision new AWS resources that are not natively supported by the Serverless Framework, or modify existing resources to meet your specific requirements. We have demonstrated how business requirements can be painful if the wrong tooling has been selected in the early stages of product or service development. Finally, we have seen, that even if Serverless Framework has drawbacks in comparison with more general-purpose tools like Terraform, there is still some room to improve the IaC-generated resources in a way that allows us to keep consistent with our tool of choice without having to switch to another level of abstraction whenever a new requirement finds its way to our businesses.

If you've learned something new, you may be interested in the following articles too: