Building a Real-World Use Case with Step Functions and the Callback Pattern
Table of contents
In this article, we talk about Step Function Workflows. One major use case that Step Functions enable is the Wait for Callback pattern.
Code Included
This edition comes with a special bonus - for the first time, we're including a GitHub repository just for you! Clone the repo and deploy it in your AWS Account.
What are even Step Functions?
Step Functions orchestrate workflows with different AWS Services. They are all about making different AWS Services work together in harmony.
You can make use of direct integrations like:
💾 Scanning items in DynamoDB
⚡️ Invoking Lambda functions
📤 Sending WebSocket Messages
📊 Getting data from external APIs
You can have dedicated error & retry mechanisms for each step within a Step Function, making it a powerful orchestrator.
Today, we will build such a workflow together.
The Workflow Designer
AWS offers you a great tool to get started with Step Functions: the workflow designer.
It's a bit like playing with digital LEGO on your screen, where you can drag and drop pieces to build your workflow. The workflow designer is a low-code tool. Meaning that you don't need to code your workflow. Instead, you'll design it.
Here is an example from the workflow designer. It is a demo workflow by AWS:
However, in the end, you need to have some sort of code to include it in your Infrastructure as Code (IAC) provider. From the workflow designer, you can export a .json file in the Amazon State Language (ASL).
Or with an IAC framework like the CDK (which we use in the GitHub repository), you can build up your whole Step Function with its own syntax.
We prefer the latter one. It can be quite cumbersome to export a huge .json file into your code base every time you update one option in your state.
To get started with Step Functions we recommend using the workflow designer first. It shows you what step functions are capable of. But if you want to have it in production, translate it into IAC.
Callback Patterns Let You Wait for User Input
Today, we're diving into the magic of the callback pattern with Step Functions, a feature that's like putting your workflow on pause. Think of it as a "Hold, please! 👋" button for your tasks. The callback pattern lets a part of your workflow wait quietly until it's given the green light to go again.
Picture this: you grab some data from DynamoDB, send it over to a Lambda function, and then—here's where the magic happens—you tell the next Lambda task to hang tight until it gets a special signal, a callback.
This setup is perfect when you need a human interaction, like a manual check or approval, before moving forward. Want to see how we put this together? Check out the Step Function Definition on GitHub and follow along with us.
Real-world Application: Content Moderation
Our use case for implementing the callback pattern is a Content Moderation System. Imagine hosting a blogging platform such as Hashnode or Medium. You don't want to enable everybody to publish anything. This is why you put some safeguards into place.
We've created a streamlined model for this concept. Here's the scenario: when someone submits an article that includes a specific no-go word (yep, "forbidden" is our trigger word), we don't just push it live. Instead, we pause and call in a human – a manual reviewer – to give it the once-over.
Is it okay to publish, or does it cross the line?
This is how the end state will look like:
An article title is submitted.
Our Step Function kicks into gear.
If the title includes "forbidden", everything pauses, waiting for a special nod – a task token.
Over in the admin dashboard, there's a new article flagged for review.
Upon approval, the Step Function springs back into action, moving the process forward.
It's a simple but effective way to keep tabs on content.
Diving Into the Architecture
The architecture consists of several parts:
Frontend - a Next.JS/React frontend that handles the submissions and admin dashboard
Lambda API - a REST API based on a Lambda Function URL that handles all business logic
Step Function - a Step Function to handle the moderation workflow
DynamoDB - all workflow executions that need to be reviewed including all URLs
Lambda Approval Receiver - another API to approve or reject the Step Function
This is a birds-eye view. Let's dive into it.
Triggering Step Functions
Once the user submits an article we start a new Step Function with the given title.
Content Check: First up, the Step Function scans the title. If the "forbidden" word is present, we'll pause the Step Function.
Requesting a Review: Once paused a Lambda function is executed that sets the workflow into its waiting state. The execution ARN and some other information are saved in DynamoDB.
The Wait for Callback Explained
This waiting game means the Step Function halts until a Task Token is received from the Step Function API. This token is essential; it's the green light our Lambda function needs to proceed.
In the workflow designer, the Lambda Function looks like this:
We've configured the input of the task token (see taskToken.$: $$.Task.Token), and we set the configuration Wait for callback, which pauses the whole workflow.
In CDK the Lambda Task looks like this:
Once a Step Function has the word forbidden in its title the Step Function will halt and look like this:
Our Lambda Function Ask For Manual Approval, standing by for a callback, also creates URLs for approving or rejecting the article in question.
These URLs, along with the details of the waiting Step Function, are stored in DynamoDB. The following item is stored in DynamoDB:
With this system in place, retrieving the list of pending reviews and their respective decision-making URLs becomes a breeze.
Decision Time: Approve or Reject?
When our Step Function waits, admins jump in. They head to a simple dashboard at the /admin route in our frontend. Here, they see articles waiting for a thumbs up or down, each with its link.
Clicking approve or reject sends their choice straight to a special function. This function tells the Step Function what to do next:
Once this API call is sent the Step Function continues and stores a boolean value for decision in the output of the Lambda function.
Recap & Resources
Step Functions are a great service in AWS to orchestrate and handle complex workflows. Using the Callback Pattern is one of the best use cases for Step Functions. While the input and output transformations of the data can be quite complex, it is a very good way to build observable workflows.
Key Takeaways:
Step Functions are your workflow orchestrators, ensuring each step of your process harmonizes with the next.
The Workflow Designer is a good way to get started, but you need IAC in the end.
Callback Patterns offer the flexibility to pause your workflows, awaiting human input or external triggers to proceed.
Through our Content Moderation System example, you've seen how these concepts apply in real-world scenarios, ensuring quality and control.
Go ahead, deploy the GitHub Repository to your AWS account, and change a few things. For example:
store the article in a DynamoDB with a direct integration
check the input for more words than just forbidden
send the user a notification based on WebSockets, Emails, or SMS
Resources to Explore: