notification-system.svg

Case Study

Webhooks

### Problem

The possibility to send a high number of emails without keeping your own server for email delivery.
Goals and challenges
- Track user interaction with the email (up to 10K notifications per minute)
- Reduce load on 3rd party app that sends high number of emails daily
- Webhook receiver’s downtime shouldn’t affect main application flow
- System has to be scalable and guarantee to deliver within one day, when hook target isn’t available
- Convenient and timely email delivery status reporting and displaying this info on a dashboard
- Service is easy to use and customizable to meet changing requirements
- Availability and comprehensiveness of documentation

### How WE helped

We've implemented the hooks delivery system with Amazon Simple Notification Service (SNS) and proxy server that processes, subscribes, and redirects requests. Little coding was required and responsibility of webhook delivery is assigned to an external service.

### Results

- Email delivery system with the user’s activity tracking feature that processes a high number of events (up to 20K notifications per minute)
- Information about user interaction with the email is captured and stored
- Redelivery solution is available when webhook receiver is unreachable
- System is separated and does not affect main app services
- Solution is easy to scale
- 2 times higher level of throughput'

### Implementation process

To achieve project objectives, we’ve considered several implementation options:
- Naive implementation
- Implementation via SQS queue
- Webhooks delivery SNS notification

We have the following components that are responsible for events in our system:
1. Event consumer
2. Pull queue and decide what we need to do with events.
3. Event storage
4. Store all events in our system.
5. 3rd party app
6. Have to be informed about events related to the app account.

### Native implementation

The simple way to inform a 3rd party API is just to make HTTP request the hook URL directly from the consumer, as displayed on the following diagram:

Pros:
- Easy to implement

Cons:
- Main application flow is affected, thus if a 3rd party app won't respond, we will receive an error in our internal consumer
- Сonsumer cannot be scaled separately from webhooks which are sent to 3rd party app
- Impossible to implement webhooks redelivery correctly

Accordingly, we came to the conclusion that this approach will not give us expected results or satisfy our needs in a comprehensive manner.

Therefore, we’ve started to consider other options.

![text](/src/case_studies/notification_system/diagram_1.png)
          
### Implementation via SQS Queue

We use a lot of AWS services at the moment and already have a bunch of SQS Messaging Queues. The main idea for the solution it is to use a separate queue for hooks (webhooks) and consumer to process this queue. Take a look at the schema that shows how SQS queue works:
In this manner our main app flow does not depend on the working state of an external app and we are able to implement retries via SQS queue.

The flow of a retry is as follows:
1. Pull webhooks events from queue
2. Send HTTP request with events in the body to 3rd party app
3. If request is completed successfully, then remove message from queue.
4. If request has failed, retry with the specified delay and send a message once again.
5. Set maximum receive count that equals delivery retry count'

![diagram_2](/src/case_studies/notification_system/diagram_2.png)


The flow is displayed on diagram below:
Pros:
- Main application flow is not affected
- Redelivery can be easily implemented
- Independent main event consumer can be scaled

Cons:
- More complex to implement
- Max redelivery delay is 15 minutes (since max delay is 15 minutes in SQS queues), thus we can’t make a logarithmic retry for our messages
- Logic of retries and responsibility for webhook delivery has to be managed manually

This option was way better, however, not all of our primary goals would be met by implementation via SQS queue.'

![diagram_3](/src/case_studies/notification_system/diagram_3.png)

### Webhooks delivery by SNS Notification

Another way to implement webhooks delivery is by using AWS SNS. The service was developed for delivering notifications and can send them via HTTP, the same way as webhooks do. Also, if notification delivery has failed, SNS is responsible for retry delivery, and we can use linear or logarithmic function to manage retry frequency.

This solution will give us desirable outcomes; however, it cannot be applied due to these two reasons:

1. SNS send confirmation requests to subscribe endpoint (webhooks destination)
2. SNS provides redundant metadata to delivered messages

The SNS was designed for communication between different components within one system and not for communication among different components of different systems with different owners. Hence, we need a proxy component inside our system that SNS can communicate with

Let's take a look at the scheme of proposed webhooks implementation using SNS:

![diagram_4](/src/case_studies/notification_system/diagram_4.png)

Flow of hooks delivery using SNS:

1. The consumer sends events to a certain SNS topic.
2. We subscribe a 3rd party app to the topic and the URL should contain an endpoint that we need a redirect for (http://proxy.example.com/http%3A%2F%2Fwebhook.endoint%2Ftarget)
P.S.: We’ve encoded the URL with Base64 instead of simple URL encoding.
3. When the proxy receives a request for subscription confirmation, it will send a request to SNS with a proper confirmation token.
4. When the proxy receives a request with a notification (an event, in our case), it will format event data and redirect to a URL that contains the URL path part.
5. All responses from the hooks’ endpoints pass as responses to SNS requests.

Schema of proxy API is shown below:

Pros:

 - Main application flow isn’t affected by the state of the hooks’ receiver application
- The responsibility for message delivery is shifted to SNS.
- Redelivery strategy is already implemented by AWS.
- The simple proxy has good throughput, very lightweight and can be scaled.
- Little code is needed (less code, fewer bugs)

Cons:
- The proxy has to be developed.
        
![text](/src/case_studies/notification_system/diagram_5.png)

### Solution

After considering each implementation option in terms of benefit it will bring, and required resources for its execution, we’ve chosen to go with the webhooks delivery by SNS notification. With SNS and the simple proxy server, we’ve implemented the hooks delivery system with minimum coding involved and shifted responsibility of webhook delivery to an external service. As a result, we are able to send about 20,000 events per minute without any scaling, which is even higher compared to the planned 10,000 events per minute. The major bottleneck is the proxy, but we designed this one for scaling so throughput can be easily increased.

### Additional info

We wrote our proxy using Node.js platform. The proxy hosted on EC2 T2 small instance and can process about 4,000 concurrent requests with throughput of about 400 req/sec. The proxy doesn’t have any interaction with the database since all data for redirects are passed in URLs. This gives us an option to do a horizontal scaling. If we place our proxy behind a load balancer, then we will be able to add more server instances and as a result obtain two times higher throughput.

Vertical scaling won’t give us the same effect. When we scale node processes within one server instance, we can get about 30% higher throughput for each additional processor kernel. Therefore, horizontal scaling is a more suitable and effective approach.

We don’t need to worry about scaling SNS because AWS does it automatically, we only have to send the proper count of messages.

Creating webhooks notification system

scheduled-emails.svg

### Why?

Our main target is to develop a system for sending large numbers of emails and then tracking users’ interaction with a received mail (e.g. opens, link clicks, unsubscribes, etc.) Users with large amounts of daily sends prefer use scheduling for sending use.

### Why we need scheduling

Using scheduling, instead of send-immediately flows, gives us several advantages.
- Better control for sending process
- If a user needs to send emails after his workday is finished, then he won’t be able to monitor the sending process and fix them if something goes wrong. With scheduling, users can prepare all emails for sending before the actual send date, and our ESP (Email Service Provider) handles the rest.Less resources required
- If you already have content for emails that you need to send (for example tomorrow), you are able to send the emails gradually. For example, if you need to send 5M emails at 8 PM you can start sending at 10 AM and continue sending in small batches the whole day. We collect the emails from our side and cover the delivery in time.No personal CRON.

If you have a notification email that needs to be sent after some time (for example, a subscription reminder), just send the notification to our ESP and we cover the scheduling. You don’t need to handle that case with personal Cron scripts.

### Problems that we faced when building scheduling

We propose pretty simple API for working with scheduling, but it’s not easy to organize simultaneously sending even 1M emails. We needed to think about scalability and our system in the future. We had to solve several problems.

- Proper storage for scheduled emails
- Limited throughput of the network for one process
- Split scheduled emails that were ready to send for different workers
- Syncing statuses of scheduled emails and avoiding duplication on send

### Proper storage for scheduled emails

We should able to filter “ready for send” emails as fast as possible among large amounts of records that don’t need to send yet. At the time we already used ClickHouse database as storage for our analytics events (unsubscribe, clicks, open, etc.).

Clickhouse is a column-oriented database that is used mostly for analytics. But with the read/write throughput that ClickHouse provides for us we are able to use the solution as scheduling storage.

The most valuable storage requirements for us were the following:
- Local and distributed joins
- Data compression
- Parallel and distributed query execution
- Cross-datacenter replication
- High availability
- SQL support
- Fetch speed not less than 500K email records per second.

### Performance testing

We made some performance tests and got results that completely satisfied our requirements:

- Read speed is 1.2 GB/s
- Write speed is 50 to 200 MB/s
 - Finding 335K scheduled emails on a dataset with the size of 5M takes 0.061 second

As a result, with network latency for transfers, we were able to fetch 3.5M emails per second. That was a very good result for us.

### Limited throughput of the network for one process

We resolved the problem of choosing our data storage and then faced another issue. We used NodeJS as our main platform, the platform, together with SQS queues, didn’t provide the desired performance for us. Then we wrote simple prototypes for testing using Golang and Elixir (we hoped that Erlang with its lightweight threads helps us) we got some throughput increases, but still weren’t reaching the desired results and scalable solutions.

Results of our benchmark’s queue inserting time for 50K messages to queue was as follows:
- NodeJS 1 thread = 95.408 sec
- NodeJS 2 thread = 75.757 sec
- Go = 35.564 sec

Elixir was about 2 times slower than NodeJS with 2 threads and the same kind of problem.

So obviously, we needed to find ways to scale horizontally.

   
### Split scheduled emails that are ready to send for different workers

Obviously, we needed to make our scheduling scalable and found a way to split all scheduled emails for workers. Workers should have the same role and know nothing about each other. It’s allowing us to run more workers on more servers when we need to process the largest amount of emails.

When workers are fully independent from each other, we decided to create one more component in our system.

The component is named Ranger and is responsible for splitting ranges for our workers. Each range shouldn’t intersect with other ranges.

With Ranger and Clickhouse working together, we created the following architecture for our scheduling feature:
1. Emails send to REST API, and if a parameter with a future send date exists, API puts the email to Messaging Queue
2. Write to the consumer processing the Queue and save all emails inside Clickhouse database
3. Ranger runs query for fetching emails and the query contains the following steps:
		1. Records filtered by send date and each time we fetch all records that are ready in that moment
		2. Filter records by time-created range, we need only those that weren’t affected by the previous step. For that, we fetch only records that have a date greater than biggest date from previous execution
		3. Aggregate results by CreatedTime and COUNT() function
4. Set first fetched date as DateFrom for query
5. Wait for full batch size and set last date as DateTo
6. Update Date to as last processed range
7. Send range to Queue for processing by worker
8. Consumer pulls a Queue with ranges and fetches by CreatedTime emails for sending to user
9. Each email sends to target user

        
![diagram_1](/src/case_studies/scheduled_emails/diagram_1.png)


### Avoid duplication

AWS SQS service promises not less than one delivery for a Standard Queue. this means that sometimes we can obtain one message on two workers and send our email twice. Since each message can contain multiple emails (our current batch size contains 5K) we need to organize duplicate handling.

For that, we have the Redis database and we store each SQS messageID as a Redis key. The keys have an expiration date that's equal to message visibility timeout in the Queue. When a consumer obtains a new message from SQS, he checks for an existing messageID in Redis. If the message already exists, the consumer skips this one.

### Conclusion

Using ClickHouse and AWS Simple Queue Service, we build a scalable scheduling system for delivering emails. These flows and tools can be applied to any actions that could be scheduled, not just for emails.

At the moment, we can fetch 335K records per second, filtering by ready date takes 0.1 seconds for 5M database set (including network latency), and workers that process the emails can easily scale horizontally for the desired throughput.


We create our story every moment

Creating webhooks notification system

How to send large amount of scheduled emails simultaneously