Skip to main content

How to auto scale Kinesis stream

We have used AWS Kinesis stream in nodejs project.
It is very easy to get started with Kinesis.
You can setup Kinesis to handle any amount of traffic.
For every queue you  create stream in Kinesis.
Each stream has many shards.
One shard can handle 1000 req/sec.
To handle more traffic you can create many shards.
Every shard cost money and unused shards are waste of money.
Only problem with Kinesis is that there are not automatic way where AWS itself create and destroy shards as per current traffic.
To handle this scenario we have created one nodejs module which continuously monitor traffic and create/destroy shards.
I have proposed that since this is very common requirement we can contribute this code to open source.
We have used many open source framework and this is one of the way to give something back to community.
With permission from client i have created nodejs module and publish it on github and npm.
You can find source code at https://github.com/xaxis-open-source/amazon-kinesis-scaling.
You can find npm module at https://www.npmjs.com/package/@xaxis-open-source/amazon-kinesis-scaling.
Labels:

Comments

Post a Comment

Popular posts from this blog

Almond Automation | Codeless Web Automation Tool

I worked on many UI projects and felt the need for automation framework which can be easily learn and use by anyone(developer and qa). Manual testing is very error prone and tedious. If your project involves lots of pages and component and has complex interaction then it is very frustrating to do manual testing to make sure that everything works and new features are not breaking any old functionality. I created  Almond Automation  to create automation for UI projects using selenium grid. Almond Automation provides following feature Simple UI to create and run testcase. You can manage multiple projects from single UI. You can run testcase on any environment(local, qa, integration etc). Create TestSuite to group testcases and execute complete testsuite. Support for modular testcase. No need to change many testcases because of change in one field etc. You can also pass runtime values while running testcase. Verify execution of testcase easily in UI. View screenshot of e...
Post a Comment
Read more

Access kafka on docker from host machine

Access kafka on docker from host machine Problem You can easily create kafka container using docker. We are using docker compose to create kafka, storm container. This setup is working fine but we cannot send any message from host machine to kafka running on docker container. All ports are currently mapped to host machine. I can ssh to docker container and send message but i was not able to send any message from my host. It is very irritating as i cannot use application running in Intellij to send any message to kafka on docker. Solution When i use docker compose to create docker container i use following configuration kafka: image: docker image ports: - "2181:2181" - "9092:9092" - "9000:9000" environment: KAFKA_CREATE_TOPICS: topic1 ADVERTISED_HOST: kafka Notice that we set ADVERTISER_HOST as kafka. Now open your host file and put following entries(this depends on environment linux/w...
Post a Comment
Read more