Recently I got chance to solve an interesting as well as a challenging problem. We were working on a
Node.js / Typescript application build on top of
serverless framework. We were using aws
cloudFormation template to deploy the aws lambdas after manually packaging the build artifacts. It was all good when the codebase was small, the deployment was working fine. But as we started adding new features, the codebase started to grow exponentially. As a result , the size of the build artifact kept increasing and finally one day …… BOOM !!!!!!!
The deployment started failing because the size of the build exceeded the limit ( 250MB) AWS provides. We were nearing the first release of the application and it was a shocker for us.
By the time this issue occurred , our codebase already had over 100 files. Initially we thought we’ll have to restructure the entire codebase , rethink about the packages we have used.
Imagine you’ve to sit and restructure the codebase having 100+ files and find a light weighted alternative of each npm package being used…………… Yeah ! it was going to be painful. We needed something intelligent , quick and robust.
So this is how I managed to reduce the build size of the application from 300MB to 2–3 MB per lambda using webpack:
Webpack as you might already know , a module bundler which is primarily used in the UI frameworks i.e.
Angular/React etc. But Kudos to the creators, it is so configurable that it can be used anywhere.
Serverless provides a plugin called
serverless-webpack which rescued us.
Let’s go step by step and see how serverless can be configured to deploy the lambdas individually with drastically reduced size.
In this tutorial :
- Create a new repository and install required packages
- Create lambda handlers
- Deploy and test the lambdas
To start , open terminal , make a directory
serverless-webpack-app and go inside it.
npm init , it’ll ask you a set of questions , fill it according to your preference(or run with -y flag to have default values). Once you’re done , it will create a
package.json file in the root folder.
npm packages :
serverless globally :
npm i -g serverless
then add below packages:
npm install -D webpack typescript ts-loader serverless-webpack webpack-node-externals terser-webpack-plugin fork-ts-checker-webpack-plugin
once the installation is completed , install
npm install aws-sdk lodash @types/lodash
Now run following commands :
Create a folder
src and create 2 files with names
testLambda.tsinside it. and add following content inside it :
By this time , you have :
lambda handlers : actual code to be executed by lambda
package.json : It has all the required dependencies and scripts
webpack.config.js : Blank right now but we will add config later in this tutorial.
serverless.yml : Blank right now, will be updated by cloudFormation template
tsconfig.json : Blank right now, will be updated with typescript rules later.
Now let’s update the
typescript compilation rules , open
tsconfig.json and copy the below content :
tsconfig.json will be used by
webpack.config.js in the next step.
webpack.config.js and copy paste the below code :
lets understand what we did here :
Webpack expects a configuration file to understand how the app will be built. The configuration file can be in
in the module we are using
ts-loader to transpile the typescript. You need to write rules based on the type of file you have in your project. for example if you have
css/scss files , you need an appropriate loader to resolve such file and get your code compiled successfully. ( see loaders ).
node-externals library enables the third party dependencies to be installed separately and not included in the build file. You can avoid it if you want the the third party libraries to be included in the build file. ( I like to keep it separate).
TerserPlugin is one of the most popular code minification library. We are using this to minify the code.
ForkTsCheckerWebpackPlugin is used for Typescript’s type-checking and linting.
Next step is to setup
serverless.yml file to get the application deployed. open the
serverless.yml file and paste copy paste the content below :
serverless.yml contains the usual template for lambdas and other resources you need to deploy the application ( serverless starter ).
Apart from the followings :
this option enables the individual packaging of the lambda. When this option is set to
true , serverless will package the lambdas individually. You can set it to
false if you want a single package for all the lambdas deployment ( not recommended).
This plugin enables serverless to package the lambda using
webpack . You need to provide webpack configuration as custom property.
We are passing the configuration for the
serverless-webpack plugin. You can explicitly exclude and include third party libraries based on your requirement. Since
aws-sdk is already provided in the lambda environment by AWS , we don’t need to package it in the build hence it’s been excluded forcefully.
Now we are almost done. If you haven’t configured
aws-cli on your machine you need to configure it to be able to successfully deploy the application. Alternatively you can export your secret key as following :
serverless config credentials --provider aws --key <Access Key ID> --secret <Secret Access Key>
once this is done, run the following command in the terminal to deploy the app :
sls deploy --config serverless.yml
congratulations !!!! your serverless-webpack app has been deployed successfully.
To invoke use the following command :
sls invoke -f testLambda -l
Now to remove the stack , run the following command :
So after 2–3 weeks of rigorous efforts finally we were able to deploy our application using webpack. This not only reduced the deployment package size but also helped us catching typescript errors.
That’s it for this time, I hope it was useful for you. Give a clap if you liked it. Suggestions are always welcomed!
Here is the complete source code :
In the next blog I’m going to explain How I streamed a zip file having multiple csv files with more than 1 million records into kafka producer.
Stay tuned :)