JavaScript CI Simple Start

 JavaScript Project with hosted CI

Continuous Integration is the practice of merging all developer working copies to a shared mainline several times a day.

Developers tend to spend much time setting up the personalised working environment in a way that makes each one comfortable and productive. Some automation scripts may save ones time to perform routine tasks. These can be, running tests, compiling and copying files, sending out email, generating testing coverage reports and so on. Despite of the diversity of the programming languages and the deployment processes, all the tasks above can be centralised to a remote server. While each one can use their preferred working setup locally, the updated software can flow through the steps of the release process.

Depending on the software the team is developing, it can be a pain to deploy, update and rollback. These operations are required not only for the in-house development and testing environment but also for production. Each step needs to be carefully managed to achieve minimum or no down-time. This is not only time consuming but also stressful across the teams.

For example, let's walk through a typical process to identify the ways it can be improved. A team is working on a javaScript REST API. A new feature comes in and each developer picks up a small task and creates a git branch of master. They work locally, run unit tests, and review each other's code. Now the code needs to be tested more.

The Quality Assurance (QA) team needs this update deployed somewhere to start running the tests that they have prepared. If something is wrong, the developers need to rollback their local deployment, fix the problem, and then re-deploy to  QA's server. These steps can happen a few times. Each time a developer needs to find some time to stop what they are doing and update the deployment for QA.

Once everything is passing, the update needs to be deployed to a pre-production environment to test the integration with other services. Load tests are usually added in this step. The ping-pong between developers and QA continues here for fixing bugs or performance-downgrading issues.

At this point, we can identify three different environments. Let's describe a few custom scripts that can save some time to the developers.

• Basic configuration and running the unit tests for each commit or code review request.
• Review the commit messages and make a changelog file with description of what this feature includes.
• Cleanup each environment to make sure a clean setup will run not affecting the new version.
• Configure and deploy for mock-tests in QA servers.
• Get a report of the tests and notify the teams when something is failing.
• Configure with different settings for the integration and load tests.
• Create new reports and notify the teams for performance and test failures.

Depending on the flow and the principles a team has set the above, more things may be automated with scripts. For instance, to make it easier to gather information for the changelog, a commit should not be allowed without a comment in a specific format (git hooks)

Already we can see the how much time and effort we can save using a tool to take care of all the above custom scripts for automation. But wait, don't start working on this yet. Before moving to analysing the next steps towards production, let's see in more detail, what happens in the pre-production environment. 

The pre-production environment can be expensive not only for the resources, but also in the aspect of configuration. A smaller suit of tests should also run across all layers to make sure our application can scale with no problems. We will need to configure this server to run behind a load balancer and in clusters. Every deployment comes with configuration settings that the developer must not mix, a small mistake (hopefully not including the production environments) will lead to extra time spent to investigate and find the misaligned configuration setting. In the best case that this didn't break anything but always in manual configurations there will be a mistrust. To minimise that, we need to keep consistency by using automation scripts.

Further down the stack we need to consider if we need a separate production deployment per customer, or group of customers. In this case we need to keep track of the cluster of servers we deployed the new version and even more different configuration settings. We might want to distribute an update to some of the clusters or even start divergent branches per cluster. It can get really messy even for the most organised people and result in hours of downtime for simple mistakes if not even data loss. The risks in this series of deployments should make you reconsider adapting more features of Continuous Integration.

Similar to writing a software, you need to setup, maintain and improve the CI system.

Let's walk through a simple end-to-end setup of a JavaScript project with CI. For the REST API we are using NodeJS, and Firebase for persistence. You can find the complete example project on GitHub as Indexaki.

Key components:

• NodeJS (Server side REST API)
• Firebase (NoSQL Database)
• GitHub (version control and the source repository for our CI)
• CircleCI (The orchestrator to run tests, deliver reports, and trigger the deployment)
• CodeClimate (Automated basic review with reports including test coverage)
• WebPack (bundler to generate the final files for deployment)
• Heroku (Build, Deploy and Host)

The Flow:

• Code gets updated in your local IDE and pushed to GitHub 
• CircleCI grabs the code and runs the tests 
• If all the tests are passing in CircleCI, it pushes the test coverage report to CodeClimate 
• In the meantime, CodeClimate generates a code review report 
• CircleCI then triggers the deployment in Heroku. 
• Heroku grabs the sources and builds the project. 
• Heroku deploys the project live in a self-hosted environment with a friendly url

Picking the "right" tools

Which are the right tools after all? We have been overwhelmed in regards of the options we are given. Simple questions flooding our heads from the very beginning, like; 

• What should I use for version control? 
• How about deployment, 
• where should I host the project? 
• I should not forget about automated tests, 
• some of them might need mock services. 
• A testing environment would be ideal before publishing in production. 
• Is my solution scalable enough? 
• Will I be able to trigger load tests and get reports in one place nice and easily? 
• How about all the third party services / APIs I am consuming. 
• Oh, wait a minute, I will have secret API keys to configure, what about those and how will I keep them secret... 
• What if s**t hits the fan... I should be able to quickly roll back to a version I know it was working.

What if my choice will not be the right one... how .. where.. but if... and what about... okay let's find another similar project and copy as much as possible .. I am sure my friend mentioned that cool platform, I should find something that includes that one... but again ... that thing, is different will it be okay to tweak here and there? probably I can make it work as I want... a little workaround here, another small hacky script there... It should work like a charm in the end.

...until it piles up with "tech-dept" Jenga bricks.

Where do I start

Your brain has to work with exactly the same pattern as it does when you develop your platform. First you think your requirements, keeping in mind how the project will scale. Draft a quick flow that would work, and start thinking of the bottlenecks. What would happen if you want to scale it, how many teams are going to contribute. How would you map the current manual process to the automation. Which of the steps can be done first without blocking the current flow, while you can benefit from progressive changes. Please don't forget to document everything in one place. You will need to come back in the future and tweak the flow for optimisation, extensions and debugging.

Step 1.

First you need to think if any of the components need a specific operating system. It will affect your options for the CI tools. For instance you might need to build a Visual Studio project, or Xcode.

Start by creating the git project locally and continue with initialising the build tool you might want to use if you are going to work with a framework. For this example we are using simple JavaScript and WebPack to bundle the project. 

Step 2.

Publish your git repository on a host, and don't forget about the suggested files/folders to ignore. Usually GitHub or Bitbucket is a good choice, since they provide many integrations between other tools you might already be using. This makes your life a bit easier, as you won't need to find your way with more 'custom' solutions using specific SSH keys and working with webhooks and access restrictions.

Step 3.

Start building your project folder structure. Follow any best practices you can find regarding your framework, or research around and find a structure that makes sense to you. You can modify it later if you find the need. Remember that you need a simple structure. Split your project in many repositories in case you need to develop many services / tools.

Step 4.

Create your sample scripts, write some tests for them and hook npm to trigger the tests. For a test suite you can start with mocha, and you will find information easily how to run the tests with npm and how to generate test and coverage reports.

Let the game begin

Selecting the tools you need for the CI, is not critical at this stage to decide which are the best. A quick research if the tool you are about to pick supports your main tasks and most (or all) the integrations you will need (between other CI tools, or communication channels, e.g. Slack).

The key point is to understand the flow, and configurations you will need to do. After your first approach you will get enough insights to figure out how to migrate to another tool that might benefit you more. 

Before deciding to migrate the whole project to another flow you will get the chance to extend the initial setup with custom scripts that you can run in deployment steps. All the tools I have tried so far, support customisation at that level.

For this project we are going to use free accounts that are more than enough, and we will go end-to-end solving small problems to achieve a deployment, test and reporting pipeline.

Create account to third party tools you want to use. In this example we use:

• Firebase
• Codeclimate
• CircleCI
• Heroku

Don't start panicking with all the integration options you will see around. Also don't start randomly enabling and modifying them. Again, for your brains is like coding your platform. Imagine that you just downloaded your libraries, though it doesn't mean that you need to start initialising all the objects they provide.

Main Structure

The project consists of the following files/folders:

.
+-- .vscode                // configuration for vscode editor
+-- libs                   
|   +-- models
|       +-- document.js
|   +-- storage.js
|   +-- utils.js
+-- public                 // dummy html sample client
|   +-- index.html
+-- web                    // helper js and css files for the dummy client
|   +-- libs
|       +-- index.js
|       +-- main.css
+-- test                   // mocha test files
|   +-- test-api.js
|   +-- test-gif.js
|   +-- mocha.opts         // in-file configuration for mocha
+-- indexaki.js            // the main application file
+-- app.json               // Heroku auto generated configuration
+-- circle.yml             // CircleCI configuration
+-- .gitignore             // git to ignore files
.
.

NodeJS rest API

The main application runs with node and restify module. In this case we configured the rest API to serve the endpoints that will help us manage stored documents, and a static HTML page fore the sample client.

We need to configure the port to listen based on an environment variable (will be set from the deployment tool). Adding a fallback port for convenience help us run the project locally in any workstation with not much configuration.

server.listen(process.env.PORT || 8080)

We will need a few libraries to help us out with the integrations. These are described below or you can explore the main node file package.json. 

Firebase

Start with a separate module for Storage. This way you will be able to update / replace it in the future without changing other places than configuration and the Storage module.

For accessing the Firebase, we will need to configure some variables that won't be ideal to keep in a public repository for security reasons. As before, those need to be set in the environment variables. You will notice a fallback databaseURL, this can be used to communicate with the local Firebase development tool which will allow you to run manual and integration tests.

const Storage = function() {
    this.config = {
        apiKey: process.env.FB_APIKEY,
        authDomain: process.env.FB_AUTHDOMAIN,
        databaseURL: process.env.FB_DBURL || "ws://localhost.firebaseio.test:5000",
        storageBucket: process.env.FB_BUCKET
    };
    firebase.initializeApp(this.config)}

Following the Firebase documentation for the nodeJS library we built the corresponding wrapper functions to save/get/delete our documents

CodeClimate

Setting up Codeclimate is pretty straight forward. From the Dashboard, you can follow the basic instructions to give access for your repo in GitHub.
Under Setting you will find the Integrations menu, and you can enable those that can be useful for you. One at this stage could be the GitHub Pull Requests. This will allow the reports to run for every Pull Request automatically in your GitHub repo review page
You will also find the sub-menu Test reporting, in this tab, take a note of the Test Reporter ID

CircleCI

Follow the steps in CircleCI to enable access to your GitHub repo. You can add a project according to the instructions in the platform. Then Select Settings -> Build Settings -> Environment Variables

Add a new variable CODECLIMATE_REPO_TOKEN and set as value the Test Reporter ID from CodeClimate

The build configuration takes place in circle.yml file. You will need the Heroku project Configuration here, so we will come back to this later.

We will configure this deployment process to generate coverage reports when the tests are running, post them to CircleCI and then deploy to Heroku. For this step you will need to add to your project the reporter library:
npm i code-climate-test-reporter

You will need to link your Heroku project and also give access to CircleCI

Heroku

Following Heroku's guidelines and startup steps, you will create a deployment pipeline and link your project in a new app. Again you can select integrations with GitHub to allow Heroku to pickup the deployment files.

In the pipeline you can choose to have more than one environments. You can use this to setup another branch if you want for instance to have a QA environment from your Development branch, or if you want to deploy to a test server your master branch and manually push to the live server after you finish testing.

You can select the deployment options you wish, be aware to review the build steps after you complete all the configurations and follow your project's flow. For instance, you don't want Heroku to automatically pick the changes from your selected branch and overwrite the deployment from CircleCI once the tests have finished. You can run the tests in Heroku if you want to enable this option but be aware to read notifications as some CI tools start charging after some usage, or they can have some other limitations.

At the first steps we left out some environment variables we need to communicate with Firebase. Now it's the right time to configure those in the host environment with Heroku.
Select the pipeline -> your application -> Settings tab -> Config Variables

Set the values that you noted down from your Firebase setup. These should match the keys your node script is expecting  to read: FB_APIKEY, FB_AUTHDOMAIN, FB_BUCKET, FB_DBURL

Deployment configuration circle.yml

• Running the tests

If you have a different trigger than npm test to run your tests, you can use test.override.Here we use npm run coverage in package.json
We selected to use istanbul coverage over mocha, and this is how we trigger the tests:
"coverage": "./node_modules/.bin/istanbul cover _mocha"

• Submit the coverage report

Using the codeclimate-test-reporter we feed the coverage results to CodeClimate.
This is using the CODECLIMATE_REPO_TOKEN environment variable we set earlier.

• Development / Test deployment (optional)

You can trigger the deployment to a test environment by selecting the develop branch and the corresponding heroku appname. BEWARE you will need to use the right environment variables there not to match the same Firebase configuration. You will also need to make sure you will get the right coverage reports per branch.

• Deploy

You will need now to configure the deployment step. Give the name of your Heroku environment, in this case it was Production. Add the branch you want to deploy and the appname you have selected. You can configure CircleCI to be triggered only (or not) with specific git tags, more details are described.

	test:
	override:
1	- npm run coverage
	post:
2	- ./node_modules/.bin/codeclimate-test-reporter < ./coverage/lcov.info
	deployment:
	development:
3	branch: develop
	heroku:
	appname: betaindexaki
4	production:
	tag: /v[0-9]+(\.[0-9]+)*/
	owner: kapekost
	branch: master
	heroku:
	appname: indexaki

Takeaways

Deployment process can be a pain and needs to be designed in a flexible way to accompany the development of the software. Starting with automating the simple parts, we get the needed experience to keep extending and evolving the Continues Integration. You don't need to design all the steps/tools upfront but you need to know where you're heading and how you can improve the steps in the near future to delivery easier and more controllable. In this example we picked some free tools and hosting options to deliver a simple project from the GitHub repository to Heroku servers.
The configuration is maintainable, and allows improvements in the future. Environment Variables hold the secret keys to avoid exposure to publicly accessible files while the setup can easily be transferred to other tools and servers.

ENJOY ! :)