# Build a robust CI process for your applications

In the world of devops, there is a lot of practices or standards that are everywhere, one of them you may already heard of is “CI/CD” which stands for Continuous Integration and Continuous Delivery.

In this case I’m only going to cover the CI one, but with a lot of things, let’s start and see what we can do :)

Disclaimer: I’m going to use Github as platform to explain most of the things, but concepts are general and can be applied anywhere.

### Testing

Here we have a lot of categories, depending on your need you could add

* Unit
    
* Integration
    
* End to end
    
* Performance
    

To read more about each one of these types (and more) I suggest you read about them here

[https://www.geeksforgeeks.org/types-software-testing/](https://www.geeksforgeeks.org/types-software-testing/)

[https://martinfowler.com/articles/practical-test-pyramid.html](https://martinfowler.com/articles/practical-test-pyramid.html)

Now let’s show a basic example on github actions

```yaml
name: Docker Publish

on:
  workflow_dispatch:
  pull_request:
    branches: [ "develop", "master" ]
    paths:
      - 'src/**'
      - 'infra/docker/**'
      - 'infra/kubernetes/**'
      - '.github/workflows/publish.yml'
      - 'tests/**'
  push:
    branches: [ "develop", "master" ]
    paths:
      - 'src/**'
      - 'infra/docker/**'
      - 'infra/kubernetes/**'
      - '.github/workflows/publish.yml'
      - 'tests/**'
  
env:
  BRANCH_NAME: ${{ github.ref_name }}
  APP_NAME: infobae_api
  APP_VERSION: latest
  APP_DEV_VERSION: unstable
  AWS_ECR_REGISTRY: ${{ secrets.AWS_ECR_REGISTRY }}

jobs:

  test:
    permissions: 
      contents: read

    name: test
    runs-on: ubuntu-latest
    steps:

      - name: Checkout
        uses: actions/checkout@85e6279cec87321a52edac9c87bce653a07cf6c2

      - name: Set up Go
        uses: actions/setup-go@5a083d0e9a84784eb32078397cf5459adecb4c40
        with:
          go-version: 1.23.2

      - name: Test
        run: go test -v ./tests
```

Here I’m running a suite of tests each time there is a pull request at the develop/master branch, or at each push to develop/master

This is running these tests

[https://github.com/jd-apprentice/infobae-api/blob/master/tests/main\_test.go](https://github.com/jd-apprentice/infobae-api/blob/master/tests/main_test.go)

A larger collection of tests can be seen here

[https://github.com/jd-apprentice/waifuland-api/tree/master/**tests**](https://github.com/jd-apprentice/waifuland-api/tree/master/tests)

### Audit

Now here it depends on the language or platform but we can use/see a few things, for example if we are using `nodejs` we can see a command like `npm audit` which may not be the best ([https://overreacted.io/npm-audit-broken-by-design/](https://overreacted.io/npm-audit-broken-by-design/)) but is better than nothing in some cases.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742698452295/00add925-7b2e-4ccd-bb3e-96d2f73f3ff7.png align="center")

In the case of github we can also use dependabot

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742698540725/5ae0f5ad-3571-446b-9e65-7ea4f4c1ca76.png align="center")

We can use the manual one or automatic one

### Sast

Sast comes from [https://en.wikipedia.org/wiki/Static\_application\_security\_testing](https://en.wikipedia.org/wiki/Static_application_security_testing)

Same as `DAST` there is a LOT of tools so I’m only going to cover you, is up to you to investigate which one fits more your business needs.

A popular one could be [https://snyk.io/](https://snyk.io/)

We can login with github and add a new project there

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742699107938/4f602ef4-fbb6-4bf4-ab68-85b0fbb256c2.png align="center")

I’m going to use github to add the project

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742699142988/3d28ed86-485a-4a5a-8ac9-95109c13ec73.png align="center")

Once repository is selected

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742699273793/9b96f8e3-ca5b-4553-8632-49033abb39e2.png align="center")

We should see our project there (I know it’s destroyed, I just started with this one sob sob)

It’s useful to set automatic fix pull request to C/H

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742699425935/64b48c98-bd21-460d-a674-63795b1f1c59.png align="center")

It can be done in the Github integration section for the project itself.

We can use their extension to be able to see things earlier

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742699629829/5780ea16-02ba-4b32-9892-85624105d947.png align="center")

Connect to your account

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742699648234/98130a9b-bad9-4ea6-875f-2d47987a28bc.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742699664072/82f50d05-a130-4327-842a-d4ef0cdc4188.png align="center")

Now let’s say we want to add the scan process into our pipeline, with github we can find in the marketplace the github action for it

```yaml
# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.

# A sample workflow which sets up Snyk to analyze the full Snyk platform (Snyk Open Source, Snyk Code,
# Snyk Container and Snyk Infrastructure as Code)
# The setup installs the Snyk CLI - for more details on the possible commands
# check https://docs.snyk.io/snyk-cli/cli-reference
# The results of Snyk Code are then uploaded to GitHub Security Code Scanning
#
# In order to use the Snyk Action you will need to have a Snyk API token.
# More details in https://github.com/snyk/actions#getting-your-snyk-token
# or you can signup for free at https://snyk.io/login
#
# For more examples, including how to limit scans to only high-severity issues
# and fail PR checks, see https://github.com/snyk/actions/

name: Snyk Security

on:
  push:
    branches: ["master" ]
  pull_request:
    branches: ["master"]

permissions:
  contents: read

jobs:
  snyk:
    permissions:
      contents: read # for actions/checkout to fetch code
      security-events: write # for github/codeql-action/upload-sarif to upload SARIF results
      actions: read # only required for a private repository by github/codeql-action/upload-sarif to get the Action run status
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Snyk CLI to check for security issues
        # Snyk can be used to break the build when it detects security issues.
        # In this case we want to upload the SAST issues to GitHub Code Scanning
        uses: snyk/actions/setup@806182742461562b67788a64410098c9d9b96adb

        # For Snyk Open Source you must first set up the development environment for your application's dependencies
        # For example for Node
        #- uses: actions/setup-node@v4
        #  with:
        #    node-version: 20

        env:
          # This is where you will need to introduce the Snyk API token created with your Snyk account
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}

        # Runs Snyk Code (SAST) analysis and uploads result into GitHub.
        # Use || true to not fail the pipeline
      - name: Snyk Code test
        run: snyk code test --sarif > snyk-code.sarif # || true

        # Runs Snyk Open Source (SCA) analysis and uploads result to Snyk.
      - name: Snyk Open Source monitor
        run: snyk monitor --all-projects

        # Runs Snyk Infrastructure as Code (IaC) analysis and uploads result to Snyk.
        # Use || true to not fail the pipeline.
      - name: Snyk IaC test and report
        run: snyk iac test --report # || true

        # Build the docker image for testing
      - name: Build a Docker image
        run: docker build -t your/image-to-test .
        # Runs Snyk Container (Container and SCA) analysis and uploads result to Snyk.
      - name: Snyk Container monitor
        run: snyk container monitor your/image-to-test --file=Dockerfile

        # Push the Snyk Code results into GitHub Code Scanning tab
      - name: Upload result to GitHub Code Scanning
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: snyk-code.sarif
```

It looks like this, in my case I’m only going to use code scan so everything else if going to be deleted.

```yaml
# This workflow uses actions that are not certified by GitHub.
# They are provided by a third-party and are governed by
# separate terms of service, privacy policy, and support
# documentation.

# A sample workflow which sets up Snyk to analyze the full Snyk platform (Snyk Open Source, Snyk Code,
# Snyk Container and Snyk Infrastructure as Code)
# The setup installs the Snyk CLI - for more details on the possible commands
# check https://docs.snyk.io/snyk-cli/cli-reference
# The results of Snyk Code are then uploaded to GitHub Security Code Scanning
#
# In order to use the Snyk Action you will need to have a Snyk API token.
# More details in https://github.com/snyk/actions#getting-your-snyk-token
# or you can signup for free at https://snyk.io/login
#
# For more examples, including how to limit scans to only high-severity issues
# and fail PR checks, see https://github.com/snyk/actions/

name: Snyk Security

on:
  push:
    branches: ["master" ]
  pull_request:
    branches: ["master"]

permissions:
  contents: read

jobs:
  snyk:
    permissions:
      contents: read # for actions/checkout to fetch code
      security-events: write # for github/codeql-action/upload-sarif to upload SARIF results
      actions: read # only required for a private repository by github/codeql-action/upload-sarif to get the Action run status
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Set up Snyk CLI to check for security issues
        # Snyk can be used to break the build when it detects security issues.
        # In this case we want to upload the SAST issues to GitHub Code Scanning
        uses: snyk/actions/setup@806182742461562b67788a64410098c9d9b96adb

        # For Snyk Open Source you must first set up the development environment for your application's dependencies
        # For example for Node
        #- uses: actions/setup-node@v4
        #  with:
        #    node-version: 20

        env:
          # This is where you will need to introduce the Snyk API token created with your Snyk account
          SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}

        # Runs Snyk Code (SAST) analysis and uploads result into GitHub.
        # Use || true to not fail the pipeline
      - name: Snyk Code test
        run: snyk code test --sarif > snyk-code.sarif # || true

        # Push the Snyk Code results into GitHub Code Scanning tab
      - name: Upload result to GitHub Code Scanning
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: snyk-code.sarif
```

Remember to add the *SNYK\_TOKEN* to the secrets.

A simpler option could be CodeQL if you are using Github

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742700422157/293e79c9-9708-4286-9dd5-208e8484d431.png align="center")

### Dast

Disclaimer: DAST can affect the current system integrity y/o app, so add it at your own risk or make sure to have monitors for it.

Like mentioned before, for `DAST` there is a lot of things.

The one that I’m using right now is [https://github.com/marketplace/actions/zap-baseline-scan](https://github.com/marketplace/actions/zap-baseline-scan)

You can also use [https://github.com/marketplace/actions/zap-full-scan](https://github.com/marketplace/actions/zap-full-scan)

They give a full example on how to run it

```yaml
on: [push]

jobs:
  zap_scan:
    runs-on: ubuntu-latest
    name: Scan the webapplication
    steps:
      - name: Checkout
        uses: actions/checkout@v4
        with:
          ref: master
      - name: ZAP Scan
        uses: zaproxy/action-full-scan@v0.12.0
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          docker_name: 'ghcr.io/zaproxy/zaproxy:stable'
          target: 'https://www.zaproxy.org/'
          rules_file_name: '.zap/rules.tsv'
          cmd_options: '-a'
```

But here you can check out a lot of them

[https://www.acunetix.com/blog/web-security-zone/10-best-dast-tools/](https://www.acunetix.com/blog/web-security-zone/10-best-dast-tools/)

### Quality Gates

Our beloved Sonarqube could enter here (not the only one) but one reliant and good tool to cover this section since we can use it cloud or self hosted.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742764065771/d3e8e4b9-0e70-4037-925b-adce4389f7da.png align="center")

To add a project into sonarqube cloud, we need to have an organization

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742764090931/70c2fc5c-1ad4-4176-8bd1-123e2b6f03ad.png align="center")

Both creating an organization or adding a project could be found in the same place

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742764117706/f371896f-8c12-4add-9f3e-c8845792ad10.png align="center")

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742764139930/87c7e65a-1fc5-4352-a02e-b4c8ee2a1e8f.png align="center")

Once we select our organization, we can select a repository to import. Sonarqube is going to ask us about the method of scan on new code

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742764202448/541b8107-8ad2-4670-9dd2-35617c2a4df3.png align="center")

In my case since I’m not doing releases on my personal projects I’ve selected the second option

Also I’m using the automatic analysis (works quite well even tho sonar itself says not recommended)

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742764256314/09f9946f-8d06-43a8-b6ae-bcf33dd547fd.png align="center")

With this option enabled whenever you open a pull request it will leave a comment with the status of that new code

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742764295212/5e5e75dc-17ec-4e07-888f-ad55c8644912.png align="center")

From there we can follow and see the issues mark by sonarqube. You can also use their extension in VSCode to track things earlier before they appear on a PR

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742764374272/2558ebe1-fc96-4d13-9212-9a79ac0d583e.png align="center")

With this installed we could add a sonar server (self hosted) or the cloud one (my case)

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742764403157/e3cd29ff-a2db-4740-a956-70fd3486b627.png align="center")

We could also manually check the issues in their webpage if needed

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742764491490/eb070b9c-7ae5-4a9f-bda8-3720be70ead0.png align="center")

### Linter

In the linter section we should evaluate if this needs to run either at pre-commit level or CI one, since it can be tedious for the developers in some cases (mostly because of their gitflow)

Also depending on the lang you are working on you may need a different tool. In this example I’m going to use [https://golangci-lint.run/](https://golangci-lint.run/) in a golang project.

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742764701861/6c1535a2-b340-4450-8df2-81d323c96a31.png align="center")

In the root of my project I have a `Makefile` which I mostly use it to have shortcuts for commands (so I don’t have to remember them) or just to add a dependency (another command) previous to run that one.

In the case I’m highlighting here I’m running `path` before `lint`. To run this I have to type `make lint` and it will check these things (that are defined in my `.golangci.ymal`

```yaml
# yaml-language-server: $schema=https://golangci-lint.run/jsonschema/golangci.jsonschema.json
## Copied from https://github.com/microsoft/typescript-go
### https://golangci-lint.run/usage/linters/

run:
  allow-parallel-runners: true
  timeout: 180s

linters:
  disable-all: false
  enable-all: true
  disable:
    - tenv
    - godox
    - gci
    - gofumpt
    - godot
    - gofmt
    - wsl

linters-settings:
  depguard:
    # Rules to apply.
    #
    # Variables:
    # - File Variables
    #   Use an exclamation mark `!` to negate a variable.
    #   Example: `!$test` matches any file that is not a go test file.
    #
    #   `$all` - matches all go files
    #   `$test` - matches all go test files
    #
    # - Package Variables
    #
    #   `$gostd` - matches all of go's standard library (Pulled from `GOROOT`)
    #
    # Default (applies if no custom rules are defined): Only allow $gostd in all files.
    rules:
      # Name of a rule.
      main:
        # Defines package matching behavior. Available modes:
        # - `original`: allowed if it doesn't match the deny list and either matches the allow list or the allow list is empty.
        # - `strict`: allowed only if it matches the allow list and either doesn't match the deny list or the allow rule is more specific (longer) than the deny rule.
        # - `lax`: allowed if it doesn't match the deny list or the allow rule is more specific (longer) than the deny rule.
        # Default: "original"
        list-mode: lax
        # List of file globs that will match this list of settings to compare against.
        # Default: $all
        files:
          - "!**/*_a _file.go"
        # List of allowed packages.
        # Entries can be a variable (starting with $), a string prefix, or an exact match (if ending with $).
        # Default: []
        allow:
        - Scruticode/src/config
        - Scruticode/src/constants
        # List of packages that are not allowed.
        # Entries can be a variable (starting with $), a string prefix, or an exact match (if ending with $).
        # Default: []
        deny:
          - pkg: "math/rand$"
            desc: use math/rand/v2
          - pkg: "github.com/sirupsen/logrus"
            desc: not allowed
          - pkg: "github.com/pkg/errors"
            desc: Should be replaced by standard lib errors package

issues:
  max-issues-per-linter: 0
  max-same-issues: 0

  exclude:
    - '^could not import'
    - '^: #'
    - 'imported and not used$'
```

Since I’m using the `enable-all` rule everything is enabled and in the disable section I’m discarding some of them.

The full list of linters available for this tool are here [https://golangci-lint.run/usage/linters/](https://golangci-lint.run/usage/linters/) each one checks something different and can be customized

In my case in addition to `golangci` I’m using [https://pre-commit.com/](https://pre-commit.com/) that works as a general rule for pre-commit rules (hooks/wrappers for the ones in .git)

To start a pre-commit project we need to ofc installed, then once we have it available globally on our system we can

```makefile
pre-commit:
	pre-commit clean
	pre-commit install
	git add .pre-commit-config.yaml
```

We also need a `.pre-commit-config.yaml`

```yaml
## https://golangci-lint.run/usage/configuration/

repos:
  - repo: https://github.com/tekwizely/pre-commit-golang
    rev: v1.0.0-rc.1
    hooks:
      - id: go-imports ## go install golang.org/x/tools/cmd/goimports@latest
      - id: golangci-lint ## yay -S golangci-lint
        args: ["--fix"]
```

Here I’m saying which hooks I want to run on each commit, since most of the things are being part of my linter I only use these two.

New if the hooks run and I had something that won’t pass their validations it will be displayed like this

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742765231035/0ae26219-7491-416c-a99f-ac1171340a05.png align="center")

Like mentioned before, this could be run it `pre-commit` level or CI level. If we decided to run it at CI level we should need to install `golangci` at the pipeline. I’ll suggest using their binary

[https://golangci-lint.run/welcome/install/#binaries](https://golangci-lint.run/welcome/install/#binaries)

Then export the `golang` path with `export PATH="$PATH:$HOME/go/bin"` this will make the binary available for the shell that is running.

### Format

In the case of format I’m gonna explain using the JS ecosystem, one popular tool there is Prettier.

To install prettier we should follow their guide [https://prettier.io/docs/install/](https://prettier.io/docs/install/)

Also think this, if everyone in the team uses prettier, consider not using a extesion and have configuration at VSCode level and repository level, only use the one in the repository since is the one that follows your company needs.

One pretter.config.js could look like this

```javascript
export default {
    "arrowParens": "avoid",
    "bracketSpacing": true,
    "htmlWhitespaceSensitivity": "css",
    "insertPragma": false,
    "printWidth": 80,
    "proseWrap": "always",
    "quoteProps": "as-needed",
    "requirePragma": false,
    "semi": true,
    "singleQuote": true,
    "tabWidth": 2,
    "trailingComma": "all",
    "useTabs": false
};
```

And your `package.json` like this

```json
  "scripts": {
    "lint": "eslint ./src/**/*.ts",
    "lint:fix": "eslint ./src/**/*.ts --fix",
    "format": "prettier --check ./src/**/*.ts",
    "format:fix": "prettier --write ./src/**/*.ts",
  },
```

Now same here as the linter one, we could add with `husky` a step for pre-commit or added into the CI process.

For prettier in the github actions you could use [https://github.com/marketplace/actions/prettier-action](https://github.com/marketplace/actions/prettier-action)

Which contains an example

```yaml
name: Continuous Integration

# This action works with pull requests and pushes
on:
  pull_request:
  push:
    branches:
      - main

jobs:
  prettier:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v3
        with:
          # Make sure the actual branch is checked out when running on pull requests
          ref: ${{ github.head_ref }}

      - name: Prettify code
        uses: creyD/prettier_action@v4.3
        with:
          # This part is also where you can pass other options, for example:
          prettier_options: --write **/*.{js,md}
```

### Secrets

In the case of secrets we could check if someone uploaded sensitive information to alert the team about it.

(In the case of a git workflow in github, gitlab, azure, blabla) it could be that the portion of code remains there (azure won’t delete abandoned pull requests for example)

![](https://cdn.hashnode.com/res/hashnode/image/upload/v1742839931211/e55fc7e5-dcf9-43b3-a931-9321ee7c56d8.png align="center")

In the case of `gitleaks` I’m using a reusable workflow that I’ve built myself

```yaml
name: CI/CD

on:
  workflow_dispatch:
  push:
    branches:
      - master
      - development
    paths:
      - "src/**"
      - "Dockerfile"
      - ".github/workflows/*.yml"
  pull_request:
    branches:
      - master
      - development
    paths:
      - "src/**"
      - "Dockerfile"

env:
  BRANCH_NAME: ${{ github.ref_name }}
  APP_NAME: waifuland_api
  APP_VERSION: latest
  APP_DEV_VERSION: unstable

jobs:
  gitleaks:
    uses: jd-apprentice/jd-workflows/.github/workflows/gitleaks.yml@main
    with:
      runs-on: ubuntu-latest
      name: Gitleaks
    secrets:
      gh_token: ${{ secrets.GITHUB_TOKEN }}
```

The origin of that workflow comes from [https://github.com/jd-apprentice/jd-workflows](https://github.com/jd-apprentice/jd-workflows)

## Conclusion

Now like I mentioned in some of the points, a few steps can be done at pre-commit level and not in the pipeline itself, that’s up to you or your team. But if you want to enforce everything you can do it here without interfere with the code in their projects.