Press "Enter" to skip to content

Tag: test

A Tour of Go: Web Crawler

After playing with Go a couple of days I managed to finish the tour. Actually I learned a bit more reading related articles and linked pages. I really recommend the experience as an initial attempt to understand go.

As a last exercise you are requested to program a simple web crawler. Find the code at the end of this post (I will not be much longer).
After a couple of hours tinkering and recalling all the knowledge from the language I managed to complete the exercise. Many things of this new language to interiorize.

No big conclusions yet about Go, but what I’m pretty sure is how good it enforces recursion and concurrency in programs…

Try this code on the Go Playground

Shellcheck in Jenkins

All we have written Shell scripts. And at the speed we need to deploy is easy to make mistakes (beginner mistakes). So I decided to integrate Shellcheck in Jenkins so that when a commit is done the CI drops a complete shell check analysis of the code.

A test in your CI could be something like:

grep -rIl '^#![[:blank:]]*/bin/\(bash\|sh\|zsh\)' \
     --exclude-dir=.git --exclude=*.sw? \
     | xargs shellcheck

The exclude switches are useful for git and vim users respectively.

Is strongly adviced to check for availability of shellcheck in your build agent before the test. I use to wrap my test code with some kind of check like so:

if which shellcheck ; then
  # Do your tests here

There are other nice ways of performing automatic tests using make as your procedure executor. I will talk about that in later posts.

Happy testing!

Edit: A nice snippet for your Makefiles could be like so.

Integrating npm jsonlint in your CI

Today I received a pull request on a JSON file where the indentation looked just wrong. We decided to look for a linting solution to run on our CI with some requirements:

  • It should validate and point out broken JSON.
  • It should fix indentation.
  • It should not sort JSON objects.

After some research we found different options with different levels of quality and livelyness. Actually the most difficult part of looking for solutions sometimes is strongly influenced by those 2 factors. But I wouldn’t enter in a comparison today, so we finally decided to go for npm jsonlint.

Is easy to install:

npm install -g jsonlint

And easy to use:

Usage: jsonlint [file] [options]

file     file to parse; otherwise uses stdin

   -v, --version            print version and exit
   -s, --sort-keys          sort object keys
   -i, --in-place           overwrite the file
   -t CHAR, --indent CHAR   character(s) to use for indentation  [  ]
   -c, --compact            compact error display
   -V, --validate           a JSON schema to use for validation
   -e, --environment        which specification of JSON Schema the validation file uses  [json-schema-draft-03]
   -q, --quiet              do not print the parsed json to STDOUT  [false]
   -p, --pretty-print       force pretty printing even if invalid

Test integration

In our case we basically deploy a set of Makefiles with every project where we define some standard targets like release, test or fix so basically I decided to add another linting check to the test target like so:

test: json.validate
    @for f in $$(find . -type f -name "*.json"); do \
      jsonlint $$f -p | sed -e '$$a\' | diff $$f -; \
      if [ $$? -ne 0 ]; then \
        echo "Error validating $$f." ; \
        exit 1 ; \
      fi \

The for loop iterates through all the json files found doing a diff using jsonlint with the pretty-print flag.

      jsonlint $$f -p | sed -e '$$a\' | diff $$f -;

Note the use of sed to append a new line to the stream so that diff does not throw an error “No newline at end of file”.

After the target is set in the Makefile everything is done as in our Jenkinsfile we have someting like this already:

    stage("Test") {
      steps {
        sh 'make test'

Fix integration

As done for the testing the fix target does exactly the same operation but applying the changes to the files directly. This target is going to be executed by developers before creating a Pull request as a cleanup step so is not critical to be fast so some more checks are executed to make it clean.

fix: json.fix
    @for f in $$(find . -type f -name "*.json"); do \
      jsonlint $$f -q && jsonlint $$f -qpi && sed -i -e '$$a\' $$f ; \
      echo "Formatted $$f" ; \
      if [ $$? -ne 0 ]; then \
        echo "Error validating $$f" ; \
        exit 1 ; \
      fi \

The fixing process first validates the JSON in order to avoid errors and also to minimize the output on the CI. If jsonlint fails validation with the pretty-print option this will output all the file to stdout. That is why you see jsonlint called twice.

      jsonlint $$f -q && jsonlint $$f -qpi && sed -i -e '$$a\' $$f 

Note the ‘-i’ flag to edit the file in place. Also note the usage of sed to append the new line to the file so that after a this automatic fix the file is guaranteed to pass the validation step.