13 Jan 2018, 22:34

Huge Furnace Update


Hi folks.

In the past couple of months I’ve been slowly updating Furnace.

There are three major changes that happened. Let’s take a look at them, shall we?

Google Cloud Platform

Furnace now supports Google Cloud Platform (GCP). It provides the same API to handle GCP resource as with AWS. Namely, create, delete, status, update. I opted to leave out push because Google mostly works with git based repositories, meaning a push is literary just a push, than Google handles distributing the new code by itself.

All the rest of the commands should work the same way as AWS.

Deployment Manager

GCP has a similar service to AWS CloudFormations called Deployment Manager. The documentation is fairly detailed with a Bookshelf example app to deploy. Code and Templates can be found in their Git repositroy here: Deployment Manager Git Repository.

Setting up GCP

As the README of Furnace outlines…

Please carefully read and follow the instruction outlined in this document: Google Cloud Getting Started. It will describe how to download and install the SDK and initialize cloud to a Project ID.

Take special attention to these documents:

Initializing GCloud Tools Authorizing Tools

Furnace uses a Google Key-File to authenticate with your Google Cloud Account and Project. In the future, Furnace assumes these things are properly set up and in working order.

To initialize the client, it uses the following code:

  ctx := context.Background()
  client, err := google.DefaultClient(ctx, dm.NdevCloudmanScope)

The DefaultClient in turn, does the following:

// FindDefaultCredentials searches for "Application Default Credentials".
// It looks for credentials in the following places,
// preferring the first location found:
//   1. A JSON file whose path is specified by the
//      GOOGLE_APPLICATION_CREDENTIALS environment variable.
//   2. A JSON file in a location known to the gcloud command-line tool.
//      On Windows, this is %APPDATA%/gcloud/application_default_credentials.json.
//      On other systems, $HOME/.config/gcloud/application_default_credentials.json.
//   3. On Google App Engine it uses the appengine.AccessToken function.
//   4. On Google Compute Engine and Google App Engine Managed VMs, it fetches
//      credentials from the metadata server.
//      (In this final case any provided scopes are ignored.)
func FindDefaultCredentials(ctx context.Context, scope ...string) (*DefaultCredentials, error) {

Take note on the order. This is how Google will authenticate your requests.

Running GCP

Running gcp is largely similar to AWS. First, you create the necessary templates to your infrastructure. This is done via the Deployment Manager and it’s templating engine. The GCP templates are Python JINJA files. Examples are provided in the template directory. It’s a bit more complicated than the CloudFormation templates in that it uses outside templates plus schema files to configure dynamic details.

It’s all explained in these documents: Creating a Template Step-by-step and Creating a Basic Template.

It’s not trivial however. And using the API can also be confusing. The Google Code is just a generated Go code file using gRPC. But studying it may provide valuable insigth into how the API is structured. I’m also providing some basic samples that I gathered together and the readme does a bit more explaining on how to use them.

Your First Stack

Once you have everything set-up you’ll need a configuration file for Furnace. The usage is outlined more here YAML Configuration. The configuration file for GCP looks like this:

  project_name: testplatform-1234
  spinner: 1
  template_name: google_template.yaml
  stack_name: test-stack

Where project_name is the name you generate for your first billable Google Cloud Platform project. Template lives next to this yaml file and stack name must be DNS complient.

Once you have a project and a template setup, it’s as simple as calling ./furnace-gcp create or ./furnace-gcp create mycustomstack.


Deleting happens with ./furnace-gcp delete or ./furnace-gcp delete mycustomstack. Luckily, as with AWS, this means that every resource created with the DeploymentManager will be deleted leaving no need for search and cleanup.

Project Name vs. Project ID

Unlike with AWS Google requires your stack name and project id to be DNS complient. This is most likely because all API calls and such contain that information.

Separate Binaries

In order to mitigate some of Furnace’s size, I’m providing separate binaries for each service it supports.

The AWS binaries can be found in aws folder, and respectively, the Google Cloud Platform is located in gcp. Both are build-able by running make.

If you would like to run both with a single command, a top level make file is provided for your convinience. Just run make from the root. That will build all binaries. Later on, Digital Oceans will join the ranks.

YAML Configuration

Last but not least, Furnace now employs YAML files for configuration. However, it isn’t JUST using YAML files. It also employs a smart configuration pattern which works as follows.

Since Furnace is a distributed binary file which could be running from any given location at any time. Because of that, at first I opted for a global configuration directory.

Now, however, furnace uses a furnace configuration file named with the following pattern: .stackalias.furnace. Where stackname, or stack is the name of a custom stack you would like to create for a project. The content of this file is a single entry, which is the location, relative to this file, of the YAML configuration files for the given stack. For example:


This means, that in the directory called stacks there will a yaml configuration file for your database stack. The AWS config file looks like this:

  stackname: FurnaceStack
  spinner: 1
  code_deploy_role: CodeDeployServiceRole
  region: us-east-1
  enable_plugin_system: false
  template_name: cloud_formation.template
  app_name: furnace_app
    # Only needed in case S3 is used for code deployment
    code_deploy_s3_bucket: furnace_code_bucket
    # The name of the zip file in case it's on a bucket
    code_deploy_s3_key: furnace_deploy_app
    # In case a Git Repository is used for the application, define these two settings
    git_account: Skarlso/furnace-codedeploy-app
    git_revision: b89451234...

The important part is the template_name. The template has to be next to this yaml file. To use this file, you simply call any of the AWS or GCP commands with an extra, optional parameter like this:

./furnace-aws create mydatabase

Note that mydatabase will translate to .mydatabase.furnace.

The intelligent part is, that this file could be placed anywhere in the project folder structure; because furnace, when looking for a config file, traverses backwards from the current execution directory up until /. Where root is not included in the search.

Consider the following directory tree:

├── docs
│ ├── furnace-aws status mydatabase
├── stacks
│   ├── mystack.template
│   └── mystack.yaml
└── .mydatabase.furnace

You are currently in your docs directory and would like to ask for the status of your database. You don’t have to move to the location of the setting file, just simply run the command from where you are. This only works if you are above the location of the file. If you would be below, furnace would say it can’t find the file. Because it only traverses upwards.

.mydatabase.furnace here contains only a single entry stacks/mystack.yaml. And that’s it. This way, you could have multiple furnace files, for example a .database.furnace, .front-end.furnace and a .backend.furnace. All three would work in unison, and if want needs updating, simply run ./furnace-aws update backend. And done!

Closing words

As always, contributions are welcomed in the form of issues or pull requests. Questions anything, I tend to answer as soon as I can.

Always run the tests before submitting.

Thank you for reading. Gergely.

04 Dec 2017, 22:34

Commit-Build-Deploy With AWS CodeBuild and Lambda


Hi All.

Today I would like to write about an AWS finger practice.

Previously, I wrote about how I build and deploy my blog with Wercker. Since, I’m a cloud engineer and I dislike Oracle and it’s ever expending tenctacles into the abyss, I wanted to switch to use something else.

My build and deploy cycle is simple.

Commit to Blogsource Repo -> Wercker WebHook -> Builds my blog using Hugo -> Pushed to a Different Repository which my Github Blog.

That’s all.

It’s quiet possible to reproduce this on AWS without infering costs. Unless you publish like… a couple 100 posts / week.

I’m going to use the following services: CloudFormation, AWS Lambda, CodeBuild, S3.

To deploy the below describe architecture in your account in us-east-1 region simply click this button: Launch Stack

BEFORE doing that though you need the following created:

Have a bucket for your lambda function. The lambda function can be found here:

Lambda Repository.

Zip up the lambda folder contents by doing this:

cd lambda
zip -r gitpusher.zip *
aws s3 cp gitpusher.zip s3://your-lambda-bucket

That’s it.

To read a description of the stack, please continue.


The architecture I’m about to lay out is simple in its use and design. I tried not to complicate things, because I think the simpler something is, the less prone to failure it will be.

In its most basic form the flow is as follows:


You push something into a repository you provide. CodeBuild has a webhook to this repository so on each commit it starts to build the blog. The build will use a so called buildspec.yaml file which describes how your blog should be built. Mine looks like this:

version: 0.2

      - echo Installing required packages and Hugo
      - apt-get update
      - apt-get install -y git golang wget
      - wget -q https://github.com/gohugoio/hugo/releases/download/v0.31/hugo_0.31_Linux-64bit.deb -O /tmp/hugo.dep
      - dpkg -i /tmp/hugo.dep
      - echo Downloading source code
      - git clone https://github.com/Skarlso/blogsource.git /opt/app
      - echo Build started on `date`
      - cd /opt/app && hugo --theme purehugo
      - echo Build completed on `date`
    - /opt/app/public/**/*

When it’s finished, CodeBuild will upload everything in the public folder as a zip to a bucket. The bucket has a lambda attached which triggers on putObject event with the extension .zip. It downloads the archive, extracts it and pushes it to another repository, which is the repository for the blog.

And done! That’s it. For an architecture overview, please read on.


Now, we are going to use CloudFormation stack to deploy these resources. Because we aren’t animals to create them by hand, yes?

An overview of my current architecture is best shown by this image:

AWS Stack.

Let’s go over these components one - by - one.

Lambda Role

This is the Role which allows the Lambda to access things in your account. It needs the following service access: s3, logs, lambda; and the following permissions: logs:Create*, logs:PutLogEvents, s3:GetObject, s3:ListBucket.

Code Build Role

This is the role which allows CodeBuild to have access to services it needs. These services are the following: s3, logs, ssm, codebuild. CodeBuild also needs the following actions allowed: logs:Create*, logs:PutLogEvents, s3:GetObject, s3:PutObject, ssm:GetParameters.

Build Bucket

This is the bucket in which CodeBuild will push the generated build artifact.

Blog Pusher Function

This is the heart of this project. It contains the logic to download the zipped artifact, extract it, create a hollow repository from the extracted archive and push the changes to the repository. And just the changes.

This is achieve by a short Python 3.6 script which can be found in the linked repository.


The stack requires you to provide a couple of parameters which are described in the template. Like, bucket name, github repository, git token and such. Please refer to the template for a full description of each.


I recently push a couple of builds to test this configuration and I inferred 0.2 USD in charges. But that was like 10-15 builds a day.


In order to deploy this you can use Furnace to easily manage the template and it’s parameters. Once you copy the template to the target directory, simply run furnace aws create and provide the necessary parameters.


And that is all. A nice little stack which does the same as Wercker without costs but the leisure of simply pushing up some change to a repository of your choosing.

I hope you enjoyed this little write up as much as I enjoyed creating it.

As always, Thanks for reading! Gergely.

16 Apr 2017, 09:23

Furnace - The building of an AWS CLI Tool for CloudFormation and CodeDeploy - Part 4


Hi folks.

Previously on this blog: Part 1. Part 2. Part 3.

In this part we are going to talk about Unit Testing Furnace and how to work some magic with AWS and Go.

Mock Stub Fake Dummy Canned

Unit testing in Go usually follows the Dependency Injection model of dealing with Mocks and Stubs.

## DI

Dependency Inject in short is one object supplying the dependencies of another object. In a longer description, it’s ideal to be used for removing the lock on a third party library, like the AWS client. Imaging having code which solely depends on the AWS client. How would you unit test that code without having to ACTUALLY connect to AWS? You couldn’t. Every time you try to test the code it would run the live code and it would try and connect to AWS and perform the operations it’s design to do. The Ruby library with it’s metaprogramming allows you to set the client globally to stub responses, but, alas, this is not the world of Ruby.

Here is where DI comes to the rescue. If you have control over the AWS client on a very high level, and would pass it around as a function parameter, or create that client in an init() function and have it globally defined; you would be able to implement your own client, and have your code use that with stubbed responses which your tests need. For example, you would like a CreateApplication call to fail, or you would like a DescribeStack which returns an aws.Error(“StackAlreadyExists”).

For this, however, you need the API of the AWS client. Which is provided by AWS.

AWS Client API

In order for DI to work, the injected object needs to be of a certain type for us to inject our own. Luckily, AWS provides an Interface for all of it’s clients. Meaning, we can implement our own version for all of the clients, like S3, CloudFormation, CodeDeploy etc.

For each client you want to mock out, an *iface package should be present like this:


In this package you find and use the interface like this:

type fakeCloudFormationClient struct {
	err error

And with this, we have our own CloudFormation client. The real code uses the real clients as function parameters, like this:

// Execute defines what this command does.
func (c *Create) Execute(opts *commander.CommandHelper) {
	log.Println("Creating cloud formation session.")
	sess := session.New(&aws.Config{Region: aws.String(config.REGION)})
	cfClient := cloudformation.New(sess, nil)
	client := CFClient{cfClient}
	createExecute(opts, &client)

We can’t test Execute itself, as it’s using the real client here (or you could have a global from some library, thus allowing you to tests even Execute here) but there is very little logic in this function for this very reason. All the logic is in small functions for which the main starting point and our testing opportunity is, createExecute.

Stubbing Calls

Now, that we have our own client, and with the power of Go’s interface embedding as seen above with CloudFormationAPI, we have to only stub the functions which we are actually using, instead of every function of the given interface. This looks like this:

	cfClient := new(CFClient)
	cfClient.Client = &fakeCloudFormationClient{err: nil}

Where cfClient is a struct like this:

// CFClient abstraction for cloudFormation client.
type CFClient struct {
	Client cloudformationiface.CloudFormationAPI

And a stubbed call can than be written as follows:

func (fc *fakeCreateCFClient) WaitUntilStackCreateComplete(input *cloudformation.DescribeStacksInput) error {
	return nil

This can range from a very trivial example, like the one above, to intricate ones as well, like this gem:

func (fc *fakePushCFClient) ListStackResources(input *cloudformation.ListStackResourcesInput) (*cloudformation.ListStackResourcesOutput, error) {
	if "NoASG" == *input.StackName {
		return &cloudformation.ListStackResourcesOutput{
			StackResourceSummaries: []*cloudformation.StackResourceSummary{
					ResourceType:       aws.String("NoASG"),
					PhysicalResourceId: aws.String("arn::whatever"),
		}, fc.err
	return &cloudformation.ListStackResourcesOutput{
		StackResourceSummaries: []*cloudformation.StackResourceSummary{
				ResourceType:       aws.String("AWS::AutoScaling::AutoScalingGroup"),
				PhysicalResourceId: aws.String("arn::whatever"),
	}, fc.err

This ListStackResources stub lets us test two scenarios based on the stackname. If the test stackname is ‘NoASG’ it will return a result which equals to a result containing no AutoScaling Group. Otherwise, it will return the correct ResourceType for an ASG.

It is a common practice to line up several scenario based stubbed responses in order to test the robustness of your code.

Unfortunately, this also means that your tests will be a bit cluttered with stubs and mock structs and whatnots. For that, I’m partially using a package available struct file in which I’m defining most of the mock structs at least. And from there on, the tests will only contain specific stubs for that particular file. This can be further fine grained by having defaults and than only override in case you need something else.

Testing fatals

Now, the other point which is not really AWS related, but still comes to mind when dealing with Furnace, is testing error scenarios.

Because Furnace is a CLI application it uses Fatals to signal if something is wrong and it doesn’t want to continue or recover because, frankly it can’t. If AWS throws an error, that’s it. You can retry, but in 90% of the cases, it’s usually something that you messed up.

So, how do we test for a fatal or an os.Exit? There are a number of points on that if you do a quick search. You may end up on this talk: GoTalk 2014 Testing Slide #23. Which does an interesting thing. It calls the test binary in a separate process and tests the exit code.

Others, and me as well, will say that you have to have your own logger implemented and use a different logger / os.Exit in your test environment.

Others others will tell you to not to have tests around os.Exit and fatal things, rather return an error and only the main should pop a world ending event. I leave it up to you which you want to use. Either is fine.

In Furnace, I’m using a global logger in my error handling util like this:

// HandleFatal handler fatal errors in Furnace.
func HandleFatal(s string, err error) {
	LogFatalf(s, err)

And LogFatalf is an exported variable var LogFatalf = log.Fatalf. Than in a test, I just override this variable with a local anonymous function:

func TestCreateExecuteEmptyStack(t *testing.T) {
	failed := false
	utils.LogFatalf = func(s string, a ...interface{}) {
		failed = true
	client := new(CFClient)
	stackname := "EmptyStack"
	client.Client = &fakeCreateCFClient{err: nil, stackname: stackname}
	opts := &commander.CommandHelper{}
	createExecute(opts, client)
	if !failed {
		t.Error("expected outcome to fail during create")

It can get even more granular by testing for the error message to make sure that it actually fails at the point we think we are testing:

func TestCreateStackReturnsWithError(t *testing.T) {
	failed := false
	expectedMessage := "failed to create stack"
	var message string
	utils.LogFatalf = func(s string, a ...interface{}) {
		failed = true
		if err, ok := a[0].(error); ok {
			message = err.Error()
	client := new(CFClient)
	stackname := "NotEmptyStack"
	client.Client = &fakeCreateCFClient{err: errors.New(expectedMessage), stackname: stackname}
	config := []byte("{}")
	create(stackname, config, client)
	if !failed {
		t.Error("expected outcome to fail")
	if message != expectedMessage {
		t.Errorf("message did not equal expected message of '%s', was:%s", expectedMessage, message)


This is it. That’s all it took to write Furnace. I hope you enjoyed reading it as much as I enjoyed writing all these thoughts down.

I hope somebody might learn from my journey and also improve upon it.

Any comments are much appreciated and welcomed. Also, PRs and Issues can be submitted on the GitHub page of Furnace.

Thank you for reading! Gergely.

22 Mar 2017, 12:03

Furnace - The building of an AWS CLI Tool for CloudFormation and CodeDeploy - Part 3


Hi folks.

Previously on this blog: Part 1. Part 2. Part 4.

In this part, I’m going to talk about the experimental plugin system of Furnace.

Go Experimental Plugins

Since Go 1.8 was released, an exciting and new feature was introduced called a Plug-in system. This system works with dynamic libraries built with a special switch to go build. These libraries, .so or .dylib (later), are than loaded and once that succeeds, specific functions can be called from them (symbol resolution).

We will see how this works. For package information, visit the plugin packages Go doc page here.

Furnace Plugins

So, what does furnace use plugins for? Furnace uses plugins to execute arbitery code in, currently, four given locations / events.

These are: pre_create, post_create, pre_delete, post_delete. These events are called, as their name suggests, before and after the creation and deletion of the CloudFormation stack. It allows the user to execute some code without having to rebuild the whole project. It does that by defining a single entry point for the custom code called RunPlugin. Any number of functions can be implemented, but the plugin MUST provide this single, exported function. Otherwise it will fail and ignore that plugin.

Using Plugins

It’s really easy to implement, and use these plugins. I’m not going into the detail of how to load them, because that is done by Furnace, but only how to write and use them.

To use a plugin, create a go file called: 0001_mailer.go. The 0001 before it will define WHEN it’s executed. Having multiple plugins is completely okay. Execution of order however, depends on the names of the files.

Now, in 0001_mailer.post_create we would have something like this:

package main

import "log"

// RunPlugin runs the plugin.
func RunPlugin() {
	log.Println("My Awesome Pre Create Plugin.")

Next step is the build this file to be a plugin library. Note: Right now, this only works on Linux!

To build this file run the following:

go build -buildmode=plugin -o 0001_mailer.pre_create 0001_mailer.go

The important part here is the extension of the file specified with -o. It’s important because that’s how Furnace identifies what plugins it has to run.

Finally, copy this file to ~/.config/go-furnace/plugins and you are all set.

Slack notification Plugin

To demonstrate how a plugin could be used is if you need some kind of notification once a Stack is completed. For example, you might want to send a message to a Slack room. To do this, your plugin would look something like this:

package main

import (


func RunPlugin() {
	stackname := os.Getenv("FURNACE_STACKNAME")
	api := slack.New("YOUR_TOKEN_HERE")
	params := slack.PostMessageParameters{}
	channelID, timestamp, err := api.PostMessage("#general", fmt.Sprintf("Stack with name '%s' is Done.", stackname), params)
	if err != nil {
		fmt.Printf("%s\n", err)
	fmt.Printf("Message successfully sent to channel %s at %s", channelID, timestamp)

Currently, Furnace has no ability to share information of the stack with an outside plugin. Thus ‘Done’ could be anything from Rollback to Failed to CreateComplete.

Closing Words

That’s it for plugins. Thanks very much for reading! Gergely.

19 Mar 2017, 12:03

Furnace - The building of an AWS CLI Tool for CloudFormation and CodeDeploy - Part 2


Hi folks.

Previously on this blog: Part 1, Part 3, Part 4

In this part, I’m going to talk about the AWS Go SDK and begin do dissect the intricacies of Furnace.


Fortunately, the Go SDK for AWS is quiet verbose and littered with examples of all sorts. But that doesn’t make it less complex and less cryptic at times. I’m here to lift some of the early confusions, in hopes that I can help someone to avoid wasting time.

Getting Started and Developers Guide

As always, and common from AWS, the documentation is top notch. There is a 141 pages long developer’s guide on the SDK containing a getting started section and an API reference. Go check it out. I’ll wait. AWS Go SDK DG PDF. I will only talk about some gotchas and things I encountered, not the basics of the SDK.

aws.String and other types

Something which is immediately visible once we take a look at the API is that everything is a pointer. Now, there are a tremendous amount of discussions about this, but I’m with Amazon. There are various reasons for it, but to list the most prominent ones: - Type completion and compile time type safety. - Values for AWS API calls have valid zero values, in addition to being optional, i.e. not being provided at all. - Other option, like, empty interfaces with maps, or using zero values, or struct wrappers around every type, made life much harder rather than easier or not possible at all. - The AWS API is volatile. You never know when something gets to be optional, or required. Pointers made that decision easy.

There are good number of other discussions around this topic, for example: AWS Go GitHub #363.

In order to use primitives, AWS has helper functions like aws.String. Because &“asdf” is not allowed, you would have to create a variable and use its address in situations where a string pointer is needed, for example, name of the stack. These primitive helpers will make in-lining possible. We’ll see later that they are used to a great extent. Pointers, however, make life a bit difficult when constructing Input structs and make for poor aesthetics.

This is something I’m returning in a test for stubbing a client call:

		return &cloudformation.ListStackResourcesOutput{
			StackResourceSummaries: []*cloudformation.StackResourceSummary{
					ResourceType:       aws.String("NoASG"),
					PhysicalResourceId: aws.String("arn::whatever"),

This doesn’t look so appealing, but one gets used to it quickly.

Error handling

Errors also have their own types. An AWS error looks like this:

if err != nil {
    if awsErr, ok := err.(awserr.Error); ok {

First, we check if error is nil, than we type check if the error is an AWS error or something different. In the wild, this will look something like this:

	if err != nil {
		if awsErr, ok := err.(awserr.Error); ok {
			if awsErr.Code() != codedeploy.ErrCodeDeploymentGroupAlreadyExistsException {
				return err
			log.Println("DeploymentGroup already exists. Nothing to do.")
			return nil
		return err

If it’s an AWS error, we can check further for the error code that it returns in order to identify what to handle, or what to throw on to the caller to a potential fatal. Here, I’m ignoring the AlreadyExistsException because, if it does, we just go on to a next action.


Luckily the API doc is very mature. In most of the cases, they provide an example to an API call. These examples, however, from time to time provide more confusion than clarity. Take CloudFormation. For me, when I first glanced upon the description of the API it wasn’t immediately clear that the TemplateBody was supposed to be the whole template, and that the rest of the fields were almost all optional settings. Or provided overrides in special cases.

And since the template is not an ordinary JAML or JSON file, I was looking for something that parses it into that the Struct I was going to use. After some time, and digging, I realized that I didn’t need that, and that I just need to read in the template, define some extra parameters, and give the TemplateBody the whole of the template. The parameters defined by the CloudFormation template where extracted for me by ValidateTemplate API call which returned all of them in a convenient []*cloudformation.Parameter slice. These things are not described in the document or visible from the examples. I mainly found them through playing with the API and focused experimentation.


From other SDK implementations, we got used to Waiters. These handy methods wait for a service to become available or for certain situations to take in effect, like a Stage being CREATE_COMPLETE. The Go waiters, however, don’t allow for callback to be fired, or for running blocks, like the ruby SDK does. For this, I wrote a handy little waiter for myself, which outputs a spinner to see that we are currently waiting for something and not frozen in time. This waiter looks like this:

// WaitForFunctionWithStatusOutput waits for a function to complete its action.
func WaitForFunctionWithStatusOutput(state string, freq int, f func()) {
	var wg sync.WaitGroup
	done := make(chan bool)
	go func() {
		defer wg.Done()
		done <- true
	go func() {
		counter := 0
		for {
			counter = (counter + 1) % len(Spinners[config.SPINNER])
			fmt.Printf("\r[%s] Waiting for state: %s", yellow(string(Spinners[config.SPINNER][counter])), red(state))
			time.Sleep(time.Duration(freq) * time.Second)
			select {
			case <-done:


And I’m calling it with the following method:

	utils.WaitForFunctionWithStatusOutput("DELETE_COMPLETE", config.WAITFREQUENCY, func() {

This would output these lines to the console:

[\] Waiting for state: DELETE_COMPLETE

The spinner can be configured to be one of the following types:

var Spinners = []string{`←↖↑↗→↘↓↙`,


And with that, let’s dive into the basics of Furnace.


Directory Structure and Packages

Furnace is divided into three main packages.


Commands package is where the gist of Furnace lies. These commands represent the commands which are used through the CLI. Each file has the implementation for one command. The structure is devised by this library: Yitsushi’s Command Library. As of the writing of this post, the following commands are available: - create - Creates a stack using the CloudFormation template file under ~/.config/go-furnace - delete - Deletes the created Stack. Doesn’t do anything if the stack doesn’t exist - push - Pushes an application to a stack - status - Displays information about the stack - delete-application - Deletes the CodeDeploy application and deployment group created by push

These commands represent the heart of furnace. I would like to keep these to a minimum, but I do plan on adding more, like update and rollout. Further details and help messages on these commands can be obtained by running: ./furnace help or ./furnace help create.

❯ ./furnace help push
Usage: furnace push appName [-s3]

Push a version of the application to a stack

  furnace push
  furnace push appName
  furnace push appName -s3
  furnace push -s3


Contains the configuration loader and some project wide defaults which are as follows: - Events for the plugin system - pre-create, post-create, pre-delete, post-delete. - CodeDeploy role name - CodeDeployServiceRole. This is used if none is provided to locate the CodeDeploy IAM role. - Wait frequency - Is the setting which controls how long the waiter should sleep in between status updates. Default is 1s. - Spinner - Is just the number of the spinner to use. - Plugin registry - Is a map of functions to run for the above events.

Further more, config loads the CloudFormation template and checks if some necessary settings are present in the environment, exp: the configuration folder under ~/.config/go-furnace.


These are some helper functions which are used throughout the project. To list them: - error_handler - Is a simple error handler. I’m thinking of refactoring this one to some saner version. - spinner - Sets up which spinner to use in the waiter function. - waiter - Contains the verbose waiter introduced above under Waiters.

Configuration and Environment variables

Furnace is a Go application, thus it doesn’t have the luxury of Ruby or Python where the configuration files are usually bundled with the app. But, it does have a standard for it. Usually, configurations reside in either of these two locations. Environment Properties or|and configuration files under a fixed location ( i.e. HOME/.config/app-name ). Furnace employs both.

Settings like, region, stack name, enable plugin system, are under environment properties ( though this can change ), while the CloudFormation template lives under ~/.config/go-furnace/. Lastly it assumes some things, like the Deployment IAM role just exists under the used AWS account. All these are loaded and handled by the config package described above.


A typical scenario for Furnace would be the following:

  • Setup your CloudFormation template or use the one provided. The one provided sets up a highly available and self healing setting using Auto-Scaling and Load-Balancing with a single application instance. Edit this template to your liking than copy it to ~/.config/go-furnace.
  • Create the configured stack with ./furnace create.
  • Create will ask for the parameters defined in the template. If defaults are setup, simply hitting enter will use these defaults. Take note, that the provided template sets up SSH access via a provided key. If that key is not present in CF, you won’t be able to SSH into the created instance.
  • Once the stack is completed, the application is ready to be pushed. To do this, run: ./furnace push. This will locate the appropriate version of the app from S3 or GitHub and push that version to the instances in the Auto-Scaling group. To all of them.

General Practices Applied to the Project


For each command the main entry point is the execute function. These functions are usually calling out the small chunks of distributed methods. Logic was kept to a bare minimum ( probably could be simplified even further ) in the execute functions mostly for testability and the likes. We will see that in a followup post.


Errors are handled immediately and usually through a fatal. If any error occurs than the application is halted. In followup versions this might become more granular. I.e. don’t immediately stop the world, maybe try to recover, or create a Poller or Re-Tryer, which tries a call again for a configured amount of times.

Output colors

Not that important, but still… Aesthetics. Displaying data to the console in a nice way gives it some extra flare.


This project works with a Makefile for various reasons. Later on, once the project might become more complex, a Makefile makes it really easy to handle different ways of packaging the application. Currently, for example, it provides a linux target which will make Go build the project for Linux architecture on any other Architecture i.e. cross-compiling.

It also provides an easy way to run unit tests with make test and installing with make && make install.

Closing Words

That is all for Part 2. Join me in Part 3 where I will talk about the experimental Plugin system that Furnace employs.

Thank you for reading! Gergely.

16 Mar 2017, 21:49

Furnace - The building of an AWS CLI Tool for CloudFormation and CodeDeploy - Part 1

Other posts:

Part 2, Part 3, Part 4.

Building Furnace: Part 1


Hi folks.

This is the first part of a 4 part series which talks about the process of building a middlish sized project in Go, with AWS. Including Unit testing and a experimental plugin feature.

The first part will talk about the AWS services used in brief and will contain a basic description for those who are not familiar with them. The second part will talk about the Go SDK and the project structure itself, how it can be used, improved, and how it can help in everyday life. The third part will talk about the experimental plugin system, and finally, we will tackle unit testing AWS in Go.

Let’s begin, shall we?



If you haven’t yet read about, or know off, AWS’ CloudFormation service, you can either go ahead and read the Documentation or read on for a very quick summary. If you are familiar with CF, you should skip ahead to CodeDeploy section.

CF is a service which bundles together other AWS services (for example: EC2, S3, ELB, ASG, RDS) into one, easily manageable stack. After a stack has been created, all the resources can be handled as one, located, tagged and used via CF specific console commands. It’s also possible to define any number of parameters, so a stack can actually be very versatile. A parameter can be anything, from SSH IP restriction to KeyPair names and list of tags to create or in what region the stack will be in.

To describe how these parts fit together, one must use a CloudFormation Template file which is either in JSON or in YAML format. A simple example looks like this:

        Description: The EC2 Key Pair to allow SSH access to the instance
        Type: AWS::EC2::KeyPair::KeyName
        Type: AWS::EC2::Instance
          - Ref: InstanceSecurityGroup
          - MyExistingSecurityGroup
            Ref: KeyName
          ImageId: ami-7a11e213
        Type: AWS::EC2::SecurityGroup
          GroupDescription: Enable SSH access via port 22
          - IpProtocol: tcp
            FromPort: '22'
            ToPort: '22'

There are a myriad of these template samples here.

I’m not going to explain this in too much detail. Parameters define the parameters, and resources define all the AWS services which we would like to configure. Here we can see, that we are creating an EC2 instance with a custom Security Group plus and already existing security group. ImageId is the AMI which will be used for the EC2 instance. The InstanceSecurityGroup is only defining some SSH access to the instance.

That is pretty much it. This can become bloated relatively quickly once, VPCs, ELBs, and ASGs come into play. And CloudFormation templates can also contain simple logical switches, like, conditions, ref for variables, maps and other shenanigans.

For example consider this part in the above example:

        Ref: KeyName

Here, we use the KeyName parameter as a Reference Value which will be interpolated to the real value, or the default one, as the template gets processed.


If you haven’t heard about CodeDeploy yet, please browse the relevant Documentation or follow along for a “quick” description.

CodeDeploy just does what the name says. It deploys code. Any kind of code, as long as the deployment process is described in a file called appspec.yml. It can be easy as coping a file to a specific location or incredibly complex with builds of various kinds.

For a simple example look at this configuration:

    version: 0.0
    os: linux
      - source: /index.html
        destination: /var/www/html/
      - source: /healthy.html
        destination: /var/www/html/
        - location: scripts/install_dependencies
          timeout: 300
          runas: root
        - location: scripts/clean_up
          timeout: 300
          runas: root
        - location: scripts/start_server
          timeout: 300
          runas: root
        - location: scripts/stop_server
          timeout: 300
          runas: root

CodeDeploy applications have hooks and life-cycle events which can be used to control the deployment process of an like, starting the WebServer; making sure files are in the right location; copying files, running configuration management software like puppet, ansible or chef; etc, etc.

What can be done in an appspec.yml file is described here: Appspec Reference Documentation.

Deployment happens in one of two ways:


If the preferred way to deploy the application is from GitHub a commit hash must be used to identify which “version” of the application is to be deployed. For example:

    rev = &codedeploy.RevisionLocation{
        GitHubLocation: &codedeploy.GitHubLocation{
            CommitId:   aws.String("kajdf94j0f9k309klksjdfkj"),
            Repository: aws.String("Skarlso/furnace-codedeploy-app"),
        RevisionType: aws.String("GitHub"),

Commit Id is the hash of the latest release and repository is the full account/repository pointing to the application.


The second way is to use an S3 bucket. The bucket will contain an archived version of the application with a given extension. I’m saying given extension, because it has to be specified like this (and can be either ‘zip’, or ‘tar’ or ‘tgz’):

    rev = &codedeploy.RevisionLocation{
        S3Location: &codedeploy.S3Location{
            Bucket:     aws.String("my_codedeploy_bucket"),
            BundleType: aws.String("zip"),
            Key:        aws.String("my_awesome_app"),
            Version:    aws.String("VersionId"),
        RevisionType: aws.String("S3"),

Here, we specify the bucket name, the extension, the name of the file and an optional version id, which can be ignored.


So how does code deploy get either of the applications to our EC2 instances? It uses an agent which is running on all of the instances that we create. In order to do this, the agent needs to be present on our instance. For linux this can be achieved with the following UserData (UserData in CF is the equivalent of a bootsrap script):

    "UserData" : {
        "Fn::Base64" : { "Fn::Join" : [ "\n", [
            "#!/bin/bash -v",
            "sudo yum -y update",
            "sudo yum -y install ruby wget",
            "cd /home/ec2-user/",
            "wget https://aws-codedeploy-eu-central-1.s3.amazonaws.com/latest/install",
            "chmod +x ./install",
            "sudo ./install auto",
            "sudo service codedeploy-agent start",
        ] ] }

A simple user data configuration in the CloudFormation template will make sure that every instance that we create will have the CodeDeploy agent running and waiting for instructions. This agent is self updating. Which can cause some trouble if AWS releases a broken agent. However unlikely, it can happen. Never the less, once installed, it’s no longer a concern to be bothered with.

It communications on HTTPS port 443.

CodeDeploy identifies instances which need to be updated according to our preferences, by tagging the EC2 and Auto Scaling groups. Tagging happens in the CloudFormation template through the AutoScalingGroup settings like this:

    "Tags" : [
            "Key" : "fu_stage",
            "Value" : { "Ref": "AWS::StackName" },
            "PropagateAtLaunch" : true

This will give the EC2 instance a tag called fu_stage with value equaling to the name of the stack. Once this is done, CodeDeploy looks like this:

    params := &codedeploy.CreateDeploymentInput{
        ApplicationName:               aws.String(appName),
        IgnoreApplicationStopFailures: aws.Bool(true),
        DeploymentGroupName:           aws.String(appName + "DeploymentGroup"),
        Revision:                      revisionLocation(),
        TargetInstances: &codedeploy.TargetInstances{
            AutoScalingGroups: []*string{
            TagFilters: []*codedeploy.EC2TagFilter{
                    Key:   aws.String("fu_stage"),
                    Type:  aws.String("KEY_AND_VALUE"),
                    Value: aws.String(config.STACKNAME),
        UpdateOutdatedInstancesOnly: aws.Bool(false),

CreateDeploymentInput is the entire parameter list that is needed in order to identify instances to deploy code to. We can see here that it looks for an AutoScalingGroup by Physical Id and the tag labeled fu_stage. Once found, it will use UpdateOutdatedInstancesOnly to determine if an instance needs to be updated or not. Set to false means, it always updates.


Where does Furnace fit in, in all of this? Furnace provides a very easy mechanism to create, delete and push code to a CloudFormation stack using CodeDeploy, and a couple of environment properties. Furnace create will create a CloudFormation stack according to the provided template, all the while asking for the parameters defined in it for flexibility. delete will remove the stack and all affiliated resources except for the created CodeDeploy application. For that, there is delete-application. status will display information about the stack: Outputs, Parameters, Id, Name, and status. Something like this:

    2017/03/16 21:14:37 Stack state is:  {
      Capabilities: ["CAPABILITY_IAM"],
      CreationTime: 2017-03-16 20:09:38.036 +0000 UTC,
      DisableRollback: false,
      Outputs: [{
          Description: "URL of the website",
          OutputKey: "URL",
          OutputValue: "http://FurnaceSt-ElasticL-ID.eu-central-1.elb.amazonaws.com"
      Parameters: [
          ParameterKey: "KeyName",
          ParameterValue: "UserKeyPair"
          ParameterKey: "SSHLocation",
          ParameterValue: ""
          ParameterKey: "CodeDeployBucket",
          ParameterValue: "None"
          ParameterKey: "InstanceType",
          ParameterValue: "t2.nano"
      StackId: "arn:aws:cloudformation:eu-central-1:9999999999999:stack/FurnaceStack/asdfadsf-adsfa3-432d-a-fdasdf",
      StackName: "FurnaceStack",
      StackStatus: "CREATE_COMPLETE"

( This will later be improved to include created resources as well. )

Once the stack is CREATE_COMPLETE a simple push will deliver our application on each instance in the stack. We will get into more detail about how these commands are working in Part 2 of this series.

Final Words

This is it for now.

Join me next time when I will talk about the AWS Go SDK and its intricacies and we will start to look at the basics of Furnace.

As always, Thanks for reading! Gergely.

17 Apr 2016, 00:00

Minecraft world automatic backup to AWS S3 bucket - Part 2 (Custom functions)

Hi folks.

Got an update for the backup script. This time, you’ll have the ability to implement your own upload capabilities. I provide a mock implementation for the required functions.

Here is the script again, now modified and a bit cleaned up. I hope it’s helpful.


if [[ -t 1 ]]; then
    colors=$(tput colors)
    if [[ $colors ]]; then

if [[ -z ${MINECRAFT_BUCKET} ]]; then
    printf "Please set the env variable %bMINECRAFT_BUCKET%b to the s3 archive bucket name.\n" "${RED}" "${NC}"
    exit 1

if [[ -z ${MINECRAFT_ARCHIVE_LIMIT} ]]; then
    printf "Please set the env variable %bMINECRAFT_ARCHIVE_LIMIT%b to limit the number of archives to keep.\n" "${RED}" "${NC}"
    exit 1

if [[ -z ${MINECRAFT_WORLD} ]]; then
    printf "Please set the env variable %bMINECRAFT_WORLD%b to specify what world to back-up.\n" "${RED}" "${NC}"
    exit 1

archive_name="${backup_world}-$(date +"%H-%M-%S-%m-%d-%Y").zip"

function create_archive {
    printf "Creating archive of %b${backup_world}%b\n" "${RED}" "${NC}"
    zip -r $archive_name $backup_world

function amazon_bak {


    printf "Checking if bucket has more than %b${backup_limit}%b files already.\n" "${RED}" "${NC}"
    content=( $(aws s3 ls s3://$backup_bucket | awk '{print $4}') )

    if [[ ${#content[@]} -eq $backup_limit || ${#content[@]} -gt $backup_limit  ]]; then
        echo "There are too many archives. Deleting oldest one."
        # We can assume here that the list is in cronological order
    	printf "%bs3://${backup_bucket}/${content[0]}\n%b" "${RED}" "${NC}"
        aws s3 rm s3://$backup_bucket/${content[0]}

    printf "Uploading %b${archive_name}%b to s3 archive bucket.\n" "${RED}" "${NC}"
    state=$(aws s3 cp $archive_name s3://$backup_bucket)

    if [[ "$state" =~ "upload:" ]]; then
        printf "File upload %bsuccessful%b.\n" "${LIGHT_GREEN}" "${NC}"
        printf "%bError%b occured while uploading archive. Please investigate.\n" "${RED}" "${NC}"

function custom {
    if [[ -e custom.sh ]]; then
        source ./custom.sh
        echo "custom.sh script not found. Please implement the apropriate functions."
        exit 1

    echo "Checking for the number of files. Limit is: $backup_limit."
    files=( $(list) )
    if [[ ${#files[@]} -eq $backup_limit || ${#files[@]} -gt $backup_limit ]]; then
        echo "Deleting extra file."
        delete ${files[0]}
        if [[ $? != 0 ]]; then
            printf "%bFailed%b to delete file. Please investigate failure." "${RED}" "${NC}"
            exit $?

    echo "Zipping world."

    echo "Uploading world."
    upload $archive_name

    if [[ $? != 0 ]]; then
        printf "%bFailed%b to upload archive. Please investigate the error." "${RED}" "${NC}"
        exit $?

    printf "Upload %bsuccessful%b" "${LIGHT_GREEN}" "${NC}"

function help {
    echo "Usage:"
    echo "./backup_world [METHOD]"
    echo "Exp.: ./backup_world aws|./backup_world custom|./backup_world dropbox"
    echo "Each method has it's own environment properties that it requires."
    echo "Custom: Have a file, called 'custom.sh' which is sourced."
    echo "Implement these three functions: upload | list | delete."
    echo "upload -> should return exit code 0 on success, should return exit code 1 on failure."
    echo "list -> should return a list of cronologically ordered items."
    echo "delete -> should return exit code 0 on success, should return exit code 1 on failure."

case $1 in
    aws )
    custom )
    * )

And here is the sample implementation for the custom upload functionality.


function upload {
    echo "uploading"
    local result=0
    return $result

function delete {
    echo "deleting $1"
    local result=0
    return $result

function list {
    local arr=("file1" "file2" "file3")
    echo "${arr[@]}"

Thanks for reading!


16 Apr 2016, 00:00

Minecraft world automatic backup to AWS S3 bucket

Hi Folks.

Previously we created a Minecraft server using Docker. After my server got popular in the family, and a lot of stuff started to pile up on it, as a good IT person, I’m backing up the world once in a while.

For that, I’m using AWS S3 with the CLI and a little bash script which runs once a week.

The script is really straightforward. I’m doing manual versioning, although S3 does provide one out of the box. However, amazon’s S3 versioning doesn’t allow limiting the number of versions being kept. And since I’m doing that anyways, might as well take care of the rest.

Without further ado, here is the script:


if [[ -t 1 ]]; then
    colors=$(tput colors)
    if [[ $colors ]]; then

if [[ -z ${MINECRAFT_BUCKET} ]]; then
	printf "Please set the env variable ${RED}MINECRAFT_BUCKET${NC} to the s3 archive bucket name.\n"
	exit 0

if [[ -z ${MINECRAFT_ARCHIVE_LIMIT} ]]; then
	printf "Please set the env variable ${RED}MINECRAFT_ARCHIVE_LIMIT${NC} to limit the number of archives to keep.\n"
	exit 0

printf "Creating archive of ${RED}${world}${NC}\n"
archive_name="${world}-$(date +"%H-%M-%S-%m-%d-%Y").zip"
zip -r $archive_name $world

printf "Checking if bucket has more than ${RED}${backup_limit}${NC} files already.\n"
content=( $(aws s3 ls s3://$backup_bucket | awk '{print $4}') )

if [[ ${#content[@]} -eq $backup_limit || ${#content[@]} -gt $backup_limit  ]]; then
    echo "There are too many archives. Deleting oldest one."
    # We can assume here that the list is in cronological order
	printf "${RED}s3://${backup_bucket}/${content[0]}\n"
    aws s3 rm s3://$backup_bucket/${content[0]}

printf "Uploading ${RED}${archive_name}${NC} to s3 archive bucket.\n"
state=$(aws s3 cp $archive_name s3://$backup_bucket)

if [[ "$state" =~ "upload:" ]]; then
    printf "File upload ${LIGHT_GREEN}successful${NC}.\n"
    printf "${RED}Error${NC} occured while uploading archive. Please investigate.\n"

It uses environment properties to define where to upload the given world and how many versions to keep.

I’m calling this from a cron job, and it’s sitting next to where the Minecraft world is.

That’s it folks.

I’ll start expanding on this idea and implement various services, like your own server address, or dropbox, or what have you.

Happy backing up.