06 Oct 2016, 00:00

RScrap scraper

Intro

Hey folks.

So, there is this project called Huginn which I absolutely love.

But the thing is, that for a couple of scrappers ( at least for me ), I don’t want to spin up a whole rails app.

Hence, I’ve come up with RScrap. Which is a bunch of Ruby scripts run as cron jobs on a raspberry pi. And because I dislike emails as well, and most of the time, I don’t read them, I opted for a nicer solution. Enter the world of Telegram. They provide you with the ability to create bots. You basically get an API key, and than using that key, you can send private messages, or even create an interactive bot which you can send messages too.

In my simple example, I’m using it to send private messages to myself, but I could just as well, make it interactive and than tell it to run one of the scripts.

The Code

Let’s take a look at what we got.

The main scraper

The main scraper, is simply bunch of convenience methods that wrap handling and working with the database and the telegram bot. That’s all. It’s very simple. Very short. The Telegram part is just this bit:

def send_message(text)
  Telegram::Bot::Client.run(@token) do |bot|
    bot.api.send_message(chat_id: @id, text: text)
  end
end

Straightforward. Creating an interactive bot, would look something like this:

#!/usr/bin/env ruby
require 'telegram/bot'

token = 'YOUR_TELEGRAM_BOT_API_TOKEN'

Telegram::Bot::Client.run(token) do |bot|
  bot.listen do |message|
    case message.text
    when '/start'
      bot.api.send_message(chat_id: message.chat.id, text: "Hello, #{message.from.first_name}")
    when '/stop'
      bot.api.send_message(chat_id: message.chat.id, text: "Bye, #{message.from.first_name}")
    end
  end
end

Basically, it will listen, and than you can send it messages and based on the parsed message.text you can define functions to call. For example, for rscrap I could define something like run_script(script). And the command would be: /run reddit. Which will execute my reddit script. The possibilities are endless.

The scripts

The scripts use nokogiri to parse a web page, and than return a URL which will be sent by the TelegramBot. They are also saved in the database so that when a new comic strip comes out, I know that it’s new. For reddit, I’m saving a timestamp as well, and I collect everything after that timestamp through the reddit API as JSON, and send it as a bundled message with shortified links to the posts using bit.ly.

The scraping is most of the times the same for every comic. Thus, there is a helper method for it. The script itself, is very short. For example, lets look at gunnerkrigg court.

require_relative '../rscrap'
require 'nokogiri'
require 'open-uri'

url = 'http://www.gunnerkrigg.com'
scrap = Rscrap.new
page = Nokogiri::HTML(open(url))
comic_id = page.css('img.comic_image')[0].select { |e| e if e[0] == 'src' }[0][1]
new_comic = "#{url}#{comic_id}"
scrap.send_new_comic(url, new_comic)

The interesting part of it is this bit: comic_id = page.css('img.comic_image')[0].select { |e| e if e[0] == 'src' }[0][1]. It extracts the URL for the comic image, and stores it as an “id” of the comic. This than, is sent as a message which Telegram will embed. There is no need to visit the web page, the image is in your feed and you can view it directly. Just like an RSS ready.

Cron

These scripts are best used in a cron job. The comics are usually running with a daily frequency, where as the reddit gatherer is running with an hour frequency. Basically, I’m receiving updates on an hourly basis if there are new posts by then. Running ruby from cron was a bit tricky. I’m using bundler for the environment, and came up with this:

0 6-23 * * * /bin/bash -l -c 'cd /home/<youruser>/rubyproj/rscrap && bundle exec ruby scripts/reddit.rb'
0 8,22 * * * /bin/bash -l -c 'cd /home/<youruser>/rubyproj/rscrap && bundle exec ruby scripts/gunnerkrigg.rb'
0 8,22 * * * /bin/bash -l -c 'cd /home/<youruser>/rubyproj/rscrap && bundle exec ruby scripts/aws_blog.rb'
0 5,23 * * * /bin/bash -l -c 'cd /home/<youruser>/rubyproj/rscrap && bundle exec ruby scripts/goblinscomic.rb'
0 6,20 * * * /bin/bash -l -c 'cd /home/<youruser>/rubyproj/rscrap && bundle exec ruby scripts/xkcd.rb'
0 7,19 * * * /bin/bash -l -c 'cd /home/<youruser>/rubyproj/rscrap && bundle exec ruby scripts/commitstrip.rb'
0 8 * * * /bin/bash -l -c 'cd /home/<youruser>/rubyproj/rscrap && bundle exec ruby scripts/sequiential_art.rb'

And a telegram message for all these things, looks like this: Reddit: TelegramIMReddit Comics: TelegramIMComics

Conclusion

That’s it folks. Adding a new scraper is easy. I added the aws blog as a new entry as well by just copying the comics scripts. And I’m also getting Weather Reports delivered every morning to me.

Have fun. Any questions, please feel free to leave a comment!

Thanks, Gergely.

17 Sep 2016, 00:00

Budget Home Theather with a Headless Raspberry Pi and Flirc for Remote Controlling

Intro

Hello folks.

Today, I would like to tell you about my configuration for a low budget Home Theater setup.

My tools are as follows:

TL;DR

Use Flirc for remote control, omxplayer for streaming the movie from an SSD on a headless PI controller via SSH and enjoy a nice, cold Lemon - Menta beer.

Flirc

First, the remote control. So, I like to sit in my couch and watch the movie from there. I hate getting up, or having a keyboard at arm length to control the pi. Flirc is a very easy way of doing just that with a simple remote control.

It costs ~$22 and is easy to setup. Works with any kind of remote control. Setting up key bindings for the control, is as simple as starting the Flirc software and pressing buttons on the remote to map to keyboard keys. Now, my pi is running headless, and the Flirc binary isn’t quite working with raspbian; so to do the binding, I just did that on my main machine. When I was done, I just plugged in the Flirc, and proceeded to setup the pi.

Raspberry Pi 2

The pi 2 is a small powerhouse. However, the SD card on which it sits is simply not fast enough. From time to time, I experienced lateness in sound, or stutter in video. So, instead of having the movie on the pi, I’m streaming through a faster SSD with SSHFS. For playing, I’m using omxplayer. With omxplayer, I had a few problems, because sound was not coming through the HDMI cable. A little bit of research lead me to this change in the pi’s boot config. Uncomment this line:

#hdmi_driver=2

After rebooting, I also, did this thing:

sudo apt-get install alsa-utils
sudo modprobe snd_bcm2835
sudo amixer -c 0 cset numid=3 2

This saved my bacon. The whole answer can be found here: Stackoverflow.

Once SSHFS was working, and HDMI received sound, I just executed this command: omxplayer -o hdmi /media/stream/my_movie.mkv. This told omxplayer to use the local HDMI connection for video output.

All this was from my computer through an SSH session so I never controlled the pi directly. Once done, I proceeded to sit down with a nice, cold Lemon - Menta beer and a remote control.

Once little gotcha – omxplayer is controlled through the buttons + (volume up), - (volume down), (stop, play), and q for quitting. Flirc is able to map any key combinations on a keyboard as well to any button on the remote. Combinations can be done by selecting a control key and pressing another key. So mapping + to the volume up button was by pressing shift and then ‘=’.

Wrapping Up

I enjoyed the movie while being able to adjust the volume, or pause it, when my popcorn was ready, and close the player when the movie was done. There are a number of other ways to do this, like using kodi + yatse. Which lets you remote control a media software with your mobile phone. But I’m using the pi for a number of other things and the GUI is rather resource heavy.

There you have it folks. Might not be the easiest setup, but it’s pretty awesome anyways.

Cheers, Gergely.

19 Aug 2016, 00:00

Always Go with []byte

Another quick reminder… Always go with []byte if possible. I said it before, and I’m going to say it over and over again. It’s crucial.

Here is a little code from exercism.io. First, with strings:

package igpay

import (
    "strings"
)

// PigLatin translates reguler old English into awesome pig-latin.
func PigLatin(in string) (ret string) {
    for _, v := range strings.Fields(in) {
        ret += pigLatin(v) + " "
    }

    return strings.Trim(ret, " ")
}

func pigLatin(in string) (ret string) {
    if strings.IndexAny(in, "aeiou") == 0 {
        ret += in + "ay"
        return
    }

    for i := 0; i < len(in); i++ {
        vowelPos := strings.IndexAny(in, "aeiou")

        if (in[0] == 'y' || in[0] == 'x') && vowelPos > 1 {
            vowelPos = 0
            ret = in
        }
        if vowelPos != 0 {
            adjustPosition := vowelPos

            if in[adjustPosition] == 'u' && in[adjustPosition - 1] == 'q' {
                adjustPosition++
            }

            ret = in[adjustPosition:] + in[:adjustPosition]
        }
    }
    ret += "ay"
    return
}

Than with []byte:

package igpay

import (
    // "fmt"
    "bytes"
)

// PigLatin translates reguler old English into awesome pig-latin.
func PigLatin(in string) (ret string) {
    inBytes := []byte(in)
    var retBytes [][]byte
    for _, v := range bytes.Fields(inBytes) {
        v2 := make([]byte, len(v))
        copy(v2, v)
        retBytes = append(retBytes, pigLatin(v2))
    }

    ret = string(bytes.Join(retBytes, []byte(" ")))
    return
}

func pigLatin(in []byte) (ret []byte) {
    if bytes.IndexAny(in, "aeiou") == 0 {
        ret = append(in, []byte("ay")...)
        return
    }

    for i := 0; i < len(in); i++ {
        vowelPos := bytes.IndexAny(in, "aeiou")

        if (in[0] == 'y' || in[0] == 'x') && vowelPos > 1 {
            vowelPos = 0
            ret = in
        }
        if vowelPos != 0 {
            adjustPosition := vowelPos

            if in[adjustPosition] == 'u' && in[adjustPosition - 1] == 'q' {
                adjustPosition++
            }

            in = append(in[adjustPosition:], in[:adjustPosition]...)
            ret = in
            // fmt.Printf("%s\n", ret)
        }
    }
    ret = append(ret, []byte("ay")...)
    return
}

And than,the benchmarks of course:

BenchmarkPigLatin-8          	  200000	     10688 ns/op
BenchmarkPigLatinStrings-8   	  100000	     15211 ns/op
PASS

The improvement is not massive in this case, but it’s more than enough to matter. And in a bigger, more complicated program, string concatenation will take a LOT of time away.

In Go, the bytes package has a 1-1 map compared to the strings packages, so chances are, if you are doing strings concatenations you will be able to port that piece of code easily to []byte.

That’s all folks.

Happy coding, Gergely.

16 Aug 2016, 00:00

Global variable for never changing regex

Quick reminder. If you have a never changing regex in Go, do NOT put it into a frequently called function. ALWAYS put it into a global variable. I’ll show you why.

Benchmark for code with a variable in a frequently called function:

BenchmarkNumber-8     	   30000	     41633 ns/op
BenchmarkAreaCode-8   	   50000	     27736 ns/op
BenchmarkFormat-8     	   50000	     29263 ns/op
PASS
ok  	_/phone-number	5.110s

Benchmark for code with the same variable outside in a global scope:

BenchmarkNumber-8     	  300000	      5618 ns/op
BenchmarkAreaCode-8   	  500000	      3884 ns/op
BenchmarkFormat-8     	  300000	      4696 ns/op
PASS
ok  	_/phone-number	5.197s

Notice the magnitude change in ns/op! That’s something to keep an eye out for.

Thanks for reading! Cheers, Gergely.

13 Aug 2016, 00:00

Drupal missing ToolBar and settings not saving

Hi folks.

Quick gotcha, when working with Drupal. If you just freshly installed it, and everything seems to work fine, and yet you are experiencing things like, the admin toolbar is randomly disappearing, or configuration is not saved; than you might not have modrewrite enabled on your apache server.

Because, by default, Drupal has clean url enabled, that needs URL rewriting on apache.

So, step one.

Have this in your .htaccess file:

<IfModule mod_rewrite.c>
  RewriteEngine on
  ... # and than a bunch of rewrite rules according to your leisure

Than look up this line in your httpd.conf file and remove the prefix ‘#’.

#LoadModule rewrite_module libexec/apache2/mod_rewrite.so

That is all. From there on, everything should work. If, you don’t want the clean url setting, yet you can’t disable it, and don’t want to restart the server and edit the settings.php file; use drush like this:

drush vset clean_url 0 --yes

This should disable it and bust the cache in the process so it’s immediately visible.

That is all folks.

Cheers, Gergely.

28 Jul 2016, 00:00

Jenkins Best Practices Talk

Hi folks.

I wanted to take the time to share with you a talk that I recently did.

The slides and the source I used, can be found here: Github.

And then, there is also a docker image which contains all the plugins, job configurations and all the practices which I did during the talk. Please feel free to have a go with it. DockerHub - Jenkins Best Practices.

For easy access and reading, here are the slides on Slideshare: Jenkins Best Practices Slides.

I, gladly answer any questions which should arise.

Thanks! Gergely.

12 Jul 2016, 00:00

Ruby Sieve

Though it could be done better, I’m sure, but I’m actually pretty satisfied with this one. It loops only twice as opposed to filtered ranges and whatnot other solutions to the sieve. I was thinking of rather creating a list and deleting elements from it, but that’s already three loops.

Maybe I’ll do a benchmark later on more solutions.

# Sieve contains a function to return a set of primes
class Sieve
  def initialize(n)
    @n = n
  end

  # Returns a list of primes up to a certain limit
  # @param n limit
  # @return list of primes
  def primes
    marked = []
    primes = []
    (2..@n).each do |e|
      unless marked.include?(e)
        primes.push e
        (e..@n).step(e) { |s| marked.push s }
      end
    end
    primes
  end
end

Cheers, Gergely.

12 Jul 2016, 00:00

Simple hook to rid of trouble

Hi folks.

This is but a simple git hook to run a test in order to ensure you can push. It also ignores the vendor folder if you happen to have on in your directory.

Edit the file under .git/hooks/pre-push.sample and add this at the end before the exit 0.

go test $(go list ./... |grep -v vendor)
RESULT=$?
if [ $RESULT -ne 0 ]; then
    echo "Failed test run. Disallowing push."
    exit 1
fi

After this, rename the file to pre-push removing the .sample from it.

If you now, mess something up, you should see something like this before your push:

# github.com/Skarlso/goprogressquest
./create.go:40: undefined: sha1 in sha1.Sum
./create.go:41: undefined: fmt in fmt.Sprintf
./create.go:115: undefined: json in json.Unmarshal
./create.go:130: undefined: json in json.Unmarshal
FAIL	github.com/Skarlso/goprogressquest [build failed]
Failed test run. Disallowing push.
error: failed to push some refs to 'git@github.com:Skarlso/goprogressquest.git'

That is all.

Cheers, Gergely.

12 Jun 2016, 00:00

How to do Google sign-in with Go

Hi folks.

Today, I would like to write up a step - by - step guide with a sample web app on how to do Google Sign-In and authorization.

Let’s get started.

EDIT: A sample project of this, and Part 2, can be found here or here.

Setup

Google OAuth token

First what you need is, to register your application with Google, so you’ll get a Token that you can use to authorize later calls to Google services.

You can do that here: Google Developer Console. You’ll have to create a new project. Once it’s done, click on Credentials and create an OAuth token. You should see something like this: “To create an OAuth client ID, you must first set a product name on the consent screen.”. Go through the questions, like, what type your application is, and once you arrive at stage where it’s asking for your application’s name – there is a section asking for redirect URLs; there, write the url you wish to use when authorising your user. If you don’t know this yet, don’t fret, you can come back and change it later. Do NOT use localhost. If you are running on your own, use http://127.0.0.1:port/whatever.

This will get you a client ID and a client secret. I’m going to save these into a file which will sit next to my web app. It could be stored more securely, for example, in a database or a mounted secure, encrypted drive, and so and so forth.

Your application can now be identified through Google services.

The Application

Libraries

Google has a nice library to use with OAuth 2.0. The library is available here: Google OAth 2.0. It’s a bit cryptic at first, but not to worry. After a bit of fiddling you’ll understand fast what it does. I’m also using Gin, and Gin’s session handling middleware Gin-Session.

Setup - Credentials

Let’s create a setup which configures your credentials from the file you saved earlier. This is pretty straightforward.

// Credentials which stores google ids.
type Credentials struct {
    Cid string `json:"cid"`
    Csecret string `json:"csecret"`
}

func init() {
    var c Credentials
    file, err := ioutil.ReadFile("./creds.json")
    if err != nil {
        fmt.Printf("File error: %v\n", err)
        os.Exit(1)
    }
    json.Unmarshal(file, &c)
}

Once you have the creds loaded, you can now go on to construct the OAuth client.

Setup - OAuth client

Construct the OAuth config like this:

conf := &oauth2.Config{
  ClientID:     c.Cid,
  ClientSecret: c.Csecret,
  RedirectURL:  "http://localhost:9090/auth",
  Scopes: []string{
    "https://www.googleapis.com/auth/userinfo.email", // You have to select your own scope from here -> https://developers.google.com/identity/protocols/googlescopes#google_sign-in
  },
  Endpoint: google.Endpoint,
}

It will give you a struct which you can then use to Authorize the user in the google domain. Next, all you need to do is call AuthCodeURL on this config. It will give you a URL which redirects to a Google Sign-In form. Once the user fills that out and clicks ‘Allow’, you’ll get back a TOKEN in the code query parameter and a state which helps protect against CSRF attacks. Always check if the provided state is the same which you provided with AuthCodeURL. This will look something like this http://127.0.0.1:9090/auth?code=4FLKFskdjflf3343d4f&state=lhfu3f983j;asdf. Small function for this:

func getLoginURL(state string) string {
    // State can be some kind of random generated hash string.
    // See relevant RFC: http://tools.ietf.org/html/rfc6749#section-10.12
    return conf.AuthCodeURL(state)
}

Construct a button which the user can click and be redirected to the Google Sign-In form. When constructing the url, we must do one more thing. Create a secure state token and save it in the form of a cookie for the current user.

Random State and Button construction

Small piece of code random token:

func randToken() string {
	b := make([]byte, 32)
	rand.Read(b)
	return base64.StdEncoding.EncodeToString(b)
}

Storing it in a session and constructing the button:

func loginHandler(c *gin.Context) {
    state = randToken()
    session := sessions.Default(c)
    session.Set("state", state)
    session.Save()
    c.Writer.Write([]byte("<html><title>Golang Google</title> <body> <a href='" + getLoginURL() + "'><button>Login with Google!</button> </a> </body></html>"))
}

It’s not the nicest button I ever come up with, but it will have to do.

User Information

After you got the token, you can construct an authorised Google HTTP Client, which let’s you call Google related services and retrieve information about the user.

Getting the Client

Before we construct a client, we must check if the retrieved state is still the same compared to the one we provided. I’m doing this before constructing the client. Together this looks like this:

func authHandler(c *gin.Context) {
    // Check state validity.
    session := sessions.Default(c)
    retrievedState := session.Get("state")
    if retrievedState != c.Query("state") {
        c.AbortWithError(http.StatusUnauthorized, fmt.Errorf("Invalid session state: %s", retrievedState))
        return
    }
    // Handle the exchange code to initiate a transport.
  	tok, err := conf.Exchange(oauth2.NoContext, c.Query("code"))
  	if err != nil {
  		c.AbortWithError(http.StatusBadRequest, err)
          return
  	}
    // Construct the client.
    client := conf.Client(oauth2.NoContext, tok)
    ...

Obtaining information

Our next step is to retrieve information about the user. To achieve this, call Google’s API with the authorised client. The code for that is:

...
resp, err := client.Get("https://www.googleapis.com/oauth2/v3/userinfo")
if err != nil {
    c.AbortWithError(http.StatusBadRequest, err)
    return
}
defer resp.Body.Close()
data, _ := ioutil.ReadAll(resp.Body)
log.Println("Resp body: ", string(data))
...

And this will yield a body like this:

{
  "sub": "1111111111111111111111",
  "name": "Your Name",
  "given_name": "Your",
  "family_name": "Name",
  "profile": "https://plus.google.com/1111111111111111111111",
  "picture": "https://lh3.googleusercontent.com/asdfadsf/AAAAAAAAAAI/Aasdfads/Xasdfasdfs/photo.jpg",
  "email": "your@gmail.com",
  "email_verified": true,
  "gender": "male"
}

Parse it, and you’ve got an email which you can store somewhere for registration purposes. At this point, your user is not yet Authenticated. For that, I’m going to post a second post, which describes how to go on. Retrieving the stored email address, and user session handling with Gin and MongoDB.

Putting it all together

package main

import (
    "crypto/rand"
    "encoding/base64"
    "encoding/json"
    "io/ioutil"
    "fmt"
    "log"
    "os"
    "net/http"

    "github.com/gin-gonic/contrib/sessions"
    "github.com/gin-gonic/gin"
    "golang.org/x/oauth2"
    "golang.org/x/oauth2/google"
)

// Credentials which stores google ids.
type Credentials struct {
    Cid     string `json:"cid"`
    Csecret string `json:"csecret"`
}

// User is a retrieved and authentiacted user.
type User struct {
    Sub string `json:"sub"`
    Name string `json:"name"`
    GivenName string `json:"given_name"`
    FamilyName string `json:"family_name"`
    Profile string `json:"profile"`
    Picture string `json:"picture"`
    Email string `json:"email"`
    EmailVerified string `json:"email_verified"`
    Gender string `json:"gender"`
}

var cred Credentials
var conf *oauth2.Config
var state string
var store = sessions.NewCookieStore([]byte("secret"))

func randToken() string {
	b := make([]byte, 32)
	rand.Read(b)
	return base64.StdEncoding.EncodeToString(b)
}

func init() {
    file, err := ioutil.ReadFile("./creds.json")
    if err != nil {
        log.Printf("File error: %v\n", err)
        os.Exit(1)
    }
    json.Unmarshal(file, &cred)

    conf = &oauth2.Config{
        ClientID:     cred.Cid,
        ClientSecret: cred.Csecret,
        RedirectURL:  "http://127.0.0.1:9090/auth",
        Scopes: []string{
            "https://www.googleapis.com/auth/userinfo.email", // You have to select your own scope from here -> https://developers.google.com/identity/protocols/googlescopes#google_sign-in
        },
        Endpoint: google.Endpoint,
    }
}

func indexHandler(c *gin.Context) {
    c.HTML(http.StatusOK, "index.tmpl", gin.H{})
}

func getLoginURL(state string) string {
    return conf.AuthCodeURL(state)
}

func authHandler(c *gin.Context) {
    // Handle the exchange code to initiate a transport.
    session := sessions.Default(c)
    retrievedState := session.Get("state")
    if retrievedState != c.Query("state") {
        c.AbortWithError(http.StatusUnauthorized, fmt.Errorf("Invalid session state: %s", retrievedState))
        return
    }

	tok, err := conf.Exchange(oauth2.NoContext, c.Query("code"))
	if err != nil {
		c.AbortWithError(http.StatusBadRequest, err)
        return
	}

	client := conf.Client(oauth2.NoContext, tok)
	email, err := client.Get("https://www.googleapis.com/oauth2/v3/userinfo")
    if err != nil {
		c.AbortWithError(http.StatusBadRequest, err)
        return
	}
    defer email.Body.Close()
    data, _ := ioutil.ReadAll(email.Body)
    log.Println("Email body: ", string(data))
    c.Status(http.StatusOK)
}

func loginHandler(c *gin.Context) {
    state = randToken()
    session := sessions.Default(c)
    session.Set("state", state)
    session.Save()
    c.Writer.Write([]byte("<html><title>Golang Google</title> <body> <a href='" + getLoginURL(state) + "'><button>Login with Google!</button> </a> </body></html>"))
}

func main() {
    router := gin.Default()
    router.Use(sessions.Sessions("goquestsession", store))
    router.Static("/css", "./static/css")
    router.Static("/img", "./static/img")
    router.LoadHTMLGlob("templates/*")

    router.GET("/", indexHandler)
    router.GET("/login", loginHandler)
    router.GET("/auth", authHandler)

    router.Run("127.0.0.1:9090")
}
}

This is it folks. I hope this helped. Any comments or advices are welcomed.

Google API Documentation

The documentation to this whole process, and MUCH more information can be found here: Google API Docs.

Thanks for reading, Gergely.

17 Apr 2016, 00:00

Minecraft world automatic backup to AWS S3 bucket - Part 2 (Custom functions)

Hi folks.

Got an update for the backup script. This time, you’ll have the ability to implement your own upload capabilities. I provide a mock implementation for the required functions.

Here is the script again, now modified and a bit cleaned up. I hope it’s helpful.

#!/bin/bash

if [[ -t 1 ]]; then
    colors=$(tput colors)
    if [[ $colors ]]; then
        RED='\033[0;31m'
        LIGHT_GREEN='\033[1;32m'
        NC='\033[0m'
    fi
fi

if [[ -z ${MINECRAFT_BUCKET} ]]; then
    printf "Please set the env variable %bMINECRAFT_BUCKET%b to the s3 archive bucket name.\n" "${RED}" "${NC}"
    exit 1
fi

if [[ -z ${MINECRAFT_ARCHIVE_LIMIT} ]]; then
    printf "Please set the env variable %bMINECRAFT_ARCHIVE_LIMIT%b to limit the number of archives to keep.\n" "${RED}" "${NC}"
    exit 1
fi

if [[ -z ${MINECRAFT_WORLD} ]]; then
    printf "Please set the env variable %bMINECRAFT_WORLD%b to specify what world to back-up.\n" "${RED}" "${NC}"
    exit 1
fi

backup_world=${MINECRAFT_WORLD}
backup_bucket=${MINECRAFT_BUCKET}
backup_limit=${MINECRAFT_ARCHIVE_LIMIT}
archive_name="${backup_world}-$(date +"%H-%M-%S-%m-%d-%Y").zip"

function create_archive {
    printf "Creating archive of %b${backup_world}%b\n" "${RED}" "${NC}"
    zip -r $archive_name $backup_world
}

function amazon_bak {

    create_archive

    printf "Checking if bucket has more than %b${backup_limit}%b files already.\n" "${RED}" "${NC}"
    content=( $(aws s3 ls s3://$backup_bucket | awk '{print $4}') )

    if [[ ${#content[@]} -eq $backup_limit || ${#content[@]} -gt $backup_limit  ]]; then
        echo "There are too many archives. Deleting oldest one."
        # We can assume here that the list is in cronological order
    	printf "%bs3://${backup_bucket}/${content[0]}\n%b" "${RED}" "${NC}"
        aws s3 rm s3://$backup_bucket/${content[0]}
    fi

    printf "Uploading %b${archive_name}%b to s3 archive bucket.\n" "${RED}" "${NC}"
    state=$(aws s3 cp $archive_name s3://$backup_bucket)

    if [[ "$state" =~ "upload:" ]]; then
        printf "File upload %bsuccessful%b.\n" "${LIGHT_GREEN}" "${NC}"
    else
        printf "%bError%b occured while uploading archive. Please investigate.\n" "${RED}" "${NC}"
    fi
}

function custom {
    if [[ -e custom.sh ]]; then
        source ./custom.sh
    else
        echo "custom.sh script not found. Please implement the apropriate functions."
        exit 1
    fi

    echo "Checking for the number of files. Limit is: $backup_limit."
    files=( $(list) )
    if [[ ${#files[@]} -eq $backup_limit || ${#files[@]} -gt $backup_limit ]]; then
        echo "Deleting extra file."
        delete ${files[0]}
        if [[ $? != 0 ]]; then
            printf "%bFailed%b to delete file. Please investigate failure." "${RED}" "${NC}"
            exit $?
        fi
    fi

    echo "Zipping world."
    create_archive

    echo "Uploading world."
    upload $archive_name

    if [[ $? != 0 ]]; then
        printf "%bFailed%b to upload archive. Please investigate the error." "${RED}" "${NC}"
        exit $?
    fi

    printf "Upload %bsuccessful%b" "${LIGHT_GREEN}" "${NC}"
}

function help {
    echo "Usage:"
    echo "./backup_world [METHOD]"
    echo "Exp.: ./backup_world aws|./backup_world custom|./backup_world dropbox"
    echo "Each method has it's own environment properties that it requires."
    echo "Global: MINECRAFT_WORLD|MINECRAFT_BUCKET|MINECRAFT_ARCHIVE_LIMIT"
    echo "Custom: Have a file, called 'custom.sh' which is sourced."
    echo "Implement these three functions: upload | list | delete."
    echo "upload -> should return exit code 0 on success, should return exit code 1 on failure."
    echo "list -> should return a list of cronologically ordered items."
    echo "delete -> should return exit code 0 on success, should return exit code 1 on failure."
}

case $1 in
    aws )
        amazon_bak
        ;;
    custom )
        custom
        ;;
    * )
        help
        ;;
esac

And here is the sample implementation for the custom upload functionality.

#!/bin/bash

function upload {
    echo "uploading"
    local result=0
    return $result
}

function delete {
    echo "deleting $1"
    local result=0
    return $result
}

function list {
    local arr=("file1" "file2" "file3")
    echo "${arr[@]}"
}

Thanks for reading!

Gergely.