Tuesday, March 4, 2014

Learning Go with Martini - Working with MongoDB

This is the second post in a series of posts on creating Go based web applications/APIs. If you missed the first post Learning Go with Martini - The Basics Go check out and then come back to this post

Intro

Now that I understand how to handle GET requests its time to add some database interaction. In this post I will walk you through how to add a datgabase connectivity middleware to each request. Once I have that in place I will convert the /attributes/:resource into a ‘real’ GET, one that reads from a database and returns the results fo the query. I will add a POST route, storing the data that was ‘POSTed’ to the app in a MongoDB collection.

Setup

Before we get started you will need to have MongoDB installed or have access to an instance of MongoDB. To install it locally visit this MongoDB’s download page. If you’d like to use MongoDB in the ‘cloud’ checkout Mongolab.

Also, if you haven’t already done so you’ll need to have a functional Go language environment. Visit the golang.org’s install page and follow their instructions for your platform of choice.

Lastly, The code for the beginning of this post is stored Gist style and can be found here.

Creating the Middleware

Before I start working with the database I need a way to ensure that each handler functions has access to the database connection. To do that I’m going to create a middleware function that middleware that will make a database connection available to each request handler. Before I jump right into that piece of middleware I'm going to start off with a 'Hello Middleware' function.

Step 0. The Hello Middleware Edition

To make sure I have the handler working correctly I’m going to start off easy. I’m going to create a handler function that simply writes out “Someday I will be a MongoDB connection”. Here’s what the skeletal function looks like.

func Mongo() martini.Handler {
  return func (c martini.Context ) {
    fmt.Println("Someday I will be a MongoDB connection")
    c.Next()
  }
}

The middleware returns a martini.Handler as an anonymous function. All my function does is write a string back and call c.Next(). The Next call yields until after all of the other handlers have executed.

To add the function into the request stack I need to add the line m.Use( Mongo() ) after the martini.Classic() call in the main function.

Now, each and every request will make a call to the Mongo() handler. If you download the gist above and add the function and the m.Use call to it you’ll be able to build and run the code. Do a few requests, valid and invalid. In the window where you started the service you should see the output of the middleware for each and every request.

Step 1. Setting up for MongoDB

For the MongoDB connection I will be using the mgo library. Before you can install it you’ll need to have the bazaar tool installed. So, if you don't already have bazaar installed visit the project's go to the project’s download page. If you are on a mac and have homebrew installed you can install both by running the following:

brew install bazaar
go get labix.org/v2/mgo

Now that I have mgo installed I’m going to change the Mongo function so that it adds a database session to each request. First I need to add the mgo package to my import statement, labix.org/v2/mgo, to my imports list. After updating the imports list I updated the function so that it now looks like this:

func Mongo() martini.Handler {
  session, err := mgo.Dial( "localhost/goattrs" )
  if err != nil {
    panic( err )
  }

  return func (c martini.Context ) {
    reqSession := session.Clone()
    c.Map( reqSession.DB( "goattrs" ) )
    defer reqSession.Close()

    c.Next()
  }
}

It looks quite a bit different that the 'Hello' version. The first line in the function uses the mgo.Dial function to create the MongoDB connection, you can think of it as a database connection. The Dial function has two return values the session object and an error object if necessary. If an error occurrs then we 'panic', no sense on continuing if we have no database conection. If there was no error an anonymous function is created with a single parameter, a martini.Conext. The context object is what I will use to make the database session available to the handlers. Before that I create a clone of the original object so that each request can close its databse connection. The defer call, puts the reqSession.Close() call ‘on hold’ until the end of the handler function. Once the function has ended reqSessoin.Close() is called and the requests database connection is closed.

Converting the GET /attributes/:resource Handler

Now that the Mongo() middleware is in place I can convert the handler over to use it. First, I need to add the parameter db *mgo.Database to the getAttributes function. Now that I have a pointer to the database conection I can look for attributes for the given resource. I am going to change the if statement so that if the results of my query are not null, I will return a 200 along with a JSON version of the results. If the query results are null, I will return a 404 and an errorMsg JSON object. Here’s the updated function.

func getAttributes( params martini.Params, writer http.ResponseWriter, db *mgo.Database) (int, string) {
  resource :=  strings.ToLower( params["resource"] )
  writer.Header().Set("Content-Type", "application/json")

  var attrs []resourceAttributes
  db.C("resource_attributes").Find(bson.M{"resource": resource }).All(&attrs);

  if attrs != nil {
    return http.StatusOK, jsonString( attrs )
  } else {
    return http.StatusNotFound, jsonString( errorMsg{"No attributes found for the resource: " + resource} )
  }
}

The meat of the changes are the two lines that handle the database query. First the var attrs []resourceAttributes line declares the array that will contain the returned data if any is found. The next line is where all the magic happens. The db.C(“resource_attributes”) tells the driver that we want to work with the resource_attributes collection. If you aren’t familiar with MongoDB think of a collection like a table in a relational database. The Find call, similar to a select in rdbms world, takes a map as its parameter. Here I’m looking for an objet that has a resource property equal to the lower case version of the resource that was passed in. The .All call will take the results of my query and store them in the attrs array. If no matches are found then the attrs array will be nil. Now when I make a the /attributes/tv call I get an empty result set back. To fix that, I'm going to add a POST handler to create attributes.

Creating the POST /attributes/:resource Handler

I’m going to start off by setting up a new route in the attr-server.go file. The route will call a second function in the attribute-routes.go file that will handle the heavy lifting of creating a new attribute. Here’s what the new route looks like:

m.Post("/attributes/:resource", addAttribute  )

Not much going on here, now any post to the /attributes/:resource url will be handled by the addAttribute function. The first version of the function will be a simple placeholder, one that shows me that I have the POST support wired up correctly. Here’s the function:

func addAttribute( params martini.Params, writer http.ResponseWriter, db *mgo.Database) (int, string) {
  resource :=  strings.ToLower( params["resource"] )
  writer.Header().Set("Content-Type", "application/json")

  return http.StatusOK, "POST placeholder " + resource
}

Nothing new happening here other than the fact it handles a POST instead of a GET. After I recompiled I used the following curl command to test the code:

curl -X POST [http://localhost:3000/attributes/tv](http://localhost:3000/attributes/tv)

which returned POST placeholder tv. Now I know that the route and handler are wired up correctly, its time to add code to process the JSON data.

Adding JSON Support

In order to support the JSON POST I'm going to use another package, this one comes from the martini-contrib ecosystem, martini-contrib/binding package. The binding package will allow me to tell martini to take the submitted JSON object and magically load it into an attribute struct. To add the support I need to modify the route definition in the main() function. Specifically I need to tell the binding package which struct to bind the incoming JSON to. The updated definition looks like this:

m.Post("/attributes/:resource", binding.Json( attribute{} ), addAttribute  )

I also need to update the handler function to support the new attribute parameter.

func addAttribute( attr attribute, params martini.Params, writer http.ResponseWriter, db *mgo.Database)

Now the handler can access the posted JSON data through the attr paramter. Running the curl POST parameter with JSON data will return a JSON string that represents the submitted data.

curl -XPOST  -d '{
  "name":"location”,
  "type":"string",
  "description":"Where the TV is located, which area of a facility is it loaded in",
  "required":"true"}' 
  -H "Content-Type: application/json" 
  http://localhost:3000/attributes/tv

Returns the following:

{"name":"location","type”:"string","description":"Where the TV is located, which area of a facility is it loaded in","required":false}

The code is now parsing the JSON and its being converted into something I can use. Thats great, but what happens if I run this curl command, what do I get then?

curl -XPOST http://localhost:3000/attributes/tv

The curl command returns an empty atttribute struct. I don’t want that to happen. I want the name field to be required on all POSTs. If nothing is supplied for the type and description fields I want them to default to an empty string. Of course you can do this after the new attribute struct is given to the handler function but I want this to be done outside of the handler code. Thankfully, the binder package has that functionality built in.

Adding Validation

The martini-contrib/binding package defines a Validator interface that contains one method declaration, Validate(Errors, http.Request). In order for me to add my validation requirement I will need to create a method for the attribute struct.

func (attr *attribute) Validate( errors *binding.Errors, req *http.Request ) {
  if attr.Name == "" {
    errors.Overall["missing-requirement"] = "name is a required field";
  }

  if attr.DataType == "" {
    attr.DataType = "string"
  }           
}

In addition to the interface, the package also has an Errors struct that has two maps in it, Overall and Fields. I’m using the Overall map to report validation errors. Specifically I'm using “missing-requirement” as the key to the error message when no name is either empty or nil. While I’m checking for the required field I will also check to see what data type was provided in the JSON object’s type property. If nothing was provided I default it to string.

After my validation is complete I can check to see if there were any validation errors by calling errors.Count(). If that is greater than zero then I know a validation error occurred. To let the client know I will send a response with 409 and a JSON string representation of the ErrorMsg struct. Otherwise, I all return a 200 and a JSON string representation of the new attribute. Here’s latest version of the POST handler.

func addAttribute( attr attribute, err binding.Errors, params martini.Params, writer http.ResponseWriter, db *mgo.Database) (int, string)  {                   

  writer.Header().Set("Content-Type","pplication/json")

  if err.Count() > 0 {
    return http.StatusConflict, jsonString( errorMsg{ err.Overall["missig-reuiement"] } )
  }

  return http.StatusOK jsonString( attr )
}

You may be wondering why I chose to return HTTP Status Code 409. I chose it after reading what w3c had to say. Basically it boiled down to this line:

This code is only allowed in situations where it is expected that the user might be able to resolve the conflict and resubmit the request. The response body SHOULD include enough information for the user to recognize the source of the conflict

After grabbing the value of the resource parameter I create a query that will be used to find the document. Once the query is defined I create a mgo.Change struct. This is how I will be able modify the data if it exists or insert it if it doesn't (this is done by setting Upsert to true). The Update is what will do the actual changing of the document. If you aren’t familiar with MongoDB what my statement says is if a matching document is found only add the attribute to the attributes array if it isn’t already in there. If it already exists it will not be added to the array.

Now that I have my query and change objects in place its time to run the database update. The Apply call returns two results ChangeInfo and a Go error struct. Since I don’t care about the ChangeInfo struct, I put a _ in its place. Obviously I’m interested in any errors that occur so I grab that value and check for an error. If an error does occur I will send an errorMsg struct back to the caller. If no error occurs then the I send back a 200 status and an empty object.

The Finishing Touches

Now that I’m storing data I’ve noticed that the GET /attributes/:resource returns a JSON Array when it should return an object. I tweaked to the getAttributes section of code that returns the results of the query to look like this:

attrs := resourceAttributes{}
err   := db.C("resource_attributes").Find(bson.M{"resource": resource }).One(&attrs)

if err == nil {

I changed the attrs declaration to be a single object instead of an array, you’ll see why on the next line. I also changed from the All() function to the One() call since only one document. If a record isn’t found it, the One() will return an error object so I’ve changed the if statement to use the presence of an error to determine which JSON string to return.

Summary

I've changed the attributes services to read from a MongoDB database. I did that by adding the Mongo() middleware function so that the handler functions can interact with the database. I added a POST route that illustrated how to use the martini-contrib/binding package to bind the incoming JSON to the attribute struct. After finishing up the POST method I went back to the GET handler to change the returned data from an array to a single object.

In my next post I'm going to create my first Go package. The package will be used to interact with the etcd to retrieve the database configuration information.

Resources

Packages

Other Links

Github and Gists

Previous Posts in the Series

Wednesday, February 26, 2014

Learning Go with a Martini - The Basics

Intro

I’m working with my son to build a system to manage all sorts of data, files and devices. Our apps will need to run on OSX, Windows, Linux, and possibly Raspberry Pi machines. In addition to multiple platform support, our apps need to be fast, we will eventually be working with near real time data. So I started looking for a language that would fit the bill and Go caught my eye. Why? For one, it compiles, I mean it really compiles down to an actual executable, not byte code. Another reason is some of the speed benchmarks I saw said go was second only to C for speed. Also, if I can’t get Go to be fast enough I can load C libraries too. Lastly, Go appears to have a vibrate third party package ecosystem. So I thought I’d give Go a try. Since I most of my day job work is in web development I thought I’d learn Go while I build out a web service and share my experience with others who may be curious about Go.

Goal

In this post we will start building out a API to manage attributes. Every resource in our system will have attributes and these attributes will be different per each type of resource. The API will provide the necessary functionality to assign,remove and list attributes the available attributes assigned to a particular resource. In this post we will set up the Go development environment and create the basics for the GET /attributes/:resource.

Setup

Installing and Configuring Go

Installing Go is pretty straight forward, just follow the instructions on golang.org’s install page. To ensure Go is setup correctly in your development environment run go version and you should see output similar to this, go version go1.1.2 darwin/386

After installing Go I need to do a few tasks to setup my Go work space. The Go tools are created to work with a certain directory structure, mainly a ‘home’ directory that contains three subdirectories: src, where my source files will go; pkg, where any of 3rd party packages I install will live; and bin, where any executables I install will live. the directories have been created with in the ‘home’ directory a GOPATH environment variable should be set to that directory. In my environment my $GOPATH variable is set to ~/src/go. For a more detailed overview of the workspace layout read the “How to Write Go Code” page on the golang.org site.

Editor Setup

I use emacs to write the vast majority of my code. I would imagine that my editor choice is not the norm for most of you reading this post so I’m going to add a few other editors that have Go support.

Eclipse goclipse
Emacs go-mode.el
Sublime go plugin
Vim go language vim support

Follow the instructions and your editing environment will be ready to ‘Go’.

Installing Third Party Packages

The last step before we start writing code is to install the lone third-party package that will use to create the attributes API. Go makes it easy to install the packages by providing the go get tool. As you can see below you just add the package you wish to install after go get. When the install completes the package can be found by running ls $GOPATH/pkg in a terminal window.

go get github.com/codegangsta/martini

Now that I have Go installed, my work environment setup including my editor, installed martini, I’m ready to start coding. I will be writing the code for this blog series in the $GOPATH/src/github.com/rippinrobr/martini-to-go-posts directory.

Writing the Code

A Basic HTTP Server

One of the reasons why I chose to get familiar with Martini is it allows you to get a functional web server up and running with about 10 lines of code. So to make sure I have everything in place and I can respond to an HTTP GET / request I’m going to start with a very basic app. This app will respond to the HTTP GET / request by returning a string that reads "Where are the attributes?!?!” Here’s the code

package main

// loading in the Martini package
import "github.com/codegangsta/martini"

func main() {
  // if you are new to Go the := is a short variable declaration
  m := martini.Classic()

  // the func() call is creating an anonymous function that retuns a stringa
  m.Get("/", func() string {
    return "Where are the attributes?!?!"
  })

  m.Run()
}

Before you run it let’s go over the code. The first line defines the main package, all Go code must be in a package. This code belongs to the main package which is a special package in Go. The main package is where the main function must be for all executable Go projects. Once I’ve defined the package I need to let go know what packages I want to import. In this code I am only importing the the martini package. This statement makes the martini functions, structs, and interfaces available to my code. To call anything in this package I need to preface the call with martini. In this example I’m only using one function from martini, martini.Classic().

The martini.Classic() call creates the classic martini object that I will use to declare the supported routes and start the service. This particular line makes use of the ‘short variable declaration’ syntax. The := determines the type of the object, var, etc.. on the right side and creates a variable of that type on the left side of things. The := syntax can only be used within the body of a function.

Now that I have created my martini object I can start setting up to handle our HTTP GET / request. Adding a route is to handle is pretty straight forward. For this simple example I declare the route I want to respond to “/“ and I am using an anonymous function to handle the requests. If you are new to Go the string that follows func() is the return value of the function.

m.Get( “/”, func() string {
  return “Where are the attributes?!?!”
})

Since this is the only route I’ve declared any other route sent to the service will result in a 404 error. The last bit of code is the Run() call which starts the HTTP server. By default the server will listen on port 3000, if you want to change that port set the PORT environment variable to the new value and restart the server. Martini automatically looks for the PORT variable.

The next step is to actually see the code in action. The easiest way to do that is by calling go run.

go run attr-server.go

The go run command will compile and run the application. If there are no errors you should see a message [martini] listening on host:port :8000. My server is running on port 8000 because I've set my PORT environment variable to 8000.

Now, to make sure that the response is what I expect. I'm going to run:

curl http://localhost:8000/

And I should see should see Where are the attributes?!?! string returned. As you can see it is pretty simple to get a basic HTTP server up and running.

GET /attributes/:resource

Ok, now that I’ve shown you the basics of martini its time to build out our first ‘real’ route. Remember, the goal of this service is to track a resource’s attributes. Resources can be anything from a TV, scoreboard, ad boards, etc.. Each of these will have its own set of attributes. For this blog post the /attributes/:resource route need to do the following:

  1. If the resource requested is a TV then we will return a JSON object with all the attributes assigned to a TV and the HTTP Status code of 200
  2. If the resource is not a TV then we will return a JSON error object that we will define and a status code of 404.
New Packages

I am going to need to include a few more packages to the code to meet my needs. The first import is the net/http package. I’m importing this package so I can use http.StatusOK instead of the number 200, it will make the code a little more readable. The next new package is the strings package. I’m using this package so I can convert the requested resource to lower case so that I can ensure my string comparison is comparing the input in the same case as my test string.

import (
  “net/http” // this will allow us to use http.StatusOK and http.StatusNotFound instead of 200 and 404
  “strings”  // I’m adding this so I can ensure that we are comparing lower case   strings.
  “github.com/codegangsta/martini” 
)

Notice that the import call has changed. When there are multiple packages to import you can group them together as I have above or you could use an import call for each one. Either way works but I believe the way I have it here is the more idiomatic Go way.

The New GET Handler

The next change is that I’ve replaced the m.Get call we had previously with this one:

m.Get("/attributes/:resource", func( params martini.Params ) (int, string) {
  resource :=  strings.ToLower( params["resource"] )

  if resource  == "tv" {
    return http.StatusOK, “a TV attributes object will be returned here"
  } else {
    return http.StatusNotFound, "JSON Object here"
  }
})

The new m.Get call has a bunch of new parts to it. The first /attributes/:resource tells martini what route to look for. The :resource is used to indicate to martini that whatever value is here we want to store in params map. The value will be stored under the key ‘resource’, notice that the key does not have the leading colon. This handler’s function has one parameter, params, which will contain all route parameters. Next is the return value declaration. This version of the handler returns two values, the HTTP Status code and a string.

The guts of the function are there to determine if the resource being requested is a TV or not. If it is return OK if not return a not found error. Right now the code has placeholders in it but soon the strings will be string representations of a JSON object. If you are new to Go like me the if statement looks a little naked, there are no () around the test portion. The return statements are a little different than what I’m used to seeing also. Remember that this function has two return values and on the return lines the values are separated by a comma.

Now that we’ve talked it to death if you want to see it an action download the code form here https://gist.github.com/rippinrobr/9084362 and run it using:

go run attr-server.go

In a seperate terminal run the following curl command and you should see similar output.

curl -v http://localhost:3000/attributes/tvs

* Adding handle: conn: 0x7f911c004400
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x7f911c004400) send_pipe: 1, recv_pipe: 0
* About to connect() to localhost port 3000 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 3000 (#0)
> GET /attributes/tvs HTTP/1.1
> User-Agent: curl/7.30.0
> Host: localhost:3000
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< Content-Length: 16
< Date: Wed, 19 Feb 2014 01:27:00 GMT 
<
* Connection #0 to host localhost left intact
JSON Object here

The curl -v displayed enough output so that you can see just about everything that happened during the request. I’m using it here so you can see that the call above does in fact return a 404 code in addition to the error message. To see what happens when you pass it TV rerun the command after removing the trailing s from tvs. The HTTP/1.1 status should now be 200.

Sending the JSON Object

Now that I have the basic logic in place its time to start building out the infrastructure to support resource attributes. To help model the resource to attributes relationship I am introducing two new structs, Attribute and ResourceAttributes.

type Attribute struct {
  Name string `json:"name"`
  DataType string `json:"type"`
  Description string `json:"description"`
  Required bool `json:"required"`
}

type ResourceAttributes struct {
  ResourceName string `json: "resourceName"`
  Attributes []Attribute `json: "attributes"`
}

The Attribute struct contains all of the information I want to store about each attribute. The ResourceAttribute struct is used to represent the relationship between a resource and its attributes. Since these structs will be converted to JSON and I want to the names of the field to follow proper JSON naming conventions I’m using the “field's tag value” to convert the names to lower case during the JSON conversion process.

The updated handler will be returning a string representation of the JSON object and in order to do that I need to create a String method for each of the types I’ve declared. I want to send all of my structs back to the client as JSON objects I need to create two String() methods. The method below is used on the ResourceAttributes struct.

func (ra ResourceAttributes) String() (s string) {
  jsonObj, err := json.Marshal(ra)

  if err != nil {
    s = ""
  } else {
    s = string( jsonObj )
  }

  return
}

There are two differences in the declaration of this method from the functions I declared earlier. The first is right after the func keyword is what looks like a parameter declaration. What it does is declare what type is the 'receiver' for this method. What that means is any ResourdeAttribute object can call the String() method. The second difference is in the way the return value is declared. This method makes use of Go's named return value. What that means is whatever the value of the variable s is at the time that the method returns will be the value returned by the method.

First, the method converts the receiver into JSON. If there are no errors returned during the conversion process then the JSON representation is converted to a string and stored in s. If an error occurs then s is set to an empty string.

To be able to send all of the structs I declared as JSON I would have to create a String() method for each type. The methods would be exactly the same except for the reciever. Not exactly keeping the code DRY. Thankfully shortly after writing the code of this part of the blog I reached a section on interfaces in The Go Programming Language Phrasebook and I was happy to see that using interfaces will let me DRY up the String() methods.

A Go interface is a set of methods. Any struct that has all of the methods in the interface declaration is said to implement the interface. Interfaces can have 1, 10, or no methods. So I decided to try an empty interface declaration that would stay within my main package.

type jsonConvertible interface { }

Since the name of this type starts with a lower case character it is only visible within the package it was declared in. Now any struct I declare in the main package will implement the jsonConvertible interface. After creating the interface I moved away from using methods back to a normal function, I created a new function named JsonString. JsonString has a single parameter, a jsonConvertable struct. Now I can have one function to convert all my structs into a JSON string.

func JsonString( obj jsonConvertible ) (s string) { jsonObj, err := json.Marshal( obj )

if err != nil {
  s = ""
} else {
  s = string( jsonObj )
}

return

}

If you want to see this version of the code in action you can grab it in this gist and go run it.

Setting the Content-type to application/json

When you run the latest and greatest you see that the server does send back a JSON string but if you look at the headers you can see that the Content-Type is set to text/plain. I want the Content-Type to be application/json. In order to do that I need to set the Content-Type before I send the response. Luckily, using martini makes this as easy as adding a new parameter to my handler function, writer http.ResponseWriter. I can use the new parameter to set the correct Content-Type.

writer.Header().Set("Content-Type", "application/json")

Now whichever object is returned, it will have the correct Content-Type set. To see for yourself, clone the repository, checkout the 1st-post branch and run it. You’ll see in the headers that I now have the correct Content-Type set.

Summary

With that, I’ve completed everything that I set out to do by the end of this post. I showed you where to get Go and how to setup your environment. I’ve walked you through how to create structs, respond to HTTP GET calls and how to return a JSON string. In addition to that I introduced you to interfaces in Go.

In my next post, I will add CRUD functionality using MongoDB, showing you how to add middleware to pass along database connectivity to the request handlers. By the end of the second post the attr-server will retrieve all available attributes assigned to a TV from the database using our /attributes/:resource route.

Resourcs

Editors

Eclipse goclipse
Emacs go-mode.el
Sublime go plugin
Vim go language vim support

Go Language

GitHub & Gits

Blogs & Books