Learning Go with Martini - Working with MongoDB

This is the second post in a series of posts on creating Go based web applications/APIs. If you missed the first post Learning Go with Martini - The Basics Go check out and then come back to this post

Intro

Now that I understand how to handle GET requests its time to add some database interaction. In this post I will walk you through how to add a datgabase connectivity middleware to each request. Once I have that in place I will convert the /attributes/:resource into a ‘real’ GET, one that reads from a database and returns the results fo the query. I will add a POST route, storing the data that was ‘POSTed’ to the app in a MongoDB collection.

Setup

Before we get started you will need to have MongoDB installed or have access to an instance of MongoDB. To install it locally visit this MongoDB’s download page. If you’d like to use MongoDB in the ‘cloud’ checkout Mongolab.

Also, if you haven’t already done so you’ll need to have a functional Go language environment. Visit the golang.org’s install page and follow their instructions for your platform of choice.

Lastly, The code for the beginning of this post is stored Gist style and can be found here.

Creating the Middleware

Before I start working with the database I need a way to ensure that each handler functions has access to the database connection. To do that I’m going to create a middleware function that middleware that will make a database connection available to each request handler. Before I jump right into that piece of middleware I'm going to start off with a 'Hello Middleware' function.

Step 0. The Hello Middleware Edition

To make sure I have the handler working correctly I’m going to start off easy. I’m going to create a handler function that simply writes out “Someday I will be a MongoDB connection”. Here’s what the skeletal function looks like.

func Mongo() martini.Handler {
  return func (c martini.Context ) {
    fmt.Println("Someday I will be a MongoDB connection")
    c.Next()
  }
}

The middleware returns a martini.Handler as an anonymous function. All my function does is write a string back and call c.Next(). The Next call yields until after all of the other handlers have executed.

To add the function into the request stack I need to add the line m.Use( Mongo() ) after the martini.Classic() call in the main function.

Now, each and every request will make a call to the Mongo() handler. If you download the gist above and add the function and the m.Use call to it you’ll be able to build and run the code. Do a few requests, valid and invalid. In the window where you started the service you should see the output of the middleware for each and every request.

Step 1. Setting up for MongoDB

For the MongoDB connection I will be using the mgo library. Before you can install it you’ll need to have the bazaar tool installed. So, if you don't already have bazaar installed visit the project's go to the project’s download page. If you are on a mac and have homebrew installed you can install both by running the following:

brew install bazaar
go get labix.org/v2/mgo

Now that I have mgo installed I’m going to change the Mongo function so that it adds a database session to each request. First I need to add the mgo package to my import statement, labix.org/v2/mgo, to my imports list. After updating the imports list I updated the function so that it now looks like this:

func Mongo() martini.Handler {
  session, err := mgo.Dial( "localhost/goattrs" )
  if err != nil {
    panic( err )
  }

  return func (c martini.Context ) {
    reqSession := session.Clone()
    c.Map( reqSession.DB( "goattrs" ) )
    defer reqSession.Close()

    c.Next()
  }
}

It looks quite a bit different that the 'Hello' version. The first line in the function uses the mgo.Dial function to create the MongoDB connection, you can think of it as a database connection. The Dial function has two return values the session object and an error object if necessary. If an error occurrs then we 'panic', no sense on continuing if we have no database conection. If there was no error an anonymous function is created with a single parameter, a martini.Conext. The context object is what I will use to make the database session available to the handlers. Before that I create a clone of the original object so that each request can close its databse connection. The defer call, puts the reqSession.Close() call ‘on hold’ until the end of the handler function. Once the function has ended reqSessoin.Close() is called and the requests database connection is closed.

Converting the GET /attributes/:resource Handler

Now that the Mongo() middleware is in place I can convert the handler over to use it. First, I need to add the parameter db *mgo.Database to the getAttributes function. Now that I have a pointer to the database conection I can look for attributes for the given resource. I am going to change the if statement so that if the results of my query are not null, I will return a 200 along with a JSON version of the results. If the query results are null, I will return a 404 and an errorMsg JSON object. Here’s the updated function.

func getAttributes( params martini.Params, writer http.ResponseWriter, db *mgo.Database) (int, string) {
  resource :=  strings.ToLower( params["resource"] )
  writer.Header().Set("Content-Type", "application/json")

  var attrs []resourceAttributes
  db.C("resource_attributes").Find(bson.M{"resource": resource }).All(&attrs);

  if attrs != nil {
    return http.StatusOK, jsonString( attrs )
  } else {
    return http.StatusNotFound, jsonString( errorMsg{"No attributes found for the resource: " + resource} )
  }
}

The meat of the changes are the two lines that handle the database query. First the var attrs []resourceAttributes line declares the array that will contain the returned data if any is found. The next line is where all the magic happens. The db.C(“resource_attributes”) tells the driver that we want to work with the resource_attributes collection. If you aren’t familiar with MongoDB think of a collection like a table in a relational database. The Find call, similar to a select in rdbms world, takes a map as its parameter. Here I’m looking for an objet that has a resource property equal to the lower case version of the resource that was passed in. The .All call will take the results of my query and store them in the attrs array. If no matches are found then the attrs array will be nil. Now when I make a the /attributes/tv call I get an empty result set back. To fix that, I'm going to add a POST handler to create attributes.

Creating the POST /attributes/:resource Handler

I’m going to start off by setting up a new route in the attr-server.go file. The route will call a second function in the attribute-routes.go file that will handle the heavy lifting of creating a new attribute. Here’s what the new route looks like:

m.Post("/attributes/:resource", addAttribute  )

Not much going on here, now any post to the /attributes/:resource url will be handled by the addAttribute function. The first version of the function will be a simple placeholder, one that shows me that I have the POST support wired up correctly. Here’s the function:

func addAttribute( params martini.Params, writer http.ResponseWriter, db *mgo.Database) (int, string) {
  resource :=  strings.ToLower( params["resource"] )
  writer.Header().Set("Content-Type", "application/json")

  return http.StatusOK, "POST placeholder " + resource
}

Nothing new happening here other than the fact it handles a POST instead of a GET. After I recompiled I used the following curl command to test the code:

curl -X POST [http://localhost:3000/attributes/tv](http://localhost:3000/attributes/tv)

which returned POST placeholder tv. Now I know that the route and handler are wired up correctly, its time to add code to process the JSON data.

Adding JSON Support

In order to support the JSON POST I'm going to use another package, this one comes from the martini-contrib ecosystem, martini-contrib/binding package. The binding package will allow me to tell martini to take the submitted JSON object and magically load it into an attribute struct. To add the support I need to modify the route definition in the main() function. Specifically I need to tell the binding package which struct to bind the incoming JSON to. The updated definition looks like this:

m.Post("/attributes/:resource", binding.Json( attribute{} ), addAttribute  )

I also need to update the handler function to support the new attribute parameter.

func addAttribute( attr attribute, params martini.Params, writer http.ResponseWriter, db *mgo.Database)

Now the handler can access the posted JSON data through the attr paramter. Running the curl POST parameter with JSON data will return a JSON string that represents the submitted data.

curl -XPOST  -d '{
  "name":"location”,
  "type":"string",
  "description":"Where the TV is located, which area of a facility is it loaded in",
  "required":"true"}' 
  -H "Content-Type: application/json" 
  http://localhost:3000/attributes/tv

Returns the following:

{"name":"location","type”:"string","description":"Where the TV is located, which area of a facility is it loaded in","required":false}

The code is now parsing the JSON and its being converted into something I can use. Thats great, but what happens if I run this curl command, what do I get then?

curl -XPOST http://localhost:3000/attributes/tv

The curl command returns an empty atttribute struct. I don’t want that to happen. I want the name field to be required on all POSTs. If nothing is supplied for the type and description fields I want them to default to an empty string. Of course you can do this after the new attribute struct is given to the handler function but I want this to be done outside of the handler code. Thankfully, the binder package has that functionality built in.

Adding Validation

The martini-contrib/binding package defines a Validator interface that contains one method declaration, Validate(Errors, http.Request). In order for me to add my validation requirement I will need to create a method for the attribute struct.

func (attr *attribute) Validate( errors *binding.Errors, req *http.Request ) {
  if attr.Name == "" {
    errors.Overall["missing-requirement"] = "name is a required field";
  }

  if attr.DataType == "" {
    attr.DataType = "string"
  }           
}

In addition to the interface, the package also has an Errors struct that has two maps in it, Overall and Fields. I’m using the Overall map to report validation errors. Specifically I'm using “missing-requirement” as the key to the error message when no name is either empty or nil. While I’m checking for the required field I will also check to see what data type was provided in the JSON object’s type property. If nothing was provided I default it to string.

After my validation is complete I can check to see if there were any validation errors by calling errors.Count(). If that is greater than zero then I know a validation error occurred. To let the client know I will send a response with 409 and a JSON string representation of the ErrorMsg struct. Otherwise, I all return a 200 and a JSON string representation of the new attribute. Here’s latest version of the POST handler.

func addAttribute( attr attribute, err binding.Errors, params martini.Params, writer http.ResponseWriter, db *mgo.Database) (int, string)  {                   

  writer.Header().Set("Content-Type","pplication/json")

  if err.Count() > 0 {
    return http.StatusConflict, jsonString( errorMsg{ err.Overall["missig-reuiement"] } )
  }

  return http.StatusOK jsonString( attr )
}

You may be wondering why I chose to return HTTP Status Code 409. I chose it after reading what w3c had to say. Basically it boiled down to this line:

This code is only allowed in situations where it is expected that the user might be able to resolve the conflict and resubmit the request. The response body SHOULD include enough information for the user to recognize the source of the conflict

After grabbing the value of the resource parameter I create a query that will be used to find the document. Once the query is defined I create a mgo.Change struct. This is how I will be able modify the data if it exists or insert it if it doesn't (this is done by setting Upsert to true). The Update is what will do the actual changing of the document. If you aren’t familiar with MongoDB what my statement says is if a matching document is found only add the attribute to the attributes array if it isn’t already in there. If it already exists it will not be added to the array.

Now that I have my query and change objects in place its time to run the database update. The Apply call returns two results ChangeInfo and a Go error struct. Since I don’t care about the ChangeInfo struct, I put a _ in its place. Obviously I’m interested in any errors that occur so I grab that value and check for an error. If an error does occur I will send an errorMsg struct back to the caller. If no error occurs then the I send back a 200 status and an empty object.

The Finishing Touches

Now that I’m storing data I’ve noticed that the GET /attributes/:resource returns a JSON Array when it should return an object. I tweaked to the getAttributes section of code that returns the results of the query to look like this:

attrs := resourceAttributes{}
err   := db.C("resource_attributes").Find(bson.M{"resource": resource }).One(&attrs)

if err == nil {

I changed the attrs declaration to be a single object instead of an array, you’ll see why on the next line. I also changed from the All() function to the One() call since only one document. If a record isn’t found it, the One() will return an error object so I’ve changed the if statement to use the presence of an error to determine which JSON string to return.

Summary

I've changed the attributes services to read from a MongoDB database. I did that by adding the Mongo() middleware function so that the handler functions can interact with the database. I added a POST route that illustrated how to use the martini-contrib/binding package to bind the incoming JSON to the attribute struct. After finishing up the POST method I went back to the GET handler to change the returned data from an array to a single object.

In my next post I'm going to create my first Go package. The package will be used to interact with the etcd to retrieve the database configuration information.

Resources

Packages

• martini-contrib/binding
• mgo

Other Links

• Go Install
• MongoDB
• MongoLabs
• W3C Status Codes

Github and Gists

• This Posts code in github (branch 2nd-post)
• The Starting Gist

Previous Posts in the Series

• #1 Learning Go with Martini - The Basics