Mongodb Tip #1 : dumping bson into legit json objects

Well, it's a bit embarrassing to start off with a hack, but what the hey... it made my life instantly easier.
So here was my challenge... to dump to file 40,000 BSON (JSON-like) objects from my mongodb so that I could slurp them up with Groovy 1.8 and its new JsonSlurper.  Then I'd be able to use dot notation to get the values I want.

Here's the sequence of what I did. BTW, in case you didn't know, there's no native method within a mongo session to just say db.hats.dumpAll()... at least I haven't found it yet. So you need to get outside the normal mongodb command line session and use these utilities that come with the mongodb distribution.

/var/lib/mongodb-linux-i686-1.8.2/bin/mongodump --db mydb --port 27017 --collection hats --query '{ }' --out dumper 
/var/lib/mongodb-linux-i686-1.8.2/bin/bsondump dumper/mydb/leads.bson

First, mongodump dumps the whole collection I called "hats".  But it's not in human readable form.  You need bsondump for that.  bsondump, by default, converts every row of mongodump output to a BSON string.

So, I thought I could just write a few lines of Groovy to convert each JSON string into an object and use dot notation to dereference the values I wanted.  But I forgot I was still dealing with a BSON string.  My problem was this:

{ "_id" : ObjectId( "4e56d1c780acbde57e951402" ), "size" : "8", "color...

As you can see, the ObjectId string is not itself in quotes and therefore messes up JsonSlurper. So I submit the following solution that worked for me.

def f = new File(fname)
def lineCount = 0
f.eachLine { line ->
        def line2 = line.replaceFirst('ObjectId\\(','') .replaceFirst('\\),',',')
        try {
        def slurper = new JsonSlurper()
        def res = slurper.parseText(line2)
        println res.size + "|" + res.color
        lineCount++
        } catch (Exception e) {
                e.printStackTrace()
        }
}
println lineCount + " lines encountered."

I used replaceFirst instead of replaceAll() to avoid overkill and generally screwing up other innocent content that might contain the right matching parentheses with comma.  Unlikely, but safe(r).

By the way, the reason for the exception handling is that bsondump outputs some stats, not in BSON format,  every now and then.  Odd.  Obviously, I could have handled that more gracefully, but the exception handling did the job and the lines encountered gave me the target number I was looking for.

That's it.  Got the job done.  Enjoy.

David

mongodb goodness, so far...

Lately, I've been working pretty extensively with mongodb.  I classify it as a "JIT DB", as in Just-In-Time Schema Database.  It's perfect for lazy moments when you're writing some code and it dawns on you that you need an additional field or even an additional table (called "collections" in Mongo).

"Lazy" is the wrong word.  mongoDB is in a class of technologies and strategies that foster inspired notions and reduce barriers (like time and patience) to assert your ideas. SQL doesn't do that for me.  The level of required schema pre-work and retrofitting has nipped some cool ideas in the bud... mongoDB encourages me to do it right now because I don't see any impedance!  Throw together shell scripting, Groovy and mongoDB and let's just do it!

Here's a quick example that will hopefully illustrate for you the low impedance of mongoDB (and lots of other unSQL databases)...

Let's sort a table called myData by a timestamp field.

db.mydata.find().sort({timeStamp:-1}).

This is equivalent to

select * from mydata order by timeStamp desc;

mongodb comes back and says something to the effect of "can't do a big sort like this without an index."  Well there you go... so you type

db.mydata.ensureIndex({timeStamp:1})

You try the sort again and it works. You've just experienced something like a conversation with your database!  "I can't do this... you know what to do..."

In full disclosure, I acutally use Groovy for all my Java-style development now.  I've completely lost interest in Java because 1) Groovy is way more satisfying and productive and 2) I, currently, have no
 need to use Java for squeezing max performance out of code.  I mostly use Groovy for batch-style work, updating Salesforce.com via the Web Services API and such.

With mongoDB there's a nasty little conceptual hurdle to jump over, especially switching back and forth between using native javascript commands and Java driver programming.
In Java, there are at least 2, 3 or more ways to construct a db operation

def doc = new BasicDBObject().append("lastName","Smith").append("firstName","Jack")
myData.query(doc)

...versus...

def person = [:] // Groovy syntax
person.lastName = "Smith"
person.firstName= "Jack"
def doc = new BasicDBObject(person)
myData.query(doc)

Straightforward enough. In the native mongo language, the query looks something like this...

db.myData.find({lastName:"Smith",firstName:"Jack"})

which is equivalent to

select * from myData where lastName = 'Smith' and firstName = 'Jack';

Now, because I'm a Linux guy and I love the power of intermingling shell scripts to glue Java/Groovy together as needed, here's one way way you might integrate that mongoDB (javascript) script language in a bash shell script using a here document.

function findUser {
        lastName=$1
        firstName=$2
        mongo <<EOF
        use employeeDB

        var criteria = {lastName:"${lastName}",firstName:"${firstName}"}
        var answer = db.leads.find(criteria)
        answer.count()

EOF
}


findUser Smith Joe


Enough for now.  For the next few weeks, I'll post some of the mongodb commands and concepts I found most useful.

Theory on Time as a symptom, not the cause, coming together...

In a recent Facebook discussion, I referenced the development at http://www.physorg.com/news/2011-04-scientists-spacetime-dimension.html re: "...space-time has no time dimension..." and how it reinforces my gut-level feeling about the concept of time.  Thanks to a response by my former Lutris colleague Daryl, a few thoughts came together.

1. The folks behind this article didn't go far enough to put some visual teeth into it (for us lay people), and 
2. Once again, Julian Barbour and his work come to my rescue to explain what could be going on...

Barbour, his book called, "The End of Time" and his site http://platonia.com mean a lot to me.  In particular, he has proposed a view of existence he's dubbed Platonia. Platonia looks like a typical landscape rendering you might see in an landscape architect's office except that it represents the likelihood of events (i.e., probability).  The illusion that is created as physical space changes in sequence is the perception of time.

While looking at one of Barbour's very recent papers, I saw that he appears to be driving to a quantum theory of the universe. Or at least he's describing why the universe could be described in quantum terms. As I understand things, time prevents folks from getting to a quantum explanation... but only if you see time as a building block and not an outcome of the dynamics of things.  If you remove time from the space-time equation and replace it with the sequencing of change, then his Platonia view of things as movements powered by probability, makes a quantum thoery of space, and its coming into existence, more attainable.

So that's as far as I can run with the original article I referenced at the beginning of this post.  There's more to come now that I have a new reason for exploring Barbour's work again...

Life is so cool...

Kicking the Sugar Thing in 2011

I have been a sugar addict all my life.  Just love the taste of the stuff.  Peanut M&M's are my favorite indulgence.  I've always relied on them for reward during crazy hours behind the computer.

But, starting the day after new year's 2011, I made an impulsive switch to a low-GI (Glycemic Index) diet, a.k.a. Atkins.  I'd been on Atkins before, so I knew what it was about.  But because I've been teaching myself to cook the past year, it seems to be more doable this time.

Two months later, I'm writing to report on how things are going.  It's real simple.  My mind is more clear than it's been in decades.  I don't have incredible impulses to eat.  And I'm generally very happy about the whole thing.  Yesterday, I played some driveway basketball with my youngest daughter, Claire.  She just turned 11.  There was something different about this session.  I felt great.  I felt light, upbeat and, most wonderfully of all, full of energy.  Maybe there is something different about getting all your energy from protein and not carbs.

The toughest part of the diet has been the eggs thing in the morning.  Started to gag just thinking about them.  But I've found that, for me, sautéing some mushrooms and bell peppers, at a minimum, make eggs a wonderful thing.  Just break a few eggs on top of the aforementioned sauteed items and the result is a wonderful blend of goodness including whatever I sprinkled on top. Chives. Yes.  Garlic.  Yes.  Now I'm moving on to including spinach.  Starting to border on eggs florentine.  Without the bread, of course.

So here's what I like about by new life style after two months of this stuff.
1. Lack of appetite is wonderful.  I don't feel like something I can't see is controlling me.
2. I'm much more into water these days.  Naturally.  I have no problem with craving water.
3. Exercise seems much more natural.  Can't wait to get to the gym tonight.

The only thing I _really_ miss is sourdough bread from Santa Fe.  I'm originally from San Francisco and sourdough is in my blood.  And sourdough from the Santa Fe bakery is awesome.

I'm reading a book called "The 4 Hour Body" by Timothy Ferris.  It's informative, inspiring and hilarious.  Even mentions Chad Fowler, who was an early days champion of the Enhydra Java application server when he was at GE.  Now he's a long-time Ruby evangelist.  That was pretty cool.

I'll report on the impact that book has had on me in a later post.  And it even helps me with my sourdough jones.  

Groovy Tip and How-To (gathering specific files from a directory)

I'm working on a service in Grails that offers all kinds of methods for getting dates or formats of dates with single line calls. For example, dateTimeService.getBracketDates() which will return the first and last dates for the current month (e.g., 2/1/2011 and 2/28/2011). It's not rocket science but I have no desire to re-invent the wheel every time I'm creating a new monthly report for our marketing folks.

Tip: Keep a code snippets (experiment) file

One of the awesome things about groovy is that it's a lot like script programming. That makes it very convenient to try pieces of code and functionality you're not sure about before adding it to your project. Be forewarned. I'm a vi nut. I'm aware that dev tools have snippet support. I just find this works for me, and perhaps for you too.

The nice side-effect of this practice is that you build up a semi-structured collection of groovy and coding knowledge to reference in times of need and senior moments (not you, me!). Guess you could use it for old girl (or boy) friend names too. Another nice side-effect might be "instant book" once you insert a few paragraphs between each snippet!

Put a System.exit(0) at the end of your snippet so that your groovy interpreter doesn't try to execute all the accrued code below. This won't protect you against using variables that have already been defined lower in the file. All I usually do is attach a number to the end of the variable name to get rid of the name collision. Afterall, all I'm trying to do is validate my code. I used this little practice of mine to refinethe How-to below before subjecting you to what might have been buggy code.

How-to: Gathering list of (specifically-named) files from a directory

Here is one of many ways to accomplish this task. I found this old code snippet from my snippet file and thought I'd share it. By the way, a site I love to consult on little basics like this is Pleac Groovy. It's not the most groovy-tized site in the world, but I like it's get-it-done blue collar perspective.

So this bit of code returns a list of log files that use the the naming convention of <directory path>/<year><month><day>.log. In particular, this code winnows the list of files down to the files stored in the month of February (i.e., 201102.*\.log). The regular expression is pretty lame. I'm sure you could improve it.

def baseDir = '/tmp/logs'
def originalFiles = new File(baseDir).listFiles()
println "number of files found:"+originalFiles.size()

// here's our closure, a match expression
def screener2 = { it.name =~ /.*201102.*\.log$/}

def screenedList = originalFiles.findAll(screener2)
println "Number of files matching :"+screenedList.size()
println "screenedList :"+screenedList