A definition for "cloud" computing

Before "the cloud," you pretty much new where you data was getting stored. If you were on a business network, you probably knew that your files were stored on your local hard drive or on the network server located in the closet in the office. Or maybe you had heard that "the server" was hosted by a downtown Internet Service Provider company. You may have even heard that it was still your company's hardware. They just kept it cool and in a "cage."

So, with the cloud, how is that different?

If your files are now hosted in the cloud, then you no longer can picture where they reside. That's because instead of having one or two dedicated pieces of file-hosting hardware, your network folks have instead subscribed to a cloud service like Amazon.com. That means your network file serving has been "thrown over the wall" to a stranger like Amazon.com. What machines they use, what technology they use is their concern. All you know is that you still see your files. And you've heard your network folks love it because they let Amazon.com deal with the hardware, the backup responsibilites, the "failover" prevention, the security and so forth. In fact, they love it even more because they know they can keep adding more room for storing more files as the company grows. And all without having to order new hardware.

But what's this "iCloud" thing from Apple?

Well, it's just more of the same, except there's no network group at the office involved. Unlike Amazon.com, you don't have to deal with bringing up "an instance of a server." Instead, it's more of a consumer-focused file hosting service. It's just between you, your Apple device and the iCloud service. Again, you have no idea where they store your images or tunes. They just make sure you have constant access to what you "throw over the wall to them."

So what are the implications?

As long as Apple stays in business, you have "control" of your files. As long as Apple keeps their servers healthy, you have access to your files. You now don't have quite the concern for how big your disk drive is anymore, right? That's because your store your files on the cloud now.

Other implications? Of course, but I'm not in the mood to reflect on the big brother nature of all of this, or the house of cards scenario. It seems that this is direction the forces of business-driver evolution are taking the Internet. It's not an evil plot. It seems to make sense.

David

An adoptee's perspective on Steve Jobs...

Steve and I went to the same high school, Homestead, on the border of Sunnyvale and Cupertino. He was a freshman when I was a sophomore. Wozniak was a senior. Steve dated my next door neighbor Marla and showed up in a satin white tux with top hat at the 5 year high school reunion (according to my sister Carolyn who was in the same graduating class of '72).

As I tell people, while I hung out with girls, he was more of a "shop" guy. Our paths never crossed. Or they did but I didn't know him from Adam. But later I met him twice, once when he entertained a group of us from SCO (the Unix on Intel company) at NeXT with the purpose of convincing SCO to ship "NeXTStep", a development environment that he was trying to make more pervasive in the Unix world.

It was there, at the NeXT offices near Redwood City off of 101, visiting Jobs with two SCO VPs where I learned what a crutch powerpoint presentations are as well as Jobs' commanding knowledge of the industry. After Steve and I shortly reminisced about the good old days at Homestead High, one of our VPs got up to give him a presentation of SCO and our 2-tier distribution model, of which we were very proud. About to display the 3rd slide, Steve waved him off, asked him to sit down and proceeded to summarize the SCO business model as well as I'd ever heard from our own people. I think it's fair to say that our guy was pretty devastated to be brought down to earth so quickly... especially given how much he probably had rehearsed the days before in order to impress Jobs about SCO's unique Unix-on-Intel business model.

At the end of the meeting, it was already clear to us that he was a little nuts and, in a way, totally out of his league. He was trying to build a consortium without forming a consortium. And consortiums are difficult to control. Steve had never given the world the impression that he was a "standards kind of guy" or capable of adapting to the committee-driven nature of Unix standards. At the end of the discussion when we were trying to understand his real motivations, he finally said (to paraphrase), "Well somebody's got to stop Microsoft!" I kind of shook my head and thought, "this poor guy... he's still driven by his personal quarrel with Gates."

I later interviewed at NeXT to run their product engineering group. I again had a casual conversation with Jobs, but my interview was with Avie Tevanian, the architect of the NeXTStep Mach operating system. He impressed me as a sweet, got-it-together kind of guy. I would have loved working for him. But they never called me back and I never called them because I couldn't see leaving my wonderful cocoon of Santa Cruz and resuming that commute to Redwood City, of which I was well familiar after 4 years of Santa Cruz-to-Sunnyvale-and-back. Makes new cars old very quickly.

I'm writing this because last night, when I inadvertently brought up my browser to cnn.com and saw the news, I was devastated. Shockingly devastated. Yeah, I have a few cute stories about Jobs, and no, there was no way I would have enjoyed working for him given the butterfly nature of my love for software development.

But, as I was telling a colleague at SAMBA this morning, like me, Jobs was an adoptee. Of course, he was also a fellow graduate of Homestead High and our silly allegiances to famous people feel quite real. But way back in the 80's when he became famous, I looked up to him as a fellow adoptee because I loved how he handled it. It gave being an adoptee real respect. He loved his parents and he didn't let the adoption thing hinder his life one bit. At the time, I thought it was kind of weird that I didn't have the slightest inclination to find out who my real parents were. My harsh self-appraisal was accented by what I'd been seeing on television at the time. Groups of adoptees were easy to find on day time talk shows, whining about how incomplete their lives were having no knowledge of their "real parents" and were confronting the fact that the adoption process was somehow overly secretive. It was strange to me that, in the same reaction, I was both repelled by these people and somewhat guilty that I didn't share their protest.

Jobs made me feel like I was normal. Better than that, he legitimized some of the crazy things I'd done along my career path. His unintended stamp of approval on how you can lead an adoptee's life without driving yourself nuts and, in fact, turn it into an outlaw'ishly productive and inspired life, was important to me. I had no idea how important his inspiration was, or how deeply I had internalized that message, until my reaction last night.

Thank you Steve,
David

CNBC's "Explain This" makes KhanAcademy a Pioneer of the "HyperSchool"

Anybody think that CNBC has delivered the latest unanticipated Internet service that's a (revolutionary) turn for the better?

If you peruse my blog, you know I'm a big fan of khanacademy.org and Sail Khan's library of over 2,000 9-to-15 minutes classes. Perfect for the Web, for our rapid paced lifestyle and increasingly limited attention spans.

Why do I think this new features turns khanacademy into a new class of school I'm going to call "hyperschools"? Because it is the beginning of the destruction of our view of education as a silo where school is school and it happens during school. With "CNBC Explains", CNBC and Khan are showing that learning can be more spontaneous, more targeted, more convenient and more "in the moment."

The parallel is with hyperlinks themselves. They give the Web its ability to cross-wire information. Following hyperlinks, you can follow all kinds of trails of content (knowledge). Hyperschools, of which there is now only one -- khanacademy.org --, provide the same flexibility. Let me educate myself by attempting to read this interesting article about LIBOR (inter-bank trading rates) and I'll click on these "Explain This" icons as I see fit in order to more deeply understand what I'm reading (by following a quick 10 minute Khan class focused on specified, relevant topic (to the article).

Another metaphor is the Star Trek-like scenario of "I want to be expert enough to follow this article on Treasury Bonds, so I'll take a pill on everything I need to know to understand the article.

Hopefully, you get the idea. It feels like a subtle new B2B enhancement of a company's web site (CNBC to Khan)... but the more I thought about it, the more I like the feeling I got of yet more breaking of old barriers to education. Love it!

A more reasonable way to comment blocks of shell code...

I love using shell script for launching java and groovy apps because they're so good at setting the table in a way that keeps the application much simpler... especially when the shell script can handle common needs which is often the case for Operations-style functionality. Example shell script functionality includes

• determining if there's sufficient space on the file system,
• collecting files to process,
• configuring the environment (as in defining traps that clean up temp files and remove lock files when an app closes either naturally or via control-C interrupt, or re-directing I/O so that a control-C won't kill the process you're running).

Because operations-style applications suffer from out-of-site, out-of-mind syndrome, having a solid strategy with shell scripts that can manage and measure the operating environment, and scream bloody murder when things aren't right, makes them a worthy design component.

I've never liked "the fact" that to comment out a block of code in unix shell programming I had to insert # symbols in front of every line.

I found out over the weekend the ideal way to do it using a here document and the ":" operator, which is a no-op

# all this code inside this section document
# is now invisible to the shell interpreter
# Add to use a # anywhere.

And here's how to do the equivalent with a here document.

: <<STUFF_TO_PASS_TO_COLON
all this code inside this here document
is now invisible to the shell interpreter
Didn't have to use a # anywhere.
STUFF_TO_PASS_TO_COLON

Mongodb Tip #1 : dumping bson into legit json objects

Well, it's a bit embarrassing to start off with a hack, but what the hey... it made my life instantly easier.
So here was my challenge... to dump to file 40,000 BSON (JSON-like) objects from my mongodb so that I could slurp them up with Groovy 1.8 and its new JsonSlurper.  Then I'd be able to use dot notation to get the values I want.

Here's the sequence of what I did. BTW, in case you didn't know, there's no native method within a mongo session to just say db.hats.dumpAll()... at least I haven't found it yet. So you need to get outside the normal mongodb command line session and use these utilities that come with the mongodb distribution.

/var/lib/mongodb-linux-i686-1.8.2/bin/mongodump --db mydb --port 27017 --collection hats --query '{ }' --out dumper 
/var/lib/mongodb-linux-i686-1.8.2/bin/bsondump dumper/mydb/leads.bson

First, mongodump dumps the whole collection I called "hats".  But it's not in human readable form.  You need bsondump for that.  bsondump, by default, converts every row of mongodump output to a BSON string.

So, I thought I could just write a few lines of Groovy to convert each JSON string into an object and use dot notation to dereference the values I wanted.  But I forgot I was still dealing with a BSON string.  My problem was this:

{ "_id" : ObjectId( "4e56d1c780acbde57e951402" ), "size" : "8", "color...

As you can see, the ObjectId string is not itself in quotes and therefore messes up JsonSlurper. So I submit the following solution that worked for me.

def f = new File(fname)
def lineCount = 0
f.eachLine { line ->
        def line2 = line.replaceFirst('ObjectId\\(','') .replaceFirst('\\),',',')
        try {
        def slurper = new JsonSlurper()
        def res = slurper.parseText(line2)
        println res.size + "|" + res.color
        lineCount++
        } catch (Exception e) {
                e.printStackTrace()
        }
}
println lineCount + " lines encountered."

I used replaceFirst instead of replaceAll() to avoid overkill and generally screwing up other innocent content that might contain the right matching parentheses with comma.  Unlikely, but safe(r).

By the way, the reason for the exception handling is that bsondump outputs some stats, not in BSON format,  every now and then.  Odd.  Obviously, I could have handled that more gracefully, but the exception handling did the job and the lines encountered gave me the target number I was looking for.

That's it.  Got the job done.  Enjoy.

David

mongodb goodness, so far...

Lately, I've been working pretty extensively with mongodb.  I classify it as a "JIT DB", as in Just-In-Time Schema Database.  It's perfect for lazy moments when you're writing some code and it dawns on you that you need an additional field or even an additional table (called "collections" in Mongo).

"Lazy" is the wrong word.  mongoDB is in a class of technologies and strategies that foster inspired notions and reduce barriers (like time and patience) to assert your ideas. SQL doesn't do that for me.  The level of required schema pre-work and retrofitting has nipped some cool ideas in the bud... mongoDB encourages me to do it right now because I don't see any impedance!  Throw together shell scripting, Groovy and mongoDB and let's just do it!

Here's a quick example that will hopefully illustrate for you the low impedance of mongoDB (and lots of other unSQL databases)...

Let's sort a table called myData by a timestamp field.

db.mydata.find().sort({timeStamp:-1}).

This is equivalent to

select * from mydata order by timeStamp desc;

mongodb comes back and says something to the effect of "can't do a big sort like this without an index."  Well there you go... so you type

db.mydata.ensureIndex({timeStamp:1})

You try the sort again and it works. You've just experienced something like a conversation with your database!  "I can't do this... you know what to do..."

In full disclosure, I acutally use Groovy for all my Java-style development now.  I've completely lost interest in Java because 1) Groovy is way more satisfying and productive and 2) I, currently, have no
 need to use Java for squeezing max performance out of code.  I mostly use Groovy for batch-style work, updating Salesforce.com via the Web Services API and such.

With mongoDB there's a nasty little conceptual hurdle to jump over, especially switching back and forth between using native javascript commands and Java driver programming.
In Java, there are at least 2, 3 or more ways to construct a db operation

def doc = new BasicDBObject().append("lastName","Smith").append("firstName","Jack")
myData.query(doc)

...versus...

def person = [:] // Groovy syntax
person.lastName = "Smith"
person.firstName= "Jack"
def doc = new BasicDBObject(person)
myData.query(doc)

Straightforward enough. In the native mongo language, the query looks something like this...

db.myData.find({lastName:"Smith",firstName:"Jack"})

which is equivalent to

select * from myData where lastName = 'Smith' and firstName = 'Jack';

Now, because I'm a Linux guy and I love the power of intermingling shell scripts to glue Java/Groovy together as needed, here's one way way you might integrate that mongoDB (javascript) script language in a bash shell script using a here document.

function findUser {
        lastName=$1
        firstName=$2
        mongo <<EOF
        use employeeDB

        var criteria = {lastName:"${lastName}",firstName:"${firstName}"}
        var answer = db.leads.find(criteria)
        answer.count()

EOF
}


findUser Smith Joe


Enough for now.  For the next few weeks, I'll post some of the mongodb commands and concepts I found most useful.

Theory on Time as a symptom, not the cause, coming together...

In a recent Facebook discussion, I referenced the development at http://www.physorg.com/news/2011-04-scientists-spacetime-dimension.html re: "...space-time has no time dimension..." and how it reinforces my gut-level feeling about the concept of time.  Thanks to a response by my former Lutris colleague Daryl, a few thoughts came together.

1. The folks behind this article didn't go far enough to put some visual teeth into it (for us lay people), and 
2. Once again, Julian Barbour and his work come to my rescue to explain what could be going on...

Barbour, his book called, "The End of Time" and his site http://platonia.com mean a lot to me.  In particular, he has proposed a view of existence he's dubbed Platonia. Platonia looks like a typical landscape rendering you might see in an landscape architect's office except that it represents the likelihood of events (i.e., probability).  The illusion that is created as physical space changes in sequence is the perception of time.

While looking at one of Barbour's very recent papers, I saw that he appears to be driving to a quantum theory of the universe. Or at least he's describing why the universe could be described in quantum terms. As I understand things, time prevents folks from getting to a quantum explanation... but only if you see time as a building block and not an outcome of the dynamics of things.  If you remove time from the space-time equation and replace it with the sequencing of change, then his Platonia view of things as movements powered by probability, makes a quantum thoery of space, and its coming into existence, more attainable.

So that's as far as I can run with the original article I referenced at the beginning of this post.  There's more to come now that I have a new reason for exploring Barbour's work again...

Life is so cool...

Kicking the Sugar Thing in 2011

I have been a sugar addict all my life.  Just love the taste of the stuff.  Peanut M&M's are my favorite indulgence.  I've always relied on them for reward during crazy hours behind the computer.

But, starting the day after new year's 2011, I made an impulsive switch to a low-GI (Glycemic Index) diet, a.k.a. Atkins.  I'd been on Atkins before, so I knew what it was about.  But because I've been teaching myself to cook the past year, it seems to be more doable this time.

Two months later, I'm writing to report on how things are going.  It's real simple.  My mind is more clear than it's been in decades.  I don't have incredible impulses to eat.  And I'm generally very happy about the whole thing.  Yesterday, I played some driveway basketball with my youngest daughter, Claire.  She just turned 11.  There was something different about this session.  I felt great.  I felt light, upbeat and, most wonderfully of all, full of energy.  Maybe there is something different about getting all your energy from protein and not carbs.

The toughest part of the diet has been the eggs thing in the morning.  Started to gag just thinking about them.  But I've found that, for me, sautéing some mushrooms and bell peppers, at a minimum, make eggs a wonderful thing.  Just break a few eggs on top of the aforementioned sauteed items and the result is a wonderful blend of goodness including whatever I sprinkled on top. Chives. Yes.  Garlic.  Yes.  Now I'm moving on to including spinach.  Starting to border on eggs florentine.  Without the bread, of course.

So here's what I like about by new life style after two months of this stuff.
1. Lack of appetite is wonderful.  I don't feel like something I can't see is controlling me.
2. I'm much more into water these days.  Naturally.  I have no problem with craving water.
3. Exercise seems much more natural.  Can't wait to get to the gym tonight.

The only thing I _really_ miss is sourdough bread from Santa Fe.  I'm originally from San Francisco and sourdough is in my blood.  And sourdough from the Santa Fe bakery is awesome.

I'm reading a book called "The 4 Hour Body" by Timothy Ferris.  It's informative, inspiring and hilarious.  Even mentions Chad Fowler, who was an early days champion of the Enhydra Java application server when he was at GE.  Now he's a long-time Ruby evangelist.  That was pretty cool.

I'll report on the impact that book has had on me in a later post.  And it even helps me with my sourdough jones.  

Groovy Tip and How-To (gathering specific files from a directory)

I'm working on a service in Grails that offers all kinds of methods for getting dates or formats of dates with single line calls. For example, dateTimeService.getBracketDates() which will return the first and last dates for the current month (e.g., 2/1/2011 and 2/28/2011). It's not rocket science but I have no desire to re-invent the wheel every time I'm creating a new monthly report for our marketing folks.

Tip: Keep a code snippets (experiment) file

One of the awesome things about groovy is that it's a lot like script programming. That makes it very convenient to try pieces of code and functionality you're not sure about before adding it to your project. Be forewarned. I'm a vi nut. I'm aware that dev tools have snippet support. I just find this works for me, and perhaps for you too.

The nice side-effect of this practice is that you build up a semi-structured collection of groovy and coding knowledge to reference in times of need and senior moments (not you, me!). Guess you could use it for old girl (or boy) friend names too. Another nice side-effect might be "instant book" once you insert a few paragraphs between each snippet!

Put a System.exit(0) at the end of your snippet so that your groovy interpreter doesn't try to execute all the accrued code below. This won't protect you against using variables that have already been defined lower in the file. All I usually do is attach a number to the end of the variable name to get rid of the name collision. Afterall, all I'm trying to do is validate my code. I used this little practice of mine to refinethe How-to below before subjecting you to what might have been buggy code.

How-to: Gathering list of (specifically-named) files from a directory

Here is one of many ways to accomplish this task. I found this old code snippet from my snippet file and thought I'd share it. By the way, a site I love to consult on little basics like this is Pleac Groovy. It's not the most groovy-tized site in the world, but I like it's get-it-done blue collar perspective.

So this bit of code returns a list of log files that use the the naming convention of <directory path>/<year><month><day>.log. In particular, this code winnows the list of files down to the files stored in the month of February (i.e., 201102.*\.log). The regular expression is pretty lame. I'm sure you could improve it.

def baseDir = '/tmp/logs'
def originalFiles = new File(baseDir).listFiles()
println "number of files found:"+originalFiles.size()

// here's our closure, a match expression
def screener2 = { it.name =~ /.*201102.*\.log$/}

def screenedList = originalFiles.findAll(screener2)
println "Number of files matching :"+screenedList.size()
println "screenedList :"+screenedList