Innocence is Forever

On this Thanksgiving Day, I acknowledge it's special because of the memories of family and people I've known and loved.  Oh man, the memories of hanging out with my cousins in Hillsdale, smelling the cooking of food that teased us for hours, watching the adults become child-like and feeling the camaraderie of watching football on tv, no matter how silly it sounds now, are undeniable. There doesn't need to be anything more special than that.

I also enjoy Thanksgiving because, for whatever reasons, it's one of the few days I experience peace and quiet on a personal level.  I wrote a love letter to somebody this morning.  And while I wrote it, I was enjoying a source of innocence that doesn't show up very often, especially as I grow older.

I think that's why I enjoy finding my spirituality in the universe, which to me is simply "Life."  Whether you believe in a god, one Universe, multiverses or that it's all an illusion, it's all Life. 

The immensity of it all means good news to me.  The news? No matter what, we are all innocent.  No matter how thick your skin has become and the piles of regrets equally so, you can access your innocence to be bold, to be romantic or to start over.  It's still there, I assure you.  How can anyone not feel innocent when visualizing your place in the Universe.

So that's my Thanksgiving and perhaps a gift to those who've forgotten, or never had the feeling that it's a special day.

David

Particles versus Waves -- Beautifully explained in a 7 short minutes...

... not by me, though I try below.

So the "discrete" in "discretepassions" comes from the Neils Bohr's observation that electrons can only occupy specific orbits above a nucleus.  As the video points out, that was quite a shock to Newtonian folks who believed an orbit is an orbit and orbits had no rules associated with predefined heights.

http://www.youtube.com/watch?v=a6o9XjQOvHc
"Quantum Physics and the Nature of Reality: Neils Bohr, Charles Rutherford, Werner Heisenberg

This video caught my imagination and my desire to understand concepts at a deep level.  Something that makes sense.  Something that's beyond rote memorization.

We all know about the counter-intuitive explanation of electrons and the observation that they behave both like particles and waves.

As this short video explains, the rationalization of the dual nature of neutron behavior around a nucleous is achieved by the Heisenberg Principle.  Take some light or x-ray or some form of energy to observe an electron.  Problem: the energy you use to observe with influences the electron.  It moves it, diverts it, etc.  The end result is that you can never know the exact position or the momentum of that neutron.  At least you can't know both attributes at the same time.

The result: the wave-like display of electrons around a nucleus results from the fact that we, as observers, never know the position of an electron.  Why? Because our act of observing instrantly alters the positional attribute of that neutron. What Heisenberg concluded as that the wave explanation is really one based on possible locations of individual neutrons at any one moment in time.  Consequently the wave form that represents the presence of neutrons is really the manifestation of a spread of possible locations of those electrons.

Thus the dual nature of electron behavior, as observed by us humans. 

That was fun (and hopefully accurately summed up...)!

• 
  

Chipping away at manipulating and querying mongoDB sub-documents (with Groovy/Java)

I'm building a little library of commonly needed MongoDB Groovy scripts using the MongoDB Java driver. Sub-documents are my favorite aspect of mongodb scheming. They're how I like to illustrate a core difference between document DBs and the "joining tables" world of SQL. So I figured I'd tackle storing and fetching sub-documents.

There are more programmer/java-friendly strategies such as Morphia. My preference is to start with a reasonably low level API to understand some foundation perspective before jumping up a level or two of very helpful abstraction. It's a control thing, I'm sure... ;}

It's easy to use native MongoDB javascript to create some test data, so I started with that. Below I'm using a collection called "diary". This document identifies the activities and activity dates performed by John.

mongo
> use blog
> db.diary.insert ({name: 'john'}, {'activities':[]}); // setup activities array for subsequent content updates

> db.diary.update({name:'john'},{"$push" : {"activities" : { "date" : "20130812", "name" :"Go to school"}}});

> db.diary.update({name:'john'},{"$push" : {"activities" : { "date" : "20130817", "name" :"Bird watching on Rio Grande"}}});

> db.diary.findOne({}, {_id:0}); // select all docs; don't display the _id
{"activities" : [
       {
               "date" : "20130812",
               "name" : "Go to school"
       },
       {
               "date" : "20130817",
               "name" : "Bird watching on Rio Grande"
       }
       ],
       "name" : "john"
}

I now have a key called "activities" that collects sub-documents in array form. Each item in the array is a sub-document representing a date-stamped activity, such as "Go to school", which I do every day in one form or other.

In the code below loops through any top-level documents, looking for the presence of the "activities" key. Since we're talking MongoDB, there's no requirement that such a key exists in every document. The responsibility of whatever you decide that policy should be is implemented in your code!

Here's some tested code the performs the query I've been looking for.

// main...
    myApp.dumpActivities(fetchActivities('john'))
// ...end of main

def fetchActivities(name) {
  def activities = [:]
  def q = new BasicDBObject().append("name",name)
  def cursor = diary.find(q)
  while (cursor.hasNext()) {

    def activityDocs = (BasicDBList) cursor.next().get("activities")

    # DANGER -- This loop assumes one event per day...
    for (BasicDBObject activity: activityDocs) {
      activities.put(activity.date, activity.name)
    }
  }
  return activities
}

def dumpActivities(activities) {
  activities.each { dateStamp, activity ->
    println "Activity: "+ activity + " on " + dateStamp
  }
}

So we're looping through each document looking for a match on the key "activities". When found, we use the key casting of the value associated with the activities key to create a BasicDBList object. Basically, we're treating the activity key's sub-document as the array that it is.

The innermost loop processes each sub-document as the hash (represented by the BasicDBObject object) that it is, collecting each key-value pair ("date","activity") as encountered.

So! Now I have the basis of my standard query pattern. Next post should be about inserting new activity sub-documents with Groovy.

Magic numbers at it again: approaching critical mass of knowledge as video

When I was running Lutris Technologies in the mid-90's, everybody knew I had a fascination in two things: serendipity (as it influences business) and what I called "magic numbers."  I'll write about the serendipity stuff another time... but I used the concept of magic numbers as associated with the hiring of any new employee and the eventual impact on our Lutris culture.

• The 7th employee and, all of a sudden, the need to call meetings.
• The 10th employee and, all of a sudden, the need to hire an office manager.
• The 12th employee and, all of a sudden, the puzzling interruption in the perfect flow and distribution of knowledge (amongst all of us).

It was somewhere around 20 when I realized, "geez, we need a real CEO." That's another story in itself.

But, to get back to the real reason for this posting... and that's about what I've observed recently and that observation is that video has become a true knowledge base.

My favorite example is Charlie Rose at http://www.charlierose.com When I just feel like learning something knew or wonder if he ever interviewed somebody I'm curious about, I'll go to his website and search.

It used to be that you searched for text, such as wikipedia.  I still do that.  But if I'm in a real learning mode, I go video first.

The impact?  I used to think of Youtube as a resource for music and kitten videos.  Instead, I watch videos on Quantum Mechanics or a new salesforce.com feature.  Or, as I just did, I search Youtube for videos on "defining mongodb schemas."

It's a wonderful phenomenon.  The charlierose.com is particularly interesting to me because I have always suspected his politics and social views were similar to mine.  So I know I'm going to like his questions of those he interviews.  So it's more than a site of pure knowledge.  It's one that supports an angle that I relate to.

So, somewhere along the line, in the past 3 or so years, and maybe I'm just late to the knowledge party, but one of those magic #'s was reached.  I guess it's the # that represents a sufficient # of topics (relative to my direction of personal growth and interest) supported by a critical mass of videos.  It's a curious kind of transformation because you don't realize it until it's been there for awhile.  Fun stuff.

So back to my mongo video...

David

Tackling programming in chunks

Sometimes, out of nowhere, you discover you've acquired some wisdom over the years. Wisdom, in this case, was probably inherited from my Unix/Linux background... namely, break a problem into chunks, and attack it left to right.

For example, take a list of names, sort them and get rid of the redundant ones.

cat myNameListFile.txt | sort | uniq > myUniqListOfNames.txt

Unix has a nice way of pipe'ing the output from one app or tool to the next.  As you become familiar with the available tools, you start thinking in terms of how you can sub-divide tasks.

For example, you need to generate some customer numbers from your database.  This is a one off task, though it has the possibility of being useful further down the road.

You're not sure about the SQL.  You can do it, but it's going to take awhile to figure out that lengthy thing. And feeling comfortable with Unions and Joins can be quite a challenge. So, why not do it in chunks?  Why not do it in a series of SQL calls, feeding the results of the first query into the second.

But what if that isn't quite working for you in terms of doing so confidently.

This is where a bit of bash scripting or Groovy comes in.  There are more tricks to bash shell managing SQL queries and results than you may know.  I will address that in a future post.  It's how I survived before I discovered Groovy.

Here's the strategy:
1. Create a SQL query that gets all your customers' ID.
2. Execute the query from Groovy so that you can capture the results in a list. 

Maybe there were lots of conditions applied to that customer list.  Perhaps they're the customers who are not suspended and they reside in Ohio and they're new accounts as of 3 years ago.  Simple considerations, but nonetheless, considerations that lengthen your thought process and your SQL query.

Now you can move to the next phase or chunk.  All of a sudden that grand design of a SQL query has gotten a little bit simpler.  

Career bootstrapping for young people: an alternative strategy

At a Satellite coffee shop here in ABQ yesterday, I heard a guy I knew to be a successful local businessman giving a 24 year old fella advice on breaking into business after college.  He focused on how he should look, dress and act.  I kept waiting for him to get past the surficial stuff but, alas, he kept going on about how to shape shift into something acceptable to the corporate stereotype.

So, in a fit of my occassional righteousness, I stood up, looked at the the advice-giving guy and said, "Don't forget to tell him about deed" and walked to the other end of the coffee shop to continue my work.  To his credit, the business guy understood my point and immediately translated my point to the young man.

So, the point of all this is, yes, it's a good idea to start combing your hair, but it's also important to think beyond just "getting" that first job out of college.

1. Think in terms of "infiltrating" a business.  By that I mean, be determined to learn how the company works, from its ultimate business plan to how it implements that plan in manufacturing, sales and elsewhere.  Position yourself physically -- where you start in the organization.  Believe it or not, the mail room is a great way to achieve that, assuming there are still mail rooms in businesses.  As far as your "insertion point" goes, your ideal vision of that first job may not serve you best.  Ivory tower positions have their consequences.  "Big picture" ignorance is one of them.
2. Start on the manufacturing side of your ultimate goal.  From that perspective, you'll see the reality of things.  As a wanna-be software developer, I saw how the company hired hordes of young developers from colleges around the Bay Area and those kids never had any idea what happened to their software after it was handed off.
3. Then, the hard part, which goes hand-in-hand with big pay-offs, is to figure out the weaknesses in the system and find something you think you could improve.  Find a "coach" within the programming group, or general management, and tell them what you're seeing.  See if they'll advise you on a strategy.  For me, it was a fella named Mark Wong who said, "We're drowning in our current workload.  Here's a programming book.  Teach yourself how to program in our language and build a program for that missing process."

So, that's what worked for my career.  By shaving a couple of days off of how things went from design to manufacturing, I got noticed, promoted and even sent to school by the company to become a full engineer.  In today's world, if you're doing the software thing, I highly recommend you pick a company that has truly adopted the Agile methodology.  It's fundamental basis in transparency amongst contributing teams undermines the ivory tower thing.  Just make sure it's a true Agile culture.

So there it is.  I know times have changed.  But my gut tells me that this kind of career strategy is still a valid option.  Just be sure to comb your hair, for goodness sake!

Key to understanding Higgs Boson... field versus particle

It's a bit of a stretch going from javascript to physics, but why not?  On the web today, "answers" are all around us.  It's fun to find the ones that work for me and, perhaps, you...

I watched the video by John Ellis referenced below.  It was the first explanation by anybody that made the Higgs Boson thing click for me, conceptually speaking.

For of all, composite and elementary particles (electrons, protons, neutrons, photons, neutrinos, quarks, gluons, klingons (just kidding) and leptons) have no mass intrinsically.  From a parental perspective, they're like pre-teens.  Full of nothing but potential... It's through their daily travels that they acquire mass as you would pick up lint when you put on black clothing  and your dog wants some attention).  But some particles, like photons and gluons, never wear lint-attracting clothing...

So here are my video points from Ellis' video:

• The key is understanding that Higgs Boson introduces both a field and the particles (Higgs Bosons) that reside within that field.
• It's the particle that CERN detected, not the field.  The field is implied by the prediction and detection of the particle.
• It's the journey through the Higgs Field, which is everywhere in our universe, that gives particles their mass, such as electrons.  
• Certain particles that have no mass, like photons, can traverse the Higgs Field much like the bottom of a skier's ski... No Higgs Boson particles "stick" to the slick surface of that photon and therefore the photon acquires no mass.
• What did CERN discover?  They discovered the particles, or snow flakes, that reside throughout the Higgs field... namely the Higgs Bosons. So they did not discover the field itself.   They discovered the particles that reside there... the things that latch onto certain particles, giving those particles mass.
• So, another way of looking at it is that particles with no mass, like the photons that deliver light, are smooth and slick and therefore the Higgs Field that they plow through has little consequence in terms of build up of Higgs Boson particles.  And particles that acquire mass have a rough surface and therefore accumulate Higgs Boson particles along their journey.  The rough and smooth attributes are not necessarily attributes of particles.  I'm just using them as conceptual elements.

So, if you need to see moving pictures, watch the video below.  It's a good one.

If you want to know more about where the sub-particle name Boson came from (since Higgs gets all the press), be sure to read the web page at ibtimes.com.  Just another story from the amazing culture of India... with a great story about the role that Einstein played.

https://www.youtube.com/watch?feature=player_embedded&v=QG8g5JW64BA

David

alpha sorting (filters) with unobtrusive (jQuery) javascript

Often I see code for alpha lists that looks like the following...

<a url="#" onClick="javascript:performFilter('D')" name="D">D</a>

Pretty busy... and that's just the "D" letter...
But there's a much easier way to accomplish the same thing using jQuery (once the dom is instantiated)...
See the complete code below...

BTW, using href="javascript:void(0);" instead of "href="#" will keep the page from scrolling to the top when you click on the associated link.

David

<html>
<head>
<script type="text/javascript" src="http://ajax.googleapis.com/ajax/libs/jquery/1.7.0/jquery.min.js"></script>
<script type="text/javascript">

var letterElem
$(document).ready(function(){

  letterElem = $('#default') // "ALL" to begin with...

  div#alpha a').click(function() {

    letterElem.removeClass("chosen"); // remove previous node's green background

    letterElem = $(this) // cache the selected element node

    var letter = letterElem.text(); // grab the text node

    $('#answer').text(letter); // display the letter selected

    letterElem.addClass("chosen"); // give the selected letter a green background

    // askServerToFilter(letter); // here's your placeholding call to the server for a filtered query

  });

$('div#alpha a').attr("href",'javascript:void(0);'); // a's have to have href value...
});

</script>
<style type="text/css">
a { margin:0 8px 0 8px; padding: 1px 2px 1px 2px;}
a.chosen { background-color:green;color:#fff; text-decoration:none;}
span#answer { font-size:18px; color: blue; font-weight:bold; }
</style>

</head>
<!-- BEGIN HTML WITH NO JAVASCRIPT ANYWHERE -->
<body>
<p>An example of unobtrusive javascript... used to address alpha lists. </p>

<div id="alpha">
<p><a>A</a>|<a>B</a>|<a>C</a>|<a>D</a> ... <a id="default" class="chosen">ALL</a></p>
</div>

<p>Click on the alpha list above to get the letter selected... ANSWER: <span id="answer">?</span></p>
</html>
</body>

MongoDB Tip #2 - updating fields via Java/Groovy (Bicycle Shop Example)

First post of the new year, 2nd MongoDB tip... here we go!

Let's say that you're writing software for a bicycle repair shop. You need to update the "flat tire" status for a handful of customers because those tires were repaired by the repair shop team.

Using native MongoDB Javascript, you might craft the following:

db.bicycleJobs.update({jobId : { $in: [ 110234, 110433, 110511, 110766] }}, 
 {$set : {flatTire:false}}, false, true); 

Yes, we could cruft up a javascript file to do this, but you want to grab the customer job numbers (e.g., 110234 above) from a file or your inventory system.
So you decide it would be much more convenient to do it in Java or Groovy. Interrogating the native mongo statement we used above, you can probably guess what the update method signature looks like according to the Java Driver for MongoDB.

customers.update(document1, document2, flag1, flag2)

document1

This is our "select" that identifies the target rows based on job numbers in the provided list.

document2

This is our "action" document that actually states the operation to take place. Here we "set" the field called "flatTire" to false.

flag1

This is the "multi" flag. It indicates that the set operation should be applied to all rows that match the results of document1, not just the first match, which would be the default behavior. Having been burned in the past, that's what I call a nice default behavior!

flag2

This is an "upsert" indicator, meaning if there's no match to be had according to document1, then create the row, initialized with these two fields: jobId and flatTire. Might look good on the books, but let's keep this behavior turned off since it probably reveals a data entry error (with respect to the correct jobId).

So a more useful description of this Groovy method signature might something like:

bicycleJobs.update(findDoc, operationDoc, multiFlag, upsertFlag)

Now here's some code to tie things together...

def jobList = [] as BasicDBList

jobList << 110234 // until we're reading from a file or other data source...
jobList << 110433
jobList << 110511
jobList << 110766

def findDoc = new BasicDBObject().append('jobId', 
 new BasicDBObject().append('$in', jobList ));

if (verbose) {  // This bit gives us a little debug to see "the before"...
 def cur = leads.find(findDoc)
 while (cur.hasNext()) {
  def customerRow = cur.next()
  println customerRow['jobId'] + ': flatTire: ' + 
   customerRow['flatTire']  + ': customer: ' + 
   customerRow['name']
 }
}

def operationDoc = new BasicDBObject().append('$set', 
 new BasicDBObject().append('flatTire', false));

def multiFlag = true // update all matches
def upsertFlag = false

db.bicycleJobs.update(findDoc, operationsDoc, multiFlag, upsertFlag)

//You could repeat the "verbose" block above to confirm the expected results.

That's it for now. As I build my knowledge with native mongoDB, I'm trying to make sure I know how to do the equivalent in Java/Groovy. Once you understand how to use Hashes and BasicDBOObject operations to build a document, then it appears to be pretty straightforward.

A definition for "cloud" computing

Before "the cloud," you pretty much new where you data was getting stored. If you were on a business network, you probably knew that your files were stored on your local hard drive or on the network server located in the closet in the office. Or maybe you had heard that "the server" was hosted by a downtown Internet Service Provider company. You may have even heard that it was still your company's hardware. They just kept it cool and in a "cage."

So, with the cloud, how is that different?

If your files are now hosted in the cloud, then you no longer can picture where they reside. That's because instead of having one or two dedicated pieces of file-hosting hardware, your network folks have instead subscribed to a cloud service like Amazon.com. That means your network file serving has been "thrown over the wall" to a stranger like Amazon.com. What machines they use, what technology they use is their concern. All you know is that you still see your files. And you've heard your network folks love it because they let Amazon.com deal with the hardware, the backup responsibilites, the "failover" prevention, the security and so forth. In fact, they love it even more because they know they can keep adding more room for storing more files as the company grows. And all without having to order new hardware.

But what's this "iCloud" thing from Apple?

Well, it's just more of the same, except there's no network group at the office involved. Unlike Amazon.com, you don't have to deal with bringing up "an instance of a server." Instead, it's more of a consumer-focused file hosting service. It's just between you, your Apple device and the iCloud service. Again, you have no idea where they store your images or tunes. They just make sure you have constant access to what you "throw over the wall to them."

So what are the implications?

As long as Apple stays in business, you have "control" of your files. As long as Apple keeps their servers healthy, you have access to your files. You now don't have quite the concern for how big your disk drive is anymore, right? That's because your store your files on the cloud now.

Other implications? Of course, but I'm not in the mood to reflect on the big brother nature of all of this, or the house of cards scenario. It seems that this is direction the forces of business-driver evolution are taking the Internet. It's not an evil plot. It seems to make sense.

David

An adoptee's perspective on Steve Jobs...

Steve and I went to the same high school, Homestead, on the border of Sunnyvale and Cupertino. He was a freshman when I was a sophomore. Wozniak was a senior. Steve dated my next door neighbor Marla and showed up in a satin white tux with top hat at the 5 year high school reunion (according to my sister Carolyn who was in the same graduating class of '72).

As I tell people, while I hung out with girls, he was more of a "shop" guy. Our paths never crossed. Or they did but I didn't know him from Adam. But later I met him twice, once when he entertained a group of us from SCO (the Unix on Intel company) at NeXT with the purpose of convincing SCO to ship "NeXTStep", a development environment that he was trying to make more pervasive in the Unix world.

It was there, at the NeXT offices near Redwood City off of 101, visiting Jobs with two SCO VPs where I learned what a crutch powerpoint presentations are as well as Jobs' commanding knowledge of the industry. After Steve and I shortly reminisced about the good old days at Homestead High, one of our VPs got up to give him a presentation of SCO and our 2-tier distribution model, of which we were very proud. About to display the 3rd slide, Steve waved him off, asked him to sit down and proceeded to summarize the SCO business model as well as I'd ever heard from our own people. I think it's fair to say that our guy was pretty devastated to be brought down to earth so quickly... especially given how much he probably had rehearsed the days before in order to impress Jobs about SCO's unique Unix-on-Intel business model.

At the end of the meeting, it was already clear to us that he was a little nuts and, in a way, totally out of his league. He was trying to build a consortium without forming a consortium. And consortiums are difficult to control. Steve had never given the world the impression that he was a "standards kind of guy" or capable of adapting to the committee-driven nature of Unix standards. At the end of the discussion when we were trying to understand his real motivations, he finally said (to paraphrase), "Well somebody's got to stop Microsoft!" I kind of shook my head and thought, "this poor guy... he's still driven by his personal quarrel with Gates."

I later interviewed at NeXT to run their product engineering group. I again had a casual conversation with Jobs, but my interview was with Avie Tevanian, the architect of the NeXTStep Mach operating system. He impressed me as a sweet, got-it-together kind of guy. I would have loved working for him. But they never called me back and I never called them because I couldn't see leaving my wonderful cocoon of Santa Cruz and resuming that commute to Redwood City, of which I was well familiar after 4 years of Santa Cruz-to-Sunnyvale-and-back. Makes new cars old very quickly.

I'm writing this because last night, when I inadvertently brought up my browser to cnn.com and saw the news, I was devastated. Shockingly devastated. Yeah, I have a few cute stories about Jobs, and no, there was no way I would have enjoyed working for him given the butterfly nature of my love for software development.

But, as I was telling a colleague at SAMBA this morning, like me, Jobs was an adoptee. Of course, he was also a fellow graduate of Homestead High and our silly allegiances to famous people feel quite real. But way back in the 80's when he became famous, I looked up to him as a fellow adoptee because I loved how he handled it. It gave being an adoptee real respect. He loved his parents and he didn't let the adoption thing hinder his life one bit. At the time, I thought it was kind of weird that I didn't have the slightest inclination to find out who my real parents were. My harsh self-appraisal was accented by what I'd been seeing on television at the time. Groups of adoptees were easy to find on day time talk shows, whining about how incomplete their lives were having no knowledge of their "real parents" and were confronting the fact that the adoption process was somehow overly secretive. It was strange to me that, in the same reaction, I was both repelled by these people and somewhat guilty that I didn't share their protest.

Jobs made me feel like I was normal. Better than that, he legitimized some of the crazy things I'd done along my career path. His unintended stamp of approval on how you can lead an adoptee's life without driving yourself nuts and, in fact, turn it into an outlaw'ishly productive and inspired life, was important to me. I had no idea how important his inspiration was, or how deeply I had internalized that message, until my reaction last night.

Thank you Steve,
David

CNBC's "Explain This" makes KhanAcademy a Pioneer of the "HyperSchool"

Anybody think that CNBC has delivered the latest unanticipated Internet service that's a (revolutionary) turn for the better?

If you peruse my blog, you know I'm a big fan of khanacademy.org and Sail Khan's library of over 2,000 9-to-15 minutes classes. Perfect for the Web, for our rapid paced lifestyle and increasingly limited attention spans.

Why do I think this new features turns khanacademy into a new class of school I'm going to call "hyperschools"? Because it is the beginning of the destruction of our view of education as a silo where school is school and it happens during school. With "CNBC Explains", CNBC and Khan are showing that learning can be more spontaneous, more targeted, more convenient and more "in the moment."

The parallel is with hyperlinks themselves. They give the Web its ability to cross-wire information. Following hyperlinks, you can follow all kinds of trails of content (knowledge). Hyperschools, of which there is now only one -- khanacademy.org --, provide the same flexibility. Let me educate myself by attempting to read this interesting article about LIBOR (inter-bank trading rates) and I'll click on these "Explain This" icons as I see fit in order to more deeply understand what I'm reading (by following a quick 10 minute Khan class focused on specified, relevant topic (to the article).

Another metaphor is the Star Trek-like scenario of "I want to be expert enough to follow this article on Treasury Bonds, so I'll take a pill on everything I need to know to understand the article.

Hopefully, you get the idea. It feels like a subtle new B2B enhancement of a company's web site (CNBC to Khan)... but the more I thought about it, the more I like the feeling I got of yet more breaking of old barriers to education. Love it!

A more reasonable way to comment blocks of shell code...

I love using shell script for launching java and groovy apps because they're so good at setting the table in a way that keeps the application much simpler... especially when the shell script can handle common needs which is often the case for Operations-style functionality. Example shell script functionality includes

• determining if there's sufficient space on the file system,
• collecting files to process,
• configuring the environment (as in defining traps that clean up temp files and remove lock files when an app closes either naturally or via control-C interrupt, or re-directing I/O so that a control-C won't kill the process you're running).

Because operations-style applications suffer from out-of-site, out-of-mind syndrome, having a solid strategy with shell scripts that can manage and measure the operating environment, and scream bloody murder when things aren't right, makes them a worthy design component.

I've never liked "the fact" that to comment out a block of code in unix shell programming I had to insert # symbols in front of every line.

I found out over the weekend the ideal way to do it using a here document and the ":" operator, which is a no-op

# all this code inside this section document
# is now invisible to the shell interpreter
# Add to use a # anywhere.

And here's how to do the equivalent with a here document.

: <<STUFF_TO_PASS_TO_COLON
all this code inside this here document
is now invisible to the shell interpreter
Didn't have to use a # anywhere.
STUFF_TO_PASS_TO_COLON

Mongodb Tip #1 : dumping bson into legit json objects

Well, it's a bit embarrassing to start off with a hack, but what the hey... it made my life instantly easier.
So here was my challenge... to dump to file 40,000 BSON (JSON-like) objects from my mongodb so that I could slurp them up with Groovy 1.8 and its new JsonSlurper.  Then I'd be able to use dot notation to get the values I want.

Here's the sequence of what I did. BTW, in case you didn't know, there's no native method within a mongo session to just say db.hats.dumpAll()... at least I haven't found it yet. So you need to get outside the normal mongodb command line session and use these utilities that come with the mongodb distribution.

/var/lib/mongodb-linux-i686-1.8.2/bin/mongodump --db mydb --port 27017 --collection hats --query '{ }' --out dumper 
/var/lib/mongodb-linux-i686-1.8.2/bin/bsondump dumper/mydb/leads.bson

First, mongodump dumps the whole collection I called "hats".  But it's not in human readable form.  You need bsondump for that.  bsondump, by default, converts every row of mongodump output to a BSON string.

So, I thought I could just write a few lines of Groovy to convert each JSON string into an object and use dot notation to dereference the values I wanted.  But I forgot I was still dealing with a BSON string.  My problem was this:

{ "_id" : ObjectId( "4e56d1c780acbde57e951402" ), "size" : "8", "color...

As you can see, the ObjectId string is not itself in quotes and therefore messes up JsonSlurper. So I submit the following solution that worked for me.

def f = new File(fname)
def lineCount = 0
f.eachLine { line ->
        def line2 = line.replaceFirst('ObjectId\\(','') .replaceFirst('\\),',',')
        try {
        def slurper = new JsonSlurper()
        def res = slurper.parseText(line2)
        println res.size + "|" + res.color
        lineCount++
        } catch (Exception e) {
                e.printStackTrace()
        }
}
println lineCount + " lines encountered."

I used replaceFirst instead of replaceAll() to avoid overkill and generally screwing up other innocent content that might contain the right matching parentheses with comma.  Unlikely, but safe(r).

By the way, the reason for the exception handling is that bsondump outputs some stats, not in BSON format,  every now and then.  Odd.  Obviously, I could have handled that more gracefully, but the exception handling did the job and the lines encountered gave me the target number I was looking for.

That's it.  Got the job done.  Enjoy.

David

mongodb goodness, so far...

Lately, I've been working pretty extensively with mongodb.  I classify it as a "JIT DB", as in Just-In-Time Schema Database.  It's perfect for lazy moments when you're writing some code and it dawns on you that you need an additional field or even an additional table (called "collections" in Mongo).

"Lazy" is the wrong word.  mongoDB is in a class of technologies and strategies that foster inspired notions and reduce barriers (like time and patience) to assert your ideas. SQL doesn't do that for me.  The level of required schema pre-work and retrofitting has nipped some cool ideas in the bud... mongoDB encourages me to do it right now because I don't see any impedance!  Throw together shell scripting, Groovy and mongoDB and let's just do it!

Here's a quick example that will hopefully illustrate for you the low impedance of mongoDB (and lots of other unSQL databases)...

Let's sort a table called myData by a timestamp field.

db.mydata.find().sort({timeStamp:-1}).

This is equivalent to

select * from mydata order by timeStamp desc;

mongodb comes back and says something to the effect of "can't do a big sort like this without an index."  Well there you go... so you type

db.mydata.ensureIndex({timeStamp:1})

You try the sort again and it works. You've just experienced something like a conversation with your database!  "I can't do this... you know what to do..."

In full disclosure, I acutally use Groovy for all my Java-style development now.  I've completely lost interest in Java because 1) Groovy is way more satisfying and productive and 2) I, currently, have no
 need to use Java for squeezing max performance out of code.  I mostly use Groovy for batch-style work, updating Salesforce.com via the Web Services API and such.

With mongoDB there's a nasty little conceptual hurdle to jump over, especially switching back and forth between using native javascript commands and Java driver programming.
In Java, there are at least 2, 3 or more ways to construct a db operation

def doc = new BasicDBObject().append("lastName","Smith").append("firstName","Jack")
myData.query(doc)

...versus...

def person = [:] // Groovy syntax
person.lastName = "Smith"
person.firstName= "Jack"
def doc = new BasicDBObject(person)
myData.query(doc)

Straightforward enough. In the native mongo language, the query looks something like this...

db.myData.find({lastName:"Smith",firstName:"Jack"})

which is equivalent to

select * from myData where lastName = 'Smith' and firstName = 'Jack';

Now, because I'm a Linux guy and I love the power of intermingling shell scripts to glue Java/Groovy together as needed, here's one way way you might integrate that mongoDB (javascript) script language in a bash shell script using a here document.

function findUser {
        lastName=$1
        firstName=$2
        mongo <<EOF
        use employeeDB

        var criteria = {lastName:"${lastName}",firstName:"${firstName}"}
        var answer = db.leads.find(criteria)
        answer.count()

EOF
}


findUser Smith Joe


Enough for now.  For the next few weeks, I'll post some of the mongodb commands and concepts I found most useful.

Theory on Time as a symptom, not the cause, coming together...

In a recent Facebook discussion, I referenced the development at http://www.physorg.com/news/2011-04-scientists-spacetime-dimension.html re: "...space-time has no time dimension..." and how it reinforces my gut-level feeling about the concept of time.  Thanks to a response by my former Lutris colleague Daryl, a few thoughts came together.

1. The folks behind this article didn't go far enough to put some visual teeth into it (for us lay people), and 
2. Once again, Julian Barbour and his work come to my rescue to explain what could be going on...

Barbour, his book called, "The End of Time" and his site http://platonia.com mean a lot to me.  In particular, he has proposed a view of existence he's dubbed Platonia. Platonia looks like a typical landscape rendering you might see in an landscape architect's office except that it represents the likelihood of events (i.e., probability).  The illusion that is created as physical space changes in sequence is the perception of time.

While looking at one of Barbour's very recent papers, I saw that he appears to be driving to a quantum theory of the universe. Or at least he's describing why the universe could be described in quantum terms. As I understand things, time prevents folks from getting to a quantum explanation... but only if you see time as a building block and not an outcome of the dynamics of things.  If you remove time from the space-time equation and replace it with the sequencing of change, then his Platonia view of things as movements powered by probability, makes a quantum thoery of space, and its coming into existence, more attainable.

So that's as far as I can run with the original article I referenced at the beginning of this post.  There's more to come now that I have a new reason for exploring Barbour's work again...

Life is so cool...

Kicking the Sugar Thing in 2011

I have been a sugar addict all my life.  Just love the taste of the stuff.  Peanut M&M's are my favorite indulgence.  I've always relied on them for reward during crazy hours behind the computer.

But, starting the day after new year's 2011, I made an impulsive switch to a low-GI (Glycemic Index) diet, a.k.a. Atkins.  I'd been on Atkins before, so I knew what it was about.  But because I've been teaching myself to cook the past year, it seems to be more doable this time.

Two months later, I'm writing to report on how things are going.  It's real simple.  My mind is more clear than it's been in decades.  I don't have incredible impulses to eat.  And I'm generally very happy about the whole thing.  Yesterday, I played some driveway basketball with my youngest daughter, Claire.  She just turned 11.  There was something different about this session.  I felt great.  I felt light, upbeat and, most wonderfully of all, full of energy.  Maybe there is something different about getting all your energy from protein and not carbs.

The toughest part of the diet has been the eggs thing in the morning.  Started to gag just thinking about them.  But I've found that, for me, sautéing some mushrooms and bell peppers, at a minimum, make eggs a wonderful thing.  Just break a few eggs on top of the aforementioned sauteed items and the result is a wonderful blend of goodness including whatever I sprinkled on top. Chives. Yes.  Garlic.  Yes.  Now I'm moving on to including spinach.  Starting to border on eggs florentine.  Without the bread, of course.

So here's what I like about by new life style after two months of this stuff.
1. Lack of appetite is wonderful.  I don't feel like something I can't see is controlling me.
2. I'm much more into water these days.  Naturally.  I have no problem with craving water.
3. Exercise seems much more natural.  Can't wait to get to the gym tonight.

The only thing I _really_ miss is sourdough bread from Santa Fe.  I'm originally from San Francisco and sourdough is in my blood.  And sourdough from the Santa Fe bakery is awesome.

I'm reading a book called "The 4 Hour Body" by Timothy Ferris.  It's informative, inspiring and hilarious.  Even mentions Chad Fowler, who was an early days champion of the Enhydra Java application server when he was at GE.  Now he's a long-time Ruby evangelist.  That was pretty cool.

I'll report on the impact that book has had on me in a later post.  And it even helps me with my sourdough jones.  

Groovy Tip and How-To (gathering specific files from a directory)

I'm working on a service in Grails that offers all kinds of methods for getting dates or formats of dates with single line calls. For example, dateTimeService.getBracketDates() which will return the first and last dates for the current month (e.g., 2/1/2011 and 2/28/2011). It's not rocket science but I have no desire to re-invent the wheel every time I'm creating a new monthly report for our marketing folks.

Tip: Keep a code snippets (experiment) file

One of the awesome things about groovy is that it's a lot like script programming. That makes it very convenient to try pieces of code and functionality you're not sure about before adding it to your project. Be forewarned. I'm a vi nut. I'm aware that dev tools have snippet support. I just find this works for me, and perhaps for you too.

The nice side-effect of this practice is that you build up a semi-structured collection of groovy and coding knowledge to reference in times of need and senior moments (not you, me!). Guess you could use it for old girl (or boy) friend names too. Another nice side-effect might be "instant book" once you insert a few paragraphs between each snippet!

Put a System.exit(0) at the end of your snippet so that your groovy interpreter doesn't try to execute all the accrued code below. This won't protect you against using variables that have already been defined lower in the file. All I usually do is attach a number to the end of the variable name to get rid of the name collision. Afterall, all I'm trying to do is validate my code. I used this little practice of mine to refinethe How-to below before subjecting you to what might have been buggy code.

How-to: Gathering list of (specifically-named) files from a directory

Here is one of many ways to accomplish this task. I found this old code snippet from my snippet file and thought I'd share it. By the way, a site I love to consult on little basics like this is Pleac Groovy. It's not the most groovy-tized site in the world, but I like it's get-it-done blue collar perspective.

So this bit of code returns a list of log files that use the the naming convention of <directory path>/<year><month><day>.log. In particular, this code winnows the list of files down to the files stored in the month of February (i.e., 201102.*\.log). The regular expression is pretty lame. I'm sure you could improve it.

def baseDir = '/tmp/logs'
def originalFiles = new File(baseDir).listFiles()
println "number of files found:"+originalFiles.size()

// here's our closure, a match expression
def screener2 = { it.name =~ /.*201102.*\.log$/}

def screenedList = originalFiles.findAll(screener2)
println "Number of files matching :"+screenedList.size()
println "screenedList :"+screenedList

probability, observer and speculating "existence"

Boy, am I going to get in trouble for a title like this entry's... but this is the kind of stuff that flows through my thoughts when I've been working too much and sleep deprivation denies my brain the energy to cling to my usual collection of standard worries...  and why not end 2010 with a bang!

I'm fascinated by the following aspects of life, according to a number of sources...

• All substance is composed of mostly nothingness. The ratio of particle to the space it occupies is staggeringly unbalanced.
• Quantum physics explains the smallest particles as waves of probability.
• For these things to exist, there must be an observer.

So my thoughts on this have come this far. If you play a game of traveling as an observer of your own life and you speed by the faces of people you've met or walked past, your children as they grew up, the silly arguments you've add, the places you've been to... all blurring as you pass them, what is it that you're really experiencing besides a collection of memories? And how could it be different if you have the ability to re-wind and start over again?

It seems that these are all probabilities that you observed and, in the process of observing, also participated in. It's no wonder why life seems more and more like a dream to me.

But, if you keep digging and thinking about all of this as probability, then life starts to feel less like something tangible and more like a continuous experiment of unconscious messing with probabilities. Combinations of probabilities. Combinations of observations. It's not really like there are a bunch of universes ("multiverses") out there (or in here). It's more like there are all these (floating equations representing) probabilities and you as the observer are making them real. Making them "feel" tangible. So rather than thinking of all of this as an infinite set of what's out there. It's more like everything is being generated as we observe.

All of this brings up the "soul" thing or "consciousness" as one of those fundamental "what is it" questions. Why? Because it's this observer thing that quantum physics says must exist for things to exist. It somehow implies that a soul has special ranking, almost as though it's outside the definition or occurrence of probability. An analogy might be the patterns you see of iron filings when under the influence of a small magnet. The patterns are the assemblages of probabilities. And the magnet is You, the consciousness. (That imagery feels pretty powerful... but it feels more like an influence or steering of probabilities and not just a passive observer roll. Is the observer by definition an actuator?)

I have no authority on this topic. I'm not a mathematician. But I do love how the many topics of physics opens (and throughly energizes) my mind. And as the explanations of things these days gets wilder and wilder, I don't think there's anything wrong with letting a bit of intuition and subconscious rumination have its day. The math helps to illuminate a conundrum. But as our math and its conundrums continue to stack up over time, our ability to go back to our imaginations for speculative interpretation becomes all the more critical to create a sense of things we can conceptually grasp.

As fantastical as the concept of "god" seems, it's actually nothing compared to what this life (you, me, existence) is really all about. My intuition is screaming that.

World Series Giants and me...

This is one of those little kid inside me takes over posts.

I was lucky enough and old enough to see the Beatles from their (U.S.) beginnings, but I consider myself even more fortunate to have seen Willie Mays, Willie McCovey, Orlando Cepeda, Jimmy Davenport, Juan Marichal and Jack Sanford all through the 60's. Those Candlestick memories have dominated my thoughts the past few weeks and re-introduced my Dad in my day-to-day thoughts as well. I'm a pretty private person and don't intend to blog much about that, but I will say that I absolutely love my past as a young kid. I was pretty damn lucky.

The even better part of the whole Giants-Rangers-Phillies-Braves experience was hanging out with my New York daughter Amanda over the phone. She kicked things off by calling me from a downtown pub in Philadelphia the evening the Giants won that last game... wearing her Giants hat! Brave young woman and a TRUE Giants fan. Will Clark, Kevin Mitchell and Rod Beck were the guys during Amanda's early days. It was wonderful to feel so connected between PA, NY and ABQ! Thanks Amanda!

This is from the San Francisco Chronicle website @ http://www.sfgate.com :