Divide and Conquer all around us

In this article let me bring out the relevance of the famous problem solving principle – divide and conquer – in various disciplines all around us. While Software Engineering continues to sit in the centre, we shall see similar patterns in application of this principle with instances from different corners.

Let’s talk business

Why do we have quarterly targets in businesses? Why do we track quarterly performances?
1. We divide long term objectives into a series of quarterly objectives
2. Put-forth a quarterly plan / target
3. Work towards achieving this short term, comparatively smaller short term target
4. Measure and analyse the performance to come-up with preventive and corrective actions that go as an input to subsequent quarters
Some businesses do have monthly targets as well. All on the same lines – Conquer long term goals by dividing them into smaller, achievable, plannable, executable short term targets.

Let’s play

In one of the interview with Indian Cricket captain Mr. M.S. Dhoni on how he could execute chasing a big score so successfully and consistently, he mentioned how he divides the whole chasing into different stages and bat to achieve smaller targets in each stage.

  1. Divide final target to be achieved in 50 overs into smaller targets achievable in each stage of chasing like by end of 20, 25, 30, 35, 40, 45 overs.
  2. For each smaller target, identify when to accelerate, which bowler to target for scoring more runs
  3. At the end of each stage, based on how well we executed, decide how the next stages need to be executed

Let’s travel

As I wait for the dawn to set in a railway station, I am compelled to bring out how divide and conquer can be observed when we see a train. Accolades go to the person who was behind the design of the train

  1. Divide the whole train into set of coaches
  2. Build 1 coach at a time
  3. Connect coaches behind the engine and you get the train

I leave it to your imagination to identify various problems solved by dividing the train into set of coaches.

Let’s code

This is something very close to my heart – the art of problem solving in programming. It takes its root in writing algorithms or pseudo code and drawing a flow chart as it is difficult to solve the problem in one go.

  1. Divide the whole problem into series of tasks. When I started to program, I wrote them as series of comments. It was difficult for me initially to translate my thoughts into programming syntax. By breaking the whole problem into smaller tasks, I had to just focus on solving that simple, smaller task and get the syntax right.
  2. Implement (write code) each comment detailing the task as a separate method (preferably).
  3. Review and test individual methods

Practicing the above approach helped me to get the focus right and align the thought processes in the direction of solving the problem without worrying too much about the programming syntax.

Let’s engineer software

As promised let Software Engineering be the centre of the focus as well as be at the centre of this article. Any engineering discipline, be it software engineering or civil engineering, all advocates one process or the other that primarily divides the overall building process into a set of stages and a methodology that states what to be performed in each stage and how they work in tandem to build “concrete” things.

In case of the traditional waterfall model, the whole software development is divided into engineering phases – Requirements, Design, Development, Testing and Deployment – with goals set for each phase. When we view the complete software as one single chunk to build, we find it difficult to achieve goals within individual phases as well as the overall goal of getting the required software at the end. So, we divide the whole Software into set of smaller increments and iteratively build the whole system. There are various flavours of iterative development. But all are based on the fundamental principle of divide and conquer:

  1. Divide the complete system into set of sub-systems based on priority, complexity, relevance and other external dependencies.
  2. Analyse, elicit (requirements), design, develop, test and deploy individual sub-systems in an iterative manner.

Let’s study

Why do we study until 12th Grade in school? Why do we have 4 academic years in Engineering with each year having 2 semesters? Within each grade why do we have so many terms? Just imagine how our school life or college life would have been if we have to learn everything at once. While there will be an uncertainty of achieving what we want to learn, it will rob away the delight that we get when we proudly boast our progress from one grade to another. You can observe this pattern in every subject that we learn – be it art, music or science. We have to progress in stages.

Let’s draw

When I reflect back my very short experience (spanning very few days) in the Art and Decoration department during the initial days in college, I got a first-hand experience of how a huge poster is made to perfection. One of the key activities given to amateurs like me is to divide the huge chart into set of smaller squares by drawing cross-cutting lines on it. The expert artist then completes the huge drawing by taking it in steps, square by square.

Let’s watch time

How will our life be if we have not divided the day into 24hrs, an hour into 60 minutes, a minute into 60 seconds? Leave the rest to you. To debate one might say I shall watch the angle of Sun light like in olden days and live with it. But a Software Engineer rarely sees the Sun once he / she enters the office 

Conclusion

After observing the patterns of divide and conquer all around us, we can take a leaf out of this principle to solve any kind of problem. Guiding principle is to divide the whole problem into chewable chunks. Plan and execute each chunk making steady progress in our road to achieve the goal and see the bigger picture.
Philosophically, our life moves in only one direction – getting older day by day. To get a holistic growth in our life, we need to set forth long term, near term, short term and immediate term goals. Once you set forth these goals, you need to have the clarity on a daily basis to execute and achieve immediate term goals. Time is very critical during execution as the lost time or the time spent on some other activities can’t be bought back.

Advertisements
Posted in Software Engineering | Tagged , , , | Leave a comment

A Report on 2 day workshop on Big Data

Day 1, 16th June 2012

On a silent Saturday morning with the usual Bangalore monsoon trying to peep in, Prabhu and I slowly alighted from the 3-wheeled prominent (notorious?) entity of Bangalore traffic. At first, the small left turn that leads to the venue was blocked. To our fortune, there is a convenient way to the venue from its parallel road. After requesting the man guarding that place to direct participants coming to the workshop, we both reached the venue at 9:10.

As the training room Vanaka (a Sanskrit word meaning “the state of Brahmacharin”) greeted, I finally got the day I was waiting for. While on one side the room was getting colder with an A/C set to its minimum, our hands shared the warmth with enthusiastic participants getting in to the lab. At 9:45, with almost 90% of the participants in, I started to set the context for the 2 day workshop with an overview of Big Data – Why, What, How? and its relevance to the current decade where every one of us are contributing and consuming data in various forms.

Right from a school going kid to our grand ma’s and grand pa’s every one is a stake holder in some role or the other to the digital data that is getting crowded in multiple places. Big data is not just “big” in size, it is big in complexity due to various forms and origins it takes, it is big in processing, it is big in technology, it is big in applications and finally it is a big fascinating subject that has no horizon in the Knowledge Areas it embodies.  Being aware of the widespread documentation of “what is big data”, I consciously handed over the baton to Prabhu, a colleague and friend of Manish. While it is unfortunate Manish couldn’t join this 2-day party, he had found a very convincing talent in Prabhu (participants’ feedback reinstate this) who had worked on content extraction, searching and indexing from the initial versions of tika and solr while they were in its infant not-so-stable releases.

Prabhu started off with content extraction. Before tea and coffee could turn cold, participants were able to get their first hands-on in tika by extracting the content from PDF. Every participant system was loaded with dual OS – windows for the first day and Ubuntu for the second. Eclipse was pre-configured with Maven plugin. Maven repository was pre-loaded with all the dependencies of tika project. We didn’t want our participants to go through the suspense maven gives for a first maven project on new technology where maven has to resolve dependencies by downloading the dependent project artifacts from the configured maven repositories.

Big Data Workshop - Day 1 - Tika, Solr, Cassandra

Big Data Workshop – Day 1 – Tika, Solr, Cassandra

Big Data Tika Solr

Big Data Workshop – Day 1 – Tika, Solr, Cassandra

After a quick tea break, the once cold shivering room turned warm with everyone latched on to their eclipse. Lexicons of content extraction started to flow in. Meta data, dynamic variables, handling multiple MIME types, respecting other language were all around. We moved on to Solr witnessing the Sol(a)r (on) Eclipse.

With each one of us having our second brain in google search to often solve every problem that comes to us, this Solr definitely removed the darkness in understanding how indexing and search works. Every participant were given a handful of documents to index it. With the standard set of config files (don’t forget the main one – schema.xml – the “controller” config, if I can call it that way) obtained from Solr, participants were able to write their first method that index these documents. The terminologies of indexing – the useless (or misleading) stopwords, synonyms. Without the Solr server, isn’t Solr just a Lucene?

Solr was then running on Jetty inside Eclipse. The HTTP queries are fired from the browser to search content indexed in Solr. Starting with simple query that fetched every item displaying all the fields extracted by tika, we could move on to every single aspect of search – getting only the required fields, searching for a value present in any of the field, stemming, scoring, highlighting, proximity, faceting, Edismax. With our role model “Google” on one-side, every concept or term is explained with our search executing on our browser and the similar features implemented in Google. Here are the few queries.

http://localhost:8983/solr/select?q=content:*
http://localhost:8983/solr/select?=content:facebook&fl=content,score
http://localhost:8983/solr/select?q=content:facebook&fl=content,score&hl=true&hl.fl=content
http://localhost:8983/solr/select?q=content:google&fl=content,score&hl=true&hl.fl=content
http://localhost:8983/solr/select?q=content:facebok~0.5
http://localhost:8983/solr/select?q=content:%22facebook%20philadelphia%22~19&fl=content,score&hl=true&hl.fl=contentent
http://localhost:8983/solr/select?q=author_s:*&fl=author_s
http://localhost:8983/solr/select?q=content:*&fl=author_s&rows=2
http://localhost:8983/solr/select?q=content:*&fl=author_s,score&group=true&group.field=author_s
http://localhost:8983/solr/select?q=content:facebook&facet=true&facet.field=contenttype_shttp://localhost:8983/solr/select?q=content:facebook&facet=true&facet.field=id
http://localhost:8983/solr/select?q=content:facebook&facet=true&facet.field=author_s&fq=contenttype_s=application/pdf
http://localhost:8983/solr/select?q=content:*&fl=id&facet=true&facet.field={!ex=dt}contenttype_s&fq={!tag=dt}contenttype_s:application/pdf
http://localhost:8983/solr/select?q=content:*&fl=id&facet=true&facet.field={!ex=dt}contenttype_s&facet.field=author_s&fq={!tag=dt}contenttype_s:application/pdf
http://localhost:8983/solr/select?q=content:million%20cricket&fl=score,content&hl=true&hl.fl=content&defType=edismax&mm=2
http://localhost:8983/solr/select?q=content:facebook&mlt=true&mlt.fl=content&mlt.count=3&mlt.mindf=1&mlt.mintf=1&fl=id,score
Big Data Workshop - Day 1 - Tika, Solr, Cassandra

Big Data Workshop – Day 1 – Tika, Solr, Cassandra

As the dusk hitting the dust outside, we moved on to the key-value No-SQL database – Cassandra. It was a quick session where participants got to meet Cassandra hand-in-hand. Setting up the keyspace, coupled with column family and then firing normal and CQL queries.

The standing line on the clock forced us to part ways half-heartedly. I am sure everyone’s plate was full with food for thought that they could chew upon later in the evening.

After bidding good bye to participants, I decided to set up a pseudo installation of Hadoop on one of the Ubuntu systems. I had Ajit on Skype to ensure I get it right the first time. On a plain Ubuntu box, it started from Java installation, downloading and installing Collabnet’s Hadoop 0.20 conf-pseudo through apt-get. After formatting the name node, I had to provide permission rights to folders under the name node. After a handful of permission rights being changed or added for the folder (read from hadoop’s log file), Hadoop was running successfully and I started my run back home.

Day 2, 17th June 2012

It was a usual Sunday morning at Bangalore. Peaceful, very less traffic. I could relish the charm of Inner Ring Road that I saw 10 years back. I was still tensed if all participants could turn up at 9:30 on a cloudy and windy Sunday morning. Yes, they did. I should admit that each and every one were so enthusiastic about technology and learning. Ajit had already come by then. We both started to copy all the downloaded stuff that I got in the Linux machine that I set up last night. Participants who dropped in early like Guruprasad too rendered a helping hand. Thanks to a handful of USB drives that speeded up the whole process.

I started off with a flash back of what we did on Saturday and handed it over to Ajit. We decided to start with NoSQL and Neo4j as we thought we might not get to do justice to it if we schedule it after the two monsters – Hadoop and Mahout. Ajit started off with the concepts behind No SQL (Not-Only SQL), various forms that it takes (for example key-value, Big Table, Document, Graph), pros-and-cons of each, various industrial players currently in the market and moved on to explain Graph database with specific reference to Neo4j. With most of us heavily using facebook and linkedin, it made more sense to model the social / professional graph rather than a conventional relational one. Ajit then moved on to explain Neo4j in depth explaining the APIs, Cypher Queries, Spring-Neo4j, Neo4j entity classes, Neo4j-specific annotations and a demo. After a good deal of conceptual overview of Neo4j, participants got their first hands-on in Neo4j. The example was pretty much apt for a Sunday – Movie, Actor, Roles, Director.

Big Data Workshop - Day 2 - Neo4j, Hadoop, Mahout

Big Data Workshop – Day 2 – Neo4j, Hadoop, Mahout

Big Data Workshop - Day 2 - Neo4j, Hadoop, Mahout

Big Data Workshop – Day 2 – Neo4j, Hadoop, Mahout

We then moved on with Hadoop installation. We first downloaded CDH3 (Cloudera Distribution Including Hadoop Version 3) repository using apt.

Create a new file /etc/apt/sources.list.d/cloudera-cdh3.list with the following contents:
deb http://archive.cloudera.com/debian maverick-cdh3 contrib
deb-src http://archive.cloudera.com/debian maverick-cdh3 contrib
Update the APT package index:
$ sudo apt-get update

We started hadoop installation process and rushed to satiate our hunger.

Big Data Workshop - Day 2 - Neo4j, Hadoop, Mahout

Big Data Workshop – Day 2 – Neo4j, Hadoop, Mahout

Big Data Workshop - Day 2 - Neo4j, Hadoop, Mahout

Big Data Workshop – Day 2 – Neo4j, Hadoop, Mahout

sudo apt-get install hadoop-0.20-conf-pseudo

When we were back hadoop had already installed and was waiting for our configuration to be done. We started off with the formatting of name node.

bin/hadoop namenode -format

After the above command, we had to set the permission rights appropriately (777) for the user (in our case it was root) to name node folder and SecurityAuth.audit file under log (Check out the hadoop log file for further details). It was then time for us to start hadoop.

for x in /etc/init.d/hadoop-0.20* ; do sudo $x start; done

 Hadoop didn’t start the first time. It needed further permission rights to folders it had created under the name node. After setting the permission rights we again executed the above command. Some got to meet the big elephant (hadoop) soon, some a little later and some couldn’t at all. After troubleshooting hadoop installation with half success we proceeded with our first Map Reduce job on hadoop in the interest of time. We had in hand a working example on word count, secondary sort and patient citation.

1. Move the input files to HDFS

> bin/hadoop dfs -copyFromLocal Hadoop_Examples/src/main/resources/word_count word_count/input/

2. Deploy the Map-Reduce job on hadoop

> bin/hadoop jar  Hadoop_Examples/target/hadoopexample-1.0-SNAPSHOT-job.jar  word_count/input/ word_count/output/

3. View the results of MR job

> bin/hadoop dfs -ls word_count/output/

We now resorted to driving the yellow elephant (Mahout). Ajit started off with the concepts behind Machine Learning and its applications. How many of us have bought products just because it flashed on our screen while buying another product? How many of us got back in touch with our old school friends or colleagues with the help of facebook and linked in? It is a human intelligence derived from an unimaginable size of data.

Focusing specifically on recommendation engine Ajit started to reveal some of the secrets behind the screen. User based and Item based recommenders to start with and then continued with components of Mahout Recommender Architecture – Data Model, User and Item Similarity, User Neighbourhood and the whole abstraction of recommenders. Terms like Pearson correlation-based similarity, Euclidean distance similarity, Cosine Measure similarity, Slope-one recommender started to flow in.

Participants got to drive the yellow Elephant with a recommendation engine example using PearsonCorrelationSimilarity, GenericUserBasedRecommender. Ajit then concluded with a brief note on how to run Mahout application code as MR jobs in hadoop.

While the 2 day workshop had to end as the Sun has to set, our voyage on the Big Elephant has just began.

 

Posted in Big Data | Tagged , , , , , , | Leave a comment

Test Driven Training

The thoughts of naming the stated training methodology as Test Driven Training (TDT) came after going through Kent Beck’s book titled Test Driven Development (TDD) By Example. Once I decided the name and then related my thoughts to TDD, my thoughts became more clear and structured. At the end I am convinced that I would practice TDT in whatever training that I do through our Ram Software Engineering Labs.

Goal of Test Driven Training (TDT)

After coining the name on the lines of TDD, I am tempted to state the goal of TDT on similar lines. The goal of TDD is “Clean code that works“. After a long hunt to get a similar crisp punch line I landed up with the statement “Learning that solves”. The objective for everyone of us to undergo a formal training is to learn and acquire those specific skills that help us solve problems in that area.

There are two possible pot holes that one might fall into while undergoing a formal training

  1. Long sight: Too much focused on teaching all the concepts and building a huge foundation that attempts to solve every damn problem in the world. What I think would happen in this scenario is that participants get too bogged down with the details that they get lost in the mid way and at the end they fall short of solving simple problems in that area. I can relate this to a development team that is too much focused on building a highly robust Software while at the time of deadline the final product is not ready to solve the most critical problem of the customer.
  2. Short sight: Here the training is too focused on solving only the problems in hand without understanding and analyzing the solution – what, why and how. Participants become dependent on others for helping them solve their problems every time. They don’t think and try to relate one problem and its solution with others. On a similar angle, I have seen in many places where people code for every functionality without bothering to nicely organize and restructure the redundant piece of code.

The primary goal of TDT is to help prevent us falling prey to above two problems. It aims at solving right and required set of problems and at the same time providing the required foundation and mind set that help solve future problems as we travel down.

Rules of TDT

The goals of TDT are best achieved if we adhere to the following rules or guidelines of how TDT has to be practiced.

  1. Work with the sponsor of the training and get Acceptance Tests for the training. This is drawn on the similar lines of UAT scenarios prepared at the beginning of Software development. Acceptance tests in TDT refer to a list of solvable problems that the participants are expected to solve after they undergo the training. Sponsor is involved not only in finalizing the topics being covered but also specific problems that participants have to solve at the end.
  2. Pick a problem from the Acceptance Tests and solve it with participants. Let them feel the excitement of achievement.
  3. Analyze the problem and solution by asking
    1. WHAT each step in the solution means
    2. HOW it works
    3. WHY is it included in that step and WHY NOT other ways of performing that step
  4. Instead of trying to answer all the open questions arising from the previous step at the same time, maintain a list of “To-do” items where all open questions are logged and tracked
  5. Pick the next problem and try to solve and analyze. This time, while trying to add open questions to the To-do list, we check if the question is already part of that list. If yes, then we try to understand the concepts related to the answer of that question. Instead of learning the concept during the second iteration, we can apply the rule of three.
  6. Continue solving the remaining set of problems. At the end, learn just enough to answer the open questions in the To-do list. Knowing what is “just enough” is an art and needs to be developed iteratively through introspection and participants feedback.

Advantages of TDT

TDT ensures

  1. Participants are continuously engaged with hands-on session in solving problems
  2. No theoretical boredom
  3. Training is objective oriented
  4. Sponsors are guaranteed that their participants at the bare minimum solve the immediate set of problems that they get in their business and with good probability solve problems that come in future in the same area
  5. Optimized usage of all resources involved – time, effort, cost with an optimum quality
  6. Devising a method to formally state the acceptance criteria for training by the sponsors
Posted in Uncategorized | Leave a comment

Cooking while Programming

Software metaphors

As Steve McConnell mentioned in his famous book “Code Complete”, Metaphors help you understand software development process by relating it to other activities you already know about. In his book, he vividly illustrates subtle differences between bad, good and better metaphors. I too agree with him on “building” (like in construction industry) a software as it implies various stages of planning and execution that vary in kind and degree depending on what is being built. I also enjoyed reading oyster farming and its relation to incremental software development. In this article I have tried to relate software to cooking and in that attempt wishing that my Mom understands what I do at work.

Why cooking?

First motivation to choose cooking is the interest I had (yes, past tense:) in it when I was a lonely bachelor hosting lunch (during weekends) and dinner to my colleagues and friends. I could relate the joy I obtained when the food turned out well and was enjoyed by the people who visited me to the joy of my code or product being used by customers. I even saw a similar joy in my mom’s face when my brother and I hogged her food. On the other end, I could also relate the dejection I experienced with my own food to various defects any other person found in my software.

Second motivation is the various scale of cooking that I could relate to Software development. It ranges from cooking for oneself, to cooking for dear ones at home, to guests at a family function and cooking for a function that extends beyond the family circle (I mean here events like a South Indian wedding that lasts for 2 to 3 days).

Third motivation is the kinds of setup under which a cook prepares food. When we prepare food at home we cook for ourselves. For a bigger function we either seek the service of a caterer to prepare food as per our interests or order food from any of the restaurants or food outlets.

Final motivation that gave me confidence was when I saw various books ending with “cookbook”, like HTML cookbook, Java cookbook, etc then I thought this idea is not going to be too absurd and foolish.

From Home to Caterers to Retail Outlets

Whenever I was alone at home and cooked, I never bothered how I cooked. I am fine as long as it fills my stomach. I can relate this to the assignments I did during my under graduation. As long as it gave the necessary output that my professor asked in the assignment I was happy to submit it.

A bit more elaborate process when we cook for our dear ones at home. As people at home are usually forgiving (some exceptions still exist 🙂 to the short comings in the food, we tend to take it easy again on the exact cooking procedure that assures good food at a higher probability. I could relate this to intranet or internal applications developed in house. Again, some internal customers are very demanding and they insist on a full fledged elaborate software to be developed with strict adherence to all the standard processes.

When we come out of our sweet home and start delivering goods and services to others as part of our business, it is no longer a cake-walk. It is going to be a serious business. If we just put ourselves in the role of a caterer cooking for a South Indian wedding, we expect our customer to be delighted with our service matching or exceeding his requirements. We wish to take pride and get that elation when a guest who had our food comes to us or our customer and tells “Amazing food, really delicious and great service too!”. We also expect this great work to spread through words of good references that fetch us next set of customers. All sounds exciting, but how can we as a caterer achieve this? Will the cooking style we used in the above two setups fetch us this glory. If you are with me on this article till now then your answer should be an unanimous big “No” for this question. Right from estimation, planning, preparation till execution and delivery, we should adopt a methodical organized approach that assures (if not, at least a higher probability) of success. I can relate the service a caterer provides to the software we develop as per customer specified requirements.

A retail outlet (consider restaurants and manufacturers of packaged foods) too expects a similar kind of glory through their customers. They always wish for the demands on their products to increase. On the contrary, a retail outlet does not prepare food based on the requirements of a single customer. It generates requirements based on what people in that area look for, their individual expertise in preparing those food and their knowledge of retail outlet industry. We expect new food items or products (through innovations or creativity) to come from an outlet set up. Being the owners of what they produce, they take pride of what they produce on one hand and on the other hand they become more answerable to a larger set of diverse customers. How many of us read through or at least notice various details that are presented along with the product. It ranges from simple manufacturing and expiry date / time to full fledged ingredients, calorie chart, cooking or eating tips (like appropriate side dishes or toppings), etc. I can relate this retail outlet setup or packed foods manufacturers to a product development setup in the software industry.

Preparing to cook

Take for example, a guest has come to my home and I am planning to offer him a cup of tea. To ensure that the effort that I am going to put in doesn’t get wasted and instead should impress the guest, I will have to ask him if he likes tea in the first place, if yes then I will have to understand his or her specific preferences – green tea or black tea or with milk, if with milk should it be thick (more of tea, less of milk) or light, with sugar or without sugar, very hot or mild hot, etc. For a simple cup of tea itself there are so many details that need to be understood from a customer point of view, then imagine the level of detail we need to go into the requirements of some large scale preparation.

Before we begin cooking we need to first understand what we are going to prepare. To enthrall the customers’ taste buds it is very important to gather all their expectations. The most critical part is to analyze and dig out those expectations that are not obvious. For example what is the level of spice the customers would prefer, any specific set of items that should not be used in the cooking like garlic, mushroom (yes, I don’t like them), how it needs to be presented & served, etc.

I am sure you agree with me on the fact that it is always better to clarify all doubts we have on menu or requirements upfront before starting to cook rather than customer pointing it out after he starts eating the food. It is going to be costly in terms of time, effort and money if any changes have to be made in the menu or if some flaw in the prepared food is found out at the end. Some costly mistakes will even make the food inconsumable or wasted. The cost of change increases as we proceed with cooking – later the change arrives more is the cost to do that.

Similar to this aspect, when we build software we need to understand all the requirements that the final package has to meet. Some will be explicitly stated by the customer, some will be implied and some will have to be dug out through a thorough analysis of requirements.

Now, let us take the case of cooking something new or something that we are not used to or to identify the mistakes or shortcomings early so that we can fix them at a relatively less effort and cost. Take for example, we are cooking dhal fry (an Indian dish) for the first time. The first step is to boil dhal (a type of yellow pulses). It is better to check if the dhal has boiled well at the end of this step before adding it to the rest of the mix of dhal fry that are cooked separately. We will not be comfortable in digesting all the requirements and procedure in a single step especially when we are an amateur. In such a situation, what I think normally most of us do is cook step by step. We would like to check how at the end of each steps our dish has shaped-up and if “all is well” and at the same time prepare for the next step to be followed.

In Software development too, we have various forms of incremental or iterative development. Each form has its own way of defining what needs to be achieved and how in each step.

Before marriage, when my parents visited my wife’s place they were offered an Indian dessert called Kajar Halwa prepared by my wife. She wanted to ensure that she impresses my Mom. So, she did a couple of tries on her Mom to understand what it takes to prepare that dish before specially preparing it for my Mom. Imagine a case when a customer is not too sure about what he wants or even the previous case of cooking something new. Before actually developing the dish we have to deliver, we would like to experiment or try out or cook a sample for the customer. Customer might also use this strategy to assess how we cook. One thing that we need to set forth upfront is the objective that needs to be achieved at the end of such a trial. Sometimes we will have to be determined to throw away the food we tried and cook it afresh from scratch for the food to be delivered to customer.

We come across similar situations in Software development too where we have to either make our self comfortable on what we have to build and how to make our customers comfortable on what they want and how we build that. We sometimes use terms like prototyping or proof-of concept or piloting for the experiment or trial we do.

Estimating what we need

Let us take our previous example on Dhal fry. Let me think that my invitees have come home and waiting for the lunch to be ready. I am in the kitchen cooking this wonderful dish. Feeling guilty of making my guests wait and leave them alone, I go to them every ten minutes asking for 5 more mins. Every time I went I became more apologetic and shameful. After 20 minutes, I find that I forgot to buy curry leaves to garnish my dish. Then I run to the vegetable shop in the ground floor to get that. After 40 mins when the dhal fry is ready, even though it has turned out well, my face more prominently boasts an apologetic figure rather than a pride figure for making them wait. Do you think your guests will be equally patient next time or you want them to equate your 5 mins to 40 mins? How worse will it be if the same lapse is made in a formal setup where customers are in place of guests who have ordered Dhal fry?

Now comes the dire need to estimate cost (that includes list of all the items necessary for the dish to be cooked), time and effort required to prepare what we have to deliver. We will have to do this to tell our customer when he can expect his Dhal fry to be delivered and what price he has to pay.

Before we estimate cost, time and effort, we need to understand the requirements clearly. For example, to first find out how much Dhal fry needs to be prepared, we need to get the answers for the the following questions.

  1. How many people are expected to have the food
  2. What type of crowd is expected and there by determining what would be the average consumption of dhal among the people
  3. What type of dhal customer needs (especially customer’s special or specific requirements) or most of the customers would prefer (for retail outlets / restaurants) like more dilute / liquid, more of onions, less spicy, less salty, etc. This in a way will help us identify how much quantity of each of the ingredients would be required

In one of our family wedding we were discussing with the caterers before awarding the contract. The first thing that they try to understand and estimate is the overall size of the contract so the usual questions were: how many days the function is going to be held (usually in our type of wedding it will last anywhere between 1.5 days to 2.5 days), how many courses of food have to be provided (as some will mandate a mini meal or snack during the evening apart from the dinner), how many people are going to attend each of the course and finally the exact menu we have in our mind. Till the last part the caterer himself drives the conversation with a check list derived from his experience in conducting and / or observing such wedding party. Most of the questions will sound like “Do you want this or are you planning to do this?”. Some of the questions will even strike us with some of the tasks or details we have totally forgotten. Believe me, that is the time we start admiring and respecting the Subject Matter Expertise present in front of us in the form of caterer.

In Software too, we first try to understand the size of the software we are going to build. Things like – number of features customers are asking, how complex the processing logic are, how many screens, how complex each screen are, how many entities are managed by Software, how many external Software our system has to interact with, etc.

Once the caterer understood our need, he immediately gave an overall cost estimate along with the details what all responsiblities he will undertake and what is agreed between us on the menu and number of people who will attend. He doesn’t go into the details of how many quantities of dhal, wheat, rice, etc to be purchased, what is the rate of individual items in the market, etc. This can only be possible only if the caterer is well experienced in organizing this kind of wedding parties and seen what it takes to deliver and what is the profit margin they should get. Some caterer might use information he has collected from his past experience to arrive at a final figure. These data will tell the caterer how much it costed them to conduct similar wedding parties in the past, what were the assumptions that failed and resulted as a risk, how much margin he could get, etc.

While taking up Software projects, we too adopt a similar approach if we are familiar with the solution to the customer’s problem. Some of the adoptations have also got the name as Wideband Delphi. These techniques have included some more guidelines in them to ensure that the estimated is a well calculated, thought over gut feel estimate.

I don’t know much how the caterer is breaking his total estimated cost for the wedding party to each of the schedule he has to deliver. But I am guessing, he would allocate percentage of the total budget to each of them again based on his past similar experience and information he has collected on his and his team’s performance in the past. We call this approach of estimation as Top – down approach in Software estimation.

Sometimes for a small family function we do call our favourite cook and tell him what we want him to prepare. He will tell us what all are required to complete that job. We then took individual items in his list and put the estimate for each one of them. Included cook’s charges too. Then finally arrived at the overall estimate. We call this type of estimation as Bottom-up approach in Software estimation. The complete list of tasks / activities that describes the overall scope of what needs to be done is termed as Work Breakdown Structure. We normally use it as a starting point in our bottom-up approach.

Knowingly, with guilt, I am leaving the remaining part of estimation and planning that is done in Software as well as cooking. This normally covers how many people are required to complete the job in hand, what is their availability, which task is dependent on which task so that they are relatively ordered and finally a schedule that tells how every activity has to be performed in which order, by whom and at what time. We will then come to know how much time it is going to take to complete whatever we need to deliver.

Using ready-made goods

When I sit back and compare how my grand ma cooks, my mom cooks and my wife or me cooks today, there is a substantial change in the way we prepare our food. Lots of spices that my grandma hand picked and ground to make a rasam or sambar (typical South Indian gravy that people mix with rice and have) powder are now readily available as ready-made powders. We just read instructions written on the packet to make use of it and prepare rasam or sambar in a much lesser time and effort.

In Software too, it was taking a lot of time, effort and cost to build a website. Now all the components are readily available and it takes much lesser time to build the website and that too it is so intuitive that any amateur can make use of it. People have to assemble all the pieces together and start using them. We call these ready-made goods as reusable components, frameworks, packages, libraries, etc. Many such components are free of cost and they are often referred us open source software. When we consume or produce such components, we need to strictly follow or publish the interface specification or contract the component is bound to.

Keep mom and grand mom’s tips in mind

I still remember my conversation with mom during the days when I started cooking alone. She explained me the basic framework for cooking – heating the vessel, sorting the lentils, frying the vegetables, adding spices and frying, boiling the mixture with spices, adding the boiled stuff like pulses and boiled vegetables and finally garnishing. This basic principle helped me to cook various dishes. She also told me how to choose which vegetables to fry and which vegetables to boil. There are various common cooking tips or solutions that our ancestors have told us which we apply today to prepare various types of dishes.

In Software too, there are common strategies or approaches devised and advised for various problems having the same pattern. We call these strategies as design patterns, architectural patterns, application integration patterns, etc.

The main intent of having these patterns in every space is to not reinvent the same wheel when there is a proven solution for a similar kind of problem.

Time is over, Can we cook now?

When I set forth to right this article I never anticipated it to take this much time and this long. Finally we have reached a state where we are all set to cook. My grandma used to use ammikallu (a stone to grind or bruise things upon) to grind all the spices where as my mom brought in mixer grinder to grind the same in fraction of seconds and with very less effort. Think about a kitchen where devices are there to do all the cooking tasks at one place – right from washing the vegetables, to cut, to boil, to microwave, to clean the dishes, to exhaust the smoke etc to another kitchen where everything has to be done manually and at different places. I would compare a kitchen to Integrated Development Environment (IDE) Software that we most of us use to write a computer program or a commercial product.

The more organized the kitchen is the more easy for the cooking team to prepare the dish. When I mean organization of kitchen it is the arrangement or placement of all the things that are required to cook. Right from the placement of the gas stove to the place where dishwasher is kept help in the overall cooking process.

While building a Software proper organization of all configuration items in the repository and consistent integration & configuration of IDE in every Software Engineer’s system greatly assists in simplifying the construction part of the Software. Like how cook prepares food, we write programs or routines in any of the computer language to build the Software.

The more organized every member of the cooking team is the less is the confusion at the time of cooking. If every member of the team knows what he needs to give to others at what time, where to take things from, where to keep things back, which vessels to clean, when and how to talk to other people in the team most of the problem in cooking is solved. The only thing that will remain would be to do each one’s tasks correctly and completely and they verify the completeness before giving out what they have prepared.

While programming too we expect consistency and discipline in all the members of the Software development team. Every one should adhere to the agreed coding convention and standards, understand-implement-check-integrate their piece of code that conforms to the contract or specification stated at the time of planning (designing). We use the term unit testing to refer to the process of verifying ones own program for completeness and correctness.

If you look back at the history of Software development it is difficult to imagine how our previous generation programmers have managed to develop big applications without proper tools that are readily available today. I had given up many a times in the middle of this thought process. Though I am not too old to talk about history I will just compare what I used ten years back to complete my programming assignments to what I use today. I open VI editor type my programs, keep a book on the side to check the method signature & language syntax, use command line interface to compile a program as well as to execute it and repeatedly execute it . Now I use sophisticated IDE where all relevant tools are integrated or can be integrated through plugins. The productivity has multiplied N fold because of the advent of these tools and technologies.

Why are they poking their nose?

Let us imagine a stage portraying a cook preparing food with the mind set that he is the boss of kitchen and he knows everything to complete preparation of food. There comes a supervisor entering the kitchen and proceeding towards this cook. While the cook has not finished preparing the food, he has to respond to all the supervisor’s questions even though he doesn’t like it. Cook murmuring “I know what to do. I am taking care of everything. Don’t worry. Please allow me to finish it.” Some times a potential ego clash can also erupt. On the other hand, the Supervisor thinks “My managers have entrusted me this responsibility to ensure that all is well in the kitchen. As they are answerable to their customers, I need to do my duty to ensure that my Managers are well informed and they never land up in a state where they are taken by surprise. So let me find out from the cook if everything is taken care of”.

In Software industry too, we perform audits that are meant to identify the gaps in what is planned against what is being built so that relevant stakeholders are well informed. This enables people who are involved to address these gaps well in advance by taking appropriate corrective and preventive actions.

Oops, salt is more!

Now I managed to complete my part of cooking. I take a spoon full of Dhal fry (don’t get bored with my Dhal fry) and taste it. “My God! it is salty!”. According to me, fixing a defect in a food is more challenging to me than removing a bug (we call it this way) from the software. What do I do for the extra salt? Can I add more plain boiled dhal to the mix? Can I mix a paste of fried gram dhal to the mix? Or add more water and make the dhal dilute? I hate this situation. I should have added salt in steps. What if I have added more salt in one of the steps. But what to do. Can’t bear the shame of people criticizing my Dhal. Call up my Mom and ask her. Or hit the search engine on the browser and look for help. Somehow, I finally managed to get a better tasting Dhal fry.

Sometimes I don’t feel confident with my own taste buds. Even if I find the Dhal fry okay to pass it on to the dining table, I feel more confident to call the best connoisseur at home who critically comments on the food inside the kitchen and get his or her remarks.

In Software development too, we do various types of testing at various levels. A programmer first tests what he has written which we refer us unit testing. After successful testing of his unit he integrates his module with rest of the modules and performs integration testing to check if his module works in tandem with other modules to achieve the overall system objective. Finally we give the Software for overall System testing to Quality Assurance (QA) team or Independent Verification & Validation (IV&V) team who are considered experts in understanding what and how the Software has to perform.

Are we ready?

Once I went to a restaurant and ordered for a North Indian meal. After around 15 minutes there came a waiter carrying my North Inidan meal in a plate. When I got the plate, I came to know that a cup of gravy was missing. I brought it to his notice. He said “Sorry Sir. I forgot it. I will get it for you”. When I started to have food, I found that the spoons given to me were not washed. I called my waiter again and showed him the spoon, He said “Sorry, I didn’t check it. I will get you a fresh one.” Finally when I took the Roti to eat I found that it was charred. I called my waiter friend. This time I lost my patience to explain the problem. I just lifted the roti and showed him either side of it. I can notice the shame in his face. Just to save it, he added saying “Actually Sir, I made another Roti after seeing this charred Roti. But, while delivering the plate to you I accidentally took the old one and placed it on the plate. My bad. I am extremely sorry. Kindly excuse me”. Will you opt to come to this restaurant next time? Recently I went to a Pizza shop and ordered few pizzas for takeaway. Before they delivered the pack to me they took my order and a check list and started ticking each one of them as they go through the list. I then understood how these chaps don’t forget to keep any of the items consistently.

We normally relate this step in Software to Build & Release process. Many improvements and automation have taken place in this part. We do check if all the deliverables including for example user documentation, if promised are delivered along with the Software place. We check if all the promised features in the release are tested and cleared. Sometimes, we also randomly check to confirm if the system functions right. We ensure upfront that right versions of each of the artifacts are bundled in the version. Finally, a release note or document that puts all information and pointers related to the release. Most of the projects do have a release checklist that they use to review or audit the deliverables that are about to be delivered to the customer.

Enough of cooking

I guess I already crossed my limits of a blog article and provoked the readers to become impatient. Though thoughts of relating the process of obtaining customer feedback, change management, bug fixing, maintenance, quality accreditation, etc haunted me, I am leaving it here in order to avoid overdoing this comparison. So, enough of food for thought. Call me up and come home for a cup of tea and dhal fry self-cooked and served hot with warmth in the heart.

 

Posted in Software Engineering | 2 Comments

Observable pattern in grooming a child and grooming our subordinates

I got this book “Learning Fundamentals” by Colin Rose and Gorden Dryden for the age group 3-6 to get some tips on how to teach my elder daughter who is nearing 4. As I read through this book, I found the section “Making the most of the activities” useful not just to coach our children but also to groom professionals to the next level in the organization to take up additional responsibilities. I will take you through the ten principles cited by the author and try to relate to the aspect of grooming a professional. The quoted text in Italics are extracted from the book.

1. Ensure Success

If your child can remember three objects in a memory game, but not four, play the game again using three things, until she is ready to try four. Success breeds a positive self-image and a willingness to keep trying and learning. Failure is demotivating.

You might have caught what I am about to write. So let me resort to a non-IT instance to add some color and to avoid being trivial up front. If I wish to perform a Classical Carnatic music concert, I can’t immediately be given a task to do the improvisations like Raga Alapana, Kalpanaswaram without understanding the intricacies of the Swaras that constitute the raga. I will only fail and lose my confidence and stick to bathroom singing.

Here comes the bland one. I cannot ask a developer who aspires to be a Design Lead but exposed only to structured programming to come and design a three-tier application on Java. He need to be taken through the concepts of Object Oriented Programming making him understand how to identify classes and their responsibilities from the requirements, what framework needs to be used, etc. It is a step-by-step process.

Somehow I can’t resist myself from preaching. Here it goes. We should assess a person’s capability and load him with the responsibilities such that he doesn’t feel overloaded or stressed. The person progresses through every step towards the new role steadily, ensuring success at every step and there by building the confidence to take-up additional responsibility. This is also applicable when we conduct training. At the end of every step, we do give practical, hands-on problems just to ensure that participants are confident and positive in achieving the overall training objective. The author also warned at “hot-housing” which is also prevalent in one form or the other in an industry set-up.

2. Give ‘just enough’ help

If you take over, saying, “Let me do it”, you convey a strong hidden message that she is not competent. On the other hand, never leave her struggling. It is also very important to constantly emphasise that mistakes are part of learning.

 I became doubly conscious after reading this, both with my daughter as well as with my colleagues. It is very difficult to hold back our temptation to solve a problem rather than guiding some one to solve the problem step-by-step. I recently played a cake baking computer game with my daughter. It was so cool that I was tempted to get my hands-on. When I got my daughter to do the same, she tested my patience at every step which looked trivial to me. When I saw her delight after the computer gift-wraps her baked cake, I felt that it is worth the effort and wait.

We talk about delegation every now and then. Let me take a situation where you need to groom your subordinate ‘X’ to be the SPoC for your customer. You would like to participate in the regular status call with your customer along with X. Customer do delve deep into every single issue. You might want to jump-in with the motive of helping X. Rather you should just sit back and try to observe the thought process of X while trying to respond. This would give X a chance to think through the problem and for you to give a constructive feedback, if any, at the end. At the same time, you should also be there to render a helping hand when needed. When you feel that X is cornered by your customer, you should pitch-in to guide your X to come out of the tangle.

For a person in his debut, it is common to make mistakes while performing the responsibilities. Instead of being harsh and start criticising, we need to work with the person finding out why that mistake has occurred and how to fix it and prevent it. Yes, I am talking of causal analysis. In cricket, how many times have you watched a senior player or captain going across to a young, aggressive fast bowler on his debut who was getting hammered, to pep him up and have a couple of words. This would help your subordinate to feel that mistakes are not the end of the world.

3. Practise ‘show and tell’

She learns most, not when you are her teacher, but when you are her fellow-learner and guide. Show and tell her at the same time, then let her experiment for herself.

If I were to explain what a critical path is to a person who is groomed to prepare and manage project schedule, I cannot just talk about slack or float and network diagram. I will have to show an example, preferably a real one from my experience, assign tentative duration and dependencies for each of the tasks and take them through those terms. Similarly, if I have to explain a deadlock situation in a multi-threaded application I can not talk about resource dependency graph. I need to have a concrete example. 

In training or workshop, we often get to see various case studies trainer uses to show how the concepts can be applied and/or what result we can expect if we apply them. This would be followed with the problems to solve where participants can experiment themselves independently.

4. Give her time to work it out herself

Encourage her to think things through herself. Let her correct her own efforts if possible.

This complements the aspect Give ‘just enough’ help we discussed in point number two. One talks about the quantum of help and the other talks about the quantum of time. As the productivity taken as input for estimates take into account the skill level of the person assigned the job, we would normally have time accounted for on-the-job learning. We should just ensure that the person is time conscious and result-oriented. People especially in knowledge industry, hate micro-management and expect management to give them time to think through the problem and solve by themselves.

We also come across people who tend to give-up the fight fast without putting their heart and soul to solve the problem. One should be smart enough to gauge this laziness and accordingly work with the person to make him use all the available time to analyze on their own and come up with the result.

5. Give encouragement rather than praise

Trying to succeed in order to please herself is self-motivation. Self-motivation lasts and is what she will need in later life.

It starts becoming complicated when I try to relate it to the grooming aspect. Some thoughts came controversial too while I tried to think. At first, it was difficult for me to differentiate between encouragement and praise. Don’t we human work in the industry to be in the limelight and to get the accolades. Then I realised the stand taken by the author while putting forth this point. You don’t want your subordinate to do his work just for the sake of pleasing you. You want him to understand the value he adds to himself as well as to others by doing it. You expect him to take pride (the pride I refer in the home page) of his work rather than to take your praise.

Author vividly explains this minute differences with an example on how we need to react to ensure we encourage rather than praise.

If your response is: “Good girl” she will come to want to do things to please you rather than herself. He advises us to encourage her with a helpful advice like “Well done, you succeeded because you looked carefully” or, “You got on well that time because you did it a bit more slowly.

6. Encourage methodical thinking

If you prompt her to think for herself, deliberately and methodically, helping with the words she needs, you create self-confidence. Here are three powerful thinking steps – Help her see what’s important, Help her take time and plan, Encourage her to take care

 This section talks about the fine-tuning of thought process. Once when I took up additional responsibility, my Manager sensed that I was stressed because of lack of time. He guided me to realize and measure the actual importance of things I do and the importance I gave to them in terms of time. It helped me understand prioritization of things. This in a way is closely related to what Pareto principle, the 80-20 rule tells us so that we focus on the important, critical things.

In the beginning of my career, I used to be overly aggressive with the estimates I committed for my work because I was very short-sighted. I was worried only about the development activities. Did not think through all the activities / process steps that I need to take account of. I was then guided to think through the complete process steps right from analysis of scope of work till how I am going to deliver and deploy my completed work. It is important to point out that making a plan – and not just rushing in – saves time later.

A person on debut tends to jump in excitement to execute things in a scurry when additional responsibilities are given. We tend to miss out on finer attention to details. So, it is essential to caution the person to take complete care with full attention so that we can expect completeness at the time of completion.  

7. Avoid rewarding with treats

The effort disappears when the bribe disappears. Your aim is for internal motivation where the feeling of success in meeting a challenge – a sense of self-mastery – is its own reward.

Once again we are visiting a topic that is close to what we have discussed in section 5: Give encouragement rather than praise. Here it refers to those deals that we tend to strike to get things done or make the person perform. I have offered to get cake for my daughter if she gets ready quickly to her school. I did promise my mates to treat them with Pizzas if we deliver the release that night. On a number of occasions, I would have felt helpless or felt that people expect treat to get things done. I do think quite often that why don’t people volunteer to deliver the commitment without me offering those treats? How can I nurture that sense of internal motivation and ownership in each one of them? How can I kindle that passion in each one of them to create and sustain that internal motivation? As a leader you will exhibit that passion from the front (if not, you will have to, again excuse me to be imperative) and expect every one in the team to share the same passion. Believe me, it is not so straight forward, simple and scientific. I am still learning to master that art.

Every organization has a rewarding mechanism. According to me, rewards are there to recognize and celebrate one’s achievement and success rather than means or the so-called ‘carrot’ to get things done.

8. Take your lead from her interests

Suggest games, provide option, but let her choose what she wants to do.

If I view this principle from one viewpoint, I thought the author is asking us to find out what really thrills the person and what kind of responsibilities he would be interested to take-up and present those options to the individual. In industry, we also need to candidly assess his own strengths and weaknesses in his list of interests. I think, this can be used as a real good aid to re-motivate people who are not feeling so satisfied with the work.

On the other front, I could also relate this principle to the flexibility and empowerment we provide to people to decide how they want to execute their responsibilities. After I make a person in charge of the team while grooming him to take-up that role, I shall not ask him to do a daily stand-up meeting at 10 AM, have a joint lunch with the whole team every week, etc. You can present various styles of management, how people practice them and under what circumstances each style become more effective. He can choose how he wants to execute the same on a daily basis.

9. Encourage curiosity

Encourage her to think around at everything – to wonder, to ask, why how, what?

Again from an aspiring Carnatic professional. If I just follow what elders do by starting the concert with a varnam without understanding why they are there in the first place, in future, I might not think twice to not sing that in the beginning.

Nowadays, every industry has defined quality processes and the documents each project team has to create and maintain. A person who is getting groomed to be a Process Owner in a project has to understand and justify the existence and relevance of each of the processes and the templates attached to them. Only then, you can also expect a good tailoring coming from him. As a leader, we will have to ensure that such in-depth analysis with reasoning have gone in the work before you accept the result. Author asks us to do this best by being curious and questioning as well. In the industry, you expect people to be curious to understand the bigger picture so that they can relate the value they and their work add to the overall success of the industry.

Let me try to get the interests of the programmers. Take an instance of a programmer learning to automate unit testing with JUnit. In Software industry, one great feature it has given for itself is copy-paste and find-replace. The programmer copy-pastes the unit test case written for another module by his colleague for his module and find-replaces all the context specific nouns with the author being the first. Sorry, more often in the unit test cases that I have reviewed, wrong author used to be the first defect that I used to catch 🙂 Unless you question him, “Why do you have ‘setup’ and ‘teardown’ methods? How and when are they invoked?”, he will not make real use of those methods and might possibly miss them in his future unit test cases. A robust Software cannot be developed unless every developer is motivated and curious to break their own piece-of-code.

10. Avoid comparison with other children

There’s always someone richer, brighter, more artistic or more attractive.

This is a universal truth. So, I will not add my own explanation to it. I will just stop with a pointer to “The Cracked Water Pot” story. It was very interesting to read it. I will just present the moral of that story here. Each of us have our own unique flaws. We’re all cracked pots. But it’s the cracks and flaws we each have that make our lives together so very interesting and rewarding. You’ve just got to take each person for what they are, and look for the good in them. There is a lot of good out there. 

Summary

In this article, we discussed how to ensure your subordinate gets successfully groomed to the next role with just enough help rendered through the practise of “Show and Tell” inculcating methodical or organized inquisitive thinking along with sufficient freedom and time to solve the problem himself with enough encouragement and without external comparison. It is easy to preach, but difficult to practise. If you just check individual principles, there is no or very less science behind them. They are more of an art. It can only be learnt with conscious practice of each of the principles and constant introspection on how our techniques work with our children and our subordinates.

Posted in Softskill | 3 Comments