Technologies come and technologies go, but careers have to stay, grow and flourish. It's always challenging to stay abreast of what’s going on in the fast changing software industry. The software world reminds me of the Hip-Hop industry in music. The shelf-life of rappers is incredibly short. While there are many one-song wonders in the history of music, the chances of a rapper ending up as nothing more than a one song wonder are pretty high. This is because it’s a sub-culture that’s about a rapper's image and lifestyle as much as it is about his or her music. You may be considered real cool today, but you can become "yesterday's news" by next week. The rate of change in the software industry is almost comparable. Platforms, languages and programming paradigms are evolving at a rate that's hard to keep up with unless you are a technology obsessed "geek". 

People always ask me "what's hot now?” If you are a techie, you have probably heard that question come up several times during the course of your career during hallway conversations, parties and other social gatherings. And the answers are usually varied and different depending on the day of the week. Also, "software" is obviously too generic a term and many platforms and technologies are popular depending on your area of specialization. Recently, I see a lot of momentum building around a technology in the area of big data processing. If you are working in the areas of databases, data warehousing, cloud computing or any kind of back-end or systems engineering, you should look at and explore Hadoop. Hadoop is an open source platform for large data processing applications that's widely adopted especially in the internet data analytics field.

Hadoop started as an open source answer to Google's GFS (Google File System) and Google's implementation of the map-reduce programming paradigm. GFS is widely recognized as one of the core strengths of Google's internal platform. Once Hadoop was out there and available, the internet industry readily embraced it as companies were looking for any and all kinds of edge to compete with or mimic Google. A basic psychology in any industry is that companies like to mimic their more successful counterparts hoping to emulate them. Some companies are also afraid that the technology in question is actually giving the winning company the edge and they will be left behind if they don't follow suit. I used to work for a small company that had a few key telecom customers. Pretty soon, every telecom company adopted our technology out of fear that their competitors are benefiting from our technology and they didn't want to be the odd one out.

Hadoop benefits a lot from that psychology and the overall hype, but it has also evolved as a legitimate, cost-free alternative to large-scale data processing solutions which is big business. Hadoop also ties in nicely with cloud computing - another buzzword these days, as it enables you to process large sets of data on the cloud. Yahoo and Facebook use Hadoop extensively and startups in Silicon Valley are jumping on Hadoop in droves since those big boys have already proven that Hadoop can power successful, large-scale businesses. There are also many companies trying to build a business in the Hadoop eco-system. They are trying to build tools and other services around the basic Hadoop platform including a couple of companies that are trying to become what Red Hat is for Linux - providing support, training and integration services for a subscription to help customers adopt and use the open-source platform. Larger corporations like IBM, EMC, EBay and Apple are also starting to use or get involved with Hadoop in some way, shape or form. 

All of this means great opportunities for people with Hadoop skills. The aforementioned companies are competing with scores of other startups to hire Hadoop talent. Techies will remember the Y2K and SAP boom from the past. While Hadoop may not be at the same scale - at least not yet, it is in the growth phase of its evolution and its ceiling is pretty high. Unlike the Y2K jobs, this is not a transient opportunity based on a platform that's on its way to obsolescence. Hadoop is here to stay and the technology and the problem space will teach you skills that will always be marketable. Big data processing as a technology space will never go out of style especially given that the scale of data flowing through the back offices is growing exponentially. So feel free to download Hadoop and play around with it. There are plenty of free materials available online and a great starting point for learning more about Hadoop is its homepage on Apache - http://hadoop.apache.org. Test drive the elephant and see if it can take you and your career places.