Every time Apple makes a move on AppStore policy, there is a chorus of groans from the developer community.

Today, Apple changed terms to say that you can no longer use Location services in an app primarily for the purposes of targeting ads, if your app is not using those services already for ‘beneficial’ functionality.  While many people will spend time parsing the word ‘beneficial’ and attacking it (and I agree it’s a terrible term), the thrust of the policy change is that LBS services on the iPhone are for building features for users, not for creating a location-based ad marketplace.

I think that’s a great move, even as a developer.  It’s good for consumers, who don’t want to authorize every crossword-puzzle app to know their latitude and longitude.  And it’s good for developers, because it will help maintain consumer trust in iPhone/iPad applications and their use of personal data.

Does it limit your monetization options?  Yes.  Severely?  Meaningfully?  No.  Anyone who really thinks they’re going to offer a cute free app and monetize it with location-based ads should go read this PinchMedia presentation for starters.

I love a lot of what many people hate about the iPhone.  I love that it’s closed, and thus secure and manageable.  I love that it’s basically free of spammy, buggy, predatory applications.  I love that there are standards of utility, performance, and interface.

And I hope I have the guts to make unpopular, and right, decisions like this one in the future.

They’re not the same thing, and they’re not optimized the same way.  People throw around the word “scalability” all the time without having a clue what it really means.

So let’s talk about the components of performance.

Speed is about how fast your software works for any given operation, independent of data load and demand.  You address speed with profiling and tuning.  It’s both a code problem, and one that’s apparent without load, which makes it the easiest for both engineers and business people alike to zero in on, but in the grand scheme of things, it’s merely a constant multiplier on the performance of a system.

Capacity is how much load your system can handle.  How much data does a user generate over how much time, how fast is the CPU on your production server, and thus how many concurrent requests can it handle (given the Speed of your software).  Capacity is an offshoot of Speed, efficiency, and hardware.

Scalability is about how your system handles growth in capacity.  Does your system scale vertically (upgrade your hardware), horizontally (add more hardware) or diagonally (rewrite the system).  Vertical scaling has two problems: it’s exponentially expensive (ever bought an 8GB RAM chip?), and it’s limited by the physical capacity of individual computers.  This is why when people talk scaling these days, what they mean is being able to expand your infrastructure by adding commodity hardware (relatively) infinitely.  This also explains why the “cloud” is all the rage.

In big companies, programmers focus on Speed, systems administrators focus on Capacity, and architects on Scalability.  In startups, they all keep me up at night.

2 years ago I had never used anything in our current technology stack.  I f*cking love it.

Our first prototype at GameChanger was a more obvious LAMP incarnation, Linux+Apache+MySQL+Python, where the only variation from the original was Python over PHP (thank goodness).  Where we’re going now is starting to be a clearer departure from that stack.

The first mutation was swapping MySQL for MongoDB, which was a decision I like most days (I’m still a little terrified of data corruption or catastrophic failure, but the featureset is just so good it’s worth it).

The second is dropping Apache for Nginx, which I’m in the process of doing now.  This is really part of a more subtle architectural change, moving toward a long-polling/comet system for live in-browser updates.  Moving to a push model system wide is going to give us a much better scaling curve for live GameStreams, and is a no-brainer.

And under the covers, there are even subtler shifts going on.  We’re currently using Django for both our API layer and our Web layer, but that’s going to change too.  I’m playing with Tornado now (the FriendFeed offshoot technology) to give us a long-polling capable alternative at the API layer, and moving toward a replicated, and then sharded, MongoDB data store on the back end.

I started learning Python less than 2 years ago, when MongoDB was an infant project, and fired up Russian-made Nginx that same summer.  I can’t help but feel it’s a great time to be a technologist.

I was once turned down for a CTO job because the CEO said I was too much like him, and he needed to hire someone different to balance out his team.

As I’m looking over a crop of talented developers, I’ve been contemplating whether one needs to hire Ninjas or Soldiers, but realized that I could phrase it in these terms as well.  Do I want to hire someone with similar core-competencies as our existing team, or do I want someone who is more of a complement?

This person needs to be a cultural fit with our team, which is one vote in favor of a similar background.  And I’ve also heard that organizations tend to be weakest where their leaders are strongest- something about overly-high standards.  But on the other hand, if you’re already very strong in one area, should you specifically hire for a diverse skillset, or at least devalue your core skillset when looking at potential hires?  And it seems to me you’d also want a diversity of opinion and background on a healthy team as well.

I don’t know the final answer, but I can say it’s a first-class problem to have.

I’m reading Murakami again.  I first fell in love with his writing in a Japanese lit class at Middlebury, reading him in the English translation.  Soon after, during my year at Keio University in Tokyo, I picked up a copy of his Wild Sheep Chase (羊を巡る冒険) in the original Japanese, and started reading it.

It took me 6 months to get through the 2-volume (thin soft-cover volumes) novel the first time.  I underlined every word I didn’t know as I read, and dutifully spent hours going back over and looking each one up.  I filled 3 small college-ruled notebooks with vocabulary from that book alone (often realizing that I was looking up the same works two or three times).

When I finished, I was basically fully literate in Japanese.  Not native, mind you, but there wasn’t much that I couldn’t read.  And I should mention that I was taking the highest level Japanese courses that Keio offered during that span as well, so it wasn’t independent.  But I credit that experience heavily in pushing myself from capable to fluent as a reader (and to some degree, writer) of the language.  I went on to gobble up many of his other books, and I was able to just enjoy the writing.

So I’m reading Murakami again, his new book 1Q84 (in Japanese, it reads as a phonetically like “1-9-8-4”).  This time I’m reading the Korean translation, and I’m writing down every vocabulary word that I don’t know as I go.  I’m filling up pages of a beautiful new notebook, and hoping that on the other side, I’ll have a new level of comfort with the language.

I think this is analogous to how I have taught myself to program.  I started out maintaining a big Java app at Higher One, testing it, then tweaking it, then rewriting it.  I would look up every class and function I didn’t know, initially in the JDK documentation, and then by reading Sun’s source code.  When I later read the Design Patterns book, for instance, it was all concepts I’d already seen and used.

I think this partially explains why I use Emacs, rather than some fancier IDE, for development.  The process, at first painful, of looking up how to do each task, and what functions each class has available, is a part of how I internalize code, patterns, and documentation.  In the end, I think it makes me a better, faster, and more well-rounded programmer.

It’s been a topic of hot debate around the GameChanger offices over the past few weeks as we focus on our next hire.

Do we need, or in fact even want, a “Ninja” for our next programmer? And what does that mean?

A lot of job postings look for Ninjas, Gurus, or Hackers, and I think what we’re trying to select for is the ego and ambition of an alpha programmer. Indeed, many of the best coders I know have pretty well-developed… senses of self, if you will (yeah, me included, on occasion).

In larger companies, these are the rare breed, and they play a crucial role of injecting critical thought and frenzied productivity into an otherwise slow-moving environment. I’d even go so far as to say that every company needs at least a few of these types to be, and stay, competitive. I also have been in companies where a Ninja was the only type of programmer who could survive. To generalize, I think it’s a venture where the core problems are mathematical, algorithmic, and academic. For example, I happened to look at the team page on hunch.com, and the engineering credentials there are pretty staggering- right in line with the kind of problem they tackle.

The last two places I worked before starting GameChanger were somewhat similar: ShopWiki’s problems were semi-structured data extraction and search algorithms, with a good dose of scaling and performance. Conductor is trying to use machine learning to comprehend SEO. Those problems scream “ninja.” And this is where my hiring practices came of age. We hired two good engineers on the most arrogant, super-geek job req I’ve ever written.

In places that are more niche-/product-centric, however, the ninja-ratio is necessarily different. First, there aren’t a ton of academic problems to occupy the giant brains (egos, ahem) of a whole crew of these people. Second, the core competency that has high value in most startups is focus on customers and passion for the domain, more than pure ingenuity.

Risk/reward is also a factor. As a small company with Just Enough Funding, we cannot afford a bust. Our next hire will fill a crucial role in the upcoming months, as we hit the high-season for the first time with a live product. Another trait I’ve observed in Ninjas is that there is a higher rate of conflict and disconnect, likely because of ego clashes and inevitable differing foci among smart, opinionated people. On the reward side of the equation, I think it’s safe to say our team already has three Ninjas, and we’ve reached a great working chemistry. What we are currently missing is throughput.

This all adds up to my stance on hiring right now: hold the Ninjas, find me a Soldier (Ted’s words: “Soldier > Ninja”). Not a second-rate engineer, but a passionate pragmatist who is me-second, and for us, baseball first. It’s a change to the way I’ve hired before, and I think it’s a more mature, and properly adapted change. I’m excited about the kind of Soldiers I’m meeting in this round, and looking forward to the addition to our crew.

But a little down the road, once we have a strong base, I’ll be hunting Ninjas again for sure.

It all started in 2005.  I had just joined Eliot Horowitz (now of 10gen/MongoDB) and Dwight Merriman at ShopWiki’s new closet-sized office in SOS in NYC (funnily, I write this post from GameChanger’s digs a couple of hallways down: same floor, better light).  Eliot had been hacking for 6 months in Starbucks and his apartment (with an XServe on the coffee table, back in that brief window when they were both awesome and economical), and it was time to build more of an engineering team.

Eliot is as much a math guy as a programmer, and he wanted to lay out the formula for hiring the ideal programmer.  Literally, like, in equation-form.  So he first wrote the following on our whiteboard:

L
—
T

Eliot was most interested in pure productivity, but I immediately wondered about how quality fit in here.  He agreed, and quickly changed the formula to:

L
—-
T*B³

Bugs are really really really bad, he opined.  Like, exponential bad.  Why?  One bug, and it’s pretty obvious what’s wrong.  Two bugs, and they might interact, so it becomes quite a bit more difficult (and time-consuming) to track them down & squash them.  Three bugs or more (in one code module), and it quickly becomes totally non-obvious what’s wrong with your code.

Great, I say, but this is a recipe for short-term productivity only, isn’t it?  Where’s maintainability?  After some discussion, the equation becomes:

L*M²
——
T*B³

Now we had something we liked, and we wrote it on the top corner of our whiteboard in permanent marker.  (Ironically, you might notice that L/T, speed, is now just a multiplier on quality).  It lived on ShopWiki’s whiteboards for my entire tenure there, and I later took that same equation and another permanent marker and wrote in on a whiteboard at Conductor during my stint as engineering director.

But in our new GameChanger offices, after scrawling it again (this time in dry-erase), I’ve erased it.  I’ve come to think that while productivity is at the core of what programmers do, that Pragmatism, Communication, and Culture are equally important.  I still tell every engineer I hire about this equation, and it’s been a great tool for making discussions of individual productivity sufficiently geeky and abstract to ease them a lot.

And while I’m on the topic, we’re hiring (Python/Obj-C, pragmatic, nice people). :)

Thoughts?  Hit me in the comments.  Resumes?  jobs@gamechanger.io

I find it’s a common problem in building a startup that the pipeline of work approaches infinite length while the hours in the day and LOC output level I can sustain remain markedly finite.

There are a few tricks I’ve found for keeping a semblance of sanity in the face of it all.

First is GTD. I’ve never really been a structured-process guy, but incorporating some basic principles of Getting Things Done into my daily flow has made a huge difference. I keep meticulous lists, live to prioritize them, and try to use inbox-style mangement for keeping incoming work from overwhelming ongoing productivity.

At GameChanger we’ve also adopted a policy of respecting Maker Time (see Paul Graham post on same). We keep our meetings to the morning and early afternoon, and the office is quietly humming after 2pm.

I haven’t been involved with the movement that is Lean Startup, but the ideal of the minimal viable product (and feature) is one that has long seemed critical to me. Rather than get bogged down in the perfection of any one feature, constantly iterating gives us the ability to approximate a faster pace, by broadening the scope of what can be worked on in the short term.

So I’m still feeling dreadfully lacking in infinite capacity, but not too daunted (most of the time) by a seemingly infinite pile of work.

What helps you approach infinity?

cdixon:

Bill Gates walks into a bar, suddenly the average wealth of people in the bar goes from thousands to millions.   And also now everyone in the bar except Bill has below average wealth.  Similarly, contributors to user generated websites like Wikipedia are almost all below average.  There are a few people who contribute a ton, and a whole lot of people who contribute very little.  This is why it is often very hard for people to conceptualize how these user generated sites work- there is no “average user” to imagine yourself as.

(Basically paraphrasing from Clay Shirky’s book Here Comes Everybody)

Verdict: guilty, on two charges of Hubris in the 1st degree.

Charge the first: programming hubris.  I should know better by now, but 2 months ago I embarked on a complex refactoring of some serious offline/online synchronization code.  It nearly killed me.  I basically disappeared into a cave, talked to no one, and didn’t come up for air until, 6 weeks and 30k lines of code in, I was exhausted and still struggling to stabilize the new framework.

Finally, my wife (read: sanest person I know) tore my head off, and asked me why I hadn’t asked anyone for help.  Wow.  I went to lunch with a few smart friends, and started talking about what I was up against.  I sat down and wrote out a list of the problems that were plaguing the architecture.  Almost magically, I had a working build 2 days later.

Charge the second: management hubris.  This is one of my great flaws.  I’ve been told I’m a really good people-manager, and I pride myself on being able to inspire, focus, and teach, and I let it get to my head.  Every good manager, I think, occasionally falls into the trap of thinking that they can take a talented but somehow flawed (unmotivated, self-centered, psychotic, distracted, etc.) report and turn them into a superstar.  You convince yourself that all this person has lacked was a caring and attentive overseer who knows how to bring out the best in them.

I’ve made this mistake twice in recent memory, and it never works out for the best.  I can help someone who wants to learn and improve, but nobody can help a person in the wrong career, at the wrong company, or simply in the wrong stage of his/her life.  So I’m looking to hire, and this time, I’m not in the mood for a reclamation project.

As entrepreneurs, I think we’re used to the idea that we can move the whole world with pure force of will.  And usually that’s a good thing, except when it gets in the way of good judgement.  So I’ve resolved to talk more, ask for help earlier, and never relax my high standards for the people who work for and with me.

Oh, and to listen to my wife more.