What I learned about interviews after sitting on the other side of the desk

This week, I interviewed another programmer for the first time. Even going through this process a mere one time has given me a lot of insight into the hiring process and why it is really, really difficult to judge a candidate based on only a short period of interaction. Because my experience is new and fresh enough to remember what it’s like to not know this now-obvious information, I’m going to share my thoughts for those who haven’t experienced it.

When you bring someone in for an interview, you think they might be able to fill a team role you need, but you’re not completely sure. On a scale from “hire” to “don’t hire,” the candidate sits right at “maybe”. After the interview stage, you no longer get to pick from shades of grey: you either hire them, or you don’t. Your time to give an interview is extremely limited, so any questions you ask have the goal of converting that “maybe” to a yes or no. With that in mind, there are only a few reasons you decide to ask a question in an interview:

  1. To learn something new about the candidate: “How does he feel about working with [this technology]?”
  2. To confirm a suspicion you have: “I think she understands programming, but I get the impression she doesn’t know 3D math at all. Is that accurate?”
  3. To clarify something you’re unsure about: “He says he worked on [this project team], but what part of it did he actually create? Was it a meaningful contribution?”

What if you’re still sitting at “maybe” at the end of an interview? The safest decision is to interpret “maybe” as “no hire”. Bummer.

What can we take away from this knowledge as a candidate rather than as an interviewer?

The person interviewing you wants to hire you, they just need to confirm you can do the job. You want to prove you’re at least as skilled as they hope you are, and you need to assuage any fears or uncertainties they might have about you. If you fail to do either of these things, even if you seem like a great person who is super charismatic and fun to be around, your “maybe” status is still a “maybe,” and as I just mentioned above, maybe means no hire. If you get rejected after an interview, it doesn’t necessarily mean you weren’t good enough; it might simply mean you did not give enough information to swing your interviewer from “maybe” to “yes, hire her!”.

Unfortunately, it is extremely unlikely an interviewer will be direct and declare “I can tell you’re smart, but I’m not sure about how quickly and efficiently you’ll be able to get work done. Prove to me you can work quickly with examples from your past.” (In defense of interviewers, this is hardly a friendly or charismatic approach, and people are pretty bad at answering unexpected open-ended questions) Instead, they’ll probably say things like: “Tell me about your work on Interesting Side Project. How big was the team? How long did the project take to create? How did you handle keeping the scope of the project in check? Did you have to cut features?” As an upside, these questions are a lot easier to answer because they’re specific and direct; however, there is also a slight disconnect between what is being asked and what the interviewer actually is trying to figure out.

When reflecting on an interview, don’t just think about what you were asked, try to figure out why you were asked those questions. If you’re lucky, your name is Professor Charles Francis Xavier and reading minds is something you do on a daily basis. For us mere mortals, the best we can do is try to give it our best guess. If interviewers tend to focus on a specific section of your work experience, it could be because it’s the most impressive and interesting! …or maybe it’s the most unclear part of your resume and they’re not confident that previous job was adequate experience to prepare you for this new one. If you’re a fresh college graduate and you keep getting asked a lot of questions about your degree’s curriculum, they might be trying to figure out “I’ve never heard of this college. Is it a good program, or did they go to one of those borderline scam for-profit schools and not actually learn anything of value?”

At the end of the day, though, interviews are difficult for everyone involved. Both participants are trying to make a decision that will impact the next several months or years of their life based solely on a short 45-minute conversation. It’s far from a perfect system, and everyone’s going to make mistakes, but it’s the system we have, so just work with it as well as you can.

Advertisements

Aside: Why are programmers so expensive?

A lot of people argue it’s a supply and demand issue, and to an extent, it is. However, a business can’t afford to pay an employee more money than that employee produces in value; otherwise, they’re better off not hiring that person in the first place. So how is it a programmer fresh out of college can create over $100,000 in value every year for a company like Microsoft or Google?

Software is really, really good at two things: improving efficiency and eliminating jobs.

Labor costs are one of the largest expenses businesses have. Decent-paying jobs in an “average” town in the USA will net a salary in the $40,000-$70,000/yr range depending on the region. Then take that salary and multiple it by about 140%, because a business also has to pay for things like equipment, health care, office space, etc. Your average employee probably costs a company at least $60,000/yr just to keep them employed. A 100-person company might be spending $6,000,000/yr in labor costs alone, and that doesn’t even count businesses in super expensive cities. Now imagine the cost of labor to a company like Intel who has over 100,000 employees – it’s astronomical.

Let’s say you build a piece of software you wrote can make those employees 5% more efficient, either by improving their output by 5% or by making 1 in 20 employees redundant. To a relatively small 100-person company, that software is worth $300,000 every single year. To a larger company, it’d be worth millions. And unlike machines, software costs almost nothing to make copies of, so you could even sell this software to dozens of companies for pretty much no additional cost. Because software can be so incredibly valuable, a big tech company can easily afford to throw a six figure salary at fresh college grads with confidence that they’ll make that money back and then some.

An obvious exception here: programming a video game isn’t improving someone’s efficiency for a business. However, because a programmer could be working at Microsoft instead of on the next Call of Duty, entertainment businesses are essentially competing for the same talent pool as enterprise software companies.

Optimization, Part 2: When do you pay the cost?

(For the previous article, see Optimization, Part 1: Figuring out your problem)

Although the word optimization probably makes you think “make the program run faster,” it isn’t always about optimizing algorithms. Usually, you can make noticeable improvements by changing when your code is executed rather than trying to adjust how long it takes. In this post, I’ll discuss the options of loading screens, amortization, and performing calculations at build time.

Loading Screens

Although a loading screen is traditionally used for anything your program can’t run without–for example, a game can’t run until the upcoming level is loaded–you can also pre-load items in them. This is primarily useful for video games, as having business software take a tenth of a second to do some operation is fine. In a game, stopping for a tenth of a second is a noticeable hitch in your frame rate.

Let’s take a simple example: you’re building a platformer game similar to Mario or other games. The first time you spawn a given enemy type in a level, the game stutters. On further investigation, you discover that the game is taking some time to load all of the enemy’s animations into memory for the first time, and the combination of parsing a bunch of animation data along with transferring a bunch of data into RAM causes a brief stutter. One solution is to pre-load that animation data into your animation system before the level begins.

“But aren’t loading screens bad?” Well, yes, and moving more things to a loading screen is going to make it take even longer. However, if you get creative, you can hide a “loading screen” in your game. Have you ever played a game where you have to sit in an elevator? That elevator was used as a loading screen. You know when games have you walk through a door then lock it behind you? It’s unloading the section of the level you just passed through, and that upcoming winding, empty corridor is being used to keep the rest of the level out of line of sight until the game has had time to load it in.

Portal hides most of its loading time in elevator sequences

Portal hides most of its loading time in elevator sequences

+ Easily “fixes” most slow operations
+ With some creativity, can be hidden so it’s less noticeable
– Nobody likes sitting at loading screens doing nothing. Making them take even longer just makes it worse.
– Paying for things you aren’t using is a Bad Thing. You’ll probably need that thing loaded eventually, but you’re wasting memory on it when you don’t.

Amortization

Amortize is originally a financial term, but it just means to write off a cost of (an asset) gradually. In other words, rather than doing an expensive computation all at once right now, we’ll spread it out over time. What are some ways we can do that?

A common example can be found with streaming video. Rather than blocking everything waiting for the entire video to download ahead of time, the “cost” of downloading the video is paid over time, allowing the user to start watching right away.

If your program is multithreaded, having a worker thread do a calculation in the background accomplishes this goal. However, a single-threaded application can apply the same concept of multitasking. Let’s use a video game example: you’re building an endless runner game like Canabalt or Robot Unicorn Attack or Temple Run. Because the game goes on forever until the player dies, you need to dynamically generate more platforms, obstacles, and enemies in front of the player. You set the game up to generate more content once the player gets close to the end of what’s currently loaded, but every time you hit that level generation code, your game freezes up briefly for 100ms while the new content is created. Rather than unload the old content and load the next “chunk” all at once, you can slowly load in a couple new objects every frame. By spreading that loading time over two seconds, assuming you’re running at 30 FPS, you only need to spend an average of 100 / 60 = 1.7ms per frame rather than 100ms all at once.

Before you apply this type of optimization, dig deep enough to make sure it will actually fix your problem. If you want to smooth out loading content taking 100ms, but 95ms of it is caused by one single object, you’re still not going to get that lag spike any lower than 95ms.

+ Good for smoothing out spikes in CPU usage
– Potentially adds a lot more complexity to your code
– May require saving a lot more of your algorithm’s current state mid-calculation, using up more memory and opening up to lots of room for bugs

Performing Calculations Ahead of Time

Not having to pay a cost at all is even better than paying it at an opportune moment. If your calculation only relies on static data, you can calculate it ahead of time before your program ever runs. For a simple example, math libraries may have functions like sine and cosine pre-calculated and simply look up the correct value rather than having to calculate it at run time.

Optimizations of this nature can often be integrated directly into your tools and workflow. For example, let’s say you’re building a 3D simulation and you need to calculate some physics data from the 3D models of your objects: maybe a bounding box, or the object’s center of gravity, or generating a convex hull. Instead of waiting until your program is running to calculate this data for every object, you could instead write a custom exporter for your 3D modeling program of choice that calculates this data ahead of time. While you’re at it, you could even tweak a standard 3D model file format to instead be tailored exactly to how your engine works. This sort of optimization is why a lot of AAA games use custom file formats, and modding them often requires downloading a plugin for modeling tools.

Programmers can also have some calculations performed ahead of time. The major downside is that, unlike the above example of custom asset formats, anything you have that runs on build is going to make every single build take longer for every member of your team, but sometimes it may make sense. One straightforward choice is using pre-build and post-build scripts. If you’re using C++, template metaprogramming is a technique that can be used to perform calculations when your program builds, however, template metaprogramming code is often difficult to read and by extension can be rather difficult to maintain. In either case, compiling code already takes long enough; you don’t want to make it worse unless it’s worth the cost.

Well, okay, you _probably_ don't want to make compilation times worse

Well, okay, you _probably_ don’t want to make compilation times worse

+ Can make expensive calculations effectively free. Yay!
– May slow down developer iteration time, and developer time is expensive
– Most likely adds complexity to your workflow and/or build process, making it harder for new team members to get up to speed, and giving more potential places for things to go wrong

Optimization, Part 1: Figuring out your problem

“Dang, my game is running really sluggish. I know, I’ll just optimize it! (*opens code editor*) …ok, how do I actually do that?”

First things first: You can’t fix something until you know what to fix. You’ll probably know a few symptoms: Maybe the game freezes up every once in a while, or the game crashes on old phones because it runs out of memory, or maybe the framerate just seems to get choppy every now and then. You might even have a guess as to what’s causing the problem, but before you dive into your code to start fixing everything, make sure you’re fixing the right thing.

If you can, an easy first step is to run a profiler. Essentially, a profiler tracks how long different parts of your program take to execute, letting you drill down deep into your code to figure out how long each function call takes. While off-the-shelf tools are generally the easiest option, sometimes they’re not available or are impractical. You can still do the same thing manually, inserting code to check the current time before and after a function is done running and dumping that info out to a log file.

Unity3D has a built-in profiler in the Pro version

Unity3D has a built-in profiler in the Pro version

So, what data are we looking for? Anything that’s taking too long, of course! But what counts as “taking too long”?

To answer that, let’s talk about frames per second (FPS), also known as frame rate. This is a measurement of how many images are presented to the player in one second. From a programmer’s perspective, this measures how many times you can loop through your game code in one second. Generally speaking, your goal is to never dip below 30 FPS. If your game requires a lot of really responsive controls, like a first-person shooter game, you might even want to push for 60 FPS. Having a target FPS is important, but FPS itself isn’t a good measure of our game’s performance when optimizing because it’s non-linear. Going into the details behind that is a bit of a tangent, but you can read more about it here: http://www.mvps.org/directx/articles/fps_versus_frame_time.htm

Instead of frames rendered per second, what we really want to measure is: how long does each frame of our game take? Instead of (frames / second), what we want is (seconds / frame). A second is a really long time in terms of computation, though, so we typically use milliseconds. To convert from (frames/sec) to (sec/frame), we just take the inverse.

And that is where these numbers in Unity's profiler come from

And that is where these numbers in Unity’s profiler come from

If our goal is 30 FPS, we have (1 / 30) = 0.03333… sec = 33.33… milliseconds per frame
If our goal is 60 FPS, we have (1 / 60) = 0.01666… sec = 16.66… milliseconds per frame

Finally, we’ve reached a reference point! Assuming our target is 30 FPS, if one iteration through our game’s code takes longer than 33.3ms, it is officially “taking too long”. Huzzah!

Armed with this knowledge, what do we look for in the profiler data?

If the game briefly “freezes” every now and then, look for spikes in performance time:

This one frame has a giant jump in computation time

Not all performance hitches will be this visually obvious, but it’s the same idea: look for a sudden jump up in frame time. The frame pointed to by the arrow took a lot longer to process than the others. If we look at the color-coding, we can deduce that something in our physics setup caused the problem. Click on that frame in the window, then look at the list of function calls to see what happened that frame. Try and find a way to consistently reproduce the problem and get as much data as you can. Maybe we just created a bunch of objects? Maybe we moved around a bunch of “static” colliders that don’t have rigidbodies and forced the engine to re-calculate all the physics data?

If the performance issue comes from your scripts, look for what function calls took the most time. Dig as far down as possible to make sure you’re fixing the right problem. I recently spent a bunch of time “fixing” a performance spike that happened whenever we would load in new level data in an endless runner. “Of course, adding all of those objects at once is causing it, duh!” I said to myself. After a few hours changing the code to slowly stream in the new objects a handful at a time… the problem was still there. Oops. Turns out the performance hitches were actually caused by loading in animation data for objects we hadn’t used yet. Fixing that took all of twenty minutes to make the game pre-load animations beforehand; I didn’t need to re-work our level loading system at all. I could’ve avoided wasting that time if I had just spent a few more minutes digging deeper into the profiler data.

If your game is consistently slow and sluggish rather than just having “spikes” like the above, you’ll need to work a bit harder. If you’re really, really lucky, it’s one subsystem or function taking way too long and you can just fix it and move on with your day. More often than not, though, your update loop is just taking too long overall across… well, everything. Rather than your performance being killed by one big orcish axe to the chest, it is instead being subjected to death by ten thousand paper cuts.

The main question you now have to answer is, how much time does each subsystem get? Remember, you only get 33.3ms to spread across every single system in your game at 30 FPS. In a big AAA project, there will probably already be a performance budget given to each subsystem: “OK, AI gets [X] ms, physics gets [Y] ms…”. For a smaller team, you probably don’t have this level of formality yet. Now you have to change that. Even after you get your performance back to a manageable level, every single thing you add will potentially push it right back over, so it’s important to keep things in check. Generally, you’ll want to start with whatever subsystem takes the largest amount of time, and work your way down from there.

Once we figure out the source of our performance issues, how do we fix it? Well, that’s a big, complex problem that I’ll write about more later. In the meantime, hopefully this post has helped you figure out where to get started!

(For the next article, see Optimization, Part 2: When do you pay the cost?)

When is it worth working for low pay?

Lots of people say you should never work for lower pay than what you could be making elsewhere, but I disagree. Sometimes, it’s the right call. Beyond direct compensation (salary, vacation time, healthcare, etc), what else do you get out of your job? What are you learning? What do you want to be doing a few years from now? Does your job actually advance you toward that goal, or are you just in it for the money?

First, there is the obvious case: do you enjoy the work you do? This is something that’s important, but everyone has their price. Misery is proportional to how long you’re in it and what the tradeoffs are. Let’s take an extreme example: Many people would hate the idea of scrubbing toilets every day for a living, but I imagine most folks would be willing to do it for $300,000/yr. On the other hand, if getting that kind of pay required working 16 hours per day every day for a decade… it might not seem worth it. You don’t get those years of your life back. Figure out how much passion means to you: would you be happier earning $40k/yr at a job you love, or $60k/yr at one that’s kind of boring? There’s no shame in picking either option; everyone has different priorities.

How much are you learning from your job? Every job will teach you different skills, and some are more useful than others. Make sure to think long term. Does what you’re learning bring you towards where you want to be in five or ten years? If you’ve always dreamed of running your own business, it’s probably worth taking a pay cut to work at a startup company so that you can experience building up a business firsthand. If you’ve always wanted to build websites, but your current job is completely unrelated, would you take a pay cut to pursue that new career? Getting paid to do work is nice, but getting paid to learn is even better.

Don’t let yourself be exploited. If you’re an awesome artist, don’t work for free just because “you can use it as a portfolio piece!”. Why work on someone else’s cool idea when you could be working on your own instead? If someone can afford to pay you well, don’t settle for crap pay just so you can say you have work experience. Heath Ledger gives good advice here:

On the other hand, if you have no idea what you’re doing, and someone is still willing to pay you to learn it and figure it out… well, then you’re getting paid despite a complete lack of skills. Then, maybe, it’s worth taking really low pay while learning that skill instead of earning nothing while teaching yourself.

Most importantly, make sure you can support yourself. It’s no use pursuing your passion if you can’t take care of yourself. If you do want to pursue something that’s not financially sustainable, try building up some savings to live off of first. That could even be your main incentive for picking a well-paying but boring job: you plan to save up half your income so that for every year you work, you can take a year off to do something you love that doesn’t make money.

Sometimes it’s okay to work for lower pay. Just make sure you’re doing it for the right reasons.

Project Management: Scheduling

Task-tracking tools like Trello are very useful for organizing and prioritizing tasks and for coordinating teams. However, one area where Trello really lacks is in giving you a feel for whether or not your project is on-task. This is a big factor in why Poker Solitaire dragged on a lot longer than we would’ve liked. The app itself should have taken maybe 2 to 3 months; instead, it dragged on for over half a year.

So, yeah, the game took over twice as long as it should have. What happened?

Prioritizing tasks based on general “usefulness” is easy. “A definitely is higher priority than B”. What’s more difficult, however, is cutting features because they’ll take too long. It’s easy to say “well, this feature seems pretty important, and it’ll only take one or two days to implement, which isn’t that much time”. The problem is that, while this might seem like a reasonable trade-off in isolation, if you do this a dozen times, you just pushed your project back an entire month. Oops. Additionally, when a feature slips and takes longer than planned, you don’t really have a high-level feel of how that will impact your project.

To fix this, for Ludisto‘s next game, we’ve been using a spreadsheet that looks like this:

what-the-shell-schedule-spreadsheet

This accomplishes a few things:

  1. If a feature slips, you automatically see what other features that’s pushing back. By extension, this also adds some accountability: you actually feel the impact of not getting your work done on time. Most importantly, that feeling comes from “I want to get this awesome project done on time!” and not “ugh, if I don’t get this done, my boss is going to yell at me”.
  2. Once everything is broken down by week, you have a much better idea of when you’ll actually finish your project given its current scope. Estimates get more and more rough the further out they are, but you’ll at least be in the right ballpark.
  3. If you have a better idea of when you’ll finish, it’s much easier to keep your project scope in check. It’s a lot easier to weigh your options, too: “Alright, we’re behind schedule, what do we need to get rid of to still launch by the original deadline? Well, if we cut these three features, we can move these other ones up, and hopefully still launch before the conference… or, if we leave them in, that’ll push us back to [date]”.
  4. Because we mark who is working on what feature, we can monitor work loads at a glance. One week might seem super busy for everyone because there are lots of coding tasks, but without the high-level overview, it can be easy to miss that an artist might not have anything to do for that same week.
  5. It forces you to prioritize not only in terms of how good a feature would be to have, but also in terms of how long that feature will take.

In practice, it’s not a perfect system, but it does help a ton. The spreadsheet doesn’t completely replace Trello; you still need somewhere to track smaller details, record bugs, discuss things back-and-forth with comments, etc. Because you’re then working with two different systems, you inherently have an issue of data duplication. Keeping things in sync adds some overhead. Still, the extra time invested in a schedule like this definitely pays for itself in keeping your project on-task.

We learned all of this the hard way, but to anyone experienced with project management, everything I’ve said probably seems pretty obvious. You don’t fully appreciate what a dedicated project manager does for a team until you don’t have one anymore!

Aside: Businesses are not a hive-mind

Something common you’ll hear in response to news: “Can you believe what [business] did?”.

For example, Intel recently pulled an ad campaign from Gamasutra due to some internet drama. On my Facebook feed, there was a lot of discussion over “what an immoral company!” and “how could the marketing department at such a big company make a PR mistake like that?”. The specific example doesn’t matter; this sort of discussion happens a lot.

The thing is, businesses aren’t a big hive-mind. Most decisions are made by an individual or a small team of people. To continue with the above example, Intel has over 100,000 employees. Out of 100k people, odds are that 99,990 of them (literally 99.99%) had nothing to do with any given company decision.

Theoretically, our reaction should be “Can you believe what some guy who works at [business] did?”, but it isn’t. For better or worse, we tend to think of businesses as a singular entity. I personally find this both fascinating and terrifying.