I'm sorry for asking you out

Over the last year or so, I've asked out several girls with varying responses. Before I get into that though, I want to delineate a process, of what goes through our heads when we ask you out:

Oooh, she's cute!
*start conversation* 
Wow, she said something witty!
"How about dinner?"

I'd like to say that there's more. I'd like to say that I felt some sort of primal connection and that you're special. I'd love to say that your demeanor reminds me of a gentle ocean breeze, or that your smile melts my soul - but it doesn't, and you're not special. You are adorable, you are (decently) intelligent, and I want to get to know you better.

I used to think women realized this, so last year, I asked out a girl who I had just met that same week. She was pretty, energetic, and I found her spurts of evanescent ebullience rather refreshing. I asked her out, without any warning of the unimaginable horror that was to come. It was, in fact, this single action that led to the three most ridiculous weeks of my life. 

"Sure," she acquiesced (and guys, for all practical purposes, sure should just be taken as a no in this context). I was ecstatic; I had just conquered one of the most prevalent societal fears--and what's more, I succeeded. While the details aren't really important, we didn't really work out. In fact, the date itself never really worked out.

Now, whenever that incident comes up, our female mutual friends tend to comment, saying something like "you asked her out--but you barely even knew her!" God damn it. Why the hell do you think I asked her out!? Do you expect me just to do nothing, and somehow perhaps we'll get to know each other?

A date is nothing special; it's just me saying "I think you're worth two hours of my time." If it doesn't work out - whatever. In fact, most of my dates do not work out in romantic ways, but instead, pave the road for long-lasting friendships. I hope that those of you reading this will understand that next time a boy asks you out, it's probably not because he's in love with you. Get over yourself.

(This is one situation, but it's happened multiple times. I don't mean to isolate anyone in particular... this was just the best illustration)

twitter: @zeteg

At Interviews, Don't ...

Let me start by saying that I am by no means a recruiting expert. I don't have any recruitment psychology under my belt, and I don't have any witty questions to ask. However, I've been giving interviews for the last few days and some things have really stood out as annoying. These are my opinions, and may not apply 100% to every kind of interview. Anyway, let's proceed in chronological order:

"The Slime" (Handshake)
You'd be surprised how many people start off with a bad handshake. If you haven't interviewed much before, you might not even know what a bad handshake is. The absolute worst kind of handshake is the sweaty, slimey handshake. If your hands sweat when you're nervous, wipe them off on your shirt and let your hands air out before the interview. Don't keep keep clinched fists. Wet handshakes feel disgusting, and they make you seem cold and inconsiderate. 

"The Deadman" (Handshake)
A handshake is a mutual exchange, and just because your interviewer takes the initiative to hold your hand, doesn't mean you should let yours go limp. Not only does a limp handshake convey a lack of confidence, it also makes me want to wash my hand afterwards and leaves me with a general feeling of disgust. Also, remember you have your entire hand to shake. Don't close off half your hand - that's awkward.

"The Governator" (Handshake)
I don't know if Arnold actually does this, but the name is funny so I used it. Remember you're shaking a hand as a gesture of respect. Do not destroy your interviewer's hand. You're not trying to show how strong or violent you can be - you're trying to convey that you're a decent human being... and you won't do that by making them squirm. Watch yourself, especially if you're a big person.

 "The Sunburn" (General)
Make sure to arrive to an interview well ahead of time. I know it's trivial, but when someone walks in out of breath or really red, I can't help but notice. While we try to look past it, I wonder if some part of me subconsciously judges you for being really really red. I know I keep asking myself when you're so red, while you're talking. It's distracting, even if nothing else. Arrive ahead of time, grab a drink, and calm down. 

"The Damaged Goods" (General)
Now, a lot of people might disagree with me on this one, but my logic is the same as for The Sunburn. If you have a cut or physical deformation in an obvious but strange spot, consider describing what happened briefly. I doubt anyone will judge you for it - but if you don't tell us what happened, we're likely to spend half of our attention conjuring possible scenarios in our head, instead of listening to you. Even something simple such as "Sorry about the cut, my razor slipped this morning" would return my attention to you.  

"The Hottie" (Nervous Ticks)
Get a feel for the room before you walk into the interview, or right as you walk in. Don't wait until half way through the interview to say "I'm feeling hot", and take off your jacket. Not only is this distracting, but it makes us remember you as the guy/girl who took off a jacket. You want us to remember you for something you did. Unfortunately, it's much easier to remember things we see, than things we hear. We see few quirks, and hear a lot of descriptions.

"Super-Intensity" (Nervous Ticks)
Please do not be overly intense. I really do appreciate enthusiasm and energy, but don't over do it in the wrong ways. You should enthusiastically describe things you've done, but you shouldn't yell them at me. You should set your pace to be about the same as your interviewer, and you should keep about the same volume. I have all types of friends, ranging from super geek to it's-homecoming-every-day, and it still throws me off guard when an interviewee starts yelling in an interview. 

 "The Mystery" (Answering)
Interviews are one of the only places where being terse will work against you. I don't want you to keep talking, but don't be too laconic either. If I ask you a yes/no question, I probably expect you to elaborate at least a bit. Don't answer "have you had any other experience" with just "yes." Take questions as an opportunity to express what you want to convey, three times.

"The Wanderer" (Answering)
Some really awesome people come into interviews without the faintest idea why they're applying, or what message they want to get across. While I'd love to hire these often-talented people, I also recognize that a smart person without drive, does about as much work as a dumb person with drive. Before you walk into the interview, ask yourself why you want the position, and more importantly ask yourself how you want your interviewers to see you. Do you want to be seen as reliable? Smart? Hard-working? Pick one or two, and focus. Offer three examples or explanations of the major points you want to project. I might miss the first two, but I'll probably remember the third.

"The Know-It-All" (Finishing)
Please have at least one general purpose question prepared, that we probably won't already have answered in the interview. If you can manage two, even better. I used to believe that having questions prepared is silly, but after conducting all these interviews, it's remarkable how much smarter having questions actually makes you sound, if they're good questions. Despite what some teachers say - there is such thing as a dumb question. If you have nothing good to ask, ask me about opportunities for advancement. Questioning advancement implies commitment, and I think most recruiters <3 commitment.

That's all I can think of for now. I hope this was helpful. If anyone with lots of hiring experience wants to argue what I've said, or add stuff of their own to this list, shoot me an email ($firstName.$lastName@gmail.com). 

You can also follow me on twitter: @zeteg

How to Accost pg at YC Events

I spent a few hours last night at YCNYC, which was absolutely awesome. However, getting through to pg isn't an easy task. In fact, it seems like he's been working on his investor superpowers. Let's break it down...

Normally it's like this

Speaker
A few people crowded around a speaker/partner, and it's pretty easy to talk to them. Not so much for PG. 

pg's Delegate Barrier

Pg
People crowding him everywhere! In fact, once in a while pg asks people to back up. Why? Well we need to analyze movement to see!

Formation of pg's Delegate Barrier

Pg2
Everyone moves in to attack pg, and pg moves backwards because people keep moving in. Why do they move in? The bigger the inner circle is, the more people can fit - which means there's a higher chance someone will talk before you get a chance to. In order to increase personal exposure and mitigate competition, people naturally nudge closer and closer.

pg's Barrier Completed

Pg3
pg is crushed against the wall, and can't escape. From there, delegates are free to beat him down with questions until Renee (who is the absolutely wonderful new events coordinator at YC) or Jessica saves him. 

So - how do you make sure you get through?

  1. Approach from the side, at the wall. If he isn't at a wall, approach from the side, slightly behind him. Given some time, he'll be forced backwards towards the wall and you'll have your prime spot. Wall spots are desirable because it's the closest to pg. In a loud environment like YCNYC, it's difficult for other spots to get a good conversation going.
  2. Find the weakest link. On your approach, you can usually lodge one shoulder into any given separation. If you're not at the wall, make sure the shoulder you insert is the one on the side of the person furthest back. This causes that person to feel like he/she is imposing, and you'll get more room. You'll also stop yourself from being squeezed out.
  3. Establish a toehold. Even if you can't completely get into a space (provided there is a small opening), put a foot in there, and wobble (slowly) back and forth. This often causes people to move aside. At the very least, it stops people from completely closing you out. 
  4. Comment, and then engage. Changing topics is difficult because it's hard to interject at the end of a response, before a new question is asked. If you can add onto a comment (kind of like HN!!), and then shift the focus, you can take the opportunity to take a step into the inner circle.

This was meant to be semi-satirical. Oh, Kelsey - email me so we can discuss!

Follow me on Twitter: @zeteg

 

 

Why Linear Search is Better than Binary Search

On the second day of my Algorithms class, my professor wrote two algorithms on the board:

Linear Search(...)
(Linear search is a process where you iterate through a list from start to finish, and return the result as you find it. Doesn't work so well with large sets of numbers.)

Binary Search(...)
(Binary search is a process where you keep cutting a sorted list in half, until you find the result. It works well with large sets of numbers.)

He paused for a moment, and asked us a fairly simple question:

What's better? Linear Search or Binary Search?

Not only does my professor have a heavy accent, but it seems like he's assaulting you every time he asks a question. We all froze up, and mumbled "binary..?" The thing is, we've been taught to analyze everything through time complexity, and for large data sets, binary search has an obvious advantage. We started thinking more. "Doesn't it depend on how the data is organized?", "It depends on large the data set is!", etc etc.

He solicited responses for a full 7 minutes (he counted), and then said:

What is better anyway? What if you have a stupid boss who doesn't understand complicated code? Maybe you write a nice binary search algorithm and show it to him. He thinks it's wrong because it's complex and he fires you. You see, now linear search is better. At least you still have a job."

Hidden animosity aside, this was a very important lesson to teach. Because we were taught about time complexity, an entire class of students managed to ignore common sense. 

Takeaway: How much would you charge if someone asked you to clean every window in the United States? Why?

 

Follow me on twitter: @zeteg

Share Your Household Expenses

This isn't at all that innovative, but there aren't too many good expense sharing resources around. Basically, my house mates wanted a way to share our expenses (shared purchases, taxi, alcohol, etc) and keep track of who owes who what.

I ended up slashing together an excel sheet late at night, so feel free to use it if it helps you. I like it more than other solutions because it's pretty simple. (Click Download at the bottom)

Click here to download:
HouseMoney_2.xlsx (156 KB)
(download)

Instructions

Replace Person X with names in Data. Add expenses, who you owe it to, and look in the Totals sheet to see who owes who what. Use negative values for paybacks.

 

Twitter: @Zeteg

 

The Tale of Eternity: Part 3

In This Series...

  1. Prelude to Eternity
  2. The Tale of Eternity: Part 1
  3. The Tale of Eternity: Part 2
  4. Interlude: See-Invisibility Exploit
  5. The Tale of Eternity: Part 3
  6. (more to come)

 

If you only care about the DDoS attack, scroll down.

I remember reading Paul Graham's essays, which kept stressing the importance of finding compatible co-founders. One or two of them also mentioned that poor founder relations are often a reason for startup dying. All praise the oracle. 

 

The Golden Years

EternityRO started with a bang. Our IRC channel was flooded with regulars, and frequented by random players here and there who needed support. The work never seemed to stop. We barely slept, and implicitly divided up our availability so there was always at least one person taking charge. Our dedication paid off, and Eternity Version 1 grew to about 1,200 simultaneous players. Boom, bang--whatever you want to call it, we really couldn't have dreamed for more. I was even scared we might go over our allocated 1TB of bandwidth for the month.

 

2008-data
I didn't have Google Analytics enabled at this point, so here's the awstats from LiquidWeb (most helpful and flexible webhost I've ever worked with). These numbers aren't totally representative of the population, since this is for the website and forum, where most players don't go. I have actual game client stats which I'll post later... that's mind-blowing. 

 

The Dilemma

Alas, all good things must come to an end. After several weeks of operation, the stress etched away Ayumi's sanity, and I witnessed a cascade of poor decisions. From freaking out at players to disappearing randomly, it was obvious that something was wrong. Like any friend, I tried to talk to her - but I was met with hostility and anger. 

If this were a smaller project, I would've said, "Screw the project. You're stressed out and overreacting, and I want to help. Sit down." That's what any good friend would have done. But thousands of people were depending on us, and by this time, almost two thousand dollars had already been invested by the population. I had to weigh an emotionally unstable co-founder against a semi-business, and the enjoyment of thousands of people. 

Enter Snow and Aisha - two of the five women responsible for EternityRO's booming success. Aisha's experience and personality made her well suited for management, and Snow was a development powerhouse. After trying to lure them into Eternity for a few days, they offered their assistance... but at a price. They wanted me to fire Ayumi and Prodigy.

 

The Decision

How does one co-founder fire the others? I guess you don't - but there was no equity in this case. After deliberating and arguing with myself for a night, I begrudgingly agreed. In retrospect, I can say I made the right decision. While it may sound morally abhorent, it might be better to look at this through consequentialist lenses. The alternative would've been to let Eternity die. Hey - I saw Social Network! I'm not the only one who did this!

Removing influential community members is usually a delicate process. You need to have finesse, and yet generate enough momentum to carry the motion through. It's actually remarkably similar to the Needle Through Glass technique I learned. The trick to using a needle to penetrate a pane of glass involves throwing the needle straight, yet with enough force to break the glass. A fault on either end will either bounce the needle back or shatter the glass. I'm getting a little bit off topic; I'll explain in another post some time how this and other techniques like breaking bricks and metal bars works. For now, here's a picture from when I was learning:

P6090039

I changed the passwords on every system, and even the ssh ports. I closed down their forum accounts, disabled their SVN access, and redirected their emails to a new support email I set up - all while they were asleep. The last thing I needed was an emotional overreaction. Eternity was going strong, and I wasn't going to let it break apart from the inside. I announced their departures as mutual, and announced Snow and Aisha shortly after. The community didn't care all that much.

 

The Hiring Process

Ayumi and the others freaked and eventually disappeared from the internet (not an easy task). To compensate for the missing manpower, I charged Aisha with recruiting new support staff. She solicited applications, and to my surprise, recruited 20 new support GMs (Game Masters). Twenty is an exorbitant amount of people to introduce at once - but my objections were moot. Although Aisha was actually very small, cute, and huggable in real life, she scared me online.

Here's an approximate breakup of what happened to the twenty keen recruits:

  • 10 of them quit or stopped logging in after the first month due to stress, incompetence, and other factors
  • 5 of them dropped out between the first and second month due to stress, and real life
  • 2 of them dropped out between the second month and fourth month
  • 1 of them was fired for outright cheating
  • 1 of them was fired for conspiring with enemies/competitors
  • 1 of them stayed on and proved to be both exceptionally useful and intelligent. Hello Griffin!

Training twenty people at once was downright painful, especially since they were all volunteers. We didn't pay any of our staff, so you might be curious why anyone would work for us. Well, people like to have a sense of power, and that's really why people become Game Masters - even though they'll tell you it's because they want to help the community

So why did 85% (17/20) of our new recruits drop out? Some of you HR fanatics have probably already decided that we had a bad selection or orientation process. While this might be true of this specific incident, our future open-hiring showed similar patterns. No, the problem is deeper rooted than our practices.

To understand why we had such a ridiculous churn rate, we need to look at motivation. People are only motivated by power for so long; once the honeymoon rush is gone, they lose their incentive. Add constant player-abuse, cheating accusations, and pressure to perform - and all of a sudden the job isn't so appealing anymore. Since they don't have much to lose, most people either quit or simply disappear.

 

The Irony

However, there was one out of the 20 who stayed on board, firm and committed. In fact, after version 1, I gave him administrative powers and started sharing all my plans with him. He essentially took on the role of co-founder, and at times even put me in my place. But why didn't he quit? Why didn't he lose motivation? You could say it was because he loved the community, but I have a better answer. 

After version 3 died, we had a reminiscent chat. Apparently the only reason he applied for a Game Master position, was so he could cheat - but not in the classic way. Because all Game Masters could see which other Game Masters were online, Griffin used this to his advantage. He ran a small bot network on the server, and linked the software to his Game Master account. His script worked something like this (pseudo code):

if(GameMasters.Online.Count() > 0) { AllBots.logoff();}

Basically, because his bots avoided the inherent cheat protection, a Game Master was required to catch them. If they were never logged on when Game Masters were online, no one could ever prove he was cheating. I laughed when I found out. Obviously, Griffin stopped doing this eventually, and probably stayed on for the same reason as I did: intellectual curiosity. Several of our other staff members were also previously cheaters - ha. 

 

The DDoS Attack

Recently, a wise man told me "when you run a successful tech startup, there are two truths: you will be DDoS'd and you will be sued." One night, I was sitting peacefully at my computer, when the server froze. After frantically trying to SSH into my box after being pelted with close to a hundred calls/texts/emails/forum PMs/IRC messages, I decided to log into my control panel to check my bandwidth. Oh my f***. Here's what I saw:

Loldos2
Normal traffic, then boom - a DDoS attack. Granted, I was on a 10mbps line, but I never figured I would need more than that. As you can see, the traffic never even came close to the limit, and I didn't have enough monthly bandwidth to max consistent use of even 10mbps. I turned off my box, and waited a few minutes before turning it back on. The attack stopped, but it had somehow corrupted one of my SQL tables. Not a big deal... I fixed it, and everything was OK for a while. Then...

Ddos
If you couldn't tell from the graph... they became more persistent. Before I go any further, I should probably explain what a DDoS attack is. This is a basic overview, and I realize it's not 100% complete or accurate. 

 

Distributed Denial of Service Attack Explained

The Distributed Denial of Service Attack--otherwise known as a DDoS attack--is a cyber attack, where many different computers flood a target with data at the same time. The theory behind a DDoS attack, is if you can overwhelm the victim's resources while making your requests indistinguishable from legitimate request, then the host can't protect against it. 

DDoS attacks are usually done through compromised machines. For example, someone may send you a virus which stays dormant until it receives a command. Upon activation, the virus will use your computer to send packets of data to a server that the controller requested. This happens to all compromised machines, which means the originating IPs of these attacks could be spread all over the world. As the victim, you can't filter out IPs or subnets at risk of filtering out your legitimate users. 

At a certain point, the victim is overwhelmed with data. Either the attack overloads the bandwidth capability of the server, or it strains the hardware or software in the machine. Eventually, if the attack is successful, the server is unable to operate properly. In our case, the bandwidth capability was overloaded.

 

Containment Efforts

I googled this, asked friends, and even emailed professors and experts in security, asking how I could stop these DDoS attacks. I remember a lot of jargon about APF, firewalls, etc, but here's one of the more helpful responses I received:

So I had a brief chat with Dr. McHue about your problem and depending on how severity the DDoS attacks, you might be screwed. He actually recommend the IP caching filtering approach that I suggested (I was surprised... I was right) however he indicated that you will probably need some dedicated hardware for it because the lookups will eat an entire CPU, possibly more. You also want to make sure you store the IP lookup table entirely in memory. Finally, this machine has to be located far enough up the network so you minimize the packet lost due to high traffic; this might mean putting the machine at the ISP even.

Other things that he suggest is simply changing the IP of the machines you have and see how long it takes for the attackers to update. Depending on how long it take, you might get some clues as to how professional these attackers are. The second suggestion he made is to incorporate some sort of distributed architecture - multiple machines for logging in and clusters for the game state. The greater the distribution, the more difficult it is to get flooded with traffic.

What did you guys do to piss someone off enough to DDoS you?

Basically... we needed specialized hardware, or (and this is key): we're screwed. Even more importantly is the last sentence! 

 

My... Roundabout Solution

Well, I was stumped, but I made some assertions that proved useful in solving the problem:

  1. EternityRO is a game, so whoever is DDoSing me probably isn't really all that professional.
  2. If I make it more expensive and risky to DDoS me, they might stop.
  3. Additional bandwidth probably costs less than acquiring new compromised machines. Outputting too many packets from each machine would make their attacks detectable and then I could filter them out.

Now, at this point, I had consulted Razor Servers on what I should do. Unfortunately, their response to stemming a DDoS attack was to null route my IP - which is fancy for turning my machine connection off. This really pissed me off, but I never stopped using Razor Servers. Their services are generally fantastic and well priced, but their customer service needs a lot of work. Rather than keep my main box there, I rented out several virtual private servers for secondary functionality - but I'll get to that later.

I did a quick google search, found that Softlayer offered large uplink ports, a Cisco Guard, and a server in Washington DC. Having no alternative, I moved my server to Softlayer. They gave me a 1gbps--1000 mbps--line for just $10/mo! Obviously, I couldn't use that much all the time, but I also doubted DDoS attackers could keep their attacks up for all that long. I essentially called the bluff and won. After a while, we stopped receiving DDoS attacks altogether. 

I guess this wasn't the best solution, but it was the solution we were looking for. I'd like to think running any project is about judgment calls - and it wouldn't have been worth our effort to find a technical solution to such a big problem. For a larger corporation, that may not be the case.

 

More to come... I'll post more as I write it. Follow me on twitter (@zeteg) for updates, or email me if you have questions. 


36 Hours is NOT a Proper Response Time

Do you reply to text messages randomly, days after you receive them? Are you really old, and have no idea why people use text messaging? In this fully-annotated PG-13 post, I will try to explain why you need to crank up your phone etiquette. Let's start with several of my least favorite cases.

 

The WTF????

Usually I'm more respectful of attempts to conversation, but this is just too much. If anyone can tell me what she's trying to say, I'll buy you a drink. In case you don't know, this is an iPhone conversation where my text is in green, and my friend's text is in white. Observe:

 

Bad

Come on - seriously? I really didn't know how to ... SKDJGSHKDJHKJFSDKSDGN

 

The Delay

I end up in a lot of confused text-conversations about nothing. How can you talk about nothing? Take this wonderful example:

Nora

Thirty three hours. Thirty three. 

(Kris is my English name which some friends use... because people tend to butcher Shenglong, or ask me 5 times if that's actually my real name. More on that some other time though...)

 

The Over-Emphasis

I love it when people text me enthusiastically; it makes me feel like I'm not wasting my time replying. Once in a while though, lack of originality and emphasis can be a little... silly:

Ooo

I am tempted to model her O's as a function of messages received. 

 

Why Text Anyway?

There has been a lot of negativity surrounding text messaging - and how carriers make a ton of money from them. I had a brief detour in youth marketing at a major carrier, and I can testify that changing send-and-end users (people who just call) was a primary objective. However, I also used to be a send-and-end user.

In the United States, calling cell phones makes perfect sense, since you have nation-wide roamaing and long distance, just as if they were local minutes. This isn't the case in Canada though, and sometimes you get charged long distance, even when dialing in the same area code. The charges are so obscured, actually, that the majority of university students in Canada don't even know in what situations they would be charged for calls.

Add on background noise, multiple conversations, and texting in class or at work, and you have plenty reasons why texting is great. The main use of texting though, is the ability to carry on passive conversations. Texts aren't as long as emails, and it allows for quicker exchanges when necessary, and slower exchanges when material runs dry.

Maintaining a good text-conversation allows you to build rapport over time, and over multiple situations. You're passively inserting yourself into someone's entire life - and any good salesperson will tell you how important that is. The reason you want to have dinner with sales targets and have golf with them, is because you want to expand the field of exposure in order to gain trust. Texting isn't as good as being there in person, but it's sure a lot better than nothing. 

 

The 2-2-2 Rule

My obsessive use of phone and email has led to the invention of the 2-2-2 rule, which I often impose onto friends: pick up calls within 2 seconds, answer texts within 2 minutes, and respond to emails within 2 hours. There are obvious exceptions, but it's a rule of thumb to try and stick by whenever possible. 

More than anything, if it's important, I'll probably call you before I text you, and text you before I email you. Needless to say, I've been told to screw off on many occasions :)

 

Follow me on twitter: @zeteg 

Another chapter of The Tale of Eternity this week. Sorry for the delay. 

Interlude: The See-Invisibility Exploit

Once in a while I will take a break from the main Tale of Eternity story, and elaborate on a very interesting issue: exploits. I will overview some common hacks, how they work, what solutions are normally implemented, and how we solved the problem. I'll try to generalize more advanced concepts so everyone can understand, but some of it might be unavoidably technical. First, there are many different ways of cheating - but in today's post, I'll just outline one. 

 

See-Invisibility Hack

Otherwise known as the Maya Purple Hackthis is a fairly common hack exists in just about every game that incorporates invisibility. On RO--and probably most other games as well--this hack works because of unnecessary information being sent from the server to the client. There are semi-logical reasons for it, but in the end we have to blame it on poor programming.

 

The Reason

Normally in a visible-player setting, the client needs to tell the server "Hey, I'm at square x,y", and then the server checks for all players within the area (x +/- 14, y +/- 14). For each player it finds, it tells them, "Hey, this player is at square x,y", at which point each recipient's client translates that information and shows the player visually. Pretty simple, and it makes sense.

Of course, location data isn't the only data that's sent. At the same time, sprite data (how the player looks), class data (what class the character is), and a bunch of other data is also sent--including visibility data. With the way the RO client is programmed, received information that isn't recognized by the client results in an error dump--a message with lots of Korean code, followed by a client crash.

I'm guessing it's because of this, that some genius at Gravity Inc. decided to implement invisibility as it works now. With invisible characters, the same aforementioned process happens, with one key exception. After the client receives the location data for the invisible player, it also receives visibility data, informing the client that said player is invisible. In turn, the code tells the client to show nothing, rather than a player. Okay, this solution works as long as no one tries to tamper with the game client.

 

The Problem

There are two generic ways to exploit this: hex editing, and packet filtering. There is a third way, but it's game-specific and not worth discussing at length. 

Hex editing works with the hexadecimal makeup of a program. Just like a program can be represented in binary as a string of 0 and 1, it can be represented in hex. With unencrypted files (such as the RO client), hex editing is extremely easy, and values can be changed in a heartbeat. Unfortunately, such was the case for the RO client. For clarification, imagine the following piece of pseudo code:

if (visibility = no), then {player = not visible}

Essentially, what you're doing when you're hex editing is changing it to:

if (visibility = no), then {player = visible}

Hey, now you can see invisible players. Granted, it's a little more complicated than I've described. 

 

The second method is packet filtering. By ignoring visibility packets and filtering for position packets, a user is able to translate the packet data directly in order to see hidden players. This can be solved with a good encryption technique, but encrypting every packet for real-time gaming proves to be a real problem. Some problems originate from increased server load, and others from mirroring the decryption on the client.

If you use a basic encryption technique, it's going to get cracked; that's a fact you can't avoid. In order to encrypt and decrypt packets on the server, you need related functionality on the client. This ultimately means you're delivering your encryption system to the exploiter. He's going to diff your files, collect packets and analyze them for patterns, and eventually try to solve your encryption. Sadly, the more secure your encryption technique is, the more load it imposes on the server. 

Let me give you an example of server load: On RO, potions can be consumed at a most, 10 per second. With 2,000 players all consuming on average 5 potions per second in Siege War, there are about 20,000 MySQL inserts per second, counting inventory and logs - which is taxing enough by itself. Try to blowfish each of these 10,000 packets, and a nightmare ensues. 

 

The Solution

The traditional methodology in addressing this problem strikes an uncanny resemblance to struggling competing products: a functionality war. The builders keep improving on their old techniques marginally, and after a week or two, the hackers catch up marginally. The cycle just keeps continuing, eating up a ton of time, and not really getting anywhere. You see, when you introduce a technical problem, the exploiters--who are people who do this for fun, just because they can--are excited about the challenge.

To disengage people from breaking the encryption, you can either introduce a revolutionary technology so profound, that it's no longer fun to solve, or find an alternative method that can't be attacked directly. A technical solution was never finalized for the invisibility hack, and there came a point where it became a big problem on Eternity. 

Because we had no reliable way of catching and proving cheating, the bad players started using it - giving them an advantage over people who didn't use it. The good players, feeling the situation was unjust, decided to level the playing field and use it too. Soon, a large portion of the server was using this exploit, and we were receiving a ton of complaints.

Then, genius: Somewhere in the middle of version 2 (I haven't gotten that far yet in my story), I was discussing this issue with my co-admin, Griffin. I rebooted my brain, and purposed an idea for an alternative way of catching these cheaters - we'll codename it Project Stalker to be hip and unique. Project Stalker involved several steps of logic.

  1. There are only a few ways in the game to reveal a hidden character.
  2. By manipulating game mechanics, it is possible to make a "hacked character" undetectable by normal means.
  3. Hence, if anyone sees this character, they must be cheating. But how do we tell?
  4. When a player hovers their mouse over a character, a packet is sent to the server, requesting the name of the selected player.
  5. Thus, if we receive a name request packet on an undetectable character, the requester must be cheating.

Griffin understood immediately, and proceeded to code it into the server. He had the server spit back logs of players who try to request a specific character we hard coded in to be our undetectable character.

Next Siege War, we tested the system. Griffin ran around as Project Stalker, and the server spit back the names and IPs of every player who requested his packet - all the while without letting them know what was going on. The numbers were shocking. Over 50% of the players were cheating! Instead of banning everyone, we alerted the guild masters of cheating players (some of which were cheating themselves) and made a general announcement: "We have a way of catching you. We're letting today slide, but next time, you will be banned." 

We succeeded. People were scared, and they realized we weren't lying. The amount of invisibility-exploiters dropped from over 50% to less than 1%, and we went through several Sieges without catching a single cheater. This remarkable feat gave us a lot of credibility in terms of catching cheaters. 

 

More to come... follow me on Twitter (@zeteg) for updates

Why ISPs Shouldn't Ban MAC Addresses

I'm very fed up with the University of Toronto, and its terrible network management. Last Saturday, I was cut off from the internet at about 11:00 PM for no apparent reason. I hadn't touched any cables (I'm hooked up by a cord), and the other people in this suite had perfectly working internet. I sighed heavily, scooped some ice cream, and proceeded to diagnose the problem.

The little connectivity symbol on my startbar was acting strange. Once in a while it would appear normal:

Box
But sometimes, a yellow exclaimation mark would appear, signaling that something was wrong. After running a diagnostic test, Windows 7 told me my configurations were correct, but it couldn't connect to the primary DNS. I tried disabling and re-enabling, jiggling my cord, and even ran a bunch of ipconfig commands in hope it would be fixed. No luck. Here's what it would look like:

1
Ok - I had three ideas:

  1. My ethernet card was broken
  2. My ethernet cord or socket was really broken
  3. My MAC address been banned off the network without notice

Let me rewind. I'm currently living at the University of Toronto for the summer for an internship - I don't actually go here. My $600/month under a 4 month contract is supposed to include internet service, and I have been very careful not to violate any of their sensible rules. I say sensible, because I'm pretty sure a failing law student drew up their contracts. I point you to one of many issues with the Occupancy Agreement, for example:

 

14. The Resident will not keep any firearm, fireworks, weapon, explosive, animal, fish, reptile, insect, bird in the Room, Suite or Residence

 

Read it carefully - multiple problems should jump out at you. This is a completely insensible request, and every suite in the entire building is in violation. So once again, I obey all the sensible requests.

To test whether my ethernet card was broken, I tried a wifi connection, and also connected my computer to my friend's computer through the same ethernet cord. I then connected my computer to my friend's wall socket. This tested both #1 and #2, and showed me that for sure, I had been banned.

But why? I didn't get a knock on the door, a message, or even an email. Okay, network administrators are sensible people - I'm going to go talk to them. If they're up banning me at 11:00 at night, they should probably be awake right? Wrong. My suite mate told me that apparently (I can't confirm this) this university has software that flags individuals. Then, apparently there is a delay between flagging and banning. This means that once you've been flagged, you will be banned at some random time in the future. I really hope this is not true... but evidence thusfar corroborates the theory.

I went down stairs and I was told that the technicians "do not have a set schedule". Furthermore, I was told I could not contact them directly, and that I could fill out a form and they would find me "when they have time." However, being a long weekend, that might not be until Tuesday. I was about to have a verbal scuffle at this point with the desk staff, but realized they couldn't do anything about poor policy. I asked for a contact number, made them aware of my situation, and went up stairs to sleep.

The next day, a technician comes to my door and tells me he can't do anything to help me. Yes, he came to my door to tell me this. He further assured me that he would find out what was going on, and at least let me know by Tuesday. He couldn't even find out why I had been banned, and blamed it on the "central network" - and his tone suggested it was some sort of omnipotent power. He further suggested that I may have been banned in error, and that I should find alternative ways to access the internet.

Good point, technician! I might have been banned in error. I went back to my computer, and decided to just get myself online, because it's not like they intended to ban me - right? 

2
Good thing Microsoft smartened up and made it easy to spoof your MAC Address in Windows 7. I didn't even have to do any tinkering. So there it was, I changed my MAC Address and what do you know? This happens:

3

This is the standard page, telling you to authenticate yourself. I just used the same information I signed up with originally, and it worked. Had they banned my records, could easily have wrote a script to brute force every room-birthday combination. Of course I didn't, because I wouldn't want to do anything bad. They hadn't banned my room, so it must have been a mistake. I ran a security check, and bamn:

4

I was back online! I could hang out with all your wonderful people again:

5

 

Subsequently, I have been banned again yesterday, and just now again about 50 minutes ago. The funny thing is, I figured maybe I had broken one of their rules.. so I did a test. Between when I was first banned and now, the only things I have done are:

  • Visit HN and read articles
  • Use Wikipedia
  • Check my Gmail
  • Use Facebook / Twitter / LAL / G+ / MSN / Skype / GTalk
  • Google random things here and there

No video streaming, no downloads, not even YouTube. It's almost Thursday, and no one has gotten back to me yet. I know this isn't a big technical accomplishment, but it's a great illustration of outdated techniques and poor implementation/service. At the very least, it's a semi-funny story to read (I hope). 

The Tale of Eternity: Part 2

In This Series...

  1. Prelude to Eternity
  2. The Tale of Eternity: Part 1
  3. The Tale of Eternity: Part 2
  4. Interlude: See-Invisibility Exploit
  5. The Tale of Eternity: Part 3
  6. (more to come)

If you haven't read the prelude or first part, I strongly suggest you do before you read this segment, as this might not make sense otherwise. 

 

Life Beyond Death

I made a lot of enemies before Eternity even launched. As it turned out, our early adopters didn't just leave their previous servers - they took it upon themselves to hasten the eventual demise. With nothing left to lose, they marched in proud defiance of rules, conducting all sorts of unsolicted behavior. Advertising for Eternity, obvious botting/packet-sending, and DDoS attacks were just a small part of the mess.

I imagined most marketing managers would be ecstatic to have their customers attack their competitors unguided - It would be some sort of pre-emptive victory. I certainly felt this way, and aside from the unconvincing PR speech about how "we shouldn't cause damage on other servers - we're better than that," I did very little to stop it. 

Looking back, it's blatantly obvious that my long line of mistakes in this whole ordeal started here. I'm not going to spew moral bullshit and tell you how I should've abandoned my sanctimonious stance. Regardless of whether I erred morally, it's a failure in foresight that haunts me most.

should have pulled a Gandalf and voiced an epic proclamation that would have echoed for generations to come. "You shall not pass!" - or at least, "We will punish you here for bad behavior on other servers!" is what I should have said. Yet, I didn't. I chuckled at their misfortune, eyes gleaming and mouth watering in anticipation of the power and glory that lay ahead.

You see, back then I believed morality played no role in business - which is not what my Business Ethics class taught me. Though to be fair, I'm not really sure if that class taught anyone anything. Ethics classes tend to ramble on about what some 200-year old skeleton believed - and not once has anyone given me a convincing reason that ethical choices should be practiced in business when these choices are in opposition of obvious benefit (not just immediate profit). 

If you're in the same boat as I was, I'll share with you the two most convincing reasons I learned (It's greyed out because it's kind of boring):

  1. Modern human society is based on a sense of trust. We trust our government (or used to) to enforce the law, we trust our neighbors not to steal our things, and we trust teachers we barely know to take care of our children. It makes sense, that a sense of right and wrong--the fundamental basis of morality--is derived from our survival. One could infer, given this line of logic, that when involving multiple parties in a public setting, morality is also the superior choice. Yes, this argument has logical deficiencies, but this isn't a debate, and I did not describe what I'm trying to say perfectly. Hopefully you'll extrapolate and derive some value.
  2. Even if your business doesn't prosper, making an unexpected moral choice leaves a profound impact on those around you. Unfortunately, this only really works if it's a heavily publicized decision. I read an article today detailing Benjamin Franklin's refusal to have the Stove patented, because he believed others should have access to it as well. Wow, makes him seem like a good guy! I think it's natural for people to respect genuinely good actions, whether it's something they would've done or not, and having this sort of respect can make you life far easier in the future. 

I talk about a lack of foresight, but the truth is, throughout this project, I've made some extraordinary insightful decisions. I've received praise from players, staff, and even previous competitors. Many people say praise is a dangerous beast that consumes its target - but I rarely see a modest man swell with pride. It appears that praise only breeds haughtiness in cases where the praised are already egoistical.

That was me. I did not believe that anyone could make better decisions than I could. I thought I was the smartest guy around, and I could easily account for any amount of steps my competitors were planning. I was sure I could predict the market - and you can't imagine the depth of my smile when a player said, "you're an oracle... you just see everything." 

The disaster that my amoral actions wrought were only revealed half a year later when players started to do to Eternity, what they had done to our competitors (I'll get to this in more detail when I get to that part). If I had set precedence for how all servers would handle bad behavior back when we had leverage, Eternity version 1 would have lasted much longer. In fact, it could've been my chance to establish and lead a system akin to patent laws in the United States.

But no. I told myself I could predict player actions, and went on to enforce a set of rules that actually did work... until players adapted. Your users will always adapt if they force any sense of irrationality on them, even if it makes sense on your end. Much like viruses and bacteria, an adapted version of the original can be more harmful than the originating strain. Now armed with experience, I heavily advice against forcing adaptation on your users. I see Apple doing this, and I feel quaintly nostalgic. Much like Apple, I should have found an alternative strategy... kind of like how Diablo 3 plans to deal with gold farmers by introducing a new business model aspect.

 

What's a Server?

Just so you don't think I'm understating how much I knew - I spent a good deal of time trying to figure out the difference between shared hosting, virtual private servers (VPS), and dedicated servers. I guess some part of me always imagined that RO ran like a regular website. Permissions? Bah! Memory? Who needs that!? Wait wait... what's memory again?

By this time, I was googling things left and right, and trying to find a cheap dedicated server. After browsing around, I eventually ended up with Razor Servers both because they were cheap, and because their servers were based in DC at the time. You see, DC was the perfect spot for a gaming server, and Razor Servers in particular offered very low latency to most areas we were serving. Eastern Canada/USA had 30ms ping, west coast had <100, and Europe had less than 140 - which is much better than people were used to experiencing.

RO relies on reaction speed, and an extra 30ms of ping is often the difference between winning and losing a fight. Latency and reliability were important, and all the marketing materials on their website told me they provided both. Yeah, I realized this wasn't all true, but I also realized that because they claimed it, I could tell my players, who in turn believed me since a server provider can't possibly be exaggerating. 

But heck, a dedicated server was going to cost me a lot of money each month.

Inv

That's $100 I didn't have. Gosh - time to think outside the box again: time to go to sleep. You see, half of my thinking is done on the border of consciousness and sleep - sometimes enter a blissful state where I can think of thousands of concepts in a split second and analyze them too. I don't know if this is real or just imagined, but regardless, most of my ideas come from either when I'm falling asleep or I'm waking up. 

Never before done in the RO world - I asked for pledges. Before I explain the details of this financial instrument (I made more complicated ones later on), let me go over how finances typically work for any given private server:

 

The Problem with Old Cash Shops

Every server needs money to run, and very few server owners bootstrap their servers. Usually they make an initial investment and either go for a break-even model (rarely by choice) or profit from donations. They're called donations, but they're not really donations. They're payments, in exchange for some type of virtual currency or service. I guess Farmville has made this concept pretty popular, but when we did it, the concept of Cash Shops had just sprung up.

Most servers offered substantial rewards for donating, and gave away powerful items that otherwise couldn't be obtained. Pause and think - what could be wrong with this model? Well, the people with money usually aren't the hardcore players. To put it in more offensive terms: whales are never athletes. Because of this, every server using this method eventually developed a secondary market for the resale of donation items. 

There were two forces that eventually drove down the numerical worth donation items even in a heavily inflating market: progression and liquidity. Players will always progress faster than anyone can reasonably introduce new donation items which were at least somewhat balanced. To add to this, once donation points are spent, the resulting item loses a significiant portion of its value because it is now restricted to certain classes and builds (cut in demand). The whales never colluded, which further allowed the hardcore gamers dictate the digital worth of each dollar. This became a pivotal strategy concept for Eternity.

So, by now you're curious. How much money can an RO server make? I'm not going to reveal Eternity's financials (although I will mention them later on), but a server with 2,000 players peak and statted donation items can make upwards of $20,000 a month. Not just a "game" anymore, now is it?

 

Eternity's Decision

I told everyone we would not have any statted donation items - knowing it would significantly cut potential profit. However, it aligned with our culture, and it served as the first step to the cash shop Nash Equilibrium. Yes, this is one of the reasons we had so many potential players. We screwed over everyone including ourselves in terms of $value/player, but increased our number of players significantly. Worth it? We'll see.

Instead of items that add power, we introduced items that would add aesthetics--pretty hats.. useful and pretty hats. I came up with the idea of synthesis, and you can read my poorly written code here. It's just a NPC script, and anyone with any experience in programming should understand basically what it does. I've never given out a script before (ever), so feel special!

Synthesis was our selling point, and it's an idea that's been used (I don't dare say copied because I have no proof I was even the first to think of it) by various other games that now exist. Essentially, a player could merge a good looking item with the stats and abilities of another item. 

 

Complex Financial Instruments

I went through a phase where I really wanted to be an investment banker. Why? Well, one night, a friend and I were lying in bed chatting (platonic), and I brought up how I don't usually wear formal shoes to interviews even when I wear a suit. She asked me why, and I told her it's because I value performance over appearance, and that formal boots are not really good for anything practical. She hmph'd and declared "Goldman Sachs would never hire you!" I responded, "What's Goldman Sachs?" and from then on, I was hooked. A challenge had been issued, and I was determined to join Goldman Sachs. 

In those days, I had the pleasure of reading about the IB industry, and about financial instruments. I think it's from there, that I got my idea for pledges, and later in v2, options. I asked the population for an early donation, on the promise that if we launch, they would be rewarded double. It worked. We fund raised a good $1,000 in the first two days, and I was in financial bliss. 

 

Rush for Substance

Oh boy - now we had more willing customers than we hoped, a ton of money, but no product! This was like the reverse situation of a usual startup, and it sounds better than it really was. As I have mentioned, I had no idea how to set up a RO server, much less how to "fix lag" and make other promised things come true. Ayumi? Autumn? Prodigy? They had no idea either!

Oh right - by this time, we had added a fourth co-founder at Ayumi's request. It was actually the guy she was e-dating. I'm not sure why I willingly agreed to dilute my interest, but it probably had to do with a fear of incessant whining. Regardless, it was done, and he doesn't really play much of a role in this story. The fact remained - no one knew how to do anything. 

Enter Articulus - savior, benefactor, and all-around awesome person. Articulus graduated engineering in California, and founded the server I had played on before, so it was only natural to approach him for help. Rather than setting everything up for me, he decided I had to learn to set things up myself... starting from the basics: CentOS. Why CentOS, I still don't quite understand. I struggled and struggled, but eventually got the OS and everything set up to run RO. I set up subversion on the system, and configured the proper settings with a lot of help from him. He's the kind of developer that every company should be looking for. It's genuinely difficult to find someone smart without a massive ego, who goes out of his way to help others with time-consuming problems.

I know setting up Linux and configuration doesn't sound like a lot, but for someone who had no idea what he was doing, and still going through school, it was tiring. I had a few sleepless nights in preparation for launch, and it's safe to say my nightly dreams transformed into a dedicated server committed to configuration. QA and testing became second nature, and I I realized I had a penchant for finding esoteric bugs. 

 

The Payoff

It was launch day, and everything had been set up and tested.

5

4

3... I turned on the login server

2... I turned on the character server

1... AWW SO CUTE

Yes, if you ever have the luxury of building connective software that requires multiple parts to function together, please watch how your eager customers spam your login. I swear some people wrote bots just to log in. I had 200 people spamming the login every 5 seconds. It's interesting to note, that we later used this fact to find bots. 

After watching with a blissful smile, I remembered I still had to launch map.

0... Map launched!

I watched the character count on the server skyrocket to 350 in the first 2 minutes. Ten minutes later, user count was at 500, and peaked out at about 750 two days later. That's right - 750 people online at once. Our team rejoiced, and we honestly thought the hardest part was behind us. Boy, were we wrong. 

 

More to come... I'll post more as I write it. Follow me on twitter (@zeteg) for updates, or email me if you have questions.