komyou - Tumblr blog

komyou · 8 years ago

Text

Part 3: What Can Be Done?

This is part of a series describing the history, context, and technical details of the modding we've been doing resulting in this big En Masse / NA TERA commotion:

Introduction. What's all this about?

Part 1: The History – A Timeline. Take a walk down memory lane from 2012 to today.

Part 2: Techno Mumbo Jumbo. The technical details of how and why these mods work.

Part 3: What Can Be Done? ⬅ You're here!

Conclusion (not yet published). Obligatory closing thoughts.

Feel free to skip around to the parts that interest you.

May 1st, the first scheduled protest day, has come and gone, and EME doesn't seem to have done much of anything with this subject matter, so it's time for the last bit.

The Past

BHS has done two things to try and kill off meters in March 2016, and they're both laughable attempts at best.

Encryption Keys

The first attempt was to update the key generation. As mentioned in the previous post, the client and server need two keys to set up encryption. What I didn't cover was that these two keys are manipulated to produce the initial encryption state. It's a little out of the scope of this post to explain in more detail, but Wikipedia is there for anyone wanting to learn more.

There's one thing that's particularly relevant to this case, and that's the circular shift. If you had a 4 digit number, such as 1234, and you wanted to perform a single circular shift left, you would move the "234" left and the "1" would go back around to the end because we want to stay at 4 digits. So 1234 after one circular shift left would turn into 2341.

Likewise, 1234 after one circular shift right would turn into 4123.

Why's this of particular importance to us? Because BHS's attempt to thwart meters was to change the shift amount in three of their circular shifts.

That's... That's it.

We have entire files dedicated to dealing with BHS's custom encryption scheme, and all they did to try and shake meters off their trail was change three numbers.

Seriously? Seriously. Really?

Look. It's bad enough that they made their own cryptography. It's something that pretty much everyone advises not to do. That's not too relevant here because the prevailing concern there is that your cryptography won't be secure and will be very vulnerable to an attacker.

That's not BHS's concern here. They're not trying to stop an attacker that's trying to hijack TERA connections in a coffee shop. They're not sending sensitive data where it's a privacy breach for a player if someone can read it.

They're only trying to stop things like meters and proxies and "third party tools" from playing with the data.

And that's fair. Totally a good idea for them to do, because that's the sort of thing that stalls widespread network data manipulation for years. It clearly hasn't stopped it, and it's certainly been around for just as long if you've read the first part of the series, but the less accessible you make it, the less you have that sort of stuff running rampant.

Rolling their own crypto is, to me, indicative of another problem: BHS has absolutely no idea what they're doing.

There's an algorithm called SHA-1. It's a hashing algorithm; that means it takes any arbitrary data you want and maps it to some fixed size value. It's supposed to be as difficult as possible to recover the original data, and as difficult as possible to find two things that have the same hash.

SHA-1 is a very well-known algorithm. That thing that shows you the padlock on secure websites—like this one, or PayPal, or even En Masse's site? For almost all HTTPS websites up until a few years ago, that was with the help of SHA-1. All the links I've been making to specific changes from GitHub? In Git, each set of changes is identified by a SHA-1 hash, so you can link to a specific changeset. Point is, this algorithm is everywhere and there's too many implementations of it to count on two hands.

And BHS still got it wrong.

Shrina and tera-proxy and everyone else has to include a custom implementation of SHA-1 as part of the overall encryption code because BHS can't get a ubiquitous algorithm right.

It doesn't look intentional to me, either. There's very clearly a set of constants that could've been changed in their custom implementation to make it like-SHA1-but-not-actually-SHA1. Those are unchanged. It's more like they went down the checklist of how to do SHA-1 but then wrapped it up and called it a day without realizing they missed the very last step.

Which more or less describes the actual issue in the SHA-1 implementation, but that's not a topic for this post.

Here's my point. They botched SHA-1, and then thought changing three little numbers would be enough to stave off DPS meters.

Reverse engineering that stuff isn't anywhere close to my skillset, but even I could tell you that you're gonna need a lot more than that, because that was fixed in practically no time.

Signature Detection

Around the same time as the encryption key change, in other regions that use one of those anti-hack systems, they added a block on Shinra's "signature".

Now, I'm not very well-versed in this stuff at all, and I haven't personally had to deal with this myself nor have I seen first-hand accounts, so I'm really going off speculation on how this detection works.

Antivirus systems have done signature-based detection for a very, very long time. The principle of it is that they know how certain key parts of unwanted programs look (the signature), so when they do scans of whatever's running on your computer, if they see anything matching a known signature then they can be reasonably certain it's a bad thing and take whatever action.

Based on what Shinra did, I assume that the anti-hack in whatever TERA region used a similar concept. Again, this is speculation, but here is what I think happens: the anti-hack looks at what programs your system is running, then goes to the .exe and reads it and computes the hash, and if the hash matches anything in its blocked database, it won't let you launch TERA.

It's an okay idea, because then it stops users from doing things like just renaming the file to get by checks on program names. But you might have noticed another very simple way to bypass it, and that's what Shinra did.

They added a useless line to ShinraMeter that they were absolutely sure would be included verbatim in the exe and wouldn't affect anything else, and then added a program to randomize that specific line. Now everyone's copy of ShinraMeter could be made unique.

And... that's it. That's really it. This section is done.

The Present

Over a full year after BHS's failed attempts to ward off these network-based mods, the next major incident brings us right up to the present.

On Developers

Three major developers—Bernkastel, Pinkie Pie, and myself—were all banned. Everyone else either changed GitHub names to mask their in game identities, or they never cared in the first place because they play on EU / other regions.

Want to know a common thread among all of them, banned or not?

Nobody stopped developing.

After the ban, Bernkastel released a costume-changing script for free and continues to support it and other scripts. Pinkie Pie? I'm not sure, really. I'm not in their Discord, but I hear there were talks of developing a private server.

(As for me, I'm taking a break. I'm a grad student pursuing an MS in Computer Science, thesis track. Between working on my thesis and writing this series, there's not much time left for work on proxy stuff. That's fine; I want to see how this whole thing goes anyway before I make any moves.)

These developers were toeing the line with their identities at stake. When they wake up one day and suddenly find out they're banned, you've taken everything away from them. For EME, there's nothing left to negotiate with. For a banned developer, there's nothing left to lose.

What do you think is going to happen when you suddenly snatch away something that someone has been building up for years?

I've played TERA for five whole years. On a roleplay server. I can tell you about each and every single one of my characters' roleplay personalities. I lead a guild that's existed since May 2012 and that nobody's even heard about because we're an insular community that just wants to play the game with each other and have fun without seeking fame and glory. At some point or another, I've mained sorc, priest, lancer, archer, zerker, and mystic. There's a ton of history there.

In fact, if you tallied how much playtime I accrued since the last time I checked and compared it to how long NA TERA has been out, it comes out to about 43%.

If I logged onto TERA every day since May 1, 2012 until the date in that chart, I would have had to be online for 10 hours a day. Every single day.

And that's only the characters that were on CH.

So when I wake up one day and see an email that says, "Hey, um, we were kind of cool before but today we decided you don't get to nolife in our game anymore," it's kind of hard to not have an initial reaction that isn't along the lines of, "Oh, well, I guess I don't have to be careful with anything anymore."

In fact, that's about one of the first things I typed since I found out the news:

o i got banned well screw playing nice i'm listing cheaty shit on the directory now 🤷

In the days leading up to the ban, I did a massive revamp of some stuff and pushed a number of changes to the proxy (around 1,000 lines of code added and 250 removed) in preparation for a solid, real, actual "1.0.0" release to line up with the upcoming major content patch. As part of the big update, I was also in the middle of stepping up the website and documentation as well, which would include a listing of modules that were on the arguably benign side for proxy users. I DM'd a number of module writers on Discord to make sure they were okay with having a specific list of their GitHub modules added to the page.

So when I got the ban, it was like I didn't have a reason to make it any more difficult for people to find the controversial stuff anymore. Why not list those too? Doesn't matter if I'm already banned. Automatic mana pots? You're in. Costume changers? There you go.

Since the ban, I haven't actually made any changes there, and I still haven't published the new website either. The documentation isn't done yet and I don't want to publish something that's blatantly unfinished.

Either way, these banned developers were just players that largely wanted to fix flaws in the game that have gone unaddressed for years. We put our names to these fixes, and generally tried not to go too far with our shenanigans. And when bad people roll up, exploit the rewards store and whatever else, and then when we get swept up in the banwave with paltry attempts at communication, there's not much of a reason for us to play nice anymore. If we wanted to blatantly exploit the game, it would not have been under our own names.

I'm not trying to argue whether or not the ban was justified. Everyone's already dumped their two cents on why it was good or bad. I'm just here to point out that the ban doesn't do a whole lot to developers.

For instance, I've mentioned it a few times but Alkahest is a project that aims to do the same as tera-proxy, but in a different programming language so it's much simpler to add a proper UI. The Alkahest developer had the same idea as me with a 1.0.0 release landing on the next big content patch, and we've been talking about interoperability so that people can use both tera-proxy and Alkahest at the same time if they want to. And then we talked with ShinraMeter about interoperability with that too, so people can use all three at once with minimal issues.

So. You've banned some major developers from NA, one who had a major public community presence for other reasons, and you haven't actually stopped any of the development. In fact, you went full Streisand effect and now a lot more people are aware of the proxy, which only gives developers more incentive to support their things, now potentially with malicious intent.

Hmm.

On Players

I don't really need to cover this, do I?

Forum threads are being locked or disappearing altogether, users are being shadowbanned, and others are being straight up forum banned—sometimes for seemingly no reason (update: official reply from Spacecats).

This is a mess.

Reddit learned their lesson and replaced shadowbans with more transparent account suspensions over a year ago. Why is this still a thing?

People want clear and transparent communication and moderation. Anything else and you have them feeling silenced and censored with a company that they feel is growing increasingly out of touch for a game that's been called "dying" if not "dead" for years.

I don't even have much to comment on this. So many people already consider it such a massive PR disaster that there's not really a whole lot else for me to say.

And you know what? The players that did get banned? They all know precisely where they messed up. A meter shown on stream or in screenshots or dumped in chat, or something a little more noticeable like automatically and instantly accepting all broker offers for the listed price.

EME did scare off a few people, I'll give them that. But everyone else is either continuing as normal, or laying low for this whole fiasco to blow over before going right back to business like usual. For many people, especially the ones that really needed things like skill predictions or can't get enough of delicious quality of life mods like automatic Vanguard completions, all you've done is make them less likely to show their game to anyone. Less screenshots, less streams, and... people still using the mods.

Banned developers. The developers still develop.

Banned some players. Many still play with their mods.

Hmmmm.

In the midst of all these bans being big news, why are people still doing it anyway? Because if you play your cards right, there's no way for EME to detect these things.

"Undetectable"

That's a pretty big statement to make, but with a few footnotes, I wholly stand by it.

I'm not trying to advertise this or anything as "our product is great because we engineered ways to get around EME's detection systems". The fundamental problem is that it is a cat and mouse game, and for any method BHS or EME takes to thwart these sorts of programs, people will find a way to get around it. It's only a matter of time and effort.

How can I be so sure? Think about it like this. BHS and EME have only one thing under direct control: the game servers. We can be reasonably certain that they won't be having anyone breaking in and fiddling with some bits or changing the server code and suddenly they're cheating. The server is, for most purposes, safe.

However, they cannot fully control your computer. Once you download the game, it's in your domain, not theirs. You can start up Cheat Engine—hell, go the extra mile and custom build an OS that can run Windows programs with some tools to directly edit memory without detection—and the only thing "stopping" you is the TERA client noticing it and refusing to play nice.

So then you just trick it so that it doesn't notice. Then they have to find a new way to detect it.

It's a cat and mouse game where the odds are in the end user's favor. This is the same reason why offline DRM systems in general are doomed to fail. Someone was able to use it legitimately, so it's just a matter of figuring out how to reach the same thing illegitimately with every tool at your disposal and absolutely nothing stopping you.

Sometimes, they do a really good job, but those sorts of systems get broken eventually. It's just a question of how much time and effort people go into breaking it. Just Cause 3 was protected by a system called Denuvo, and so have a number of other games. It took over a full year for a public crack of Just Cause 3, but other games started falling in months, weeks, and then days. Was Denuvo considered a failure? No. They know they can never be uncrackable. All you can do is delay the inevitable, and often times that's enough.

The only way EME and BHS can properly attack these "third party tools" in this manner is with increasingly aggressive detection tactics that quickly land a company in hot waters. Nobody likes things like GameGuard or PunkBuster—not even legitimate users. What about subtly introducing detection mechanisms? If all they do is throw up an error message and prevent the game from launching, people will find a way to get around it and you've only stopped the problem for a few hours, or days, or if it's a real good attempt, maybe weeks.

How about going more aggressive? How about applying a permanent hardware ban instead of just showing an error message? Well, are you ready for false positives and complaints about privacy? The big GUI system I was planning for tera-proxy uses something called Electron. You know what else uses Electron? Discord. Can you imagine if they did a signature scan and got a false positive on Discord, banning everyone that uses Discord on the first day of their stealthy anti-cheat patch? Yikes.

Now, look. Their past attempts did work for a while. TERA.exe uses something called Themida to make it harder for reverse engineering. The custom encryption system raised the barrier to entry for network reading and modification. They did their job... but not for long.

It's a technique called security through obscurity. If people can't figure out how your system works, then surely it's safe from attacks, right?

That's pretty much how TERA has been operating for five years. There's so many glaring security holes that I wish I could disclose, but I'd first love to see if EME would even be open to an attempt at responsible disclosure. It might not matter anyway, seeing as security issues from 2012 are still present today.

So when it's suddenly easier to see and poke at these problems... well, you're just plain fucked.

The Proper Response

Again, the only thing under EME's direct control is their servers. There is never any guarantee that any incoming connection is from a legitimate, vanilla TERA client. It could be modded. It could be a bot simulating the protocol. You never know.

But that's okay, because the server is the central authority. No matter what any client sends, the server has to verify and approve it.

TERA's servers do a moderate amount of that already. You cannot cast a skill before its real cooldown is over. You cannot cast a skill faster or slower than what your attack speed and other buffs dictate at the time of casting. You cannot cast a skill if you are under the effects of crowd control abilities. If you try to do any of those illegally (as in, outside of the rules for TERA's combat system), the server just says, "How about no?" and doesn't do anything.

On the flip side, you have the common thread with most everything else that everyone universally disagrees on. The things where the server doesn't properly check client input.

When you click try to buy something from the emporium, the server should be checking that you have the credits and that you have the reputation tier to buy it. It's really not rocket science.

When—not if, but when—someone comes along and finds out that you're not checking it, you can sure bet it's going to be a big problem, especially with something so closely linked to real money.

And then the rewards system gets taken down for fixing, and then brought up again a few days... allegedly with the exploit still possible?

Come on. If you didn't design it well enough for that to be a relatively simple fix, that system shouldn't have gone live in the first place.

Anyone who's ever worked with anything that communicates over the internet can tell you that you never trust the client. In fact, there's a very neat article written about the topic in the context of MMO games already; Gaffer on Games discusses it in light of a massive cheater problem in The Division. This is such a core part of building any networked system that you'd just be shooting yourself if you don't follow it.

Want to know one of the many reasons why Arborean Apparel hasn't gone anywhere? I wasn't happy with the networking. If you connect to an AA server that others are using, all it sends as identification is your own character ID. Anyone can spoof any character ID and forcibly play dressup with someone else's character. That's not cool and that's not acceptable. I don't even trust client connections to my own server.

We saw a sort of similar issue already with TeraDPS. Someone found their way to the admin panel and wiped the database clean. There were no solid, proper security measures taken on any sort of administrative actions. The site was, for the most part, trusting the client too much, and for that, it got bitten hard.

When the server is the central authority, it must make sure it verifies everything. Or at least with a reasonable room for error.

Take speedhacking. It's been around for a very long time. Most people don't abuse it because it's obvious when someone is, but let me tell you how the TERA servers handle it.

If you're within a few meters (somewhere under 10m?) of your last position, the server straight up accepts the new position no matter what.

If you move too far (as in, everything not covered by the above), the server forcibly disconnects you... but then saves the position anyway. When you log back on, you'll be in the new spot like nothing ever happened.

I don't even have words for this. It's been in the game for five whole years and it's still just as broken as ever.

You don't even need the proxy for this. Cheat Engine works fine, so it's not like it's suddenly become a problem now. People are just noticing it more often because it's a side effect of abusing improper desync handling in skill predictors.

How to Actually Correct These

I can't speak too much for the server side. I don't have their code and I can't analyze their design and architecture. It may not be trivial for them to fix quickly, and that's totally fair. In a game as big as TERA, I personally don't expect anyone to have a fix done in a day, or even a week or month. But five years is pretty inexcusable for a lot of the glaring issues TERA has had since launch.

If your server or service can't accurately verify things that are critical or exploitable, then you messed up your design and need to fix that. There is no negotiating this. You can't say, "But the client shouldn't be doing that anyway." Someone will find a way. You should never be allowing the possibility in the first place. Like I said, it may not be the easiest to do, but you should be doing it anyway. Safety and security is never guaranteed.

We can, however, definitely talk about clientside approaches, which is another problem that gets overlooked. It's much easier said than done to fix issues when you also have to account for user experience. People point to the skill predictors all the time and say that if we can do it in a few weeks or months, then surely BHS can do it too.

They're not necessarily wrong, but they're missing one critical component. Are our skill predictors in a state that we would release as part of the actual game? I don't know about Bern and Pinkie, but this is a pretty big "no" from me.

I'm not saying they're bad. Just that they don't meet my standards for what I'd expect from a game that has thousands of players on a daily basis. Some people find ways to abuse them, like forcing desyncs. Even getting innocent desyncs never feels good while you're stuck watching yourself shooting Rapid Fire blanks.

The more you start predicting actions on the client, the bigger chance you have of things getting horribly desynchronized. This is not trivial to fix at all, and I myself would find it fair if BHS can't fix it properly in a year let alone even bother attempting that sort of stuff after 5 years of being live plus whatever development time before that.

As one of many people who worked on these skill predictors, here is the main issue that we face. There are only two things we can ever do with an attack animation: start one, and end one.

If we had some way to adjust where in a skill animation we're at while in the middle of it, it might look a little jank, but on the proxy side we could better update things so the player has a better idea of what they're really doing at the moment, aside from delay due to latency. As it is now, it's pretty easy to just say, "Well, we messed up and properly recovering is hard if not impossible, so let's just do whatever." Which, in my case, is nothing at all.

But what we probably want is a much better, official way to handle it, and that's something that's not so easy to answer.

On one side, you can just make the server step its foot down. "This is where I say you are and what you're doing, no questions." So the client says, "All right, player, here is where we are and what we're doing because the server said so. Doesn't matter what we thought before." Teleport the player to the new spot and animate whatever they're supposed to be doing, no matter what.

It's the easiest solution, but it provides the opposite of a good user experience. No player likes seeing that.

Instead, the client needs to get updates from the server when either side thinks there might be desync, and smoothly animate things to the correct places. If a legitimate client hears from the server that it's several meters off from where it really is, it should gently nudge the player back to the proper position over the course of a half a second or more. For very small discrepancies, it might not even matter, and for the larger ones, it's less jarring than suddenly teleporting 10 meters to the side.

For clients that are way off? Trying to teleport across the entire zone? The disconnect is fine, but please don't save the new position. Does this really need to be said? If you want to play nicer and give the benefit of doubt, just throw up a load screen and drop the player back where they really should be.

But the very fact that they have to account for these sorts of issues of not ruining user experience means it's not so easy to just magically fix these problems either.

Either way, that's my take on the prediction stuff. I'm not a big game developer. I've never had to make a game that does clientside prediction. I'm far from an expert. But as someone who has casually worked on that stuff in my spare time, that's the approach I'd try taking if I have the resources.

The root of the issues should be clear by now. Can tera-proxy and skill predictors allow people to exploit desync? Yes. Does disallowing their use and banning the developers stop people from exploiting desyncs with or without them? Not even close.

Again, I'm not saying that the proxy and predictors are suddenly okay to use. If you think they're bad, period, I'm not here to argue that. If you think it's totally justified to ban me for releasing a tool that more easily allows people to do bad things, I'm not here to argue about that either. The main point I'm making here is that the actions being taken against them are doing nothing to solve the problems being faced.

If people are cheating, then your primary concern should be to fix the exploits.

When you're in charge of the server, the sole central authority, and you accidentally authorize malicious or illicit actions and then whine that people are doing them instead of actually fixing them, it makes no sense and doesn't solve anything.

The Community

The server and client code aren't the only things that need addressing.

The (NA) community is very divided on the topic, and to me, that's very understandable. What some people consider to be cheating is being conflated with the ambiguity of "third party tools". There's a spectrum of what players believe to be acceptable, and you can find at least one person at every level. Here's a list of pretty much every viewpoint I can think of on every one of these topics to get an idea of just how hard it is to cleanly draw the line.

Note: Every line in this section has an accompanying explanation of how some people feel about it. The phrasing does not mean I personally share or endorse the sentiment.

Game File Modifications

Everything under here is not formally allowed by EME, and discussion of them is not allowed anywhere on official sites.

No edits. You run the game as-is, and any other configuration should not be allowed.

.ini edits. For the most part, these are tweaks that are harmless to the game, but do make it run significantly better for many users. Most people think this is okay.

.gpk edits. We don't get stuff that other regions get, or sometimes people want to fix glaring issues like awful UI. Let's patch that right up without directly interfering with normal gameplay. Lots of people do this too, and they are very happy about their Japanese Elin voice packs and Alice in Wonderland dresses et cetera.

Macros

All human input. No macros, period.

Non-combat macros with presence. EME used to say this was acceptable. You can log onto a priest and have a macro that presses your buffs for you, and it's acceptable as long as you're actually there at your system watching the game.

Non-combat macros without presence. Is it fair to leave that buffbot on 24/7 and never look at the game? How about if you extend it to things like gathering? Why would gathering be okay when Felicity is essentially the same thing?

In-combat macros. I'm not sure if EME ever explicitly allowed this, but it is an important distinction to make. Now you gain a direct gameplay advantage, like with shield barrage block cancels. Most people have the line here or earlier, believing that these should not be allowed.

DPS Meters

Vanilla. No packet sniffers. If you want to know how well you're doing, you either figure it out through entirely in-game means, or you just have to live without knowing. Lots of people think that DPS meters make people do awful things.

If you're not being a jerk. This is how another significant population feels about meters. It's totally acceptable to want to know, objectively, how you (and others) are performing so you can become a better player. Just don't rag on people for not being 100% perfect.

Completely fair game. You're not modifying the client, and everyone has a right to know how everyone is performing, so it's okay. "If that IMS valk is doing under 100k/s, they deserve to know that they're trash."

Proxying / "Injecting"

Vanilla. Once you start being able to directly interfere with the game, it's a no-go.

Benign quality of life modules. Auto Vanguard, cutscene skipper, chat timestamps, disabling AFK logout timer. These don't give any direct gameplay advantages, or you were already going to do them with a few clicks or keypresses that are tedium rather than indicative of skill.

Simple quality of life modules. battle-notify allows full customization of alerts for in-game events, like ShinraMeter already does with things like boss enrages or Hurricane applications. Auto Nostrum applies after accepting a rez. Modules for enrage announcements and timers in /n. These sorts of modules do provide a combat advantage to some degree, but these add nothing new to what's already on the screen, or anyone committed enough could have already run timers or mentally timed it themselves.

Macro-equivalent. You could be macroing your buffbot skills. You could also do it just as well with a proxy module. If the end result is the same, shouldn't they be treated the same?

Skill predictors. Some people believe that these provide a universal advantage since even a 10 ms player can shave off that 10 ms, and that easily categorizes it as a cheat. Others feel that the 10 ms difference, plus the cost of massive desyncs, is nothing close to a noticeable advantage, while also helping 100+ ms players have fun again.

Clientside changes only. Some believe things like playing dressup should be fine if nobody else can see it. If people can preview those sorts of changes, they might even be more likely to spend money and buy cosmetics so other people can start seeing it too. More money is good.

Everything else. Unless I'm missing any remaining categories of scripts, the rest fall under what is pretty universally regarded as cheating or exploiting.

Talking to the Community

For every choice in every category listed above, I can guarantee you can find someone who would draw the line there.

There is no easy answer, even moreso when it's so hard to enforce.

The main part where EME messed up is having some of these fly under the radar, turning a blind eye to some of these for five whole years, and all of a sudden overnight attempting to blacklist everything that isn't the vanilla game—not only that, but denouncing everything else. Any possible reason for using anything more than the vanilla game is automatically and unconditionally irrelevant because it's against the Terms of Service. Are you an archer with 200 ms ping? Tough luck. Learn and play with an objectively inferior rotation. Maybe reroll another class.

People not only want clear, open, honest, and transparent communication. They also want to be acknowledged for where they came from. Stating that the answer to any "is it okay?" modding question is a hard "No, it's not." is a big slap to anyone innocently using .ini mods to get more than 5 fps in dungeons. Saying that "an honest player who appreciates tackling the challenges TERA has to offer" is probably "not a big fan of injection" spits on the players for whom ping reduction modules are the very thing that allows them to appreciate tackling the challenges TERA has to offer because the game grows increasingly unplayable the higher you go from 50 ms—which some people already think is unplayable.

Look. I'm not saying any of these modifications are okay. I'm not trying to say my proxy is fine and I should be free of charges. But right now, the community is probably the most divided it's ever been, and that seriously needs some addressing.

My take on it? I'm no Community Manager, so I can't say my approach will even work, but as a player, I have a pretty good idea of what I want.

EME needs to do some research and actually get in touch with its playerbase. They need to realize that, yeah, it actually really super duper sucks to play with high ping so it's understandable that people would want to try fixing that. It's still not acceptable, but they get it. Unfortunately, they can't enforce anything about it very well, so their next course of action is to put their foot down and start being serious about it.

They also need to realize that a significant portion of the hardcore endgame PvE community is aware of, if not already using, this stuff. They get to push the limits. That's what's fun for them. DPS meters let them objectively gauge it, and skill predictors make an effort to even the playing field. You can bet these guys are big fans of injection—not necessarily all of it, but at least QoL modules are all the more worth it for people that play for twelve hours a day. Some players, like myself, have spent this much time with the game and been subscribed or had elite for most, if not all, of our time here, only to land tier 4 in a "loyalty" system, so we don't also need disparaging comments on how your hardcore endgame community just isn't "honest".

Then, clearly delineate everything that is and isn't acceptable, with explanations. No, .ini mods aren't officially acceptable because we know some users do things like showing hitboxes and we don't think that's okay so if we see it in screenshots we will take action, but if we see FoV edits we might leave those be. No, .gpk mods aren't acceptable because we do not have the licenses for those things you're importing from Korea and we can't allow that so if we see it in screenshots or videos we will remove them from our forums. DPS meters are never acceptable, et cetera.

Then address how you plan to approach the issues. If meters are officially gone, will we at least be getting crusade leaderboards back to replace them for the part of the community that does parse runs? If we can't pretend to play from Chicago, will there be anything done to improve the game in that regard?

I mean, we all know the answer to these questions. No, and no. EME has to politely request that BHS implement any of these changes, and there's a high chance BHS won't think it worth their time and effort. (I'd love nothing more than to be proven wrong, though.)

But when all's said and done, I think that there's just one thing that everyone would like to hear. It won't solve all of the problems, and it'll still probably attract some hate, but it'll be a significant first step.

The players want to hear that you understand them, and while these modifications are not and never were allowed, you get it. You totally recognize their intentions, but you can't enforce any of it, and that's why you unfortunately have to start seriously putting the foot down.

Right now, that's the best outcome I can hope for.

The End?

Thank you so much for sticking with me through this very long series. I've probably written enough that I could actually get it published as a book.

I'm sure a lot of people still have questions, and there are some people making controversial statements about me and my work, so I plan to address as much of those as I can in the final post. If you have anything you'd want me to address there, send me a tumblr ask. The questions will be displayed anonymously, unless you mention in your ask that you really want your name on it.

Answers to those, as well as a summary of these big walls of text, will be covered in the final part: the Conclusion (to be published sometime later this week).

Corrections

May 2, 2017, 4:06 am: Changed "I don't even trust my own clients." to "I don't even trust client connections to my own server." I think some people are interpreting this to mean that I've sold my scripts. That's a cute interpretation for your narrative, but sadly not the case. I've never sold any script, period, and here I distinctly intended it to mean "network client", or "incoming socket" if you want to get technical.

#tera #tera-proxy #en masse

19 notes · View notes

komyou · 8 years ago

Text

Part 2: Techno Mumbo Jumbo

This is part of a series describing the history, context, and technical details of the modding we've been doing resulting in this big En Masse / NA TERA commotion:

Introduction. What's all this about?

Part 1: The History – A Timeline. Take a walk down memory lane from 2012 to today.

Part 2: Techno Mumbo Jumbo ⬅ You're here!

Part 3: What Can Be Done?. What do we know, and how should we address it?

Conclusion (not yet published). Obligatory closing thoughts.

Feel free to skip around to the parts that interest you.

Last time we talked about what happened and when, but now we're going to get into the how. How do DPS meters and tera-proxy work? How even do the skill prediction and other things work? I'll be answering that in this installment.

First, a disclaimer. This post analyzes in detail the alteration of game behavior. Many places will consider this "discussion of cheats, hacks, or exploits". If you want to post a link to this post anywhere, make sure it follows any submission rules there about this sort of topic.

If it's so sensitive and controversial, why am I still posting it? Because I would argue this post has nothing new to offer to anyone with malicious intentions. Remember, TERA has been out for 5 whole years already, and there's plenty of underground communities dedicated to illicit activities. Plus, the recent storm has only brought even more attention to tera-proxy. The proxy, along with ShinraMeter, CasualMeter, and Alkahest, are all open source. Everyone can look at the code, and anyone with the technical knowledge and ability to do bad things would have already figured out what we're doing by now.

First, we'll talk about how TERA's networking works normally. Then we'll look at how we use that knowledge in a packet sniffer (such as Shinra), and then we'll extend that idea onto tera-proxy. Now, tera-proxy does nothing on its own; everything it does comes from plugins, or "modules". So to round it all off, we'll talk about how tera-proxy modules work at the end.

If you'd like, you can skip all the way down to the tera-proxy sections. Everything above is not strictly necessary for those, so that's going to cut your reading time by a decent chunk.

Throughout this post, I'll be adding a number of charts and diagrams. Some may be helpful, and some may not. If you're not sure how to read a diagram, don't fret too much. They mostly reiterate what I've already written about, so you're not missing anything if you just don't get it.

How TERA Talks to the Server

When you click the big "PLAY" button on the launcher, or when you back out to server select, TERA sends a request to a specific website to get a list of game servers. From there, it can do things like log you into the last server you played on based on what EME's account server said it was, or it can show you the server list.

Either way, once a game server is selected, the TERA client tries to connect to it. I'm going to assume the vast majority of people are familiar enough with the concept of "server" and "client", but here's a diagram anyway, which will serve as a convenient reference point for the various approaches we'll be taking later on.

In particular, we'll be zooming in on exactly what's going on in the "Client" and "Server" bubbles here, and then we'll be taking a step back to look at how other programs fit into this.

After the client establishes a connection to the server, they do something that's called a handshake; the client and server talk to each other to make sure that this is indeed for a TERA connection and they are ready, and then they share their encryption keys.

I assume most people are at least familiar with the concept of encryption, but we're going to want to define a few more terms just to be sure we know what's going on. Remember, this is going to be a very simplified explanation, so don't expect it to be 200% accurate. Just enough to help everyone understand the concepts.

Let's choose a very simple example. Let's say you want to encrypt a message like "hello". There's lots of ways (infinite, actually) to go about doing that, so we're going to need to pick a specific encryption algorithm. Let's make one up right now.

Start with a number n—for instance, 2. Then, for each letter in the original message, you turn it into the nth letter after it, and then increase n by 1 and go to the next letter. So, in this example:

You see "h". Since n = 2, you turn it into "j" (h → i → j) and make n = 3.

You see "e". Since n = 3, you turn it into "h" (e → f → g → h) and make n = 4.

... and so on, until you end up with "jhpqu".

You know how we picked the starting number of 2? That's called the key. You can sort of think of it like a password; if you know the correct key, you'll know how to correctly decrypt the result, but if it's wrong, all you get is meaningless data.

But the starting number wasn't the only thing that mattered. Our encryption also had to keep track of what n was at each step of the way. That's the encryption state. This isn't something you're supposed to modify and play around with yourself, but it is important. For "hello" with a key of 2, the last three letters came out to "pqu". If we skip the "e" and try "hllo", then the last three letters become "opt". Even though we started with the same letters, we got a very different result because we had a different encryption state when we reached them.

So, with the definitions of encryption key and encryption state out of the way, we can get back to how the client and server talk to each other.

The two establish a connection and send their their encryption keys to each other. (In this case, the encryption algorithm requires two keys.) After that, they both hold a copy of their encryption states: one state for themselves, and one state for the other. Why two states? Because this is a "duplex" connection—data goes both ways. The client can send stuff to the server, and the server can send stuff to the client, and both of those directions needs its own encryption.

The rest of the connection is then encrypted. So it's important that both the server and client are tracking the correct encryption states. As they process incoming data, they have to decrypt it, which updates the incoming encryption state. As they send outgoing data, they have to encrypt it, which updates the outgoing encryption state. If either side gets either state wrong, you end up with garbage values and eventually a disconnect.

In fact, if anyone tries to add, modify, or delete a single thing in the encrypted data as it gets passed between the client and server, the decryption will go into a wrong state and you'll never be able to recover. You'll read garbage values until you realize something isn't going right. TERA chalks it up to a disconnect and boots you right back to server select.

To the casual eye, it might sound pretty fragile, so why don't you disconnect very often at all? How is it handling when you receive data in the wrong order, or have packet loss and drop packets like the bass at an EDM concert? Well, the short answer is that you can't. TERA's network connections use something called TCP. Here's a quick summary: TCP is built to be reliable, error-free, and in-order. If you receive something, but you know there was something sent before it and you never got that, you have to ask the other side to resend the previous message and wait for it before you can continue.

TCP means that as far as TERA is concerned, packets are never received out of order, packets are always correct, and you can never miss one (unless you get totally disconnected, in which case you obviously miss all). That also explains some lag spikes. If you have packet loss, the client has to play catch up, and it can't go forward while it eagerly awaits retransmission of every single packet that it missed. All the server can do is resend those and everything else that happened while trying to recover, and for some connections that can snowball hard.

So now we know how data is encrypted, transmitted, and decrypted between a TERA client and server, but what exactly is being transmitted? To answer that, we'll need to talk about protocols.

A protocol, in terms of network communications, is a specific format, or a set of rules, for how to transmit information. We know one already, actually; TCP is a very heavily used protocol that dictates how tons of packets are being sent over the internet on a daily basis. In fact, you're most likely viewing this post thanks to a TCP connection. A TCP packet can hold any arbitrary data, but for our analysis on TERA networking, it holds TERA's network data. That data follows BHS's own custom protocol.

As I go into the specifics of the custom protocol, let's take a minute to think about why we even need a protocol. As humans, natural language is easy for us to use and process. "Move my character one step forward" is very clear to an English speaker, but computers aren't humans, and while it's a bit of a cliche to say that computers "speak in ones and zeroes", the idea of it should be enough to show why an English sentence just won't cut it. We need to represent that same idea in a way that's compact and efficient for a computer to process.

Under this custom protocol, we can construct what I personally call a "message" (think "sentence"). Each message starts with two numbers. The first number says how long the message is (as in, "how many letters?"), and the second is an opcode—short for "operation code", which defines what operation to perform. Like taking a step forward. Any additional data will depend on the opcode. For player movement, you'll want to send details like your speed and position in game, so for the message that uses the player movement opcode, there's a very specific order and format in which to put information like your in game position and other things like movement type.

So, if that's how we can construct a valid message, then the game's protocol is comprised of the rules for what opcodes mean and how to format and order additional data and all of that stuff.

Notice that I switched from "packet" to "message", even though you probably only ever heard about packets. Semantically speaking, they're different; you may want to send two actions happening at the same time, in which case you can just bunch them together under the same TCP packet. Other times, a message is just too large and you don't want to make the packet too big, so you have to split it across multiple TCP packets. But for the most part you'll see people just calling them "packets" because it's a lot less ambiguous than "message", which could also refer to things like chat messages or system messages. So from now on, I will likely use the two terms interchangeably.

Now you know pretty much all there is to know on how TERA's network communication works. One side, client or server, constructs a message following the rules set by TERA's protocol, then it gets encrypted and sent over, where it's decrypted and read according to the protocol.

Packet Sniffers and DPS Meters

The first type of publicly released mods that wanted to look at this network data used a method called packet sniffing. A packet sniffer is able to read (and only read) the data that's being transmitted over a network, and often has tools to help analyze the packets.

So we set up a packet sniffer to read the data over your net connection, but remember: TCP is a standardized and widely known protocol while BHS's TERA protocol is custom made, so our packet sniffer can only help us as far as TCP. Even then, we're seeing much more than what the server and client see. We're seeing all the other stuff that happens from TCP: packets being resent and packets being received out of order. So we have to do all this work of figuring out how to recombine the packets that we're seeing so that we can most accurately reconstruct what the server and client are sending and receiving. Sometimes, rarely, you get it wrong, and everything breaks because our encryption and decryption depend on their state, and if we mess up reconstructing a message, we end up with an incorrect encryption state. (If your meter sometimes occasionally breaks, this may be why.)

But let's assume that, as is the case the vast majority of the time, we got it right. We're reading the packets properly—but they're still encrypted! Well, that's easy to handle. We know how the handshake looks, and how the server and client exchange encryption keys, so we can just recreate the same encryption states and we're ready to decrypt.

We've rebuilt the packets being sent and we've decrypted their data, so now we're down to BHS's custom protocol.

As mentioned, each message under this protocol always begins with the message length and the opcode. That's convenient for us, because if we see an opcode that we don't care about, we can totally skip ahead to the next message. Unfortunately, opcodes generally change on every major TERA patch, and that's why you need to update meter and proxy. (A "major patch" is one where the first number of the patch version goes up; check the version list on the right side of the EME patch notes for an example.) Luckily, we can actually very easily call a function in the TERA client to convert an opcode to a readable name like C_PLAYER_LOCATION.

When we do encounter an opcode we recognize and care about, we need to figure out how to read the rest of the message. Say we found a message with the opcode corresponding to C_PLAYER_LOCATION and we want to use the info from it, but we don't know how that info was stored. That's where more work comes in, because we haven't yet figured out a way to automatically dump this from the client.

If we don't know how the rest of C_PLAYER_LOCATION is formatted, we look at the data to try to see what they correspond to. Don't forget that we're working with computers, so at the end of the day, everything is just numbers. "Looks like this number is a 1 when you start running, and a 2 when you stop, and a 3 when you jump. This number looks like it could correspond to a letter of the alphabet, so we can try converting it to text and seeing what it says." And so on.

This is a tedious process known as reverse engineering.

DPS meters are only interested in a very specific set of opcodes, and since both Shinra and CasualMeter are in popular use (and especially since one developer works closely with both), they usually just need to figure out how those handful of messages look and then they're done dealing with TERA's networking without too much hassle.

tera-proxy

But we had to go a step further. For the sake of completeness, let's first talk about what exactly a proxy server is. You can imagine the connection between a client and server as being like two people having a chat with each other. A packet sniffer will be sitting off to the side, letting the client and server have a direct chat with each other while it's up to the packet sniffer to eavesdrop and make as much sense of the conversation as they can or need, while being unable to interfere or even interact at all with the conversation.

A proxy server, the middleman, sits between them. The client must talk to the proxy and receive replies from the proxy. The server must also talk to the proxy and receive replies from the proxy. The client and server no longer interact directly with each other.

This can be used in a number of ways. Maybe the middleman is a translator, and the client and server need something like that between them in order to understand each other. Maybe the client and server are just really mad at each other because the server is based in Japan and only wants to serve clients that live in Japan, in which case the proxy can act as a Japanese client on behalf of the real client. Even if the real client is completely fluent in Japanese, everything the client sends must still go through the proxy, likely completely unmodified, because the real client is not living in Japan and those are the server's rules.

You should have a pretty good idea by now how tera-proxy fits into this. Actually, there's two proxy servers; remember how the client has to fetch the server list so it knows where to make a connection? The first thing we do here is proxy the server list, so when the client requests it, that request goes to us first. We ask the real server list what's up, and when we get the reply, we can change whatever it says before handing it back to the client. Like adding an extra entry called "Mount Tiltrannas" that points to a game proxy server.

So the game client got this new server list and now it wants to connect to this special server we've added. We know the connection handshake, and we know that the real client and real server are going to exchange encryption keys which we can save and forward very easily.

Since the proxy talks to both the client and server, it handles two connections. Each connection has two ways data can flow (towards or away from the proxy), so that means we now have four encryption states to take care of: the two between client ↔ proxy and the two between proxy ↔ server.

But if we do our job right, both sides are oblivious that there's even anyone in between them, because as far as they're concerned, the data they're receiving is getting decrypted properly. Everyone's happy.

Unlike packet sniffers, we're not just eavesdropping anymore. This data from both sides is being sent straight to the proxy and only the proxy. It's our job to decrypt according to one side, re-encrypt for the other side, and then send it to its intended destination. Since we can do that, we can also do things like modify the data, adding and removing things at our leisure because now we have control of the encryption states.

Besides everything already mentioned, there's one more thing that distinctly set tera-proxy apart from DPS meters.

tera-proxy doesn't have a specific focus like meters have with DPS. We no longer care only about meter-related packets anymore. If someone finds any opcode at all that looks interesting to them, they can try to figure out how its packet data looks, write up a definition for it, and (hopefully) add it to tera-proxy. It might get used by others too, and it can also be verified or updated by other people because that's the beauty of open source. In fact, as of the time of writing this, we have well over 400 definitions mapped out thanks to a lot of people. We've been hard at work.

That means we know how to handle a lot more opcodes, but what does that mean for our proxy?

tera-proxy Modules

By itself, tera-proxy does nothing substantial. It overrides the server list to point to itself, it decrypts and encrypts the data, and it's capable of reading and writing messages with the knowledge we've given it of TERA's protocol. But by default, it has no reason to read or write anything.

That's what modules are for. Each module essentially says, "Hey, when you see any of these particular packets, I want you to tell me the data it contains and then I want you to do these things." This is called a "hook". If you hook C_PLAYER_LOCATION, then when tera-proxy sees a C_PLAYER_LOCATION packet, it reads it and converts it to a format that's much simpler for a programmer to use.

Again, let's look at some old diagrams:

We start with this 0012 0123 0256 0390 4800 0002 message, the section outlined in the middle. After decrypting a packet, that's what tera-proxy might see, so it checks what opcode 0123 is and finds out that it's C_PLAYER_LOCATION. It knows that you registered a hook for that packet, so it first tries to make sense of the packet. Checking the definition we've written for that packet, we have names for each field like before:

So now it runs your hook, and in your hook, you can retrieve (and modify) the values by messing with a variable called position or moveType.

You can also silence the packet to prevent the intended recipient from ever seeing it, and you can also construct your own packets.

And that's pretty much all a module can do to a TERA connection.

Maybe you've noticed it by now, or maybe you haven't.

It took well over 3,000 words and plenty of graphs and charts and diagrams to explain how TERA's whole networking system works. Then we described tera-proxy modules in just a handful of sentences.

There's a reason for that—and it's not because you learned everything you needed to know.

The real reason is that you didn't need to know in the first place. That's the proxy's blessing and curse.

In our example, we wanted to get a C_PLAYER_LOCATION and modify it. We can do it in three lines of code. We never had to bother ourselves with the details of the encryption, the protocol, the proxying, or any of that.

There's more. Most languages require the use of a compiler; you write code, give it to the compiler, and you get back "machine code" that you can run. For tera-proxy, I use a language called JavaScript. It's an "interpreted language", which means it doesn't need compiling for someone to run it. Actually, you've probably already been using JavaScript quite a lot; it's the language used to make webpages interactable, and you'd be hard-pressed to find a single website that doesn't use a single line of JavaScript.

Since we don't need to compile anything, anyone can just open up a text editor, write some module code in JavaScript, run the proxy, and it just works.

Does that sound very powerful? It should. Can you make really cool stuff with it? You bet. Can you also cheat really hard with it?

Unfortunately, yeah. But that discussion is reserved for part 3.

Most modules will be rather self-explanatory (when a cutscene happens, hide it from the client and instantly send the skip packet; when a Vanguard is completed, instantly send the accept rewards packet; etc). But a lot of people are up in arms about the skill prediction stuff, so it's time to get into a little more detail about that.

Skill Prediction

First, let's pick some arbitrary ping. Say 200 ms. That means you sent a "ping" packet to the server at some time, and when you received a "pong" packet back from the server, it had taken 200 ms. For now, it will be sufficient to assume that this means the time it takes for a packet to reach the other side was 100 ms, so when we sent something and got something back, it took twice that time, or 200 ms.

There's only one thing in TERA that happens on your screen as soon as you press a button, and that's movement. This is a technique called client-side prediction. It's still going to take 100 ms for the server to receive your new position, and then half of everyone else's ping for them to see it, but at least on your own client, you see when you step forward or back instantaneously. It feels smooth and fluid.

That's just about the only thing client-side predicted as far as we're concerned.

What does that say about casting skills? Well, when you press the button, your client doesn't actually do much of anything yet. It sends a request to do it to the server, who verifies a number of things (Was it off cooldown? Was it on a valid target? Are you not stunned or slept?) and then sends back an acknowledgment along with other information like the attack speed. Only then does your client animate that skill.

Let's ignore chain skills for a moment. We've casted our first skill, but now we want to cast the second. We can't do that while we're already in a skill, so we have to wait for the first skill to finish animating.

But our first skill already took 200 extra ms to start.

And then we have to wait another 200 ms for the server to acknowledge our second skill.

That's what's colloquially known as "ping tax", and you pay it on most everything you do in TERA.

Lockons are extra awful in this department. Entering the lockon pays ping tax, acquiring a lock pays ping tax, and then firing the skill pays ping tax.

And Rapid Fire? All right, let's look at this. Let's say you've got a total of an arbitrarily chosen 120 + 60 = 180 attack speed, which translates to 150% animation speed. Each hit of Rapid Fire chains into the next after somewhere around 300 ms at base speed (I think?), so we can cut that down to 300 ms ÷ 1.5 = 200 ms per hit of RF.

First, a perfect world where latency doesn't exist. We can send each hit of RF as soon as we get the chance, so we can send the seventh and final hit after 6 animation cycles (200 ms). 6 × 200 ms = 1.2 seconds.

But we have 200 ms ping. That means to send the next hit, we first have to wait for the acknowledgement of the previous hit to reach us (200 ms), and then animate it (200 ms). Six cycles of that is 6 × 400 ms = 2.4 seconds.

In fact, it doesn't matter how much attack speed you get. In this case, you'll always take 1.2 seconds longer to reach the seventh hit of RF because you have to pay that ping tax six times no matter what. That's disgusting.

(As an aside, for someone living very close to the servers with the luxury of about 30 ms ping, their RF time comes out to 6 × 230 ms = 1.38 seconds. That's only 0.18 seconds (180 ms) more than the perfect case scenario—less than the time we'd be waiting for our very first hit at 200 ms ping. If your ping is already very low, there's very little to gain, especially because you have to balance those millisecond bonuses with just how much desyncing you're going to be dealing with. I explain that in more detail below.)

So how do we fix it?

Well, we can't get rid of latency. It'll always take 100 ms for the server to receive anything you send.

But what we can do is try our best to eliminate the ping tax on the client. When you try to cast a skill, skill predictors are really just tera-proxy modules that predict the acknowledgment reply from the server and send it to the client.

You say you want to cast Rapid Fire? The predictor instantly replies that you did it. That's a whopping 0 ms on the ping tax. No need to wait for the server when you can just assume it was successful.

Sometimes, we get it wrong. There's a lot of things the client doesn't check before sending the server a request to cast a skill, so skill predictors have to be checking if you're stunned or slept or knocked down, if you have appropriate prerequisite buffs to cast something, if you have glyphs or buffs that alter a specific skill's attack speed... If we guess any of those wrong, you get desynced. Your client might still animate it, but it might be too slow, or too fast, or it wasn't actually cast at all. Damage doesn't happen, but you're still stuck in animation.

The three main skill predictors, each written by a now banned developer, handles things very differently. Mine had lockons for lockons and fast-fire to focus solely on three skills (Rapid Fire, Burning Heart, and Burst Fire), both of which made no attempt to resolve desyncs. Both Bern and Pinkie aimed to generalize the approach for all skills in the game, but I can't discuss how they went about resolving desyncs and similar issues due to not personally examining any of their code.

But that's pretty much it. Your client asks to cast a skill and the proxy instantly replies with an acknowledgment so it doesn't have to wait. We're not actually casting anything if we couldn't already do it. We're just tricking the client into thinking it's doing it instead of waiting for the server to say it was valid.

That covers all the technical parts of how everything is working, from TERA's network connection to DPS meters and from the proxy to its modules.

I hope it wasn't too difficult to get through, but there's a ton of information to cover, and a lot of details that typically get glossed over because most people don't care to know much more beyond "tera-proxy is hacks and skill prediction is lagging servers".

Armed with all this knowledge, the next thing we'll be looking at is what EME and BHS have done to try to address these things and what they can do about it in part 3: What Can Be Done?.

#tera #tera-proxy #en masse

37 notes · View notes

komyou · 8 years ago

Note

Will you be quitting Tera? And by that I mean quit making mods and what not?

I think it’s too early for me to decide this right now. I’m going to give it a little while and see where things go. There’s still plenty of people being vocal.

1 note · View note

komyou · 8 years ago

Text

Part 1: The History – A Timeline

This is part of a series describing the history, context, and technical details of the modding we've been doing resulting in this big En Masse / NA TERA commotion:

Introduction. What's all this about?

Part 1: The History – A Timeline ⬅ You're here!

Part 2: Techno Mumbo Jumbo. The technical details of how and why these mods work.

Part 3: What Can Be Done?. What do we know, and how should we address it?

Conclusion (not yet published). Obligatory closing thoughts.

Feel free to skip around to the parts that interest you.

The Setup

First, let's get our bearings. NA and EU TERA were officially released in May 2012. The level cap was 60 and you needed to purchase a subscription to play.

It wasn't long for people to start thinking about stuff like DPS meters. In July 2012, just two months later, we started seeing a rudimentary meter (forum post, YouTube link). Back then, it just looked at your screen and tried to read numbers from your combat log. Its main issue was that it needed pixel perfect samples for each digit, otherwise it would silently skip it. Suddenly, a 1 mil crit became 100k and you had no good way to see when this was happening.

Just for reference, NA TERA went F2P in February 2013.

May 2013, a full year after TERA's release, someone put up a GitHub repository for a TERA server emulator. There was a whole lot going on with reverse engineering and emulating TERA servers, but I wasn't a part of any of the communities doing that at the time, so I have no reference points aside from this repository going up.

This was pretty big, in my opinion, for a number of reasons. First, it's a public repository on GitHub. That means it's easy to find and open source. Anyone can see that code if they go looking for it. Second, it's written in a language which I find to be fairly accessible, and the author did a pretty good job of separating things nicely.

I mention the meter and the server emulator because these became the foundation for my own project. I was unhappy with that one meter. I thought we could do better. I started trying to read the combat log off the screen, but quickly decided that I wanted something much more accurate and reliable. That led me to two options: reading TERA memory, or reading TERA's network data.

Well, I've never done anything with memory reading, but we just so happen to have this open source server emulator. If we can emulate a TERA server, surely that means we know how to read what a TERA client is sending. And if we know how the server is making whatever it's sending back, we should be able to figure out how to read it too. Hmm...

Meishu Enters the Scene

Next thing you know, by January 2014 during the Wonderholme patch, I had written a TERA packet sniffer and made my own take on a DPS parser. Granted, this one was mostly just for show, but I did post about it on the forums. Not a release, just a showcase. It got plenty of praise before being hidden from the public by moderators—simply because I discussed my methods (i.e., packet sniffing).

In February 2014, I published the framework. I didn't post about it anywhere, and it's not the DPS parser itself; it's only the thing that made the DPS parser work. As mentioned, it was just a packet sniffer (the technical details are in the next section, part 2). However, I purposely omitted one major component from my repository: the cryptography code... which just so conveniently happened to exist in another project using the same programming language I was using.

A few days later, I wrote another forum post, this time analyzing character sliders. Luckily, this thread got to stay—but you can probably tell by now just how long I'd been working with this sort of stuff.

If I recall, I didn't end up doing a whole lot with the framework itself. I used that programming language since that's what the server emulator used and I could straight up copy the cryptography code, but that language wasn't my forte. Ultimately, I was mostly just having it log the decrypted packets so I could have another program analyze it.

So I thought, what the heck, this packet sniffer code is taking a lot of effort to maintain when I could just skip the middleman and do it way simpler in the language I'm very familiar with. And maybe, just maybe, instead of being stuck with only reading stuff, we might be able to modify it too. Hmmmm...

It didn't take me long. I stopped updating that GitHub project the next month, and in April 2014 I had stated that I was working on a better version.

By the end of April I was already playing with "headless clients"—bots. For instance, a priest on MT had a macro to cycle buffs while AFK in one of the alliance halls, but then got a 3-day suspension. My response? What if it only buffed when you waved?

Besides that, the only other thing I did with it was make a chat bot that would relay everything between guild chat and IRC, which was just amazing at bringing together my guild. And what with being on the roleplay server, friends and I had this thing we'd sometimes do where we'd sit around the Velika tavern and watch the RP while idling. We called it "tavern scumming". I was already having fun playing with this guild/IRC bot and figuring out this chat stuff, so I thought, why not see if we can watch tavern RP at any time? What if the bot relayed RP to a website? And so I launched tavern-scumming.

It didn't last long. People caught wind, reported the bot and site for harrassment, and had En Masse look into it. The bot account, as well as all of my accounts, got slapped with a permaban.

I appealed. I appealed it so hard. Look—I'm a very loyal player. I have the Mark I titles for MC and KNXX. I love this game and I just want to make cool stuff. If I wanted to make anything actually malicious, I would have done it long ago. My forum and tumblr posts show the opposite of ill will.

They unbanned me, but not without warnings to be careful. Actually, the only issue was with the RP chat relay, because the GM said, verbatim, "[t]he IRC transcription function is not technically against the Terms of Service, but you are responsible for the transcribed chat".

Wow, how times have changed.

After the Ban

I went somewhat quiet for a while. The new project was gamechanging. I was thinking of all sorts of stuff and seeing what I could do with it, but I also didn't want to land myself in another sticky situation like that.

A couple of months later, on June 2014, this tumblr was made along with its first post. The stuff I was showing wasn't as practical anymore, but it was still kind of neat to see.

And by September 2014 I was just fooling around at that point. Besides, that was during the Rift's Edge patch anyway, so fooling around was about all there was to do anyway.

In October 2014 I started showing off Arborean Apparel. This was super neat to make and super neat to show... but did I really want to release it? tera-proxy is ridiculously powerful on its own, and then we want to let people play dressup for free? Well, I mostly rushed to make it look pretty to show off, but I really wasn't happy with the actual code and how I was building the data, so I figured I had time to think it over.

A few days later, tera-proxy was quietly published. Again, nothing else was there. No scripts or modules or anything like that to do anything. It was just a framework—with cryptography included this time. If you ran that code on its own, nothing in your game would've changed.

Nobody really noticed, and that was perfectly fine with me.

A lot of time passed in more or less silence. I was occasionally posting neat stuff on this blog and doing a few changes here and there with tera-proxy, while only few people noticed me (who were already from the server emulation community, like the person who wrote the emulator that got me started). Most importantly, I was just having fun playing the game.

One year passed. We saw Fate of Arun, Crusades, Sky Castles, Dreadspire, Gunner, Timescape, and Stepstone Isle.

On October 2015, Gothos published their attempt at a packet sniffing DPS meter—which, by the way, just so happens to use the same encryption code that I used. And from there, both ShinraMeter and CasualMeter showed up on GitHub.

Soon after, I remade my guild chat bot, except now the hot thing was Discord. So the Discord relay got published in November 2015.

Several months later, in May 2016, we saw TeraDPS.io. Then, I believe sometime in September 2016, Moongourd took its place.

On October 2016, just before the sorc revamp, I wrote fast-fire at the request of overseas friends who play archer, ninja, and gunner.

It went from friend to friend to very competitive friend, and with DPS leaderboards now being posted online, that was starting to mean a whole lot.

Some people didn't understand what it was doing. Some people got jealous or upset.

So in December 2016, it exploded. Big time.

After It Hit the Fan

Well, I figured, since everyone either knows about or has it now, I might as well start properly supporting it.

Lots of people noticed. Lots of people wanted to not be gimped by ping. Healers and archers hailing from Europe were sending thanks—and even people here at home, where California gets the same ping as the UK.

And people wanted more. No less than four people were trying to extend fast-fire to work with pretty much all skills in general. More quality of life mods popped up too. Work on the proxy spiked in the months following the drama thanks to a number of very helpful people contributing data and suggestions.

Unfortunately, with enough time came the exploits too. And that brings us to today. April 2017.

To my knowledge, someone found a way to buy items in the emporium from a higher tier than is accessible, and also posted a screenshot of having a metric ton of credits (although it was only the clientsided display; these were not valid, spendable credits). In response, EME took down the rewards system in order to fix it.

At the same time, it was announced that the triple drop weekend would not be including Ruinous Manor due to exploits. A lot of people thought this was linked with the rewards shop exploit, but as far as I'm aware, you didn't even need the proxy to exploit RM. You could easily do it with the vanilla game. But as soon as one person mentioned "the injector", all hell broke loose.

Then the bans started happening, alongside a public statement from En Masse with a hard stance against any and all third party tools and modifications. And lots of people are not happy.

The forums are a mess, the TERA stream chat was a mess, meme images and videos on recent events are everywhere, petitions and boycott plans are being made, and the community is divided.

That's what's made me so hesitant to release things all these years. That's why I've been so quiet on this blog and never been able to deliver on release promises. I knew that if this became a big thing, it would also become a big problem.

To know exactly why the proxy can wield so much power, I'll be explaining everything going on behind the scenes in part 2: Techno Mumbo Jumbo.

Corrections:

April 29, 7:28 pm Pacific: Fixed patch eras. Made MC parse run during WH patch, not MC patch. The fooling around / FABULOUS video was from Rift's Edge patch, not WH patch.

7:33 pm: "November 2016" for the Discord relay corrected to "November 2015".

7:35 pm: Clarified the rewards system was taken down for fixes, and added a link to the triple drop news announcement.

#tera #tera-proxy #en masse

23 notes · View notes

komyou · 8 years ago

Text

The fires are still raging, and plenty of people have stated plenty of thoughts in plenty of places. It just so happens that I'm not at home today and I have nothing else to do, so now it's my turn to dump words.

For those unaware, EME has banned me for my work with the proxy. Bernkastel was also banned and did a writeup (see justpaste.it or the original PDF download) with their thoughts on the situation, and I myself don't have much to add there. So, instead of writing yet another wall of text to explain why I agree or disagree with things for reasons everyone's heard hundreds of times by now, I'm going to write something different.

This is going to be a series of posts describing the history, context, and technical details of the modding we've been doing. I am writing this for a general audience, so if you're a casual passerby who just wants to know exactly how things work and why they came to be, this series is for you. I'll be doing my best to introduce topics and concepts in a way that even "halp how do i computer" should be able to follow along.

There's a lot of misinformation being spread around, largely because people just don't know what's really going on beyond a few technical buzzwords. And I don't expect them to know, either. So this series is for the people that care to know and want to become a little more educated on the topics involved.

I will not be trying to make any arguments about right or wrong. I'm not here to ruin anyone's reputation. I simply want to share facts and share information that may not be so easy to casually Google, and put it all into one convenient place for anyone interested in finding out more about this whole drama stuff. As with everything written, you'll be reading things through my own filters and biases, but I will attempt to remain as neutral as I can. Feel free to send me messages with questions, corrections, or suggestions because I'll happy to entertain any discourse on this subject.

However, do note that each part will be very, very, very detailed. If you're not down to read 2,000 word novellas, now's the time to back out.

This series will be published over the next few days (I'd like to do one a day) and is currently planned to be divided into these sections:

Introduction ⬅ You're here!

Part 1: The History – A Timeline. Take a walk down memory lane from 2012 to today.

Part 2: Techno Mumbo Jumbo. The technical details of how and why these mods work.

Part 3: What Can Be Done?. What do we know, and how should we address it?

Conclusion (not yet published). Obligatory closing thoughts.

Feel free to skip around to the parts that interest you.

Next ➡️ Part 1: The History – A Timeline

#tera #tera-proxy #en masse

28 notes · View notes

komyou · 8 years ago

Video

youtube

I’ll probably have a writeup of stuff sometime today. I think everyone has spoken plenty of the topic in general, but we’ve really been missing some technical writers. Might be interesting for people who are genuinely interested in exactly how everything is working.

#tera #tera-proxy #en masse

15 notes · View notes

komyou · 8 years ago

Text

So Valks are a thing now.

Hello again! It’s been too long.

I’ve been mostly quiet here but things are still progressing—by quite a bit, actually. Most of it is unfortunately invisible to everyone who just wants to use mod things, but with the help of a few people whose assistance I am very thankful for, we’ve built up a pretty solid base. And also set up a Discord.

And a website. And I updated all my documentation. https://meishuu.github.io/tera-proxy/

If you already had some old version, there’s a prepackaged build which you can just download, unzip, and then grab your old tera-proxy/bin/node_modules folder and move it over. (You might also need to move servers.json and other files, depending on what you might have had to change.)

Up next is... a lot of things:

Make a better configuration system so it’s easier for people to set things up correctly for their region / server.

Arborean Apparel is still coming! I’ve been testing a pre-alpha, but most things seem to be working and there’s a few more I’d like to add before an actual public alpha or beta release.

GUI, plus autoupdater, so no more having to mess with downloading and unzipping and moving things every update. Will also auto-update mods from GitHub.

Probably silence again up until the dungeon patch. If you want to get in touch, definitely try Discord.

Until then! 💉

#tera-proxy #baldera #tera #arborean apparel

4 notes · View notes

komyou · 9 years ago

Text

🔥

Or: How a Bunch of Drama Happened and an Alt Gmail Got Hacked in Record Time.

This is the story of how MT and Moongourd went from 0 to 100 in a day. This is a cautionary tale about securing your email accounts. And most of all, this is a story about taking things way too seriously.

This is going to be a very lengthy post giving a chronological account of my experiences, with an update at the end detailing my plans moving forward. Grab some popcorn and strap yourself in for some entertaining drama, because this is gonna take a while.

The story is under the cut, but if all you wanna know is what’s going on with me and my stuff, just skip to the very last section.

Part 1: Moongourd Memes

A couple of months ago, a European archer came to me asking about making Rapid Fire work faster, and for once, I was actually bored enough to give it a try. It worked, and I extended it to Burning Heart and Burst Fire, and then I made a post about it. Cool.

Since then, it’s been passed around to a few people here and there, and we’ve been able to identify and remove some bugs and exploits. There’s still some massive desync problems that I hoped to address sometime, but I just haven’t had the time to get around to it.

tera-proxy has always been an unstable abomination, and despite being around for a number of years, I'm still finding and fixing bugs and crashes. That extends to this new fast-fire mod as well, and that's why I've mostly been telling people to keep this stuff on the down-low. If I publicly release something with a major exploit, that's bad. If there's lots of crashes and bugs, people swarm me for support, and that's also bad.

My biggest priority for a very long time now has been "Baldera", intended to be a GUI to make it easier for people to use and configure tera-proxy while also receiving automatic updates so I don't have to track down everyone using it to tell them which files to change. This is what I was, and still am, waiting for.

Cue my surprise when suddenly this "0 ping RF" suddenly blows up. Big.

Now there was all this talk of "packet injecting" and threats of getting banned from Moongourd. A prepackaged version of tera-proxy + fast-fire was being given out in Global chats.

I had joined the Moongourd Discord a day or two prior, just to see what all the buzz was about, but I stayed completely silent. People were talking about me as if I weren't there at all.

From what I gather, teri was upset about some Amaterasu people using fast-fire, and tensions between both parties escalated this to a full Moongourd ban on all Amaterasu people.

All this drama was pretty amusing to me, so I broke my silence to say only two things, with some irrelevant messages omitted:

"Discussions" grew worse and worse until the Moongourd Discord was engulfed in memes. It was memed so hard that the admins locked it down and prevented anyone from talking.

A few minutes later, without hearing anything from anyone, I was promoted to the "Devs" rank on the Moongourd Discord. Several more minutes later, I discovered that the reason for this was to have a talk about how best to address everything going forward.

I don't have a good answer to share with anyone on that. All there is to do for now is wait and see, so that concludes this chapter.

Part 2: Gmail Hacked

4:43 am: I had had a very fun and eventful day, so sleep was a little difficult. I was still awake when I received a message on my main Gmail account saying there was a new sign-in on my alt Gmail account: Firefox on Windows, IP address 37.35.105.82.

That's weird. I don't use Firefox.

4:46 am: Just to make sure things were okay, I tried to log in on my phone, which was a terrible idea because my phone is crap and the login didn't seem like it would happen anytime soon.

4:48 am: I pulled out my laptop and logged into the alt account. Google didn't give any extra information on the IP address, but searching it revealed that it was from Switzerland, along with a couple of pages about it being the source of some forum spam.

4:50 am: The attacker searched "discord" and went to the homepage, only to be alerted of me changing the password on the Gmail account.

4:53 am: The attacker retaliates by changing the password again, changing the recovery email address, and deleting the security question.

4:55 am: The attacker sent my main Gmail a message titled "F" and containing "ICARUS HAS FOUND YOU" repeated 95 times. Unfortunately I am not very familiar with Deus Ex, otherwise the message would've been cooler.

Immediately afterwards, the attacker went and issued a password reset on the linked Discord account. Luckily, it was just a bot for my guild's Discord, but they were able to log in and change the email address linked with that account.

After doing so, they sent me a friendly greeting:

I immediately kicked the bot from the Discord, but didn't do anything else with it. I wanted to see if they'd send me more messages. (They didn't. Oh well.)

4:58 am: The attacker started signing up for a Dropbox account with the compromised Gmail account. Dropbox sent a verification email.

5:01 am: The attacker issued a password reset on the attached Twitter account and logged into it—again, Firefox on Windows. Too bad this was a Twitter RP account that I last used in 2012. Nothing juicy here, just some mildly embarrassing roleplay.

5:05 am: The attacker Googled "enmasse signup" and went to the account creation page, and then Googled "enmasse support" and went to the support page. I'm guessing they planned to reset my info here too, but I only have three alt accounts made for now defunct guild banks linked with this email so they wouldn't have found anything interesting anyway.

5:08 am: Google sends the compromised account a verification code during my first attempt to get back into the account. Since I couldn't get this code, Google prompted me to prove account ownership by providing the old password before sending a code to the old recovery email. The code didn't work for me. Pretty discouraging for it to say it couldn't verify my identity.

5:13 am: The attacker completes the Dropbox signup, and I try to get back into the Gmail account a second time. The new code succeeds, and I'm greeted with a form to explain my situation to Google in order for them to transfer control of the account back to me.

5:16 am: The attacker issues a password reset on my Newgrounds account. I didn't even remember I had a Newgrounds account. Last used 2010 at the latest, probably. All I did was post a couple of music tracks. Nothing exciting. You can keep this one, buddy, but the email was still marked unread so I don't think you finished resetting it. Not like I remember the real password anyway.

5:25 am: The attacker decides to send my main email account another message titled "HELP", containing "SEEK HELP [my real name]" repeated 175 times. Spooky. I don't like to give out my real name, but I have it in some GitHub files and a few YouTube videos on my main email account, none of which are very difficult to find, so I wasn't scared to see it pop up.

5:32 am: Google finishes manually checking the account takeover and gives me a link to recover it. It's still not a particularly important account to me, but just to secure it anyway, I immediately add a recovery phone number and enable 2-step verification.

After making sure all the security settings are in order, I take a peek at Gmail:

This was an alt Gmail account I used for places I knew I wouldn't use but still required registration, or for places that looked sketch. There were thousands of unread email, newsletters and regular updates and the like, but all of it was gone once I logged back in.

I'm guessing the attacker saw that password change email, panicked, and decided to send those three "BALDERA" emails since they knew the jig was up.

I did some digging around and Google has a form to recover deleted emails, so I filled that out to figure out which accounts were issued password resets. Since then, I've changed passwords and verified there was no other malicious activity on the Gmail and Twitter accounts. I deleted the Dropbox account, which was completely unused, and nothing seems to have happened to the three alt TERA accounts. I'm not even going to bother touching the Newgrounds account, and Discord won't do anything with the compromised account, which is fine because I just made a new one and invited it to my Discord again.

That about wraps up this chapter as well, but it was interesting to experience first-hand what that kind of cyberattack is like. Everything I've detailed above was pieced back together using a log of Google searches and the recovered emails on the compromised account.

As I said before, the Gmail account was intended to be a sort of fodder account, so it had an extremely weak, reused password with minimal extra security. The loss of the Discord account was just about the worst that happened with it, but I don't think there was anything else important on there.

My main accounts use strong passwords and two-factor authentication, and my main Gmail, Dropbox, Discord, etc. remain untouched. I might even move the Discord bot to my main Gmail, just because there's plenty of potential for very bad things if that gets compromised again.

The only thing that eludes me is how the attacker knew of that email address since I couldn't recall posting it anywhere. I could only think of one related place where it was used, and that was to register a test Moongourd account to see what the site was like before I registered with my main account.

While pondering it, I noticed emails about an old GitHub repository I contributed to under the alt email account, and I remembered that meishuu was an alt GitHub account which subsequently became my main account and was linked with my main email instead. So I'm guessing they noticed I had used this alt email on meishuu at some point.

The Gmail password is extremely easy to brute force, although Have I Been Pwned? says the account was in the Nexus Mods breach. I probably used the same password there too. Who knows?

But at the end of the day, the main takeaway is just to practice good computer security. Don't let something like this happen to you on an account that has all your sensitive information on it. Even though there wasn't much lost for me, it was still an eye-opening hour in an otherwise sleepless night. I'm glad I was awake to ward it off before it got too bad.

Aftermath

All of that covers about a 12 hour period where an average day for me turned into a rollercoaster of events. After all this, there's been a brief spike in people messaging me for tera-proxy help. Sorry, but I know absolutely nothing about any of the other regions, so I can't help with EU, TW, etc., and if you don't know how to set up tera-proxy and install rapid-fire by yourself, I'm currently not offering support unless I know you.

The GUI is still next on my plate, and after (or maybe even during) that, I'd like to have a go at my own meter and site. I've had plenty of ideas, both new features to add as well as criticisms with how old and current meters and parse archives do things, so it's just down to a matter of setting aside time to do it.

In the meantime? I just want to play the game with friends and enjoy the rest of my holiday season. Aiming for an Ambush bow and maybe doing HH on my mystic, but CH is a sad place for HH. (Though on that note, if anyone wants to RP, hit me up!)

There's been some talk of what EME/BHS might do in the future, and honestly? I'll deal with that when we get there.

That's all for now. Thanks for everyone's support; I'm sorry it's taking me so long to work on these things, but now we've seen what happens when it goes full public. Just remember to have fun, enjoy the rest of the year, and try not to get too mad about video games.

Peace. 💕

#tera #tera-proxy #baldera #fast-fire #moongourd

18 notes · View notes