michaelok

[michaelok]

17. Reindeer

“I’ll go see the Lapps. They’re our best chance.”

After having it on my list for years, I finally read a David Howarth’s We Die Alone, about a Norwegian commando, Jan Baalsrud, who escaped from the Nazis and by walking, rowing, skiing, and finally by sleigh, made it across the far north Norway, in the Arctic, to finally make it to safety in Sweden. Wikipedia puts it succinctly;

This mission, Operation Martin, was compromised when Baalsrud and his fellow soldiers, seeking a trusted Resistance contact, accidentally made contact with an unaligned civilian shopkeeper, with the same name as their contact, who betrayed them to the Germans.

To set the scene, take a look at this map, from a recent expedition retracing Jan’s exact route:

https://gjeldnes.com/in-the-footstep-of-jan-baalsrud/

Early in his journey, he makes a mistake by trying to charge through a town and outrunning the Germans on skis instead of taking his time and waiting for the all-clear. In doing so, he gets lost in a snowstorm, and nearly dies in an avalanche. He survives, but suffers from frostbite and is unable to walk, making his escape considerably more difficult.

There are many interesting aspects to the story, but I’ll focus on one part I found fascinating: while he was certainly assisted greatly by Norwegian partisans, he never would have made it out without the help of the Sámi people, who, with their sleigh pulled by reindeer, brought Jan to safety.

The Sámi people (also spelled Sami or Saami) are an indigenous Finno-Ugric people inhabiting Sápmi, which today encompasses large northern parts of Norway and Sweden, northern parts of Finland, and the Kola Peninsula within the Murmansk Oblast of Russia.

_ _

The Sámi people of Arctic Europe have lived and worked in an area that stretches over the northern parts of the regions currently known as Norway, Sweden, Finland, and the Russian Kola Peninsula. They have inhabited the northern arctic and sub-arctic regions of Fennoscandia for 3,500 years. Before that they were probably living in the Finno-Ugric homeland.

Source: Wikipedia

In fact, much of the book is about the time he spends waiting for help out in the snow on the side of a mountain, for the Sámi. The Norwegians, afraid of being caught assisting Jan, essentially leave him buried alive in the snow, while arranging for the Sámi to arrive with their herd of reindeer. The plan is for them to take Jan on their way to their summer pastures in Sweden. While the forbidding Arctic winter poses many problems for Jan, including a snowstorm which nearly ends his quest, the snow, providing a winter road for the reindeer, is also his only chance to avoid the Germans and get to Sweden. Sweden, being neutral, would be a safe haven for Jan.

What of the mysterious Sámi, who despite their significance, are relegated to a but a single chapter (17) in the book?

József K.

Perhaps I am interested not only as they are the key helpers in the story but also because my wife’s grandfather, József, a paratrooper in WWII, supposedly made it out of a Siberian prison camp on a sled pulled by reindeer. This is a story passed down by family members and seems almost mythical, especially to someone who grew up in more temperate lands and only knew that reindeer pulled Santa’s sled. And they could fly.

Now, there are other escape stories. Two I am familiar with are Lajos M., a Hungarian who is imprisoned near the Don river. A more popularized story is the “The Long Walk”, about a Polish officer, Slavomir Rawicz, who escapes from a Siberian prison camp in a snowstorm and treks 4000 miles to freedom. Now, this story is controversial, but that is because Rawicz seems to have borrowed the story, so it still likely has some elements of truth, while the names and places might be invented.

And there is more evidence that even if Rawicz didn’t do the walk, someone else did.

We learned of a British intelligence officer who said he had interviewed a group of haggard men in Calcutta in 1942 - a group of men who had escaped from Siberia and then walked all the way to India.

And then from New Zealand came news of a Polish engineer who had apparently acted as an interpreter for this very same interview in Calcutta with the wretched survivors.

These stories are second-hand, and far from conclusive proof, but for Mr Weir, they convinced him that there was an essential truth in the story that he wanted to retain.

“There was enough for me to say that three men had come out of the Himalayas, and that’s how I dedicate my film, to these unknown survivors. And then I proceed with essentially a fictional film.”

_https://www.bbc.com/news/world-11900920 _

Lajos M., 42 éves / Louis M, age 42

This is a book I picked up at my wife’s house, and is a powerful story of a Hungarian, also a paratrooper I think, that is caught behind enemy lines and send to a prison camp. This book solemnly relates Lajos M.’s incredible suffering as he somehow manages to stay alive in a prison camp. Somehow it was published during the communist times, perhaps that’s why it reads a bit darkly, fatalistically (Stoically?) like The Good Soldier Schweik, Catch-22, Chickenhawks, and Slaughterhouse Five, and other war stories of this nature.

He spends 13 years there until they are finally pardoned. I think at one point, like Dostoevsky, he is sentenced to death, but pardoned at the last minute. The story starts out with Lajos, now working at a factory in Hungary, stealing a blanket, so a co-worker wants to find out why one would steal a blanket, and so unravels the long story. You get the gist of the powerful, dreamlike quality of the story in this summary:

M. Lajos magyar közkatona elment a Donig. 1943-ban esett hadifogságba. Fogolyként elment az ember-lakta világ legeslegszéléig. És onnan is továbbment. Tizenhárom év múltán visszafelé is megtette ezt az utat. Hazafelé. Mindig azt tette amit mondtak neki. Csak közben M. Lajos eltévedt a történelemben.

Hungarian Lajos M. went to Don. He was captured in 1943. As a prisoner, he went to the very edge of the human-inhabited world. And from there he went on. Thirteen years later, he made this journey backwards. He always did what he was told. Meanwhile, M. Lajos was lost in history.

Unfortunately Lajos M. is an obscure Hungarian book, and I haven’t been able to figure out if this a true account, or some kind of merging of a bunch of stories, but no question, like Lajos M., József went to the eastern front, and he made it back, somehow.

Now with these precedents, especially with Jan’s story, could this amazing story of a Hungarian rescued by a reindeer be true? Now, it starts to make sense. Why, of course, the only way to escape would be by sled, it is as normal as a car is to us suburbanites, or skis are to Norwegians in the far north.

I have few details of this story other than this grandfather József K. was (as is often the case with paratroopers) captured behind enemy lines. Theorizing here, but the prison camp, being in Siberia, was so remote (perhaps an island as in Lajos M.’s case, or across 7 time zones like Magadan) that it was not as tightly secured as other, more urban camps might be, since it would be suicide to attempt to escape. Alcatraz was not protected by the prison walls, but by the San Francisco bay, with cold waters, strong currents and sharks.

In a recent New York Times article on Magadan, the caption to one of the photos echoed this idea

Clearing snow by a lighthouse. Residents refer to the rest of Russia as “the mainland,” a sign of how isolated the city feels.

The story goes that somehow József was assisted by some locals, and placed on a reindeer sled. No one “drove” the sled, according to my wife, “the reindeer knew the way to the next town.” And this is how Grandfather returned home. Now, why would these locals help a prisoner, a foreign invader? Given that they were herdsman, perhaps they realized József was a long-lost cousin, from their nomadic past? This is me entirely theorizing here, but there are tribes in Siberia that live a semi-nomadic existence, almost like American Indians in the States used to, and their language is (distantly) related to Hungarian.

The core Khanty vocabulary still contains numerous examples of vocabulary inherited from the Finno-Ugric proto-language (Collinder, 1962). Khanty is predominantly an agglutinative language with no prepositions and numerous affixes, each of which expresses a particular function.

Jan

Now, getting back to Jan’s epic tale.

With Jan’s tale, exhaustively corroborated in We Die Alone, as Howarth retraced Jan’s steps to validate the story, and even talked with the villagers who helped him, surely there is truth to József’s story too.

This certainly leads to the question of how much help the Norwegians were in the first place. For starters, the ethereal patriotism of the Norwegians is the main reason why the commandos were exposed. The commandos had a list of helpful partisans, so when they landed in Norway they made the unfortunate mistake of talking to the owner of a grocery store whose name matched one of the partisans on their list. However, in a twist of fate, it turned out he was not the person they were thinking of, he just happened to have the same name and now owned the same grocery store after the owner died. Surely some of the fault goes to the commandos, who aren’t careful enough to vet the supposed partisan, and they further compound the error the mistake by being nice and letting this chap go, and depend on his honor not to turn them in.

The grocer wrestles with the matter a bit before making a compromise and calling the town mayor or some other official to inform him, instead of calling the Nazi party directly.

Here’s a movie based on Howarth’s book:

Howarth notes the theory that the locals take their time in alerting the authorities, on the notion that this will give the commandos a chance to escape. Sadly, this nice gesture was not communicated to the commandos, but even if it had, they had no chance of escape in their sluggish fishing boat.

The Pathfinder?

This gets to an interesting side question on the subject of whistle blowers, much in the news these days, and I propose two simple types:

(1) the whistle blower who squawks because they fear for their safety if they don’t squawk and get caught

(2) The whistle blower who, at the risk of his own safety, blow the whistle because of a greater calling.

The Norwegian grocer and others who compromise the mission are “whistle blowers” in the former sense, allying themselves with the foreign Nazi menace due to fear of persecution. Here’s the reasoning of the Norwegian official who turns the commandos in:

“The story was bound to spread, and the Germans were bound to hear it; and then the official himself would be the first to suffer.”

Howarth does not discuss what happens to the few commandos who survive, but we can infer their suffering was great.

Now, as is the case with many pondering this story from the comfort of their armchair like myself, it is easy to blame the weak-willed grocer and local officials for talking, but we were not there, and likely, when push comes to shove, I would do the same I suppose.

Who knows what makes people like Oskar Schindler, a onetime wealthy businessman and Nazi sympathizer, to later put their life on the line? Christopher McDougall explored this idea in “Natural Born Heroes”, when he described a diminutive woman who disrupted the attack of a school shooter, despite, until then, having done nothing out of the ordinary. For another excellent study of this idea, see The Pathfinder, a film by Nils Gaup, in which a young man has to decide whether to save himself, or help a tribe chased by Russian invaders.

Most of the Norwegians Jan meets as he attempts to escape, while they are trustworthy and helpful, are very careful in extending any assistance. For example, rather than hiding Jan in their homes, the partisans keep Jan lodged first in an abandoned, unheated hut, and then nestled in a snowbank at the top of a valley.

To me, this alone is one of the most astounding challenges Jan faced. Clearly, the Norwegians were assuming the Sámi would arrive “any day now”, but as they are to find out, the Sámi have a different timetable. Howarth, like the frustrated Norwegians, seems to take issue with the Sámi and their concept of “the vague imponderable future”

Heroes

Howarth’s book focuses on the many Norwegian’s who assist Jan, so don’t need to mention them here as their story can be found in better detail in the book. There is the midwife who helps him recuperate after his initial escape, and the old Norwegian, Lockertsen, who puts himself greatly at risk, because he rows Jan across the fjord, going at night to evade any Germans potentially on watch, and drops Jan off at Ullsfjord.

The Sámi

No question the locals put their lives at risk, in fact over 100 Norwegians assisted Jan, but my thesis is that the Sámi are the great heroes of this story, next to Jan Baalsrud. Yet the Sámi are not even mentioned in the summary on the book’s back cover, and only are fully covered in the last chapter.

Howarth seems to take a dim view of the Sámi, considering them opportunists simply out for the trinkets that the Norwegians give them as payment. Further, he notes their strange concept of time “the imponderable future”, and other puzzling differences. Now, anyone who knows a strange culture, say the Hungarians, well, aren’t these admirable characteristics instead? As shown below, the Sámi herdsman who ultimately takes Jan to safety ponders the task, as he stands still for 3-4 hours, watching Jan.

Had Jan taken a similar amount of time to plan his route, rather than barreling through the middle of the town, depending on the prowress of the Norwegians in skiing, and the Germans lack of ability in the same, he might not have gotten himself in such a predicament.

I can’t imagine how he finally made up his mind to take him, but certainly the difficult state that Jan was in, and that he would surely die if he stayed in Norway, perhaps swayed him. 

[Scene: Jan is in his snowbank refuge. After waiting there, above the village for almost a month, he has given up all hope. It took every ounce of strength of a few villagers to get him to a plateau where they hoped the Sámi would be. For the villagers, the Sámi with their reindeer would be like a helicopter to one of us, stuck on a mountain. But the snow is melting…]

When he opened his eyes there was a man standing looking at him. Jan had never seen a Lapp before.. the man stood there on skis, silent and perfectly motionless, leaning on his ski-sticks [ski poles, my British friend]. He had a lean swarthy face and narrow eyes with a slant. He was wearing long tunic of dark blue embroidered with red and yellow, and leather leggins, and embroidered boots with hairy reindeer skin and turned-up pointed toes.

In fact, this was one of the Lapps whom the ski-runner from Kaafjord had gone to see on his journey a month before. He had just arrived with his herds and his tents and family in the mountains at the head of Kaafjord, and must have been thinking over the message all the time. When he had first been asked, the whole matter was in a vague imponderable future….He did stand looking at Jan for three or four hours.

Howarth goes on to mention the blankets coffee brandy and tobacco procured at “enormous prices” to give to as payment for the escape. Other Sámi had turned down the offer, so we can gather they understood the risk, and knew the trinkets would not pay for a loss of life, but luckily for Jan, this Sámi (we never learn his name) agrees.

The next thing that brought Jan to his sense was a sound of snorting and shuffling, unlike anything he had ever heard before, hoarse shouts, the clanging of bells and a peculiar acrid animal smell, and when he opened his eyes the barren snow-field around him which hadd been empty for weeks was teeming with hundreds upon hundreds of reindeer milling around him with an unending horde, and he was lying flat on the ground among all their trampling feet. Then two Lapps were standing over him talking their strange incomprehensible tongue….They muffled him up to his eyes in blankets and skins, and lashed him and everything down … There was a jerk, and the sledge began to move.

A lapp on skis was leading it. It was one of the bell deer of the heard, and as it snorted and pawed the snow and the sledge got under way and the bell on its neck began a rhythmic clang, the herd fell behind it, five hundred strong.. the mass of deer flowed on behind, it streamed out in the hurrying narrow column when the sledge flew fast on the level snow.

All day the enormous beasts swept on across the snow.. the most strange and majestic escort ever offered to a fujitive of war.

Hurrah! How exhilarating! I picture the start of the American Birkebeiner ski race, with thousands of skiers churning up the snow, or the giant Vasaloppet in Sweden with 20,000 skiers.

I imagine that was a swell ride too, for József K, to finally leave the prison camp for home.

The Man Who Never Gave Up

https://nordnorge.com/en/artikkel/jan-balsrud-is-the-man-that-never-gave-up/

As Far as My Feet Will Carry Me

Here a German escapes from a Soviet prison camp in Kamchatka and eventually finds his way to Iran. As with “The Long Walk”, there are doubts as to the authenticity of the story.

https://www.youtube.com/watch?v=Zaq88ixc-FY

The Pathfinder

https://en.wikipedia.org/wiki/Pathfinder_(1987_film)

How The Long Walk became The Way Back

https://www.bbc.com/news/world-11900920

** **

Lajos M., 42 éves

https://www.antikvarium.hu/konyv/csalog-zsolt-m-lajos-42-eves-lajos-m-aged-42-37420

**The Good Soldier Švejk **

_The unfinished novel breaks off abruptly before Švejk has a chance to be involved in any combat or enter the trenches, though it appears Hašek may have conceived that the characters would have continued the war in a POW camp, much as he himself had done. _

The book includes numerous anecdotes told by Švejk (often either to deflect the attentions of an authority figure, or to insult them in a concealed manner) which are not directly related to the plot. [Source: wikipedia]

The 585 people appearing in The Good Soldier Schweik: http://honsi.org/svejk/?page=4&lang=en

Magadan, Russia

Nine Lives / Ni liv (1957)

[michaelok]

Fun with Java Deserialization

Down the Rabbit Hole

I’ve just been scrambling down the rabbit hole to patch an app that Qualys is complaining has a deserialization vulnerability. What should have been a simple effort has turned into a mystery because, while we appear to have the correct libraries already in place, Qualys is still complaining about the error. A report that should be clean, to indicate compliance with GDPR, is instead “yellow”, or “orange”, or “red”, so fingers point, tempers flare, e-mails fly about, cc’ing higher and higher ups, so now we have assumptions, and based on those assumptions, tersely written orders, involvement by 3rd party vendors. Time to panic? Shall we be careful and tip-toe through the eggs?[0]

Well, it turns out to be a rather interesting mystery.

What is Java serialization?

First, some definitions are in order. What is Java serialization and why is it important? Perhaps Wikipedia[1] defines it the simplest:

A method for transferring data through the wires

Java serialization is a mechanism to store an object in a non-object form, i.e. a flat, serial stream rather than an object, so that it can be easily sent somewhere, such as to a filesystem, for example. It is also known as “marshaling”, “pickling”, “freezing” or “flattening”. Java programmers should be familiar with the concept, and with the Serializable interface, since it is required in various situations. For example, this technique is used for Oracle Coherence’s “Portable Object Format” to improve performance and support language independence.

Early Days of Java Serialization

Amazing to think that, back in the day, we used all the various tools required for distributed communication, whether simple like RMI and JMX, or more involved specs like CORBA and EJB, and we never thought much about the security aspects. I’m sure if I peruse my copy Henning and Vinoski’s definitive work on C++ and CORBA, I’ll find a chapter or so focusing on security[1], but I’m figuring, we, like everyone else, focused on the business details, getting the apps to communicate reliably, adding features, improving stability, etc, and not on whether there were any security holes, such as tricking a server into running cryptocurrency mining malware[2]. Yes, Bitcoin and the like did not even exist then.

The Biggest Wave of Remote Execution Bugs in History

Well, times change, and the twenty-year-old Java deserialization capability is the source of “nearly half of the vulnerabilities that have been patched in the JDK in the last 2 years” [3], so Oracle has plans in the works to completely revamp object serialization. Further note that this is not solely Oracle’s issue, nor is it limited to Java. Many other software vendors, and open source projects, whether tools or languages, have this weakness, such as Apache Commons Collections, Google Guava, Groovy, Jackson, and Spring.

It seems all the excitement, at least in the Java world, started when Chris Frohoff and Garbriel Lawrence presented their research on Java serialization “ultimately resulting in what can be readily described as the biggest wave of remote code execution bugs in Java history.” [6] However, it is important to note that this flaw is not limited to Java. While Frohoff and Lawrence focused on Java deserialization, Moritz Bechler wrote a paper that focuses on various Java open-source marshalling libraries:

Research into that matter indicated that these vulnerabilities are not exclusive to mechanisms as expressive as Java serialization or XStream, but some could possibly be applied to other mechanisms as well.

I think Moritz describes the heart of the issue the best:

Giving an attacker the opportunity to specify an arbitrary type to unmarshal into enables him to invoke a certain set of methods on an object of that type. Clearly the expectation is that these will be well-behaved – what could possibly go wrong?

Java deserialization

For our purposes, we focused on Java serialization and Apache Commons Collections. From the bug report COLLECTIONS-580[4]:

With InvokerTransformer serializable collections can be build that execute arbitrary Java code. sun.reflect.annotation.AnnotationInvocationHandler#readObject invokes #entrySet and #get on a deserialized collection.

If you have an endpoint that accepts serialized Java objects (JMX, RMI, remote EJB, …) you can combine the two to create arbitrary remote code execution vulnerability.

The Qualys report didn’t have much in the way of details, other than a port and the commons-collections payloads that illustrated the vulnerability, but I guessed from that info that the scanner simply uses the work done by the original folks (Frohoff and Lawrence) [5] that discovered the flaw available as the ysoserial project below.

https://www.youtube.com/watch?v=KSA7vUkXGSg

Source code here: https://github.com/frohoff/ysoserial

Now, in the flurry of trying to fix this error, given the annoyingly vague details from Qualys, I had looked at all sorts of issues, after noticing a few extra JVM arguments in the Tomcat configuration that happened to be set for the instances that were failing with this error, but were not set on other instances. Apparently someone had decided to add these, without informing our team. Interesting.

Now, remember that according to the original bug report, this exploit requires (1) untrusted deserialization, it (2) some way to send a payload, i.e. something listening on a port, such as a JMX service. In fact, These extra JVM args were for supporting remote access via JMX, so unraveling the thread, I researched Tomcat 8 vulnerabilities especially related to JMX. While it turns out that JMX is a weak point (JBoss in particular had quite a well-known major JMX flaw), I did have any luck convincing the customer that they should shut down the port. It is used to gather monitoring metrics useful in determining application performance such as CPU load, memory, and even cache information. Ok, easy but drastic solutions were off the table. I was back to the drawing board.

Next, I tried to see why it was flagging Apache collections in the first place. Going back to the ysoserial project, was it incorrectly flagging Apache Commons Collections 3.2.2, or Collections4-4.1, despite the fact that the libs were fixed? Further looking at the specific payloads, Qualys/Ysoserial was complaining about Collections 3.2.1, which limited the test scenarios to try to get working

Now here’s the interesting part: with ysoserial, I was unable to get the exploit to work, as depicted in the Marshalling Pickles video. It was failing with a strange error I hadn’t seen before, something about filter setting a “rejected” status. Now, this led me to finding info about Oracle’s critical patch update (_121). I was running with latest & greatest JDK, release _192, however our production servers were running a very out-of-date version - surprise surprise.

Apparently, with Oracle JDK at release 121 or later, Oracle has started to address this vulnerability in an official way, rather than what exists currently which is a bunch of ad-hoc solutions, mainly whitelisting/blacklisting, which is a difficult without library support. Some would call this ‘whack-a-mole’, but I think this illustrates quite well the idea of a “patch”, i.e. there’s a leak, so run over and put some tape over it, but we aren’t solving the fundamental issue. In other words, the current defense against this attach is limited because we can’t possibly know what libraries customers will use, so the library maintainer has to scramble to plug the holes whenever they are discovered. Note that even the best of libraries like Groovy, Apache and Spring have had to fix this flaw.

So kudos to Oracle for taking some much needed steps in solving this problem. Here’s a little detail on the new feature that works to make the deserialization process more secure:

The core mechanism of deserialization filtering is based on an ObjectInputFilter interface which provides a configuration capability so that incoming data streams can be validated during the deserialization process. The status check on the incoming stream is determined by Status.ALLOWED, Status.REJECTED, or Status.UNDECIDED arguments of an enum type within ObjectInputFilter interface.

https://access.redhat.com/blogs/766093/posts/3135411

While it is the “official” way to deal with the deserialization issue, it remains to be seen how well this strategy will work. As a further research project, I’m curious whether this model might be used beyond Java serialization, i.e. in projects like Jackson. Does it add anything more than Jackson already has, or does it simplify it, etc.

This feature is targeted for Java 9, but was backported to 8, though it looks like it doesn’t have all the functionality that Java 9 supports.

So you are probably wondering what happened? Did we fix all of the above, and even throw in an upgrade Tomcat, like the Monty Python “Meaning of Life” movie “everything, with a cherry on top!” Well, finally, given a little guidance on where to look, the 3rd party developers - turned out that not only had they added the JVM args, they had also added in some extra code to handle the authentication. Which used - you guessed it - the _old_ 3.2.1 version of commons-collections. This code was also manually maintained, so while the app our team maintained received the updated commons jar in an automated fashion along with all the other updates, this little bit of code, tucked away on the server, was never updated.

Lessons learned? Off-the-wall custom authentication? Don’t do this. But if you do, don’t leave manually updated chunks of code lying around, and further, keep up with the patches!

[0] Yes, I’m reading William Finnegan’s “Barbarian Days: The Surfing Life”, Finnegan’s hilarious and fascinating account of being a surfer in the early days of the sport. At one point, he complains to his friend and fellow surfer, who is getting on his nerves, that he is tired of walking on eggs around him. Of course, in his anger, he mixed up the quote, and meant “walking on eggshells”.

[1] https://en.wikipedia.org/wiki/Serialization

[2] A quick look at the omniORB doc shows it has a feature called the “Dynamic Invocation Interface…Thus using the DII applications may invoke operations on any CORBA object, possibly determining the object’s interface dynamically by using an Interface Repository.” Sounds like reflection doesn’t it? I’m not aware of any specific vulnerabilities, but it does seem we’ve traded a bit of the security that invoking statically-compiled objects brings for convenience.

https://www.cl.cam.ac.uk/research/dtg/attarchive/omniORB/doc/3.0/omniORB/omniORB011.html

[3] https://www.siliconrepublic.com/enterprise/cryptocurrency-malware-monero-secureworks

The Java Object Serialization Specification for Java references a good set of guidelines on how to mitigate the vulnerability:

https://www.oracle.com/technetwork/java/seccodeguide-139067.html#8

[4] https://www.securityinfowatch.com/cybersecurity/information-security/article/12420169/oracle-plans-to-end-java-serialization-but-thats-not-the-end-of-the-story

[5] https://issues.apache.org/jira/browse/COLLECTIONS-580

[6]Which seems to be a pretty standard strategy these days, i.e. proprietary companies like Qualys leveraging open source and adding it to their toolset. AWS does this to great effect, and we, as the consumer, benefit by getting simple interfaces. However, we should not forget that much of the code we use today is Open-source software, in some way or another.

Stratechery, as usual, has a very thoughtful post about this very idea:

It’s hard to not be sympathetic to MongoDB Inc. and Redis Labs: both spent a lot of money and effort building their products, and now Amazon is making money off of them. But that’s the thing: Amazon isn’t making money by selling software, they are making money by providing a service that enterprises value, and both MongoDB and Redis are popular in large part because they were open source to begin with.

[snip]

That, though, should give pause to AWS, Microsoft, and Google. It is hard to imagine them ever paying for open source software, but at the same time, writing (public-facing) software isn’t necessarily the core competency of their cloud businesses. They too have benefited from open-source companies: they provide the means by which their performance, scalability, and availability are realized. Right now everyone is winning: simply following economic realities could, in the long run, mean everyone is worse off.

https://stratechery.com/2019/aws-mongodb-and-the-economic-realities-of-open-source/

[7] https://www.github.com/mbechler/marshalsec/blob/master/marshalsec.pdf?raw=true

[8] https://medium.com/@cowtowncoder/on-jackson-cves-dont-panic-here-is-what-you-need-to-know-54cd0d6e8062

[michaelok]

Heroes

The engineer who figured out the problem with the water in Flint teaches a course on “heroes” (see link below.)

He teaches a course on ethics and heroism at Virginia Tech. He tells his students that everyone has it in them to be heroic.

“I feel like I’m doing the job I was born to do,” he said. “I get up every day with such a sense of purpose I wish everyone could experience something like that once in their life.”

“Everyone has it in them to be heroic.” This is the same message from McDougall’s Natural Born Heroes, about the courageous Cretans, make life hard for the German occupiers.

Here’s a quote from a similar course at Stanford, taught by Meehan:

Another part of the course will examine a professional’s obligation to society. Meehan’s said his personal view is that “The professional, scientist, or any technical person has an obligation that goes beyond their employer,” to make known the hazards and implications of a particular technology.

Based on Beowulf, it was the same with our Viking neighbors, and so to with the Irish — consider St. Brendan, who sailed to the Americas in a leather boat, narrowly avoiding giants throwing fireballs at him, or castles of ice toppling over and crushing his boat.

Now, from reading Bánffy, we know that the nobility, carried with them a sense of duty, a responsibility that came with their elevated status. They knew they were from a long line of heroes, perhaps stretching back to the Magyar chieftain himself, Geza. In their estates, the walls were lined with pictures of leaders who fought in great battles. Does this create a deeper connection to the land, to the people, that, for example in Abady (the main character), causes him to go to lengths to help the peasants, even though these are the same peasants that are letting their cattle graze on his tree plantation?

Should we return to a focus on the Greek ideals in schools? The Greeks valued heroes, and told stories about their heroic feats, and passed the stories down orally from generation to generation.

Now, why wait for a course like this as a college student? Why not kindergarten? Or at least high school?

The heroic professor who helped uncover the Flint lead water crisis​ has been asked to fix it In Flint, Mich., there is a famous block of concrete that for decades has served as a community message board. Like an…www.washingtonpost.com

Stanford Course:

The Stanford Daily 22 April 1981 - The Stanford Daily The Stanford Daily > The Stanford Daily, 22 April 1981stanforddailyarchive.com

[michaelok]

Regex Tips and Tricks

The intent for this page is to provide a list of handy regular expressions that are useful for a particular task, or can be a base that can be modified. The idea will also to show how regular expressions can be subtly different across systems, for example platforms like Microsoft Windows, Oracle, and Vim, or languages like Ruby and Java.

Microsoft Windows

Let’s set the scene first. You have been pulled in to help another team, and need to make a hot fix to an unfamiliar build. Sometimes you find yourself in the situation where, somehow, your build process has failed, and for a variety of reasons, you can’t determine the source of the failure. It is 1 am, your hot fix build just failed for the third time. Each build runs about 75 minutes, and sometimes it works, and sometimes it doesn’t. Other developers and staff are depending on you to get the build working, yet it runs fine when your run it locally. You know it is possibly an indeterminate test, as it seems to be failing in the section with the unit tests, but you can’t be sure as there is no summary page for the unit tests, just a bunch of XML output files with the results for 10,000 tests, and the output from the main build log looks like a Microsoft .doc file. (Yes, I just learned that the .doc format is essentially a raw dump of memory. In other words, there is no format. Have fun.)  You, being a lowly developer, have no access to the build server, so you are unable to fix the build script. That will take much going back and forth and writing change tickets. The only tool you have is a way to copy the unit test files to your local Windows machine. You seem to remember mentioning to a friend, back when you worked on Xenix, a version of Unix that ran on the x86, and noted the coolness of commands like grep, and your friend replied “oh, but Windows has something like that too.”

In essence, you are in a dark maze. Tunnels lead to your left and right. A cool wind blows across your face. What to do?

Well, it turns out Windows has, like everything, it’s own take on regular expressions, and has what some would call rather rudimentary, non-standard, and bizzare (ok, let’s say creative) support for regex. 

1. Copy all the test output to a local directory.

2. Open a command prompt and cd to the local directory

3. Run the below command.

findstr /s /r “failures=\”[1-9]*\” errors=\”[1-9]*\”” *.xml

Here is a line-by-line description:

Oh, that’s right, it is a one-liner. Let’s make that a word-by-word description.

First we have the venerable Windows findstr command. 

The /s means we want to search current directory and subdirectories.

The /r means we want to treat the pattern as a regular expression, of course keeping it in mind that this is a Microsoft’s idea of a regex. 

More detail at Microsoft Technet:

https://technet.microsoft.com/en-us/library/bb490907.aspx

Now the details. We are looking for the keyword “error” or “failure” followed by an equal sign, and a non-zero value, that can be greater than 10, i.e. something like the following:

errors=“1″

Now, you would think we could use the “not” operand which is ^ even in this case, and included and explained in the documentation, but it does not work, or perhaps it does work, given some special magic sprinkling of flour as you utter the correct mantra. Suffice it to say, without the Microsoft source code, it’s easier to figure out the workaround.

There is one more interesting nuance with Microsoft’s interpretation of regular expressions. With your standard regular expression background, you’d also think that if you are listing the alternatives, where we want either “error” or “failure”, they would be separated by the pipe symbol “|” but no, in findstr-land, you use a space. Interesting.

And so there you go. This should list any lines in the raw XML files that have either an error value of non-zero, or the failure value of non-zero, and the file name.

Runaway Trains

As you get better with regexes, don’t get too confident. Here’s a fun story that shows the power of regular expressions, and the need to use some care in their design, especially when the particular expression you are using will be deployed to a production system.

An endless loop is a rather typical rookie mistake - maybe the sentinel value never got flipped so you end up with:

while (true) {

// an endless loop

}

As programmers get better, usually they can add a test and spot these mistakes easily. What’s interesting about regexes is that, due to their rather obscure (hard to read) nature, and their power, even coders with a master degree in CS, who consider themselves fluent in the language, can fall into the trap of creating a regex that can spiral out of control, so that it is like a runaway train, and can cause serious performance issues on even powerful production machines.

Yes, I first learned of this interesting capability while working on a performance issue. Every so often, our prod machines would get stuck, CPU would spike, and they would not free up until the entire web app was killed. Logs weren’t much use, so you ask to see the thread and heap dumps. Now, sometimes it can be very difficult to review these (unless you are a JVM wizard) but in this case, the source of the problem was fairly obvious. On the failing machines, there were several threads with long stack traces having to do with regexes. Long, as in 500-1000 lines. Now, stack traces are complex, as every programmer seems to be following the mantra of “I can solve this problem by adding another class”, so even what would seem to be fairly simple operations can sometimes involve many, many classes and methods, but 500-1000, at least at the time of this writing, and for our particular app, is a bit unusual.

We dug into the code, and it turned out to have to do with a regex which was triggered for particular customers, and a particular bit of search text (this had to do with a Search app) which of course made it difficult to troubleshoot. In fact, this bug had been in the code for a few years, and there were various attempts to solve it. With tricky bugs like this, the simplest solutions should always be applied first, i.e. give it more memory or faster hardware. But just as typically, these solutions usually don’t work, or only temporarily solve the issue.

Upon looking at the particular regex, and testing it out, we were able to reproduce the behavior locally. Spin up the app, run our integration test, and ‘bam’, the machine would churn away for 45 minutes, which is the point your say “ok, this may not be an endless loop, but there’s a problem here.” A little research brought up this excellent post about this rather interesting phenomenon: 

Runaway Regular Expressions: Catastrophic Backtracking

The author some nice examples and describes how to avoid these issues. 

Clearly we have an exponential complexity of O(2^n) here. At 21x the debugger bows out at 2.8 million steps, diagnosing a bad case of catastrophic backtracking.

RegexBuddy is forgiving in that it detects it's going in circles, and aborts the match attempt. Other regex engines (like .NET) will keep going forever, while others will crash with a stack overflow (like Perl, before version 5.10). Stack overflows are particularly nasty on Windows, since they tend to make your application vanish without a trace or explanation. Be very careful if you run a web service that allows users to supply their own regular expressions. People with little regex experience have surprising skill at coming up with exponentially complex regular expressions.

Well, as the above story proves, even very experienced coders, probably under pressure, and without the chance to test every combination, can write exponentially complex regexes.

So, then what did we do? Well (as a reminder to myself), I should at some point print the particular regex, and the issue (a group with a star (kleene closure!)* that itself was starred. With a bit of juggling (who knew that coders have so much in common with jugglers?), we managed to create a regex that accomplished the same thing without the exponential issue causing the machines to stop in their tracks, or rather, go off spinning CPU cycles like a runaway train!

Have lots of fun! 

*One of my profs was proud to note that the person who invented the * had taught at the same school. Yes, it’s pretty cool when you get a mathematical concept named after you :)

[michaelok]

Now

Here is a summary of my current status, as an ex-New Englander, of Irish/German extraction, living in Minnesota, and therefore somewhat like a hobbit from the days of old, living amongst these giant, but quiet Vikings.

For entertainment here, we enjoy going out for a cross-country ski, and then cooking ourselves in a sauna, and finally diving into a frozen lake. Others like to go ice fishing, some of us like to swim with the fish.

At the moment I am preparing for a long-distance skate ski race (I’m better at classic) and pondering rowing, windsurfing, and/or cyclocross as a way to stay in shape during the off-season. Yoga is fantastic - I have an amazing teacher and it is great for handling stress, but it does not prepare one for “The Wall”, as the local high school skiers call the massive (for the midwest) hill on a popular nearby trail. (I love this name, as it reminds of the name of a mountain, or mountain range, in Austria called “Hohe Wand” which means “High wall”.

[Odd that I put sports first in this list. Hmm.]

In an effort to be a well-rounded person, like an Ancient Greek, I have been learning to play a penny whistle, with grand ambitions to pick up the guitar again. I find I spend a surprising amount of time watching YouTube videos of The Great Ones, from Angus Young, to Jimmy Page, to Bert Jansch (inspired Page to write Black Mountain Side).

As can be seen on these pages, I also have an interest in Hungarian culture and literature. You can already find a little about the Rock Opera, Steven the King, on these pages, and I will also put some posts up about Miklós Bánffy’s incredible Transylvanian Trilogy, which my uncle gave me.

I also am continuing to learn the Hungarian language, more than just so I can understand what my in laws are talking about. It is a hard language for an English-speaker to learn, and I would love to speak it at a fluent level. Right now I am at a low level, but there is a school we take the kids to, so we all go, which is quite fun.

I am taking a gaming course on Udemy, which covers the Unity framework and C#. I learned how to program by copying in games from Creative Computing, and writing a few of my own, with a little help from my high school buddy who worked on Halo, and a few other games for the XBox (that Kinect thingy), PopCap, and now Facebook. Definitely interested in all the “new new stuff” (analytics, machine language, cloud, security), but after the gaming course I will probably take an algorithms class, so I can finish up a graph class I started a few years ago. I’ve pondered going back to school, perhaps getting a masters, but, as someone once told me about engineering school, to make it through you really have to want it. So I take second-fiddle while my wife studies her herbal courses, as she has way more focus than me.

On the softer-skill side, I try to get out to a local shop called DevJam that serves as a part-time coffee house, part-time platform for tech wizards to show off their stuff. For example, the talk this Wednesday will be a guy from Netflix’s Chaos Engineering team.

[michaelok]

István, a király / Stephen, the king

The question is, what would be good training music for some x-skiing?  Or a long drive, say to Marquette, or another mining town, Butte? An article in Cross-Country Skier Magazine suggested "Come Sail Away".  The author wrongly crediting Bee Gees, but this is a song written by Dennis De Young with Styx. Perhaps István, a király (Stephen the King) would be a good choice. This is a very popular Hungarian rock opera produced back in the day of the great rock operas like Pink Floyd's "The Wall", "Hair", and "Jesus Christ, Superstar" and is still performed live. The music is beautiful. It was written by members of ILLÉS, one of the most popular bands in Hungary:

"Written by Levente Szörényi (music) and János Bródy (lyrics), whose work is a masterful melting of numerous and rather different musical styles, ranging from Gregorian chant, through folk music, to hard rock." It opens with Beethoven: "The full title of Beethoven's score is "King Stephen, Hungary's first Benefactor, a Prologue in one act by Kotzebue, Music by Ludwig van Beethoven, written for the Opening of the New Theater in Pesth, February 9, 1812." Da Pacem, Domine. 

I think I first heard Gregorian Chants when on an epic drive to Butte, Montana. The other tape we listened to on that drive was Handel's Water Music. 

There is an interesting mixture of dances and music in the film, and one can gain an understanding of the current music styles popular in Hungary at the time. The pagans, the followers of Koppány, dance the men's dance, which is a traditional style, somewhat like step or tap dancing except you slap your boots with your hands. The music is heavy guitars, heavy metal, and the leaders dress as if they could be pro-wrestlers or bikers if they weren't such good singers, dressed in black leather and heavy chains

The Christians are lead by István looking like he was probably the star of Jesus Christ Superstar also, in his white frock. His followers dance in their bobby-soxer look, and are folk-oriented too. Remember, Hungary, in the 80s was like traveling into a time-warp, with the strong musical traditions from the Habsburg Empire such as Franz Liszt played at the Báls, and thus the melody from the prologue by Beethoven plays throughout the musical, but there is also the "newer" music (American 50s music) played at New Years Parties. And of course always the folk music - listen and you can hear the flute and bag pipes too.. Here is a short summary "Transition from paganism to Christianity. In 997 A.D., the Magyars' ruling prince, Géza, dies. The nation is torn between two religions, represented by the clashes of his son and chosen successor, István, and his pagan relative, Koppány, who claims himself to be the new leader, based on ancient pagan rules." In the film version, not all the music is sung by the actors and actresses, notably István, and Koppány's daughter,  Réka, (singing in front of the fire half way through in the clip below), but the voice is Márta Sebestyén's, later to gain world wide fame for the music in the English Patient.  However, it is impressive, judging by the credits at the end, than many, including Bernadett Sára (playing the Bavarian princess, Gizella, István's wife, part of the original cast) do sing and act. They might have been part of the original stage version.  Even the bishop, Asztrik, has an unusual and powerful voice. To give you some of the flavor, here's a couple versions of "Halld meg uram a kérésem", where Réka pleads for her father, so she can give him an honorable burial.  I believe they are dancing a csárdás which is a traditional folk dance. Her request, sadly, is not to be honored.  It seems István's mother, Sarolt, still wields much power, and wants to make an example Koppány, like the Romans planned to make of Jesus. Here's the original, Kovács Ottillia and sung by Sebestyén Márta http://www.youtube.com/watch?v=5q-mfveVABE&feature=related This is a newer version,with Herczeg Flóra playing Réka: http://www.youtube.com/watch?v=5jzg-joPnOY&feature=related Emese ("meh-she") sings the part beautifully also. http://www.youtube.com/watch?v=xx1vws9O4sA&feature=related From the comments, it seems everyone thought Mese was better, but Herczeg got the part, perhaps because she was a better actress.  Here are the lyrics, with a translation by a non-professional, but native speaker ;) RÉKA: Gy��ztél, uram, atyám felett, Ki sötét vihart kavart ellened. Fény és dicsőség a neved, De ne gyűlöld a legyőzötteket! (My majesty you won over my father, who stirred up a dark storm against you, your name is light and glory, don't hate the conquered) Halld meg uram a kérésem, Halott atyám add ki nékem! Hadd temessem el a földbe, Nyugodjék békétlen lelke! (Give me my dead father so I can bury him in the earth and he can be in peace) Légy az igaz, megértő urunk, Ha már e földön szolgáid vagyunk, Nyújts vigaszt a szenvedőknek, Bocsáss meg a vétkezőknek! (Give us consolation and pardon the sinners) ISTVÁN: Ne félj, a kérésed nem szégyened, A hangod a szívemhez szól. (Don't worry your question is not a shame, my heart is listening to your voice) RÉKA: Jó uram, a szívem tiéd, A szív a szívnek így válaszol. (Your majesty, my heart is yours and this is how the heart answers ) SAROLT: Mit nem képzelsz, balga lányom, kérésed már árulás! Hitszegőknek itt a földön megbocsátni nem szokás. Koppány testét felnégyeljük, s várainkra tűzzük fel. Ez legyen a lázadóknak mindörökre intő jel! Pusztulj vagy meghalsz! (What do you think, foolish daughter, your request is betrayal! We don't pardon traitors on this earth, we'll quarter Koppány's body and string up the parts on the castle, and this will be a reminder for those who rebel for ever, and leave or you will perish!)

The great thing about youtube is - some fans have uploaded portions of the play and the film version to youtube. Film here, with subtitles:

https://www.youtube.com/watch?v=6OOubj4UAX4

Nice translation here:

https://www.youtube.com/watch?v=ZKMrXuedxY8

[michaelok]

Follow the goal creep

A journey of a thousand miles begins with a single step.

-Lao Tzu

This is just a 50-mile race, then another 50-mile race after.  

-Dusty Oslon, close friend and running partner of ultra-marathoner Scott Jurek, telling Scott not to worry about his first 100-mile race, the Angeles Crest 100.

David Heinemeier Hansson, also known as DHH, the creator of Rails, has an interesting post on how he got to where he is.  We've probably heard this advice before (divide and conquer), but DHH provides an interesting take on this idea as he explains how he implemented this practice in his own life.  He gives a little insight into how he went from a Danish kid with a business degree (?) to founding RubyOnRails, one of the most important open source projects out there.  Rails is a framework used to build many of the most popular sites around, such as Twitter.  In this post, I will highlight a few of the points I found interesting from DHH's post, and a few other inspiring individuals like Scott Jurek and Tim Severin.

David got the ball rolling on RubyOnRails by putting "down the Xbox controller" and learning Ruby.  DHH is a talented programmer, but action was key.  How easy it is to set aside work on a novel, and watch a movie, or surf the web.

It's also important to note that one shouldn't set the bar too low.  DHH, a skilled perl and PHP programmer was hired by Chicago developer named Jason Fried who had started a web site business.  (Fried had customers, but lacked the technical depth DHH had, so he simply hired DHH to help out.)  However, he didn't know anything about Ruby other than he thought it was a neat language.  To develop an entirely new framework, in an entirely new language, is a serious task.

Most likely, DHH approached this task itself in small, maintainable chunks.  Instead of the goal being "write a perfect, awesome framework that will solve every problem", his goal was much more the open source mantra: scratch an itch.  

One thing I've realized, while observing how successful open source projects get started, is that you have to be at the right place, with the right people.  I don't know the exact lore, but I do know that DHH started out as a PHP programmer, and met his business partner, Jason Fried, while trying to help Fried with some web sites he was working on.  Fried finally decided it was easier to simply hire DHH, and history was made.  I'm guessing Fried was an important part of the success, giving DHH the freedom to spend the time on writing a better mousetrap, which surely took away time from billable projects.

Finally, one idea DHH touches open is flow:

After all, racing to me was all about getting access to long stretches of flow, that sensation of being so completely engrossed in an activity that you lose track of time and place.

DHH is referring to some ideas popularized by  Mihaly  Csikszentmihalyi who, according to the bio on the TED talk above:

has contributed pioneering work to our understanding of happiness, creativity, human fulfillment and the notion of "flow" -- a state of heightened focus and immersion in activities such as art, play and work.

Clearly, incredibly productive individuals like DHH, Frank Lloyd Wright, and computer security expert Mikko Hyponnen, to name a few, are not only smart people, but they also share the characteristic that they are being extremely efficient and focused.  How else can they seem to, as Wright once said, "shake designs out of their sleeves"?

Another person I greatly admire is Tim Severin.  Now a famous explorer and writer, he got his start by organizing a little trip, via motorcycles, to China.  While at Oxford, (or was it Cambridge?), he happened to hear that another student was planning on doing the same thing he was thinking of, which was to visit the forbidden realm of China, at the time a closed country, and very difficult to visit.  He went to visit the fellow student, and told him he thought they should join forces, and do the trip together.  The student said "Okay, why don't you join me for tea?" and that was that.  

In fact, they never did make it to China, having never gotten the proper permission, but there is one notable part of the trip that illustrates what was to become a common theme across all of Severin's trips, from sailing a boat made of cow skins to America, to sailing a raft from China to America.  He is asked at one point for his papers by threatening security guards in a difficult part of the world, perhaps Iran.  The usually thorough Severin happened not to have the correct papers in this case.  But he happened to have a worn, credit card size card that said he was a member of the Geographic Society.  Suddenly the tense environment changed immeasurably.  The terse guards, familiar with the Geographic Society, immediately give Severin the necessary passes or approval for wherever he was going, perhaps thinking he was a famous explorer.  They were right, in a way, because very soon, he would indeed be a famous explorer.

Finally, there is Scott Jurek.  In his book, Eat & Run, he chronicles his journey from the backwoods of Minnesota to become one of the greatest of the ultra-marathoners.  Giving his story as an example, his main point is that each of us has it within ourselves to do something great.  It just takes some ambition, some inspiration, and some perseverance and belief in yourself.

In conclusion, in whatever you do, take that first step, on your thousand mile journey.  You may not have it all figured out, or have all the necessary papers, or even a Geographic Society membership card, but you can do it.

Follow the Goal Creep

[michaelok]

MAN, THIS IS GOING TO BE SICK. DITCH THESE BRAKES, MAKE IT A FIXIE, SNAG SOME NITTO DROP BARS, ALL SHIMANO COMPONENTS, BROOKS SADDLE, CHRIS KING HEADSET, CLIPLESS PEDALS, MAYBE SOME AEROSPOKES IN A SWEET COLORWAY … I JUST NEED A COUPLE THOUSAND DOLLARS AND I’LL BE SET.

FOR A FUTURE IN COMPETITIVE VELODROME RACING?

NO, TO LIKE … WALK IT TO PARTIES AND STUFF.

[michaelok]

Which programming books to use to learn Ruby and Rails?

Many Java developers, and programmers in general, once they have mastered one technology, are always searching for an easier, or better, way of doing things.  Michael Lewis captured this idea simply in the title of his book about Sun's Jim Clark, titled <a href="http://www.amazon.com/New-Thing-Silicon-Valley-Story/dp/0393048136">"The New New Thing"</a>.  

This is kind of the nature of the trade, and perhaps any trade.  Even if you are a circuit designer, fluent in C, or a low level transport wizard, expert in TCP/IP, or a COBOL guru, speaking CICS and DASD, or a .NET, usually, unless the field is just a paycheck, you want to challenge yourself.  Is there a better way?  Am I learning something that will be obsolete tomorrow?  So naturally, as each new technology comes along, you evaluate it.  Should I invest my time and become an expert in technolgoy X?  Is it CORBA?  Is it EJBs? Spring?  Hibernate?  SOA?  Hint: As a general rule, when IDEs come out to offer an easier way to manage technology X (where X might be EJB or Spring), consider that an opportunity to start freshening your search for "the new new thing".

Fortunately, in my opinion, and perhaps yours, I have only had to dabble in the aforementioned technologies.  And the tremendous, how should I say, what is the proper technical term, blah?  associated with those technologies is the very thing that pushes one to consider a language like Ruby, and a framework like Rails.

Consider Java.  It seems that here we are, ten years with Java, and it seems it is still a ridiculously complex effort to create a simple web app.  The typical Java stack your typical web site, and therefore typical Java guru will push, is a nightmare of complexity.

Start with a dash of Java.  A pinch of IDE - Eclipse, Netbeans, so far so good.  Roll in some framework - Struts, Spring, etc.  Toss in a database, and an ORM layer.  And of course we need a javascript library.  Oh, and almost forgot!  The web application server!  And we need libraries upon libraries.  Libraries galore!  But some work with this JDK, and this database, and some don't work at all.

Ok, so you get the picture,  And this is why Java folks are very busy, and figure, it's always been like this, so whatever, this is the Java way.  And others shake their heads in disbelief.  So you're telling me to add a record to a table, I've got to download Hibernate?  Partially, this is because with the established Java stack, you have an embarrassment of riches.  Honestly, the Java environment is mature, and scalable, and powerful and you can't go wrong.  And once you get your perfect stack, and everything is happy and it's all working, at that point you realize you need Maven and then, and then, that's when perhaps you have an epiphany and say "I've spent so much time figuring out the stack, I've totally lost my thought on the very thing I was going to build.  And it was a very simple thing."

A passionate .NET developer, and author, Roy Osherove, has made this same step.  And many other Java luminaries, such as James Duncan Davidson, author of Ant.

So the first step is to start with a book.  Now, it makes sense that to learn Rails, one must learn Ruby first.  And this is already exciting, because the author of Ruby, "Matz" (Yukihiro Matsumoto) thinks that programming should be fun!  Well, that sounds nice!

We'll take a look at two books below, and discuss the pro's and con's of each:

Programming Ruby by Dave Thomas and Chad Fowler

This is probably the book that started it all, at least for the English-speaking world.  Therefore, I had high hopes for the book.  However, I was not wildly impressed with his other book, "The Pragmatic Programmer".  For books of this nature, my favorite has always been Steve Macquire's <a href= "http://www.amazon.com/Writing-Solid-Code-Microsoft-Programming/dp/1556155514">Writing Solid Code</a>.  The Pragmatic Programmer has decent ideas, but nothing dazzling like Steve's book.  Steve draws upon his work done at Microsoft, for example creating a core code base for the Windows and Mac versions of Word.  He includes many fascinating real-life examples to back up the lessons in the book.  

In contrast to Macquire, Thomas draws upon his experience as a (Ricky Gervais flips over the intro card for Tim Allen, makes a perplexed, puzzled look) web site developer?  One of his examples in the Programming Ruby book is how he didn't performance test a web site for some contest, and how easy it was to fix because - voila, he used Ruby.  Thomas instantly reminds you of the type of the sloppy, unthinking programmer Macquire is talking about in his book.

His lack of valid experience, and the fact he's built some web sites for mom and pop is fine, but the real problem is the irritating attitude he writes with.  He comes across as one of your smarty-pants developers, which distracts you from the content of the book.  That would all be fine, but then, please, do not reference "compiler issues" for the reason you dislike languages like Java.  At first I thought he meant he was putting the java compiler through it's paces, perhaps adding weird arguments to javac, or running into the limits of the byte-code compiler.  Then I realized - he's simply talking about compiling the code! OMG!  Dude - you have trouble compiling a program?  Well - IDK - read a book or something, and fix your code.  Of course - the whole thing about Ruby is it brings back those interactive days when you tapped in something like "print 2+2" and the answer would pop out.  If that doesn't work, well, that's because of an "interpreter issue" I suppose.

His Ruby book is sprinkled with annoyances like "Ruby is an object oriented language.  Everything is an object.  Let me repeat that.  Everything is an object."  Sigh.  We, the reader, can read.  We got the point.  The first time.  But no, we are listening to a lecture, his listeners/readers are idiots, that need to be told twice.  Elsewhere, he says "And astute readers (and I'm talking about *all* of you) will notice that".  He does this at least *twice* in the book.  Yeah, great joke, got it the first time.  We are obviously not astute at all, thus a special point has to me made about this fact, many times.

But the most grievous flaw in the book is that it's a poorly written, thousand page tome in the format of one of those "bible" books with Everything and the Kitchen Sink thrown into it, like the entire Ruby API, and other stuff that gets out of date before it's printed.  So you end up with maybe 100 pages of real text.  And even that isn't too good, because it jumps all over the place - there's no real story line.

So nothing against the author, he's apparently done great things.  Certainly this is true in his own mind, since he calls himself the "cornerstone" of the Ruby community.  Surely that would be Matz for Ruby, but go elsewhere if you actually want to learn Ruby, or learn anything about programming.  Relegate the "Pragmatic" line of books to the "Dummies" series: It's ok, but there are better ones.  This is *not* Kernigan and Richies "The C Programming Language", or Gosling's The Java Language".  Those readers who are not "astute" or not requiring someone to tell them something twice, should moving on....

Learning Ruby by Michael Fitzgerald

This book hits the mark just right.  Fitzgerald aims it somewhere between the beginner and expert programmer, so it contains both introductory and more advanced material.  It covers the basics of the language including variables, conditional statements, Strings, Math, Arrays, Hashes, Files, and Classes.  It is a little over 200 pages, and that includes a chapter with extras such as XML processing, and another about rails.  I found the examples useful, and there was just enough text to explain the concept.  Therefore, it hit just the right balance, giving the Ruby beginner more context and helpful explanations than the O'Reilly Nutshell series, but without 500+ pages you probably won't need, like in the pickaxe book.  This book is written in a friendly manner, treating the reader with respect, and is concise and to the point.  

How to test the quality of the book?  Taking a real-world example and solving it is a great way to learn a language.  Armed with this book, I was able to quickly write a ruby script to solve a business problem, which included updating a database, in less than a day.  This provided a nice way to explore the language and the book.  With this small set of programs, a variety of interesting Ruby elements came into play, including Strings, regular expressions, SQLite API, method calls, closures, Classes, and file management.

This attests to the power and simplicity of Ruby.  Depending on how a Java programmer might chose to accomplish this same task, even with a fairly skilled programmer, this might have taken equal or more time, given dependency problems between various libraries, assuming an equivalent stack was installed as part of the task.

Granted, a programming background certainly helps, and there are many things you will skim through in this book, having a solid Java background.  Certainly there is much more to learn about Ruby, but to make a general hypotheses about programming Ruby, it appears to be a language designed to work in a natural way, with the programmer, so that it handles as you would expect it to handle.  One simple example of this is that when you require a variable, you simply use it.  No need to go back to some place in your class file, find the appropriate section (public/private) and declare it, and then recompile - it works the first time!  Now think about how many times this has tripped you up in Java.

Certainly there are likely to be areas covered that you will want to explore more on, such as closures, mixins, and exceptions.  Some sections are quite surprising.  A tiny excerpt about $LOAD_PATH?  Surely it can't be this simple?  Some things you will puzzle over.  All class variables are public by default - interesting.  And being able to drop out to the command line and try a line of code out?  Very nice.  This is especially useful when trying to get a regular-expression correct.  

Well, there you have it.  The perfect book to get you started in Ruby.  Stay tuned, and I will post more articles on the interesting and creative world of Ruby and Ruby on Rails!

Happy Trails!

[michaelok]

The 4-hour Automatic Backup System

Introduction:

Have you ever lost a document you've been working on in Word because you didn't turn on the "AutoSave" feature, or have you turned the autosave feature on, only to discover it turns Word, already a bit ponderous, into a very slow word processor?

Well, there's a better way.  Ideally, you should have some sort of backup system, perhaps a separate drive (it's easy with the Mac OS X and TimeMachine), or perhaps an online service like Carbonlite.  Definitely good ideas.

But with a little bit of time, and a few free tools, and the Windows Scheduler task (on Windows XP), or cron or similar on Unix/Mac OS X, you can configure your own automated backup system in as little as 4 hours.  Okay, maybe even less, but I liked the snappy title Tim Ferris came up, with a little research, for his 4-hr Workweek book (highly recommended).

Overview:

Long ago, I used a wonderful tool called “Lone-tar” (a play on the “tar” (‘t’ape ‘ar’chive) command in Unix. This tool was written by this guy in Maryland (despite the Texas/cowboy influence in the name). I’m not sure how big the company is now, but years ago, when I called support, a woman answered, and said, “Hold on, he’s out mowing the lawn, just a minute..”

The person that installed the software for a retail system I was supporting had set it up very simply, to do an incremental backup.  Finally, we started running out of space, and I got heavily involved in reviewing the configuration of lone-tar.  For one thing, the installer had never provided any instructions, and I realized we had a few years worth of incremental backups!  A restore would have been difficult - where was the master backup?

But the main issue was how to fit the backup on one tape.  (I realize a simpler solution would be to use the "reboot/reinstall" line of thought and get a bigger tape backup system.  But as folks in the industry know, sometimes it's not so simple to upgrade when - everything works.  So was there another way?  I started customizing the tool use bash scripts.

Rather than have some fancy GUI, everything with Lone-tar was command-line driven, and therefore endlessly configurable, but also very nice for the Unix programmer, being familiar old shell programming.  So what was brilliant about the tool was it was endlessly customizable.  Don't want to back up the entire drive?  You could specify 'include' or 'exclude' directories, customize which file types were backed up, which weren't, and many other powerful customizations.

Since then, however, I have gone towards a simpler, and faster backup system, which is to use the Mac OS X time machine and a separate drive. I have not researched it much, but there seems to be about zero configuration supported by Time Machine.  But fine, it works.  And if I run out of space, well, time to buy a new drive.

Well, that works nice, but what about Windows?  Note that lone-tar is only available on Unix and Linux, so even if you're on the Mac, you are out of luck.

After I unwittingly erased one of the interview notes I had been carefully crafting for a few hours on Wednesday, and after trying to find a backup copy, trying the 9 or 10 different ways suggested by Microsoft, and even using tools that apparently rebuilt the filesystem directly from low level calls to the Microsoft inode table, or whatever it's called in Windows, and failing, I decided to set up some form of version control that would be better than what Word provides.

Steps.

1.  There’s pretty neat tool developed by Linus Torvalds, the guy behind Linux, called “git”. This is used for version control.  Download and install this tool

 ======= Git ======= Learn how to use git.

Some git basics: 1. Start git shell (git bash)

2. cd to directory to version

3. git init // creates the git structure, but no files are added

4. git add . // adds all files recursively

5. git commit // commit changes

6. << do some more changes >>

7. git status // long version git status -s // summary git diff git diff --stat // statistics 8. git add . // now add next set of changes

9. git status // show what will be committed

10. git commit // or git commit -a, but still must add

11. git log --stat // show the commits

Next, create a schedule in Windows XP:

1. Start->Settings->Control Panel->Scheduled Tasks

2. Add a scheduled task

3. Set the "Run" command to: C:\WINDOWS\system32\cmd.exe /c ""C:\Program Files\Git\bin\sh.exe" --login -i c:\doc\backup.sh"

Next, create the actual backup script:

1. From the Git bash shell, fire up vi (a windows editor could be used as long as you make sure to correct the carriage returns)

2. Add the git commands to backup the files, for example:

#!/bin/sh

cd /c/doc // whereever you have the files you want to backup

git add . // add any new files

git commit // commit the changes

git log --stat > git.log // output the changes to a log file

1. Test the lines in the file to make sure they work

2. Then test the entire script.

a. Make a minor change to a file

b. run the script or line

c. use any of the git tools to verify the change was committed (git diff, git log)

3. Repeat 2, except instead of directly invoking the script, restart the computer and login

Have fun!

[michaelok]

Mac OS X User Installs Snort From Source

If you are coming from a Windows world and are new to OS X, you might be unfamiliar with the standard process for installing software when it is not available on iTunes, or you do not have a .dmg file, as you would for an Adobe upgrade, for example.  In this post, the goal is to educate the reader on the general process for how to go about installing an application on OS X by compiling it from the source code.  We'll use Snort in our example, however the general steps should be similar regardless of the software you are installing.

Now, while writing this rather long and wordy article, I did receive much better and simpler instructions from the good folks that develop Snort, so if you are reading this purely to figure out the simplest and fastest way to install Snort on OS X, here are the exact steps.  Otherwise, if you are interested in going a little more in detail and learning the fun things (and challenges) available on OS X compared to Windows, then the rest of this article might be useful to you.

Simple way to install Snort on OS X:

1) Install Xcode (XCode includes the compiler and other libraries required to compile Snort)

2) Install MacPorts from http://www.macports.org/

3) Install PCRE via MacPorts

'sudo port install pcre'

4) Install wget via MacPorts

'sudo port install wget'

5) Download Snort

'wget --no-check-certificate -O snort-2.8.6.1.tar.gz

http://www.snort.org/ports/snort-current/snort-2.8.6.1.tar.gz'

NOTE: The URL will change as Snort release numbers change, go to

snort.org to check the latest version!

6) Unpack Snort

'tar zxvf snort-2.8.6.1.tar.gz && cd snort-2.8.6.1'

7) Make Snort

'./configure && make'

8) Install Snort

'sudo make install'

At that point you should be ready to start configuring for a test run.

If your account has the 'Allow user to administer this comptuter"

check-box checked you really don't need to setup a root account, you

can just sudo for the parts you need root access and run as a normal

user otherwise.  It's recommended that you drop privileges on the

process at runtime if you're running a production sensor, if there's a

security bug in the code you don't want an attacker going after Snort

to get root automatically.

Learning how to install software on OS X by installing Snort

Installing software on a UNIX system like OS X is something most Linux, Solaris, AIX, and other UNIX users are familiar with.  With abundant varieties of UNIX, the particular binary for your version might not be available.  In this particular post, the idea is to give the reader, who may be used to installing a disk image (dmg) on Mac OS X, or using Install Shield to install something on Windows, a little insight into how to move up to the next level, and install software using the original source code.  We will be loosely following the instructions contained in Andrew Lockhart's "Network Security Hacks" book.  According to Lockhart, Snort is considered the "undisputed champion of open sources NIDs".

High-level Overview:

Here's a high-level summary if you want to get the quick overview of what we are going to do:

1.  Install and configure your compiler (gcc).

2.  Download snort

3.  Download pcre

4.  Configure your path so gcc is able to compile

5.  Set up the root user

6.  Run the configure and makefile scripts to install pcre

7.  Run the configure and makefile scripts to install snort

8.  Install the rules

9.  Fire up Snort for a quick spin!

Without further ado, here is our use case:

Use Case Name: Mac OS X User Installs Snort From Source.

Actor:

First we describe the actor:

Mac OS X Beginner User:

This might be of varying interest to folks with skills ranging anywhere someone familiar with Unix, but not Mac OSX, to a Windows user with some Windows programming skills such as Visual C++ or .NET.  Perhaps you've compiled CVS from scratch on a Solaris server, and you'd like to do the same for Snort on Mac OS X?  Or even someone simply interested in how one compiles and installs a nicely packaged Unix application.

Objective:

The main objective is to install Snort by compiling the source code.  Installing a binary is easier, but sometimes finding a recent binary is not easy, and furthermore, it may not have the best performance.  Here is a quote from the Snort Cookbook:

IDS systems are critical on efficiency. The precompiled packages are easy and quick, but they fail to optimize the system to your exact hardware. If you start to hit performance related issues with your binary install, try recompiling from source, which may solve the problem.

However, along the way, as an additional benefit, you will learn much that will be useful when installing any software from source on Mac OS X.  

Trigger:

[EDIT: More info on IDS and benefits]

You have a requirement to run an Intrusion Detection System on your network, and have chosen Snort.  However, there's no binary available for your particular OS, or you wish to compile from source to ensure good performance.

Precondition (Requirements):

You'll need some way to uncompress files.  This is Unix, so we already have standard tools for this - tar and gzip.  Simple instructions for using these are specified in the section on installing the rules at the end of this document.  On the other hand, there are nice GUI tools available, which make things a little simpler if you are new to Unix, and save you such questions how do I use tar, or gzip?  What are the arguments?  As this guide is intended to be a gentle introduction into both Snort and finding your way around a Unix system, we have strived to minimize the amount of different things you need to learn - that can come later.  Therefore, I like StuffIt, and OS X has a built in archive tool also, however it is not as powerful and flexible as StuffIt.

Ok then, let's get started….

Use case flow:

1.  Install Xcode.  If you are doing any sort of programming on the Mac, you might already have this installed, and it includes gcc, so is the copy of gcc I used.  You can certainly download gcc on it's own, of course, since the two copies should reside peacefully together, but I've taken the approach that, where possible, we try the simplest path first.  XCode takes care of installing gcc for us, although, as we'll see, there are a few changes that need to be made.  If later you realize you need a specific version of gcc than that which comes with Xcode, then you will of course need to download gcc separately.

2.  Download Snort (I am using 2.8.6.1),  expand it with some utility like StuffIt, and copy the directory from your downloads directory to a directory such as your Development directory.  Who knows, now that you have the source, perhaps you can contribute a patch to the snort team?

3.  Now we need to do some work from the command line, so we'll need to open a shell.  If you are a complete Mac newbie, here is how to do that:

From the finder, open up, got to your Applications->Utilities directory, and find terminal.app, and drag that to your "dock".  Clicking on that will open up a window which allows you to key in commands, much like you would at a DOS prompt.  This is a Unix Shell, which is, according to the Bash Reference Manual, "a macro processor that executes commands".

4.  In your shell, cd to the snort directory and run ./configure && make.  According to my O'Reilly book, that should be it.  In a few seconds, I will be off and happily running Snort?  The best laid plans.

Instead, the script fails with an error, and prints text similar to the lines below:

checking for gcc... no

checking for cc... no

checking for cl.exe... no

Executing your configure script (part 1: Configuring your compiler)

Let's see, checking for gcc... no.  That means it didn't find the GNU C compiler.  With the next line, it checks for a standard C compiler, I believe this is Sun's version.  Then it seems to think we might be on Windows and looks for  cl.exe, which is the Microsoft C++ compiler.  Ok, this script isn't too smart, we are not on a Windows machine.

A compiler - that would help.  That should be easy enough.  Surely Mac OS X has a C compiler?  I know it's somewhere, let's see..

We will simplify the install and put aside the question of *which* C compiler would be best to use.  GNU gcc is probably a good choice for most open source software because it is probably the de facto standard compiler out there.  One advantage is that it runs on nearly every version of Unix, and is well known and documented.  There are many other good compilers - IBM's XL C is a good choice for AIX, Sun's cc Sun Solaris, Intels (do they have one for the mac?).  But gcc is the standard, and many, many apps use this, so we'll go with that for now.  Why make things unnecessarily harder?  Now, if you realize later that it simply won't compile with your favorite compiler, we might have to come back to this question.  But for now, we are focusing on gcc.

Turns out, from a little web surfing,  that gcc should be installed with XCode

XCode installation guide is a little dated, but tells you where gcc should be if you installed it with xcode (I downloaded xcode, it was not from a cd):

http://developer.apple.com/mac/library/documentation/Xcode/Conceptual/XcodeCoexistence/Contents/Resources/en.lproj/Basics/Basics.html

So, now we know gcc is in the directory below:

 /Developer/usr/bin/gcc

1.  Ok, now we add it to our path:

crank up a text editor and create a file called .profile in your home directory (/Users/<username>)

Add the following lines:

GCC_PATH=/Developer/usr/bin

PATH=$PATH:$GCC_PATH

export $PATH

or you can of course do this from the shell, by firing up your favorite editor like vi:

cd ~ (to home dir)

vi .profile

and now you can add the lines.  If you're not ready to dive in to the intricacies of vi, then simply use your favorite text editor.

2.  To reload the updated .profile, type in the following command (you can also exit out from the terminal app and then reopen it):

source .profile

3.  Let's echo the path, to make sure it's configured right:

echo $PATH

You should see the gcc directory now in your path.  Good.  Now we have gcc there, and make sure we didn't spell a directory wrong:

gcc --version

excellent -- we have now have a C compiler.  Well, we always had one, now it's configured to run from the command line.

Executing the configure script (part 2):

First, what does the configure script do?  The top of the configure script for Snort states the following:

"Guess values for system-dependent variables and create Makefiles."

Well, literally, it runs whatever is in the configure script.  The idea for the configure script is to verify that all the supporting machinery and configuration is available to compile the source, and create a makefile with all the applicable compiler flags and other settings appropriate for the particular "target" environment, in this case, Mac OS X.   The configure script will take care of setting all the compiler flags to provide for the best performance possible, and this is why your custom-compiled binary will perform better than a generic binary that you've downloaded.  This is to accommodate thousands (well, almost) different varieties of Unix.  For example, for shared objects, what is the suffix?  Usually it is .so, but -- not always.  Another good example is that you can compile the application for particular chips, for example[EDIT: could use a few lines here to better describe the purpose of the configure script.]

Probably the best way to understand the configure script is to browse through one.  Therefore, browse through the file quickly to get the gist, here's an example:

   msvisualcpp | msvcmsys)

      # This compiler won't grok `-c -o', but also, the minuso test has

      # not run yet.  These depmodes are late enough in the game, and

      # so weak that their functioning should not be impacted.

      am__obj=conftest.${OBJEXT-o}

      am__minus_obj=

Marvel that there are actual useful comments in the script!  One of the nice things about working with open source code is to get the chance to work with nicely written code that is well-documented.

1.  From the snort directory, execute the following command:

./configure

Actually, if you want to save yourself a little work, send the argument --enable-zlib to the configure script:

./configure --enable-zlib

As we'll see later on when editing the configuration file, this is required.  You can type ./configure --help to see the various configuration changes available.  For example, if you will be logging to a database, you might want to add the --with-oracle=DIR argument.

Well, it still breaks, this time with the error "c compiler can't make executables see config.log"

So we google first, and for this bland, generic, cryptic error we get suggestions like "install XCode again".  Fair enough, given the limited detail.  Seriously, this is probably a good option.  On the other hand, you have a really new version of XCode, so you don't have that much faith in this.  Besides, it's kind of the "reboot your computer and try again" answer, unless the version of XCode really is different, or you do the install differently, you're not really doing anything differently.  And what is that saying?  Doing the same thing twice and expecting a different outcome is, uh, not too brilliant.  Something along those lines.

Right, instead, let's do this.  Let's see what the error is by reviewing config.log:

Looks like there is some confusion over what is an arg, but the big problem is gcc is failing to compile a simple one line C program.

Ok, let's try their simple test, and see if gcc really works at the simplest level, can we compile a 1 line program?

touch configtest.c, start up vi and add the line (or, again, using TextEdit or the like)

gcc configtest.c

--> here's the error:

ld: library not found for -lcrt1.10.5.o

the ".o" suffix tells us this is an object file.  [Add more detail here.]

It's failing in the linker (ld), which is not finding a library.  There is a dependency here, for some reason this file isn't there.  Go back to Google, maybe this additional detail will be more easily fixable than installing XCode again (although in many cases, this is probably the simplest solution, crazy as it seems - so we'll keep this idea as a strong plan b):

Ok, here's an interesting post on mac site, gives us the hint that the "-l" is an argument and the lib is actually crtl1.10.5.o.  I'm using the same platform (Mac OS X Snow Leopard, XCode 3.2.1).

http://code.google.com/p/remail-iphone/issues/detail?id=3

Hold on a moment.  At this point, you might be a little mystified at this point: So I'm downloading an App (XCode), with some tools, and the darned thing doesn't even compile a one line program?  If your standard configure script has this nice, simple test, why couldn't the XCode installer have done something similar, and updated the path correctly?  [Note: This is could be because the version of Xcode I used was somewhere between OS X 10.5 and 10.6.  I have since updated the paths to point to the 10.6 libraries, and it still works (simply changing from the MacOSX10.5 sdk to the MacOSX10.6 sdk).  I have not tried compiling without these paths.]

They show two workarounds, one involving setting up a link to point any references to 1.10.6 to 1.10.5.  First we need to find the library, and then we'll create a symbolic link.  [Edit: add post on symbolic links].  Wait a minute - maybe this is a path problem, and the library really is somewhere?  Let's see:

cd /Developer

# the following command is an immensely useful Unix command.  In this case, it tells use to find all the files matching the name "crt1.10.5.o", starting from the current directory, indicated by the "." symbol.

find . -name crt1.10.5.o -print

Here we are:

/SDKs/MacOSX10.5.sdk/usr/lib/crt1.10.5.o

It's in the 10.5 directory, but we are using 10.6, and with ls, we see there is no MacOSX10.6.sdk

Let's see where that is.

Wait a sec.

So maybe this directory is not in the path configured by XCode?  Like Gabor, I'm not an expert on gcc, so I'm not sure how we tell it where the right libraries are.  Let's see, in Visual C++, this would be through the options->Settings (or something like that), or in the VCVARS bat file for the command line, but that is the Windows world.  We'll have to figure out the equivalent for gcc.

Instead of our default response to any error being more googling, let's try a smarter search this time by bring up the manual page for gcc.  In Unix, we can instantly find a short summary about nearly any Unix utility by typing in "man" followed by the utility your are interested in.  The following command, for example, will bring up the "man" page for gcc:

man gcc

Another useful feature, once we are in the man page, is to hit the forward slash, and then type in some text we want to search for.  In this case, we want to search for "library".

Hm, now we now what the -l stands for - library.  Maybe we could pass the full path to this library?  Set that idea aside, easy to do, but it's likely there could be many other missing libraries, so better to set the path to a directory.

Ok, back to Google.  Using man has been useful as now we have some more information on the right search query.  Something like "gcc library path mac os x".  Sure enough, here's a link from StackOverflow.com:

http://stackoverflow.com/questions/1365211/error-in-xcode-project-ld-library-not-found-for-lcrt1-10-6-o

Most of the answers don't apply, since they are configuring the path through XCode, but here we are, Stefan has a way to do this via the command line.  Back to our profile, and we add the environment variables in.

#use MacOSX10.6.sdk if you are using > Xcode 3.2.1, which includes

#the 10.6 directories.

SDK_PATH=/Developer/SDKs/MacOSX10.5.sdk

export C_INCLUDE_PATH=$SDK_PATH/usr/include

export LIBRARY_PATH=$SDK_PATH/usr/lib

It's good to take a moment and post a comment at stackoverflow, or even simply using your "karma" to indicate that this answer was right (or wrong).  This requires us to login at stackoverflow, no anon permitted - argh.  (Stackoverlow is a bit overzealous sometimes.)  But it is worth it, let's contribute back, since we've learned something, right?

Compiling and Installing pcre (Perl-compatible regular expression library)

That complete, we run configure again, and then we pause to consider the new error message:

checking for pcre.h... no

   ERROR!  Libpcre header not found.

   Get it from http://www.pcre.org

Now, one thing that's very nice is that you get a great error message from the snort config script - what a welcome change from the typical cryptic error message!  Thank you, snort!  You are now congratulating yourself on having selected snort, as having good error messages (and good comments) in their install script indicates this code is well-thought out, and probably is nicely designed.  And with open source, we can actually verify this by browsing the source code.

Cool, this is the library mentioned in the library book, so we should somewhat expect this.  PCRE stands for Perl-Compatible Regular Expressions and allows a C program to do the same advanced regular expression matching that Perl is known for, and even uses the same syntax and semantics as in Perl.  Being a coder, you gambled somewhat on the possibility that this regular expression library might be included with OS X, and if not, the compile will fail.

Which it does, as we have seen.  Surely there's a version of pcre *somewhere* on OS X, that is available in my path?  In your best Steve Martin impression),  "but nooooooooooo".

Alright, no complaining - with Mac OS X you do not need to install another key library, which is called libpcap.  The O'Reilly book actually spends a paragraph going into a little detail on libpcap - which we happen not to need, so be content that running on this Unix flavor has gotten you a little bit at least.  [EDIT: this is a key library.  Add some detail, and a link to Martin's paper on Snort, which discusses libpcap.]

Besides, we've gotten the compiler running, and installing pcre should be very similar to snort itself, you're a seasoned pro by now - it'll be a snap, right?

So here we go, same thing, got to the directory mentioned above, and download the bzp file.

[Text like this could be in a sidebar.  It really isn't relevant to the main purpose of this document, but it is interesting by itself.  Perhaps the subject of another article.]

After browsing a few web pages at the PCRE site, you see they developed this for some mail program, and now, it's a widely used utility, far beyond their original expectations probably, and certainly much more used than their mail program.  Interesting, reminds me of Ruby on Rails.  No matter how big BaseCamp is, it will never approach the level where Rails is.  Do you use, or even know the names of any of their actual commercial software?  Ok, interesting, but let's keep focused.

We are close (I think) to getting snort up and running.  Well, let's say 80% - it's always that last 10 or 20% isn't it though?

Next, uncompress with StuffIt, copy it to your ~/Developer directory.

1.  cd to the directory and run ./configure

And voila, it works the first time.  Maybe we are really on our way here!  Nice.

NOTE: The following does not currently work, so stick with 32 bit for now!

For extra credit, since Snow Leopard supports 64 bit code, you can also compile the library and create a 64 bit version instead of 32 bit:

From the Apple instructions for GCC:

You must specify a 64-bit architecture (PPC64 or x86-64) with -arch ppc64 or -arch x86_64. You can also compile binaries with multiple architectures, such as -arch ppc -arch ppc64 -arch x86_64.

According to the PCRE Configure script help, we can set the flags as environment variables:

  CXXFLAGS    C++ compiler flags

Use these variables to override the choices made by `configure' or to help

it to find libraries and programs with nonstandard names/locations.

Therefore, we can add this environment variable to our .profile

export CPPFLAGS='-arch x86_64'

One thing though.  I wonder if this is in our build path now?  Is that done by the configure script?  Maybe, but you're not fluent enough about pcre, or configure scripts in general to know.  Let's take a look at the pcre readme file.  It happens to be very detailed and well written, and has some good suggestions on how you might want to tweak the build.  Past that section, it tells us what configure did.  Aha, here we go - run make install!  According to the pcre readme file, this will

"install PCRE into live directories on your

system."

Perfect!  Note to open source developers - this is how to write a readme file!  You should also note that, like Snort, PCRE comes with a wealth of good documentation, demos and test programs.

Ok - we didn't execute make yet.  Configure just, well, makes sure you have a compiler and things like that, and creates a makefile according to your environment.  But it didn't actually execute make yet, and build the libraries libpcre and libpcreposix that we need for snort.

So we run make.  And it fails.  Argh.  See, thought you were 80% of the way there, now we have a component that is failing to build, not even our main program!  Sigh.  So close.

Oh well.  So we walk up the output till we see what seems to be the first error.  Hm, can't find a standard C library, how can that be?

pcrecpp.cc:36:20: error: stdlib.h: No such file or directory

pcrecpp.cc:37:19: error: stdio.h: No such file or directory

pcrecpp.cc:38:19: error: ctype.h: No such file or directory

In file included from pcrecpp.cc:39:

/Developer/usr/bin/../lib/gcc/i686-apple-darwin9/4.0.1/include/limits.h:10:25: error: limits.h: No such file or directory

Ok, here we go, some path problem again, these are very core header files.  Standard lib and Standar I/O.  Even the most basic C program would not compile without these.  But if we had to add the library to our path, maybe we need to add a directory with the header files?  Again, I know how to do this in Windows Visual C++, so let's see if we can replicated this, following the library example above, in which Stefan showed us how to add the lib file to the path.  Wait a minute, wait a minute!  Didn't we just add the include path previously, per Stefan?  Did we key it in wrong somehow?

First, we need to find where these header files are.  Back to our find command:

find . -name stdio.h -print

Ok there it is, exactly in the path we have specified in our .profile.  We test the variable name in the following way:

ls -lrt $C_INCLUDE_PATH/stdio.h

and indeed, the file is displayed, so that's not the problem.  Back to the drawing board!

So what is going on?  It's almost like everything is running smoothly, and then somehow this path is wiped out - surely this header file is referenced in the makefile previously?

We go back through the readme more closely this time.  Anything about mac os x?  Nope, just some tidbit about hp.  Unix-like systems.  BSD.  Ok, review the configure output, anything blow up that looks important?  Nope, and more interestingly, we see configure is checking for some of the header files later claimed to be not found, very strange.  Here it finds stdlib.h, here it finds limits.h.  Back to readme - what about the very first item here, about using the --disable-cpp?  We certainly don't need that, aren't we using gcc?  On a whim, you try that - and it works!  Or was it merely that you ran it again and it worked?  Ran 'make clean'?  Ok, strange.  [Note: Using the newer version of Xcode (>3.2.1) I did not need to do this, but I am leaving this here in case anyone runs into a similar issue.]

Setting up the root user in Mac OS X

All ready, now we need to install as root.  Uh oh.  what's our root password?  If I su to root from my regular user account, it fails with a login error.  Wait a minute, I'm sure I configured root, why doesn't this work?  Time for the Apple manual, we hit google again.  (No wonder Google is making billions).

Here we go, a knowledgeable article on setting up the root password.  Looks fairly convoluted, so I won't waste time replicating the steps here.  Simply follow Apple's instructions, and note that once you authenticate, you need to enable the root user from the Directory Utilities *menu* bar (at the top), not in one of the dialog windows.  Or you can reset the password, if you've already enabled it, but never wrote it down, like someone did.

Log out of the admin account, log back in as your regular user, since you already have the profile configured for that user.  Huh, same error.  That's the right password, what's the problem?  Does my user need admin rights, you say?  Aha, let's log out of regular user, log in as the admin user - whatever user you have set up in your preferences, considering you have followed best practices by using a special "admin" account.  Now, from the shell window, copy the profile from your regular users home directory, execute source to run it, and then su to root.  Ahah, this time you get in.  Hopefully you will remember that next time ;)

You run make install for pcre, and it works.  Take a moment to look through the output as usual.  Hm, there's a message about the libraries have been copied to /usr/local/lib - is that a standard directory?  Well, it is on most versions of Unix, but it might not be for the flavor Mac OS X uses, so we keep that item in mind.  Anyway, pcre is installed and ready to go, you are now got your second wind back, it's the home stretch!  Whew!

Configuring and Installing Snort

Back to Snort.  Now run make.  Huh, no make file?  I thought configure worked, but make was failing?  Let's read through our notes (wasn't it a good idea making these notes?), did we ever actually run configure successfully?  Oh.  It gets a little confusing building two libs, and keeping them straight.  We did fail in the configure script for snort, so let's run that again, now that we should have all the underlying components ready.  Hm - what about the /usr/local/bin?  Being a programmer, we can run the script again and see if it works, at the relatively low cost of typing in the command.  Besides, we are also banking on the idea that the configure script is smart enough to know about the /usr/local/bin dir for pcre, after all, that's the purpose of the config script right?

And so indeed, it is, and it works!  Glance through the output, and we see there are a bunch of Makefiles created, including src/win32/Makefile.  That's interesting you say.  Isn't my configure script smart enough to indicate this is not a Windows OS so I have no need for a win32 makefile.  Interesting.  Well, in any case, we now are ready and have the makefiles.  Makefiles, in that it looks like rather than one grand file, we have a bunch of smaller makefiles, organized by directory.  That's nice, always good to split things up a little.

In your best Staples impression.. "That was easy".

Finally, let's install this.  From the two lines covering the install in our O'Reilly manual, we need to su to root - but you knew that already, from installing pcre right?  Excellent!

Still as root, we execute snort using our standard test, which is to print out the version:

#snort --version

This time, everything works and we are off and running!

Now, hold on a minute.  Surely you want to be able to do more with snort than print the version number, cool as that may be?  Well, it turns out we need to configure a few more things.  Snort is very nice in that you can do a minimal install (no web server, no database), but you do need some rules, and those are now a separate install.

Installing the Rules

So let's grab a copy of the rules.  You need to be a paying subscriber in order to get the most recent rules, but as a registered user, you can get the rules for free thirty days after they are released.

Now that you are a pretty familiar with Unix, we'll introduce a couple more to help us easily install the rules all from the shell.  The rules are downloaded with the suffix of "gz", meaning it is a compressed tar file.  So from the shell, in your download directory, execute the following command:

gzip -d snortrules-snapshot-xxxx.gz

This unzips the file and you'll have a file with the same name, except with the "tar" suffix, which stands for "tape archive".  A tar file is basically a bunch of files packaged together in one file.

Now copy this file to your base snort directory, and we'll execute the following command, which will add all the files and directories from the tar file to your current directory.  The -x stands for "eXtract", v for verbose, and f to indicate the file is the next argument.

tar -xvf snortrules-snapshot-xxxxtar

now you should have a "rules" directory,  Browse a few of the rules to see the structure and how they work.

Now we can really put snort through it's paces.

Let's try this simple command from the Snort manual:

snort -v

Halt little pot, halt!

We stop the process in the usual way, with ctrl-z.  Note this is not the command key, but the control key.  A summary will print out, showing a few statistics that were collected while snort was running.

Let's use netstat to get a little idea about which IP addresses are which:

netstat

Installing the Rules

Per the manual, we can test the configuration with the following command:

sudo -T -c ~/Development/snort-xxx/etc/snort.conf

ERROR: parser.c(5165) Could not stat dynamic module path "/usr/local/lib/snort_dynamicrules": No such file or directory.

Open up snort.conf and find the section on dynamic rules.  We're not exactly sure what's breaking, so copy each directory or file and see if it exists:

ls /usr/local/lib/snort_dynamicpreprocessor/

Ok, everything's fine except the last line:

# path to dynamic rules libraries

dynamicdetection directory /usr/local/lib/snort_dynamicrules

Hm, we did install a directory called "so_rules" when we downloaded the rules, but that's for the shared objects, and here it looks like we need the actual rules themselves.  Instead of Googling, let's do a "smart" search, searching directly for snort_dynamicrules on the Snort.org site.  About four links are returned for this search.  Here's an interesting link:

http://www.snort.org/snort-rules/about-so_rules

This article states:

This command tells snort to use the snort.conf file where it will find the dynamic rule files (thanks to the configuration options above) and then use those files to generate the stub files and put them into /usr/local/etc/snort/so_rules/

After this is complete, the rule files appear in the directory.

# ls /usr/local/etc/snort/so_rules/

We've already got the stub files in the right location, so perhaps we can comment out the line that is breaking.  One other thing we see from this article is that the snort.conf defines a bunch of includes for the dynamic library rules, and in our snort.conf these are commented out, so, now that we've verified we actually have the files, we can uncomment them.  It might be start to try one change at a time, rather than making both changes, but let's see what happens.  We comment the line out, and add a comment to indicate the change we made in case we have to roll it back later.  We also uncomment the last lines in snort.conf, where the includes are defined.

#MOK 2010-Sep-14 Comment out line below since no such directory

# and we already have stub files created

#dynamicdetection directory /usr/local/lib/snort_dynamicrules

[snip]

# dynamic library rules

# MOK - 2010 Sept 14 - uncomment

include $SO_RULE_PATH/bad-traffic.rules

[etc]

Excellent!  Snort is able to get much farther in processing the snort.conf, but it still fails with a new error.  Again, not the nice error message, giving us a line number and an error message in English, instead of something like "NullPointerException".

ERROR: ./snort.conf(195) => Invalid keyword 'compress_depth' for 'global' configuration.

Fatal Error, Quitting..

Back to the snort search.  This time only one link is returned, which is a link to the Snort manual.  In the manual, we find the following:

[compress_depth <num>] [decompress_depth <num>] \ disabled

For some reason, our build seems to not understand compress_depth.  We observe that this can be disabled, so maybe it can be removed without any ill effects, but that should be the last option.

Instead, googling tells us we needed to run configure with the enable-zlib argument:

./configure --enable-zlib

Before making that change, while we have the snort.conf open, there are a couple minor tweaks we can make.  We note that somehow the stream5_tcp policy setting is set to "windows", we see from the README.stream5 that this can be changed to "macos".  Similarly, the frag3_engine is set to "windows", we see from the associated readme file this should be changed to "first".

Then run the makefile again, making sure to "clean" so that the complete build will be done.  Otherwise, Make is smart enough (or lazy and dumb enough, depending on your perspective), not to waste time recompiling any files it doesn't think it needs to.  And only some scripts changed, like the makefile, and none of the source files changed, nothing will get built and the compile will run very quickly.  Ordinarily, if you only change a file or two, this is what you want to happen, but not in this case.

make clean

Now you are ready to install, and this time it does work successfully, so we are *finally* ready.  According to the O'Reilly book:

sudo snort -Dd -l /private/var/log -A console -c ~/Development/snort-2.8.6.1/etc/snort.conf

A brief description of the arguments:

We are executing Snort with "sudo", which allows Snort to run with Administrator privileges.  [EDIT: Per Martin's e-mail, as a security item, should drop admin privileges, once we have everything configured.  For one thing, we need to set up a log directory that Snort can write to without root privilege.]

The -D tells snort to run in "daemon" mode, so it runs as a background process.  Even if you log out, Snort will continue to run until you shut off the computer.

The -l tells Snort where to send the logs

-c tells Snort to use the configuration file we've updated to fix a few minor items

Instead of printing the startup information to the console, the output will now be directed to the system log.  

A good way to browse the logs is to use the Console application, which can be configured the same way as you set up the Terminal application.  Under /private/var/log, you should now see the Snort logs.  

If permissions are not set correctly, you will see a message that says "You do not have permissions to read this file" and the display will be blank.  Also note that the logs are in binary format by default, so see the command below for how to display the log in a readable format.

If you go to a shell, you can use the tcpdump utility to view the logs:

tcpdump -r snort.log.xxx

If you don't see any logs, check the system.log for any startup problems with snort. You should see a successful message like the following:

Snort initialization completed successfully

Another useful log is the "secure" log.  This tells you security events, such as if someone tried to run su but failed.

The Grand Conclusion

And that is the mere tip of the iceberg.  Start with the manual above, or a good book like the O'Reilly Snort Cookbook, or Brett D. McLaughlin's article on configuring Snort, Richard Bejtlich's excellent Snort Report series, and continue from there.

Happy Snorting!