pr0tectus

[pr0tectus]

Protectus is Shutting Down

This email was sent to all of our customers on July 1st, 2020:

Good afternoon,

Please accept my apologies in advance.  This is a long and perhaps meandering email.  The short version is this: Protectus will cease operations in December of 2020. The rest of this writing is to enumerate the recent history leading to this decision, and to attempt to answer some of the questions you may have.

As you know, Pete Garvin, founder and chief engineer of Protectus, died last year after a short battle with an aggressive cancer.  In the immediate aftermath of his death, our concern for the company was to make sure that our customers were not left hanging.  We worked closely with our longtime partners at Palitto Consulting Services to hand off to them all client-facing operations, and with their rock-solid support managed to keep the lights on.

After a few months of professional and personal survival, we were able to give more thoughtful consideration to the future of the company.  We consulted with our friends and business partners.  We prayed.  In our deliberations, a number of determinations were made around the skills needed to keep the company going, the size and type of the business that Protectus represents, and the way Protectus looks to outside investors.  

We recognize that Protectus, and in particular the Sentry, is a unique technical animal.  While the value proposition of our product and services is straightforward, the IT infrastructure, Sentry development life cycle, and corresponding operations are very deep and idiosyncratic. To really own the Protectus technologies, you need a good understanding of everything from the Xen hypervisor and management of a fleet of VMs, to SSH tunnels, Debian packaging, Bash scripting, Python webapp development, MongoDB, Suricata Intrusion Detection, Cython C extensions, PF_RING, d3.js, Javascript frontend development, tcpdump, a very deep understanding of both IP and transport layer protocols, to say nothing of actual information security.  Under normal circumstances, it would take a team of people to keep up with what Pete had been doing on his own.  A first large concern, then, is that filling technical shoes so deep and so idiosyncratic would be difficult, at best. 

Pete played the role of IT lead, security engineer, software engineer, account representative, everything.  While he tried to grow the company larger than himself from time to time, like many engineers sales was the least favorite part of his work. As a result, Protectus grew to a point where it sustained him and his family well, but not to the point where he could comfortably hire full-time help at market rates. This points to a second determination, that the profit realized by Protectus is not very large by the standards of many technology ventures.

These two determinations point to the last.  In our conversations with advisers, friends in the industry, and candidate buyers, it is clear that the potential profit available through Protectus does not meet the amount of risk and investment required to keep it running.  We believe the best option is to wind down the company gracefully before technical and organizational debts cause significant service outages for our customers.

There are some specifics and details to be addressed:

What will happen to my sentry? Your sentry will disconnect from the Protectus for the last time in early or mid December.  By default, we will shut down the Sentry, but can leave the Sentry on if requested.

What will happen to my weekly traffic analysis? Weekly delivery will cease on or around the end of November. I've spoken to almost all of the customers who have this level of service, but if I've missed you or if something remains unclear, please reach out to me.

What will happen to the Sentry hardware? Protectus intends to abandon the hardware.  If you desire to ship the hardware to us, we can provide an address for you to do so.

When will my bills stop? We intend to send out the last invoices in November.

Can I use the Sentry after Protectus ceases operations? The Sentry was not designed to be used without periodic updates from Protectus, so we cannot advise you to use the Sentry out of support.  That said, if you do choose to run the Sentry after December, we expect that the underlying Debian Linux OS will continue to receive automatic security updates until June of 2022.

Dad started Protectus in 2003.  17 years is not that bad of a run.  Thank you all for your patience as we've worked to determine the correct path, and thank you so much for your years of working with Pete.  He valued his relationships with you dearly, and thought of many of you as friends.

Posted by Tim Garvin

[pr0tectus]

Workplace [Cyber] Safety

  In March of 1867, a 29-year-old Scottish immigrant was working in a factory in Indianapolis. His right eye was injured in a workplace accident and he subsequently became completely blind. John Muir eventually recovered his sight, became a well known naturalist, and was instrumental in the creation of Yosemite and other national parks.

Workplace safety has come a long way since then. Even though everyone wants a safe work environment, ongoing education and training are still needed. Why? Partly because new employees enter the workforce and partly because experienced employees forget. In our human nature, we sometimes lose track of what is important by the distraction of what appears to be urgent.

So we should not be surprised that safety education and awareness training are needed in the relatively new cyber world where unfamiliar threats can lurk unseen and unheard. According to the 2019 Verizon Data Breach Investigations Report (DBIR) phishing is the top threat action in cyber breaches.

How can this education and training be accomplished? Two good options are phishing simulation and cyber awareness videos.

Phishing simulation involves sending spam-like emails to your organization. A training opportunity arises when someone clicks on or responds to the email. Many tools and services are available to send simulated phishing emails with some entry-level options available at no cost.

Cyber awareness / training videos have come a long way. Some of them are actually entertaining! Do a web search for 'cyber security awareness video' and you'll find plenty to try out.

Some cyber security awareness training services integrate phishing emails and training videos into one convenient package.

Cyber safety today is where workplace safety was many decades ago. We're improving but we still have a long way to go. Recognizing and accepting the need is the first step towards a safer cyber workplace.

  Posted by Pete Garvin

Photo by Tzogia Kappatou

[pr0tectus]

Why Cyber Safety Will Improve

Photo by Jan Vašek.

A childhood memory I have from years ago is traveling along the Ohio Turnpike in my parent's station wagon. The speed limit was 70mph so, just like today, we were probably traveling faster than that. Unlike today, not a single person was wearing a seatbelt. The car had no air bags or other safety features we now take for granted. Child car seats were practically unheard of. Other cars with the similar safety features and habits were speeding past us.

This memory is almost unbelievable by today's standards but it was the norm back then. Motivated by the human pain and suffering from traffic accidents, our collective habits and expectations have changed. Thankfully, automotive safety has improved.

Similarly, cyber safety will also continue to improve. Younger generations are more savvy about online threats and scams. Cyber security regulations are being introduced in more industries. Mechanisms to patch software vulnerabilities are becoming more streamlined. Many colleges are introducing cyber security curricula.

Some people still do not wear seat belts and there will always be people who knowingly take unnecessary risks - human nature has not changed. But cyber education and awareness can reduce the number of people unknowingly taking online risks.

Many education and awareness resources are available and some are even entertaining. Do a web search for 'humorous cyber security awareness video' and watch one. Then, share it with others.

Together, we can keep public cyber safety moving forward.

Posted by Pete Garvin

[pr0tectus]

Defensive Manufacturing

Posted by Pete Garvin

Photo by Şafak Cakır

While listening to MAGNET’s Bob Schmidt speak, I heard a useful analogy comparing new cyber security requirements with ISO. You might remember about 30 years ago when many organizations were just coming to grips with ISO 9000 compliance  An attitude common at many organizations was, “If we ignore it long enough, it will go away.”  It didn’t go away. 

In a similar way, manufacturers for the US Dept. of Defense are facing compliance with Defense Federal Acquisition Regulation Supplement (DFARS) cyber security requirements.  Spoiler alert - just like ISO, DFARS will not go away.

A January 2019 memo indicates how these new cyber security requirements will be enforced. The memo states that the government will "...leverage its review of a contractor's purchasing system....in order to....assess compliance of their Tier 1 Level Suppliers with DFARS...."

Some homework may be needed to understand what DFARS means for your organization and what your compliance timeline looks like. Talk to your contracting officer, customers, and technology staff. We’re also available to help.

Industries like finance and health care have been navigating cyber security requirements for a long time.  Other industries are following suit.  Even though you may not be facing imminent loss of business, why not take steps to be prepared for the future?

If you haven’t already done so, start the discussion.  If you’re not already on the journey, then take the first step now.

[pr0tectus]

Not All Cyber News is Bad

Image by David Carillet

The Ohio Data Protection Act (Senate Bill 220) went into effect on Nov. 2, 2018.  This legislation offers a unique, positive incentive for businesses to proactively improve cyber security.  Specifically, civil liability from a data breach can be negated when a business implements and maintains a cyber security program that reasonably conforms to an industry standard framework.

Want more legal details?  Check out the many law firm blog posts on the topic.  We’re going to touch on some of the 11 designated frameworks.

For those in certain industries (e.g. financial, healthcare, and retail) the acronym-filled framework names (e.g. GLBA, HITECH, and PCI) will look familiar.  Not familiar with these acronyms or subject to these frameworks?  Then the Center for Internet Security Controls is a great starting point.

Why so many frameworks?  Consider an analogy with building codes.  There was a time when each locality had their own building code.  Over time, a more uniform national code came into being and was adopted by local municipalities.

Lets hope that similar consolidation and simplification will occur with the cyber security frameworks as there is certainly a lot of overlap.  Cyber security fundamentals don’t change when applied to specific industries.

If you are new to all this and feel overwhelmed, there is more good news - you already have some controls (safeguards) in place.  Start by identifying what is already in place and then put together a plan to fill in the gaps.  Sure, we can help as can other companies but your organization knows its technology infrastructure and processes better than anyone. Use that knowledge and get started.

No matter where your organization is located, your cyber security posture can benefit from following a framework. If you're located in Ohio, then you can also reap the legal benefits provided by the Ohio Data Protection Act.

[pr0tectus]

Warm Up to a Security Freeze

Posted by Pete Garvin

Photo by Larisa Koshkina  

The latest cyber-security breach headlines catch our attention briefly but then fade from memory.  Life goes on and everything seems fine ..... You received no unexpected charges or payment notices; your tax refund wasn’t claimed by someone else, and you're not aware of traffic violations from a fake driver's license in your name.  A little voice says, “Maybe I dodged a bullet and am not affected.”

After 2017, it’s time to wake up and smell the burning electronics.  Your personal information is probably for sale on the dark web and if not, it likely will be soon.  Get over it and move on.

Haven’t noticed any problems yet?  Don’t get too complacent.  Consider a few possibilities:

There is a glut of stolen identity information.

Your identity is being used but the symptoms are still below your radar.

You are really, really lucky and your luck will never run out.

Stolen identity information becomes available as a commodity on the dark web.  An identity is eventually purchased by someone who will ‘invest’ time to exploit the information before receiving benefit at your expense.  From the perspective of the identity thief or attacker, it’s a business.  The millions of identity records known to be stolen in 2017 are in various stages of the exploitation pipeline.

So what to do?  One important step is to place a security freeze on your credit files.  When an identity thief applies for credit in your name, the potential creditor will check your credit file.  A freeze will prevent the credit check from occurring and therefore stop the attempted application.

Rather than repeat a lot of details on how to initiate a security freeze, here are two excellent blog posts on the topic:

https://krebsonsecurity.com/2015/06/how-i-learned-to-stop-worrying-and-embrace-the-security-freeze/

https://www.sans.org/security-awareness-training/blog/what-communicate-about-equifax-hack

There are some downsides - paying fees to initiate the freeze, unfreezing for legitimate credit checks, and keeping track of the PINs or passwords needed to unfreeze. For me, these are minor disadvantages compared to cleaning up an identity theft mess. There are also less effective, but more convenient, options like fraud alerts.

Take a few minutes, get informed, decide what's best for you, and most importantly, take action!  You’ll be glad you did.

[pr0tectus]

Where is The Cloud?

Posted by Pete Garvin

Photo courtesy of NASA 

Cloud /kloud/ noun

Condensed water floating in the atmosphere

A computer belonging to someone else

A computer belonging to someone else - this is a great definition of the cloud. Using someone else's computer has become commonplace. We enjoy the benefits and we also accept the risks.

Ever wonder where all those computers are? We did, so we let our deployed Sentry devices collectively tell us where they see the cloud.

Hover over the interactive map to see results for each country

Geographies with minimal land area and substantial traffic are unfortunately not well represented in this type of infographic - Hong Kong and Singapore are two examples. Both have artificially modified coordinates for easier location but can still be difficult to find. For a better view, open this interactive map in a new tab.

A few lessons emerge from this visualization. First, you may be surprised to learn how many countries your computer communicates with. This is partly due to advertising content, software updates, and other legitimate activity.

Second, some countries may rank higher than expected. The top traffic source for Ireland explains why it ranks as the #2 country.

Third, the answer to 'Where is The Cloud?' depends on your vantage point. Just as a view of clouds in the sky depends on your location, the map will look different depending on your network's traffic.

Network traffic visualization is a tool for for patrolling your network. A better understanding of 'normal' allows for easier identification of 'abnormal'. Learn more about identifying abnormal activity on your network.

[pr0tectus]

Detecting Ransomware on The Network

Posted by Pete Garvin

Photo by Joel Garvin 

A simple action - website visited, email opened, mouse clicked. Later, you notice that something is wrong.....why can’t I get to my files anymore?

Ever felt the pain of ransomware? If so, you understand. If not, you are fortunate but not immune.

In case you don’t know, ransomware is a type of malware that encrypts your files and provides instructions on how to purchase the decryption key - that’s the ransom part.

There are defenses but as always, nothing is perfect. The best defense starts with good online habits but anyone can make a mistake.

Since security is best implemented in layers, detecting ransomware from the network traffic it generates seemed interesting - another layer of security. Also, it was a good topic for a paper to renew my SANS Incident Handler certification.

The paper is about ransomware identification using network traffic and is available for download, but if you’re not interested in all the gory details, the idea is to divide ransomware-generated traffic into three groups depending on its destination:

Internet

File server

Broadcast

Traffic analysis is then performed to find patterns in each group. If you want more then that, the paper’s introduction and conclusion sections are your next steps.

The bits zipping around on your network may seem very obscure and hidden, but they can provide real insight into what is actually happening. It's just a matter of transforming those 1's and 0' into human understanding - something that will become more commonplace as technology improves.

[pr0tectus]

Network Traffic by Country

This video demonstrates network traffic analysis by country. You frequently access computers around the world whether you know it or not. Surprised? See for yourself!

Trouble viewing this video? Try http://www.youtube.com/protectuschannel

[pr0tectus]

What’s Your Threshold?

Don’t want to read this introduction? Cut to the chase and download the Thresholds and Monitoring white paper now.

Photo by Joel Garvin 

We’ve all been there - you’re being poked and prodded at the doctor’s office. But how is a diagnosis made? To start with, simple stats like height, weight, temperature, pulse rate, and blood pressure are taken.

In a similar way, simple stats can be measured on your network. Simple stats are useful, but just like at the doctor’s office, they don’t tell the whole story.

Would you like a doctor that diagnosed only using simple stats like height, weight, temperature, pulse rate, and blood pressure? Alternatively, would you go to a doctor that never used these simple stats?

How do we know if a simple stat like blood pressure might indicate a problem? By using thresholds. If your blood pressure exceeds a threshold, then more follow-up is needed.

Knowing exactly what’s happening on a computer network is difficult with billions of bits zipping around at light speed. So we sometimes use simple stats, like peak bandwidth for example, to extract useful insight. Just like with the medical analogy, we establish thresholds to help indicate a potential problem requiring follow-up.

Simple stats and thresholds are useful but understanding their limitations is important. To better understand the effect of thresholds on network insight, check out our Thresholds and Monitoring white paper.

Don’t want to register with your contact info? No worries - registration is optional!

[pr0tectus]

The Sentry - A Quick Look

This video introduces the Protectus Sentry - a network traffic visualization and analysis tool useful for security, performance, and troubleshooting. You know what a security camera and DVR are. The Sentry is like a security camera and DTR (Digital Traffic Recorder) for your network traffic. Have a look!

Trouble viewing this video? Try http://www.youtube.com/protectuschannel

[pr0tectus]

Monitoring vs. Insight

Don’t want to read this introduction? Cut to the chase and download the Network Monitoring vs. Network Insight white paper now.

The phrase ‘network monitoring’ can mean so many different things. To illustrate this fact, lets use a sports analogy. Think of a few ways you might follow along with (i.e. monitor) your favorite sporting event:

Periodically check the score

Receive a text when the score changes

Listen to a play-by-play announced on the radio

Record and watch on a Digital Video Recorder (DVR)

All of these can be considered ‘monitoring’ the sporting event, but which ones give more insight?

Now lets complete the analogy by taking the four sports examples listed above and connecting each one back to the world of computer networks:

Periodically check the score: Simple Network Monitoring Protocol (SNMP), more accurately called Infrastructure Monitoring.

Receive a text when the score changes: Syslog and event management.

Listen to a play-by-play announced on the radio: NetFlow, IPFIX, or related vendor-specific flow protocols.

Record and watch on a Digital Video Recorder (DVR): Capture packets from the network and create interactive visualizations.

Each of these ways to monitor a network can be valuable. To understand the pros and cons of each option, check out our Network Monitoring vs. Network Insight white paper.

Yea - we don’t like to give out our contact information either……..so registration is optional!

[pr0tectus]

Network Doors vs. Physical Doors

Posted by Pete Garvin

Security cameras are a simple, cost effective way to help secure a building's door.  If a break-in occurs, the security camera shows what is happening and what did happen.  It provides visual information to fill-in the details.

What if a break-in occurs on your network?  We don't normally talk about networks having doors, but networks do have virtual openings for packets to flow in and out.....a door of sorts.

So if you know or even suspect a network break-in, an unexpected opening of your network 'door' so to speak, what can fill-in the details?  What can provide visual information to show what is happening or did happen?

Visualizing the packets flowing in and out of a network 'door', is analogous to the security camera.  Traffic analysis means reviewing the network traffic visualizations to understand what happened, plan a response, and verify the response was effective.

Let's take a simple password guessing attack as an example.  Notice the rectangle-shaped, dark-blue area in the bandwidth graph below which represents a sustained increase in network traffic. Data used to create the bandwidth graph is shown in a text-table format beneath the graph.

Sustained traffic increase starting early morning on Saturday January 3 and continuing into the afternoon. Country column shows geo-location of the corresponding Source IP address.

User interaction allows for a deeper analysis of the sustained traffic.  Selecting the first row in the table highlights that row along with the corresponding traffic in the graph.

Below is the same graph shown above, only this time the traffic from a particular Source IP address has been highlighted.

Password guessing traffic highlighted. Attacker using IP address in Taiwan is guessing passwords on a Secure Shell (SSH) server using the default port 22.

The highlighted rectangle in the graph was caused by password-guessing attempts.  There are no legitimate users in Taiwan for this Secure Shell (SSH) server. Since no human user will re-enter their password continuously for twelve hours, the attacker must be using an automated tool to guess passwords repeatedly.

Adding in a graph of Intrusion Detection System (IDS) alerts provides additional insight. The login attempts are occurring frequently enough to trigger IDS alerts. Notice how the IDS alerts correlate in time with the traffic.

Can you see the second, shorter-duration password-guessing attempt in the IDS visualization below?

Highlighted IDS alerts (bottom graph) correlate with highlighted password-guessing traffic (top graph).

Notice that the both the highlighted network traffic and the highlighted IDS alerts have the same Source IP address. A review of the SSH server logs (not shown here) reveals that this particular attacker was attempting password log-ins with three usernames:  root, www-data, and nobody.

Visually understanding what is happening and did happen at your network door is just the beginning.   Check out our website for more examples of the benefits of network traffic analysis and visualization!

[pr0tectus]

Bandwidth Usage When Streaming Video

Posted by Pete Garvin

Ever wonder how much bandwidth is consumed by a video stream?  This seems like a simple question to answer.  Just watch an online video, measure the number of bits flowing during a certain time window, and do the math.  You know, bits per second - simple right? Maybe, maybe not.

Let's give it a try.  The graphic below shows a video stream from watching a football game online.  

Bandwidth usage for a football game on espn.com. Each vertical bar is approx. 1 minute wide. Brief times of low bandwidth around 8:30PM and 9:15PM are when our team fumbled and we paused the game in frustration!

It seems pretty straightforward. The bandwidth averages around 1.5 Mega-bits-per-second or Mbps and has some occassional spikes.

But think for a moment about the time scale or x-axis on this graph. At this resolution, the width of one vertical bar represents about 1 minute. But the bar height or Y-axis is in units of Mbps. So each individual veritcal bar represents the average Mbps for that minute.

This importance of time scale becomes clear if we zoom-in to a point where the width of each vertical bar represents 1 second.

Bandwidth usage for football game from espn.com. Each vertical bar is 1 second wide. The data's bursty nature is obvious at this time scale.

At this higher resolution, it becomes clear that the bandwidth is not really 1.5 Mbps but actually varies from near 0 to spikes well over 5 Mbps.  Similar results are seen when streaming video from other sites like Netflix and YouTube.

Of course if the bandwidth did not drop to near zero, then the first graph and the second graph would look more alike. This is actually what happens when spikes occur in the first graph.

You might be thinking.....why should I care if application data is bursty?  If you don't have any realtime traffic on your network - like Voice Over IP for example - then maybe you don't care.  But to solve problems like VoIP call quality and other intermittent problems with realtime applicaiton traffic, a detailed understanding of what's really happening on a network is important.

Check out our website to learn how the Sentry can show you what's really happening on your network!

[pr0tectus]

Who is Picking the Lock on Your Network?

Posted by Pete Garvin

Can anyone on the Internet send packets onto your network? Does your network have an opening to the Internet.....a virtual door of sorts?  If so, is someone trying to pick the lock?

The answer is yes if any Internet-accessible web sites, applications, email servers, or other types of servers are hosted on your network.

The answer is, or at least should be, no if Internet-accessible servers are not hosted on your network. Even then, remote access mechanisms can still leave an open virtual door but that's a topic for another blog post.

If your answer is yes or if your not sure, that leads to some questions which have traditionally been difficult to answer. All the silent and invisible data zipping and zapping around at light speed makes it difficult to see:

What the doors to your network are.

Who is using the doors legitimately and who isn't.

Where Internet user's and attacker's computers are located.

When are the doors being used.

How much traffic is flowing through the doors.

The Sentry can help answer all these questions but in this blog post, we're going to focus on Who and Where.

Let's talk about incoming connections. Any Internet-connected network is constantly being probed by attackers. Once a virtual door is found, an attacker will rattle the door knobs to see if the door is locked and check how easily the lock can be picked. The attacker's rattling and picking blends in with legitimate user's traffic and appears on your network as incoming connections. When someone good or bad wants to get on your network, they do so by initiating an incoming connection.

This may all sound very abstract so let's look at a specific network as an example - we'll call it network A. This network is in the United States and hosts an Internet-accessible email server (port 25) and a Remote Desktop server (port 3389).  You might see those destination ports (Dest. Ports) again so keep them in mind.  

Incoming Connections view for network A showing Destination Port and source Country Note: This graphic is from a production network so we hid the IP addresses.

You might be thinking that putting a Remote Desktop server directly on the Internet is not a best practice.....and you're right.  But we all know that we don't live in a perfect world.  Sometimes an administrator inherits a network, sometimes meeting a business need requires an out-of-the-ordinary solution, sometimes convenience trumps security.  Every organization has opportunities for improvement.  Identifying those opportunities and encouraging our customers to move towards them makes us (and you!) part of the solution.

With a few mouse clicks, we can eliminate connections originating from within the United States to show only non-US traffic which simplifies the example.

Non-US Incoming Connections for network A showing Source IP address, Destination Port, and source Country Note: Internal IP addresses are still hidden but we don't care if you see the external (attackers) IPs.

Check out this video showing how the Sentry allowed us to easily transition from the first graphic to the second graphic. The video also shows how highlighting can be used to visualize the traffic distribution over time for each Source IP address.

What really stands out visually is the ongoing Remote Desktop (port 3389 - remember?) traffic from many Source IP addresses around the world. These are login attempts. Attackers are trying to open the virtual door, pick the lock so-to-speak, by guessing usernames & passwords. If the target were a web application, a different type of lock picking would occur. Low-level, unseen, ongoing  probes / attacks are part of life on the Internet and time is on the attacker's side. If you really don't believe us, just throw a Sentry on your network and see for yourself. By the way, visualizing incoming connections is useful for educating decision makers on the reality of network security threats.  People tend to be come very interested in security upon seeing their network being attacked.  

In the absence of painful and expensive consequences of a successful breach, it is easy to make the mistake of thinking that:

We just make widgets.....who would want to bother our network?

There are so many computers on the Internet.....we'll just blend in.

I don't need a strong password.....security by obscurity is good enough.

The good news is that fundamentals go along way towards preventing your network from being low-hanging fruit for attackers.  Fundamentals include keeping systems patched, configuring networks properly, having good computer-use habits, and using strong passwords.  

Just as a point of comparison, let's look at the incoming connections on another production network we'll call network B.  This network is also located in the United States and is configured to allow incoming connections to a Remote Web Workspace server on port 443.

Non-US Incoming Connections for network B showing Source IP address, Destination Port, and source Country

Note how there are no hours-long attempts to open network B's virtual door.  Why do you think this is?  Stay tuned-in for a future blog post explaining why.

Until then, check out our website to learn how the Sentry is making unprecedented network security, performance, and troubleshooting insight a reality!

[pr0tectus]

What the ALS Ice Bucket Challenge does to a Small Office Network

Posted by Tim Garvin

This week, the ALS Ice Bucket Challenge swept though many of our social circles here at Protectus.  (And yes, some of us took part.)  What's interesting is the effect the Challenge had on our network performance.

One of our office networks can be likened to many places of business: a handful of humans with two or three network devices each. The data usage isn't obscene, and we survive happily with a modest pipe to the internet.

Last week's internet traffic for the network in question. I've got $10 that says you can't find the ALS-related traffic in there -- yet.

Now, a small handful of times this week, the network became just oppressively slow.  You know what I'm talking about.  VoIP calls going bad, sluggish email refreshes, web browsing, chatting, everything just being held back for a split second or two, or three. Although I am a good and patient man, I began to wonder just what the heck was going on.

Enter the Sentry!  One of the features of the Sentry is simply this: it will ping the host(s) of your choice indefinitely, and graph the round trip time.  We like to call these "Ping Graphs".  As luck and careful planning would have it, we maintain a Ping Graph to the Google public DNS machine at 8.8.8.8, which typically gives us a pretty stable read of how our internet pipe is doing.

Last week's traffic, along with last week's ping graph. Note that in the Sentry UI, all currently displayed graphs are lined up in time.

See those spikes?  Those are times when the round trip time to 8.8.8.8 has risen to an average of hundreds of milliseconds!  Beyond that, if you look closely, you can see from the light-pink ranges that pings were seen as high as 500ms.  If you've ever played online games with friends, you know that such long ping times can be deadly.

What's causing these dastardly delays? When we compare the ping graph to the traffic graph, it's not clear that there's any correlation at all! This is useless!  Or is it?  Let's take a closer look at one of those spikes.

"Zoom, Enchance. Zoom. Enhance."

Here's where we end up:

We're zoomed in now.  Note that the highlighted row is also highlighted in the graph, so we know what we're looking at.

Lets analyse this data:

On top, we see the traffic (blue) on our internet pipe.  We also see some traffic that's highlighted.

Next, we see the ping graph (purple) to Google.  Look at that high latency!

Now, in the table, we've highlighted the top row.  The highlighted table row corresponds to the highlighted traffic graph.

The highlighted row has a little black arrow pointing from the source address to the destination address.  This means that most of the traffic is going out, not coming in.

Finally, we see from the tooltip that the IP address being talked to is a Facebook IP address.

Ta-da! An upload to Facebook correlates perfectly with our network issue. But why does an upload to Facebook drag down our network?  After all, we can clearly see lots of other traffic going on throughout the week.

The answer is Asymmetric DSL, which (for today's purposes) is another way of saying that our beloved Internet Service Provider has sold us a plan that provides more download bandwidth than upload bandwidth. 

Most of the time this is fine, as we don't upload much besides code around here, but sometimes (when half the people on the network pour ice water on their heads and post the video online) it wreaks havoc.

Thanks for reading!   I challenge you to do two things:

Donate to a good cause. (Or pour ice water on your head, or something.)

Check out the Sentry.  It's helping all sorts of companies learn about their network from the standpoints of Performance, Security, and Troubleshooting.

[pr0tectus]

Like A Security / Traffic Camera for your Network

Posted by Pete Garvin

Do you really know what's happening on your network?

Most of us don't, but that's about to change.

We are pleased to announce availability of the Sentry, a new tool for network security, performance, and troubleshooting.  The Sentry provides a new way to see network traffic both historically and in real time.  It's like having a traffic and security camera watching what’s coming into and out of your network.

What does that mean?  Think of it this way:

Your firewall is like the locked door at the edge of your network. It's great, because it makes sure that bad guys aren't using your network to do bad things.  But it's not perfect:

Sometimes people hold the door open from the inside for the bad guys.

Sometimes people pick the lock, or guess the combination, or find other ways through.

Sometimes lots of people (good or bad) use the door at the same time, and create a traffic jam.

So what do you do?  You install a CCTV system, a security camera, or something like that.  The Sentry is like that Security Camera.

You know your network better than anyone. The Sentry creates intuitive, interactive visualizations of what's happening on your network so your brain can do what it does best - see patterns.  Once you are familiar with ‘normal’ for your network, the ‘abnormal’ really stands out.  With the Sentry, you can view traffic in aggregate or drill down and look at individual network connections. Same goes for Intrusion Detection alerts.  We’re not talking traditional network monitoring…..we’re talking next generation traffic analysis.

In the coming days, we'll tell you about the Sentry, how it's being used in networks today, and how knowing the shape of your network makes all sorts of once-difficult problems much easier.

Check it out!  Or follow us on Tumblr, Twitter, or Facebook to stay in the loop!