#CrowdHandler

[madehq]

The Bots Come Out At Night

nline ticket scalping (or touting) is the act of using abusive software (known as “bots”) to automate the purchase of tickets and sell them on at inflated prices. This practice damages the reputation of our clients, who want to offer tickets at reasonable prices. Since tickets have been available online, industrial-scale scalpers have used automated bots to systematically profit within the secondary ticket market, at the expense of fans.

Over the few last years, we’ve seen mounting public pressure putting the spotlight on those individuals and companies who profit from selling on the secondary ticket market, which is estimated to be worth over £1 billion in the United Kingdom. Meanwhile, proponents within the ticketing industry, notably the pressure group Fan Fair Alliance, are actively working to generate awareness of the problem on a multiple of different fronts.

Legislation is beginning to catch up with scalpers too. In the United States, the Better Online Ticket Sales Act, or the BOTS Act, was signed into law in 2016. This law effectively makes it illegal to use automated software to buy tickets in order to circumvent the “control measures” used by ticket sellers. In other words, if a ticket seller has actively employed measures to stop scalpers, then you’re breaking the law by using bots to get around them. This does however mean that in order for ticket sellers to prosecute scalpers, they need to provide evidence that they have put adequate measures in place, as well as an audit trail of the accused illicit behaviour.

This month Ticketmaster filed a $10 million suit against Prestige Entertainment, who are alleged to have used bots to scope up thousands of tickets for the Broadway smash hit Hamilton, amongst other high demand ticketed events. It is one of the first major lawsuits since the introduction of the BOTS act, and comes after two years of monitoring and tracking automated transactions, and subsequent sales on the secondary market.

Similar legislation has been lobbied in the U.K. Houses of Parliament, resulting in the independent report into the secondary ticketing market by Professor Michael Waterson. The U.K. moved towards criminalising the use of “digital purchasing software” with the Digital Economy Act 2017, which received royal assent in April and has now become law. It gives the Government the power to make it a criminal offence to use bots in order to bypass maximum ticket purchases set by event organisers.

So, as the gatekeeper for over $500 million worth of online ticket sales for non-profits every year, how can we at Made Media do our bit in the battle against scalpers? Let’s discuss how these bots actually work.

There are three main categories of bots that we see attempting to infiltrate our systems:

1. Drop Checkers or Spinner Bots. Most of the traffic we see are bots waiting for tickets to be released for sale. Usually, we’ve found these bots come out at night. They constantly probe ticketing pages which display availability both before an event that is due to go on sale and after the event is sold out (in case additional performances are put on-sale, or new tickets are released). Amusingly, we had one bot spinning around the booking page for Kylie Minogue for months after the event had sold out. The goal of these bots is to detect when tickets are released and initiate an army of…

2. Acquisition Bots. The goal of these bots is to find the best inventory available for an event and reserve it for purchase. This is the most dangerous kind of bot, because once tickets are held in a bot’s shopping cart, they’re unavailable to other customers. The game is effectively lost at this point. As the ticket scalpers have potentially hundreds of bots at their disposal, they can effectively pick and choose the most valuable tickets for resale at leisure. Causal scalpers can finish the checkout process manually, but larger scale scalpers use...

3. Expediting Bots. Once tickets are selected to buy, scalpers use these kind of bots to automate their actual purchase, they effectively login, enter all the required information to pass through the ticket purchase path, and complete the order from a batch of credit cards. These are usually from a selection of accounts of pre-made accounts created in advance (also farmed using a different kind of bot), and can use fraudulent credit cards for the actual purchase.

These bots are categorised and defined under OAT-005 in the OWASP Automated Threat Handbook, which goes into further detail of their characteristics and traits. Depending on the kind of purchase path, automated software may use CAPTCHA Defeat bots if necessary, which utilise comprehensive databases of image-word mappings to bypass CAPTCHA, which can be beaten easily (although we have high hopes for Google’s Invisible reCAPTCHA, which is on the horizon.)

These bots are usually coordinated by a single purchasing script or custom built software. For example, the website TicketBots sells off the shelf software (currently discounted at $10,000) to completely automate this process of holding inventory and purchasing tickets online via Ticketmaster.

Example user interface for executing spinner bots

We currently integrate with a number of different web solutions that identify and block traffic identified as bots. Given that we develop, host, and maintain many of our clients’ ecommerce solutions, we are well positioned to help keep inventory away from resale websites. Our CrowdHandler platform, for example, sits between customers and the ecommerce path; customers (or bots for that matter) cannot access tickets without first passing through the queue.

In computer science, the task of distinguishing between a bot and genuine customers can be viewed as a classification problem with the goal of, given a large multidimensional dataset, identifying the authenticity of the end user in real time. And, although the tech industry’s latest buzzword, Machine Learning, combined with domain driven heuristics, could play a role in tackling this issue. We’ve found that for these AI-driven solutions to be effective, large training datasets (e.g. web traffic) are required, and a reliable feedback mechanism to inform a learning algorithm.

Machine Learning allows us to identify patterns in bot behaviour that even the most sophisticated bots exhibit—trails they do not realise they’re leaving behind.

One approach we are currently working on uses (anonymous) data we’ve mined, adding AI to the way customers are allowed through the website through the use of Artificial Intelligence and Amazon Web Services. Machine Learning allows us to identify patterns in bot behaviour that even the most sophisticated bots exhibit—trails they do not realise they’re leaving behind. It will better classify between users in the virtual queue who are bots, and those who are genuine fans. And with the use of feedback mechanisms (i.e. supplying the algorithm with data confirmed to be scalpers), this prevention mechanism will learn and adapt as scalpers change their methods and strategies to circumvent detection.

In this way, CrowdHandler effectively acts like a bouncer outside your local nightclub. The algorithm scans users waiting in the queue and analyses multiple attributes about their identity, but also their behaviour up until the point of entering the queue. These scalpers/touts are effectively “bounced” to the back of the queue or blocked completely, allowing real fans access to tickets. And whilst we have some useful heuristic-based blocking rules in place (e.g. frequency of requests combined with number of sessions from a single IP address, user agent strings) there is certainly more experimentation to be done in order to combat the degree of sophisticated attacks these bots present.

So, the battle against bots is an ongoing one. Legislative action is allowing primary ticket sellers (who have preventative measures in place) to seek the prosecution of scalpers/touts, who could be subject to unlimited fines or incarceration. Greater transparency of sellers on secondary ticket markets will help us develop better algorithms for stopping scalpers at the box office. There are other fronts opening up on this fight too, for example Ticketmaster’s Paperless Ticketing program, which challenges how we identify ticket holders at point of entry. All of this provides incentive to invest and enhance our tout / scalper detection system, with the aim of protecting the reputation of our clients.

[madehq]

Presenting Our Latest Work: Dr. Phillips Center for the Performing Arts

We were very proud last month to launch the new website and Tessitura-integrated ticketing path for Orlando’s Dr. Phillips Center for the Performing Arts, one of our newest clients. This release was the product of a six-month partnership and represents the first step in Made’s redesign of the Center’s digital presence.

The Need

Dr. Phillips Center first approached us with an urgent and specific need: to provide them with a ticketing path that would give their customers an excellent experience at all times—even during major onsales, where demand for popular events is very high. Several very large onsales were planned for later in 2017, and it was clear that slow loading times and error pages during these key selling periods were out of the question.

The Center’s previous website had been built by a local design shop, and featured a completely custom Tessitura integration. This is no small feat—as Tessitura is such a powerful tool, we know firsthand the learning curve required by developers new to its APIs—but the way the former site was built, with minimal caching and constant polling of the Tessitura APIs even on pages outside of the purchase path, meant that the site struggled to handle even moderate traffic and could easily overstress the Tessitura servers.

The Center needed web developers with hard-won experience building extremely performant Tessitura-integrated websites, and that’s where Made came in.

“Arts For Every Life”

Made started the project, as we usually do, by spending a week on site in Orlando with key members of the Dr. Phillips Center staff. It immediately struck us that this was not your typical arts presenter.

As one of the country’s newest performing arts centers—they first raised their curtain only in 2014—the Center’s founders and leadership wanted to learn from the mistakes of older arts centers, and were intent to avoid projecting any sense of inaccessibility or elitism. We could see this in their internal culture (even back-office staff wear on-brand name badges), in their broad and populist programming, and even in the architecture of the building, which was designed such that the front doors are flush with the street: a visitor doesn’t have to take a step up to come inside.

It was clear to us that the staff really lived their “Arts For Every Life” slogan, and that this attitude was as much a part of the brand as their modernist typography and Florida sunrise–inspired color palette. Our brief, then, was more than just building a website that could handle a major onsale—it also needed to be as welcoming and thoughtful as the Center itself.

The Design

Our mantra when making aesthetic design choices and developing the user experience was always “Simplicity & Beauty.” We wanted a to create a digital experience that was an extension of the friendly, open, and accessible atmosphere a patron would enjoy when visiting the Center. We wanted the experience to reflect the beauty and attention to detail evident throughout the physical space.

To create an arts center that extends well beyond our walls and flows throughout our entire community.

The first morning of our onsite Discovery week, we were taken for a full tour of the Center by the core team. Being shown around the space, meeting staff and hearing anecdotes, we saw how the the Center’s entire philosophy had suffused the architecture and design of the building. This really helped us to quickly develop a deeper understanding of the Center, and informed and inspired much of our initial design approach.

For example, wayfinding signage had been impressively integrated into the building: it never dominates the space, and yet every time you might start to wonder which way you need to walk next, your eyes land on the very directional signage you need. This thoughtfulness fed directly into our approach to the new site’s core navigation. A simple and clean fixed sidebar offers the user immediate entry to ticket purchase, and for further exploration, a single click on the menu icon reveals the entire top two levels of navigation.

As a presenter, the Center often doesn’t have control over imagery provided for given events. For some shows, they might only have a single image, while other shows might provide them with an extensive range of high-quality assets. We worked closely with the in-house design team to be sure that both extremes could be gracefully accommodated, starting by selecting a base aspect ratio that worked well across mobile devices, thumbnail images, and with the Center’s onsite video screens.

Taking further inspiration from the building’s ability to use lighting to alter the tone of and ambience of the space (for example, by lighting the entire space with green during a run of Wicked), we gave CMS users the ability to apply their own theme colors to individual pages, helping to seamlessly integrate imagery into key production pages.

The Center’s previous design was “responsive,” but suffered from oversized imagery and typography on desktop devices, often overpowering and pushing important content below the fold—a common affliction of mistaking “mobile first” to mean “mobile-only.” With this in mind, we created the facility for an optional full-bleed hero image to be added when appropriate assets were available. This image can be extremely impactful while stopping short of dominating the viewport, and is only served to larger screen devices.

The importance of color for the institution can’t be overstated. Their brand colors were inspired by the Orlando sunrise and Florida citrus tones. (Perhaps that’s no accident—Dr. Philip Phillips was a citrus magnate.) It was agreed that use of large blocks of these vibrant color should be eschewed, in favor of simple whites and subtle grays. Using the vivid color palette sparingly affords it greater visual weight when drawing the user’s eye to key interaction points and calls to action.

And of course, for an institution that prides itself on being accessible to all, Made (as always) ensured the site was delivered to adhere to the WCAG 2.0 accessibility guidelines.

Technology

The new site is built with Made’s preferred CMS for arts organizations, the open-source SilverStripe; CrowdHandler, a Made application that provides capacity control and bot protection; and the ticketing path is powered by a customized implementation of BlocksOffice, Made’s software-as-a-service Tessitura-integrated commerce platform. We find this to be a winning combination.

Like most of our projects (and like the website it replaced), the new site is hosted with Amazon Web Services. Given our long track record with that platform, we know that AWS is not a magic bullet. A combination of careful caching technology, including Amazon’s CloudFront CDN, and carefully structured code keeps the servers happy and the pages loading quickly for the end user.

Results

There are more load testing and larger onsales yet to come, but early indications show that the new website is far more performant than the one it replaced.

The site had its first real-world test only a few days after launch, during an onsale for the band Chicago. At peak, with a high level of users were in the ticketing path, page load times stayed below half a second, and the CPU load across the system stayed very comfortably below 10%. This is a far cry from before the new website; excessive API polling on the previous site routinely drove load on their servers as high as 60% even during low-traffic periods.

Joel Schwalbe, Vice President of Technology for Dr. Phillips Center, says: “Made Media was chosen to design, develop, and launch our new website not only because they understand our space, but because they understand our customers. Oftentimes the website creates the first impression for the guest, and it’s essential that this interaction be frictionless and intuitive. The new mobile responsive website absolutely accomplished these things and many more including enhanced navigation as well as improved capacity and performance. We are very pleased with our partnership with Made Media.”

Our thanks go to the Dr. Phillips Center team for a great partnership.

[madehq]

Made Conference Diary: Annual Ticketing Innovations Summit, Lisbon

Trip to Lisbon in June? Yes, please! However, while the venue was beautiful and the temperature nice and toasty, this year’s Ticketing Innovations Summit was more than just a pretty place. 

The organisers made a concerted effort to bring together a cross section of industries under the general ticketing umbrella to maximise the learning and knowledge sharing opportunities. And in keeping the total delegate numbers under 100, it meant I was able to meet almost every single person there over the two days.

Kicking off the first day was an all-group session of case study presentations focusing on “personalising the fan experience”.  My talk focused on on-sale survival, with cautionary tales - all names changed to protect identities! -  and tips for being better prepared, including discussion of CrowdHandler, our virtual waiting room and queuing system. I was also lucky enough to be on a panel alongside professionals representing transportation, software development and festival organisation, debating the future of ticketing technology. There was unanimous agreement that while the rise of mobile is very much a hot topic now, it is also the future of ticketing and is likely to continue evolving with new technology. We predicted increasingly pervasive hands free mobile technology, with computerised glasses, Amazon Echo and Apple’s upcoming HomePod all likely to revolutionise ticket purchasing in the next year.

Opportunities like this, to see ticketing through the lens of two distinct, but related, industries, do not come up enough, and we need to do more to learn from each other.

The remainder of the conference was split into two separate tracks — one for transportation and one for live entertainment. As tempted as I was by the transport avenue, I couldn’t help but think that my colleagues might expect me to report back on the entertainment side of things. Themes across the sessions included: digital transformation, how to deal with the secondary market, attendee vs. Purchaser data capture challenges, and empowering customers by moving away from the traditional event ticketing models which put promoters at the forefront. Notable speakers included our client Sarah Woods, Director of External Affairs at the Royal Albert Hall, the co-founder of ticketing system Skiddle and the Marketing and Ticketing Manager of Rockhal in Luxembourg. It was great to hear people from different disciplines, sharing their experiences and their insights on how we can improve ticketing.

Opportunities like this, to see ticketing through the lens of two distinct, but related, industries, do not come up enough, and we need to do more to learn from each other. We will definitely be returning to the Ticketing Innovations summit next year for more of the same, and we hope to see you there!