gaur4vgaur-blog

[gaur4vgaur-blog]

Scala Arrays and Mutable Collections

Scala always wants you to work with immutable collection, however, there are times when you need your collections to be mutable. The video talks about basics of Arrays and Mutable collections in Scala

[gaur4vgaur-blog]

Introduction to Scala Maps

Introduction to Scala: The video talks about Maps in Scala. Maps are one of the most used collections in Scala. It is a core collections module in Scala. Maps can best be understood as a set of key-value pairs.

[gaur4vgaur-blog]

Options in Scala

Introduction to Scala: The video talks about Option in Scala. Scala Option is a way to provide type-safety and error handling in Scala.

[gaur4vgaur-blog]

Scala Lists

The video is an introduction to Scala List. List is one of the most commonly used collection. The video starts with a basic syntax of creating lists and move forward to show few higher order functions such as foreach, map, zip etc.

[gaur4vgaur-blog]

Tuples in Scala

The video gives an introduction to tuples in Scala. We will talk about syntax, ways to access elements in tuple and various other useful methods available in tuples.

Tuples are a bunch of values enclosed within parentheses. You can think of tuples as lightweight containers to pass data around. 

Elements in tuple can be of different data types. Hence, it is an excellent alternative to small and simple classes that we used to declare in Java to pass multiple values.

[gaur4vgaur-blog]

Functions in Scala.

Let us talk about function in Scala. There is a subtle difference between methods and function in Scala. The video is small walk through on introduction to functions in Scala.

Scala prefers expressions over statements. An expression is a piece of code that has a return value. You can think of a function as a named reusable expression. Functions are used to store and pass behaviour in Scala. A method can be converted to a function very easily and hence some poeple use the term functions and methods interchangeably. Unlike methods functions are objects in Scala. Let's have a quick look at the syntax of functions in Scala.

[gaur4vgaur-blog]

Day at Dynamo Hack

On 3rd July, I got an opportunity to attend Dynamo Hackathon. During the second week of June I received a mail from Accenture about the Dynamo summit and hackathon. I registered my interested and, soon I received few emails and an invite to join Dynamo slack. Even though this was not my first hackathon, I was growing anxious. With numerous conversations over the slack, emails and Skype, I came to know that I would be placed in a small team of 5-6 people and would be expected to produce a minimal viable product(MVP) within 8 hours. The MVP would then be judged by a panel from Dynamo. The theme of the hackathon was to promote and showcase the technical skills and technical capabilities within North East.

The team composed of people from various companies, such as Accenture, CapGemini, Sage, DWP, and HMRC; and with multiple skills. I was excited to be a part of team that was equally talented, and eager to connect with each other. The team conceptualised an idea before joining the hack. After multiple rounds of discussion, we settled to create visualisation of the skill gap in NE.

Although BBC had predicted rain on the day, it turns out to be a lovely sunny day. It was a scenic drive from Newcastle to Durham. Thanks to Adnrew, who offered me a lift from Central Station. We reach the historic city of Durham by 7:45AM and joined other team members at Durham university.

As planned, we started to develop a visualisation portal to aid new graduates or job seekers to hunt jobs within a specific region. The visualisation would show various companies available within a specific region and their job requirements. Along with the job vacancies, the portal would link to free/paid online training available to bridge the skill gap.

After a rigorous brain storming, we chose to subscribe to Indeed API to fetch the job requirements and Google API/MS tableau to visualise the same. The team decided to fetch the data from Indeed API and populate it in a local instance of Mongo. The plan was to utilise Mongo db to create some more stats and trends if we left with some extra time. We broke the application into 3 components: - Component 1 - batch script that would subscribe Indeed API and populate local Mongo Instance, - Component 2 - a front end that would visualize the data as the per the user requests, and, - Component 3 - a Scala backend that would query local Mongo instance to provide filtered results to the front end

The great thing about the team was that we all had different strengths and we worked incredibly well together. After few initial technical hiccups all the three components were available. It was funny to observe how quickly the time passes when you are surrounded by determined and equally caffeinated people.

We were all set to integrate the components, but we hit our biggest road block. Among all of us, two of us were Ubuntu, one was linux Mint, another one was Windows 10 and the last one was MacOS. We were couple of hours away from the demo and there was no single machine that could run all the three components. By random selection and trust in Steve Jobs, we all chose to work/fix on MacOS. We managed to complete our MVP, when clock turned red to show last 10 min remaining for the hack.

It was certainly an event full of learning and I enjoyed it a lot. All the teams came up with some amazing MVPs. I will look forward for Dynamo to arrange such events in future.

[gaur4vgaur-blog]

Pattern match in Scala

Pattern matching is one of the most commonly used constructs in Scala. Pattern matching is a powerful form of a switch statement from Java or C or C++. As with many language features in Scala, pattern matches are carefully designed. And there are no break statements in Scala. You can pattern match can be on a variety of different conditions, on type, value, or on a specific boolean expression.

[gaur4vgaur-blog]

Designing Microservices: Resilience

Another design principle for microservices is resilience. A microservice should embrace the failure, whenever it happens. A microservice should be designed for all known failures. Few of the common known failures are:

Timeout from downstream microservice or a third party system failed to respond

An exception occurred with in a microservice or in one of the downstreams

There can be network delays or network failure etc.

Whatever is the type of failure, a well designed microservice needs to embrace the failure. There are many ways in which a microservice can handle failures:

microservice can fallback to default functionality. For ex consider an e-commerce website that proposes new products on side pane based on user's past searches and order history. The search prediction can be built as a separate microservice to analyse the user data and feed the side pane. If the search prediction service is down, then e-commerce website will propose the most popular products rather than user preferred products.

microservice can degrade the functionality or direct users with alternatives. For ex pulling basic details from alternative APIs. The page render to the user will not be as rich as it is supposed to be but the page will have some basic information for the user. Along with these basic details, microservice can put up a small error message to direct user to comeback for full details.

Another way to make whole distributed systems, having multiple instance of microservices, resilient is by making the failed microservice(if it cannot be recovered) to deregister itself, so the system is only aware about fully functioning microservices.

We can observe all these errors/exceptions, using logging and monitoring tools, to make service more resilient.

[gaur4vgaur-blog]

Introduction to Test Driven Development

Testing is an important phase of software development life cycle. You certainly cannot release a code without testing it. Test reports, covering all the business features, help to gain the confidence of the client. Tests are repeated frequently, hence, you should automate as much tests as possible. This can be achieved by writing unit test cases.

What is TDD?

Test Driven Development is a practice of Extreme Programming where a developer writes failing unit test cases first. The approach helps the programmer to think about all the scenarios that could fail when he/she starts implementing the functionality. This helps the developer to understand the requirements better before he/she starts coding.

Steps to perform TDD

TDD is a 3 steps process.

The first step is to write a failing test. If you write a test for a functionality, yet to be written, or even a test case that fails compilation, because the method doesn’t exist, that’s actually a failing test. The intent is to write a minimal test that breaks the code.

The next step is to fix the failing test case. Again, the intent is to write the minimal code that fixes the test case. It may require you to define a method or to write a new functionality. If all the tests does not pass then you have implemented something wrong. Fix the problem and run all the test cases until the report is successful.

The last step is to refactor. This is an opportunity to clean and restructure the code if required, while the test cases are still passing.

You follow the above 3 steps until you have implement the whole feature and test reports are successful.

Advantages

TDD provides good overall code coverage, that helps the team to gain confidence in system.

It is easier to refactor the code without impacting the functionality, as the code is covered by test cases.

The test cases create feedback loop that helps you to catch bugs in early development life cycle.

Practical Example of TDD

http://unitmm.sourceforge.net/fibonacci_example.shtml

[gaur4vgaur-blog]

Designing Microservices: Automations

Another key design principle for microservices architecture is automation. Microservice architecture enables the teams to have frequent releases and shorter time to production. But you certainly cannot ship your code into production without testing it. Whenever the code is released to production it must be tested. So we need automated tools that can test our code every time we deploy it to production.

There are two categories of automation tools available. Continuous integration(CI) tools integrates well with the source control systems. The tools compile the code, run unit test cases and run integration test cases whenever they detect a change in code base. The tools provide an immediate feedback on the change made to the code. The tools provide confidence to the team that recent code changes integrate well with the entire system. The tools are particularly helpful when many developers are working on the same code base, as the tools will alert the team if any breaking change is made. However, it is the onus of the team to write exhaustive test cases for the functionalities. These CI tools can also help to generate the code quality reports. The tools not only fires the test cases but also labels/version the code for next step, if all test cases are successful.

Another set of tools are Continuous Deployment(CD) tools. The team can configure CD tools to deploy the microservice to various environments such as Dev, QA or Production. These tools work along with CI tools to deploy the code. Once all the test cases are successful, CD tools pick up the CI build and deploy it to the targeted environment.

CI/CD tools help the teams to identify errors early by running automated tests and deployment pipelines. You can consider the tools such as Jenkins, Travis, MS Team foundation server etc to perform these jobs for your team.

[gaur4vgaur-blog]

Designing Microservices: Autonomous

A microservice must be autonomous. A microservice interacts with many other microservices or external systems to serve the user. But any change in external systems should not force a change in the microservice. Similarly any change in the microservice should not impact other microservices. All the microservices must adhere to the contracts and interfaces to their clients. A clearly defined contracts between services enable multiple teams to develop microservices in parallel and to deploy them independently.

One way to make a microservice autonomous, i.e., independently deployable and changeable, is by making them loosely coupled. The microservice should interact with other microservices over the network and should be stateless. We can use Open Communication Protocols such as REST over HTTP and architecture components such as message brokers, queues, publish-subscribe mechanism to minimize the dependency between the services. The standard formats such as XMLs and JSONs can help to standardize the communication between the services. We should also avoid using shared resources such as shared libraries and shared databases.

Another way to make microservices autonomous is by introducing versioning. The systems that we are working on today are increasing in size and complexity. It is not possible to upgrade each and every part of the system in a single release. As a result there are scenarios when one of the microservice is upgraded while its client are still waiting for business decisions. In such scenarios we need to make sure that even if there are changes in microservices, then those changes should be backwards compatible. The team working on microservice should avoid making any breaking changes. One way to deal with this problem is to introduce versions. The clients that have upgraded themselves can use later versions while others can keep using the older version of the microservice. This allows consumers to slowly migrate over time from old version to new version. You may have to deploy both old and new version of microservice. Another way is to include both old endpoints and new endpoints in the service. These old points are no more than a wrapper around the new endpoints. This will prevent breaking any existing contract. However, over the period of time all the clients should move to new endpoints.

[gaur4vgaur-blog]

Designing Microservices: Observable

The microservice architecture leads to a distributed system that is scalable, flexible and reliable. These microservices interact with each other to complete user transactions. Since there is a lot going on in the system, keeping a check on the system health is vital. So another key consideration to design microservice is that the microservice should be observable. The service should push the information about the activities that are happening with in the system.

All the microservices with in the system should have centralised monitoring and logging. The data is useful to diagnose problems, if any, and provide quick resolutions. There are many tools available in the market that can help you to monitor the microservices. Some of the popular tools are Kibana, Graphana, Splunk, ELK stack etc. These tools help you to collect the system stats and logs at a centralised location. This collected data then cab be utilised for:

capacity planning, scaling up or down the system

user behaviors, to diagnose if users are facing any issue in any specific flow

potential system heath issues. CPU usage, memory usage, disk usage etc can provide valuable feedback to the team about the bottle necks in system, if any

causes of timeouts, exceptions, and errors happening in the system

While designing a microservice, the team should decide upon:

the information that needs to be logged. Team should consult security architects to verify the sensitivity of data getting logged

structure of logs. Few of the important considerations are to put use rid, date, time and module name in the logs. The information would help the team to debug any potential bug

valid correlation id for transactions. The correlation id would help the team to trace the transactions in distributed environment.

The team should then carefully insert meaningful logs and audit events within the code. A consistent log format will allow team to query the logging information. The team can then query for specific patterns and specific error codes using logging tool.

You can also configure alert monitoring tools such as pagerduty, opsgenie, slack, webhooks etc for your microservices. The tools will send out emails and sms to alert the team for ex if there are 25 timeout exceptions in last 10 min then team will receive an email or sms. The tools can also help the team to log incidents.

[gaur4vgaur-blog]

Introduction to for loops in Scala

The video is an introduction to for loops in Scala. The video will talk through on how to use forEach and for loop in Scala.

forEach and for loop is quite common and is used by almost all the programming languages.

[gaur4vgaur-blog]

Designing Microservices: Business Centric

A microservice should be business centric or domain centric. It should be designed to represent a business function or a business domain. This helps to scope the service and to control the size of the service. The idea is borrowed from domain driven design where primary focus is on core domain logic with a bounded context. For ex HMRC is a huge organisation and it needs to update to all tax payers across UK. Hence, HMRC has to create an email functionality. But email functionality itself has many business functions such as verifying an email, drafting an email based on updates, maintaining templates based on users etc. So HMRC has created each of these functions as a microservice. You can refer to hmrc/email-verification as an example.  

In order to design a microservice, business domain is coarsely analysed. The business domain is then further split to represent separate business functions or bounded context. Every bounded context can then be made as a microservice. These extracted microservices can interact with each other using well defined APIs. As the various microservices will be in constant interaction, we need to think about the inputs and outputs for each microservice. The microservices are sometimes designed based on technical boundaries for ex a common piece of functionality is extracted as separate service as the same calculation is used by various modules or a microservice is designed to access common reference data for all other microservices. As an example let us have a look at hmrc/gmp-frontend and hmrc/gmp services. The services are used to calculate Guaranteed Minimum Pension calculation for a member of a contracted-out scheme under the certain circumstances. These services are split into two business functions, where gmp-frontend takes care of all the user interactions and gmp handles all the calculations. The two services constantly talk to each other to serve the users.

[gaur4vgaur-blog]

Designing Microservice: High Cohesion

A microservice must have high cohesion. Microservice should focus on single business functionality and should be consistent in terms of functionality, input and outputs. 

Let us refer to hmrc/email-verfication as an example. The service has a single focus of verifying the emails. The microservice verifies the email address by firing verification link to provided email address. This service encapsulates all the logic/functionality of email verification. The microservice has only one reason to change. The service will be modified only if it needs to be upgraded or if there is a change in the business logic to verify the email. 

The idea of Single Repsonsibility Principle is adopted from object oriented world. The principle the service to be focused on specific business reason and to be highly cohesive. As the mircoservice is cohesive it is scalable, flexible and reliable.

In order to design a microservice, we need to first identify single focus of the microservice. We can determine the responsibility of the microservice either:by business requirements for ex email-verification needs to verify the email, orby domain, where the focus is to create, update, fetch or delete the data. Let us refer to another service hmrc/tamc. The service allows married couple to check if they are eligible for Marriage Allowance. The service concentrates on creating/updating/listing the relationships. The service requires the changes when government rules regarding the marriage allowance changes for ex allowances going up or down. The microservice deals only with the calculation of marriage allowance, however, there is a separate microservice written to display marriage allowance data to users.

This enables marriage allowance team to enhance and deploy specific parts of system. 

[gaur4vgaur-blog]

Microservices

What are Microservices?

Microservices are gaining popularity everywhere. The digital platform I work on now is build on Microservice architecture. Microservices refers to an architectural approach that is intended to decompose application into finely grained, highly cohesive loosely coupled services. These microservices are language-agnostic and platform-agnostic, enabling different systems to talk to each other.

The microservice architecture can be used to break down a big monolithic application into smaller, simpler and manageable services. These smaller services can be then managed by cross-functional teams, without stepping on each other toes. The respective teams can then deliver the functionality based on their velocity and priorities as business needs.

The microservice architecture is best suited for a complex application that has potential to grow into large code base over a period of time. A microservice architecture goes hand in hand with Agile and DevOps culture. A digital platform may start with a single basic search functionality of serving the customers. Later on, with the customer feedback, the platform can then be extended to include updates, deletes, email, reconciliation etc. It gives business a flexibility to include the module without impacting the existing functionalities on platform.

Microservices enables a team to have quick turnaround and continuous deliveries.