Interesting interview questions about Architecture and scaling of Web Applications.

Once I had an interview with a company in quite unusual form. Thay were looking forward for Ruby on Rails Tech lead and as first stage of interview sent to me a set of questions to answer.

There are questions end up being worth to publish, as I believe might be quite useful for anyone interested in software architecture. I tried to be as concise as possible, so each answer is short.

1. Describe your approach for developing a GraphQL API. Include libraries, code structure.

I don’t have production experience with GraphQL but I do know that is a standard for building flexible API, and the main goal is to eliminate the so called “exploding API problem”. In a few words, if we need to pull the same information in a different form in REST API we would need to create a new endpoint. GrapQL allows you to describe your data with types and form a schema, so you can run `querie` and in that query you can explicitly define request parameters as well as, what json fields do you want to be in your response.

I would use gem “graphql” pretty standard ruby integration to build my own server, there are god “rspec” integration with graphql, for example “rspec-graphql_matchers”. In terms of security I would use JWT authentication as both secure and flexible.

2. Explain how you would debug server requests that take too long.

This question is really broad.

It depends what monitoring system we use. I would assume we have a server, database and a reverse proxy or a load balancer, deployed in the cloud, with something like AWS cloudwatch metrics, have some logs on the server, and have access to those logs (whether it is logstash, or some log files on the server)

Usually those 2 instruments allow us to narrow failure to a particular request, that cause a delay.

I would check servers, wether we have enough memory, and cpu. If not, it can be some expensive computation going, which we can move to a dedicated server, find more efficient algorithms etc.

Memory leaks are a whole set of problems, we should investigate them some time, especially for real time communication via, websockets

If that is ok, the next thing to check is SQL queries, logs can tell us if we have slow queries, or N+1 query problem.

We could use “New Relic” or something similar to track down slow queries. Also db logs are helpful.

There is a whole set of tools in databases for performance checking.

If this is ok, then we can check the database, maybe the database server has not enough, resources, RAM, CPU, hard drive. Maybe it has too many connections, and we need to consider scaling.

The last but not least is reverse proxy configuration, tuning parameters may be userful. Are

configs correct, does caching works, are worker processes ok, is rate limiting working?

Those are probably main steps to debug slow requests in the system.

3. Explain how you would debug database requests that take too long.

I have most experience with relational databases, so let me write about them.

I would use ‘EXPLAIN’ sql statement for slow request(Postgresql ) also has ‘EXPLAIN ANALIZE’

I would check first of all how much data are we pulling in (maybe we are joining lots of tables for example). Next very important part is indexes, does our query use index, did not it fall back to sequential scan, do we have index for all required rows in the table?

Next thing I would check, if we did some aggregation in an efficient manner? Even simple ‘cunt(*)’ query can be expensive.

Those are probably main aspects.

4. Describe the advantages and disadvantages between monolithic app vs microservices.

One more really broad question. I will try to be very concise.

Monolithic application

Pros:

- Easier to start development,

- Simple deployment, and infrastructure maintenance

- Easier to share the logic between parts of the application.

Cons:

- Harder to scale efficiently. Some parts of app might be under high load, but we still need to scale the whole app.

- Harder to share between developers, and develop different parts of the application in parallel

- Test suites and CI tends to grow.

- Code complexity tends to grow over time.

Microservices

Pros:

- Easier to scale efficiently. We can scale only those parts of the app that need more resources.

- Easier to split responsibilities for a parts of the app, between developers.

- Easier t use different technologies in one project.

- Each microservice is logically simpler that the whole app, so i tis easier for a new engineer to start being productive.

- Testing of microservices is simpler and faster.

Cons:

- Hard to maintain infrastructure and deployment (kubernetes is a big deal, but finding people who can work with it costs money)

- Requires planning on how microservices will communicate. Communication between microservices is one of the biggest topics outh there.

- Fault tolerance is a problem. Each microservice has to be built as an independent service, and any other connected services can fails any time. We need to handle those failures in every individual service.

- Microservices are eventually consistent. Data consistency is another big topic in the engineering community.

In general Monolithic architecture is easy to start, and more robust, in terns of developers discipline and planning.

But code of the monolith tends to become very complex over time, hard to scale and maintain.

Microservices are easy to scase, but move code complexity to networking complexity. It brings in all the problems of building distributed systems.

5. Describe advantage and disadvantage of using SOLID

Advantages of SOLID is that it makes what it is intended to. Enforces code maintainability, readability and structure.

I would say that most obvious and important is ingle Responsibility principle, code that violates this principle is very hard to refactor and debug.

The problems can occur when SOLID is abused. It happens more often with junior developers. They tend to create new classes hierarchy even in simple cases where one class or function will be enough.

Writing very clean code is time consuming and not always the best choice. It doesn’t matter how good the code is, if we deliver it later then business needs it.

Also some principles are really hard to follow in practice, for example Open/Close. It is easy when we deal with few classes, but in a real app, the code base can be much bigger.

My general rule is “Keep it as simple as possible”, sometimes it includes violation of solid principles to simply save some time.

6. List some APM and analytics tools you use for your applications and what insights you get from them

I used New Relic on 3 projects, to track down slow requests, I used AWS CloudWatch to monitor EC2 instances, I used DataDog to monitor API usage, find most popular API endpoints and prioritize them, or find endpoints which was not used at all. I have a plan to learn terraform as it is a popular monitoring solution for K8s.

7. Describe AWS or related cloud service you’ve used in past

I used AWS on most of the projects. Deployed services on EC2 within the autoscaling group, created VPC, added users, groups, roles. Used CloudWatch for monitoring, added RAM metric to a few EC2 instances. Also worked with AWS serverless computing.

Now I am half way through preparation for the AWS Certified Solution Architect Associate exam.

8. Describe how you set up tests for your application to ensure full test coverage.

Usually I use Rspec and write unit tests, controller test and request specs for API. My preference is not to work with feature specs/integrated tests as they tend to be very fragile and have high coupling with UI.

Front end team also writes their own specs for UI components using some `jest` or other dedicated npm package.

We usually checked the code coverage with simplecov, we were adding it to CircleCI so we can check how code coverage changes over time with every PR.

9. Describe any non-relational databases you’ve used and your reason for using it (ex: ElasticSearch, DynamoDB)? Also describe what the performance boost was from using a service like this versus a traditional relational database.

I used ElasticSearch to build an admin system, with full text search for capabilities. The app had ~ 4 millions of DAU, and originally was using Mysql for primitive search, with “LIKE” queries. After we moved all those queries to elasticsearch the response become ~10 times smaller.

I have used Mongodb to build IoT system that was capable to process upto 1 TB of data per week, this was unstructured json data so it was nearly impossible to use RDBMS in this case.

However my current project is using Postgresql and serves up to 30 * 10⁶ requests per day which is 30 000 000 / 86400 = 347 requests per second.

10. How would you stress test your application? What is the maximum load you have had to support for an application in the past (ex: requests per second)?

Don’t have much experience, but I know there are tools to do so like Apache Jmetter https://jmeter.apache.org/

11. Describe your processes for ensuring your application and data is secure.

App security can divided into 2 parts, secure infrastructure and secure code.

In general Rails apps are relatively secured by default.

I would use brakeman gem in CI to ensure security on the code level is maintained.

I would check if we do not use parameters concatenation, in the app, sanitize parameters, ise strong params, use corgery protection where possible, that we hash passwords, and have safe authentication with session invalidation, or JWT.

For infrastructure security I would host application in private VPC, hide databases and datastores in private subnet, setup Network ACL and Security Groups for services in public and private subnets with minimum possible permissions.

One important thing is also limit developers’ access to the infrastructure and give them only permissions required to do the job, not more.

12. Have you had to architecture an application before? Briefly describe the application requirements and you solution.

I had to build the service for gathering data and generating reports for individual user and for set of users, who passed certifications (Proexam)

Functional requirements were:

1. The admin should be able to calculate statistical values, generate reports for one user, set of users, set of questions, individual questions of the exam.

2. Multiple admin should be able to approve reports, in a series of steps.

3. System should be able to accept data in different formats(paper based tests, computer based tests, fax, etc)

Non functional requirement:

4. One report can have up to 40000 rows of data and should be generated within 15 seconds.

5. Database size, and scalability was not considered a problem.

6. Time of data upload and preprocessing was not critical (it’s not a problem if it takes longer to preprocess data) if the approval process takes days.

So Consistency was the key, availability was important, but partition tolerance could be ignored.

We split the app into 2 services: Data, and Reporting, each having its own database.

Reporting service was responsible for reports generation.
Req. 1 means we have questions , which form sets. Sets of questions form groups, groups form test, given to a user. Also the format of the data was well defined as each category of question has clear structure. All the above led us to shoes PostgreSQL as a database, as we can have a schema, and Req (1, 2) basically means we should use transactions and data consistency is a high priority.

We built the rails app, with postgres database, and UI, that was able to generate pdf reports.

The other part of the system was data preprocessing. We created it a separate service, another rails app.

We had to ensure all the relations between different entities are correct, so we use Postgresql, with the same database schema as the reporting application and lots of database constraints ensuring data integrity.

This allowed us to work on 2 parts of the application in parallel, and scale without problem.

We used code reviews to synchronise the schemas between apps, with a plan to automate the process.

Services were communicating via JSON API.

Every time the admin was generating reports, the Reporting app was making a series of requests to the Data app to make sure it has the latest data.

To decrease the load on the system at this moment, and speed up the reports generator, we optimized sql for that request, used raw sql, instead of ActiveRecord. We added validation to that sql as well, to filter out inconsistent data.

Also we created a cron job that synchronizing the databases in the night, and we could check whether we have the data for the last day before fetching new data.

Also to make code maintainable and reports generation faster and more flexible we leveraged SQL views.

13. How would you build fault tolerance service/application

Let’s say we want to build and deploy a standard Rails app with a relational database. Then I would dockerize the app. Then we have lots of options how to deploy, we can deploy it to elastic beanstalk if I need to do a quick prototype.

If we need more control I would deploy the app to EC2 instances in different AZ, then create an AutoScaling group for them and hide them behind Elastic Load Balancer.

For a database I would use RDS with Multi AZ deployment.

If applicable to our business logic, we can build the app as a series of serverless components which are also highly durable.

Or we can use K8s with Replica Sets to deploy fault tolerant app (not an expert in kubernetes but I am working on it right now.)