Wise blog_Custom ecommerce development-46.svg

E-commerce

Product scaling

Check out the top 10 aspects of custom E-commerce development that can help you ensure better UX and contribute the most to your E-commerce business growth. 

Key components in custom E-commerce development

Learn what aspects of a custom E-commerce solution can contribute the most to your business growth.

dedicated_team_preview.svg

Startups

A dedicated software development team is a jack-of-all-trades model in terms of outsourcing. It is like hiring your own IT department of like-minded specialists who are ready to kick off your project, share the same values, and are passionate about your product. However, the key is not the model but the right people behind it.
 
When it comes to hiring such teams, we witness a variety of choices caused by the growth of the outsourcing industry, the value of which is expected to reach <a href="https://moderngentlemen.net/outsourcing-statistics/" target="_blank">$405.6 billion by 2027</a>. Over the years, more and more vendors have entered the market offering software development services under various collaboration models. How to recognize and hire a dedicated team that is suitable for you?
 
In this article, we'll look at the latest trends in the outsourcing market to provide you with a full picture of what is relevant today. Besides, you'll get a list of characteristics that define an expert dedicated tech team. Finally, we'll share some advice for hiring, so you can select the right experts to contribute to your project.
 
## Trends & overview of the IT outsourcing industry
 
It is good practice to do some research before selecting a partner. The technologies, as well as software outsourcing companies, are changing rapidly. The things seen as progressive in the previous year may already be outdated. So what’s the picture of today’s outsourcing software development services? Let’s see more closely.
 
- Market growth & tendency to reduce costs
 
The outsourcing market is gaining momentum. According to Statista, <a href="https://www.statista.com/statistics/1091189/outsourcing-penetration-rate-north-america-country/" target="_blank">66% of US companies</a> outsource at least some of their operations. The reason for this is obvious: outsourcing helps to accelerate the development of a project and increases the return on investment. It reduces costs <a href="https://moderngentlemen.net/outsourcing-statistics/" target="_blank">by 15%</a> on average and <a href="https://whattobecome.com/blog/outsourcing-statistics/" target="_blank">up to 60%</a> in some cases.
 
- Tech trends
 
According to Deloitte, <a href="https://www2.deloitte.com/global/en/pages/operations/articles/gx-global-outsourcing-survey.html" target="_blank">68% of companies</a> use cloud services as one of the primary enablers in their outsourcing journey, such as SaaS (Software as a Service), PaaS (Platform as a Service), and IaaS (Infrastructure as a Service). **RPA (Robotic Process Automation)** is ranked second among the major outsourcing enablers, with 64% of organizations using it. 55% stated they use a single instance of **ERP (Enterprise Resource Planning)**.
 
- New communication models
 
Today, we all have to work in increasingly dynamic working environments. All this is because of quick changes in the market and the fact that the majority of development outsourcing companies switched to remote work because of the COVID-19 pandemic. This requires the use of communication tools like Slack and Google Meet, Zoom, and more sophisticated approaches for <a href="https://wise-engineering.com/blog/post/how-to-manage-remote-team-in-increasingly-dynamic-environments/" target="_blank">managing remote teams</a>.
 
Another important tendency is a significant shift in communication between product owners and the team itself. Previously, project managers served as a bridge between the two sides. But today, we have a <a href="https://wise-engineering.com/blog/post/new-communication-model-in-software-development-outsourcing/" target="_blank">new communication model in outsourcing</a>, under which clients are more willing to attend meetings and interact with the developers. Actually, it works pretty well because it provides faster feedback and sets the development on the right track.
 
## Dedicated team: definition, benefits, and when to hire
 
Now, let’s see in more detail what a dedicated software development team actually is and how it differs from other collaboration models? Generally, there are three main outsourcing models that describe the relationships between the business and the IT service vendor.

![Collaboration models in outsourcing](https://wise-site.fra1.digitaloceanspaces.com/dedicated_team_models_6d28ada73d.svg)
 
A dedicated development team is a goldmine when you want to scale your team with top talented people. No matter if you need to extend your existing team (use it as team augmentation) or build your product from scratch with an entirely new team, this model will help you reach the desired goals.
 
**What is the structure of a dedicated team?**
 
The structure is totally customizable per your needs. In most cases, it includes front-end, back-end, full-stack developers, DevOps engineers, project managers, UI/UX designers, and any other additional experts like data scientists, solution architects, marketing consultants, or business analysts.
 
**The benefits of a dedicated software development team**
 
Let’s take a closer look at the possible advantages when a dedicated development team model works successfully.
 
1. Established work processes
 
You obtain self-organized teams that may already have previous experience with different projects. You don't have to spend time building a team. It will consist of people who are used to working together and will be operating at their full potential from day one.
 
2. Hassle-free recruitment and talent management
 
You are not required to find, attract, recruit, and onboard new staff. All you have to do is review candidate profiles and select those that have the necessary skills and are a good fit for your company culture. The vendor covers all costs, including paid time off, private healthcare, hardware, and other perks.
 
3. Cost-effectiveness 
 
Hiring a dedicated development team cuts your business running costs. Among other things, you can save money on equipment, and other administrative costs. In addition, you won't have to pay for any recruitment or hiring expenses, training, or taxes.
 
4. Your project is the main focus
 
You can be sure your project is always the priority. Dedicated software developers keep the project on track without any interruptions. They are completely focused on your assignments and do not switch to anything else. Furthermore, as time passes, their expertise in your product becomes deeper and more sophisticated, increasing the likelihood of gaining better results throughout development.
 
5. Ability to focus on your core activities
 
Once you delegate the scope of your work to a dedicated team, you can focus on more strategic things. You simply know that the work is in responsible hands. As a result, you have more time to work on other aspects of your company's operations.

![Benefits of hiring dedicated development team](https://wise-site.fra1.digitaloceanspaces.com/dedicated_team_benefits_5cafb91c2c.svg)
 
## What does a decent dedicated team look like?
 
The best way to hire the right people is to have a clear vision of what a strong team looks like. How to evaluate the effectiveness of such a team, and what is relevant today? Here are some important aspects to consider.
 
- Flexible

The specifics of such a model provide quick adaptability to new projects. That’s why a great team should be able to select **the right approach** for your project and be able to pivot when business requirements change.
 
- Involved

A great team has a specific attitude towards the work they do, and when they onboard a project, take **full ownership** of it. With some level of autonomy and good communication set up, an effective team will initiate changes both to your product and engineering processes.

- Open to challenges 

Motivational and ready-for-challenge people are another important aspect. People may have the skills, but if they aren't motivated enough, their contributions to the project won't have the results. The team should be ready to go through the path of developing the workload to the end, no matter how thorny it may be and no matter what challenges it has ahead.

- With solid expertise 

This is obvious, but nevertheless, this list could not exist without this point. Technical expertise can have a wide range of domains, such as [custom E-commerce development](https://wise-engineering.com/expertise/e-commerce/), [custom web development](https://wise-engineering.com/expertise/), and [mobile development](https://wise-engineering.com/expertise/mobile-development/). Be narrowly focused on what exactly you want to develop and what technologies you need to utilize. For example, if you want to build a web platform, you need to look for a dedicated web development team. In this case, your chances of selecting the right team will be much higher.

- Building partnership

Every organization has its own practices and vision of how they develop business relationships with its outsourcing teams. We believe effective collaboration is similar to friendship. It is important to feel that you are on the same page and share things that are important to each of you.

- Agile and easy to follow

Always look at the project management methods used by the dedicated team. Scrum or Kanban-based agile methodologies are currently the most popular (almost 90% of software development projects work in Scrum).

The most significant benefit of agile adoption is the project success rate. Statistics show that <a href="https://www.zippia.com/advice/agile-statistics/" target="_blank">91% of agile-led projects</a> succeed in achieving their objectives. With Scrum or Kanban, a team delivers a product iteratively and provides regular updates such as prototypes or live demo versions depending on the phase of the project. Weekly or bi-weekly reports and burn-down charts help track the progress and obtain a clear understanding of what is happening in the development.

- Taking risks into account

The development of any product is not a straight line from point A to point B; it is a winding road with many challenges and risks. A dedicated team should react quickly and eliminate the risk before it becomes a problem. 

Brainstorming regularly with your team helps you identify where the risk factors lie. As for the WE team, we typically identify risks early in the development cycle, and often the main factors are discovered during the product development kick-offs. If a risk is discovered, we set up a meeting with all the stakeholders and people involved and develop a plan to prevent the identified issue.

- Providing honest feedback

The greatest product development teams communicate ideas and comments openly. Your project will only succeed if developers are not afraid to offer constructive judgment. As a result, there should be no hidden issues. In many cultures, it's considered rude to question the decisions of top management, but real experts are often free to provide constructive feedback and suggest improvements.

- Investing in knowledge

It is a good sign if a dedicated team spends a considerable amount of time learning and broadening their expertise. It is always beneficial if a tech team has space for growth in skills, shares knowledge continuously, and experiments with the most innovative technologies.

## Best practices to hire a dedicated software development team

Once you know how a dedicated team model works and what a great partnership looks like, you can start hiring. Based on our experience in building dedicated teams and collaborating with multiple customers, we identified useful practices that can improve the hiring process. This advice will increase your chances of making a deal with a reliable and competent partner.

### 1. Communicate and set clear expectations

Communication is the fuel of any collaboration. It is important to set interactions on the right track and make sure that both sides are ready to care and listen. The golden rule for further effective collaboration is a mutually equal exchange of resources. Ensure a dedicated team can handle all of your expectations, and you, in turn, as a customer, can cover all the expectations of the team as well.

For example, the Wise Engineering team approaches all forms of communication very attentively, with special attention to minor details. This attitude allows us to be on the same page with the customer. As a result, it allows us to quickly take full control over a project and align with all the requirements and overall product vision.

### 2. Define key goals and requirements
Ensure that all of your goals and requirements are clearly defined so that a vendor can understand them. Make a list of questions ahead, and avoid broad or cliché questions, such as *What technology is best for my application?* Better prepare the answers for the following narrowly focused questions: 

- *What is the core product functionality?*
- *Who are the users? Where are they located?*
- *What type of data should be processed? What is their structure?*
- *Are there marketing campaigns planned? When do we expect user base growth?*
- *How soon will we need to expand the engineering team?*
- *What are the time restrictions? Are there due dates set?*

Read our recent article on <a href="https://wise-engineering.com/blog/post/tech-choices-in-startup-mvp-development/" target="_blank">MVP tech stack</a> to discover more information on what questions to ask a vendor, how to approach the first step of communication, and building the first MVP. The ideal scenario is to clarify all the functional and non-functional requirements for your project. The vendor obtains a clear picture of what is needed to be done and then presents more precise proposals. 

### 3. Request proposals from several vendors

To receive detailed information from a vendor about your collaboration, you can write so-called RFx documents:

- **Request for Information.** In this document, you can describe the general problem that needs to be solved and ask a vendor for a detailed answer about which technologies are best to use to put your idea into action.
- **Request for Proposal. **In an RFP, you describe your project in greater detail (perfectly with functional, non-functional, and business requirements). In response, the vendor gives you a detailed implementation plan that shows how you can meet each requirement.
- **Request for Quote. **If you want to evaluate what budget you need to allocate to the entire project, you can send an RFQ, and the vendor will reply with a document with full pricing.

These documents announce a project, describe it, and solicit bids and ideas from vendors. It's a good idea to outline all of your needs in writing before signing a contract with a vendor. Besides, you can receive several proposals from different vendors, compare them and evaluate who could be more beneficial for you.

### 4. Use security measures

As far as you will communicate a lot about processes, one of the most significant concerns is security. This is primarily a source of worry because you’re trusting newly found entities with sensitive information. Pay attention to signing the Non-Disclosure Agreement (NDA) right from the start. 
Check if the dedicated team provides you with an opportunity to sign an NDA and what additional data privacy and security measures it provides. You can sign a unilateral (one-way or one-sided NDA and use it when only you are sharing materials) or a bilateral NDA (two-way type, when both of you provide access to confidential information).

### 5. Define the scope of work to be done
It is good practice to describe all the work that will be performed. To do this, another portion of routine yet useful paperwork can serve you well. For example, as a customer, you can write a Software Development Statement of Work (SoW) or ask a vendor to output a Software Requirements Specification (SRS).

**Software Development Statement of Work (SoW)** covers all the details and expectations of collaboration. The aim is to make the agreement valid and legally protected. A correctly written SoW can serve as a reliable blueprint and confirmation that you and your vendor share the same vision. The document contains:
- An overview of a project.
- The scope of work to be done by a vendor (technical details, documentation).
- Conditions of collaboration, Key Performance Indicators (KPIs), deadlines, payment, and other acceptance criteria.

SoW is like a key for security and guarantees that the product will be realized precisely as you wanted it to be. Thus, there are more chances that collaboration will end up successfully.

**Software Requirements Specification (SRS)** is a roadmap to the entire project and the cornerstone of proper software development. Usually, SRS is written by a tech lead from the vendor’s side, and developers from the dedicated team use it as a reference or a clear plan of all the tasks they need to perform.

### 6. Follow the right steps
Once you interview potential candidates and choose one, it is time to proceed to the hiring stage. But this process is a set of different procedures, so the right algorithm is a must. Here are the recommended steps for hiring a dedicated team.

![Steps hiring a dedicated development team](https://wise-site.fra1.digitaloceanspaces.com/dedicated_team_steps_8859a9156b.svg)

## Wrapping up

To select and hire dedicated development team, it is not enough just to check technical expertise. The success of any collaboration is also defined by relationships. Try to find people who possess the needed soft skills and see if there is a synergy between you.

People who share the same values are more likely to be passionate about your product idea. In this case, you have the best chance to create a powerful collaborative tandem. As a result, you can trust and shift some part of your burden to the team so that you could focus on other important areas of your business. 
The Wise Engineering team supports sophisticated software development and project management methodologies. We attempt to collaborate in the same way as we would do with our closest friends. We always go the extra mile to help you interweave developers into your internal team and establish a good rapport with the dedicated team. 

[Contact us](https://wise-engineering.com/contacts/) to learn more about how we approach [custom software development](https://wise-engineering.com/expertise/) at Wise and let's discuss the specifics of your future project. 







Check out IT outsourcing trends in 2022, learn when to hire a dedicated development team, and check out aspects that should influence your choice >>

Dedicated software development team. Best practices to hire

Learn what aspects to consider when hiring a dedicated team and when you will benefit from this collaboration model the most.

DevOps_interview_preview.svg

At Wise, we love friendly chats over coffee in our office kitchen. This morning routine always helps to tune up for a productive day and usually is accompanied by discussions on some trending topics. One day we had a small talk about the connection between DevOps and business, and this blog post is a continuation of that conversation. We decided to discuss the value DevOps can bring to business, and no one understands it better at Wise than our DevOps engineer Ihor Mozil.

Ihor has 12 years of background in system administration and now 5,5 years working as a DevOps engineer. Ihor’s ability to dive deeply into the problem and learn fast fascinates our team. His persistence helped him grow in his domain in immense steps. Ihor's tech knowledge is very broad, and his major expertise includes CI/CD pipelines, infrastructure, and system security. Also, among other Wisers, he is known for his hobby – Ihor constructs racing drones and flies them as well. He is a mentor at Wise Engineering School, takes an active part in our Tech Talks, and never gets tired of talking about his hobby. 

This interview is not about building drones, which requires great system thinking by the way. Yet it will help you see how DevOps can help you fly your business better. Or, better to say, faster. 

![](https://wise-site.fra1.digitaloceanspaces.com/Dev_Ops_interview_drones_a7ee48fd26.jpg)

## Specifying the nature of DevOps 

**Q. What system administration didn't give you that you headed towards DevOps eventually? **

I. Automation would be the most critical, I suppose. When working as a system administrator, I always looked for opportunities to automate my work. All the repetitive tasks were eating up most of my time, and I always wanted to spare more of this resource for more complex issues. So I intuitively started to automate some processes already in system administration, and then the transfer to DevOps was very logical. 

The ability to minimize human errors is the second reason. No matter how attentive or scrupulous you are, in routine processes that are complex, at the same time, the machine performs better, and the risk of an error is minimized.

**Q. During the five years, did you get the moment that helped you reconsider the DevOps model? And what was it?**

I. In fact, reconsideration and reevaluation are happening all the time, because, as a DevOps engineer, you get new experience continuously working with different projects. Within some projects, you have all the freedom to try new technologies and see their potential in practice. Some projects have extensive legacy parts that give you a lot of restrictions. Still, it makes you think twice harder and make up a working solution from all the minuses you have. This allows you to reconsider the overall process and introduce approaches that simplify complex issues or make these complex issues convenient for other people that work with you in a team.

I can't say this is a process of constant reconsideration of the DevOps approaches, but with time you get to see a more broad picture of how DevOps work influences business development, that’s for sure.

**Q. Are there any myths or misinterpretations of DevOps you’d like to dispel?**

I. The key idea to keep in mind is that DevOps is about the adoption of methods, best practices, and processes, and it is continuous in its nature. Thus, the first misinterpretation of DevOps I would address is that it's not about the tools your team uses. You can not say that by introducing a particular service, you’d have the DevOps processes arranged. This is not how it works.

DevOps strategy is based on setting up effective collaboration between people responsible for creating software and people whose work is to maintain it. Hence, a major task of a DevOps engineer is to streamline this process to help a business achieve goals faster.

### “DevOps engineers are an integral part of the engineering team. Not a separate unit that works in a vacuum.”

Another important aspect is that DevOps engineers are an integral part of the engineering team. Not a separate unit that works in a vacuum. Only if DevOps engineers work closely with all the development team, they understand all the tech context and if they have a high-level business roadmap for at least a year, DevOps specialists can be most effective. If working separately, they can bring only more headaches to a project.

And finally, this comes from what I’ve mentioned above, one should never underestimate the soft skills a DevOps engineer should possess. Communication is vital in DevOps. I always speak of it during the DevOps mentorship program at Wise School. If you can not clearly communicate the value the change brings to the overall development process and the business, this change will never get approved and never happen eventually.

Last but not least, the DevOps methodology has a business focus at its core. And this is critical to understanding both for business owners and engineers themselves. All the optimizations that are about to be introduced face a particular business challenge, and help achieve business goals from both short- and long-term perspectives.

## Business value driven by the DevOps team

**Q. Let’s focus on this part – the business value that the DevOps model brings. If you had to think of one core thing, what would it be?**
 
I. Faster time to market. This is the key task behind DevOps above anything else. Or better to say, everything that a DevOps engineer or a DevOps team strives to achieve is to make sure the software the engineering team builds is delivered to the customers ASAP and the management immediately starts to receive data on how it is performing.   

### “The specific appeal of DevOps is that business no longer has to trade speed for product stability, and vice versa.”

Hence, speeding up time to market should be at the core of any task a DevOps engineer receives and the key aspect a DevOps specialist should focus on when introducing any changes and optimizations. Most importantly, this applies to all – startups, more mature companies, and enterprises. By building a new product or modifying and improving the existing one, the main task is to bring this product or product updates to end-users as quickly as possible, at the same time not neglecting its stability, security, and quality. The specific appeal of DevOps is that business no longer has to trade speed for product stability, and vice versa.

**Q. So there are multiple components of DevOps work that lead to shortening time to market. Can you name them more specifically?**

I. First of all, DevOps creates and automates a pipeline of software shipping to a customer. 

A specific pipeline is created that launches automatic tests after each commit, then automatically or manually deploys these changes to where they belong, assigns QAs to test them and sends it for product owner review, and, finally, deploys them to production. Exactly how long it takes for this process to complete defines time-to-market speed. DevOps engineers’ major task is to speed up this process as much as possible. Gone are the days when deploys were happening once in a quarter. Today engineering teams need to be able to deploy changes to production multiple times a week, if not a day or hour. 

The other aspects that are in the DevOps competence and influence the business maneuverability include:

- Standardization and virtualizing infrastructure assets (servers, databases, testing environments).
- Monitoring and measuring system performance and health, minimizing downtimes.
- Fostering tighter collaboration between all tech teams and sharing resources across tech departments.
- Smooth integration and automation of low-value processes.
- Eliminating ineffective processes and cost optimization. 

![](https://wise-site.fra1.digitaloceanspaces.com/Dev_Ops_interview_portrait_903eded0a4.jpg)

**Q. What are the major business risks DevOps work should address?**

1. Losing market competition

DevOps culture allows businesses, first of all, to win a race with competitors and get more market share in the long run. Because if several companies are operating in the same market, and one of them have all the processes streamlined, and the others need weeks or months to introduce changes, it’s clear who will end up as number one.

2. Lack of relevant analytics

Working on the crossing between technology and business, DevOps engineers can set up custom analytics that helps business stakeholders keep flexibility in business decisions and base them on data rather than predictions. Because with a streamlined software delivery pipeline, your customers get access to your product or its new functionality before the competitors, and as a decision-maker, you need to start gathering data faster and adjust the business roadmap accordingly. 

3. System vulnerability

DevOps engineer analyzes the project for vulnerabilities and suggests the relevant permissions policy for everyone involved in product development. Personally, I practice forbidding access to everything right from the start and, then, granting only those access levels required for efficient work. No more, no less.

4. Lost profit

During the periods when your product or its parts were not available for customers, you lose money, and a business analyst will help you count the exact amount in no time. The efficient work of a DevOps engineer ensures that downtimes are minimized, and if any incidents happen, the work is done to prevent them in the future.

5. Lost opportunity

This is about system autoscaling and downscaling. In today's dynamic environment, you should be sure that if any load increase to your system happens, it can handle it, and once it is over, downscale back so you would not spend extra. Imagine that some celebrity shared a link to your website, and it went down due to the user high load. Would you get such a chance any time soon? Don’t think so.

## Underestimated aspects of DevOps work

**Q. You’ve mentioned previously the importance of communication between DevOps and the tech departments. How can it be arranged and how it may impact the overall teamwork?**

For a DevOps engineer, offering a single solution and stating that this is the only solution and this is what we'll do is a faulty approach at its core. Setting up DevOps processes is about communication and finding win-win solutions for all teams, offices, and departments. Everyone involved.

If as a DevOps engineer, you work with multiple tech teams, you can gather all the tech leads to discuss the solution you are offering. Depending on the project or company size, the form of how you’d communicate will vary. In small teams, creating an RFC document would be good practice. It can contain a short problem description, its technical resolution, architecture, and possible solution. DevOps can share it in a communication channel, ask all counterparts to review the suggested solution and add their comments if this solution is relevant or what aspects are not taken into account. Then in the communication that comes after it, the win-win solution is born – a solution that would address a specific problem and specific business goals.

### “Setting up DevOps processes is about communication and finding win-win solutions for all teams, offices, and departments.”

If the team is bigger the idea of communicating details remains the same, only a more complex document is created, based on it a meeting is set up where all the counterparts decide: what we are doing, why we are doing it, what are the benefits, what project part require modifications to get it implemented.

In situations when an incident happens, DevOps engineers’ soft skills matter the most. In such situations, it is critical not to find people to blame but to make sure all the aspects are examined, and all the necessary work is done so this won’t repeat in the future.

Overall, improved communication across tech teams that can be initiated by implementing the DevOps model leads to more motivated engineers and more productive teamwork. Moreover, the time that a DevOps engineer saves for the development team struggling with any routine inefficient processes can now be used to innovate and helps to take things off the engineering team’s plate.

**Q. How can a business ensure that key information on DevOps settings is well-preserved and transferred when DevOps engineer rotation takes place?** 

I. This is a very extensive and urgent topic. I’ve made some notes on it for my future speech at Wise Tech Talks. But I’ll try to expalin briefly here. 

Knowledge-sharing is a big and important aspect that DevOps methodology drives. DevOps engineer makes sure the project documentation, first of all, is written. This does not mean every single move should be described in detail. No. The documentation describes the project specifics on a higher level. So that every engineer that joins the project after reading it would understand how the system works, what system components are interdependent, how they communicate with each other, and that’s all. A good practice is when project documentation contains information on why particular approaches or technologies were not chosen, and why.

When the change in the DevOps team is taking place, *Infrastructure as a code* is one of the greatest practices that ensure the knowledge is transferred from one specialist to another efficiently. The process of managing and provisioning software infrastructure through code ensures that no settings are made manually. DevOps engineer writes code and via API of some cloud sets up infrastructure and its configurations, and develops the whole CI/CD process through code. When new engineers onboard a project, if they have efficient tech expertise, they will understand how everything works.

Because a DevOps engineer collaborates across tech teams and finds common ground for sharing knowledge, this specialist can also initiate some regular tech discussions, updates meetings, TWIL (*This week I learned*) initiatives, or other forms of knowledge sharing. The management team, then, can turn this initiative into some corporate training programs and help the company embrace the knowledge-sharing environment on a more global level.

![](https://wise-site.fra1.digitaloceanspaces.com/Dev_Ops_interview_knowledge_sharing_93d184f9be.jpg)

## New horizons for DevOps

**Q. What are the biggest challenges DevOps engineers face today?**

That depends on the project, I suppose. I’d speak of the projects I’ve been part of recently.

More and more these days, I come across tech problems that, at first, look very complex. And to solve them properly, you need to dive into business specifics. Such solutions usually require a lot of preparational work, examining similar cases, and finding unique solutions for the specific context by combining approaches that other teams already implemented. So the main challenge, DevOps tasks can not be divided from a business perspective and require a business context that is not always provided from the start.

As we’ve spoken already, DevOps practices can drive a lot of changes. The biggest challenge is to be able to communicate their value to the management and for the latter to understand how all these processes are interconnected. So, openness and efficient communication on different company levels are challenging. I am lucky to work with teams where I can initiate a lot of improvements, and it motivates me a lot.

### “DevOps specialists become the ones who think about multiple aspects even if not asked to do it directly.”

Also, cool DevOps engineers do a lot of volunteer work today. What I mean is that dynamic teams that deal with startup projects can not afford to hire multiple specialists responsible for some deep tech expertise. Such people as DevSecOps engineers, site reliability engineers, FinOps experts, and more. DevOps specialists in such teams become the ones who think about multiple aspects even if not asked to do it directly. Because a DevOps engineer sees how the system operates as a whole and understands its weak and strong sides better. This results in a high workload for DevOps engineers and requires great stress resistance.

**Q. Where is it all heading? Are there any trends that you observe?**

With the numerous new DevOps branches such as DevSecOps, FinOps, GitOps, and others, more and more engineers will dive into narrow expertise. I see both benefits and disadvantages here. I still believe a good DevOps engineer should know a bit from everywhere and have some things that make up most of the tech expertise. 

Another trend is the standardization of DevOps practices, frameworks, and approaches. I believe this is the major trend I observe these days. And Kubernetes plays the major role here. Of course, there won’t be a one-size-fits-all approach developed, but DevOps work will get more systematic and consistent across projects.

One more tendency that is changing the way businesses launch tech products is the emergence of low-code solutions that allow businesses to create tech products without hiring an engineering team. This applies only to some cases. But still, the rise of low-code solutions influences the market, and the way tech businesses start their journeys. What I mean, is that not every team should worry about the DevOps model from the start. This also helps companies to test business hypotheses faster, and this is great.

I also believe that no matter what the future holds for tech businesses, DevOps will continue to become more and more connected with a business perspective and, thus, will pivot a lot. The great thing about DevOps is that its maneuverability allows you to fly those routes that you thought were out of reach before. You just need the right team on board.






Our DevOps engineer Ihor Mozil speaks of the value to business the DevOps model can bring and the main aspects that influence its realization.

Interview with Wise’s DevOps engineer: Business value of DevOps

A conversation about the business value of DevOps

email-generation.svg

Graph API

## Problem
Each user from the marketplace of 10 million users has an individual set of products that are recommended for purchase. Actual up-to-date data on products recommended for purchase must be collected, stored, and delivered to each user in a “Resources you may like” (RYML) email. A way to generate RYML emails with the relevant content and send them to users on a regular basis has to be developed.

### Goals and challenges
- Gather actual up-to-date lists of products recommended for purchase for each user.
- This data must be stored and then delivered to all subscribed users via “Resources you may like” (RYML) emails on a regular basis.
- Delegate email creating and sending sequence functions to email sending service.
- Maintain business logic of collecting recommended data and providing it to the email sending service.

## How Wise Engineering helped
The Recommendations Email Generation Pipeline solution was developed. It allows us to gather accurate information about products recommended for purchase for each user and store this info afterward for a needed period of time.
Results
- Information about recommended products is being accurately collected.
- Email Pipeline solutions gives an ability to transfer collected information regarding 10 million users to the email sending service within only 5 hours.
- Email sending service gets the information if a particular user has any recommended products, and can easily send that list in email messages on a regular basis.

## Background
There is a marketplace with 10 million users and each user has specific products that are recommended for their purchase. This information must be collected and delivered to the user on a regular basis.'

## Implementation process

To begin with, we focused on how to orchestrate composition of microservices and to queue extracted products’ recommendations data. Next, we’ve evaluated options for transforming this data into a format that is readable for third-party services. Subsequently, we’ve designed a way to load this data into an email sending service.

The following tools and techniques were applied to achieve these goals:

1. PHP script to go over all users to group them and chunk by task type (if it has recommendation or not). After that, assigned user ID chunks into queue.
2. PHP workers process the queue with user’s ID data and retrieves recommendations for each of them.
3. Elixir Graph API application exposes data on recommended products to PHP workers. Next, it processes products’ recommendation data and applies some internal rules and limitations on it.
4. Python REST API application generates product recommendation data for a particular user ID on demand.

## Solution

We came up with the solution and called it “Email Pipeline.” It starts with running PHP script to go over all users and creates a bunch of async tasks. Each task contains information about:
- User IDs that should be processed
- What kind of operation needs to be done

It may happen that some users do not have any recommendations. In this case, we have to clean up their data on the third-party email service. The rest of them should be updated with new recommendations data. This diversification happens on the fly and group users based on the type of task.

The tasks creation process also has a time ramp up feature. Task processing jobs are heavy and time costly operations. To mitigate that, pipeline services run on Kubernetes’ cluster with autoscaling, depending on their load. So, to make sure that autoscaler has enough time to ramp up the cluster size, we create tasks with ramp up time. We increase the rate of tasks being created in 1 hour. After that time, we spawn tasks as fast as we can as all services are scaled and ready to process increased loads created by task processing jobs.

While tasks are being created, workers start to pick them up and process. The process starts with defining the type of task. In case a user doesn’t have any recommendations, it goes straight to a third-party email service and nulls recommendation data for this user. If the user has products recommendations data, then we need to fetch it fir

To fetch this data, we call Elixir Graph API service which represents a middleware between tasks, or in our case, front-end and recommendation storage. Recommended products data itself is stored in a non-precalculated state hold by Python microservice. This service can multiply matrices on the fly and calculate scores for possible product recommendations and return the data to the caller. After Graph API calls and receives data from Python microservice, the data needs to be processed and cleaned up as this service just keeps the list of recommended products and doesn’t know anything about the user. The cleanup process means removing items that were bought by a user already, were wishlisted, added to the cart, or even seen somewhere else on the site. This information is accessible by Graph API so we can easily clean up recommendations we don’t want to be served to the user next time.

We now have information about the set of products recommended to each user in the task, regardless whether it is nulled data or new recommendations. The only step left is to transfer this data to a third-party email sending service. We will use API of the third-party email sending service to perform this.

As a result, the email sending service has the information if a particular user has any recommended products, list of recommended products if there are any, and can easily send that list in email messages on a regular basis.

## Results

The Recommended Products Emails Generation Pipeline gives the ability to transfer collected information about relevant recommended products concerning 10 million users to the email sending service in 5 hours. This allows us to send “Resources You May Like” emails on a daily basis. All tasks that are related to email composing have been delegated to a third party email sending service. This gives us an opportunity to send emails to all the users with one click after the info on recommended products has been imported.


Learn how our team developed recommendations email generation pipeline and what results we achieved.

Recommendations email generation pipeline

scheduled-emails.svg

## Why?

Our main target is to develop a system for sending large numbers of emails and then tracking users’ interaction with a received mail (e.g. opens, link clicks, unsubscribes, etc.) Users with large amounts of daily sends prefer use scheduling for sending use.

## Why we need scheduling

Using scheduling, instead of send-immediately flows, gives us several advantages.
- Better control for sending process
- If a user needs to send emails after his workday is finished, then he won’t be able to monitor the sending process and fix them if something goes wrong. With scheduling, users can prepare all emails for sending before the actual send date, and our ESP (Email Service Provider) handles the rest.Less resources required
- If you already have content for emails that you need to send (for example tomorrow), you are able to send the emails gradually. For example, if you need to send 5M emails at 8 PM you can start sending at 10 AM and continue sending in small batches the whole day. We collect the emails from our side and cover the delivery in time.No personal CRON.

If you have a notification email that needs to be sent after some time (for example, a subscription reminder), just send the notification to our ESP and we cover the scheduling. You don’t need to handle that case with personal Cron scripts.

## Problems that we faced when building scheduling

We propose pretty simple API for working with scheduling, but it’s not easy to organize simultaneously sending even 1M emails. We needed to think about scalability and our system in the future. We had to solve several problems.

- Proper storage for scheduled emails
- Limited throughput of the network for one process
- Split scheduled emails that were ready to send for different workers
- Syncing statuses of scheduled emails and avoiding duplication on send

## Proper storage for scheduled emails

We should able to filter “ready for send” emails as fast as possible among large amounts of records that don’t need to send yet. At the time we already used ClickHouse database as storage for our analytics events (unsubscribe, clicks, open, etc.).

Clickhouse is a column-oriented database that is used mostly for analytics. But with the read/write throughput that ClickHouse provides for us we are able to use the solution as scheduling storage.

The most valuable storage requirements for us were the following:
- Local and distributed joins
- Data compression
- Parallel and distributed query execution
- Cross-datacenter replication
- High availability
- SQL support
- Fetch speed not less than 500K email records per second.

## Performance testing

We made some performance tests and got results that completely satisfied our requirements:

- Read speed is 1.2 GB/s
- Write speed is 50 to 200 MB/s
 - Finding 335K scheduled emails on a dataset with the size of 5M takes 0.061 second

As a result, with network latency for transfers, we were able to fetch 3.5M emails per second. That was a very good result for us.

## Limited throughput of the network for one process

We resolved the problem of choosing our data storage and then faced another issue. We used NodeJS as our main platform, the platform, together with SQS queues, didn’t provide the desired performance for us. Then we wrote simple prototypes for testing using Golang and Elixir (we hoped that Erlang with its lightweight threads helps us) we got some throughput increases, but still weren’t reaching the desired results and scalable solutions.

Results of our benchmark’s queue inserting time for 50K messages to queue was as follows:
- NodeJS 1 thread = 95.408 sec
- NodeJS 2 thread = 75.757 sec
- Go = 35.564 sec

Elixir was about 2 times slower than NodeJS with 2 threads and the same kind of problem.
So obviously, we needed to find ways to scale horizontally.


## Split scheduled emails that are ready to send for different workers

Obviously, we needed to make our scheduling scalable and found a way to split all scheduled emails for workers. Workers should have the same role and know nothing about each other. It’s allowing us to run more workers on more servers when we need to process the largest amount of emails.

When workers are fully independent from each other, we decided to create one more component in our system.

The component is named Ranger and is responsible for splitting ranges for our workers. Each range shouldn’t intersect with other ranges.

With Ranger and Clickhouse working together, we created the following architecture for our scheduling feature:

1. Emails send to REST API, and if a parameter with a future send date exists, API puts the email to Messaging Queue.
2. Write to the consumer processing the Queue and save all emails inside Clickhouse database.
3. Ranger runs query for fetching emails and the query contains the following steps: Records filtered by send date and each time we fetch all records that are ready in that moment. Filter records by time-created range, we need only those that weren’t affected by the previous step. For that, we fetch only records that have a date greater than biggest date from previous execution. Aggregate results by CreatedTime and COUNT function.
4. Set first fetched date as DateFrom for query.
5. Wait for full batch size and set last date as DateTo.
6. Update Date to as last processed range.
7. Send range to Queue for processing by worker.
8. Consumer pulls a Queue with ranges and fetches by CreatedTime emails for sending to user.
9. Each email sends to target user.


![scheduled emails scheme](https://wise-site.fra1.digitaloceanspaces.com/scheduled_emails_scheme_94e7ec9efe.svg)

## Avoid duplication

AWS SQS service promises not less than one delivery for a Standard Queue. this means that sometimes we can obtain one message on two workers and send our email twice. Since each message can contain multiple emails (our current batch size contains 5K) we need to organize duplicate handling.

For that, we have the Redis database and we store each SQS messageID as a Redis key. The keys have an expiration date that's equal to message visibility timeout in the Queue. When a consumer obtains a new message from SQS, he checks for an existing messageID in Redis. If the message already exists, the consumer skips this one.

## Conclusion

Using ClickHouse and AWS Simple Queue Service, we build a scalable scheduling system for delivering emails. These flows and tools can be applied to any actions that could be scheduled, not just for emails.

At the moment, we can fetch 335K records per second, filtering by ready date takes 0.1 seconds for 5M database set (including network latency), and workers that process the emails can easily scale horizontally for the desired throughput.


Check out how our team built a scalable scheduling system to deliver emails via ClickHouse and AWS Simple Queue Service.

How to send large amount of scheduled emails simultaneously

s3-file-upload.svg

## Problem

Users could not upload all their products to the marketplace because most of them were files that are larger than 200 MB or were video files. Both were not supported for upload; there was a size limitation for digital files, and video files were not supported at all.

## Goals and challenges

- Allow users to upload files that are larger than 200 MB.
- Enable users to continue file upload after the network error from user side has occurred.
- Scan uploaded files for malware.

## How WE helped

Direct upload of files to Amazon S3 was designed and successfully applied in the existing service (marketplace) within a shorter period of time than the upload system that was used before: half a month compared to 2 months. Moreover, it also enhanced security for downloaded files.

## Results

- Users could upload files up to 500 MB or up to 1 GB (depending on association with a specific user group).
- New items (Video Files) became available for upload.
- S3 server usage decreased the time and resources required for server maintenance compared to the hardware servers that were used before.
- The project received enhanced security of downloaded files.

## Background

Existing service allowed users to upload files and attach them to products. However, the service put a solid limit on the size of files that were selected for upload. As a result, it was not possible to upload a file that is larger than 200 MB. Also, it worked in a way that when a file upload failed (due to the network error), the user was forced to start the entire process all over again. A major problem was that many users had files usually larger than 200 MB. Consequently, there was a lot of requests to upload such types of files. Besides, the NGINX module which was used for file upload was very old and unsupportable.

## Implementation process

### Old upload schema

Previously, when the user uploaded a file, we used an old NGINX module. This NGINX module simply read all the content of the file and stored it on our server in a temporary directory with a temporary file name. The following problems emerged:

1. This mechanism read the whole file within one operation. This approach didn't work correctly for the big files and caused a significant load on our servers.
2. The unique temporary file name was not, in fact, unique. Sometimes, when we had lot of users on the website, we had conflicts when two or more users uploaded files with the exact same temporary file name.
3. Our upload progress bar displayed the upload process only. We could not add a progress bar for virus check, consequently users just saw a stalled progress bar while we checked the file for viruses. Also, we generated preview files, which were a reduced version of the original user’s files; because of this, users could not monitor the progression of the upload process.
4. After the upload, we stored all files on our own data servers. It required more attention from our side and resources to maintain them. Moreover, it made our codebase more complicated since we required some extra fields in the database in table item properties; we called it mask of servers, which contained information where a file for a specified item was stored. It increased complexity to select all items which stored files on specific data servers only.
5. This upload process was not secure:
- If someone hacked our system, they could catch and replace files.
- We checked files for viruses by Cron after the upload was completed. During the time between upload and check, an infected file could corrupt our system and harm other users who later bought the product and downloaded the infected file.

![direct s3 file upload old scheme](https://wise-site.fra1.digitaloceanspaces.com/direct_s3_file_upload_old_scheme_a8fadb287c.svg)

## Solution

### New upload schema

We decided to upload files directly to S3 and no longer store them on our servers. Also, we’ve chosen to use a 3rd party JavaScript library to upload files to S3 directly instead of creating our own library. We selected [https://github.com/TTLabs/EvaporateJS.](https://github.com/TTLabs/EvaporateJS. ) This library had already implemented and tested features for all known web browsers:

- Secured direct S3 file upload.
- Resume failed file upload.

Now all files were uploading to S3 into a temporary bucket for each user’s item. Also, we used unique paths and file names. We decided to split files into a smaller parts before upload and run uploads to S3 in parallel threads, each part was signed by our server so it couldn't be caught and replaced by someone else. Now when a user selects a file to upload to S3:

1. Evaporate JS uploads a file into a temporary bucket directly. During the upload process, it makes a request to our servers only for signing each part of the file. This is done to avoid cases when somebody can catch part of the uploaded file and replace it.
2. After uploading files into the temporary bucket, we start to process the files in the following order:
   - Check for viruses: If a virus is detected, we stop processing the file, move it into a quarantine bucket (it clears automatically after 4 days), and notify the user about this event.
   - Generate preview files.
   - Collect information about the file (size, checksum, type of file, etc.).
3. User fills in required text fields and clicks on submit button.
4. Process push job of funneling item into a queue:
When the user wants to create several items, we organize all the processes into a queue. Once the first item is submitted for creation/update, the user is able to select the next one and doesn't have to wait until the create/update process is finished for the first item. Once the item is created/updated, we display a message to inform the user.
5. In the background, worker process takes jobs from the queue and performs the next job in parallel threads: Save item details into database.
6. In case the background saving process has failed (network issue, Amazon issue, etc.), the queue process always restarts failed tasks a couple of times.

![direct s3 file upload old scheme](https://wise-site.fra1.digitaloceanspaces.com/direct_s3_file_upload_new_scheme_b28e226c13.svg)

## Results

The new upload process was designed and successfully applied. We gave our users an opportunity to upload files up to 500 MB. For premium users, we’ve added a new feature – files upload up to 1 GB. Also, we decided to allow upload of video files as a new item. This upload process was successfully applied in the existing service (marketplace) within a shorter period of time than the upload system that was used before: half a month compared to 2 months. The S3 server also was easier to maintain, and took less time and effort to maintain, compared to the hardware servers that were used before. Moreover, it also reinforced security for downloaded files.

Wise Engineering team helps businesses solve complex tech challenges like [migration to cloud](https://wise-engineering.com/expertise/cloud-migration/), [internal search integration](https://wise-engineering.com/expertise/elasticsearch-integration/), optimization for high load, and more. Check out our expertise in [custom software development](https://wise-engineering.com/expertise/) and contact us to discuss your future project.



Learn how our tech team developed a direct upload system to S3 and allowed users to upload large files to the marketplace seamlessly. 

Direct S3 File Upload | Wise Engineering Blog

Learn how we developed a direct upload system to S3 and allowed users to upload large files to the marketplace seamlessly. 

Direct S3 file upload

notification-system.svg

## Problem

The possibility to send a high number of emails without keeping your own server for email delivery.
Goals and challenges
- Track user interaction with the email (up to 10K notifications per minute).
- Reduce load on 3rd party app that sends high number of emails daily.
- Webhook receiver’s downtime shouldn’t affect main application flow.
- System has to be scalable and guarantee to deliver within one day, when hook target isn’t available.
- Convenient and timely email delivery status reporting and displaying this info on a dashboard.
- Service is easy to use and customizable to meet changing requirements.
- Availability and comprehensiveness of documentation.

## How WE helped

We've implemented the hooks delivery system with Amazon Simple Notification Service (SNS) and proxy server that processes, subscribes, and redirects requests. Little coding was required and responsibility of webhook delivery is assigned to an external service.

### Results

- Email delivery system with the user’s activity tracking feature that processes a high number of events (up to 20K notifications per minute).
- Information about user interaction with the email is captured and stored.
- Redelivery solution is available when webhook receiver is unreachable.
- System is separated and does not affect main app services.
- Solution is easy to scale.
- 2 times higher level of throughput.

## Implementation process

To achieve project objectives, we’ve considered several implementation options:
- Naive implementation.
- Implementation via SQS queue.
- Webhooks delivery SNS notification.

We have the following components that are responsible for events in our system:
1. Event consumer.
2. Pull queue and decide what we need to do with events.
3. Event storage.
4. Store all events in our system.
5. 3rd party app.
6. Have to be informed about events related to the app account.

## Native implementation

The simple way to inform a 3rd party API is just to make HTTP request the hook URL directly from the consumer, as displayed on the following diagram:

Pros:
- Easy to implement.

Cons:
- Main application flow is affected, thus if a 3rd party app won't respond, we will receive an error in our internal consumer.
- Consumer cannot be scaled separately from webhooks which are sent to 3rd party app.
- Impossible to implement webhooks redelivery correctly.

Accordingly, we came to the conclusion that this approach will not give us expected results or satisfy our needs in a comprehensive manner.

Therefore, we’ve started to consider other options.

![Native implementation](https://wise-site.fra1.digitaloceanspaces.com/webhoooks_native_implementation_2f8d52c370.svg)

## Implementation via SQS Queue

We use a lot of AWS services at the moment and already have a bunch of SQS Messaging Queues. The main idea for the solution it is to use a separate queue for hooks (webhooks) and consumer to process this queue. Take a look at the schema that shows how SQS queue works:
In this manner our main app flow does not depend on the working state of an external app and we are able to implement retries via SQS queue.

The flow of a retry is as follows:

1. Pull webhooks events from queue.
2. Send HTTP request with events in the body to 3rd party app.
3. If request is completed successfully, then remove message from queue.
4. If request has failed, retry with the specified delay and send a message once again.
5. Set maximum receive count that equals delivery retry count.

![Implementation via SQS Queue](https://wise-site.fra1.digitaloceanspaces.com/webhooks_implememntation_via_SQS_queue_abcecd5e9e.svg)

The flow is displayed on diagram below:
Pros:
- Main application flow is not affected.
- Redelivery can be easily implemented.
- Independent main event consumer can be scaled.

Cons:
- More complex to implement.
- Max redelivery delay is 15 minutes (since max delay is 15 minutes in SQS queues), thus we can’t make a logarithmic retry for our messages.
- Logic of retries and responsibility for webhook delivery has to be managed manually.

This option was way better, however, not all of our primary goals would be met by implementation via SQS queue.

![Implementation via SQS Queue general scheme](https://wise-site.fra1.digitaloceanspaces.com/webhooks_implememntation_via_SQS_queue_general_c43aaf6c6e.svg)

## Webhooks delivery by SNS Notification

Another way to implement webhooks delivery is by using AWS SNS. The service was developed for delivering notifications and can send them via HTTP, the same way as webhooks do. Also, if notification delivery has failed, SNS is responsible for retry delivery, and we can use linear or logarithmic function to manage retry frequency.

This solution will give us desirable outcomes; however, it cannot be applied due to these two reasons:

1. SNS send confirmation requests to subscribe endpoint (webhooks destination).
2. SNS provides redundant metadata to delivered messages.

The SNS was designed for communication between different components within one system and not for communication among different components of different systems with different owners. Hence, we need a proxy component inside our system that SNS can communicate with

Let's take a look at the scheme of proposed webhooks implementation using SNS:

![Webhooks delivery by SNS Notification](https://wise-site.fra1.digitaloceanspaces.com/webhooks_delivery_by_SNS_notification_ed2086657b.svg)

Flow of hooks delivery using SNS:

1. The consumer sends events to a certain SNS topic.
2. We subscribe a 3rd party app to the topic and the URL should contain an endpoint that we need a redirect for.
3. When the proxy receives a request for subscription confirmation, it will send a request to SNS with a proper confirmation token.
4. When the proxy receives a request with a notification (an event, in our case), it will format event data and redirect to a URL that contains the URL path part.
5. All responses from the hooks’ endpoints pass as responses to SNS requests.

Schema of proxy API is shown below:

Pros:

 - Main application flow isn’t affected by the state of the hooks’ receiver application.
- The responsibility for message delivery is shifted to SNS.
- Redelivery strategy is already implemented by AWS.
- The simple proxy has good throughput, very lightweight and can be scaled.
- Little code is needed (less code, fewer bugs).

Cons:
- The proxy has to be developed.

![Webhooks delivery by SNS Notification scheme 2](https://wise-site.fra1.digitaloceanspaces.com/webhooks_delivery_by_SNS_notification_scheme2_1eb5d905fe.svg)

## Solution

After considering each implementation option in terms of benefit it will bring, and required resources for its execution, we’ve chosen to go with the webhooks delivery by SNS notification. With SNS and the simple proxy server, we’ve implemented the hooks delivery system with minimum coding involved and shifted responsibility of webhook delivery to an external service. As a result, we are able to send about 20,000 events per minute without any scaling, which is even higher compared to the planned 10,000 events per minute. The major bottleneck is the proxy, but we designed this one for scaling so throughput can be easily increased.

## Additional info

We wrote our proxy using Node.js platform. The proxy hosted on EC2 T2 small instance and can process about 4,000 concurrent requests with throughput of about 400 req/sec. The proxy doesn’t have any interaction with the database since all data for redirects are passed in URLs. This gives us an option to do a horizontal scaling. If we place our proxy behind a load balancer, then we will be able to add more server instances and as a result obtain two times higher throughput.

Vertical scaling won’t give us the same effect. When we scale node processes within one server instance, we can get about 30% higher throughput for each additional processor kernel. Therefore, horizontal scaling is a more suitable and effective approach.

We don’t need to worry about scaling SNS because AWS does it automatically, we only have to send the proper count of messages.


Here’s how we implemented the hooks delivery system with Amazon Simple Notification Service (SNS) and a proxy server.