Archives
-
The week in charts
The Week in Charts
Share this email
Share these insights
Did you enjoy this newsletter? Forward it to colleagues and friends so they can subscribe too. Was this issue forwarded to you? Sign up for it and sample our 40+ other free email subscriptions here.
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to The Week in Charts newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Week in Charts" <publishing@email.mckinsey.com> - 03:30 - 2 Nov 2024 -
EP136: The Ultimate DevOps Developer Roadmap
EP136: The Ultimate DevOps Developer Roadmap
Forwarded this email? Subscribe here for moreGenerate Handwritten SDKs (Sponsored)
Invest hundreds of hours your team doesn't have in maintaining SDKs by hand or generate crappy SDKs that leave a bad impression on your users. That's two bad options. Fortunately, you can now use Speakeasy to create ergonomic type-safe SDKs in over 10 languages. We've worked with language experts to create a generator that gets the details right. With Speakeasy, you can build SDKs that your team is proud of and offer code-native API docs as well.
This week’s system design refresher:
Everything You NEED to KNOW About Web Applications (Youtube video)
The Ultimate DevOps Developer Roadmap
6 Software Architectural Patterns You Must Know
Top Eventual Consistency Patterns You Must Know
SPONSOR US
Everything You NEED to KNOW About Web Applications
The Ultimate DevOps Developer Roadmap
Programming Languages
Pick and master one or two programming languages. Choose from options like Python, JavaScript, Go, Ruby, etc.Operating Systems
Master the ins and outs of major operating systems like Linux, Windows, Mac, and so on.Source Control Management
Learn about source control management tools such as Git, GitHub, GitLab, and Bitbucket.Networking
Master the basics of networking concepts such as DNS, IP, TCP, and SSH.CI/CD
Pick tools like GitHub Actions, Jenkins, or CircleCI to learn about continuous integration and continuous delivery.Scripting and Terminals
Learn scripting in bash, and PowerShell along with knowledge of various terminals and editors.Hosting and Platforms
Master multiple hosting platforms such as AWS, Azure, GCP, Docker, Kubernetes, Digital Ocean, Lambda, Azure Functions, etc.Infrastructure as Code
Learn infrastructure as code tools like Terraform, Pulumi, Ansible, Chef, Puppet, Kubernetes, etc.Monitoring and Logging
Master the key tools for monitoring and logging for infrastructure and applications such as Prometheus, Elasticsearch, Logstash, Kibana, etc.Basics of Software Development
Learn the basics of software development such as system availability, data management, design patterns, and team collaboration.
Over to you: What else would you add to this roadmap?
New Relic unveils the industry's first Intelligent Observability Platform (Sponsored)
The New Relic Intelligent Observability Platform, transforms observability from ensuring uptime and reliability into a key driver of business growth and developer velocity for enterprises worldwide. With new innovations such as New Relic AI and GitHub Copilot integration, New Relic Pathpoint Plus for business observability, and New Relic Retail Solution.
The Ultimate Redis 101
Redis is one of the most popular data stores in the world and is packed with features.
Here are 8 simple steps that can help you understand the fundamentals of Redis.
What is Redis?
Redis (Remote Dictionary Server) is a multi-modal database that provides sub-millisecond latency. The core idea behind Redis is that a cache can also act as a full-fledged database.Redis Adoption
High-traffic internet websites like Airbnb, Uber, Slack, and many others have adopted Redis in their technology stack.How Redis Changed the Database Game?
Redis supported main memory read/writes while still supporting fully durable storage. Read and writes are served from the main memory but the data is also persisted to the disk. This is done using snapshots (RDB) and AOF.Redis Data Structures
Redis stores data in key-value format. It supports various data structures such as strings, bitmaps, lists, sets, sorted sets, hash, JSON, etc.Basic Redis Commands
Some of the most used Redis commands are SET, GET, DELETE, INCR, HSET, etc. There are many more commands available.Redis Modules
Redis modules are add-ons that extend Redis functionality beyond its core features. Some prominent modules are RediSearch, RedisJSON, RedisGraph, RedisBloom, RedisAI, RedisTimeSeries, RedisGears, RedisML, and so on.Redis Pub/Sub
Redis also supports even-driven architecture using a publish-subscribe communication model.Redis Use Cases
Top Redis use cases are Distributed Caching, Session Storage, Message Queue, Rate Limiting, High-Speed Database, etc.
Over to you: What else will you add to get a better understanding of Redis?
6 Software Architectural Patterns You Must Know
Choosing the right software architecture pattern is essential for solving problems efficiently.
Layered Architecture
Each layer plays a distinct and clear role within the application context.
Great for applications that need to be built quickly. On the downside, source code can become unorganized if proper rules aren’t followedMicroservices Architecture
Break down a large system into smaller and more manageable components.
Systems built with microservices architecture are fault tolerant. Also, each component can be scaled individually. On the downside, it might increase the complexity of the application.Event-Driven Architecture
Services talk to each other by emitting events that other services may or may not consume.
This style promotes loose coupling between components. However, testing individual components becomes challengingClient-Server Architecture
It comprises two main components - clients and servers communicating over a network.
Great for real-time services. However, servers can become a single point of failure.Plugin-based Architecture
This pattern consists of two types of components - a core system and plugins. The plugin modules are independent components providing a specialized functionality.
Great for applications that have to be expanded over time like IDEs. However, changing the core is difficult.Hexagonal Architecture
This pattern creates an abstraction layer that protects the core of an application and isolates it from external integrations for better modularity. Also known as ports and adapters architecture.
On the downside, this pattern can lead to increased development time and learning curve.
Over to you: Which other architectural pattern have you seen?
Top Eventual Consistency Patterns You Must Know
Eventual consistency is a data consistency model that ensures that updates to a distributed database are eventually reflected across all nodes. Techniques like async replication help achieve eventual consistency.
However, eventual consistency can also result in data inconsistency. Here are 4 patterns that can help you design applications.
Pattern#1 - Event-based Eventual Consistency
Services emit events and other services listen to these events to update their database instances. This makes services loosely coupled but delays data consistency.
Pattern#2 - Background Sync Eventual Consistency
In this pattern, a background job makes the data across databases consistent. It results in slower eventual consistency since the background job runs on a specific schedule.
Pattern#3 - Saga-based Eventual Consistency
Saga is a sequence of local transactions where each transaction updates data with a single service. It is used to manage long-lived transactions that are eventually consistent.
Pattern#4 - CQRS-based Eventual Consistency
Separate read and write operations into different databases that are eventually consistent. Read and write models can be optimized for specific requirements.
Over to you: Which other eventual consistency patterns have you seen?SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com
Like
Comment
Restack
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:35 - 2 Nov 2024 -
How to future-proof your leadership
by "McKinsey Highlights" <publishing@email.mckinsey.com> - 11:09 - 2 Nov 2024 -
The new arenas of competition
The Shortlist
Share this email
Curated by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of the CEO Shortlist, a biweekly newsletter of our best ideas for the C-suite. This week, we feature our latest research on leadership and the emerging industries that will define dynamic growth in the coming years. We appreciate the opportunity to connect and hope you find our perspectives novel and insightful. Please let us know what you think! You can reach us at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com. Thank you.
—Alex and Axel
Ten years ago, CEOs and top teams typically focused on four or five critical issues at any one point in time. Today, it’s eight or ten. In this fragmented landscape, leading a global organization takes a new kind of focus. Our latest research highlights two important ways CEOs can level up. First, recognize that the personal attributes and best practices of leadership are distinctly different from what they used to be. Second, build a deep bench of talented leaders, using new leadership development approaches to shape, develop, and mentor the next generation of managers.
Certain industries create more value and have a greater impact than others. We call these outperforming industries arenas of competition. A new report from the McKinsey Global Institute identifies 18 future arenas that could reshape the global economy between now and 2040, including AI software and services, future air mobility, drugs for obesity, and robotics. All told, they could generate up to $6 trillion in profits. These fast-growth industries are also defined by high dynamism, meaning market share within them changes hands to an outsize degree. This spells opportunity for modern, focused leadership teams.
We hope you find these ideas inspiring and helpful. See you next time with more McKinsey ideas for the CEO and others in the C-suite.
Share these insights
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to The CEO Shortlist newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey CEO Shortlist" <publishing@email.mckinsey.com> - 04:57 - 1 Nov 2024 -
How much could the space economy be worth in 2035?
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
The expanding space economy. Activity in space is accelerating, and along with it, growth in the burgeoning space economy. By 2035, the global space economy could be worth $1.8 trillion, according to senior partner Ryan Brukardt, who leads McKinsey’s work in the space sector, and coauthors. That’s up from $630 billion in 2023. Key factors propelling this growth include higher demand for greater internet connectivity through satellites, positioning and navigation services on mobile phones, and AI-based insights.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:36 - 1 Nov 2024 -
You're invited! Join us for a virtual event on the next big arenas of competition
Share this email
New from McKinsey & Company
Which industries will reshape the global economy in the next 15 years? Over the past two decades, a remarkably small number of industries have significantly transformed the global business landscape. These ‘arenas of competition’ are marked by their outsized growth and shifts in market share.
In a new report, the McKinsey Global Institute used the criteria of high growth and dynamism to identify 18 industries with potential to become the arenas of competition by 2040, spanning AI, space, modular construction, and nonmedical biotech. These arenas are set to reshape many aspects of the global economy and society, including data consumption, health, communication, and mobility, with broader implications for ethics, data privacy, and sustainability. By 2040, these arenas could generate $29-$48 trillion in revenues, $2-$6 trillion in profits, and as much as one-third of global GDP growth.
Join us on Wednesday, November 13 at 12:30PM-1:30PM ET (6:30PM-7:30PM CET) for a virtual event to explore the findings of a new report, The next big arenas of competition, featuring a presentation by the report authors and leading McKinsey experts. The session will explore:•
The specific factors that contribute to the extraordinary growth and dynamism of arenas of competition compared to other industries
•
What strategies companies should adopt to effectively navigate and compete within these emerging arenas of competition, considering the key ingredients of arena creation
•
Lessons learned from the performance of the 12 arenas of today that can be applied to understanding and leveraging the future arenas
•
How the emergence of these 18 future arenas will impact global economic trends and societal structures over the next 15 years
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Global Institute alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey & Company" <publishing@email.mckinsey.com> - 12:25 - 31 Oct 2024 -
[PROJECT FUNDING]
Dear,
I hope this email finds you well.
I'm writing to offer my expertise in estimation, design, and architectural project management. If you're experiencing an overflow of work, I would be delighted to assist you with these tasks.
Please feel free to reach out if you require any support in these areas. I look forward to your response and the opportunity to discuss how I can contribute to your projects.
Respectfully.
by "Jonson Grey" <jonson.globalbids@gmail.com> - 11:43 - 31 Oct 2024 -
Why Executives Seem Out of Touch, and How to Reach Them
Why Executives Seem Out of Touch, and How to Reach Them
Forwarded this email? Subscribe here for moreLatest articles
If you’re not a subscriber, here’s what you missed this month.
To receive all the full articles and support ByteByteGo, consider subscribing:
For this week's issue, I am pleased to introduce our guest author, Ethan Evans, an ex-Amazon VP who played a pivotal role in creating some of the groundbreaking services we use today, such as Prime Video, Amazon Video, the Amazon Appstore, Prime Gaming (formerly Twitch Prime), and Twitch Commerce.
Beyond his experience as an engineering leader, Ethan is also a career growth coach who has assisted numerous individuals to get unstuck and level up their careers.
If you would like to connect with Ethan, you can do so on Twitter and LinkedIn. Also, be sure to check out his newsletter and career community, Level Up (ByteByteGo readers get a 25% discount off the annual subscription—highly recommended for more deep executive insights), and consider his highly rated live course on how to get unstuck and break through to the next level in your career.
Employees in larger organizations often feel surprised and disconnected from the actions and decisions of their leaders. The bigger the organization and the more distance to the leader, the more likely it is that their choices will be surprising and seem “out of touch.”
Comic by Work Chronicles for ByteByteGo... 
Continue reading this post for free in the Substack app
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:35 - 31 Oct 2024 -
Do your company’s operations help it reach peak productivity?
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
Declining productivity. Simply put, the world needs more productivity growth; it’s the best antidote to wealth inequality, inflation, and debt. But productivity growth has declined since the 2008 global financial crisis. McKinsey senior partner Richard Sellschop and partners Erik Schaefer and Joris Wijpkema explain some reasons why. Technological improvements may be becoming more incremental, while the returns from companies’ investments in restructuring and offshoring have been declining.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:35 - 31 Oct 2024 -
How generative AI is transforming B2B sales
Re:think
FRESH TAKES ON BIG IDEAS
ON GENERATIVE AI AND B2B SALES
How generative AI can help B2B sales become more effectiveThe buzz over the past couple of years about AI, especially generative AI (gen AI), has been, at times, overwhelming. There have been a lot of bold predictions about how it will play out. But in the B2B world, people are asking questions about what’s practical. They want to know what to focus on, how to prioritize, how fast to move, and where to make investments. It ultimately comes down to basic business principles. Where is the value in improving the customer experience, getting pricing right, and speeding up responses to customers? And how does it line up with growth?
Gen AI’s value can vary widely across business models and companies. B2B companies that sell to small and medium-size businesses need to focus on customer acquisition and how to grow their customer base, since new entities start up frequently. Companies that sell to larger enterprises need to maintain and expand relationships. Then there are companies that may want to concentrate on personalization and fostering loyalty because their customers want a consumer-like experience focused on brand and products. There’s a lot of power in identifying customer opportunities.
Companies can pull together information about customer needs and their propensity to buy, then use it to autogenerate content. Gen AI can help identify small-business owners who are likely to be interested in a specific product. It can also generate a slide presentation for the pitch and draft an email to send before a meeting. It could even do this in multiple languages. An English speaker who wants to sell to someone who works mostly in Spanish can use gen AI to craft and edit an email, translate it into Spanish, and send it automatically.
A couple of things have surprised me as companies increase their use of gen AI. The first is how hard it is to go from an initial idea to scaled use. Human behavior is really hard to change. When hundreds or thousands of people need to adapt, it can be challenging to instill confidence and trust in new capabilities and get people to effectively refocus their work. The other surprising and amazing thing is how quickly the quality of gen AI output is improving. Even just a few months ago, the level of hallucination (when an AI model produces false or misleading results) was much higher. It’s getting a lot harder to tell the difference between what was generated by a computer or a human.“For every dollar a company spends on technology, it needs to be spending the same, if not more, on the people side.”
There are two common misconceptions I see from B2B companies in the early stages of their gen AI journey. The first is that it’s a straightforward issue for their IT teams. That outlook sets companies up for failure because gen AI has to be addressed deep at the core of how a business is run and how human capital is deployed. The second false notion is that legacy companies often think they don’t need technology solutions because they sell widgets. A decade ago, there were similar beliefs about e-commerce: many B2B players thought it was a consumer thing that wouldn’t impact them. Well, fast-forward to today and we see that e-commerce has grown explosively and is now the preferred B2B channel. That same dynamic is happening now. A lot of folks question whether gen AI is relevant for them. But given the exponential curve with which gen AI capabilities are improving, it’s important to be in the game no matter what the business is.
What could the future look like? So far, the focus has been on how gen AI can make it easier for humans to do certain tasks by automating communications and saving time. The conversation is now pivoting toward improving the customer experience and finding opportunities where a human doesn’t even need to be involved, like using an AI agent chatbot in customer service. Further out, when a commercial organization really embraces gen AI holistically, marketing, sales, and pricing could blend together. That would create an opportunity for people to craft more personalized experiences, helping customers really get value across multiple channels.
It’s important to stay grounded in what matters for customers and what drives them to purchase. For every dollar a company spends on technology, it needs to be spending the same, if not more, on the people side. That means not just upgrading capabilities but also rethinking processes and resource allocation. If companies are clear on how gen AI is going to help the customer, then they can focus on doing what their workforce needs to get the full value out of it.Share Steve Reis’s insights
ABOUT THIS AUTHOR
Steve Reis is a senior partner in McKinsey’s Atlanta office.
MORE FROM THIS AUTHOR
UP NEXT
Tamara Charm on holiday shopping
Consumers might have full wallets, but their sentiment is subdued. Retailers that focus on value and their omnichannel strategy are poised to win this holiday season.
Follow our thinking
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Quarterly alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Quarterly" <publishing@email.mckinsey.com> - 01:07 - 30 Oct 2024 -
Supply chains are still vulnerable. What can companies do?
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
Continuing challenges. Between trade tensions, attacks on commercial shipping, and delays in automotive production, the supply chain disruptions keep on coming. The latest McKinsey Global Supply Chain Leader Survey suggests that problems such as these remain the norm, not the exception. According to McKinsey partners Knut Alicke and Tacy Foster, nine in ten respondents say they have encountered supply chain challenges in 2024. At the same time, overall investment in supply chain digitization is leveling off after rapid growth between 2020 and 2023.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:50 - 30 Oct 2024 -
New Sales Order - Ref[4] - (Maqabim Distributors & British Columbia Liquor Distribution Branch)
Hello Fabien,It would be great to talk to you on phone. Hope we can get to discuss some of our mutual customers and suppliers.Ship fromShip toMaqabim DistributorsBritish Columbia Liquor Distribution BranchI am looking forward talking to you and to give you a top-class unmatched Rates with a GUARUANTEE. Our rates will be better than your carriers that you work with regularly. Just add us to your list.Regards
Scott Adams
Director National Sales
HEAD OFFICE
Phone: +1 (289) 295-1640 ext 102
Address: 2275 Upper Middle Road East, Suite 101
Oakville, ON L6H 0C3
WE GOT CANADA-USA-MEXICO COVERED
by "Scott Adams -MGCFREIGHT" <scott@mgcfreight.com> - 03:26 - 29 Oct 2024 -
1.8 Trillion Events Per Day with Kafka: How Agoda Handles it
1.8 Trillion Events Per Day with Kafka: How Agoda Handles it
Forwarded this email? Subscribe here for moreCut Code Review Time & Bugs into Half with CodeRabbit (Sponsored)
CodeRabbit is an AI Code Reviewer that helps you or your team merge your code changes faster with superior code quality. CodeRabbit doesn’t just point out issues; it suggests fixes and explains the reasoning behind the suggestions. Elevate code quality with AI-powered, context-aware reviews and 1-click fixes.
CodeRabbit provides:
• Automatic PR summaries and file-change walkthroughs.
• Runs popular linters like Biome, Ruff, PHPStan, etc.
• Highlights code and configuration security issues.
• Enables you to write custom code review instructions and AST grep rules.
To date, CodeRabbit has reviewed more than 5 million PRs, is installed on a million repositories, has 15k+ daily developer interactions, and is used by 1000+ organizations.
PS: CodeRabbit is free for open-source.
Disclaimer: The details in this post have been derived from the Agoda Engineering Blog. All credit for the technical details goes to the Agoda engineering team. The links to the original articles are present in the references section at the end of the post. We’ve attempted to analyze the details and provide our input about them. If you find any inaccuracies or omissions, please leave a comment, and we will do our best to fix them.
Agoda sends around 1.8 trillion events per day through Apache Kafka.
Since 2015, the Kafka usage at Agoda has grown tremendously with a 2x growth YOY on average.
Kafka supports multiple use cases at Agoda, which are as follows:
Analytical data management
Feeding data into the data lake
Near real-time monitoring and alerting solutions
Building asynchronous APIs
Data replication across data centers
Serving data to and from Machine Learning pipelines
As the scale and Kafka usage grew, multiple challenges forced Agoda’s engineering team to develop solutions.
In this post, we’ll examine some key challenges that Agoda faced and the solutions they implemented.
Simplifying how Developers Send Data to Kafka
One of the first changes Agoda made was around sending data to Kafka.
Agoda built a 2-step logging architecture:
A client library writes events to disk. It handles file rotations and manages the write file locations.
A separate daemon process (Forwarder) reads the events and forwards them to Kafka. It is responsible for reading the files, sending the events to Kafka, tracking file offsets, and managing the deletion of completed files.
See the diagram below:
The architecture separates operational concerns away from development teams, allowing the Kafka team to perform tasks like dynamic configuration, optimizations, and upgrades independently. The client library has a simplified API for producers, enforces serialization standards, and adds a layer of resiliency.
The tradeoff is increased latency for better resiliency and flexibility, with a 99-percentile latency of 10s for analytics workloads. For critical and time-sensitive use cases requiring sub-second latency, applications can bypass the 2-step logging architecture and write to Kafka directly.
Splitting Kafka Clusters Based On Use Cases
Agoda made a strategic decision early on to split their Kafka clusters based on use cases instead of having a single large Kafka cluster per data center.
This means that instead of having one massive Kafka cluster serving all kinds of workloads, they have multiple smaller Kafka clusters, each dedicated to a specific use case or set of use cases.
The main reasons for this approach are:
By having separate clusters for different use cases, any issues that arise in one cluster will be contained within that cluster and won’t affect the others.
Different use cases may have different requirements in terms of performance, reliability, and data retention.
For example, a cluster used for real-time data processing might be configured with lower data retention periods and higher network throughput to handle the high volume of data.
In addition to splitting Kafka clusters by use case, Agoda also provisions dedicated physical nodes for Zookeeper, separate from the Kafka broker nodes. Zookeeper is a critical component in a Kafka cluster, responsible for managing the cluster's metadata, coordinating broker leader elections, and maintaining configuration information.
Stop renting auth. Make it yours instead.(Sponsored)
Developer-Centric: True API first design, quick integration, built on standards, highly flexible & customizable.
Hosting Flexibility: You host or we host - the choice is yours with no loss of features.
Unlimited: Unlimited IDPs, unlimited users, unlimited tenants, unlimited applications, always free.
Total Control: Deploy on any computer, anywhere in the world and integrate easily with any tech stack.
Scale Confidently: Lightning-fast performance for 10 users or 10 million users (or more).
Data Isolation: Single tenant by design means your data is physically isolated from everyone else’s.
FusionAuth is a complete auth & user platform that has 15M+ downloads and is trusted by industry leaders.
Monitoring and Auditing Kafka
From a monitoring point of view, Agoda uses JMXTrans to collect Kafka broker metrics.
JMXTrans is a tool that connects to JMX (Java Management Extensions) endpoints and collects metrics. These metrics are then sent to Graphite, a time-series database that stores numeric time-series data.
The collected metrics include things like broker throughput, partition counts, consumer lag, and various other Kafka-specific performance indicators.
The metrics stored in Graphite are visualized using Grafana, a popular open-source platform for monitoring and observability. Grafana allows the creation of customizable dashboards that display real-time and historical data from Graphite.
For auditing, Agoda implemented a custom Kafka auditing system. The primary goal of this auditing system is to ensure data completeness, reliability, accuracy, and timeliness across the entire Kafka pipeline.
Here’s how it works:
Audit counts are generated at various points throughout the pipeline.
A separate thread runs in the background on Agoda’s client libraries as part of the 2-step logging architecture we discussed earlier. This thread asynchronously aggregates message counts across time buckets to generate audits.
The generated audit data is stored in a separate Kafka cluster dedicated to audit information. This ensures the audit data doesn’t interfere with the main data pipelines.
The audit information ultimately ends up in two places:
Whitefalcon: Agoda’s internal near real-time analytics platform
Hadoop: For longer-term storage and analysis.
Authentication and ACLs
Initially, Agoda’s Kafka clusters were used primarily for application telemetry data, and authentication wasn’t deemed necessary.
As Kafka usage grew exponentially, concerns arose about the inability to identify and manage users who might be abusing or negatively impacting Kafka cluster performance. Agoda completed and released its Kafka Authentication and Authorization system in 2021.
The Authentication and Authorization system consists of the following components:
Core Kafka Authentication: It likely uses SASL (Simple Authentication and Security Layer) mechanisms supported by Kafka.
ACLs: Access Control Lists for fine-grained permission management.
Credential Generation: A custom component for creating and managing user credentials.
Credential Assignment: A system to associate credentials with specific users or teams.
Self-Service Portal: An interface allowing teams to request Kafka credentials and ACLs without direct intervention from the Kafka team.
Kafka Load Balancing
Agoda, as an online travel booking platform, aims to offer its customers the most competitive and current prices for accommodations and services from a wide range of external suppliers, including hotels, restaurants, and transportation providers.
To achieve this, Agoda's supply system is designed to efficiently process and incorporate a vast number of real-time price updates received from these suppliers. A single supplier can provide 1.5 million price updates and offer details in just one minute. Any delays or failures in reflecting these updates can lead to incorrect pricing and booking failures.
Agoda uses Kafka to handle these incoming price updates. Kafka partitions help them achieve parallelism by distributing the workload across mple partitions and consumers.
See the diagram below:
The Partitioner and Assignor Strategy
Apache Kafka's message distribution and consumption are heavily influenced by two key strategies: the partitioner and the assignor.
The partitioner strategy determines how incoming messages are allocated across partitions during production. Common approaches include round-robin distribution and sticky partitioning.
On the consumer side, the assignor strategy dictates how partitions are distributed among consumers within a consumer group. Examples include range assignments and round-robin assignments.
See the diagram below for reference:
Traditionally, these strategies were designed with the assumption that all consumers have similar processing capabilities and that all messages require roughly the same amount of processing time.
However, Agoda's real-world scenario deviated from these assumptions, leading to significant load-balancing challenges in their Kafka implementation.
There were two primary challenges:
Hardware Heterogeneity: Agoda's use of a private cloud infrastructure with Kubernetes resulted in pods being deployed across servers with varying hardware specifications. Benchmark tests revealed substantial performance disparities between different hardware generations.
Inconsistent Message Workloads: The processing requirements for different messages varied considerably. Some messages necessitated additional steps such as third-party API calls or database queries, leading to unpredictable processing times and latency fluctuations.
These challenges ultimately resulted in an over-provisioning problem, where resources were inefficiently allocated to compensate for the load imbalances caused by hardware differences and varying message processing demands.
Overprovisioning Problem at Agoda
The over-provisioning involves allocating more resources than necessary to handle the expected peak workload efficiently.
To illustrate this, let's consider a scenario where Agoda's processor service employs Kafka consumers running on heterogeneous hardware:
They have two high-performance workers, each capable of processing 20 messages per second.
Additionally, they have one slower worker that can only handle 10 messages per second.
Theoretically, this setup should be able to process a total of 50 messages per second (20 + 20 + 10). However, when using a round-robin distribution strategy, each worker receives an equal share of the messages, regardless of their processing capabilities. If the incoming message rate consistently reaches 50 messages per second, the following issues arise:
The two faster workers can comfortably handle their allocated share of approximately 16.7 messages per second each.
The slower worker, on the other hand, struggles to keep up with its assigned 16.7 messages per second, resulting in a growing lag over time.
See the diagram below
To maintain acceptable latency and meet processing SLAs, Agoda would need to allocate additional resources to this setup.
In this example, they would have to scale out to five machines to effectively process 50 messages per second. This means that they are overprovisioning by two extra machines due to the inefficient distribution logic that fails to consider the varying processing capabilities of the hardware.
A similar scenario can occur when the processing workload for each message varies, even if the hardware is homogeneous.
In both cases, this leads to several negative consequences:
Higher hardware costs due to the need for additional resources.
Inefficient utilization of resources, with some consumers being underutilized while others are overburdened.
Increased maintenance overhead to manage the overprovisioned infrastructure.
The round-robin distribution strategy, while ensuring an equal distribution of messages across consumers, fails to account for the heterogeneity in hardware performance and message processing workload.
Agoda’s Dynamic Lag-Aware Solution
To solve this, Agoda adopted a dynamic, lag-aware approach to solving the Kafka load balancing challenges. They didn’t opt for static balancing solutions like weighted load balancing due to messages having non-uniform workloads.
They implemented two main strategies:
Lag-aware Producer
Lag-aware Consumer
Lag-Aware Producer
A lag-aware producer is a dynamic approach to load balancing in Apache Kafka that adjusts message partitioning based on the current lag information of the target topic.
It works as follows:
The producer maintains a cached copy of partition lag data to minimize the frequency of requests to Kafka brokers for this information.
The producer uses the lag data to intelligently distribute messages across partitions using a custom algorithm. The algorithm is designed to send fewer messages to partitions with high lag and more messages to partitions with low lag. They use algorithms like the same-queue length algorithm and outlier detection algorithm.
When the lags across partitions are balanced and stable, the lag-aware producer ensures an even distribution of messages.
Let's consider an example scenario in Agoda's supply system, where an internal producer publishes task messages to a processor.
The target topic has 6 partitions with the following lag distribution:
Partition 1: 110 messages
Partition 2: 150 messages
Partition 3: 80 messages
Partition 4: 400 messages
Partition 5: 120 messages
Partition 6: 380 messages
In this situation, the lag-aware producer would identify that partitions 4 and 6 have significantly higher lag compared to the other partitions. As a result, it would adapt its partitioning strategy to send fewer messages to partitions 4 and 6 while directing more messages to the partitions with lower lag (partitions 1, 2, 3, and 5).
By dynamically adjusting the message distribution based on the current lag state, the lag-aware producer helps to rebalance the workload across partitions, preventing further lag accumulation on the already overloaded partitions.
Lag-Aware Consumer
Lag-aware consumers are a solution employed when multiple consumer groups are subscribed to the same Kafka topic, making lag-aware producers less effective.
The process works as follows:
In a downstream service, such as Agoda's Processor, if a particular consumer instance detects that it has fallen significantly behind in processing messages (i.e., it has a high lag), it can voluntarily unsubscribe from the topic. This action triggers a rebalance operation.
During the rebalance, a custom partition Assigner, developed by Agoda, reassigns the partitions across all the remaining consumer instances. The redistribution takes into account each consumer's current lag and processing capacity, ensuring a more balanced workload.
To minimize the performance impact of rebalancing, Agoda leverages Kafka 2.4's incremental cooperative rebalance protocol. This protocol allows for more frequent partition reassignments without causing significant disruptions to the overall processing flow.
Let's illustrate this with an example.
Suppose Agoda's Processor service has three consumer instances (workers) that are consuming messages from six partitions of a topic:
Worker 1 is responsible for processing messages from Partitions 1 and 2
Worker 2 handles Partitions 3 and 4
Worker 3 processes messages from Partitions 5 and 6
If Worker 3 happens to be running on older, slower hardware compared to the other workers, it may struggle to keep up with the message influx in Partitions 5 and 6, resulting in higher lag. In this situation, Worker 3 can proactively unsubscribe from the topic, triggering a rebalance event.
During the rebalance, the custom Assigner evaluates the current lag and processing capacity of each worker and redistributes the partitions accordingly. For example, it may assign Partition 5 to Worker 1 and Partition 6 to Worker 2, effectively relieving Worker 3 of its workload until the lag is reduced to an acceptable level.
Conclusion
In conclusion, Agoda's journey with Apache Kafka has been one of continuous growth, learning, and adaptation.
By implementing strategies such as the 2-step logging architecture, splitting Kafka clusters based on use cases, developing robust monitoring and auditing systems, and Kafka load balancing Agoda has successfully managed the challenges that come with handling 1.8 trillion events per day.
As Agoda continues to evolve and grow, its Kafka setup will undoubtedly play a crucial role in supporting the company's ever-expanding needs. The various solutions also provide great learning for other software developers in the wider community when it comes to adapting Kafka to their organizational needs.
References:
SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:36 - 29 Oct 2024 -
The McKinsey Quarterly is turning 60, and you’re invited to the party
Since 1964, the McKinsey Quarterly has brought our firm’s most distinctive, cross-cutting insights to global executives. We’re now delighted to kick off a yearlong celebration commemorating the Quarterly’s 60th birthday. Sign up for a free digital Quarterly membership today and unlock access to "Q60," a yearlong series. Here's what you'll get:
•
Q60 special issues: Dive deep into themes shaping the future of business, with each issue dedicated to a critical topic.
•
Bonus digital offerings: Engage with special interactive features and explore compilations of timeless Quarterly classics, many of which were available only in print until now.
•
Immersive digital reading experience: Enjoy each Quarterly issue online.
The first issue of the celebration, “The future of technology,” dives deep into the technologies reshaping the business world. In its cover package, leading experts offer insights on how companies can strategically implement technology. For example, “The gen AI skills revolution: Rethinking your talent strategy,” by Alharith Hussin, Anna Wiesinger, Charlotte Relyea, Martin Harrysson, and Suman Thareja, with Prakhar Dixit and Thao Dürschlag, highlights how companies can manage a historic shift in the kind of talent they need. And Aamer Baig, Jeffrey Lewis, Klemens Hjartar, Rob Cain, and Sven Blumberg’s “A new dawn for the technology officer” explains how leaders can ensure that the entire organization is united behind the push for digital success.
And don’t miss the newly released interactive timeline “Sixty years of innovation: Key moments in business technology” and its exploration of the 25 tech breakthroughs that changed business forever.
A membership to the digital edition of the McKinsey Quarterly also includes downloads of more than 100 reports in The McKinsey Insights Store as well as access to the previous digital issues of the Quarterly.
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Quarterly alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey & Company" <publishing@email.mckinsey.com> - 07:47 - 29 Oct 2024 -
1 month free
1 month free
Hi there,
If you think Your Telecoms Consultant could benefit from being listed on 45 directories, I have a deal for you!
Sign up within 7 days, and I will give you your first month free, taking the price down from £425 to just £390 for a whole year!
If you’re interested in getting started, let me know and we can arrange a time to call and get you set up.
Thank you,
Darren Dale
Digital Marketing Consultant
0800 089 0879
infoserve.com
Darren Dale <darren@infoserve.uk> wrote:
Hi there,
If you want Your Telecoms Consultant to be found by local customers, business listings are a must.
Business listings are a great way of putting your business in front of people who are searching for a service in your area.
It can be costly and time-consuming to identify, set up and manage high-quality online listings though, not to mention all the online reviews to manage.
Our unique Listing Manager allows you to showcase your business across 45 leading directories, search engines, maps, social media and sat-navs - think a digital version of the Yellow Pages.
So whether your customers are searching on Google, Bing, Apple Maps, Uber, Sat Navs and so on - if your business is listed through our Listing Manager you will come up as a local provider.
Three key benefits of Listing Manager:
All managed in one dashboard - Update your business’ details once, and it will instantly sync to over 45 listings.
Review management - Reply to any review from any listing, from your online dashboard.
Improves Google ranking - Listing Manager boosts your local search engine optimisation, getting you found quicker.
All for just £425 for a full year!
If you'd like to find out more, drop me an email back or give me a call on 0800 089 0879.
Best regards,
Darren Dale
Digital Marketing Consultant
0800 089 0879
infoserve.com
P.S. Here's what Jonathan Anelli had to say recently:
“Been working with infoserve on and off for a number of years. I have found MASSIVELY USEFUL is LISTINGS MANAGER. My business isn't the sort that people search for through listings. However, the one-stop-shop approach for dozens of high-quality links has proved instrumental in my site developing a high authority score! I'm ranking better than competitors simply because I have higher authority... (please don't tell them about this)”
Infoserve Ltd, Southside Aviation, Leeds Airport, LS19 7UG. Registered in England No. 04209122
We have sent this email to info@learn.odoo.com, having found your company details online, as we believe our digital marketing services may be of legitimate interest to your business. If you would no longer like to get emails from us you can stop them here.
by "Darren Dale" <darren@infoserve.uk> - 04:07 - 29 Oct 2024
