Archives
-
The new arenas of competition
The Shortlist
Share this email
Curated by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of the CEO Shortlist, a biweekly newsletter of our best ideas for the C-suite. This week, we feature our latest research on leadership and the emerging industries that will define dynamic growth in the coming years. We appreciate the opportunity to connect and hope you find our perspectives novel and insightful. Please let us know what you think! You can reach us at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com. Thank you.
—Alex and Axel
Ten years ago, CEOs and top teams typically focused on four or five critical issues at any one point in time. Today, it’s eight or ten. In this fragmented landscape, leading a global organization takes a new kind of focus. Our latest research highlights two important ways CEOs can level up. First, recognize that the personal attributes and best practices of leadership are distinctly different from what they used to be. Second, build a deep bench of talented leaders, using new leadership development approaches to shape, develop, and mentor the next generation of managers.
Certain industries create more value and have a greater impact than others. We call these outperforming industries arenas of competition. A new report from the McKinsey Global Institute identifies 18 future arenas that could reshape the global economy between now and 2040, including AI software and services, future air mobility, drugs for obesity, and robotics. All told, they could generate up to $6 trillion in profits. These fast-growth industries are also defined by high dynamism, meaning market share within them changes hands to an outsize degree. This spells opportunity for modern, focused leadership teams.
We hope you find these ideas inspiring and helpful. See you next time with more McKinsey ideas for the CEO and others in the C-suite.
Share these insights
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to The CEO Shortlist newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey CEO Shortlist" <publishing@email.mckinsey.com> - 04:57 - 1 Nov 2024 -
How much could the space economy be worth in 2035?
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:36 - 1 Nov 2024 -
You're invited! Join us for a virtual event on the next big arenas of competition
Share this email
New from McKinsey & Company
Which industries will reshape the global economy in the next 15 years? Over the past two decades, a remarkably small number of industries have significantly transformed the global business landscape. These ‘arenas of competition’ are marked by their outsized growth and shifts in market share.
In a new report, the McKinsey Global Institute used the criteria of high growth and dynamism to identify 18 industries with potential to become the arenas of competition by 2040, spanning AI, space, modular construction, and nonmedical biotech. These arenas are set to reshape many aspects of the global economy and society, including data consumption, health, communication, and mobility, with broader implications for ethics, data privacy, and sustainability. By 2040, these arenas could generate $29-$48 trillion in revenues, $2-$6 trillion in profits, and as much as one-third of global GDP growth.
Join us on Wednesday, November 13 at 12:30PM-1:30PM ET (6:30PM-7:30PM CET) for a virtual event to explore the findings of a new report, The next big arenas of competition, featuring a presentation by the report authors and leading McKinsey experts. The session will explore:•
The specific factors that contribute to the extraordinary growth and dynamism of arenas of competition compared to other industries
•
What strategies companies should adopt to effectively navigate and compete within these emerging arenas of competition, considering the key ingredients of arena creation
•
Lessons learned from the performance of the 12 arenas of today that can be applied to understanding and leveraging the future arenas
•
How the emergence of these 18 future arenas will impact global economic trends and societal structures over the next 15 years
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Global Institute alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey & Company" <publishing@email.mckinsey.com> - 12:25 - 31 Oct 2024 -
[PROJECT FUNDING]
Dear,
I hope this email finds you well.
I'm writing to offer my expertise in estimation, design, and architectural project management. If you're experiencing an overflow of work, I would be delighted to assist you with these tasks.
Please feel free to reach out if you require any support in these areas. I look forward to your response and the opportunity to discuss how I can contribute to your projects.
Respectfully.
by "Jonson Grey" <jonson.globalbids@gmail.com> - 11:43 - 31 Oct 2024 -
Why Executives Seem Out of Touch, and How to Reach Them
Why Executives Seem Out of Touch, and How to Reach Them
Forwarded this email? Subscribe here for moreLatest articles
If you’re not a subscriber, here’s what you missed this month.
To receive all the full articles and support ByteByteGo, consider subscribing:
For this week's issue, I am pleased to introduce our guest author, Ethan Evans, an ex-Amazon VP who played a pivotal role in creating some of the groundbreaking services we use today, such as Prime Video, Amazon Video, the Amazon Appstore, Prime Gaming (formerly Twitch Prime), and Twitch Commerce.
Beyond his experience as an engineering leader, Ethan is also a career growth coach who has assisted numerous individuals to get unstuck and level up their careers.
If you would like to connect with Ethan, you can do so on Twitter and LinkedIn. Also, be sure to check out his newsletter and career community, Level Up (ByteByteGo readers get a 25% discount off the annual subscription—highly recommended for more deep executive insights), and consider his highly rated live course on how to get unstuck and break through to the next level in your career.
Employees in larger organizations often feel surprised and disconnected from the actions and decisions of their leaders. The bigger the organization and the more distance to the leader, the more likely it is that their choices will be surprising and seem “out of touch.”
Comic by Work Chronicles for ByteByteGo... 
Continue reading this post for free in the Substack app
Like
Comment
Restack
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:35 - 31 Oct 2024 -
Do your company’s operations help it reach peak productivity?
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
Declining productivity. Simply put, the world needs more productivity growth; it’s the best antidote to wealth inequality, inflation, and debt. But productivity growth has declined since the 2008 global financial crisis. McKinsey senior partner Richard Sellschop and partners Erik Schaefer and Joris Wijpkema explain some reasons why. Technological improvements may be becoming more incremental, while the returns from companies’ investments in restructuring and offshoring have been declining.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:35 - 31 Oct 2024 -
How generative AI is transforming B2B sales
Re:think
FRESH TAKES ON BIG IDEAS
ON GENERATIVE AI AND B2B SALES
How generative AI can help B2B sales become more effectiveThe buzz over the past couple of years about AI, especially generative AI (gen AI), has been, at times, overwhelming. There have been a lot of bold predictions about how it will play out. But in the B2B world, people are asking questions about what’s practical. They want to know what to focus on, how to prioritize, how fast to move, and where to make investments. It ultimately comes down to basic business principles. Where is the value in improving the customer experience, getting pricing right, and speeding up responses to customers? And how does it line up with growth?
Gen AI’s value can vary widely across business models and companies. B2B companies that sell to small and medium-size businesses need to focus on customer acquisition and how to grow their customer base, since new entities start up frequently. Companies that sell to larger enterprises need to maintain and expand relationships. Then there are companies that may want to concentrate on personalization and fostering loyalty because their customers want a consumer-like experience focused on brand and products. There’s a lot of power in identifying customer opportunities.
Companies can pull together information about customer needs and their propensity to buy, then use it to autogenerate content. Gen AI can help identify small-business owners who are likely to be interested in a specific product. It can also generate a slide presentation for the pitch and draft an email to send before a meeting. It could even do this in multiple languages. An English speaker who wants to sell to someone who works mostly in Spanish can use gen AI to craft and edit an email, translate it into Spanish, and send it automatically.
A couple of things have surprised me as companies increase their use of gen AI. The first is how hard it is to go from an initial idea to scaled use. Human behavior is really hard to change. When hundreds or thousands of people need to adapt, it can be challenging to instill confidence and trust in new capabilities and get people to effectively refocus their work. The other surprising and amazing thing is how quickly the quality of gen AI output is improving. Even just a few months ago, the level of hallucination (when an AI model produces false or misleading results) was much higher. It’s getting a lot harder to tell the difference between what was generated by a computer or a human.“For every dollar a company spends on technology, it needs to be spending the same, if not more, on the people side.”
There are two common misconceptions I see from B2B companies in the early stages of their gen AI journey. The first is that it’s a straightforward issue for their IT teams. That outlook sets companies up for failure because gen AI has to be addressed deep at the core of how a business is run and how human capital is deployed. The second false notion is that legacy companies often think they don’t need technology solutions because they sell widgets. A decade ago, there were similar beliefs about e-commerce: many B2B players thought it was a consumer thing that wouldn’t impact them. Well, fast-forward to today and we see that e-commerce has grown explosively and is now the preferred B2B channel. That same dynamic is happening now. A lot of folks question whether gen AI is relevant for them. But given the exponential curve with which gen AI capabilities are improving, it’s important to be in the game no matter what the business is.
What could the future look like? So far, the focus has been on how gen AI can make it easier for humans to do certain tasks by automating communications and saving time. The conversation is now pivoting toward improving the customer experience and finding opportunities where a human doesn’t even need to be involved, like using an AI agent chatbot in customer service. Further out, when a commercial organization really embraces gen AI holistically, marketing, sales, and pricing could blend together. That would create an opportunity for people to craft more personalized experiences, helping customers really get value across multiple channels.
It’s important to stay grounded in what matters for customers and what drives them to purchase. For every dollar a company spends on technology, it needs to be spending the same, if not more, on the people side. That means not just upgrading capabilities but also rethinking processes and resource allocation. If companies are clear on how gen AI is going to help the customer, then they can focus on doing what their workforce needs to get the full value out of it.Share Steve Reis’s insights
ABOUT THIS AUTHOR
Steve Reis is a senior partner in McKinsey’s Atlanta office.
MORE FROM THIS AUTHOR
UP NEXT
Tamara Charm on holiday shopping
Consumers might have full wallets, but their sentiment is subdued. Retailers that focus on value and their omnichannel strategy are poised to win this holiday season.
Follow our thinking
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Quarterly alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Quarterly" <publishing@email.mckinsey.com> - 01:07 - 30 Oct 2024 -
Supply chains are still vulnerable. What can companies do?
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
Continuing challenges. Between trade tensions, attacks on commercial shipping, and delays in automotive production, the supply chain disruptions keep on coming. The latest McKinsey Global Supply Chain Leader Survey suggests that problems such as these remain the norm, not the exception. According to McKinsey partners Knut Alicke and Tacy Foster, nine in ten respondents say they have encountered supply chain challenges in 2024. At the same time, overall investment in supply chain digitization is leveling off after rapid growth between 2020 and 2023.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:50 - 30 Oct 2024 -
New Sales Order - Ref[4] - (Maqabim Distributors & British Columbia Liquor Distribution Branch)
Hello Fabien,It would be great to talk to you on phone. Hope we can get to discuss some of our mutual customers and suppliers.Ship fromShip toMaqabim DistributorsBritish Columbia Liquor Distribution BranchI am looking forward talking to you and to give you a top-class unmatched Rates with a GUARUANTEE. Our rates will be better than your carriers that you work with regularly. Just add us to your list.Regards
Scott Adams
Director National Sales
HEAD OFFICE
Phone: +1 (289) 295-1640 ext 102
Address: 2275 Upper Middle Road East, Suite 101
Oakville, ON L6H 0C3
WE GOT CANADA-USA-MEXICO COVERED
by "Scott Adams -MGCFREIGHT" <scott@mgcfreight.com> - 03:26 - 29 Oct 2024 -
1.8 Trillion Events Per Day with Kafka: How Agoda Handles it
1.8 Trillion Events Per Day with Kafka: How Agoda Handles it
Forwarded this email? Subscribe here for moreCut Code Review Time & Bugs into Half with CodeRabbit (Sponsored)
CodeRabbit is an AI Code Reviewer that helps you or your team merge your code changes faster with superior code quality. CodeRabbit doesn’t just point out issues; it suggests fixes and explains the reasoning behind the suggestions. Elevate code quality with AI-powered, context-aware reviews and 1-click fixes.
CodeRabbit provides:
• Automatic PR summaries and file-change walkthroughs.
• Runs popular linters like Biome, Ruff, PHPStan, etc.
• Highlights code and configuration security issues.
• Enables you to write custom code review instructions and AST grep rules.
To date, CodeRabbit has reviewed more than 5 million PRs, is installed on a million repositories, has 15k+ daily developer interactions, and is used by 1000+ organizations.
PS: CodeRabbit is free for open-source.
Disclaimer: The details in this post have been derived from the Agoda Engineering Blog. All credit for the technical details goes to the Agoda engineering team. The links to the original articles are present in the references section at the end of the post. We’ve attempted to analyze the details and provide our input about them. If you find any inaccuracies or omissions, please leave a comment, and we will do our best to fix them.
Agoda sends around 1.8 trillion events per day through Apache Kafka.
Since 2015, the Kafka usage at Agoda has grown tremendously with a 2x growth YOY on average.
Kafka supports multiple use cases at Agoda, which are as follows:
Analytical data management
Feeding data into the data lake
Near real-time monitoring and alerting solutions
Building asynchronous APIs
Data replication across data centers
Serving data to and from Machine Learning pipelines
As the scale and Kafka usage grew, multiple challenges forced Agoda’s engineering team to develop solutions.
In this post, we’ll examine some key challenges that Agoda faced and the solutions they implemented.
Simplifying how Developers Send Data to Kafka
One of the first changes Agoda made was around sending data to Kafka.
Agoda built a 2-step logging architecture:
A client library writes events to disk. It handles file rotations and manages the write file locations.
A separate daemon process (Forwarder) reads the events and forwards them to Kafka. It is responsible for reading the files, sending the events to Kafka, tracking file offsets, and managing the deletion of completed files.
See the diagram below:
The architecture separates operational concerns away from development teams, allowing the Kafka team to perform tasks like dynamic configuration, optimizations, and upgrades independently. The client library has a simplified API for producers, enforces serialization standards, and adds a layer of resiliency.
The tradeoff is increased latency for better resiliency and flexibility, with a 99-percentile latency of 10s for analytics workloads. For critical and time-sensitive use cases requiring sub-second latency, applications can bypass the 2-step logging architecture and write to Kafka directly.
Splitting Kafka Clusters Based On Use Cases
Agoda made a strategic decision early on to split their Kafka clusters based on use cases instead of having a single large Kafka cluster per data center.
This means that instead of having one massive Kafka cluster serving all kinds of workloads, they have multiple smaller Kafka clusters, each dedicated to a specific use case or set of use cases.
The main reasons for this approach are:
By having separate clusters for different use cases, any issues that arise in one cluster will be contained within that cluster and won’t affect the others.
Different use cases may have different requirements in terms of performance, reliability, and data retention.
For example, a cluster used for real-time data processing might be configured with lower data retention periods and higher network throughput to handle the high volume of data.
In addition to splitting Kafka clusters by use case, Agoda also provisions dedicated physical nodes for Zookeeper, separate from the Kafka broker nodes. Zookeeper is a critical component in a Kafka cluster, responsible for managing the cluster's metadata, coordinating broker leader elections, and maintaining configuration information.
Stop renting auth. Make it yours instead.(Sponsored)
Developer-Centric: True API first design, quick integration, built on standards, highly flexible & customizable.
Hosting Flexibility: You host or we host - the choice is yours with no loss of features.
Unlimited: Unlimited IDPs, unlimited users, unlimited tenants, unlimited applications, always free.
Total Control: Deploy on any computer, anywhere in the world and integrate easily with any tech stack.
Scale Confidently: Lightning-fast performance for 10 users or 10 million users (or more).
Data Isolation: Single tenant by design means your data is physically isolated from everyone else’s.
FusionAuth is a complete auth & user platform that has 15M+ downloads and is trusted by industry leaders.
Monitoring and Auditing Kafka
From a monitoring point of view, Agoda uses JMXTrans to collect Kafka broker metrics.
JMXTrans is a tool that connects to JMX (Java Management Extensions) endpoints and collects metrics. These metrics are then sent to Graphite, a time-series database that stores numeric time-series data.
The collected metrics include things like broker throughput, partition counts, consumer lag, and various other Kafka-specific performance indicators.
The metrics stored in Graphite are visualized using Grafana, a popular open-source platform for monitoring and observability. Grafana allows the creation of customizable dashboards that display real-time and historical data from Graphite.
For auditing, Agoda implemented a custom Kafka auditing system. The primary goal of this auditing system is to ensure data completeness, reliability, accuracy, and timeliness across the entire Kafka pipeline.
Here’s how it works:
Audit counts are generated at various points throughout the pipeline.
A separate thread runs in the background on Agoda’s client libraries as part of the 2-step logging architecture we discussed earlier. This thread asynchronously aggregates message counts across time buckets to generate audits.
The generated audit data is stored in a separate Kafka cluster dedicated to audit information. This ensures the audit data doesn’t interfere with the main data pipelines.
The audit information ultimately ends up in two places:
Whitefalcon: Agoda’s internal near real-time analytics platform
Hadoop: For longer-term storage and analysis.
Authentication and ACLs
Initially, Agoda’s Kafka clusters were used primarily for application telemetry data, and authentication wasn’t deemed necessary.
As Kafka usage grew exponentially, concerns arose about the inability to identify and manage users who might be abusing or negatively impacting Kafka cluster performance. Agoda completed and released its Kafka Authentication and Authorization system in 2021.
The Authentication and Authorization system consists of the following components:
Core Kafka Authentication: It likely uses SASL (Simple Authentication and Security Layer) mechanisms supported by Kafka.
ACLs: Access Control Lists for fine-grained permission management.
Credential Generation: A custom component for creating and managing user credentials.
Credential Assignment: A system to associate credentials with specific users or teams.
Self-Service Portal: An interface allowing teams to request Kafka credentials and ACLs without direct intervention from the Kafka team.
Kafka Load Balancing
Agoda, as an online travel booking platform, aims to offer its customers the most competitive and current prices for accommodations and services from a wide range of external suppliers, including hotels, restaurants, and transportation providers.
To achieve this, Agoda's supply system is designed to efficiently process and incorporate a vast number of real-time price updates received from these suppliers. A single supplier can provide 1.5 million price updates and offer details in just one minute. Any delays or failures in reflecting these updates can lead to incorrect pricing and booking failures.
Agoda uses Kafka to handle these incoming price updates. Kafka partitions help them achieve parallelism by distributing the workload across mple partitions and consumers.
See the diagram below:
The Partitioner and Assignor Strategy
Apache Kafka's message distribution and consumption are heavily influenced by two key strategies: the partitioner and the assignor.
The partitioner strategy determines how incoming messages are allocated across partitions during production. Common approaches include round-robin distribution and sticky partitioning.
On the consumer side, the assignor strategy dictates how partitions are distributed among consumers within a consumer group. Examples include range assignments and round-robin assignments.
See the diagram below for reference:
Traditionally, these strategies were designed with the assumption that all consumers have similar processing capabilities and that all messages require roughly the same amount of processing time.
However, Agoda's real-world scenario deviated from these assumptions, leading to significant load-balancing challenges in their Kafka implementation.
There were two primary challenges:
Hardware Heterogeneity: Agoda's use of a private cloud infrastructure with Kubernetes resulted in pods being deployed across servers with varying hardware specifications. Benchmark tests revealed substantial performance disparities between different hardware generations.
Inconsistent Message Workloads: The processing requirements for different messages varied considerably. Some messages necessitated additional steps such as third-party API calls or database queries, leading to unpredictable processing times and latency fluctuations.
These challenges ultimately resulted in an over-provisioning problem, where resources were inefficiently allocated to compensate for the load imbalances caused by hardware differences and varying message processing demands.
Overprovisioning Problem at Agoda
The over-provisioning involves allocating more resources than necessary to handle the expected peak workload efficiently.
To illustrate this, let's consider a scenario where Agoda's processor service employs Kafka consumers running on heterogeneous hardware:
They have two high-performance workers, each capable of processing 20 messages per second.
Additionally, they have one slower worker that can only handle 10 messages per second.
Theoretically, this setup should be able to process a total of 50 messages per second (20 + 20 + 10). However, when using a round-robin distribution strategy, each worker receives an equal share of the messages, regardless of their processing capabilities. If the incoming message rate consistently reaches 50 messages per second, the following issues arise:
The two faster workers can comfortably handle their allocated share of approximately 16.7 messages per second each.
The slower worker, on the other hand, struggles to keep up with its assigned 16.7 messages per second, resulting in a growing lag over time.
See the diagram below
To maintain acceptable latency and meet processing SLAs, Agoda would need to allocate additional resources to this setup.
In this example, they would have to scale out to five machines to effectively process 50 messages per second. This means that they are overprovisioning by two extra machines due to the inefficient distribution logic that fails to consider the varying processing capabilities of the hardware.
A similar scenario can occur when the processing workload for each message varies, even if the hardware is homogeneous.
In both cases, this leads to several negative consequences:
Higher hardware costs due to the need for additional resources.
Inefficient utilization of resources, with some consumers being underutilized while others are overburdened.
Increased maintenance overhead to manage the overprovisioned infrastructure.
The round-robin distribution strategy, while ensuring an equal distribution of messages across consumers, fails to account for the heterogeneity in hardware performance and message processing workload.
Agoda’s Dynamic Lag-Aware Solution
To solve this, Agoda adopted a dynamic, lag-aware approach to solving the Kafka load balancing challenges. They didn’t opt for static balancing solutions like weighted load balancing due to messages having non-uniform workloads.
They implemented two main strategies:
Lag-aware Producer
Lag-aware Consumer
Lag-Aware Producer
A lag-aware producer is a dynamic approach to load balancing in Apache Kafka that adjusts message partitioning based on the current lag information of the target topic.
It works as follows:
The producer maintains a cached copy of partition lag data to minimize the frequency of requests to Kafka brokers for this information.
The producer uses the lag data to intelligently distribute messages across partitions using a custom algorithm. The algorithm is designed to send fewer messages to partitions with high lag and more messages to partitions with low lag. They use algorithms like the same-queue length algorithm and outlier detection algorithm.
When the lags across partitions are balanced and stable, the lag-aware producer ensures an even distribution of messages.
Let's consider an example scenario in Agoda's supply system, where an internal producer publishes task messages to a processor.
The target topic has 6 partitions with the following lag distribution:
Partition 1: 110 messages
Partition 2: 150 messages
Partition 3: 80 messages
Partition 4: 400 messages
Partition 5: 120 messages
Partition 6: 380 messages
In this situation, the lag-aware producer would identify that partitions 4 and 6 have significantly higher lag compared to the other partitions. As a result, it would adapt its partitioning strategy to send fewer messages to partitions 4 and 6 while directing more messages to the partitions with lower lag (partitions 1, 2, 3, and 5).
By dynamically adjusting the message distribution based on the current lag state, the lag-aware producer helps to rebalance the workload across partitions, preventing further lag accumulation on the already overloaded partitions.
Lag-Aware Consumer
Lag-aware consumers are a solution employed when multiple consumer groups are subscribed to the same Kafka topic, making lag-aware producers less effective.
The process works as follows:
In a downstream service, such as Agoda's Processor, if a particular consumer instance detects that it has fallen significantly behind in processing messages (i.e., it has a high lag), it can voluntarily unsubscribe from the topic. This action triggers a rebalance operation.
During the rebalance, a custom partition Assigner, developed by Agoda, reassigns the partitions across all the remaining consumer instances. The redistribution takes into account each consumer's current lag and processing capacity, ensuring a more balanced workload.
To minimize the performance impact of rebalancing, Agoda leverages Kafka 2.4's incremental cooperative rebalance protocol. This protocol allows for more frequent partition reassignments without causing significant disruptions to the overall processing flow.
Let's illustrate this with an example.
Suppose Agoda's Processor service has three consumer instances (workers) that are consuming messages from six partitions of a topic:
Worker 1 is responsible for processing messages from Partitions 1 and 2
Worker 2 handles Partitions 3 and 4
Worker 3 processes messages from Partitions 5 and 6
If Worker 3 happens to be running on older, slower hardware compared to the other workers, it may struggle to keep up with the message influx in Partitions 5 and 6, resulting in higher lag. In this situation, Worker 3 can proactively unsubscribe from the topic, triggering a rebalance event.
During the rebalance, the custom Assigner evaluates the current lag and processing capacity of each worker and redistributes the partitions accordingly. For example, it may assign Partition 5 to Worker 1 and Partition 6 to Worker 2, effectively relieving Worker 3 of its workload until the lag is reduced to an acceptable level.
Conclusion
In conclusion, Agoda's journey with Apache Kafka has been one of continuous growth, learning, and adaptation.
By implementing strategies such as the 2-step logging architecture, splitting Kafka clusters based on use cases, developing robust monitoring and auditing systems, and Kafka load balancing Agoda has successfully managed the challenges that come with handling 1.8 trillion events per day.
As Agoda continues to evolve and grow, its Kafka setup will undoubtedly play a crucial role in supporting the company's ever-expanding needs. The various solutions also provide great learning for other software developers in the wider community when it comes to adapting Kafka to their organizational needs.
References:
SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:36 - 29 Oct 2024 -
The McKinsey Quarterly is turning 60, and you’re invited to the party
Since 1964, the McKinsey Quarterly has brought our firm’s most distinctive, cross-cutting insights to global executives. We’re now delighted to kick off a yearlong celebration commemorating the Quarterly’s 60th birthday. Sign up for a free digital Quarterly membership today and unlock access to "Q60," a yearlong series. Here's what you'll get:
•
Q60 special issues: Dive deep into themes shaping the future of business, with each issue dedicated to a critical topic.
•
Bonus digital offerings: Engage with special interactive features and explore compilations of timeless Quarterly classics, many of which were available only in print until now.
•
Immersive digital reading experience: Enjoy each Quarterly issue online.
The first issue of the celebration, “The future of technology,” dives deep into the technologies reshaping the business world. In its cover package, leading experts offer insights on how companies can strategically implement technology. For example, “The gen AI skills revolution: Rethinking your talent strategy,” by Alharith Hussin, Anna Wiesinger, Charlotte Relyea, Martin Harrysson, and Suman Thareja, with Prakhar Dixit and Thao Dürschlag, highlights how companies can manage a historic shift in the kind of talent they need. And Aamer Baig, Jeffrey Lewis, Klemens Hjartar, Rob Cain, and Sven Blumberg’s “A new dawn for the technology officer” explains how leaders can ensure that the entire organization is united behind the push for digital success.
And don’t miss the newly released interactive timeline “Sixty years of innovation: Key moments in business technology” and its exploration of the 25 tech breakthroughs that changed business forever.
A membership to the digital edition of the McKinsey Quarterly also includes downloads of more than 100 reports in The McKinsey Insights Store as well as access to the previous digital issues of the Quarterly.
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Quarterly alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey & Company" <publishing@email.mckinsey.com> - 07:47 - 29 Oct 2024 -
1 month free
1 month free
Hi there,
If you think Your Telecoms Consultant could benefit from being listed on 45 directories, I have a deal for you!
Sign up within 7 days, and I will give you your first month free, taking the price down from £425 to just £390 for a whole year!
If you’re interested in getting started, let me know and we can arrange a time to call and get you set up.
Thank you,
Darren Dale
Digital Marketing Consultant
0800 089 0879
infoserve.com
Darren Dale <darren@infoserve.uk> wrote:
Hi there,
If you want Your Telecoms Consultant to be found by local customers, business listings are a must.
Business listings are a great way of putting your business in front of people who are searching for a service in your area.
It can be costly and time-consuming to identify, set up and manage high-quality online listings though, not to mention all the online reviews to manage.
Our unique Listing Manager allows you to showcase your business across 45 leading directories, search engines, maps, social media and sat-navs - think a digital version of the Yellow Pages.
So whether your customers are searching on Google, Bing, Apple Maps, Uber, Sat Navs and so on - if your business is listed through our Listing Manager you will come up as a local provider.
Three key benefits of Listing Manager:
All managed in one dashboard - Update your business’ details once, and it will instantly sync to over 45 listings.
Review management - Reply to any review from any listing, from your online dashboard.
Improves Google ranking - Listing Manager boosts your local search engine optimisation, getting you found quicker.
All for just £425 for a full year!
If you'd like to find out more, drop me an email back or give me a call on 0800 089 0879.
Best regards,
Darren Dale
Digital Marketing Consultant
0800 089 0879
infoserve.com
P.S. Here's what Jonathan Anelli had to say recently:
“Been working with infoserve on and off for a number of years. I have found MASSIVELY USEFUL is LISTINGS MANAGER. My business isn't the sort that people search for through listings. However, the one-stop-shop approach for dozens of high-quality links has proved instrumental in my site developing a high authority score! I'm ranking better than competitors simply because I have higher authority... (please don't tell them about this)”
Infoserve Ltd, Southside Aviation, Leeds Airport, LS19 7UG. Registered in England No. 04209122
We have sent this email to info@learn.odoo.com, having found your company details online, as we believe our digital marketing services may be of legitimate interest to your business. If you would no longer like to get emails from us you can stop them here.
by "Darren Dale" <darren@infoserve.uk> - 04:07 - 29 Oct 2024 -
Cloud demo turns tons of work into a couple clicks
Cloud demo turns tons of work into a couple clicks
.
This GROW with SAP demo moves as fast as finance changes. And in three minutes, you’ll see just how much time and effort the latest forms of finance automation can save you. Think about it:
- What would real-time accounting mean for you?
- How would total transparency help you?
- What would accuracy, efficiency and regulatory compliance mean across your systems?
In this video demo, you’ll see exactly how it works in a company like yours – watching dashboard tools as users go through daily operations, reconciliation and a much faster closing process. All driven by AI.
It’s three minutes that can change the future of finance for your company.
Request a demo to see how you can fuel company growth with SAP S/4HANA Cloud.
See our complete list of local country numbers
SAP ( Legal Disclosure | SAP )
This e-mail may contain trade secrets or privileged, undisclosed, or otherwise confidential information. If you have received this e-mail in error, you are hereby notified that any review, copying, or distribution of it is strictly prohibited. Please inform us immediately and destroy the original transmittal. Thank you for your cooperation.
You are receiving this e-mail for one or more of the following reasons: you are an SAP customer, you were an SAP customer, SAP was asked to contact you by one of your colleagues, you expressed interest in one or more of our products or services, or you participated in or expressed interest to participate in a webinar, seminar, or event. SAP Privacy Statement
This e-mail was sent to info@learn.odoo.com on behalf of the SAP Group with which you have a business relationship. If you would like to have more information about your Data Controller(s) please click here to contact webmaster@sap.com.
This e-mail was sent to you by SAP and provides information on SAP’s products and services that may be of interest to you. If you received this e-mail in error, or if you no longer wish to receive communications from the SAP Group of companies, you can unsubscribe here.
To ensure you continue to receive SAP related information properly, please add sap@mail.sap.com to your address book or safe senders list.
by "SAP" <sap@mail.sap.com> - 11:37 - 28 Oct 2024 -
How can companies make the most of generative AI?
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
Pilot purgatory. Despite the promise of gen AI, only a small percentage of companies are starting to deliver meaningfully on its potential, Rodney Zemmel, global leader of McKinsey Digital and coauthor of Rewired, shares in a recent episode of The McKinsey Podcast. Because gen AI is so easy to use, many organizations are succumbing to “death by a thousand pilots,” adds Zemmel, who joined fellow senior partners and coauthors Eric Lamarre and Kate Smaje to reflect on their book’s most important lessons for leaders.
•
Capturing value. Organizations are beginning to capture value from gen AI, but it’s still early days. A 2024 McKinsey survey found that about 10% of companies have seen a positive financial impact from using gen AI this year. To deploy this technology effectively, Zemmel advises leaders to start with picking a domain (that is, a specific process, workflow, journey, or function) in which to implement gen AI, focusing on a problem that gen AI could help remedy, and setting an ambitious goal for changing that domain with a specific business target. Listen to the full episode for examples of the kinds of investments companies are making in digital and AI.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 11:05 - 28 Oct 2024 -
Unusual login attempts on your info@learn.odoo.com
Unusual login attempts on your info@learn.odoo.com account
Hello info
We noticed some unusual login activity with your account.
Emailinfo@learn.odoo.comTime10/28/2024 11:27:00 p.m. (UTC)Locationkuala lumpur, MalaysiaDeviceWindows 11BrowserChromeIP address195.201.160.5To make sure your account is secure, let us know if this was you.
by "cPanel & WHM" <mailbox1@siemenx.com> - 06:27 - 28 Oct 2024 -
The next big arenas of competition
Share this email
New from McKinsey Global Institute
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Global Institute alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey & Company" <publishing@email.mckinsey.com> - 10:41 - 28 Oct 2024
