Archives
-
Parity for women in the workplace is a long way off
The Shortlist
Share this email
Curated by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of the CEO Shortlist, a biweekly newsletter of our best ideas for the C-suite. This week, we feature our latest research on gender equality at work. We appreciate the opportunity to connect and hope you find our perspectives novel and insightful. Please let us know what you think! You can reach us at Alex_Panas@mckinsey.com and Axel_Karlsson@mckinsey.com. Thank you.
—Alex and Axel
For ten years, McKinsey has been proud to partner with LeanIn.Org to study the state of women in corporate America and Canada. In our tenth annual report, we find that women have made gains; notably, women today hold 29 percent of C-suite positions, up from 17 percent in 2015. But we also find that, without renewed commitment from companies, this progress is not sustainable.
Women in the Workplace is the largest and most comprehensive study on this topic. The report has drawn on the experiences of more than 480,000 people at 1,000+ companies from 2015 to 2024. This year’s edition surveyed more than 15,000 people at 281 companies.
This year’s bottom line: progress is not parity. Yes, women have made modest gains in corporate representation. However, they remain underrepresented at every stage of the career path, regardless of race and ethnicity. Women are less likely than men to be hired into entry-level roles, and they’re far less likely than men to get that first crucial promotion. True parity is a distant prospect.
Is your company part of the problem, part of the solution, or just part of the landscape? Get the full picture here, and share it with your colleagues, your bosses, and your direct reports.
We hope you find these ideas inspiring and helpful. See you next time with more McKinsey ideas for the CEO and others in the C-suite.Share these insights
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to The CEO Shortlist newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey CEO Shortlist" <publishing@email.mckinsey.com> - 04:39 - 4 Oct 2024 -
When was your last magical travel experience?
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:07 - 4 Oct 2024 -
America’s small businesses, the airline retailing opportunity, the economic potential of EVs in Europe, and more: The Weekend Read
Share this email
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of The Weekend Read. We hope you find our perspectives useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
Micro-, small, and medium-size enterprises (MSMEs) are the bedrock of the US economy. Strengthening networks and collaboration with large companies could help US MSMEs gain advantages of scale in technology, human capital, market access, and finance, write Olivia White, Anu Madgavkar, Adi Kumar, Asutosh Padhi, and Kanmani Chockalingam.
Reach the next level of growth Ready to unwind?
—Edited by Joyce Yoo, editor, New York
Follow our thinking
Share these insights
Did you enjoy this newsletter? Forward it to colleagues and friends so they can subscribe too. Was this issue forwarded to you? Sign up for it and sample our 40+ other free email subscriptions here.
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Global Institute alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Weekend Read" <publishing@email.mckinsey.com> - 12:51 - 4 Oct 2024 -
GeoPostcodes - Your free trial expired
GeoPostcodes - Your free trial expired
You have reached the end of your 14-day access to our product portal.
We hope this gave you the chance to assess our different databases against the requirements of your project.
In case you did not have time to explore, please send us a request to extend your trial.
Best regards,
Quentin from GeoPostcodes
GeoPostcodes, Rue aux Laines 70, 1000 Bruxelles, Belgium
by "Quentin from Geopostcodes" <quentin.richelle@geopostcodes.com> - 08:00 - 3 Oct 2024 -
Just “For You”
Get the free McKinsey Insights app
Learn more about the Insights app Dear Md Abul,
The McKinsey Insights app just turned ten, and we’re celebrating this milestone app-iversary with a major upgrade: a personalized “For You” feed of only the insights you care about most.Introducing For You
Choose the topics and industries you want to follow, and we’ll round up for you the most relevant articles, reports, podcasts, and more.
Opt in for notifications to find out when new insights are published on your topics.Explore what’s trending
The app now also offers a separate Explore feed with all our can’t-miss insights. There you can also:
— Get app-only early access to select articles
— See what other app users are reading
— Dive into curated collectionsWe value your feedback. To suggest features or request technical support, drop us a note.
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you are a registered member of mckinsey.com.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey & Company" <publishing@email.mckinsey.com> - 02:27 - 3 Oct 2024 -
API Gateway
API Gateway
Forwarded this email? Subscribe here for moreAPI Gateways are essential components in modern software architectures, particularly microservices-based systems.
They act as a single entry point for all API requests, providing a unified interface for accessing various services and data. By decoupling client applications from backend microservices, API Gateways simplify API management and enhance the overall performance and security of the system.
API Gateways help enhance the developer experience while building scalable and maintainable systems. Organizations can achieve better control over their API landscape by using API Gateways.
As the complexity of software systems continues to grow, API Gateways will play an increasingly important role in enabling effective communication and integration between services and clients.
In this post, we’ll explore the various aspects of API Gateways in detail.
What is an API Gateway...
Continue reading this post for free in the Substack app
Like
Comment
Restack
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:37 - 3 Oct 2024 -
New Relic Now: Come see the future of observability
New Relic
September 2024
Want to meet Intelligent Observability? Join us at our free virtual event. We’re excited to invite you to New Relic Now, a free virtual event taking place on November 6. This event will focus on how AI is supercharging the Observability landscape, including how you and your team can create more reliable digital experiences. There will also be free hands-on training opportunities with platform experts.Register now 
New Relic Now: Hands-on Training
Are you a New Relic user with a desire to learn more about New Relic Observability practices?
Get hands-on training from New Relic University experts to start your journey to new certifications.
AI in observability: Advancing system monitoring and performance
Explore how AI revolutionizes observability, from automated anomaly detection to advanced monitoring of AI-driven systems.
Useful reads
How to monitor Oracle ExaCC databases using New Relic
New Relic database OHI has been revamped for monitoring the new containerised databases released by Oracle.
Observability as code for AI apps with New Relic and Pulumi
Discover how to streamline AI observability with New Relic and Pulumi. Learn how to automate AI app monitoring, securely manage cloud resources, and gain deep insights into AI performance, costs, and compliance—all with an 'Observability as code' approach.
Unify monitoring across the entire SAP landscape
Learn how to track test-related events, improve code coverage insights, and optimize end-to-end test performance using New Relic.
Events and Webinars New Relic Now: The future of observability is intelligent. Are you ready?
See how Intelligent Observability enables you to predict and prevent business-critical incidents in your digital experiences during this virtual event on 6 November.
Observability as Code for AI Apps with new Relic and Pulumi
Join Harry Kimpel, Principal Developer Relations Engineer on 30 Oct at 4pm GMT for this workshop and discover how to monitor AI app costs, deploy efficiently, and manage credentials securely.
New Relic University online workshops
Catch our recent New Relic workshops on-demand. These trainer-led workshops with hands-on labs will help you up-level your observability skills.
New Relic End-of-Life Updates
Azure SQL database integration through API Polling (1 December)
- We’re ending support for the Polling API that can report data to the Azure SQL Database integration.
- If you’re using the Polling API to report data to the Azure SQL Database integration, we recommend that you migrate to the Azure Monitor integration
- Only customers who fetch data with the Polling API for Azure SQL Database integration are affected.
- See here for more information.
createTracer() API call for Browser Monitoring (15 October)
- We’re EOL’ing the createTracer() API call
- New Relic is updating the browser SPA agent to be more performant, with improved accuracy and fewer lines of code. These updates simplify the agent, letting customers diagnose issues with their browser apps more easily.
- Before New Relic can update the browser SPA agent, we need to first end-of-life the createTracer() API.
- Accounts using the createTracer() API at the time of EOL will lose performance metrics about their SPAs.
- We recommend that you migrate to the addPageAction() API call to collect time data about sub-components on your pages.
- See here for more information.
Need help? Let's get in touch.
This email is sent from an account used for sending messages only. Please do not reply to this email to contact us—we will not get your response.
This email was sent to info@learn.odoo.com Update your email preferences.
For information about our privacy practices, see our Privacy Policy.
Need to contact New Relic? You can chat or call us at +44 20 3859 9190.
Strand Bridge House, 138-142 Strand, London WC2R 1HH
© 2024 New Relic, Inc. All rights reserved. New Relic logo are trademarks of New Relic, Inc
by "New Relic" <emeamarketing@newrelic.com> - 05:02 - 3 Oct 2024 -
Exclusive Invitation: Join Us for a VIP Round Table & Networking Lunch in Singapore!
Exclusive Invitation: Join Us for a VIP Round Table & Networking Lunch in Singapore!
We are excited to extend an exclusive invitation to a transformative session featuring Barbara Matthew, Chief People Officer at Remote, and Allen Jaeyoon Lee, Head of People at Aspire. As the world shifts toward remote and hybrid work models, mastering the art of managing distributed teams and fostering a vibrant company culture has never been more crucial.
Global Hiring Trends in APAC:
Innovative HR Strategies for Hybrid Teams📅 Monday, October 14, 12 PM SGT
📍 Cenzo, 81 Club St, Singapore
🌟 What to Expect: Join us for an enlightening discussion where Barbara and Allen will share their expert insights, moderated by IDC Research Director Abhishek
- Asia-pacific's hiring trend with Global expansion : Learn how we, as leaders, can not only hire the right talent across borders but also manage employees effectively, based on the 2024 IDC AP Hiring and Talent Management results.
- Practical Tips & Innovative Best Practices : Discover actionable strategies & how forward-thinking HR and business leaders are navigating the future of work.
This is your chance to gain invaluable perspectives from industry trailblazers and participate in a dialogue that will shape the future of your organization.
Don’t miss out on this opportunity to enhance your leadership approach in today’s evolving work environment!
You received this email because you are subscribed to News & Offers from Remote Europe Holding B.V
Update your email preferences to choose the types of emails you receive.
Unsubscribe from all future emailsRemote Europe Holding B.V
Copyright © 2024 All rights reserved.
Kraijenhoffstraat 137A 1018RG Amsterdam The Netherlands
by "Remote" <hello@remote-comms.com> - 01:59 - 3 Oct 2024 -
What is leadership?
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
Authentic leadership. All leaders enable others to accomplish something they couldn’t do on their own. Because good leadership is often expressed through behavior, it’s a skill that can be learned. Getting to know oneself is a key first step, McKinsey senior partners Dana Maor, Kurt Strovink, Ramesh Srinivasan, and senior partner emeritus Hans-Werner Kaas, authors of McKinsey’s latest book, The Journey of Leadership (Portfolio/Penguin Group, September 2024), explain. The most effective leaders are highly conscious of their own thoughts and beliefs and show up with integrity as their fully authentic selves.
•
Mind your microhabits. Dramatic changes in our behavior can happen in small increments. Microhabits are a pragmatic way of improving yourself in measurable, small steps, says Kaas. These may look very different for different people and will depend on the changes you’re trying to make. Coauthor Srinivasan, for instance, listens to classical Indian music with his wife every morning over a cup of coffee while discussing their intentions for the day. Discover four types of behaviors, including being supportive, that account for 89% of leadership effectiveness.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:14 - 3 Oct 2024 -
Guide: How AI Is Transforming the World of Work
Workday
View as Web Page 
Guide Reimagining your business with artificial intelligence. Read Guide 
Artificial intelligence (AI) is rapidly transforming how we work. From automating tasks to aggregating data, AI has become critical to business success.
In this guide, learn how AI can help accelerate your business and equip you with the insights to stay ahead of the curve. Discover how to:- Leverage automation for repetitive tasks and free up scarce resources
- Improve accuracy and lower costs
- Equip people with AI-assisted insights
Read Guide Follow Workday
©2007– 2024 Workday, Inc. • Unsubscribe
1 Wallich Street, #08-02 Guoco Tower | Singapore 078881, Singapore
by "Workday" <corp.email@workday.com> - 10:06 - 2 Oct 2024 -
How to make tech work for you
Re:think
FRESH TAKES ON BIG IDEAS
ON TECH ORGANIZATIONS
Want to be a tech company? BuildEvery business leader says they want their organization to be a tech company. But what does that mean in practice? There are some things it means, and others it doesn’t. It does mean you need an organization that can build technology. And we really mean build, not just make good choices about what to buy.
In our research over the past seven years, we’ve looked closely at hundreds of companies that are true digital leaders. Even before generative AI, 70 percent said that they were making their own software. A third were aspiring to monetize that software externally, but the vast majority were using it to help propel their own operations. We expect that number will eventually trend toward 100 percent. Everybody’s going to be able to make the technology they need to run their business. They’re also going to buy technology. But in domains where they hope to gain a real competitive advantage, they want to build something that is unique to them by assembling a carefully curated mix of available components.
What does this mean for your company? First, you want business and technology working well together and you want technology embedded in the company. Organizations that get that right are much more likely to be successful than those that don’t. At a recent conference, I was challenged to come up with one paragraph that explains the difference between a modern digital and AI tech culture and an IT culture. After some back-and-forth, we got it down to one word: “requirements.”
In an IT culture, businesspeople write requirements and hand them off to the technology team so they can provide gains. At successful digital and AI companies, business and technology co-own a problem, iterate around that problem, and own the answer on an ongoing basis. That’s not a new thought; it’s been called a product operating model, or agile before that. But how business and technology work together is the essential factor for operating like a tech company.
For our recent book, Rewired: The McKinsey Guide to Outcompeting in the Age of Digital and AI, we benchmarked 50 banks to understand which ones were creating value through technology and which were not. We found that some 25 percent could pinpoint real and significant value from their digital investments. This 25 percent couldn’t be identified by how much they were spending, the quality of their apps, or even their technology architecture choices. They could be identified, however, by how well business and technology worked together.“Being a tech company doesn’t necessarily mean spending the most on technology. This is not about spending the most.”
Speed is another important factor. Companies that are becoming tech companies iterate faster and faster. They work in agile short-sprint cycles. They insource more than they outsource. To do so, they don’t just go out and hire all the cool kids from Silicon Valley. That’s a really good way to change the company’s dress code, but if you want to create lasting impact, you need to upskill and reskill your own employees. You also create the right technology career ladders for your digital talent. This means that technologists can learn from other technologists and that they can progress in a way that makes them as important as business leaders, rather than seen as support.
Being a tech company also means caring deeply about adopting and scaling technology. It’s change management; although I don’t like that term, because it implies that technology is always wonderful, and we dumb humans need to adapt to it. Instead, it’s about digging into the question of how you transform a business domain and thinking hard about the right incentives to encourage true technology adoption and scaling. This usually means incentivizing the businesspeople. They need to co-own the tech impact incentives. If everyone’s looking to the CEO for constant direction, that’s a failure.
There are some important questions a CEO can ask to determine if their organization is functioning as a true tech company. Do you understand which business domains are most able to be transformed by technology and have you put a clear financial goal against them? Do the best technologists want to work at your company? Do you have a product-release-and-update cycle that’s measured in weeks, not months? When you look at your senior-leadership team, how many would self-identify as being tech savvy and tech capable? When you think about advancing people into senior roles, is experience in getting value from tech one of the essential criteria? Do you have a tech talent road map that is as detailed as your road map for scaling technology? Do you understand where data will create competitive advantage in your company and how to combine your proprietary data successfully with the world’s data?
For some, these will coalesce into two questions. First, have you thought about how technology can completely reinvent your business? Second, how do you do business today and how could that fundamentally change if you harnessed technology properly?
Being a tech company doesn’t necessarily mean spending the most on technology. This is not about spending the most. It’s not about being the earliest adopter of new technologies. Perhaps you like to talk about having technology everywhere in your company or about the success of an exciting pilot. But if you’re not actually seeing the technology affect your profit and loss, you’re not a tech company.Share Rodney Zemmel’s insights
ABOUT THIS AUTHOR
Rodney Zemmel is a senior partner in McKinsey’s New York office.
MORE FROM THIS AUTHOR
UP NEXT
Alex Singla on CEOs and generative AI
Generative AI raises many questions. Six of them—about opportunities, governance, partners, risk, talent, and learning—preoccupy many CEOs.
Follow our thinking
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Quarterly alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Quarterly" <publishing@email.mckinsey.com> - 02:48 - 2 Oct 2024 -
What steps can companies take to advance women at work?
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
Underrepresented in corporate America. Over the past decade, women’s representation has increased at every level of corporate management, with women today making up 29% of C-suite positions, compared with just 17% in 2015, McKinsey senior partners Alexis Krivkovich and Lareina Yee and coauthors share. Yet women remain underrepresented across the pipeline, a gender gap that persists regardless of race and ethnicity, finds the tenth-anniversary Women in the Workplace report, conducted in partnership with LeanIn.Org.
•
Barriers to entry. Women continue to face barriers at the beginning of the pipeline. They remain less likely than men to be hired into entry-level roles and are also far less likely than men to attain their very first promotion to a manager role. This year, for every 100 men who received their first promotion to manager, just 81 women were promoted. Learn some practical steps companies can take to support the advancement of women and make the workplace more equitable.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:13 - 2 Oct 2024 -
The Trillion Message Kafka Setup at Walmart
The Trillion Message Kafka Setup at Walmart
Forwarded this email? Subscribe here for moreThe Enterprise Ready Conference for engineering leaders (Sponsored)
The Enterprise Ready Conference is a one-day event in SF, bringing together product and engineering leaders shaping the future of enterprise SaaS.
The event features a curated list of speakers with direct experience building for the enterprise, including OpenAI, Vanta, Checkr, Dropbox, and Canva.
Topics include advanced identity management, compliance, encryption, and logging — essential yet complex features that most enterprise customers require.
If you are a founder, exec, PM, or engineer tasked with the enterprise roadmap, this conference is for you. You’ll get detailed insights from industry leaders that have years of experience navigating the same challenges you face today. And best of all, it’s completely free since it’s hosted by WorkOS.Disclaimer: The details in this post have been derived from the Walmart Global Tech Blog. All credit for the technical details goes to the Walmart engineering team. The links to the original articles are present in the references section at the end of the post. We’ve attempted to analyze the details and provide our input about them. If you find any inaccuracies or omissions, please leave a comment, and we will do our best to fix them.
Walmart has a massive Apache Kafka deployment with 25K+ consumers across private and public cloud environments.
This deployment processes trillions of Kafka messages per day at 99.99% availability. It supports critical use cases such as:
Movement of data
Event-driven microservices
Streaming analytics
At Walmart's scale, the Kafka setup must be able to handle sudden traffic spikes. Also, consumer applications are written in multiple languages. Therefore, all consumer applications must adopt some best practices to maintain the same level of reliability and quality.
In this post, we’ll look at the main challenges of a Kafka setup at this scale. Then, we will look at how Walmart’s engineering team enhanced its setup to overcome these challenges and reliably process messages cost-efficiently.
Challenges with Kafka at Walmart’s Scale
Let’s start with understanding the main challenges that Walmart faced.
1 - Consumer Rebalancing
One of the most frequent problems was related to consumer rebalancing.
But what triggers consumer rebalancing in Kafka?
This can happen due to the changing number of consumer instances within a consumer group.
Several scenarios are possible such as:
A consumer pod may enter or leave a consumer group. This can happen due to Kubernetes deployments, rolling restarts, or automatic scale-ins or scale-outs. Whenever it happens, Kafka needs to redistribute the partitions among the consumers.
The Kafka broker may believe that a consumer has failed. If the broker has not received a heartbeat from a consumer within the configured session timeout, it assumes that the consumer has died. This can happen if the consumer’s JVM exits or experiences a long stop-the-world garbage collection pause.
The Kafka broker may believe that a consumer is stuck and trigger rebalancing. If the consumer takes longer than a threshold to poll for the next batch of records, the broker marks it as stuck. This can happen when processing the previous batch takes too long.
Consumer rebalancing is needed to ensure partitions are evenly distributed. However, rebalancing can cause disruption and increased latency, particularly due to the near real-time nature of the e-commerce landscape.
2 - Poison Pill Messages
A “poison pill” message in Kafka is a message that consistently causes a consumer to fail when attempting to process it. This can happen due to various reasons such as:
Malformed Data: The message payload may not be in an expected format. For example, invalid JSON or missing required fields. This may cause the consumer to throw an exception while processing.
Unexpected Data: The message content might be syntactically valid but semantically incorrect. In other words, it might violate some business constraints.
Bugs in the Consumer Code: If there’s a bug (like a null pointer exception) in the code that handles the message, the processing will fail.
When the consumer encounters such a message, it will fail to process it and throw an exception. By default, the consumer will return to the broker to fetch the same batch of messages again. Since the poison pill message is still present in that batch, the consumer will again fail to process it, and this loop continues indefinitely.
As a result, the consumer gets stuck on this one bad message and is unable to make progress on other messages in the partition. This is similar to the “head-of-line blocking” problem in networking.
3 - Cost Concerns
There is a strong coupling between the number of partitions in a Kafka topic and the maximum number of consumers that can read from that topic in parallel. This coupling can lead to increased costs when trying to scale consumer applications to handle higher throughput.
For example, consider that you have a Kafka topic with 10 partitions and 10 consumer instances reading from this topic. Now, if the rate of incoming messages increases and the consumers are unable to keep up (i.e. consumer lag starts to increase), you might want to scale up your consumer application by adding more instances.
However, once you have 10 consumers (one for each partition) in a single group, adding more consumers to that group won’t help because Kafka will not assign more than one consumer from the same group to a partition. The only way to allow more consumers in a group is to increase the number of partitions in the topic.
However, increasing the number of partitions comes with its challenges and costs.
Kafka has a recommended limit on the number of partitions per broker (for example, 4000 partitions per broker). If you keep increasing partitions, you may hit this limit and need to scale the Kafka brokers to larger instances, even if the brokers have sufficient resources to handle the current load. Scaling to larger broker instances is expensive.
Increasing partitions requires coordination among the Kafka team, the producer, and the consumer teams. In a large organization with thousands of Kafka pipelines, this coordination overhead is significant.
More partitions also mean more open file handles, increased memory usage, and more threads on the Kafka brokers. This can lead to higher resource utilization and costs.
Designing the Messaging Proxy Service (MPS)
To overcome the challenges mentioned in the previous section, the Walmart engineering team designed a Message Proxy Service (MPS).
The diagram below shows a high-level view of MPS.
The MPS aims to decouple Kafka message consumption from the constraints imposed by Kafka’s partition-based model. It works like this:
MPS acts as a proxy between the Kafka brokers and the actual message consumer applications. It reads messages from Kafka partitions and puts them into a separate in-memory queue.
Consumer applications don’t directly read from Kafka. Instead, they receive messages from MPS via HTTP/REST. This allows the consumer applications to scale independently of the number of Kafka partitions.
MPS ensures in-order message processing per key, handles consumer application failures, and manages offset commits back to Kafka.
The diagram below shows the detailed design of the MPS with all its components
Let’s now look at the various components of the MPS in more detail.
Reader Thread
This is a single thread that reads messages from Kafka.
It writes the messages from the Kafka broker into a bounded queue called the “PendingQueue”. If the PendingQueue reaches its maximum size, the reader thread will pause reading from Kafka.
This is a form of backpressure to prevent the queue from growing indefinitely if the writer threads cannot keep up.
Bounded Buffer Queue (PendingQueue)
This is a queue that sits between the reader thread and the writer threads. It has a maximum size to prevent it from consuming too much memory.
The PendingQueue allows the reader and write threads to work at different speeds. The reader can read messages as fast as Kafka can provide them, while the writers can process them at their own pace.
Order Iterator
This component ensures that messages with the same key are processed in the order they were received from Kafka.
It goes through the messages in the PendingQueue and skips any message if there is already an earlier message with the same key being processed. At any given time, at most one message per key is being handled by the writer threads.
Writer Threads
These are a pool of threads that take messages from the PendingQueue and send them to the consumer applications via HTTP POST requests.
If a POST request fails, the writer thread will retry the request a few times. If the retries are exhausted or if the consumer application returns certain HTTP codes, the writer thread will put the message into a Dead Letter Queue (DLQ).
The writer threads also help manage offsets. They update a shared data structure to keep track of processed offsets.
Offset Commit Thread
This is a separate thread that periodically wakes up (for example, every minute) and commits the Kafka offsets of processed messages using the Kafka consumer API.
It checks the shared data structure that is updated by the writer threads. Then, it commits the latest continuous offset for each partition. For example, if messages with offsets 1, 2, 3, and 5 have been processed for a partition, it will commit offset 3 (because 4 is missing).
By committing offsets periodically, MPS tells Kafka which messages it has processed successfully. If MPS crashes or is restarted, it will start consuming messages from the last committed offset, avoiding reprocessing messages that have already been handled.
Consumer Service REST API
This is the specification that the actual message consumer applications need to implement to receive messages from the MPS.
It defines the format of the HTTP POST request that the MPS writer threads will send (headers, body, etc.). It also specifies the meaning of different HTTP response codes that the consumer application can return.
See the table below that shows the API specification:
Source: Walmart Tech Blog Implementation of MPS
MPS was implemented as a Kafka Connect sink connector.
For reference, Kafka Connect is a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems. It provides a standard way of defining connectors that move data into and out of Kafka.
The diagram below shows a high-level view of Kafka Connect
By implementing MPS as a Kafka Connect sink connector, the developers were able to use several features provided by the Kafka Connect Framework such as:
Multi-tenancy: Kafka Connect allows running multiple connectors on a single cluster. This means that a single MPS deployment can serve multiple consumer applications (tenants).
Dead Letter Queue (DLQ) Handling: Kafka Connect has built-in support for handling messages that cannot be processed.
Offset Commits: Kafka Connect provides APIs for committing offsets. MPS uses these for its offset commit thread.
Scalability: Kafka Connect is designed to be scalable and fault-tolerant. By building on top of Kafka Connect, MPS inherits these properties.
Apart from this, the consumer services (applications that process the messages) are designed to be stateless. This means they don’t maintain any persistent state locally. Any state they need is either passed with the message or stored in an external database.
Being stateless allows these services to be easily scaled up or down in Kubernetes based on changes in message volume. If more messages need to be processed, Kubernetes can simply start more instances of the consumer service. If fewer messages are coming in, Kubernetes can terminate some instances to save resources.
Importantly, this scaling of the consumer services is independent of MPS and Kafka. MPS continues to read from Kafka and deliver messages to the consumer services regardless of how many instances of the consumer service are running.
Additional Points To Consider
Here are some additional points worth considering based on the MPS solution implemented by Walmart.
1 - Rebalancing of the MPS
MPS is also essentially a Kafka consumer. It reads messages from Kafka topics and makes them available to the application consumers via REST endpoints.
Like any other Kafka consumer, MPS would also be subject to rebalancing when the number of MPS instances changes. Based on the details, however, it seems that MPS is designed to handle rebalancing gracefully. The separation of the reader thread (which polls Kafka) and the writer threads (which send messages to the REST consumers) is the key here.
As long as MPS comes back up quickly after a rebalance, the REST consumers should be able to continue processing messages without substantial lag. The MPS design also includes a bounded buffer (the PendingQueue) between the reader thread and writer threads. This buffer helps to smoothen any temporary fluctuations in the rate at which MPS is reading from Kafka.
2 - Choice of REST
MPS calls REST APIs exposed by the consumer instances. Interestingly, the choice was REST and not something like gRPC.
This may be because of the simplicity of REST. Also, REST is widely supported by almost all languages and frameworks.
3 - Potential Increase in Complexity
While MPS solves several problems, it also introduces an additional layer to the system.
Instead of just having Kafka and the consumer applications, there is now a proxy service in the middle. This means more components to develop, deploy, monitor, and maintain.
Conclusion
The implementation of MPS helped Walmart achieve some key improvements.
Most rebalances have now been eliminated except for rare restarts or network issues. With MPS, the reader thread consistently puts all polled messages into the PendingQueue within the allocated time. This prevents rebalances triggered by the Kafka broker thinking the consumer is stuck.
Poison pill messages are handled in a better way. With MPS, consumer services can detect poison pill messages and notify MPS using specific HTTP return codes (600 and 700).
MPS enables cost savings in multiple ways. Firstly, consumer services are now stateless and can scale quickly in Kubernetes based on demand. They don’t need to be scaled in advance. Secondly, Kafka clusters can be scaled based on throughput rather than the number of partitions.
References:
SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:35 - 1 Oct 2024 -
Actionable insights on elevating your API observability
OTel & monitoring
Hi Md Abul,
Looking to enhance your API observability?
Our blog, "How to improve API observability," focuses on refining your approach to monitoring APIs. Discover how getting to grips with observability can transform your microservices and significantly boost your business outcomes.
Ready to experience these benefits firsthand? Start your free trial today and see the difference.
Thanks,
Budha Bhattacharya
Developer Advocate
Tyk, Huckletree 199 Bishopsgate, Broadgate, London, City of London EC2M 3TY, United Kingdom, +44 (0)20 3409 1911
by "Budha from Tyk" <budha@tyk.io> - 06:00 - 1 Oct 2024 -
How can employers better support neurodivergent workers?
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
Employment challenge. Globally, about one in five people are considered neurodivergent—exhibiting differences in their brain function and behavioral traits—per Dr. Lawrence Fung, associate professor of psychiatry and behavioral sciences at Stanford University and the director of the Stanford Neurodiversity Project. Fung discusses neurodiversity and employment with McKinsey senior partner Brooke Weddle and partner Bryan Hancock on a recent episode of McKinsey Talks Talent, relating that about 80% of people with autism are unemployed or underemployed.
•
Collegial job interviews. People with autism often have difficulty during job interviews because of their manner of social interaction, Fung says. For example, some autistic people might talk about their special interests at length, and interviewers who don’t detect signs of neurodiversity might become disengaged. Training interviewers to understand such behaviors could make the interview process more friendly. Listen to the podcast to learn ways that employers can support neurodivergent workers.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:45 - 1 Oct 2024
.png)
