Archives
-
The Trillion Message Kafka Setup at Walmart
The Trillion Message Kafka Setup at Walmart
Forwarded this email? Subscribe here for moreThe Enterprise Ready Conference for engineering leaders (Sponsored)
The Enterprise Ready Conference is a one-day event in SF, bringing together product and engineering leaders shaping the future of enterprise SaaS.
The event features a curated list of speakers with direct experience building for the enterprise, including OpenAI, Vanta, Checkr, Dropbox, and Canva.
Topics include advanced identity management, compliance, encryption, and logging — essential yet complex features that most enterprise customers require.
If you are a founder, exec, PM, or engineer tasked with the enterprise roadmap, this conference is for you. You’ll get detailed insights from industry leaders that have years of experience navigating the same challenges you face today. And best of all, it’s completely free since it’s hosted by WorkOS.Disclaimer: The details in this post have been derived from the Walmart Global Tech Blog. All credit for the technical details goes to the Walmart engineering team. The links to the original articles are present in the references section at the end of the post. We’ve attempted to analyze the details and provide our input about them. If you find any inaccuracies or omissions, please leave a comment, and we will do our best to fix them.
Walmart has a massive Apache Kafka deployment with 25K+ consumers across private and public cloud environments.
This deployment processes trillions of Kafka messages per day at 99.99% availability. It supports critical use cases such as:
Movement of data
Event-driven microservices
Streaming analytics
At Walmart's scale, the Kafka setup must be able to handle sudden traffic spikes. Also, consumer applications are written in multiple languages. Therefore, all consumer applications must adopt some best practices to maintain the same level of reliability and quality.
In this post, we’ll look at the main challenges of a Kafka setup at this scale. Then, we will look at how Walmart’s engineering team enhanced its setup to overcome these challenges and reliably process messages cost-efficiently.
Challenges with Kafka at Walmart’s Scale
Let’s start with understanding the main challenges that Walmart faced.
1 - Consumer Rebalancing
One of the most frequent problems was related to consumer rebalancing.
But what triggers consumer rebalancing in Kafka?
This can happen due to the changing number of consumer instances within a consumer group.
Several scenarios are possible such as:
A consumer pod may enter or leave a consumer group. This can happen due to Kubernetes deployments, rolling restarts, or automatic scale-ins or scale-outs. Whenever it happens, Kafka needs to redistribute the partitions among the consumers.
The Kafka broker may believe that a consumer has failed. If the broker has not received a heartbeat from a consumer within the configured session timeout, it assumes that the consumer has died. This can happen if the consumer’s JVM exits or experiences a long stop-the-world garbage collection pause.
The Kafka broker may believe that a consumer is stuck and trigger rebalancing. If the consumer takes longer than a threshold to poll for the next batch of records, the broker marks it as stuck. This can happen when processing the previous batch takes too long.
Consumer rebalancing is needed to ensure partitions are evenly distributed. However, rebalancing can cause disruption and increased latency, particularly due to the near real-time nature of the e-commerce landscape.
2 - Poison Pill Messages
A “poison pill” message in Kafka is a message that consistently causes a consumer to fail when attempting to process it. This can happen due to various reasons such as:
Malformed Data: The message payload may not be in an expected format. For example, invalid JSON or missing required fields. This may cause the consumer to throw an exception while processing.
Unexpected Data: The message content might be syntactically valid but semantically incorrect. In other words, it might violate some business constraints.
Bugs in the Consumer Code: If there’s a bug (like a null pointer exception) in the code that handles the message, the processing will fail.
When the consumer encounters such a message, it will fail to process it and throw an exception. By default, the consumer will return to the broker to fetch the same batch of messages again. Since the poison pill message is still present in that batch, the consumer will again fail to process it, and this loop continues indefinitely.
As a result, the consumer gets stuck on this one bad message and is unable to make progress on other messages in the partition. This is similar to the “head-of-line blocking” problem in networking.
3 - Cost Concerns
There is a strong coupling between the number of partitions in a Kafka topic and the maximum number of consumers that can read from that topic in parallel. This coupling can lead to increased costs when trying to scale consumer applications to handle higher throughput.
For example, consider that you have a Kafka topic with 10 partitions and 10 consumer instances reading from this topic. Now, if the rate of incoming messages increases and the consumers are unable to keep up (i.e. consumer lag starts to increase), you might want to scale up your consumer application by adding more instances.
However, once you have 10 consumers (one for each partition) in a single group, adding more consumers to that group won’t help because Kafka will not assign more than one consumer from the same group to a partition. The only way to allow more consumers in a group is to increase the number of partitions in the topic.
However, increasing the number of partitions comes with its challenges and costs.
Kafka has a recommended limit on the number of partitions per broker (for example, 4000 partitions per broker). If you keep increasing partitions, you may hit this limit and need to scale the Kafka brokers to larger instances, even if the brokers have sufficient resources to handle the current load. Scaling to larger broker instances is expensive.
Increasing partitions requires coordination among the Kafka team, the producer, and the consumer teams. In a large organization with thousands of Kafka pipelines, this coordination overhead is significant.
More partitions also mean more open file handles, increased memory usage, and more threads on the Kafka brokers. This can lead to higher resource utilization and costs.
Designing the Messaging Proxy Service (MPS)
To overcome the challenges mentioned in the previous section, the Walmart engineering team designed a Message Proxy Service (MPS).
The diagram below shows a high-level view of MPS.
The MPS aims to decouple Kafka message consumption from the constraints imposed by Kafka’s partition-based model. It works like this:
MPS acts as a proxy between the Kafka brokers and the actual message consumer applications. It reads messages from Kafka partitions and puts them into a separate in-memory queue.
Consumer applications don’t directly read from Kafka. Instead, they receive messages from MPS via HTTP/REST. This allows the consumer applications to scale independently of the number of Kafka partitions.
MPS ensures in-order message processing per key, handles consumer application failures, and manages offset commits back to Kafka.
The diagram below shows the detailed design of the MPS with all its components
Let’s now look at the various components of the MPS in more detail.
Reader Thread
This is a single thread that reads messages from Kafka.
It writes the messages from the Kafka broker into a bounded queue called the “PendingQueue”. If the PendingQueue reaches its maximum size, the reader thread will pause reading from Kafka.
This is a form of backpressure to prevent the queue from growing indefinitely if the writer threads cannot keep up.
Bounded Buffer Queue (PendingQueue)
This is a queue that sits between the reader thread and the writer threads. It has a maximum size to prevent it from consuming too much memory.
The PendingQueue allows the reader and write threads to work at different speeds. The reader can read messages as fast as Kafka can provide them, while the writers can process them at their own pace.
Order Iterator
This component ensures that messages with the same key are processed in the order they were received from Kafka.
It goes through the messages in the PendingQueue and skips any message if there is already an earlier message with the same key being processed. At any given time, at most one message per key is being handled by the writer threads.
Writer Threads
These are a pool of threads that take messages from the PendingQueue and send them to the consumer applications via HTTP POST requests.
If a POST request fails, the writer thread will retry the request a few times. If the retries are exhausted or if the consumer application returns certain HTTP codes, the writer thread will put the message into a Dead Letter Queue (DLQ).
The writer threads also help manage offsets. They update a shared data structure to keep track of processed offsets.
Offset Commit Thread
This is a separate thread that periodically wakes up (for example, every minute) and commits the Kafka offsets of processed messages using the Kafka consumer API.
It checks the shared data structure that is updated by the writer threads. Then, it commits the latest continuous offset for each partition. For example, if messages with offsets 1, 2, 3, and 5 have been processed for a partition, it will commit offset 3 (because 4 is missing).
By committing offsets periodically, MPS tells Kafka which messages it has processed successfully. If MPS crashes or is restarted, it will start consuming messages from the last committed offset, avoiding reprocessing messages that have already been handled.
Consumer Service REST API
This is the specification that the actual message consumer applications need to implement to receive messages from the MPS.
It defines the format of the HTTP POST request that the MPS writer threads will send (headers, body, etc.). It also specifies the meaning of different HTTP response codes that the consumer application can return.
See the table below that shows the API specification:
Source: Walmart Tech Blog Implementation of MPS
MPS was implemented as a Kafka Connect sink connector.
For reference, Kafka Connect is a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems. It provides a standard way of defining connectors that move data into and out of Kafka.
The diagram below shows a high-level view of Kafka Connect
By implementing MPS as a Kafka Connect sink connector, the developers were able to use several features provided by the Kafka Connect Framework such as:
Multi-tenancy: Kafka Connect allows running multiple connectors on a single cluster. This means that a single MPS deployment can serve multiple consumer applications (tenants).
Dead Letter Queue (DLQ) Handling: Kafka Connect has built-in support for handling messages that cannot be processed.
Offset Commits: Kafka Connect provides APIs for committing offsets. MPS uses these for its offset commit thread.
Scalability: Kafka Connect is designed to be scalable and fault-tolerant. By building on top of Kafka Connect, MPS inherits these properties.
Apart from this, the consumer services (applications that process the messages) are designed to be stateless. This means they don’t maintain any persistent state locally. Any state they need is either passed with the message or stored in an external database.
Being stateless allows these services to be easily scaled up or down in Kubernetes based on changes in message volume. If more messages need to be processed, Kubernetes can simply start more instances of the consumer service. If fewer messages are coming in, Kubernetes can terminate some instances to save resources.
Importantly, this scaling of the consumer services is independent of MPS and Kafka. MPS continues to read from Kafka and deliver messages to the consumer services regardless of how many instances of the consumer service are running.
Additional Points To Consider
Here are some additional points worth considering based on the MPS solution implemented by Walmart.
1 - Rebalancing of the MPS
MPS is also essentially a Kafka consumer. It reads messages from Kafka topics and makes them available to the application consumers via REST endpoints.
Like any other Kafka consumer, MPS would also be subject to rebalancing when the number of MPS instances changes. Based on the details, however, it seems that MPS is designed to handle rebalancing gracefully. The separation of the reader thread (which polls Kafka) and the writer threads (which send messages to the REST consumers) is the key here.
As long as MPS comes back up quickly after a rebalance, the REST consumers should be able to continue processing messages without substantial lag. The MPS design also includes a bounded buffer (the PendingQueue) between the reader thread and writer threads. This buffer helps to smoothen any temporary fluctuations in the rate at which MPS is reading from Kafka.
2 - Choice of REST
MPS calls REST APIs exposed by the consumer instances. Interestingly, the choice was REST and not something like gRPC.
This may be because of the simplicity of REST. Also, REST is widely supported by almost all languages and frameworks.
3 - Potential Increase in Complexity
While MPS solves several problems, it also introduces an additional layer to the system.
Instead of just having Kafka and the consumer applications, there is now a proxy service in the middle. This means more components to develop, deploy, monitor, and maintain.
Conclusion
The implementation of MPS helped Walmart achieve some key improvements.
Most rebalances have now been eliminated except for rare restarts or network issues. With MPS, the reader thread consistently puts all polled messages into the PendingQueue within the allocated time. This prevents rebalances triggered by the Kafka broker thinking the consumer is stuck.
Poison pill messages are handled in a better way. With MPS, consumer services can detect poison pill messages and notify MPS using specific HTTP return codes (600 and 700).
MPS enables cost savings in multiple ways. Firstly, consumer services are now stateless and can scale quickly in Kubernetes based on demand. They don’t need to be scaled in advance. Secondly, Kafka clusters can be scaled based on throughput rather than the number of partitions.
References:
SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com
Like
Comment
Restack
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:35 - 1 Oct 2024 -
Actionable insights on elevating your API observability
OTel & monitoring
Hi Md Abul,
Looking to enhance your API observability?
Our blog, "How to improve API observability," focuses on refining your approach to monitoring APIs. Discover how getting to grips with observability can transform your microservices and significantly boost your business outcomes.
Ready to experience these benefits firsthand? Start your free trial today and see the difference.
Thanks,
Budha Bhattacharya
Developer Advocate
Tyk, Huckletree 199 Bishopsgate, Broadgate, London, City of London EC2M 3TY, United Kingdom, +44 (0)20 3409 1911
by "Budha from Tyk" <budha@tyk.io> - 06:00 - 1 Oct 2024 -
How can employers better support neurodivergent workers?
Only McKinsey Perspectives
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:45 - 1 Oct 2024 -
[You're Invited, Md Abul] | Create Stylish and Standout Documents Faster with GenAI
Adobe
Adobe Live Webinar
Create Stylish and Standout Documents Faster with GenAI 10 October 2024, Thursday
Option 1: 11am AEDT | 8am SGT
Option 2: 4pm AEDT | 1pm SGT | 10:30am IST![[You're Invited] | Create Stylish and Standout Documents Faster with GenAI](https://engage.adobe.com/rs/360-KCI-804/images/apac-fy24q4-wbr-transform-documents-mcz-banner-1200x540.jpg?version=0)
In today’s fast-paced work environment, efficiency and style are key. Discover how Adobe Acrobat AI Assistant can help you master PDF styling and boost productivity, making your document more polished and professional.
Join Callum Swain, APAC Solution Customer Success Manager for Document Cloud at Adobe, to learn how to create standout PDFs using the advanced features of Acrobat AI Assistant. Learn how to give your documents a professional look with the easy-to-use tools from Adobe Express.
Take the first step toward smarter, more stylish document management! Register today to secure your spot!
Two time options available. Register for the session that best suits your schedule and timezone:
Register for option 1: 11am AEDT | 8am SGT
Register for option 2: 4pm AEDT | 1pm SGT | 10:30am IST
Who should attend
This session is ideal for professionals who work extensively with PDFs and seek to enhance their document management skills. If you're a project manager, team leader, marketing professional, or anyone involved in document preparation and review, you'll find valuable insights on how to leverage Adobe Acrobat's advanced features. Discover how to streamline workflows, polish your documents, and improve collaboration—all while boosting your productivity.
Can't make it to the live event? Register anyway so you receive the session recording afterwards!
Angus Cheung
APAC B2B Product Marketing & GTM Manager, Document Cloud
Adobe
Callum Swain
APAC Solution Customer Success Manager, Document Cloud
AdobeAcrobat's got it. 
Adobe and the Adobe logo are either registered trademarks or trademarks of Adobe in the United States and/or other countries. This is not a comprehensive list of all Adobe trademarks. For a full list, refer to the Adobe List of Trademarks. All other trademarks are the property of their respective owners.
By clicking on some of the links in this email, you might be redirected to forms that will be pre-populated with your contact information.
This is a marketing email from Adobe Systems Software Ireland Limited, 4‑6 Riverwalk, Citywest Business Park, Dublin 24, Ireland.
Click here to unsubscribe or send an unsubscribe request to the postal address above. Please review the Adobe Privacy Policy:
Australia
New Zealand
Indonesia
Malaysia
Philippines
Vietnam
Singapore
India
Hong Kong
Thailand
To ensure email delivery, add demand@info.adobe.com to your address book, contacts, or safe sender list.
If you have a privacy-related complaint, send it to: privacy@adobe.com
View in browser
by "Adobe Document Cloud" <demand@info.adobe.com> - 10:07 - 30 Sep 2024 -
[ New Estimate Bid ]
Dear,I hope you're doing well. At AIM ESTIMATION, we specialize in providing precise cost estimation and take-off services to General Contractors, Sub-Contractors, and Home Builders. Our team brings extensive expertise across trades, including Civil, Electrical, Plumbing, HVAC, and more.
We help ensure your projects are accurately budgeted and well-prepared for success. I'd love to discuss how we can assist with your upcoming work.
Best regards,
[ AIM ESTIMATION 'Company' ]
[ On Time, On Target ]
by "David Millan" <david.aimestimator@gmail.com> - 10:02 - 30 Sep 2024 -
Join me to learn how to monitor your AI applications with New Relic
Hi MD,I’d like to personally invite you to my upcoming 60 minute online webinar, How to monitor your AI applications with New Relic, on Tuesday at 10am BST / 11am CEST.
I will share practical insights on how to overcome the unique challenges generative AI applications present for those engineers responsible for building and managing AI-powered technologies.
Attend and learn how to:- Accelerate AI initiatives: Proactively monitor and resolve issues to keep your AI projects on track with New Relic.
- Optimise LLM selection and management: Choose the right LLMs for your application and manage their performance and costs effectively.
- Ensure compliance: Build generative AI solutions that adhere to compliance standards and regulations.
- Optimise performance: Track the performance of your models with telemetry data and choose the right model.
I hope to see you then,Harry Kimpel
Principal Developer Relations Engineer
New Relic
This email was sent to info@learn.odoo.com as a result of subscribing or providing consent to receive marketing communications from New Relic. You can tailor your email preferences at any time here.Privacy Policy © 2008-24 New Relic, Inc. All rights reserved
by "Harry Kimpel, New Relic" <emeamarketing@newrelic.com> - 05:02 - 30 Sep 2024 -
Is your organization poised for hypergrowth? A leader’s guide
Leading Off
Share this email
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Leading Off. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
In business, hypergrowth typically means maintaining a rapid rate of growth over time. Start-ups often have the agility to do this, but they may become more stratified as they grow larger—often because their hiring practices may not have kept pace with the scaling needs of the business. For their part, established businesses may not have the flexibility to push growth to the levels they need. This week, we look at how both start-ups and incumbent companies can deploy talent to position their organizations for consistent and rapid expansion.
Start-ups generally pay close attention to hiring the best talent to ensure the new company’s success. But as the business expands, the recruiting process can become more challenging, say McKinsey’s Claudy Jules, Shahar Markovitch, and Charlotte Seiler in a new article. “Talent needs inevitably change, skill gaps become more apparent, and it becomes increasingly difficult for founders and leaders to stay as personally involved in the hiring process,” they say. One possible solution is to conduct a formal assessment of the organization’s plans to scale and compare the current talent mix against those plans to identify any gaps. It may also help to hire talent in “functional areas that make the most sense strategically,” suggest the authors. “For instance, bringing new talent into sales, legal, and compliance teams may help scaling technology start-ups identify and lock in new accounts more quickly.”
That’s the number of critical questions company founders should ask when they seek to bring new leaders into the ranks, according to McKinsey senior partners Alok Kshirsagar and Arne Gast and colleagues. One core question to consider as the company grows is: Who leads? “While leaders may wear multiple hats in the early stages, organizational growth will likely call for more structure and clearer roles,” observe the McKinsey experts. “Moreover, the small circle of early leaders must acknowledge that the expanding enterprise will demand leadership and people skills that may be outside of their current knowledge and experience.”
That’s McKinsey senior partner Vincent Bérubé and colleagues on creating an integrated talent approach to power growth. Designing a talent win room involves pulling together leaders from across different functions and HR to develop a holistic view of the candidate experience. This could include revising job descriptions, looking at candidates with nontraditional backgrounds and experiences, and using generative AI and other analytics tools to match candidates with skills pools. For example, a software and services company redesigned its jobs architecture and review process to offer career paths usually available only at technology companies—and experienced not only a swift increase in the number of employees hired but also a reduction in the time spent from application to hiring.
There’s no unique magic that makes start-up employees different from those in a traditional corporate setting, says venture capitalist Florian Heinemann in a discussion with McKinsey partner Philipp Hillenbrand on how incumbent companies can incorporate a start-up-like approach to hypergrowth. What’s most important, he says, is for large companies to offer employees the incentives and flexibility that entrepreneurial ventures tend to nurture. “What you need is a structure parallel to the current business, plus some kind of independent, small, multidisciplinary group that mimics the start-up setting,” he suggests. Small cross-functional teams are also essential to ensure effective marketing and sales: “Very few people can create a superior product/market fit,” says Heinemann. “But if you get these people together in a small, fully or partially cross-functional team, that really gets things going. That’s why I like this kind of team so much, because the intensity with which you can iterate and experiment really creates energy.”
Your organization may have invested in top-notch people—but there’s no guarantee that they’ll stay. One reason could be that many organizations pay lip service to culture and values but may not adhere to these principles in practice. As a leader, are you always late for meetings? Do you advocate for work–life balance but consistently expect your teams to work evenings and weekends? McKinsey senior partner Dana Maor and colleagues suggest ten timeless questions for leaders to consider before filling jobs. For example, you may want to determine the level of cultural cohesiveness to establish or whether you wish to hire generalists or experts.
Lead by planning for hypergrowth.
— Edited by Rama Ramaswami, senior editor, New York
Share these insights
Did you enjoy this newsletter? Forward it to colleagues and friends so they can subscribe too. Was this issue forwarded to you? Sign up for it and sample our 40+ other free email subscriptions here.
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Leading Off newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Leading Off" <publishing@email.mckinsey.com> - 04:36 - 30 Sep 2024 -
What new style of leadership is needed for today’s business landscape?
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
‘Inside out’ leadership. As organizations grapple with advancing technologies, climate change, and geopolitical complexity, CEOs can no longer rely on an imperial style of leadership, McKinsey senior partner Ramesh Srinivasan, senior partner emeritus Hans-Werner Kaas, and coauthors explain in their new book, The Journey of Leadership: How CEOs Learn to Lead from the Inside Out. To engage stakeholders effectively, leaders need to reflect on how they show up in the world, Srinivasan shares in a recent episode of The McKinsey Podcast.
•
Up close and personal. For leaders to inspire their teams and the institutions that they’re leading, they may need to examine their vulnerabilities, which could be scary. To ease this process, the book offers personal stories of 24 former and current CEOs who reveal their insecurities, their struggles, and how they’ve addressed them, Kaas explains. Listen to the podcast to learn how modern CEOs strike a balance between confidence and humility, and read The Journey of Leadership to get a step-by-step reinvention guide that can help leaders transform themselves.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:13 - 30 Sep 2024 -
The power of introspection
On Point
Share this email
ESSENTIALS FOR LEADERS AND THOSE THEY LEAD
This month marks a significant milestone for Author Talks. Since its inception in 2020, we have produced 200 Author Talks interviews, gleaning fresh insights from CEOs, former CEOs, and other authors of new books. To commemorate this milestone, we’re sharing our top reads across several categories, as well as some summer highlights.
Do you find yourself wishing you could rewrite a chapter of your life? Maybe you’d change how you showed up at a pivotal moment—whether you stood your ground or lost your voice. While the ebb and flow of life rarely allows for do-overs, it does offer something greater: the opportunity for self-reflection. Rather than rewriting history, you could change gears by leaning into your strengths, being open to change, and leading with authenticity. It’s never too late to meet the moment.
This commemorative edition of Readers & Leaders offers insights on self-reflection, including embracing possibility at any age, advocating for yourself and others, addressing gender parity and finding purpose at work, being mindful of societal contributions, and becoming an active learner. Finally, in McKinsey’s latest book, The Journey of Leadership, senior partners Dana Maor, Kurt Strovink, and Ramesh Srinivasan and senior partner emeritus Hans-Werner Kaas provide a playbook for human-centric leadership that highlights the value of introspection, humility, and authenticity.
Check out these interviews and look out for our next edition of Readers & Leaders—now a quarterly newsletter—in early 2025.
Want early access to these interviews? Download the McKinsey Insights app to read the latest Author Talks now.it bears repeating
in case you missed it
TURN BACK THE PAGE: OUR TOP AUTHOR TALKS
bookmark this
BUSINESS BESTSELLERS TOP
8
Catch up on the latest and greatest reads for the month. Explore business bestsellers, prepared exclusively for McKinsey by Circana.
business overall
BUSINESS hardcover
economics
decision making
organizational behavior
workplace culture
COMPUTERS AND AI
sustainability
If you'd like to propose a book or author for #McKAuthorTalks, please email us at Author_Talks@Mckinsey.com. Due to the high volume of requests, we will respond only to those being considered.
—Edited by Emily Adeyanju, editor, Carolinas
Share these insights
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Readers & Leaders newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Readers & Leaders" <publishing@email.mckinsey.com> - 03:24 - 29 Sep 2024 -
GLA Middle-East Logistics Forum 2024
Dear agent,
Greeting from Cara and GLA family. Trust you are doing great.
GLA Global Logistics Alliance is proud to host the GLA Middle-East Logistics Forum 2024 in the vibrant city of Riyadh, Saudi Arabia!This event offers an unparalleled opportunity to discover new business opportunities in the logistics world.
Date: 15th October 2024
Location: Riyadh, Saudi Arabia
𝗥𝗲𝗴𝗶𝘀𝘁𝗲𝗿 𝗻𝗼𝘄: https://glafamily.com/forum/v2/
Limited Seats Available!
Please feel free to contact me if you are interested to attendJ
Best regards,
Cara Chen
GLA Overseas department
Mobile:
(86) 190-7616-6926
Email:
Company:
GLA Co.,Ltd
Website:
Address:
No. 2109, 21st Floor, HongChang, Plaza, No. 2001, Road Shenzhen, China
The 11th GLA Conference - Bangkok Thailand on 22th November 2024 – click here for registration
Ø The 10th GLA Conference in Dubai UAE, online album
Ø The 9th GLA Conference in Hainan China, online album
【Notice Agreement No 7】
7. GLA president reserves the right to cancel or reject membership or application.
company shall cease to be a member of GLA if:
a) the Member does not adhere to GLA terms and conditions
b) the Member gives notice of resignation in writing to the GLA.
c) No good reputation in the market.
d) Have bad debt records in the GLA platform or in the market
by "Cara" <member226@glafamily.com> - 05:26 - 29 Sep 2024 -
The week in charts
The Week in Charts
Share this email
Share these insights
Did you enjoy this newsletter? Forward it to colleagues and friends so they can subscribe too. Was this issue forwarded to you? Sign up for it and sample our 40+ other free email subscriptions here.
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to The Week in Charts newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Week in Charts" <publishing@email.mckinsey.com> - 03:29 - 28 Sep 2024 -
EP131: How Uber Served 40 Million Reads with Integrated Redis Cache?
EP131: How Uber Served 40 Million Reads with Integrated Redis Cache?
Forwarded this email? Subscribe here for moreThis week’s system design refresher:
Top Kafka Use Cases You Should Know (Youtube video)
How Uber Served 40 Million Reads with Integrated Redis Cache?
What makes AWS Lambda so fast?
Why do we need to use a distributed lock?
SPONSOR US
The Enterprise Ready Conference for engineering leaders (Sponsored)
The Enterprise Ready Conference is a one-day event in SF, bringing together product and engineering leaders shaping the future of enterprise SaaS.
The event features a curated list of speakers with direct experience building for the enterprise, including OpenAI, Vanta, Checkr, Dropbox, and Canva.
Topics include advanced identity management, compliance, encryption, and logging — essential yet complex features that most enterprise customers require.
If you are a founder, exec, PM, or engineer tasked with the enterprise roadmap, this conference is for you. You’ll get detailed insights from industry leaders that have years of experience navigating the same challenges you face today. And best of all, it’s completely free since it’s hosted by WorkOS.Top Kafka Use Cases You Should Know
How Uber Served 40 Million Reads with Integrated Redis Cache?
There are 3 main parts of the implementation:
CacheFront Read and Writes with CDC
Uber built CacheFront - an integrated caching solution with Redis, Docstore, and MySQL.
Rather than the microservice, Docstore’s query engine communicates with Redis for read requests.
For cache hits, the query engine fetches data from Redis. For cache misses, the request goes to the storage engine and the database.
In the case of writes, Docstore’s CDC service (Flux) invalidates the records in Redis. It tails MySQL binlog events to trigger the invalidation.
Multi-Region Cache Warming with Redis Streaming
A region fail-over can result in cache misses and overload the database.
To handle this, Uber’s engineering team uses cross-region Redis replication. This is done by tailing the Redis write stream to replicate keys to the remote region.
In the remote region, the stream consumer issues read requests to the query engine that reads the database and updates the cache.
Redis and Docstore Sharding
All teams in Uber use Docstore and some generate a huge number of requests.
Both Redis and Docstore instances are sharded or partitioned to handle the load. But a single Redis cluster going down may create a hot DB shard.
To prevent this, they partitioned the Redis cluster using a scheme that was different from the DB sharding. This ensures that the load is evenly distributed.
Over to you: Would you have done something differently?
Latest articles
If you’re not a paid subscriber, here’s what you missed.
To receive all the full articles and support ByteByteGo, consider subscribing:
What makes AWS Lambda so fast?
There are 4 main pillars:
Function Invocation
AWS Lambda supports synchronous and asynchronous invocation.
In synchronous invocation, the caller directly calls the Lambda function using AWS CLI, SDK, or other services.
In asynchronous invocation, the caller doesn’t wait for the function’s response. The request is authorized and an event is placed in an internal SQS queue. Pollers read messages from the queue and send them for processing.Assignment Service
The Assignment Service manages the execution environments.
The service is written in Rust for high performance and is divided into multiple partitions with a leader-follower approach for high availability.
The state of execution environments is written to an external journal log.Firecracker MicroVM
Firecracker is a lightweight virtual machine manager designed for running serverless workloads such as AWS Lambda and AWS Fargate.
It uses Linux’s Kernel-based virtual machine to create and manage secure, fast-booting microVMs.Component Storage
AWS Lambda also has to manage the state consisting of input data and function code.
To make it efficient, it uses multiple techniques:Chunking to store the container images more efficiently.
Using convergent encryption to secure the shared data. This involves appending additional data to the chunk to compute a more robust hash.
SnapStart feature to reduce cold start latency by pre-initializing the execution environment
Over to you: Which other features do you think make AWS Lambda fast?
Why do we need to use a distributed lock?
A distributed lock is a mechanism that ensures mutual exclusion across a distributed system.
Top 6 Use Cases for Distributed Locks
Leader Election
Distributed locks can be used to ensure that only one node becomes the leader at any given time.Task Scheduling
In a distributed task scheduler, distributed locks ensure that a scheduled task is executed by only one worker node, preventing duplicate execution.Resource Allocation
When managing shared resources like file systems, network sockets, or hardware devices, distributed locks ensure that only one process can access the resource at a time.Microservices Coordination
When multiple microservices need to perform coordinated operations, such as updating related data in different databases, distributed locks ensure that these operations are performed in a controlled and orderly manner.Inventory Management
In e-commerce platforms, distributed locks can manage inventory updates to ensure that stock levels are accurately maintained when multiple users attempt to purchase the same item simultaneously.Session Management
When handling user sessions in a distributed environment, distributed locks can ensure that a user session is only modified by one server at a time, preventing inconsistencies.
SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:36 - 28 Sep 2024 -
Top 10 reports this quarter
McKinsey&Company
Share this email
Top Ten Reports │ Third Quarter 2024
Our top ten reports this quarter look at women in the workplace, climate change, asset management, and more. At No. 1, McKinsey's Humayun Tai, Sven Smit, and coauthors explore the physical realities of the energy transition.
2. Women in the Workplace 2024: The 10th-anniversary report
In the tenth year of our Women in the Workplace research, in partnership with LeanIn.Org, we reflect on the notable gains women have made—and how their experiences at work are, in many ways, the same or worse than ten years ago. Sustainable progress toward parity requires that companies recommit to change. Recommendations for companies
Share these insights
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you are a registered member of the Top Ten Most Popular newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Top Ten" <publishing@email.mckinsey.com> - 11:36 - 28 Sep 2024 -
Odoo Address Validation Solution
Hi MD Abul,
You're now halfway through your GeoPostcodes' trial. Have you been able to explore our data, and what questions do you have?
We're already working with 124 Fortune 500 companies to service their location data needs. Let me know what you're working on, and we can discuss how we can support you with one of our datasets.
To discuss further with our experts, find a time that works for you here.
Best regards,Kashif AliBusiness Development+32 25 86 22 31Rue Aux Laines 70, Brussels 1000, BELGIUM
mercredi 25 septembre 2024, 17:43:24 +0200, Kashif Ali <kashif.ali@geopostcodes.com>:
Hey MD Abul,
We understand the struggle posed by inaccuracies, manual verification complexities, and the constant need for reliable data. GeoPostcodes is designed to be your comprehensive remedy, ensuring precision, saving valuable time, and providing a dependable solution.
Our Address Validation Service ensures the accuracy and completeness of your address database. By integrating GeoPostcodes, companies like DB Schenker have significantly improved their delivery processes, reduced errors, and enhanced customer satisfaction. You can find more information in this blog post on how DB Schenker validates postal data 300X faster
When suits you for a quick call with our experts?
Best regards,
Kashif AliBusiness Development+32 25 86 22 31Rue Aux Laines 70, Brussels 1000, BELGIUM
jeudi 19 septembre 2024, 10:54:32 +0200, Kashif Ali <kashif.ali@geopostcodes.com>:
Hi MD Abul,
Thank you for registering on our customer portal. We noticed your interest in address validation and capture, and we're excited to introduce you to GeoPostcodes. It is a solution that can optimize your address data seamlessly.
GeoPostcodes has helped companies like DB Schenker and Amazon streamline their address validation and capture processes. This has resulted in improved efficiency and customer satisfaction.
Would you like to chat briefly about how GeoPostcodes can enhance your address data management?
Best regards,
Kashif AliBusiness Development+32 25 86 22 31Rue Aux Laines 70, Brussels 1000, BELGIUM
by "Kashif Ali" <kashif.ali@geopostcodes.com> - 07:16 - 27 Sep 2024 -
What changes are tech officers making to realize value from tech?
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
Mindset shifts. In an era of technological change, CIOs, chief technology officers (CTOs), and chief digital and information officers (CDIOs) are looking to turn the promise of tech into real value. How can they realize that goal? Conversations with dozens of business leaders, as well as an analysis of recent research, reveal a set of shifts that tech leaders can make to meet executives’ demands, McKinsey senior partner Aamer Baig and coauthors explain.
•
Orchestrating change. Creating significant gains from AI and tech often means integrating multiple facets of the organization. As orchestrators, tech officers should move from delivering tech to shaping how their companies generate value, directing business and tech teams, and taking on accountability for business outcomes. As one CTO shares, “We’re moving into being IT strategists and setting the direction for the organization.” Discover four types of roles that tech officers can embody as they build value from AI, generative AI, and other technologies.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:38 - 27 Sep 2024 -
Remote's 2024 State of Payroll Report is out 📄
Remote's 2024 State of Payroll Report is out 📄
Featured news
Get Remote's new 2024 State of Payroll Report
“Payroll is a highly strategic asset. If you’re treating payroll as a back office function, you’re already behind the times." – Jonathan Goldsmith, VP of Payroll at Remote.
Download the report and learn how to make payroll work for you.
New masterclass
How to Nail a Remote Interview
Whether you're beginning your job search or aiming to refine your skills, this Masterclass is designed to give you a competitive edge in your remote job hunt by helping you nail your next round of remote interviews.
Product updates
Manage HR Contracts & Documents in Remote
Now you can easily create, send, sign, and store contracts and documents for your employees and contractors without the need for third-party software or manual messiness.
Partnership news
Save time and reduce errors with the Hibob time off integration
%20(1).png?width=1040&upscale=true&name=1200x630%20(13)%20(1).png)
Streamline your HR processes with our expanded Hibob integration! Now you can sync approved time off and Global Payroll employee data from Hibob to Remote, eliminating the need for duplicate work and reducing errors. Get started today and experience the benefits of a seamless HR and time off data management system.
Enable today or Take a self-guided tour
Upcoming events
📍🇦🇹 2 days filled with HR expertise in a magical location
Join the HR Inside Summit on October 9th and 10th at the Hofburg Vienna and experience a day that brings together 2000 participants! Visit us in our Remote Concept Lounge and learn how you can benefit from Remote's experience helping other European start-ups and scale-ups to hire and expand globally.
Register here or reach out to us to get a 15% discount on any ticket category. (*limited availability)
📍🇦🇺 Remote is coming to SXSW Sydney
.png?width=1040&upscale=true&name=Email%20Header(1).png)
Join the Remote team in Sydney where we’ll be hosting a series of events during SXSW Sydney; Unlock global talent during the HR professionals Meet up, learn how to navigate ESOPs in the Australian Startup Ecosystem happy hour or how to scale Growth in the Screen, Games, and XR Industries happy hour
Bookmark these sessions on your SXSW Sydney schedule now
📍🇫🇷 Meet Remote at Unleash World Paris
Remote is returning to Unleash World taking place in Paris 16-17 October. Come by the Remote booth C128 and meet the Remote team.
📍🇳🇱 Join us on HR Day to discover the latest in the HR tool box for HR professionals in the Netherlands
Join Remote at HR Day in Bussum for a day filled with conferences and workshops about the latest HR solutions. Jeroen Kuijer from Remote will be hosting a workshop at 3pm about preserving culture and employee engagement in a remote work environment.
Don’t have your ticket yet? Get a 20% discount by registering using this direct link (*places are limited)
📍🇧🇪 Two days of innovative HR in the heart of Antwerp
.png?width=1040&upscale=true&name=1200x630%20(8).png)
Meet Remote at HR Tech at the Handelsbeurs in Antwerp for two days of innovative HR awards, conferences and networking. Join more than 400 HR decision makers from Belgium and register for Remote’s lunch and learn or break out session on October 3rd.
Remote is the global HR platform you deserve
Onboard, pay, and manage employees and contractors around the world with Remote. You focus on finding the best hires — we'll handle the rest.
You received this email because you are subscribed to
News & Offers from Remote Europe Holding B.V
Update your email preferences to choose the types of emails you receive.
Unsubscribe from all future emailsRemote Europe Holding B.V
Copyright © 2024 Remote Europe Holding B.V All rights reserved.
Kraijenhoffstraat 137A 1018RG Amsterdam The Netherlands
by "Remote" <hello@remote-comms.com> - 11:24 - 26 Sep 2024 -
A human-centric approach to CEO leadership, board–CEO collaboration, inclusive care for people with disabilities, and more highlights
Share this email
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of The Weekend Read. We hope you find our perspectives useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
Ready to unwind?
—Edited by Joyce Yoo, editor, New York
Follow our thinking
Share these insights
Did you enjoy this newsletter? Forward it to colleagues and friends so they can subscribe too. Was this issue forwarded to you? Sign up for it and sample our 40+ other free email subscriptions here.
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Quarterly Five Fifty alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Weekend Read" <publishing@email.mckinsey.com> - 10:11 - 26 Sep 2024
