Archives
-
Just “For You”
Get the free McKinsey Insights app
Learn more about the Insights app Dear Md Abul,
The McKinsey Insights app just turned ten, and we’re celebrating this milestone app-iversary with a major upgrade: a personalized “For You” feed of only the insights you care about most.Introducing For You
Choose the topics and industries you want to follow, and we’ll round up for you the most relevant articles, reports, podcasts, and more.
Opt in for notifications to find out when new insights are published on your topics.Explore what’s trending
The app now also offers a separate Explore feed with all our can’t-miss insights. There you can also:
— Get app-only early access to select articles
— See what other app users are reading
— Dive into curated collectionsWe value your feedback. To suggest features or request technical support, drop us a note.
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you are a registered member of mckinsey.com.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey & Company" <publishing@email.mckinsey.com> - 02:27 - 3 Oct 2024 -
API Gateway
API Gateway
Forwarded this email? Subscribe here for moreAPI Gateways are essential components in modern software architectures, particularly microservices-based systems.
They act as a single entry point for all API requests, providing a unified interface for accessing various services and data. By decoupling client applications from backend microservices, API Gateways simplify API management and enhance the overall performance and security of the system.
API Gateways help enhance the developer experience while building scalable and maintainable systems. Organizations can achieve better control over their API landscape by using API Gateways.
As the complexity of software systems continues to grow, API Gateways will play an increasingly important role in enabling effective communication and integration between services and clients.
In this post, we’ll explore the various aspects of API Gateways in detail.
What is an API Gateway...
Continue reading this post for free in the Substack app
Like
Comment
Restack
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:37 - 3 Oct 2024 -
New Relic Now: Come see the future of observability
New Relic
September 2024
Want to meet Intelligent Observability? Join us at our free virtual event. We’re excited to invite you to New Relic Now, a free virtual event taking place on November 6. This event will focus on how AI is supercharging the Observability landscape, including how you and your team can create more reliable digital experiences. There will also be free hands-on training opportunities with platform experts.Register now 
New Relic Now: Hands-on Training
Are you a New Relic user with a desire to learn more about New Relic Observability practices?
Get hands-on training from New Relic University experts to start your journey to new certifications.
AI in observability: Advancing system monitoring and performance
Explore how AI revolutionizes observability, from automated anomaly detection to advanced monitoring of AI-driven systems.
Useful reads
How to monitor Oracle ExaCC databases using New Relic
New Relic database OHI has been revamped for monitoring the new containerised databases released by Oracle.
Observability as code for AI apps with New Relic and Pulumi
Discover how to streamline AI observability with New Relic and Pulumi. Learn how to automate AI app monitoring, securely manage cloud resources, and gain deep insights into AI performance, costs, and compliance—all with an 'Observability as code' approach.
Unify monitoring across the entire SAP landscape
Learn how to track test-related events, improve code coverage insights, and optimize end-to-end test performance using New Relic.
Events and Webinars New Relic Now: The future of observability is intelligent. Are you ready?
See how Intelligent Observability enables you to predict and prevent business-critical incidents in your digital experiences during this virtual event on 6 November.
Observability as Code for AI Apps with new Relic and Pulumi
Join Harry Kimpel, Principal Developer Relations Engineer on 30 Oct at 4pm GMT for this workshop and discover how to monitor AI app costs, deploy efficiently, and manage credentials securely.
New Relic University online workshops
Catch our recent New Relic workshops on-demand. These trainer-led workshops with hands-on labs will help you up-level your observability skills.
New Relic End-of-Life Updates
Azure SQL database integration through API Polling (1 December)
- We’re ending support for the Polling API that can report data to the Azure SQL Database integration.
- If you’re using the Polling API to report data to the Azure SQL Database integration, we recommend that you migrate to the Azure Monitor integration
- Only customers who fetch data with the Polling API for Azure SQL Database integration are affected.
- See here for more information.
createTracer() API call for Browser Monitoring (15 October)
- We’re EOL’ing the createTracer() API call
- New Relic is updating the browser SPA agent to be more performant, with improved accuracy and fewer lines of code. These updates simplify the agent, letting customers diagnose issues with their browser apps more easily.
- Before New Relic can update the browser SPA agent, we need to first end-of-life the createTracer() API.
- Accounts using the createTracer() API at the time of EOL will lose performance metrics about their SPAs.
- We recommend that you migrate to the addPageAction() API call to collect time data about sub-components on your pages.
- See here for more information.
Need help? Let's get in touch.
This email is sent from an account used for sending messages only. Please do not reply to this email to contact us—we will not get your response.
This email was sent to info@learn.odoo.com Update your email preferences.
For information about our privacy practices, see our Privacy Policy.
Need to contact New Relic? You can chat or call us at +44 20 3859 9190.
Strand Bridge House, 138-142 Strand, London WC2R 1HH
© 2024 New Relic, Inc. All rights reserved. New Relic logo are trademarks of New Relic, Inc
by "New Relic" <emeamarketing@newrelic.com> - 05:02 - 3 Oct 2024 -
Exclusive Invitation: Join Us for a VIP Round Table & Networking Lunch in Singapore!
Exclusive Invitation: Join Us for a VIP Round Table & Networking Lunch in Singapore!
We are excited to extend an exclusive invitation to a transformative session featuring Barbara Matthew, Chief People Officer at Remote, and Allen Jaeyoon Lee, Head of People at Aspire. As the world shifts toward remote and hybrid work models, mastering the art of managing distributed teams and fostering a vibrant company culture has never been more crucial.
Global Hiring Trends in APAC:
Innovative HR Strategies for Hybrid Teams📅 Monday, October 14, 12 PM SGT
📍 Cenzo, 81 Club St, Singapore
🌟 What to Expect: Join us for an enlightening discussion where Barbara and Allen will share their expert insights, moderated by IDC Research Director Abhishek
- Asia-pacific's hiring trend with Global expansion : Learn how we, as leaders, can not only hire the right talent across borders but also manage employees effectively, based on the 2024 IDC AP Hiring and Talent Management results.
- Practical Tips & Innovative Best Practices : Discover actionable strategies & how forward-thinking HR and business leaders are navigating the future of work.
This is your chance to gain invaluable perspectives from industry trailblazers and participate in a dialogue that will shape the future of your organization.
Don’t miss out on this opportunity to enhance your leadership approach in today’s evolving work environment!
You received this email because you are subscribed to News & Offers from Remote Europe Holding B.V
Update your email preferences to choose the types of emails you receive.
Unsubscribe from all future emailsRemote Europe Holding B.V
Copyright © 2024 All rights reserved.
Kraijenhoffstraat 137A 1018RG Amsterdam The Netherlands
by "Remote" <hello@remote-comms.com> - 01:59 - 3 Oct 2024 -
What is leadership?
Only McKinsey Perspectives
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:14 - 3 Oct 2024 -
Guide: How AI Is Transforming the World of Work
Workday
View as Web Page 
Guide Reimagining your business with artificial intelligence. Read Guide 
Artificial intelligence (AI) is rapidly transforming how we work. From automating tasks to aggregating data, AI has become critical to business success.
In this guide, learn how AI can help accelerate your business and equip you with the insights to stay ahead of the curve. Discover how to:- Leverage automation for repetitive tasks and free up scarce resources
- Improve accuracy and lower costs
- Equip people with AI-assisted insights
Read Guide Follow Workday
©2007– 2024 Workday, Inc. • Unsubscribe
1 Wallich Street, #08-02 Guoco Tower | Singapore 078881, Singapore
by "Workday" <corp.email@workday.com> - 10:06 - 2 Oct 2024 -
How to make tech work for you
Re:think
FRESH TAKES ON BIG IDEAS
ON TECH ORGANIZATIONS
Want to be a tech company? BuildEvery business leader says they want their organization to be a tech company. But what does that mean in practice? There are some things it means, and others it doesn’t. It does mean you need an organization that can build technology. And we really mean build, not just make good choices about what to buy.
In our research over the past seven years, we’ve looked closely at hundreds of companies that are true digital leaders. Even before generative AI, 70 percent said that they were making their own software. A third were aspiring to monetize that software externally, but the vast majority were using it to help propel their own operations. We expect that number will eventually trend toward 100 percent. Everybody’s going to be able to make the technology they need to run their business. They’re also going to buy technology. But in domains where they hope to gain a real competitive advantage, they want to build something that is unique to them by assembling a carefully curated mix of available components.
What does this mean for your company? First, you want business and technology working well together and you want technology embedded in the company. Organizations that get that right are much more likely to be successful than those that don’t. At a recent conference, I was challenged to come up with one paragraph that explains the difference between a modern digital and AI tech culture and an IT culture. After some back-and-forth, we got it down to one word: “requirements.”
In an IT culture, businesspeople write requirements and hand them off to the technology team so they can provide gains. At successful digital and AI companies, business and technology co-own a problem, iterate around that problem, and own the answer on an ongoing basis. That’s not a new thought; it’s been called a product operating model, or agile before that. But how business and technology work together is the essential factor for operating like a tech company.
For our recent book, Rewired: The McKinsey Guide to Outcompeting in the Age of Digital and AI, we benchmarked 50 banks to understand which ones were creating value through technology and which were not. We found that some 25 percent could pinpoint real and significant value from their digital investments. This 25 percent couldn’t be identified by how much they were spending, the quality of their apps, or even their technology architecture choices. They could be identified, however, by how well business and technology worked together.“Being a tech company doesn’t necessarily mean spending the most on technology. This is not about spending the most.”
Speed is another important factor. Companies that are becoming tech companies iterate faster and faster. They work in agile short-sprint cycles. They insource more than they outsource. To do so, they don’t just go out and hire all the cool kids from Silicon Valley. That’s a really good way to change the company’s dress code, but if you want to create lasting impact, you need to upskill and reskill your own employees. You also create the right technology career ladders for your digital talent. This means that technologists can learn from other technologists and that they can progress in a way that makes them as important as business leaders, rather than seen as support.
Being a tech company also means caring deeply about adopting and scaling technology. It’s change management; although I don’t like that term, because it implies that technology is always wonderful, and we dumb humans need to adapt to it. Instead, it’s about digging into the question of how you transform a business domain and thinking hard about the right incentives to encourage true technology adoption and scaling. This usually means incentivizing the businesspeople. They need to co-own the tech impact incentives. If everyone’s looking to the CEO for constant direction, that’s a failure.
There are some important questions a CEO can ask to determine if their organization is functioning as a true tech company. Do you understand which business domains are most able to be transformed by technology and have you put a clear financial goal against them? Do the best technologists want to work at your company? Do you have a product-release-and-update cycle that’s measured in weeks, not months? When you look at your senior-leadership team, how many would self-identify as being tech savvy and tech capable? When you think about advancing people into senior roles, is experience in getting value from tech one of the essential criteria? Do you have a tech talent road map that is as detailed as your road map for scaling technology? Do you understand where data will create competitive advantage in your company and how to combine your proprietary data successfully with the world’s data?
For some, these will coalesce into two questions. First, have you thought about how technology can completely reinvent your business? Second, how do you do business today and how could that fundamentally change if you harnessed technology properly?
Being a tech company doesn’t necessarily mean spending the most on technology. This is not about spending the most. It’s not about being the earliest adopter of new technologies. Perhaps you like to talk about having technology everywhere in your company or about the success of an exciting pilot. But if you’re not actually seeing the technology affect your profit and loss, you’re not a tech company.Share Rodney Zemmel’s insights
ABOUT THIS AUTHOR
Rodney Zemmel is a senior partner in McKinsey’s New York office.
MORE FROM THIS AUTHOR
UP NEXT
Alex Singla on CEOs and generative AI
Generative AI raises many questions. Six of them—about opportunities, governance, partners, risk, talent, and learning—preoccupy many CEOs.
Follow our thinking
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Quarterly alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Quarterly" <publishing@email.mckinsey.com> - 02:48 - 2 Oct 2024 -
What steps can companies take to advance women at work?
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
Underrepresented in corporate America. Over the past decade, women’s representation has increased at every level of corporate management, with women today making up 29% of C-suite positions, compared with just 17% in 2015, McKinsey senior partners Alexis Krivkovich and Lareina Yee and coauthors share. Yet women remain underrepresented across the pipeline, a gender gap that persists regardless of race and ethnicity, finds the tenth-anniversary Women in the Workplace report, conducted in partnership with LeanIn.Org.
•
Barriers to entry. Women continue to face barriers at the beginning of the pipeline. They remain less likely than men to be hired into entry-level roles and are also far less likely than men to attain their very first promotion to a manager role. This year, for every 100 men who received their first promotion to manager, just 81 women were promoted. Learn some practical steps companies can take to support the advancement of women and make the workplace more equitable.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:13 - 2 Oct 2024 -
The Trillion Message Kafka Setup at Walmart
The Trillion Message Kafka Setup at Walmart
Forwarded this email? Subscribe here for moreThe Enterprise Ready Conference for engineering leaders (Sponsored)
The Enterprise Ready Conference is a one-day event in SF, bringing together product and engineering leaders shaping the future of enterprise SaaS.
The event features a curated list of speakers with direct experience building for the enterprise, including OpenAI, Vanta, Checkr, Dropbox, and Canva.
Topics include advanced identity management, compliance, encryption, and logging — essential yet complex features that most enterprise customers require.
If you are a founder, exec, PM, or engineer tasked with the enterprise roadmap, this conference is for you. You’ll get detailed insights from industry leaders that have years of experience navigating the same challenges you face today. And best of all, it’s completely free since it’s hosted by WorkOS.Disclaimer: The details in this post have been derived from the Walmart Global Tech Blog. All credit for the technical details goes to the Walmart engineering team. The links to the original articles are present in the references section at the end of the post. We’ve attempted to analyze the details and provide our input about them. If you find any inaccuracies or omissions, please leave a comment, and we will do our best to fix them.
Walmart has a massive Apache Kafka deployment with 25K+ consumers across private and public cloud environments.
This deployment processes trillions of Kafka messages per day at 99.99% availability. It supports critical use cases such as:
Movement of data
Event-driven microservices
Streaming analytics
At Walmart's scale, the Kafka setup must be able to handle sudden traffic spikes. Also, consumer applications are written in multiple languages. Therefore, all consumer applications must adopt some best practices to maintain the same level of reliability and quality.
In this post, we’ll look at the main challenges of a Kafka setup at this scale. Then, we will look at how Walmart’s engineering team enhanced its setup to overcome these challenges and reliably process messages cost-efficiently.
Challenges with Kafka at Walmart’s Scale
Let’s start with understanding the main challenges that Walmart faced.
1 - Consumer Rebalancing
One of the most frequent problems was related to consumer rebalancing.
But what triggers consumer rebalancing in Kafka?
This can happen due to the changing number of consumer instances within a consumer group.
Several scenarios are possible such as:
A consumer pod may enter or leave a consumer group. This can happen due to Kubernetes deployments, rolling restarts, or automatic scale-ins or scale-outs. Whenever it happens, Kafka needs to redistribute the partitions among the consumers.
The Kafka broker may believe that a consumer has failed. If the broker has not received a heartbeat from a consumer within the configured session timeout, it assumes that the consumer has died. This can happen if the consumer’s JVM exits or experiences a long stop-the-world garbage collection pause.
The Kafka broker may believe that a consumer is stuck and trigger rebalancing. If the consumer takes longer than a threshold to poll for the next batch of records, the broker marks it as stuck. This can happen when processing the previous batch takes too long.
Consumer rebalancing is needed to ensure partitions are evenly distributed. However, rebalancing can cause disruption and increased latency, particularly due to the near real-time nature of the e-commerce landscape.
2 - Poison Pill Messages
A “poison pill” message in Kafka is a message that consistently causes a consumer to fail when attempting to process it. This can happen due to various reasons such as:
Malformed Data: The message payload may not be in an expected format. For example, invalid JSON or missing required fields. This may cause the consumer to throw an exception while processing.
Unexpected Data: The message content might be syntactically valid but semantically incorrect. In other words, it might violate some business constraints.
Bugs in the Consumer Code: If there’s a bug (like a null pointer exception) in the code that handles the message, the processing will fail.
When the consumer encounters such a message, it will fail to process it and throw an exception. By default, the consumer will return to the broker to fetch the same batch of messages again. Since the poison pill message is still present in that batch, the consumer will again fail to process it, and this loop continues indefinitely.
As a result, the consumer gets stuck on this one bad message and is unable to make progress on other messages in the partition. This is similar to the “head-of-line blocking” problem in networking.
3 - Cost Concerns
There is a strong coupling between the number of partitions in a Kafka topic and the maximum number of consumers that can read from that topic in parallel. This coupling can lead to increased costs when trying to scale consumer applications to handle higher throughput.
For example, consider that you have a Kafka topic with 10 partitions and 10 consumer instances reading from this topic. Now, if the rate of incoming messages increases and the consumers are unable to keep up (i.e. consumer lag starts to increase), you might want to scale up your consumer application by adding more instances.
However, once you have 10 consumers (one for each partition) in a single group, adding more consumers to that group won’t help because Kafka will not assign more than one consumer from the same group to a partition. The only way to allow more consumers in a group is to increase the number of partitions in the topic.
However, increasing the number of partitions comes with its challenges and costs.
Kafka has a recommended limit on the number of partitions per broker (for example, 4000 partitions per broker). If you keep increasing partitions, you may hit this limit and need to scale the Kafka brokers to larger instances, even if the brokers have sufficient resources to handle the current load. Scaling to larger broker instances is expensive.
Increasing partitions requires coordination among the Kafka team, the producer, and the consumer teams. In a large organization with thousands of Kafka pipelines, this coordination overhead is significant.
More partitions also mean more open file handles, increased memory usage, and more threads on the Kafka brokers. This can lead to higher resource utilization and costs.
Designing the Messaging Proxy Service (MPS)
To overcome the challenges mentioned in the previous section, the Walmart engineering team designed a Message Proxy Service (MPS).
The diagram below shows a high-level view of MPS.
The MPS aims to decouple Kafka message consumption from the constraints imposed by Kafka’s partition-based model. It works like this:
MPS acts as a proxy between the Kafka brokers and the actual message consumer applications. It reads messages from Kafka partitions and puts them into a separate in-memory queue.
Consumer applications don’t directly read from Kafka. Instead, they receive messages from MPS via HTTP/REST. This allows the consumer applications to scale independently of the number of Kafka partitions.
MPS ensures in-order message processing per key, handles consumer application failures, and manages offset commits back to Kafka.
The diagram below shows the detailed design of the MPS with all its components
Let’s now look at the various components of the MPS in more detail.
Reader Thread
This is a single thread that reads messages from Kafka.
It writes the messages from the Kafka broker into a bounded queue called the “PendingQueue”. If the PendingQueue reaches its maximum size, the reader thread will pause reading from Kafka.
This is a form of backpressure to prevent the queue from growing indefinitely if the writer threads cannot keep up.
Bounded Buffer Queue (PendingQueue)
This is a queue that sits between the reader thread and the writer threads. It has a maximum size to prevent it from consuming too much memory.
The PendingQueue allows the reader and write threads to work at different speeds. The reader can read messages as fast as Kafka can provide them, while the writers can process them at their own pace.
Order Iterator
This component ensures that messages with the same key are processed in the order they were received from Kafka.
It goes through the messages in the PendingQueue and skips any message if there is already an earlier message with the same key being processed. At any given time, at most one message per key is being handled by the writer threads.
Writer Threads
These are a pool of threads that take messages from the PendingQueue and send them to the consumer applications via HTTP POST requests.
If a POST request fails, the writer thread will retry the request a few times. If the retries are exhausted or if the consumer application returns certain HTTP codes, the writer thread will put the message into a Dead Letter Queue (DLQ).
The writer threads also help manage offsets. They update a shared data structure to keep track of processed offsets.
Offset Commit Thread
This is a separate thread that periodically wakes up (for example, every minute) and commits the Kafka offsets of processed messages using the Kafka consumer API.
It checks the shared data structure that is updated by the writer threads. Then, it commits the latest continuous offset for each partition. For example, if messages with offsets 1, 2, 3, and 5 have been processed for a partition, it will commit offset 3 (because 4 is missing).
By committing offsets periodically, MPS tells Kafka which messages it has processed successfully. If MPS crashes or is restarted, it will start consuming messages from the last committed offset, avoiding reprocessing messages that have already been handled.
Consumer Service REST API
This is the specification that the actual message consumer applications need to implement to receive messages from the MPS.
It defines the format of the HTTP POST request that the MPS writer threads will send (headers, body, etc.). It also specifies the meaning of different HTTP response codes that the consumer application can return.
See the table below that shows the API specification:
Source: Walmart Tech Blog Implementation of MPS
MPS was implemented as a Kafka Connect sink connector.
For reference, Kafka Connect is a framework for connecting Kafka with external systems such as databases, key-value stores, search indexes, and file systems. It provides a standard way of defining connectors that move data into and out of Kafka.
The diagram below shows a high-level view of Kafka Connect
By implementing MPS as a Kafka Connect sink connector, the developers were able to use several features provided by the Kafka Connect Framework such as:
Multi-tenancy: Kafka Connect allows running multiple connectors on a single cluster. This means that a single MPS deployment can serve multiple consumer applications (tenants).
Dead Letter Queue (DLQ) Handling: Kafka Connect has built-in support for handling messages that cannot be processed.
Offset Commits: Kafka Connect provides APIs for committing offsets. MPS uses these for its offset commit thread.
Scalability: Kafka Connect is designed to be scalable and fault-tolerant. By building on top of Kafka Connect, MPS inherits these properties.
Apart from this, the consumer services (applications that process the messages) are designed to be stateless. This means they don’t maintain any persistent state locally. Any state they need is either passed with the message or stored in an external database.
Being stateless allows these services to be easily scaled up or down in Kubernetes based on changes in message volume. If more messages need to be processed, Kubernetes can simply start more instances of the consumer service. If fewer messages are coming in, Kubernetes can terminate some instances to save resources.
Importantly, this scaling of the consumer services is independent of MPS and Kafka. MPS continues to read from Kafka and deliver messages to the consumer services regardless of how many instances of the consumer service are running.
Additional Points To Consider
Here are some additional points worth considering based on the MPS solution implemented by Walmart.
1 - Rebalancing of the MPS
MPS is also essentially a Kafka consumer. It reads messages from Kafka topics and makes them available to the application consumers via REST endpoints.
Like any other Kafka consumer, MPS would also be subject to rebalancing when the number of MPS instances changes. Based on the details, however, it seems that MPS is designed to handle rebalancing gracefully. The separation of the reader thread (which polls Kafka) and the writer threads (which send messages to the REST consumers) is the key here.
As long as MPS comes back up quickly after a rebalance, the REST consumers should be able to continue processing messages without substantial lag. The MPS design also includes a bounded buffer (the PendingQueue) between the reader thread and writer threads. This buffer helps to smoothen any temporary fluctuations in the rate at which MPS is reading from Kafka.
2 - Choice of REST
MPS calls REST APIs exposed by the consumer instances. Interestingly, the choice was REST and not something like gRPC.
This may be because of the simplicity of REST. Also, REST is widely supported by almost all languages and frameworks.
3 - Potential Increase in Complexity
While MPS solves several problems, it also introduces an additional layer to the system.
Instead of just having Kafka and the consumer applications, there is now a proxy service in the middle. This means more components to develop, deploy, monitor, and maintain.
Conclusion
The implementation of MPS helped Walmart achieve some key improvements.
Most rebalances have now been eliminated except for rare restarts or network issues. With MPS, the reader thread consistently puts all polled messages into the PendingQueue within the allocated time. This prevents rebalances triggered by the Kafka broker thinking the consumer is stuck.
Poison pill messages are handled in a better way. With MPS, consumer services can detect poison pill messages and notify MPS using specific HTTP return codes (600 and 700).
MPS enables cost savings in multiple ways. Firstly, consumer services are now stateless and can scale quickly in Kubernetes based on demand. They don’t need to be scaled in advance. Secondly, Kafka clusters can be scaled based on throughput rather than the number of partitions.
References:
SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:35 - 1 Oct 2024 -
Actionable insights on elevating your API observability
OTel & monitoring
Hi Md Abul,
Looking to enhance your API observability?
Our blog, "How to improve API observability," focuses on refining your approach to monitoring APIs. Discover how getting to grips with observability can transform your microservices and significantly boost your business outcomes.
Ready to experience these benefits firsthand? Start your free trial today and see the difference.
Thanks,
Budha Bhattacharya
Developer Advocate
Tyk, Huckletree 199 Bishopsgate, Broadgate, London, City of London EC2M 3TY, United Kingdom, +44 (0)20 3409 1911
by "Budha from Tyk" <budha@tyk.io> - 06:00 - 1 Oct 2024 -
How can employers better support neurodivergent workers?
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
Employment challenge. Globally, about one in five people are considered neurodivergent—exhibiting differences in their brain function and behavioral traits—per Dr. Lawrence Fung, associate professor of psychiatry and behavioral sciences at Stanford University and the director of the Stanford Neurodiversity Project. Fung discusses neurodiversity and employment with McKinsey senior partner Brooke Weddle and partner Bryan Hancock on a recent episode of McKinsey Talks Talent, relating that about 80% of people with autism are unemployed or underemployed.
•
Collegial job interviews. People with autism often have difficulty during job interviews because of their manner of social interaction, Fung says. For example, some autistic people might talk about their special interests at length, and interviewers who don’t detect signs of neurodiversity might become disengaged. Training interviewers to understand such behaviors could make the interview process more friendly. Listen to the podcast to learn ways that employers can support neurodivergent workers.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:45 - 1 Oct 2024 -
[You're Invited, Md Abul] | Create Stylish and Standout Documents Faster with GenAI
Adobe
Adobe Live Webinar
Create Stylish and Standout Documents Faster with GenAI 10 October 2024, Thursday
Option 1: 11am AEDT | 8am SGT
Option 2: 4pm AEDT | 1pm SGT | 10:30am IST![[You're Invited] | Create Stylish and Standout Documents Faster with GenAI](https://engage.adobe.com/rs/360-KCI-804/images/apac-fy24q4-wbr-transform-documents-mcz-banner-1200x540.jpg?version=0)
In today’s fast-paced work environment, efficiency and style are key. Discover how Adobe Acrobat AI Assistant can help you master PDF styling and boost productivity, making your document more polished and professional.
Join Callum Swain, APAC Solution Customer Success Manager for Document Cloud at Adobe, to learn how to create standout PDFs using the advanced features of Acrobat AI Assistant. Learn how to give your documents a professional look with the easy-to-use tools from Adobe Express.
Take the first step toward smarter, more stylish document management! Register today to secure your spot!
Two time options available. Register for the session that best suits your schedule and timezone:
Register for option 1: 11am AEDT | 8am SGT
Register for option 2: 4pm AEDT | 1pm SGT | 10:30am IST
Who should attend
This session is ideal for professionals who work extensively with PDFs and seek to enhance their document management skills. If you're a project manager, team leader, marketing professional, or anyone involved in document preparation and review, you'll find valuable insights on how to leverage Adobe Acrobat's advanced features. Discover how to streamline workflows, polish your documents, and improve collaboration—all while boosting your productivity.
Can't make it to the live event? Register anyway so you receive the session recording afterwards!
Angus Cheung
APAC B2B Product Marketing & GTM Manager, Document Cloud
Adobe
Callum Swain
APAC Solution Customer Success Manager, Document Cloud
AdobeAcrobat's got it. 
Adobe and the Adobe logo are either registered trademarks or trademarks of Adobe in the United States and/or other countries. This is not a comprehensive list of all Adobe trademarks. For a full list, refer to the Adobe List of Trademarks. All other trademarks are the property of their respective owners.
By clicking on some of the links in this email, you might be redirected to forms that will be pre-populated with your contact information.
This is a marketing email from Adobe Systems Software Ireland Limited, 4‑6 Riverwalk, Citywest Business Park, Dublin 24, Ireland.
Click here to unsubscribe or send an unsubscribe request to the postal address above. Please review the Adobe Privacy Policy:
Australia
New Zealand
Indonesia
Malaysia
Philippines
Vietnam
Singapore
India
Hong Kong
Thailand
To ensure email delivery, add demand@info.adobe.com to your address book, contacts, or safe sender list.
If you have a privacy-related complaint, send it to: privacy@adobe.com
View in browser
by "Adobe Document Cloud" <demand@info.adobe.com> - 10:07 - 30 Sep 2024 -
[ New Estimate Bid ]
Dear,I hope you're doing well. At AIM ESTIMATION, we specialize in providing precise cost estimation and take-off services to General Contractors, Sub-Contractors, and Home Builders. Our team brings extensive expertise across trades, including Civil, Electrical, Plumbing, HVAC, and more.
We help ensure your projects are accurately budgeted and well-prepared for success. I'd love to discuss how we can assist with your upcoming work.
Best regards,
[ AIM ESTIMATION 'Company' ]
[ On Time, On Target ]
by "David Millan" <david.aimestimator@gmail.com> - 10:02 - 30 Sep 2024 -
Join me to learn how to monitor your AI applications with New Relic
Hi MD,I’d like to personally invite you to my upcoming 60 minute online webinar, How to monitor your AI applications with New Relic, on Tuesday at 10am BST / 11am CEST.
I will share practical insights on how to overcome the unique challenges generative AI applications present for those engineers responsible for building and managing AI-powered technologies.
Attend and learn how to:- Accelerate AI initiatives: Proactively monitor and resolve issues to keep your AI projects on track with New Relic.
- Optimise LLM selection and management: Choose the right LLMs for your application and manage their performance and costs effectively.
- Ensure compliance: Build generative AI solutions that adhere to compliance standards and regulations.
- Optimise performance: Track the performance of your models with telemetry data and choose the right model.
I hope to see you then,Harry Kimpel
Principal Developer Relations Engineer
New Relic
This email was sent to info@learn.odoo.com as a result of subscribing or providing consent to receive marketing communications from New Relic. You can tailor your email preferences at any time here.Privacy Policy © 2008-24 New Relic, Inc. All rights reserved
by "Harry Kimpel, New Relic" <emeamarketing@newrelic.com> - 05:02 - 30 Sep 2024
.png)
