Archives
-
Meet the partners behind our best 2024 insights on resilience
Share this email
New from McKinsey & Company
To see more essential reading on topics that matter, visit McKinsey Themes.
— Edited by Esther Chung, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Global Institute alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey & Company" <publishing@email.mckinsey.com> - 01:00 - 22 Dec 2024 -
The quarter’s top themes
McKinsey&Company
Share this email
In the fourth quarter of 2024, our top ten posts from McKinsey Themes highlight generative AI, the organization of the future, interviews with industry leaders, and more. At No. 1 is “18 industries that could reshape the global economy,” which features must-read insights from McKinsey’s Kweilin Ellingrud, Homayoun Hatami, and more. Read on for our full top 10.
3. The skills revolution and the future of learning and earning
October is National Learning and Development Month in the US, but technology is changing the nature of work all around the world. As organizations deploy these new technologies—including automation and artificial intelligence—how can they support current employees and prepare the future workforce for success? Fly above competitors
“Today’s leaders must master complex issues such as digital transformation, inflation, disrupted global supply chains, scarce talent, a lack of diversity, cybersecurity, and climate change, as well as an awakened search for purpose among employees,” say McKinsey senior partners Dana Maor, Kurt Strovink, Ramesh Srinivasan, and senior partner emeritus Hans-Werner Kaas. Read more
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you are a registered member of the Top Ten Most Popular newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Top Ten" <publishing@email.mckinsey.com> - 10:50 - 22 Dec 2024 -
The week in charts
The Week in Charts
Share this email
Share these insights
Did you enjoy this newsletter? Forward it to colleagues and friends so they can subscribe too. Was this issue forwarded to you? Sign up for it and sample our 40+ other free email subscriptions here.
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to The Week in Charts newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Week in Charts" <publishing@email.mckinsey.com> - 03:47 - 21 Dec 2024 -
Is your entire company thinking like a marketer? It should be.
Share this email
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of McKinsey Classics. We hope you find our perspectives useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
December 2024
In honor of the 60th birthday of McKinsey Quarterly, the McKinsey Classics newsletter is dedicating a full year to enduring Quarterly articles that were ahead of their time. Enjoy, and check out other celebration highlights such as special themed issues and bonus digital features.
One thing is certain: you can never be too certain about consumer behavior. Where and how shoppers interact with a company’s product is always evolving. An organization’s marketing function should evolve as well. Even back in 2011, the authors of this McKinsey Quarterly classic observed that a successful marketing function can no longer rely on the traditional approach of disparate marketing efforts. Instead, marketers must engage effectively with customers across the entire consumer journey. And to achieve that, everyone in an organization needs to think like a marketer.
Ensuring that a company’s marketing function is firing on all cylinders across the entire organization involves four crucial dimensions: embed core marketing activities across multiple functions; establish marketing councils to help boost management coordination; prioritize the gathering of customer insights and behavior; and engage in comprehensive data analysis to act on new consumer data.
Today’s consumer environment is only getting more complex, and consumers themselves ever savvier. To learn how to meet the marketing moment, read our 2011 classic “We’re all marketers now.”Read our latest thinking on this classic topic
— Edited by Drew Holzfeind, editor, Chicago
Is there a classic business topic you’d like us to feature? Send us an email—we’d love to hear from you.
Share these insights
Did you enjoy this newsletter? Forward it to colleagues and friends so they can subscribe too. Was this issue forwarded to you? Sign up for it and sample our 40+ other free email subscriptions here.
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Classics newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Classics" <publishing@email.mckinsey.com> - 12:12 - 21 Dec 2024 -
EP143: DNS Record Types You Should Know
EP143: DNS Record Types You Should Know
Forwarded this email? Subscribe here for moreThis week’s system design refresher:
8 Most Important System Design Concepts You Should Know (Youtube Video)
DNS Record Types You Should Know
Polling Vs Webhooks
API Vs SDK!
How Netflix Really Uses Java?
Big O Notation 101: The Secret to Writing Efficient Algorithms
The Ultimate Kubernetes Command Cheatsheet
SPONSOR US
8 Most Important System Design Concepts You Should Know
DNS Record Types You Should Know!
Here are the 8 most commonly used DNS Record Types.
A (Address) Record
Maps a domain name to an IPv4 address. It is one of the most essential records for translating human-readable domain names into IP addresses.CNAME (Canonical Name) Record
Used to alias one domain name to another. Often used for subdomains, pointing them to the main domain while keeping the actual domain name hidden.AAAA Record
Similar to an A record but maps a domain name to an IPv6 address. They are used for websites and services that support the IPv6 protocol.PTR Record
Provides reverse DNS lookup, mapping an IP address back to a domain name. It is commonly used in verifying the authenticity of a server.MX Record
Directs email traffic to the correct mail server.NS (Name Server) Record
Specifies the authoritative DNS servers for the domain. These records help direct queries to the correct DNS servers for further lookups.SRV (Service) Record
SRV record specifies a host and port for specific services such as VoIP. They are used in conjunction with A records.TXT (Text) Record
Allows the administrator to add human-readable text to the DNS records. It is used to include verification records, like SPF, for email security.
Over to you: Which other DNS Record Type have you seen?
Polling Vs Webhooks
Polling
Polling involves repeatedly checking the external service or endpoint at fixed intervals to retrieve updated information.
It’s like constantly asking, “Do you have something new for me?” even where there might not be any update.
This approach is resource-intensive and inefficient.
Also, you get updates only when you ask for it, thereby missing any real-time information.
However, developers have more control over when and how the data is fetched.
Webhooks
Webhooks are like having a built-in notification system.
You don’t continuously ask for information.
Instead you create an endpoint in your application server and provide it as a callback to the external service (such as a payment processor or a shipping vendor)
Every time something interesting happens, the external service calls the endpoint and provides the information.
This makes webhooks ideal for dealing with real-time updates because data is pushed to your application as soon as it’s available.
So, when to use Polling or Webhook?
Polling is a solid option when there is some infrastructural limitation that prevents the use of webhooks. Also, with webhooks there is a risk of missed notifications due to network issues, hence proper retry mechanisms are needed.
Webhooks are recommended for applications that need instant data delivery. Also, webhooks are efficient in terms of resource utilization especially in high throughput environments.API Vs SDK!
API (Application Programming Interface) and SDK (Software Development Kit) are essential tools in the software development world, but they serve distinct purposes:
API:
An API is a set of rules and protocols that allows different software applications and services to communicate with each other.It defines how software components should interact.
Facilitates data exchange and functionality access between software components.
Typically consists of endpoints, requests, and responses.
SDK:
An SDK is a comprehensive package of tools, libraries, sample code, and documentation that assists developers in building applications for a particular platform, framework, or hardware.Offers higher-level abstractions, simplifying development for a specific platform.
Tailored to specific platforms or frameworks, ensuring compatibility and optimal performance on that platform.
Offer access to advanced features and capabilities specific to the platform, which might be otherwise challenging to implement from scratch.
The choice between APIs and SDKs depends on the development goals and requirements of the project.
Over to you:
Which do you find yourself gravitating towards – APIs or SDKs – Every implementation has a unique story to tell. What's yours?How Netflix Really Uses Java?
Netflix is predominantly a Java shop.
Every backend application (including internal apps, streaming, and movie production apps) at Netflix is a Java application.
However, the Java stack is not static and has gone through multiple iterations over the years.
Here are the details of those iterations:API Gateway
Netflix follows a microservices architecture. Every piece of functionality and data is owned by a microservice built using Java (initially version 8)BFFs with Groovy & RxJava
Using a single gateway for multiple clients was a problem for Netflix because each client (such as TV, mobile apps, or web browser) had subtle differences.
To handle this, Netflix used the Backend-for-Frontend (BFF) pattern. Zuul was moved to the role of a proxyGraphQL Federation
The Groovy and RxJava approach required more work from the UI developers in creating the Groovy scripts. Also, reactive programming is generally hard.
Big O Notation 101: The Secret to Writing Efficient Algorithms
From simple array operations to complex sorting algorithms, understanding the Big O Notation is critical for building high-performance software solutions.
O(1)
This is the constant time notation. The runtime remains steady regardless of input size. For example, accessing an element in an array by index and inserting/deleting an element in a hash table.O(n)
Linear time notation. The runtime grows in direct proportion to the input size. For example, finding the max or min element in an unsorted array.O(log n)
Logarithmic time notation. The runtime increases slowly as the input grows. For example, a binary search on a sorted array and operations on balanced binary search trees.O(n^2)
Quadratic time notation. The runtime grows exponentially with input size. For example, simple sorting algorithms like bubble sort, insertion sort, and selection sort.O(n^3)
Cubic time notation. The runtime escalates rapidly as the input size increases. For example, multiplying two dense matrices using the naive algorithm.O(n logn)
Linearithmic time notation. This is a blend of linear and logarithmic growth. For example, efficient sorting algorithms like merge sort, quick sort, and heap sortO(2^n)
Exponential time notation. The runtime doubles with each new input element. For example, recursive algorithms solve problems by dividing them into multiple subproblems.O(n!)
Factorial time notation. Runtime skyrockets with input size. For example, permutation-generation problems.O(sqrt(n))
Square root time notation. Runtime increases relative to the input’s square root. For example, searching within a range such as the Sieve of Eratosthenes for finding all primes up to n.
Over to you: What else will you add to better understand the Big O Notation?
The Ultimate Kubernetes Command Cheatsheet
Kubernetes is an open-source container orchestration platform. It automates the deployment, scaling, and management of containerized applications.
Initially developed by Google, Kubernetes is now maintained by CNCF (Cloud Native Computing Foundation).
This cheat sheet contains the most important Kubernetes commands for different purposes:Kubernetes Setup
General Cluster Management
Kubernetes Deployments
Kubernetes Pod Inspection
Troubleshooting and Configuration
Miscellaneous commands related to services, config maps, and managing ingresses.
Over to you: Which other Kubernetes command will you add to the list?
SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com.
Like
Comment
Restack
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 12:08 - 21 Dec 2024 -
Introducing McKinsey’s Davos Daily: a limited-edition email series
Share this email
New from McKinsey & Company
As next month’s World Economic Forum Annual Meeting approaches, we invite you to take advantage of our first-ever limited-run email series: McKinsey’s Davos Daily. This set of emails is available only for the duration of the conference and offers McKinsey expert insights to enhance your understanding of the year’s convening theme, which in 2025 is “Collaboration for the Intelligent Age.”
Each day you will receive a carefully curated roundup of McKinsey insights that delve into the core priorities of the conference. Series topics this year include AI transformation, geopolitics, space, 21st-century leadership, and more. Here’s what you can expect next month:•
Day 1: A forward-looking pre-conference bundle to set the stage
•
Days 2-7: A daily roundup of articles and reports on the conference’s main themes
•
Day 8: Takeaways and reflections
Sign up in one click to receive this free, limited-run series that dives deep into the themes of the Annual Meeting to help you parse today’s most critical global challenges.
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you are a registered member of mckinsey.com.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey & Company" <publishing@email.mckinsey.com> - 04:18 - 20 Dec 2024 -
The future of wellness
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
Prioritizing health. The business of health is thriving. McKinsey’s latest Future of Wellness research, which surveyed more than 5,000 consumers in China, the UK, and the US, finds that the global consumer wellness market is generating $1.8 trillion per year. For McKinsey partner Anna Pione, the fact that consumers are increasingly concentrating on their health is a given. However, “[w]hat surprised us and continues to surprise us every year is how much of a priority it is,” Pione remarks.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:19 - 20 Dec 2024 -
🎉 Cheers to a Bright New Year with Remote!
🎉 Cheers to a Bright New Year with Remote!
.png?width=204&upscale=true&name=Logo%20Horizontal%20(3).png)
Featured news
Happy Holidays from all of us at Remote 🎄
As the holiday season approaches, we want to take a moment to express our heartfelt gratitude for being a part of our community. 2024 was a big year for Remote, and we couldn't have done it without your support and trust.
Here's to a wonderful season and an even brighter new year ahead! Keep your eyes peeled for big things coming in 2025 👀
Featured blogs
.png?width=300&upscale=true&name=image(2).png)
Maximize Your End-of-Year Planning
With the year winding down, it's time to align your goals and set your team up for success. Discover actionable strategies to make the most of year-end planning and start the next year strong.
.png?width=300&upscale=true&name=image(3).png)
What’s Next for US Labor Policy?
From workplace trends to shifting regulations, staying ahead of labor policy changes is critical. Explore our forecast for what US labor policy changes could mean for businesses in 2024.
Upcoming events
Join Remote at HR Technologies France
Join Remote at HR technologies France at Paris Exhibition Center, Porte de Versailles, on January 29 and 30th for the first HRtech and EdTech tradeshow of the year. The Remote team will welcome you at booth L41 to discuss your international employment needs and discuss how to quickly recruit, pay, and manage teams anywhere in any currency using our unique HR platform.
On-demand webinar
Watch Now: From Complexity to Simplicity – Managing Global Benefits
Discover how our integrated platform streamlines global benefits and payroll. Watch our on-demand webinar, From Complexity to Simplicity, and transform the way you support your employees.
The HR platform for global businesses
Remote makes running global teams simple. Hire, manage, and pay anyone, anywhere. Book a personal consultation with an expert to discuss your unique HR needs.
You received this email because you are subscribed to
News & Offers from Remote Europe Holding B.V
Update your email preferences to choose the types of emails you receive.
Unsubscribe from all future emailsRemote Europe Holding B.V
Copyright © 2024 Remote Europe Holding B.V All rights reserved.
Kraijenhoffstraat 137A 1018RG Amsterdam The Netherlands
by "Remote" <hello@remote-comms.com> - 07:43 - 19 Dec 2024 -
The Sidecar Pattern Explained: Decoupling Operational Features
The Sidecar Pattern Explained: Decoupling Operational Features
Forwarded this email? Subscribe here for moreLatest articles
If you’re not a subscriber, here’s what you missed this month.
Database Performance Demystified: Essential Tips and Strategies
Mastering Modern Authentication: Cookies, Sessions, JWT, and PASETO
Stateless Architecture: The Key to Building Scalable and Resilient Systems
Distributed Caching: The Secret to High-Performance Applications
To receive all the full articles and support ByteByteGo, consider subscribing:
Design patterns are reusable solutions to common problems in software design. They provide a structured approach to solving architectural challenges without reinventing the wheel each time.
The sidecar pattern is one such design pattern that has gained prominence in modern software engineering.
At its core, the sidecar pattern pairs a secondary process or service (the "sidecar") with a primary application to handle complementary tasks. These tasks include logging, monitoring, proxying, security, or configuration management. The sidecar runs alongside the main application, sharing the same host or container, but remains logically and operationally independent.
The sidecar pattern can be compared to a motorcycle with a sidecar. The motorcycle (the primary service) is the main driver, responsible for the core functionality, like transporting a person. The sidecar (the auxiliary service) carries additional tools or passengers, assisting the main vehicle without interfering with its operation.
Similarly, in software systems, the sidecar extends the capabilities of the primary application without being tightly coupled to it.
In this article, we’ll learn about the sidecar pattern in detail and understand how it works. In the end, we will also look at its benefits and challenges that can help us make better decisions when using the pattern.
Continue reading this post for free in the Substack app
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:36 - 19 Dec 2024 -
Unleashing Composable Commerce: Everything Retailers Need to Know
Unleashing Composable Commerce: Everything Retailers Need to Know
Unleashing Composable Commerce: A Guide to Navigating Tech Challenges & Opportunities
Md Abul,
The demand for flexibility, scalability, and seamless customer experiences in retail has never been higher. Composable commerce is revolutionizing how brands build and scale their digital infrastructure, offering a modular approach that swiftly adapts to market shifts. But how can your business leverage this powerful technology without sacrificing profitability or stability in the customer experience?
Our latest guide, "Unleashing Composable Commerce: Navigating Opportunities and Challenges in Commerce Technology," breaks down this essential strategy, offering insights on building a future-proof digital commerce framework that’s agile, scalable, and customer-centric.
Here's what you’ll discover:
- The Fundamentals of Composable Commerce: Learn how modularity can be your competitive advantage.
- Real-World Success Stories: See how industry leaders like Samsung and Carrefour are innovating with composable frameworks.
- Practical Guidance for Decision-Makers: Evaluate vendors, implementation partners, and technologies tailored to your business.
Download the guide today to gain a comprehensive understanding of composable commerce and see how VTEX can help your business adapt faster, personalize experiences, and scale effortlessly—while driving measurable impact.
VTEX, 11 E 44th St 11th fl, New York, NY, USA 10017, New York, NY, USA
by "Alissa O’Carroll" <alissa.ocarroll@vtex.com> - 11:08 - 19 Dec 2024 -
Over 12 new features being showcased Feb 27th. Save your seat.
New Relic
Observability is evolving fast—join us at New Relic Now 2025, on February 27th (9am GMT / 10am CET), and see how Intelligent Observability drives developer velocity and business impact.
This free virtual event will showcase 12+ new features to:
- Elevate digital experiences: Discover predictive remediation that ensures seamless interactions with customers.
- Automate with AI: See how agentic AI handles both common and uncommon tasks, empowering developers.
- Limit costs: Identify unnecessary cloud and operational costs that you can avoid.
Save your seat now and see how Intelligent Observability redefines the landscape, and opens up unexpected possibilities for your business.
Register Now
by "New Relic" <emeamarketing@newrelic.com> - 05:31 - 19 Dec 2024 -
How to promote your business
How to promote your business
Hi there,
At Infoserve, we've been working with businesses on a similar trajectory to Your Telecoms Consultant for over 2 decades, helping them to promote their business and attract new customers.
We understand the focus required to deliver the highest levels of service, managing tight deadlines and keeping your customers happy.
Making sure you keep on top of everything can make it hard to find time to spend on Marketing, lead generation and brand awareness.
That's where we come in!
We are a full-service digital marketing agency based in Leeds. Here are just few ways we could help Your Telecoms Consultant:
Showcase Your Services - Showcase your expertise with a professionally designed website tailored to highlight your products, services, and industry knowledge, or enhance your existing site to maximise its impact and effectiveness We can include your range of services with case studies, reviews from satisfied customers and content to position you as thought leaders in your field.
Boost your visibility - Whether you market your services on localy, nationally or internationally, we’ll ensure your business is easily discoverable. With targeted SEO, we'll ensure your company stands out and attracts potential clients across all digital platforms.
Expand your reach - We can implement targeted digital marketing strategies aimed at increasing brand visibility and attracting new clients. By leveraging online advertising, content marketing, and strategic partnerships, we can extend your reach and attract business from new markets.
I'd love to schedule a brief intro call to get a better understanding of your specific needs and challenges to show you how we could help.
Or if you'd prefer, I can just send you some more information, examples of other businesses we've helped and some idea of costs.
Either way, just reply to this email and I'll get back to you as soon as possible.
Thank you,
Matt Budd
Digital Marketing Consultant
0800 089 0879
infoserve.com
Infoserve Ltd, Southside Aviation, Leeds Airport, LS19 7UG. Registered in England No. 04209122
We have sent this email to info@learn.odoo.com, having found your company details online, as we believe our digital marketing services may be of legitimate interest to your business. If you would no longer like to get emails from us you can stop them here.
by "Matt Budd" <matt@infoserve.uk> - 03:15 - 19 Dec 2024 -
These industries could transform the world economy by 2040
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
The industries of tomorrow. Which emergent industries could reshape the global economy? McKinsey Global Institute directors Kweilin Ellingrud and Chris Bradley and their colleagues have identified 18 arenas of competition—including AI software and services, cybersecurity, drugs for obesity, and future air mobility—that could generate $29 trillion to $48 trillion in revenues by 2040. These industries are characterized by high growth and high dynamism, meaning they are very competitive; the market share of companies within these sectors frequently changes hands.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 11:07 - 18 Dec 2024 -
It’s not enough to just build AI. Companies should also ensure it’s trustworthy AI.
Re:think
FRESH TAKES ON BIG IDEAS
ON AI TRUST
Why investing in AI trust pays offIt’s becoming increasingly clear that AI can bring superpowers to people in many roles and will change how they work. Soon, all software that companies build and buy will become AI software, which will be more flexible and adaptive than today’s rule-based systems. This will shrink time from opportunity discovery to action. Integration will get easier as agents dynamically bridge systems so that fragmented business processes come together in interconnected flows. The shift will unlock immense value for enterprises of all kinds, enhancing velocity, productivity, and innovation.
But to get there, trust is critical. Stakeholder expectations are rising as perceived risks also rise, and that’s leading to rapid experimentation with new regulation. But the regulatory patterns and postures around the world are far from settled. And while compliance is essential, building AI trust goes well beyond compliance.
The companies that derive the most value from AI will be those that create trust with their customers, employees, and stakeholders. Fundamentally, people must trust AI enough to hand over tasks. Enhanced evaluations, transparency, and explainability can all contribute—as well as flexible governance that puts principles into practice while encouraging innovation. Organizations can start with a principled approach to deciding not just what they can build but what they should build. These ethical decisions must be rooted in the values unique to each organization and the values of a society that places humans at the center of the AI ecosystem. This approach to building trust is responsible AI, or RAI. And when implemented well, RAI leads to real ROI.
Our research has found that the majority of large companies (72 percent) are implementing AI today in at least one business process, but just 18 percent have an RAI council with decision authority. Making AI governance work requires bringing people together who can offer complementary cross-functional perspectives.
Getting RAI right means implementing guidelines for all and operationalizing formal AI trust policies. This can create a psychologically safe environment where employees feel empowered to innovate boldly. But organizations also need technical guardrails to ensure their AI systems can be driven fast but safely. Across industries, we have repeatedly seen how the right AI guardrails can accelerate innovation, not impede it.
Trusted data is also key to AI innovation. AI builders should constantly ask themselves: “How do we create the right metadata to track the provenance of data sets, how they were collected, and how they therefore can be used?” Deploying a data operating guide ensures that AI builders have well-curated and documented data for responsible innovation.“The majority of large companies are implementing AI today in at least one business process, but just 18 percent have a responsible AI council with decision authority.”
In parallel with strengthening data governance and guardrails, leaders must tackle the hard work of building and implementing trusted AI deployment processes. I often advise leaders to take three steps to move fast.
First, educate the enterprise. Create a clear communication plan about what AI trust means for the entire organization and why everyone should be committed to it. Define how executives should lead in an AI era—and then roll out structured reskilling and upskilling programs. Even top technologists can be new to many aspects of RAI and will need to learn new human-centered AI engineering practices.
Second, invest in AI trust. Allocating the right resources requires treating it as an asset that is built up and not a cost of compliance to be “managed down” in the presence of regulatory scrutiny. This means creating a multiquarter road map for enhancing RAI maturity that embraces people, processes, and technology in a well-orchestrated action plan.
Third, engage cross-functional teams to deploy a strong governance platform, including registries for the software and the data resources that must be built or bought, as well as end-to-end workflows to ensure the right controls are in place. In addition, at the model and product level, AI tools should be continuously monitored by machine learning operations, or MLOps, for performance, quality, and risk. These “engine room” technologies are critical to ensuring that leaders “on the bridge” can make confident decisions.
Getting AI trust right is a shared responsibility between the organizations deploying AI and the platform providers, governments, international organizations, and standards bodies aiming to ensure that AI is safe and reliable. In this dynamic environment, academic researchers, open-source communities, and developers also play a big role in building AI that is more trustworthy, transparent, and explainable. CEOs and chief technology officers can do their part by getting their data houses in order, empowering their teams to innovate safely, and monitoring all their AI deployments for signs of bias or misinformation.Share Roger Roberts’ insights
ABOUT THIS AUTHOR
Roger Roberts is a partner in McKinsey’s Bay Area office.
MORE FROM THIS AUTHOR
UP NEXT
Dana Maor on leadership
Most CEOs still have a lot to learn about themselves when they step into the top job. This ongoing journey of self-discovery is critical for effective leadership.
Follow our thinking
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Quarterly alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Quarterly" <publishing@email.mckinsey.com> - 02:11 - 18 Dec 2024 -
What leading UK companies do to thrive
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
Positive economic indicators. Although the UK’s FTSE hasn’t performed as strongly as some other global stock exchanges, McKinsey senior partners Andrew Goodman and Tunde Olanrewaju note that the UK economy still has a lot of bright spots. In an episode of The McKinsey Podcast, they cite the amount of private capital flowing into the UK market as one such example. In fact, over the past 15 years, the UK has attracted about $1.8 trillion in private equity investments and nearly $1 trillion in foreign direct investment.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:48 - 18 Dec 2024 -
Dive deeper: Highlights from 2024
Intersection
Share this email
We’ll see you in 2025! And if you’re looking to spread some extra holiday cheer, forward this email to a friend or colleague so they can subscribe to the Five Fifty.
We’re commemorating our flagship publication’s milestone birthday with a year of themed issues, special interactive features, and timeless Quarterly classics newly released from the archives. It’s all available with a free digital Quarterly membership. Sign up here.
Share these insights
Did you enjoy this newsletter? Forward it to colleagues and friends so they can subscribe too. Was this issue forwarded to you? Sign up for it and sample our 40+ other free email subscriptions here.
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Quarterly Five Fifty alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Quarterly Five Fifty" <publishing@email.mckinsey.com> - 04:30 - 17 Dec 2024 -
How Statsig Streams 1 Trillion Events A Day
How Statsig Streams 1 Trillion Events A Day
Forwarded this email? Subscribe here for moreRead the acclaimed Designing Data-Intensive Applications book [O’REILLY] (Sponsored)
Discover new ways of thinking about your distributed data system challenges.
In this practical and comprehensive guide, Martin Kleppmann helps you navigate the diverse and fast-changing landscape of approaches to processing and storing data for data-intensive applications.
Peer under the hood of the systems you already use, and learn how to use and operate them more effectively
Make informed decisions by identifying the strengths and weaknesses of different approaches and technologies
Understand the distributed systems research upon which modern databases are built
Go behind the scenes of major online services and learn from their architectures
Get this 145-page ebook and start reading the book that’s become essential for anyone working with data systems.
The newsletter is a collaboration between ByteByteGo and members of the Statsig engineering team (Pablo Beltran and Brent Echols).
Statsig is a modern feature management, experimentation, and analytics platform that enables teams to improve their product velocity.
Over the past year, Statsig has seen a staggering 20X growth in its event volume. This growth has been driven by high-profile customers such as OpenAI, Atlassian, Flipkart, and Figma, who rely on Statsig’s platform for experimentation and analytics.
Statsig currently processes over a trillion events daily, which is a remarkable achievement for any organization, particularly one of their size as a startup.
Source: Statsig Technical Blog However, this rapid growth comes with immense challenges.
Statsig must not only scale its infrastructure to handle the sheer volume of data but also ensure that its systems remain reliable and maintain high uptime. It also needs to keep the costs in check to stay competitive.
In this post, we’ll look at Statsig’s streaming architecture, which helps it handle the event volume. We’ll also look at the cost-efficiency steps taken by Statsig’s team.
Statsig’s Streaming Architecture
On a high level, Statsig’s streaming architecture consists of 3 main components:
Request Recorder: This stage of Statsig’s pipeline is designed with one main priority: ensuring that no data is ever lost. The top goal here is to make sure that every piece of data gets captured and stored, no matter what.
Log Processing: In this stage, the raw data collected during request recording gets refined and prepared for use. This stage applies specific rules and logic to the raw data while also ensuring that the data is accurate.
Routing: This stage acts like a smart traffic controller for data. Once the data has been recorded and processed, this stage ensures it reaches the right destination. It allows the pipeline to handle varied customer requirements without creating separate systems for each use case.
See the diagram below for a quick overview of the architecture:
The Architectural Components
While the high-level view gives a broad look at Statsig’s pipeline architecture, let us also look at each component in more detail to gain a better understanding.
1 - Data Ingestion Layer
The data ingestion layer in Statsig's pipeline is the first and one of the most crucial stages of the system.
It is responsible for receiving, authenticating, organizing, and securely storing data in a way that prevents loss, even under challenging conditions.
The request recorder is a key functionality of the data ingestion layer, with its specific role being the first step in handling incoming data. However, the data ingestion layer includes the load balancer, authentication, rebatching, and persistence-related functionalities.
See the diagram below:
Here’s a simple breakdown of the various steps:
From Client SDKs to Load Balancer:
The process starts when data is sent from client SDKs. This data is routed through a load balancer.
The load balancer ensures that incoming requests are evenly distributed across servers to avoid overloading any single server. This helps maintain system stability and performance.
Service Endpoint & Authorization:
Once the data reaches the server, it is passed to a service endpoint where the system performs authentication. This step ensures that only valid and authorized data batches are processed.
After authentication, the data is added to an internal queue. This queue holds the data temporarily until it can be processed further, ensuring no data is lost even if the processing system experiences delays.
Rebatching for Efficiency:
At this stage, smaller data batches are combined into larger ones, a process called rebatching. Handling fewer, larger batches is more efficient than dealing with many small ones, both in terms of performance and cost.
Statsig uses a dynamic configuration system to determine the optimal size for these batches.
The rebatching process then writes the optimized data batches to Google Cloud Storage (GCS). Using GCS for this purpose is a cost-effective solution compared to directly streaming smaller batches through high-cost pipelines.
Persistence for Reliability:
After rebatching, the data is securely stored in a persistence layer, which serves as a semi-permanent storage system. This layer is designed with multiple fallback options, meaning it has backup mechanisms to store data in case the primary system encounters issues.
The processing units, known as pods, are equipped with sufficient memory to handle temporary failures. This ensures the system can continue operating for up to an hour during an outage without losing data or affecting performance.
These fallback mechanisms extend across different regions and cloud platforms, providing an extra layer of reliability to the system.
2 - Message Queue Layer
The Message Queue Layer is a critical stage in Statsig’s pipeline that manages how data flows between different components. This layer is designed to handle enormous volumes of data efficiently while keeping operational costs low.
See the diagram below:
As you can see, it consists of two main components:
The Pub/Sub Topic
Pub/Sub is a serverless messaging system that facilitates communication between different parts of the pipeline. Since it is serverless, there’s no need to worry about maintaining servers or managing complex deployments. This reduces overhead for the engineering team.
Pub/Sub receives metadata about the data stored in Google Cloud Storage (GCS). Instead of directly storing all the event data, it acts as a pointer system, referring downstream systems to the actual data stored in GCS.
GCS Bucket
Directly using Pub/Sub for all data storage would be prohibitively expensive. Therefore, Statsig offloads most of the data to GCS to reduce storage and operational costs.
The pipeline writes bulk data into GCS in compressed batches. Pub/Sub stores only the metadata (like file pointers) needed to locate this data in GCS. Downstream components can then use these pointers to retrieve the data when required.
The GCS Bucket stores the actual data in a compressed format, using Zstandard (ZSTD) compression for efficiency. For reference, Zstandard compression is highly efficient, providing better compression rates (around 95%) than other methods like zlib, with lower CPU usage. This ensures data is stored in a smaller footprint while maintaining high processing speeds.
3 - Business Logic Layer
The Business Logic Layer is where the heavy lifting happens in Statsig's pipeline.
This layer is designed to process data while ensuring accuracy and preparing it for final use by various downstream systems. It handles complex logic, customization, and data formatting.
See the diagram below that shows the various steps that happen in this layer:
Let’s look at each step in more detail:
Rebatching
This step combines smaller batches of incoming data into larger ones for processing. By handling larger batches, the system reduces the overhead of dealing with multiple small data chunks.
The system is designed with an “at least once” guarantee. This means that even if something goes wrong during processing, the data is not lost. It will be retried until successfully processed.
Stateful Processing To Remove Redundancy
This step focuses on deduplication, which involves filtering out repeated or redundant data. For instance, if the same event gets recorded multiple times, this step ensures only one instance is kept.
To achieve this, the system uses caching solutions like Memcached. Memcached provides quick access to previously processed data, enabling the system to identify duplicates efficiently.
Ultimately, deduplication reduces unnecessary processing.
Business Logic Plugins
This layer allows different teams within Statsig to insert custom business logic tailored to their specific needs. For example, one team might add specific tags or attributes to the data, while another might modify event structures for a particular customer.
By using plugins, the system can support diverse use cases without requiring a separate pipeline for each customer. This makes the pipeline both scalable and versatile.
Writer
Once the data has been cleaned, transformed, and customized, the Writer finalizes it by writing it to the appropriate destination.
This could be a database, a data warehouse, or an analytics tool, depending on where the data is needed.
4 - Routing and Integration Layer
The routing and integration layer in Statsig's pipeline is responsible for directing processed data to its final destination.
See the diagram below:
Let’s look at each branch in more detail:
Warehouse Router
The Warehouse Router is responsible for deciding where the data should go based on factors like customer preferences, event types, and priority. It dynamically routes data to various destinations such as BigQuery or other data warehouses.
Here’s how it works:
For data that does not require immediate access (latency-insensitive), the router uses load jobs. These batch jobs process and upload data in bulk, significantly reducing costs compared to real-time streaming.
However, for critical, time-sensitive data, the router leverages the storage write API to deliver data in real-time, ensuring low latency but at a higher cost.
The Warehouse Router guarantees efficient resource utilization by distinguishing between latency-sensitive and latency-insensitive data. It saves costs without compromising on performance for urgent tasks.
The Side Effects Service
This service handles external integrations triggered by specific events in the pipeline. For example:
Sending data to third-party systems via webhooks.
Triggering actions or workflows in external platforms based on predefined rules.
It supports any kind of event-level trigger, making it highly customizable for customer-specific workflows.
The Real-Time Event Stream
This service is designed for situations where data needs to be accessed almost instantaneously. For example:
Real-time dashboards that display user activity.
Monitoring tools that require immediate updates.
It uses Redis, a fast in-memory data store, to cache and retrieve data in real-time so that customers querying the data experience minimal delays.
The Shadow Pipeline
The Shadow Pipeline is an important testing feature in Statsig’s event streaming system.
It acts as a safety net to ensure that any updates or changes to the system don’t disrupt its ability to process over a trillion events a day.
Here’s a closer look at how the shadow pipeline works:
Parallel Testing Environment: The shadow pipeline runs alongside the main production pipeline but doesn’t affect live data or customer systems. It mirrors the production setup, processing the same kinds of data with a new version of the software (release candidate). This allows engineers to test updates in a controlled environment without risking disruptions to the main pipeline.
Simulating Real Traffic: To test the shadow pipeline’s performance, a couple of million events are sent through it in a large burst. This mimics the high-traffic conditions the system handles daily, providing a realistic scenario for evaluation.
Comparing Results: The outputs of the shadow pipeline are carefully compared with those from the production pipeline. Automated checks look for discrepancies, such as missing or incorrect data, and alert engineers if anything falls outside expected tolerances.
Stress Testing Infrastructure: The traffic burst also tests the system’s ability to handle sudden spikes in data volume. It evaluates features like horizontal pod autoscaling (HPA) that automatically adjusts computing resources to match demand.
Statsig’s Cost Optimization Strategies
Statsig employed multiple cost optimization strategies to handle the challenge of processing over a trillion events daily while keeping operational expenses as low as possible.
These strategies involve a mix of technical solutions, infrastructure choices, and design decisions.
Let’s break down each key effort in more detail:
GCS Upload via Pub/Sub
Instead of sending all event data directly into Pub/Sub, Statsig writes the majority of the data to Google Cloud Storage (GCS) in a compressed format.
Using GCS is significantly cheaper than relying solely on Pub/Sub for storing large amounts of data. It helps reduce costs while maintaining scalability. Pub/Sub is used only to pass file pointers (metadata) that direct downstream systems to retrieve the data from GCS.
Async Workloads on Spot Nodes
Statsig runs non-time-sensitive tasks (asynchronous workloads) on spot nodes, which are temporary virtual machines offered at a lower price.
Leveraging spot nodes reduces VM costs without compromising performance for less urgent processes. Also, since these workloads don’t require constant uptime, occasional interruptions don’t impact the system’s overall functionality.
Deduplication with Memcache
A large portion of incoming events may include duplicates, which add unnecessary processing overhead. Deduplication is a key feature that saves processing resources and ensures downstream systems only handle unique data.
To handle deduplication, Statsig uses Memcache to identify and discard duplicate events early in the pipeline.
Zstandard (zstd) Compression
Statsig switched from using zlib compression to ZSTD, a more efficient compression algorithm.
ZSTD achieves better compression rates (around 95%) while using less CPU power, compared to zlib’s 90% compression. This improvement reduced storage requirements and processing power.
Batching Efficiency via CPU Optimization
Statsig also adjusted the CPU allocation for its request recorder (from 2 CPU to 12 CPU), enabling it to handle larger batches of data more efficiently.
This is because larger batches reduce the number of write operations to storage systems, improving cost efficiency while maintaining high throughput.
Load Jobs vs. Live Streaming
For data that doesn’t need to be processed immediately, Statsig uses load jobs to process and upload data in bulk, which is much cheaper.
On the other hand, for time-sensitive data, they use the storage write API, which provides low-latency delivery but at a higher cost.
Differentiating between these two types of data saves money while meeting customer requirements for both real-time and batch processing.
Optimized CPU and Memory Utilization
Statsig tunes CPU and memory usage based on actual host utilization rather than pod utilization.
Also, pods are configured without strict usage limits, allowing them to make full use of available resources when needed. This prevents underutilization of expensive hardware resources and maximizes cost-effectiveness.
Aggressive Host-Level Resource Stacking
Statsig stacks multiple pods onto a single host aggressively to use every bit of available CPU and memory.
By fine-tuning flow control and concurrency settings, they prevent resource contention while maintaining high performance. This approach helps achieve cost efficiency at the host level by reducing the number of machines needed.
Conclusion
Statsig’s journey to streaming over a trillion events daily shows how a company can achieve massive scale without compromising efficiency through innovative engineering.
By designing a robust data pipeline with key components like a reliable ingestion layer, scalable message queues, and cost-optimized routing and integration layers, Statsig has built an infrastructure capable of supporting rapid growth while maintaining high reliability and performance. Also, leveraging features and tools like Pub/Sub, GCS, and advanced compression techniques, the platform balances the challenges of low latency, data integrity, and cost-effectiveness.
A key differentiator for Statsig is its approach to cost optimization and scalability, achieved through strategies such as using spot nodes, implementing deduplication, and differentiating latency-sensitive from latency-insensitive workloads. These efforts not only ensure the system's resilience but also allow them to offer their platform at competitive prices to a wide range of customers.
Reference:
SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com.
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:36 - 17 Dec 2024
