Archives
-
How Uber Uses Integrated Redis Cache to Serve 40M Reads/Second?
How Uber Uses Integrated Redis Cache to Serve 40M Reads/Second?
Forwarded this email? Subscribe here for more80% automated E2E test coverage in 4 months (Sponsored)
Were you aware that despite allocating 25%+ of budgets to QA, 2/3rds of companies still have less than 50% end-to-end test coverage? This means over half of every app is exposed to quality issues.
QA Wolf solves this coverage problem by being the first QA solution to get web apps to 80% automated E2E test coverage in weeks instead of years.
How's QA Wolf different?
We don't charge hourly.
We guarantee results.
We provide all of the tooling and (parallel run) infrastructure needed to run a 15-minute QA cycle.
→ Learn more about QA Wolf today!
In 2020, Uber launched their in-house, distributed database named Docstore.
It was built on top of MySQL and was capable of storing tens of petabytes of data while serving tens of millions of requests per second.
Over the years, Docstore was adopted by all business verticals at Uber for building their services. Most of these applications required low latency, higher performance, and scalability from the database, while also supporting higher workloads.
Challenges with Low Latency Database Reads
Every database faces a challenge when dealing with applications that need low-latency read access with a highly scalable design.
Some of these challenges are as follows:
Data retrieval speed from a disk has a threshold. Beyond that, you cannot squeeze out more performance by optimizing an application’s data model and queries to improve latency.
Vertical scaling can take you far but assigning more resources by upgrading to better hosts has limitations. Ultimately, the database engine turns into a bottleneck.
Horizontal scaling by splitting your database into multiple partitions is a promising approach. However, it also gets operationally more complex over time and doesn’t eliminate issues such as hot partitions.
Both vertical and horizontal scaling strategies are costly in the long term. For reference, costs get multiplied 6X to handle three stateful nodes across two regions.
To overcome these challenges, microservices typically make use of caching.
Uber started offering Redis as a distributed caching solution for the various teams. They followed the typical caching design pattern where the service writes to the database and cache while serving reads directly from the cache.
The below diagram shows this pattern:
However, the normal caching pattern where a service takes care of managing the cache has a few problems at the scale of Uber.
Each team has to manage its own Redis cache cluster.
The cache invalidation logic is duplicated across multiple microservices and there are chances of deviation.
Services must maintain cache replication to stay hot in case of region failovers.
The main point is that every team that needed caching had to spend a large amount of effort to build and maintain a custom caching solution.
To avoid this, Uber decided to build an integrated caching solution known as CacheFront.
Latest articles
If you’re not a paid subscriber, here’s what you missed this month.
To receive all the full articles and support ByteByteGo, consider subscribing:
Design Goals with CacheFront
While building CacheFront, Uber had a few important design goals in mind:
Reduce the need for vertical or horizontal scaling to support low-latency read requests
Improve the P50 and P99 latencies and stabilize latency spikes
Bring down the resource allocation for the database engine layer
Replace the plethora of custom-built caching solutions created by individual teams for their needs. Instead, move the ownership for maintaining and supporting Redis to the Docstore team.
Make caching transparent from the point of view of the service, allowing teams to just focus on the business logic
Decouple the caching solution from Docstore’s partitioning schema to avoid hot partitions
Support horizontal scalability of the caching layer with cheap host machines and make the entire operation cost-effective
High-Level Architecture with CacheFront
To support these design goals, Uber created its integrated caching solution tied to Docstore.
The below diagram shows the high-level architecture of Docstore along with CacheFront:
As you can see, Docstore’s query engine acts as the entry point for services and is responsible for serving reads and writes to clients.
Therefore, it was the ideal place to integrate the caching layer, allowing the cache to be decoupled from the disk-based storage. The query engine implemented an interface to Redis to store cached data along with mechanisms to invalidate the cache entries.
Handing Cached Reads
CacheFront uses a cache aside or look aside strategy when it comes to reads.
The below steps explain how it works:
The query engine layer receives a read request for one or more rows
The query engine tries to get the rows from Redis and streams the response to the users
Next, it retrieves the remaining rows from the database (if needed)
The query engine asynchronously populates Redis with the rows that are not found in the cache.
Stream the remaining rows to the users.
Refer to the below diagram that explains the process more clearly:
Cache Invalidation with CDC
As you may have heard a million times by now, cache invalidation is one of the two hard things in Computer Science.
One of the simplest cache invalidation strategies is configuring a TTL (Time-to-Live) and letting the cache entries expire once they cross the TTL. While this can work for many cases, most users expect changes to be reflected faster than the TTL. However, lowering the default TTL to a very small value can sink the cache hit rate and reduce its effectiveness.
To make cache invalidation more relevant, Uber leveraged Flux, Docstore’s change data capture and streaming service. Flux works by tailing the MySQL binlog events for each database cluster and publishes the events to a list of consumers. It powers replication, materialized views, data lake ingestions, and data consistency validations among various nodes.
For cache invalidation, a new consumer was created that subscribes to the data events and invalidates/upserts the new rows in Redis.
The below diagram shows the read and write paths with cache invalidation.
There were some key advantages of this approach:
They could make the cache consistent with the database within seconds of the database change as opposed to minutes (depending on the TTL).
Also, using binlogs made sure that uncommitted transactions couldn’t pollute the cache.
However, there were also a couple of issues that had to be ironed out.
1 - Deduplicating Cache Writes
Since writes happen to the cache simultaneously between the read and write paths, it was possible to write a stale row to the cache by overwriting the newest value.
To prevent this, they deduplicated writes based on the timestamp of the row set in MySQL.
This timestamp served as a version number and was parsed from the encoded row value in Redis using the EVAL command.
2 - Stronger Consistency Requirement
Even though cache invalidation using CDC with Flux was faster than relying on TTL, it still provided eventual consistency.
However, some use cases required stronger consistency guarantees such as the reading-own-writes guarantee.
For such cases, they created a dedicated API to the query engine that allowed users to explicitly invalidate the cached rows right after the corresponding writes were completed. By doing so, they didn’t have to wait for the CDC process to complete for the cache to become consistent.
Scale and Resiliency with CacheFront
The basic requirement of CacheFront was ready once they started supporting reads and cache invalidation.
However, Uber also wanted this solution to work at their scale. They also had critical resiliency needs around the entire platform.
To achieve scale and resiliency with CacheFront, they utilized multiple strategies.
Compare cache
Measurements are the key to proving that a system works as expected. The same was the case with CacheFront.
They added a special mode to CacheFront that shadows read requests to cache, allowing them to run a comparison between the data in the cache and the database to verify that both were in sync. Any mismatches such as stale rows are logged as metrics for further analysis.
The below diagram shows a high-level design of the Compare cache system.
Based on the results from this system, Uber found that the cache was 99.99% consistent.
Cache Warming
In a multi-region environment, a cache is only effective if it is always warm. If that’s not the case, a region fail-over can result in cache misses and drastically increase the number of requests to the database.
Since a Docstore instance spawned in two different geographical regions with an active-active deployment, a cold cache meant that you couldn’t scale down the storage engine to save costs since there was a high chance of heavy database load in the case of failover.
To solve this problem, the Uber engineering team used cross-region Redis replication.
However, Docstore also had its own cross-region replication. Since operating both replication setups simultaneously could result in consistent data between the cache and database, they enhanced the Redis cross-region replication by adding a new cache warming mode.
Here’s how the cache warming mode works:
They tail the Redis write stream to replicate keys to the remote region
In the remote region, they don’t directly update the cache. Instead, they issue read requests to the query engine layer for the replicated keys
In the case of a cache miss, the query engine layer reads the data from the database and writes it to the cache. The response stream is discarded.
The below diagram shows this approach in detail:
Replicating keys instead of values makes sure that the data in the cache is consistent with the database in its respective region. Also, it ensures that the same set of cached rows is present in both regions, thereby keeping the cache warm in case of a failover.
Sharding
Some large customers of Docstore within Uber can generate a very large number of read-write requests. It was challenging to cache all of it within a single Redis cluster that’s limited by the maximum number of nodes.
To mitigate this, they allowed a single Docstore instance to map to multiple Redis clusters. This helped avoid a massive surge of requests to the database in case multiple nodes in a single Redis cluster go down.
However, there was still a case where a single Redis cluster going down may create a hot shard on the database. To prevent this, they sharded the Redis cluster using a scheme that was different from the database sharding scheme. This makes sure that the load from a single Redis cluster going down is distributed between multiple database shards.
The below diagram explains this scenario in more detail.
Circuit Breaker
When a Redis node goes down, a get/set request to that node generates an unnecessary latency penalty.
To avoid this penalty, Uber implemented a sliding window circuit breaker to short-circuit such requests. They count the number of errors on each node for a particular bucket of time and compute the number of errors within the sliding window’s width.
See the below diagram to understand the sliding window approach:
The circuit breaker is configured to short-circuit a fraction of the requests to a node based on the error count. Once the threshold is hit, the circuit breaker is tripped and no more requests can be made to the node until the sliding window passes.
Results and Conclusion
Uber’s project of implementing an integrated Redis cache with Docstore was quite successful.
They created a transparent caching solution that was scalable and managed to improve latency, reduce load, and bring down costs.
Here are some stats that show the results:
The P75 latency went down by 75% and the P99.9 latency went down by over 67% while also limiting latency spikes
Cache invalidation using flux and compare cache provided a cache consistency of 99.9%.
Sharding and cache warming made the setup scalable and fault-tolerant, allowing a use case with over 6M reads per second with a 99% cache hit rate to successfully failover to a remote region.
Costs were down significantly as the same use case of 6M reads per second approximately required 60K CPU cores for the storage engine. With CacheFront, they achieved the same results with just 3K Redis cores.
At present, CacheFront supports over 40M requests per second in production and the number is growing every day.
References:
SPONSOR US
Get your product in front of more than 500,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing hi@bytebytego.com.
Like
Comment
Restack
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:35 - 26 Mar 2024 -
Elevate your log storage strategy with New Relic live archives
New Relic
Say goodbye to tradeoffs between balancing the cost and accessibility of logs. Unlike hot storage solutions, New Relic live archives ensures long-term log retention without breaking the bank. You’ll still have ease of use without the complexities of cold storage maintenance.
Live archive key features:
- Regulatory compliance: Store logs in an active, hydrated, and enriched state for up to seven years, meeting regulatory requirements effortlessly.
- Streamlined analysis: Empower your engineers to analyse logs swiftly and effectively, eliminating the hassles of rehydration, reloading, and re-indexing.
- Cost-effective: Priced at just a quarter of other long-term storage solutions, it provides unbeatable value for your investment.
Ready to revolutionize your log storage strategy? Download the datasheet to learn more about New Relic live archives and how it can elevate your observability game.
Learn More Need help? Let's get in touch.
This email is sent from an account used for sending messages only. Please do not reply to this email to contact us—we will not get your response.
This email was sent to info@learn.odoo.com Update your email preferences.
For information about our privacy practices, see our Privacy Policy.
Need to contact New Relic? You can chat or call us at +44 20 3859 9190.
Strand Bridge House, 138-142 Strand, London WC2R 1HH
© 2024 New Relic, Inc. All rights reserved. New Relic logo are trademarks of New Relic, Inc
by "New Relic" <emeamarketing@newrelic.com> - 10:13 - 26 Mar 2024 -
What do the world’s greatest CEOs have in common?
On Point
by "Only McKinsey" <publishing@email.mckinsey.com> - 01:35 - 26 Mar 2024 -
Generative AI: A leader’s guide to capturing value fast
Share this email
Brought to you by Liz Hilton Segel, chief client officer and managing partner, global industry practices, & Homayoun Hatami, managing partner, global client capabilities
By now, the potential economic payoff of generative AI (gen AI) is almost undisputed: our research shows that the technology could add trillions of dollars in value to the global economy. But despite its beneficial impact on a range of functions and industries, gen AI poses considerable challenges and risks that leaders may need to prepare for even as they pilot the technology. Here’s a roundup of our latest thinking on gen AI and its implications—positive and negative—for organizations.
The buzz around gen AI has been increasing, but so has concern about its risks—and that can be a problem for leaders who wish to take advantage of its potential for immediate positive impact. In a recent survey, we found that 63 percent of respondents rated gen AI implementation as a high priority, yet 91 percent did not feel ready to manage the process in a responsible manner. We suggest that leaders address four primary sources of “inbound” risk—threats likely to affect organizations regardless of whether they deploy gen AI. For example, it’s important to establish a governance structure that includes expertise, oversight, and training for end users. “Organizations should develop a set of guiding principles agreed on by the executive team and the board that will guide AI adoption and serve as a guardrail for acceptable use cases,” note McKinsey senior partners Ida Kristensen and Lareina Yee and their colleagues.
That’s McKinsey experts on what it takes for a worker to be truly prepared for gen AI on a day-to-day basis. A successful AI team member may need to have design and collaboration skills, contextual understanding to ensure that the most high-quality answers are generated, and forensic skills to figure out the reasons for common breakdowns. “We see a rapid market emerging for people who have learned these skills over the past year,” observe senior partners Eric Lamarre, Alex Singla, Alexander Sukharevsky, and Rodney Zemmel. “If your company is just starting its gen AI journey, you could consider hiring two or three senior engineers who have built a gen AI shaper product for their companies.”
“What we know from our product development research is that, oftentimes, the problem is you close the funnel too quickly,” says McKinsey partner Michael Chui in an episode of The McKinsey Podcast on how companies can use gen AI to gain competitive advantage. “You don’t continue to look at multiple alternatives.” Chui adds that gen AI enables organizations to consider a variety of creative options, as well as enabling individual knowledge workers to increase their productivity. “You can end up with better products at the end of the day, create whole new categories, create new drugs, create systems that are more sustainable and produce less carbon emissions,” he says. “All the kinds of things we want to do with R&D, we can create a superpower by using these technologies.”
In his play Henry VI, Shakespeare may have joked about doing away with lawyers, but today, gen AI isn’t here to replace the legal profession but rather could help its members thrive. From automating repetitive tasks to helping create business strategy, the powerful new technology can potentially transform the daily work of lawyers, according to McKinsey associate general counsel Ilona Logvinova. “I see this as a tremendously exciting time in our profession,” she says. “This newly emerging automation stack can free up real time that lawyers use on heavily time-consuming work with lower payoff.” For example, gen AI can search for and compile defined terms, find applicable precedents, and take a first cut at editing forms and documents. “With legal tech as the copilot, we have an opportunity for automation and acceleration in the way we work,” notes Logvinova.
Lead by capturing gen AI’s value.
– Edited by Rama Ramaswami, senior editor, New York
Share these insights
Did you enjoy this newsletter? Forward it to colleagues and friends so they can subscribe too. Was this issue forwarded to you? Sign up for it and sample our 40+ other free email subscriptions here.
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Leading Off newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Leading Off" <publishing@email.mckinsey.com> - 04:09 - 25 Mar 2024 -
How are manufacturing leaders using digital twins?
On Point
Brought to you by Liz Hilton Segel, chief client officer and managing partner, global industry practices, & Homayoun Hatami, managing partner, global client capabilities
•
Factory of the future. Global manufacturers face intense pressure. Amid challenges such as constraints on material and labor, more industry leaders are turning to factory digital twins—real-time virtual representations of the factory. Their use supports faster, smarter, and more cost-effective decision making. In a McKinsey survey of 75 senior leaders in industrials, 86% say that a digital twin is applicable to their organization, McKinsey partner Kevin Goering and colleagues share.
•
Real-time decisions. Factory digital twins simulate outcomes from real-time factory conditions, enabling what-if analyses across production scenarios. Manufacturing leaders already use digital twins to increase efficiency and reduce costs. One industrial company recently employed a factory digital twin to redesign its production schedule. By reducing an assembly plant’s overtime requirements, the company saved 5 to 7% in costs each month. Consider how to get started with digital twins, and visit McKinsey Digital to learn more about AI and tech in business.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this newsletter because you subscribed to the Only McKinsey newsletter, formerly called On Point.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey" <publishing@email.mckinsey.com> - 01:48 - 25 Mar 2024 -
Top 10 reports this quarter
McKinsey&Company
Share this email
Our top ten reports this quarter look at the geometry of global trade, generative AI in the pharmaceutical industry, and more. At No. 1, McKinsey’s Kweilin Ellingrud, Lucy Pérez, and Valentina Sartori, in collaboration with the World Economic Forum, explore how closing the women’s health gap could provide a $1 trillion boost to the global economy.
Share these insights
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you are a registered member of the Top Ten Most Popular newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Top Ten" <publishing@email.mckinsey.com> - 03:09 - 24 Mar 2024 -
The week in charts
The Week in Charts
Share this email
Our McKinsey Chart of the Day series offers a daily chart that helps explain a changing world—as we strive toward sustainable and inclusive growth. In case you missed them, this week’s graphics explored AI in manufacturing, global M&A activity, shoppers’ shift to online channels, corporate deposits in transaction banking, and mobility’s tech trends.
Share these insights
Did you enjoy this newsletter? Forward it to colleagues and friends so they can subscribe too. Was this issue forwarded to you? Sign up for it and sample our 40+ other free email subscriptions here.
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to The Week in Charts newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Week in Charts" <publishing@email.mckinsey.com> - 03:50 - 23 Mar 2024 -
EP104: How do Search Engines Work?
EP104: How do Search Engines Work?
Forwarded this email? Subscribe here for moreThis week’s system design refresher:
System Design: Why is Kafka so Popular? (Youtube video)
How do Search Engines Work?
Top 9 Website Performance Metrics You Cannot Ignore
Top 6 Data Management Patterns
Comparing Different API Clients
SPONSOR US
😘 Kiss Bugs Goodbye! Rapid Software Delivery with Unparalleled Test Coverage (Sponsored)
For 1/3rd the cost of a full-time hire, QA Wolf is the first solution that gets web apps to 80% automated E2E test coverage in just 4 months.
They take care of automating, maintaining, and running your test suite in 100% parallel, offering unlimited runs with zero flakes guaranteed. This means your developers can ship faster and with greater confidence than ever before.
Curious about the results? Check out their case studies showcasing customers who have saved $200k+/year in QA engineering and infrastructure costs.
Learn more about QA Wolf today!
System Design: Why is Kafka so Popular?
How do Search Engines Work?
The diagram below shows a high-level walk-through of a search engine.
▶️ Step 1 - Crawling
Web Crawlers scan the internet for web pages. They follow the URL links from one page to another and store URLs in the URL store. The crawlers discover new content, including web pages, images, videos, and files.
▶️ Step 2 - Indexing
Once a web page is crawled, the search engine parses the page and indexes the content found on the page in a database. The content is analyzed and categorized. For example, keywords, site quality, content freshness, and many other factors are assessed to understand what the page is about.
▶️ Step 3 - Ranking
Search engines use complex algorithms to determine the order of search results. These algorithms consider various factors, including keywords, pages' relevance, content quality, user engagement, page load speed, and many others. Some search engines also personalize results based on the user's past search history, location, device, and other personal factors.
▶️ Step 4 - Querying
When a user performs a search, the search engine sifts through its index to provide the most relevant results.Latest articles
If you’re not a paid subscriber, here’s what you missed this month.
To receive all the full articles and support ByteByteGo, consider subscribing:
Top 9 Website Performance Metrics You Cannot Ignore
Load Time: This is the time taken by the web browser to download and display the webpage. It’s measured in milliseconds.
Time to First Byte (TTFB): It’s the time taken by the browser to receive the first byte of data from the web server. TTFB is crucial because it indicates the general ability of the server to handle traffic.
Request Count: The number of HTTP requests a browser has to make to fully load the page. The lower this count, the faster a website will feel to the user.
DOMContentLoaded (DCL): This is the time it takes for the full HTML code of a webpage to be loaded. The faster this happens, the faster users can see useful functionality. This time doesn’t include loading CSS and other assets
Time to above-the-fold load: “Above the fold” is the area of a webpage that fits in a browser window without a user having to scroll down. This is the content that is first seen by the user and often dictates whether they’ll continue reading the webpage.
First Contentful Paint (FCP): This is the time at which content first begins to be “painted” by the browser. It can be a text, image, or even background color.
Page Size: This is the total file size of all content and assets that appear on the page. Over the last several years, the page size of websites has been growing constantly. The bigger the size of a webpage, the longer it will take to load
Round Trip Time (RTT): This is the amount of time a round trip takes. A round trip constitutes a request traveling from the browser to the origin server and the response from the server going to the browser. Reducing RTT is one of the key approaches to improving a website’s performance.
Render Blocking Resources: Some resources block other parts of the page from being loaded. It’s important to track the number of such resources. The more render-blocking resources a webpage has, the greater the delay for the browser to load the page.
Over to you - What other website performance metrics do you track?
Reference: Cloudflare DocsHow do we manage data? Here are top 6 data management patterns
Cache Aside
When an application needs to access data, it first checks the cache. If the data is not present (a cache miss), it fetches the data from the data store, stores it in the cache, and then returns the data to the user. This pattern is particularly useful for scenarios where data is read frequently but updated less often.Materialized View
A Materialized View is a database object that contains the results of a query. It is physically stored, meaning the data is actually computed and stored on disk, as opposed to being dynamically generated upon each request. This can significantly speed up query times for complex calculations or aggregations that would otherwise need to be computed on the fly. Materialized views are especially beneficial in data warehousing and business intelligence scenarios where query performance is critical.CQRS
CQRS is an architectural pattern that separates the models for reading and writing data. This means that the data structures used for querying data (reads) are separated from the structures used for updating data (writes). This separation allows for optimization of each operation independently, improving performance, scalability, and security. CQRS can be particularly useful in complex systems where the read and write operations have very different requirements.Event Sourcing
Event Sourcing is a pattern where changes to the application state are stored as a sequence of events. Instead of storing just the current state of data in a domain, Event Sourcing stores a log of all the changes (events) that have occurred over time. This allows the application to reconstruct past states and provides an audit trail of changes. Event Sourcing is beneficial in scenarios requiring complex business transactions, auditability, and the ability to rollback or replay events.Index Table
The Index Table pattern involves creating additional tables in a database that are optimized for specific query operations. These tables act as secondary indexes and are designed to speed up the retrieval of data without requiring a full scan of the primary data store. Index tables are particularly useful in scenarios with large datasets and where certain queries are performed frequently.Sharding
Sharding is a data partitioning pattern where data is divided into smaller, more manageable pieces, or "shards", each of which can be stored on different database servers. This pattern is used to distribute the data across multiple machines to improve scalability and performance. Sharding is particularly effective in high-volume applications, as it allows for horizontal scaling, spreading the load across multiple servers to handle more users and transactions.
Reference: Data Management Patterns by Microsoft
Comparing Different API Clients
Postman is a widely used API lifecycle platform. It emerges as a comprehensive and versatile API client suitable for enterprise-level development. Its support for a wide range of protocols, robust feature set, and strong performance make it a top choice for complex projects. With an intuitive design, collaboration features, and a large community, Postman excels in scenarios requiring extensive functionality and community support.
Insomnia is a powerful API client with extensive features and being completely open-source makes it a good choice for developers seeking flexibility and continuous growth. Insomnia is suited for those who value an open-source environment and an active community.
ReadyAPI, with its simplicity and focus on smaller projects, is an ideal choice for scenarios where a lightweight and responsive tool is preferred. It provides essential features, making it suitable for projects with less complexity. However, it may not be the best fit for larger, more intricate endeavors that require extensive functionality.
ThunderClient, a VS Code plugin, is free and user-friendly, catering to developers who prefer an integrated testing environment. However, it lacks extensive features and community support, crucial for larger or complex projects, rendering it more appropriate for smaller teams with simpler requirements. Additionally, its reliance on Visual Studio Code may restrict its appeal to users who prefer alternative development environments. Experienced users accustomed to feature-rich tools may encounter a learning curve and might find ThunderClient lacking in certain functionalities.
Hoppscotch, a free and open-source tool, focuses on functionality over design, offering a lightweight web version with support for various protocols. While it lacks extensive documentation and community support, it provides a cost-effective solution for developers seeking simplicity.
Over to you: Which API client do you prefer?SPONSOR US
Get your product in front of more than 500,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing hi@bytebytego.com.
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:35 - 23 Mar 2024 -
Gen AI: Slow your roll
The Shortlist
Share this email
Curated by Liz Hilton Segel, chief client officer and managing partner, global industry practices, & Homayoun Hatami, managing partner, global client capabilities
New and powerful technologies like generative AI are always a thrill. But what comes next? It’s never just about the tech: companies need much more—like strategy, talent, risk management—to get real value out of their shiny new objects. That’s especially true when building a new tech-based business. In this edition of the CEO Shortlist, we highlight four articles that preach the gospel of caution. We hope you enjoy the read.
—Liz and Homayoun
Trend with caution. Lots of IT executives have likely fielded excited calls in recent months from their C-suite colleagues, who’ve read somewhere (maybe here!) that gen AI is the next big thing and want to implement it right away. But chasing trends can be a fool’s errand. The smart tech exec relies on a clear set of parameters for evaluating new trends—and responding appropriately.
Walk before you run. Learn how with “False friends or good ends? The CIO’s four-point guide to navigating technology trends,” a new article by Oliver Bossert, Klaas Ole Kürtz, and Gérard Richter.It’s not all sunshine and lollipops. Gen AI can be fun. But new risks are cropping up every day, with more on the way. Hallucinations are one thing. Deepfakes, intellectual property theft, and gen-AI-powered malware are another. Are you ready for the dark side of gen AI?
Go on defense with “Implementing generative AI with speed and safety,” by Oliver Bevan, Michael Chui, Ida Kristensen, Brittany Presten, and Lareina Yee.Nobody ever created a unicorn by having another meeting. But as companies rush to take new businesses to market, risk management and cybersecurity tend to be afterthoughts. That plays right into the hands of hackers and bad actors.
It can happen to you. Build your defenses early with “New business building: Six cybersecurity and digital beliefs that can create risk,” by Justin Greis, Ari Libarikian, Patrick Rinski, Joy Smith, and Marc Sorel.Gen AI isn’t a gold rush. After a flurry of initial excitement, executives are starting to realize that capturing value through gen AI is going to be a long game. Employing gen AI effectively in the long term requires what we call organizational surgery: a broad set of changes to the way work actually gets done.
Scrub in with “A generative AI reset: Rewiring to turn potential into value in 2024,” by Eric Lamarre, Alex Singla, Alexander Sukharevsky, and Rodney Zemmel.We hope you find these ideas inspiring and helpful. See you next time with four more McKinsey ideas for the CEO and others in the C-suite.
Share these insights
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to The CEO Shortlist newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey CEO Shortlist" <publishing@email.mckinsey.com> - 04:23 - 22 Mar 2024 -
Why does empathy matter at work?
On Point
Brought to you by Liz Hilton Segel, chief client officer and managing partner, global industry practices, & Homayoun Hatami, managing partner, global client capabilities
•
The power of empathy. Decades of evidence show that empathy is a superpower in the workplace, Stanford University research psychologist Jamil Zaki shares on an episode of the McKinsey Talks Talent podcast. Workers who think that their organizations and managers care about them report less burnout, better mental health and morale, and a greater intent to stay at their companies, says Zaki, who joined McKinsey talent leaders Bryan Hancock and Brooke Weddle on the podcast. They also tend to be more innovative and take more creative risks.
•
Strengthening empathy. Empathy isn’t a fixed trait: rather, it’s a skill that can be learned and improved upon with practice, says Zaki, author of The War for Kindness: Building Empathy in a Fractured World. One large tech company, for instance, designed a curriculum for a management academy that focused on building trust and empathy. The managers’ customer satisfaction scores improved twice as fast as those of a placebo group. Learn how leaders can create a more empathic, connected work culture.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this newsletter because you subscribed to the Only McKinsey newsletter, formerly called On Point.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey" <publishing@email.mckinsey.com> - 01:48 - 22 Mar 2024 -
Taskeye - Field Employee Task Tracking Software Offering Real-Time Visibility into Employees’ Activities on the Field.
Taskeye - Field Employee Task Tracking Software Offering Real-Time Visibility into Employees’ Activities on the Field.
Experience the power of our field employe task tracking software that offers real-time visibility into employees’ activities on the field. With our innovative device-less tracking system, effortlessly monitor the workforce and optimize their performance like never before.
Advanced tool to reduce workload and increase employee productivity.
Uffizio Technologies Pvt. Ltd., 4th Floor, Metropolis, Opp. S.T Workshop, Valsad, Gujarat, 396001, India
by "Sunny Thakur" <sunny.thakur@uffizio.com> - 08:00 - 21 Mar 2024 -
A Brief History of Scaling Netflix
A Brief History of Scaling Netflix
Forwarded this email? Subscribe here for moreLatest articles
If you’re not a subscriber, here’s what you missed this month.
To receive all the full articles and support ByteByteGo, consider subscribing:
Netflix began its life in 1997 as a mail-based DVD rental business.
Marc Randolph and Reed Hastings got the idea of Netflix while carpooling between their office and home in California.
Hastings admired Amazon and wanted to emulate their success by finding a large category of portable items to sell over the Internet. It was around the same time that DVDs were introduced in the United States and they tested the concept of selling or renting DVDs by mail.
Fast forward to 2024, Netflix has evolved into a video-streaming service with over 260 million users from all over the world. Its impact has been so humongous that “Netflix being down” is often considered an emergency.
To support this amazing growth story, Netflix had to scale its architecture on multiple dimensions.
In this article, we attempt to pull back the curtains on some of the most significant scaling challenges they faced and how those challenges were overcome.
The Architectural Origins of Netflix
Like any startup looking to launch quickly in a competitive market, Netflix started as a monolithic application.
The below diagram shows what their architecture looked like a long time ago.
The application consisted of a single deployable unit with a monolithic database (Oracle). As you can notice, the database was a possible single point of failure.
This possibility turned into reality in August 2008.
There was a major database corruption issue due to which Netflix couldn’t ship any DVDs to the customers for 3 days. It suddenly became clear that they had to move away from a vertically scaled architecture prone to single points of failure.
As a response, they made two important decisions:
Move all the data to the AWS cloud platform
Evolve the systems into a microservices-based architecture
The move to AWS was a crucial decision.
When Netflix launched in 2007, EC2 was just getting started and they couldn’t leverage it at the time. Therefore, they built two data centers located right next to each other.
However, building a data center is a lot of work. You’ve to order equipment, wait for the equipment to arrive and install it. Before you finish, you’ve once again run out of capacity and need to go through the whole cycle again.
To cut through this cycle, Netflix went for a vertical scaling strategy that led to their early system architecture being modeled as a monolithic application.
However, the outage we talked about earlier taught Netflix one critical lesson - building data centers wasn’t their core capability.
Their core capability was delivering video to the subscribers and it would be far better for them to get better at delivering video. This prompted the move to AWS with a design approach that can eliminate single points of failure.
It was a mammoth decision for the time and Netflix adopted some basic principles to guide them through this change:
Buy vs Build
First, try to use or contribute to open-source technology wherever possible.
Only build from scratch what you absolutely must.
Stateless Services
Services should be built in a stateless manner except for the persistence or caching layers.
No sticky sessions.
Employ chaos testing to prove that an instance going down doesn’t impact the wider system.
Scale-out vs scale up
Horizontal scaling gives you a longer runway in terms of scalability.
Prefer to go for horizontal scaling instead of vertical scaling.
Redundancy and Isolation
Make more than one copy of anything. For example, replica databases and multiple service instances.
Reduce the blast radius of any issue by isolating workloads.
Automate Destructive Testing
Destructive testing of the systems should be an ongoing activity.
Adoption of tools like Chaos Monkey to carry out such tests at scale.
These guiding principles acted as the North Star for every transformational project Netflix took up to build an architecture that could scale according to the demands.
The Three Main Parts of Netflix Architecture
The overall Netflix architecture is divided into three parts:
The Client
The Backend
The Content Delivery Network
The client is the Netflix app on your mobile, a website on your computer or even the app on your Smart TV. It includes any device where the users can browse and stream Netflix videos. Netflix controls each client for every device.
The backend is the part of the application that controls everything that happens before a user hits play. It consists of multiple services running on AWS and takes care of various functionalities such as user registration, preparing incoming videos, billing, and so on. The backend exposes multiple APIs that are utilized by the client to provide a seamless user experience.
The third part is the Content Delivery Network also known as Open Connect. It stores Netflix videos in different locations throughout the world. When a user plays a video, it streams from Open Connect and is displayed on the client.
The important point to note is that Netflix controls all three areas, thereby achieving complete vertical integration over their stack.
Some of the key areas that Netflix had to scale if they wanted to succeed were as follows:
The Content Delivery Network
The Netflix Edge
APIs
Backend Services with Caching
Authorization
Memberships
Let’s look at each of these areas in more detail.
Scaling the Netflix CDN
Imagine you’re watching a video in Singapore and the video is being streamed from Portland. It’s a huge geographic distance broken up into many network hops. There are bound to be latency issues in this setup resulting in a poorer user experience.
If the video content is moved closer to the people watching it, the viewing experience will be a lot better.
This is the basic idea behind the use of CDN at Netflix.
Put the video as close as possible to the users by storing copies throughout the world. When a user wants to watch a video, stream it from the nearest node.
Each location that stores video content is called a PoP or point of presence. It’s a physical location that provides access to the internet and consists of servers, routers and other networking equipment.
However, it took multiple iterations for Netflix to scale their CDN to the right level.
Iteration 1 - Small CDN
Netflix debuted its streaming service in 2007.
At the time, it had over 35 million members across 50 countries, streaming more than a billion hours of video each month
To support this usage, Netflix built its own CDN in five different locations within the United States. Each location contained all of the content.
Iteration 2 - 3rd Party CDN
In 2009, Netflix started to use 3rd party CDNs.
The reason was that 3rd-party CDN costs were coming down and it didn’t make sense for Netflix to invest a lot of time and effort in building their own CDN. As we saw, they struggled a lot with running their own data centers.
Moving to a 3rd-party solution also gave them time to work on other higher-priority projects. However, Netflix did spend a lot of time and effort in developing smarter client applications to adapt to changing network conditions.
For example, they developed techniques to switch the streaming to a different CDN to get a better result. Such innovations allowed them to provide their users with the highest quality experience even in the face of errors and overloaded networks.
Iteration 3 - Open Connect
Sometime around 2011, Netflix realized that they were operating at a scale where a dedicated CDN was important to maximize network efficiency and viewing experience.
The streaming business was now the dominant source of revenue and video distribution was a core competency for Netflix. If they could do it with extremely high quality, it could turn into a huge competitive advantage.
Therefore, in 2012, Netflix launched its own CDN known as Open Connect.
To get the best performance, they developed their own computer system for video storage called Open Connect Appliances or OCAs.
The below picture shows an OCA installation:
Source: Netflix OpenConnect Website An OCA installation was a cluster of multiple OCA servers. Each OCA is a fast server that is highly optimized for delivering large files. They were packed with lots of hard disks or flash drives for storing videos.
Check the below picture of a single OCA server:
Source: Open Connect Presentation The launch of Open Connect CDN had a lot of advantages for Netflix:
It was more scalable when it came to providing service everywhere in the world.
It had better quality because they could now control the entire video path from transcoding, CDN, and clients on the devices.
It was also less expensive as compared to 3rd-party CDNs.
Scaling the Netflix Edge
The next critical piece in the scaling puzzle of Netflix was the edge.
The edge is the part of a system that’s close to the client. For example, out of DNS and database, DNS is closer to the client and can be thought of as edgier. Think of it as a degree rather than a fixed value.
Edge is the place where data from various requests enters into the service domain. Since this is the place where the volume of requests is highest, it is critical to scale the edge.
The Netflix Edge went through multiple stages in terms of scaling.
Early Architecture
The below diagram shows how the Netflix architecture looked in the initial days.
As you can see, it was a typical three-tier architecture.
There is a client, an API, and a database that the API talks to. The API application was named NCCP (Netflix Content Control Protocol) and it was the only application that was exposed to the client. All the concerns were put into this application.
The load balancer terminated the TLS and sent plain traffic to the application. Also, the DNS configuration was quite simple. The idea was that clients should be able to find and reach the Netflix servers.
Such a design was dictated by the business needs of the time. They had money but not a lot. It was important to not overcomplicate things and optimize for time to market.
The Growth Phase
As the customer base grew, more features were added. With more features, the company started to earn more money.
At this point, it was important for them to maintain the engineering velocity. This meant breaking apart the monolithic application into microservices. Features were taken out of the NCCP application and developed as separate apps with separate data.
However, the logic to orchestrate between the services was still within the API. An incoming request from a client hits the API and the API calls the underlying microservices in the right order.
The below diagram shows this arrangement:
Continue reading this post for free, courtesy of Alex Xu.
A subscription gets you:
An extra deep dive on Thursdays Full archive Many expense it with team's learning budget © 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:39 - 21 Mar 2024 -
What does it take to succeed with digital and AI?
On Point
Brought to you by Liz Hilton Segel, chief client officer and managing partner, global industry practices, & Homayoun Hatami, managing partner, global client capabilities
•
‘Rewiring’ companies. As gen AI sweeps through the business world, new developments regularly appear in the news. The technology’s potential to change how people live and work is profound. Yet the buzz surrounding gen AI may distract business leaders from other vital goals. After all, long-term success with digital transformation comes when leaders “rewire” their companies—changing their talent, operating model, and technical capabilities to support continuous innovation, McKinsey senior partner Kate Smaje and McKinsey Digital leader Rodney Zemmel explain.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this newsletter because you subscribed to the Only McKinsey newsletter, formerly called On Point.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey" <publishing@email.mckinsey.com> - 01:47 - 21 Mar 2024 -
‘Unbundling’ and expanding banks could boost their value
Re:think
FRESH TAKES ON BIG IDEAS
ON THE FUTURE OF BANKING
How banks can take control of their futureMiklós Dietz
The banking system, supported by higher profit margins, has a historic opportunity in the next few years to reinvent its business model—something it needs to do. Despite margins strengthened by elevated interest rates, banking is the lowest-valued sector in the world, trading at a 0.8 price-to-book ratio, while the rest of the global economy trades at 2.7. Price to book reflects the theoretical amount that shareholders would get if all assets were liquidated and all debts repaid. Trading below 1.0 means the markets see the banking system in a negative light.
Clearly, the banking model needs future-proofing. Banks are losing share in the global intermediary landscape to nonbanks, which are cherry-picking high-profitability businesses. One reason banks are trading so low is that they are very complex. Banking is a mix of three things: distribution (branches and sales staff), transactions, and balance sheet management, which measures how banks are transforming deposits into loans and managing credit risk. Distribution and transactions are profitable and require little capital, so they create value. Keeping things on the balance sheet does not create value, so the banking system needs to consider a new approach to balance sheet management.
Banks could consider separating the core balance sheet from distribution and transaction, following a path taken by utilities and telcos. You don’t necessarily have to break up the bank, but by unbundling, you could create more transparency for investors into what the bank is doing. Banks could speed up the metabolism of balance sheet management through faster securitization. Technology, especially AI, can reinvent every layer of banking, making it more cost-efficient. One more thing we can’t forget: risk. Ultimately, the best-performing banks are the ones that get risk right. It’s not just traditional credit risk but also new types, including cyber and geopolitical risks.
Banks today have it tough. They are expected to operate with incredible cost efficiency and robustness; to run the general ledger and balance sheet; to do asset liability management, matching everything; and to never make a mistake. That alone is a huge task. But banks are also expected to focus on innovation and growth and produce personalized services for every single customer. These two sets of expectations create tension.“The best US banks operate at a cost-to-asset ratio of 200 basis points, while the best European banks can do 70 basis points or even less.”
In the next few years, banks will need to get the basics right and at the same time reinvent themselves. On one hand, they need to be digital and AI-driven, low-cost, and efficient. On the other hand, they need to be visionary, go into new sectors, and create end-to-end customer journeys.
To create powerful customer journeys, banks can really own the customer relationship. This may enable them to understand clients’ needs and serve them more effectively, bringing higher margins and stronger customer attachment. For example, instead of just offering mortgages, banks could help buyers find a home, move in, and finance it. In payments, banks could offer coupons, e-gifting, online marketplaces, and location-based services. For business clients, banks could add a unified, finance- and payment-enabled platform that integrates services including administrative, tax, accounting, business intelligence, benchmarking, and B2B marketplaces.
An ideal banking model might involve strategies from different geographies. In some areas of banking, the United States is more advanced, because it disintermediates more and does more securitization. In others, Europe is more advanced, digital, and efficient. The best American banks operate at a cost-to-asset ratio of 200 basis points, meaning they incur costs of two cents for every $1 of assets managed. The best European banks can operate at a cost-to-asset ratio of 70 to 80 basis points or even less. In Europe, the best banks do 70 to 80 percent of their sales digitally. Some European banks can approve a mortgage application within a day. At the best American banks, it still takes weeks to get a mortgage. Asian banks, especially Indian banks, are the stars of how to go beyond banking and discover new avenues and differentiate. They operate almost like tech companies in many ways.
A provocative vision for the bank of 2035 might be a platform of networks—essentially a holding company for a collection of businesses—including e-commerce, payments, consumer lending, real estate, and a truly personalized advisory business, moving beyond financial into insurance, healthcare, and other things. This could be a $1 trillion market cap bank. It would still do everything that banking is doing now—it would still be very heavily regulated and very stable—but it would be unbundled to create value.Share Miklós Dietz’s insights
ABOUT THIS AUTHOR
Miklós Dietz is a senior partner in McKinsey’s Vancouver office.
MORE FROM THIS AUTHOR
UP NEXT
Duwain Pinder on Black Americans’ prosperity
Black Americans’ economic well-being varies greatly depending on where they live. But even where they are most prosperous, there is a significant breach between them and their White neighbors. Progress requires an all-hands-on-deck approach, in which many solutions are operating at scale for a long time.
Follow our thinking
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Quarterly alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Quarterly" <publishing@email.mckinsey.com> - 02:21 - 20 Mar 2024 -
Join me on Thursday for Dashboard techniques to visualise system and business performance
Hi MD,
It's Liam Hurrell, Manager of Customer Training at New Relic University, here.
Are you looking to improve your dashboard skills, or use techniques that enable you to visualise patterns, correlate important metrics and make better data-driven decisions? If so, you can register to attend this free online workshop I'll be hosting on Thursday 21st March at 10 am GMT/ 11 am CET, “Dashboard techniques to visualize system and business performance”.
You can find the full agenda on the registration page here. While we recommend attending the hands-on workshop live, you can also register to receive the recording.
Hope to see you then,
Liam HurrellManager, Customer TrainingNew Relic
This email was sent to info@learn.odoo.com as a result of subscribing or providing consent to receive marketing communications from New Relic. You can tailor your email preferences at any time here.Privacy Policy © 2008-24 New Relic, Inc. All rights reserved
by "Liam Hurrell, New Relic" <emeamarketing@newrelic.com> - 06:06 - 20 Mar 2024 -
The energy industry can make changes to attract younger workers
On Point
Brought to you by Liz Hilton Segel, chief client officer and managing partner, global industry practices, & Homayoun Hatami, managing partner, global client capabilities
•
Something old; something new. Energy companies have their work cut out for them amid the transition to clean energy. They need to maintain their core businesses but also move into low-carbon offerings, power, renewables, and retail. To be successful, they should be agile, efficient, and fast—all while operating in an industry for which talent is scarce, potential M&A is looming, and generative AI is poised to shake things up, share McKinsey partners Ignacio Fantaguzzi and Christopher Handscomb.
—Edited by Jana Zabkova, senior editor, New York
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this newsletter because you subscribed to the Only McKinsey newsletter, formerly called On Point.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey" <publishing@email.mckinsey.com> - 01:35 - 20 Mar 2024 -
The human side of generative AI: Creating a path to productivity
Share this email
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Quarterly alert list.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Quarterly" <publishing@email.mckinsey.com> - 12:51 - 20 Mar 2024
