Archives
-
Our most-read leadership advice
Leading Off
Share this email
Great leaders know the value of taking time to rest and recharge—and we’re following their advice by taking a break from our usual Leading Off delivery schedule. We hope you’re also finding some space to slow down this summer and enjoy time with friends and family.
On September 9, we’ll be back in your inbox with never-before-heard reflections from leaders in The Journey of Leadership by Dana Maor, Kurt Strovink, Ramesh Srinivasan, and senior partner emeritus Hans-Werner Kaas. While we’re away, you can browse through past issues here—and be sure you didn’t miss recent favorites:If you have friends or colleagues who might enjoy Leading Off, consider forwarding this email to them or sharing it on LinkedIn, X, or Facebook. They can sign up for this or any of our 40+ other free email subscriptions at mckinsey.com/subscriptions. (And you might also want to revisit that page to see our full newsletter lineup.)
Share these insights
Did you enjoy this newsletter? Forward it to colleagues and friends so they can subscribe too. Was this issue forwarded to you? Sign up for it and sample our 40+ other free email subscriptions here.
This email contains information about McKinsey’s research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Leading Off newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Leading Off" <publishing@email.mckinsey.com> - 02:02 - 19 Aug 2024 -
Guest Post article published Offered
Hello,I am a professional SEO outreach specialist (blogger) providing high quality websites for publishing guest posts and building backlinks. We are accepting guest (sponsored) posts on our websites.Fashion General and Technology Sites Checkhttps://stylesium.com/
https://theladyposh.com/
https://trskins.com/
http://theasmr.com/
https://stylevanity.com/
https://lovetravellife.com/
https://www.homeopathy360.com/
https://www.noragouma.com/
https://thepicnob.com/
https://techbullion.com/
https://sguru.org/If you are interested in guest posting or writing with us , send us an email.
All the best,
by hussanalig397@gmail.com - 02:32 - 18 Aug 2024 -
How companies can make big moves to beat the odds
Share this email
The power of big moves in crafting a successful strategy
In today’s ever competitive business environment, crafting the right strategy is crucial for any company to distance itself from the rest of the pack. But all too often, leaders are stymied once they enter the strategy room, where biases, competing agendas, and short-term projections result in unrealistic expectations and ineffective planning. This is the “social side of strategy,” as the authors of this 2018 classic put it, and it’s preventing your company from moving up the economic power curve.
So how do you reduce the social noise and boost your odds of setting a successful strategy? Start by making big moves. The authors highlight five that, according to their research, make the biggest difference—for example, engaging in a dynamic reallocation of resources, where organizations feed the business units that are likely to produce substantial moves up the power curve (and starve the ones that aren’t), and having a strong productivity program, one that puts you in at least the top 30 percent of your industry.
To uncover the other big moves that can help make your strategic-planning sessions more worthwhile, read Chris Bradley and Sven Smit’s 2018 McKinsey Quarterly classic, “Strategy to beat the odds.”Revamp your strategic planning 
Share these insights
Did you enjoy this newsletter? Forward it to colleagues and friends so they can subscribe too. Was this issue forwarded to you? Sign up for it and sample our 40+ other free email subscriptions here.
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to our McKinsey Classics newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Classics" <publishing@email.mckinsey.com> - 12:55 - 17 Aug 2024 -
EP125: How does Garbage Collection work?
EP125: How does Garbage Collection work?
Forwarded this email? Subscribe here for moreThis week’s system design refresher:
Linux Performance Tools! (Youtube video)
How does Garbage Collection work?
A Cheat Sheet for Designing Fault-Tolerant Systems
10 System Design Tradeoffs You Cannot Ignore
SPONSOR US
WorkOS: Modern Identity Platform for B2B SaaS (Sponsored)
Start selling to enterprises with just a few lines of code.
→ WorkOS provides a complete user management solution along with SSO, SCIM, Audit Logs, & Fine-Grained Authorization.
→ Unlike other auth providers that rely on user-centric models, WorkOS is designed for B2B SaaS with an org modeling approach.
→ The APIs are flexible, easy-to-use, and modular. Pick and choose what you need and integrate in minutes.
→ User management is free up to 1 million MAUs and comes standard with RBAC, bot protection, MFA, & more.
Linux Performance Tools!
How does Garbage Collection work?
Garbage collection is an automatic memory management feature used in programming languages to reclaim memory no longer used by the program.
Java
Java provides several garbage collectors, each suited for different use cases:
1. Serial Garbage Collector: Best for single-threaded environments or small applications.
2. Parallel Garbage Collector: Also known as the "Throughput Collector."
3. CMS (Concurrent Mark-Sweep) Garbage Collector: Low-latency collector aiming to minimize pause times.
4. G1 (Garbage-First) Garbage Collector: Aims to balance throughput and latency.
5. Z Garbage Collector (ZGC): A low-latency garbage collector designed for applications that require large heap sizes and minimal pause times.Python
Python's garbage collection is based on reference counting and a cyclic garbage collector:
1. Reference Counting: Each object has a reference count; when it reaches zero, the memory is freed.
2. Cyclic Garbage Collector: Handles circular references that can't be resolved by reference counting.GoLang
Concurrent Mark-and-Sweep Garbage Collector: Go's garbage collector operates concurrently with the application, minimizing stop-the-world pauses.
Latest articles
If you’re not a paid subscriber, here’s what you missed.
To receive all the full articles and support ByteByteGo, consider subscribing:
A Cheat Sheet for Designing Fault-Tolerant Systems
Designing fault-tolerant systems is crucial for ensuring high availability and reliability in various applications. Here are six top principles of designing fault-tolerant systems:
Replication
Replication involves creating multiple copies of data or services across different nodes or locations.Redundancy
Redundancy refers to having additional components or systems that can take over in case of a failure.Load Balancing
Load balancing distributes incoming network traffic across multiple servers to ensure no single server becomes a point of failure.Failover Mechanisms
Failover mechanisms automatically switch to a standby system or component when the primary one fails.Graceful Degradation
Graceful degradation ensures that a system continues to operate at reduced functionality rather than completely failing when some components fail.Monitoring and Alerting
Continuously monitor the system's health and performance, and set up alerts for any anomalies or failures.
10 System Design Tradeoffs You Cannot Ignore
If you don’t know trade-offs, you DON'T KNOW system design.
Vertical vs Horizontal Scaling
Vertical scaling is adding more resources (CPU, RAM) to an existing server.Horizontal scaling means adding more servers to the pool.
SQL vs NoSQL
SQL databases organize data into tables of rows and columns.NoSQL is ideal for applications that need a flexible schema.
Batch vs Stream Processing
Batch processing involves collecting data and processing it all at once. For example, daily billing processes.Stream processing processes data in real time. For example, fraud detection processes.
Normalization vs Denormalization
Normalization splits data into related tables to ensure that each piece of information is stored only once.Denormalization combines data into fewer tables for better query performance.
Consistency vs Availability
Consistency is the assurance of getting the most recent data every single time.Availability is about ensuring that the system is always up and running, even if some parts are having problems.
Strong vs Eventual Consistency
Strong consistency is when data updates are immediately reflected.Eventual consistency is when data updates are delayed before being available across nodes.
REST vs GraphQL
With REST endpoints, you gather data by accessing multiple endpoints.With GraphQL, you get more efficient data fetching with specific queries but the design cost is higher.
Stateful vs Stateless
A stateful system remembers past interactions.A stateless system does not keep track of past interactions.
Read-Through vs Write-Through Cache
A read-through cache loads data from the database in case of a cache miss.A write-through cache simultaneously writes data updates to the cache and storage.
Sync vs Async Processing
In synchronous processing, tasks are performed one after another.In asynchronous processing, tasks can run in the background. New tasks can be started without waiting for a new task.
Over to you: Which other tradeoffs have you encountered?
SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com
Like
Comment
Restack
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:35 - 17 Aug 2024 -
Look back at some of our readers’ favorite recent charts
The Week in Charts
Share this email
Thanks for reading The Week in Charts as we aim to provide you with the visuals you need to understand today’s biggest business and management challenges. Now the data shows that it’s time for a short hiatus, though we’ll resume our weekly send schedule on September 3. Stay tuned for new charts when we return, including ones on leadership in honor of The Journey of Leadership by Dana Maor, Kurt Strovink, Ramesh Srinivasan, and senior partner emeritus Hans-Werner Kaas.
While we’re away, take a look back at some recent charts that resonated most with our readers:If you have friends or colleagues who might enjoy The Week in Charts, consider forwarding this email to them or sharing it on LinkedIn, X, or Facebook. They can sign up for this or any of our 40+ other free email subscriptions at mckinsey.com/subscriptions. (And you might also want to revisit that page to see our full newsletter lineup.)
Share these insights
Did you enjoy this newsletter? Forward it to colleagues and friends so they can subscribe too. Was this issue forwarded to you? Sign up for it and sample our 40+ other free email subscriptions here.
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to The Week in Charts newsletter.
Copyright © 2024 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "McKinsey Week in Charts" <publishing@email.mckinsey.com> - 11:14 - 17 Aug 2024 -
RE :: QUOTE 2024
Harry potter
Hello ,
Dear Web Owner,
I hope you are doing well.
Want more clients and customers?
We will help them find you by putting you on the 1st page in Google.
Can I send you best price list?
Thanks!
by "Krishna" <svhjvx@gmail.com> - 02:17 - 17 Aug 2024 -
The physical realities of the energy transition, customer delight, customer-centric insurance, and more highlights
Share this email
Ready to unwind?
—Edited by Joyce Yoo, editor, New York
Follow our thinking
Share these insights
Did you enjoy this newsletter? Forward it to colleagues and friends so they can subscribe too. Was this issue forwarded to you? Sign up for it and sample our 40+ other free email subscriptions here.
by "McKinsey Daily Read" <publishing@email.mckinsey.com> - 10:16 - 15 Aug 2024 -
Re: Bidding & Takeoff Services
I am not sure whether you received my below attached email which I sent you.
We take great pride in completing our projects on schedule and exceeding our client’s expectations. Kindly give us a chance to work with you in preparing your bids. Plus, we have a special discount available!
Also, our estimating fees range from $75 to $450, with specific amounts of $250, $350, and $450. We're happy to take on jobs in the $75 to $100 range as well.
If you have any plans/drawings, send them to benjamin.quickbidestimating@gmail.com
Thanks for your time!
Best Regards,
Benjamin Goldberg | Project Executive
QuickBid Estimating
P: (917) 300-1079
W: www.quickbidestimating.comOn Tue, Aug 13, 2024 at 2:14 PM <quickbidestimating23@gmail.com> wrote:Good Day,
QuickBid Estimating is dedicated to providing thorough Cost Estimation, Takeoffs, and Scheduling services tailored to your project's specific needs. Utilizing industry-leading software like Planswift, On-Screen Takeoff, Bluebeam, and MS Project, our expert team delivers accurate and reliable quantity and material takeoffs.
We work with General Contractors, Subcontractors, Project Managers, Developers, Investors, Architects, Homebuilders, Owners, and vendors, presenting our detailed analyses in an editable Excel format to ensure precision and promptness. Plus, for a limited time, we’re offering a discount on our services.
Share your project plans with us at benjamin.quickbidestimating@gmail.com, and let’s start our collaboration!
Looking forward to your reply.
Best Regards,
Benjamin Goldberg | Project Executive
QuickBid Estimating
P: (917) 300-1079
W: www.quickbidestimating@gmail.com
by "Benjamin Goldberg" <quickbidestimating23@gmail.com> - 01:13 - 15 Aug 2024 -
A Crash Course on Architectural Scalability
A Crash Course on Architectural Scalability
Forwarded this email? Subscribe here for moreLatest articles
If you’re not a subscriber, here’s what you missed this month.
To receive all the full articles and support ByteByteGo, consider subscribing:
In today's interconnected world, software applications have a global reach, serving users from diverse geographical locations.
With the rapid growth of social media and viral content, a single tweet or post can lead to a sudden and massive surge in traffic to an application. The importance of building applications with scalable architecture from the ground up has never been higher.
Being prepared for unexpected traffic spikes is indispensable for development teams building applications. A sudden increase in user demand may be just around the corner. Not being prepared for it can put immense pressure on the application's infrastructure. It not only causes performance degradation but can also, in some cases, result in a complete system failure.
To mitigate these risks and ensure a good user experience, teams must proactively design and build scalable architectures.
But what makes scalability such a desirable characteristic?
Scalability allows the application to dynamically adapt to changing workload requirements without compromising performance or availability.
In this post, we’ll understand the true meaning of scalability from different perspectives followed by the various techniques and principles that can help you scale the application’s architecture. Also, in subsequent posts in the coming weeks, we’ll take deeper dives into the scalability of each layer and component of a typical architecture.
What is Scalability?...
Unlock this post for free, courtesy of Alex Xu.
A subscription gets you:
An extra deep dive on Thursdays Full archive Many expense it with team's learning budget © 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:36 - 15 Aug 2024 -
Efficient and Professional Logistics Services
If you don't want to receive our emails, you can easily unsubscribe here.
Services: Ocean Freight | Air Freight | Projects | Trucking | Insurance | Rail Freight | Custom Brokerage | Warehousing & Distributionowen│Sales manager
Shunshunfa International Logistics Co.,Ltd
Add: B1, 7th Floor, Building 3, Lehui Science and Technology Innovation Center, No. 489 Jihua Road, Bantian Street, Longgang District, Shenzhen
Email: owen@ssf-logistics.com Web:https//:www.ssf-logistics.comMob: +86 13751768263 QQ:625927164
by "OWEN" <owen@ssf-logistics.com> - 10:35 - 15 Aug 2024 -
Estimating & Take-Off Services
Hi,
We provide residential & commercial estimates & take-offs. Do you have any project for estimation? Send me the plans/drawings for a quote.
If you would like to see some examples of our work, we would be happy to share them with you.ThanksBest Regards,Mason
by "Mason" <mason.premiumestimate@gmail.com> - 09:39 - 15 Aug 2024 -
Free log data ingest will change the way you work
Sumo Logic
Free data ingestion supercharges your budget.
The single source of truth you need to succeed at your job shouldn’t be limited by questioning how much (or what) data can be ingested and analyzed.
Excluded information might be vital - and analysis and innovation could suffer.
Learn more about the economics of log management and analytics in our new blog - Log it all and eliminate visibility gaps.
Don’t let budget be a barrier to development success.
Our analytics-based pricing model meets today's needs - pay for insights with no extra fees.
Included at no cost: unlimited log ingestion and indexing with unlimited, full-access users.
You only pay for analysis and storage.
Calculate flex pricing
Don’t sacrifice performance for budget when you can have both.Sumo Logic, Aviation House, 125 Kingsway, London WC2B 6NH, UK
© 2024 Sumo Logic, All rights reserved.Unsubscribe 
by "Sumo Logic" <marketing-info@sumologic.com> - 09:00 - 14 Aug 2024 -
Hi there, can I send you some examples?
Hi there, can I send you some examples?
Hi there,
If this may be of interest to Your Telecoms Consultant, I'd like to send you some examples of the systems we've created for other clients.
Just drop me an email back if you're happy for me to send something over.
Thank you,
Andrew Miller
CRM Management Ltd
01759 361 845
Andrew Miller <Andrew@crmcompany.co.uk> wrote:
Hi there,
At CRM Management Ltd, we specialise in offering bespoke development services tailored to your company’s unique needs. If you haven’t found a solution that fits your requirements, our team of experts will work with you to design and develop the perfect system to support and grow your business.
With many years of experience across a broad range of solutions, we can create anything from a simple Customer Relationship Management system to a comprehensive multi-channel ERP system. Here are some of the solutions we offer:
- Ticketing systems
- Booking systems
- Estate Agency systems
- Recruitment systems
- Customer Relationship Management (CRM) systems
- E-commerce systems
- Enterprise Resource Planning (ERP) systems
- Bespoke systems
Do you have any projects in mind? Simply reply to this email, and we can schedule a call.
Kind regards,
Andrew Miller
CRM Management Ltd
01759 361 845
I have sent this email to info@learn.odoo.com. If you don't want to get any more emails from us, you can stop them here.
CRM Management Ltd - English House, Railway St, Pocklington, York YO42 2QZ - Company no. 10684534
by "Andrew Miller" <Andrew@crmcompany.co.uk> - 04:06 - 14 Aug 2024 -
Counting Billions of Content Usage at Canva
Counting Billions of Content Usage at Canva
Forwarded this email? Subscribe here for moreHands-on Rust Developer Workshop: Build a Low-Latency Social Media App (Sponsored)
During this free interactive workshop oriented for developers, engineers, and architects, you will learn how to:
Create and compile a sample social media app with Rust
Connect the application to ScyllaDB (NoSQL data store) and Redpanda (streaming data)
Negotiate tradeoffs related to data modeling and querying
Manage and monitor the database for consistently low latencies
If you’re an application developer with an interest in Rust, Tokio, and event-driven architectures this workshop is for you! This is a great way to discover the NoSQL strategies used by top teams and apply them in a guided, supportive environment.
Disclaimer: The details in this post have been derived from the Canva Engineering Blog. All credit for the technical details goes to the Canva engineering team. The links to the original articles are present in the references section at the end of the post. We’ve attempted to analyze the details and provide our input about them. If you find any inaccuracies or omissions, please leave a comment, and we will do our best to fix them.
What if incorrect counting results in incorrect payments?
Either the company making the payment or the users receiving that payment based on the incorrect count lose money. Both scenarios are problematic from the business perspective.
This is exactly the situation that Canva faced when they launched the Creators Program.
As you might already know, Canva is a tool that makes design accessible to everyone worldwide. One of the main ways they make this possible is through the Canva Creators Program.
In the three years since its launch, the use of content from this program has doubled every 18 months. They process billions of content uses monthly based on which the creators are paid. This includes the use of templates, images, videos, and more.
It is a critical requirement for Canva to count the usage data of this content accurately since the payments made to the creators depend on this data. However, it also presents some big challenges:
Accuracy: The count has to be correct to maintain creator trust and ensure fair pay.
Scalability: The system needs to handle rapidly growing amounts of data
Operability: As data increases, so does the complexity of maintenance and problem-solving.
In this post, we will look at the various architectures Canva’s engineering team experimented with to implement a robust counting service and the lessons they learned in the process.
The Initial Counting Service Design
Canva's original design for the content usage counting service was built on a MySQL database, a familiar and widely-used technology stack.
This initial design comprised several key components:
A MySQL database for storing all usage data (including raw events, deduplicated usage, and aggregated counts).
Separate worker services handling different pipeline stages (data collection, deduplication, and aggregation).
The persistence of multiple layers of reusable intermediary output at various stages.
The process flow for the solution could be broken down into three main steps:
Data Collection: Usage events from various sources (web, and mobile apps) were collected and stored in MySQL.
Deduplication: A worker service identifies duplicated usage events and matches them with a specific set of rules.
Aggregation: Another worker scanned the updated deduplication table, incrementing counters in a separate table.
The diagram below shows the architecture on a high level.
This architecture employed a single-threaded sequential process for deduplication, using a pointer to track the latest scanned record. While this approach made it easier to reason about and verify data processing, especially during troubleshooting or incident recovery, it faced significant scalability challenges.
The system required at least one database round trip per usage record, resulting in O(N) database queries for N records, which became increasingly problematic as data volume grew.
The initial MySQL-based architecture prioritized simplicity and familiarity over scalability. It allowed for quick implementation but created substantial challenges as the system expanded.
The heavy reliance on MySQL for both storage and processing created a bottleneck, failing to leverage the strengths of distributed systems or parallel processing.
MySQL RDS does not horizontally scale through partitioning by itself. Every time they needed more storage, they doubled the RDS instance size. This happened every 8-10 months, resulting in significant operational overhead.
Once the MySQL RDS instance reached several TBs, maintaining it became costly.
Lastly, finding and fixing issues in case of incidents was difficult because engineers had to look into databases and manually fix the incorrect data.
Latest articles
If you’re not a paid subscriber, here’s what you missed.
To receive all the full articles and support ByteByteGo, consider subscribing:
Migration to DynamoDB
Faced with the scalability limitations of the MySQL-based counting service, the team initially looked to DynamoDB as a potential solution.
This decision was primarily driven by DynamoDB's reputation for handling large-scale, high-throughput workloads - a perfect fit for Canva's rapidly growing data needs.
The migration process began with moving raw usage events from the data collection stage to DynamoDB, which provided immediate relief to the storage constraints. This initial success prompted the team to consider moving the entirety of their data to DynamoDB. It was a move that would have necessitated a substantial rewrite of their codebase.
However, after careful evaluation, Canva decided against a full migration to DynamoDB.
While DynamoDB could have effectively addressed the storage scalability issues, it wouldn't have solved the fundamental problem of processing scalability. The team found it challenging to eliminate the need for frequent database round trips, which was a key bottleneck in their existing system.
This reveals a crucial lesson in system design: sometimes, what appears to be a storage problem is a processing problem in disguise.
Canva's approach clearly shows the importance of thoroughly analyzing the root causes of system limitations before committing to major architectural changes. It also highlights the complexity of scaling data-intensive applications, where the interplay between storage and processing capabilities can be subtle and non-obvious.
The OLAP-based Counting Service
Canva's latest architecture for the content usage counting service shows a shift from traditional OLTP databases to an OLAP-based solution, specifically using Snowflake.
The change came after realizing that previous attempts with MySQL and DynamoDB couldn't adequately address their scalability and processing needs. The new architecture altered how Canva processed and stored the usage data, adopting an ELT (Extract, Load, Transform) approach.
The diagram below shows the new architecture:
In the extraction phase, Canva pulled raw usage data from various sources, including web browsers and mobile apps. This data was then loaded into Snowflake using a reliable data replication pipeline provided by Canva’s data platform team. The reliability of this data replication was crucial, as it formed the foundation for all subsequent processing.
The transformation phase used Snowflake's powerful computational capabilities. It also utilized DBT (Data Build Tool) to define complex transformations.
These transformations were written as SQL-like queries, allowing for end-to-end calculations directly on the source data. For example, one transformation aggregated usages per brand using a SQL query that selected data from a previous step named 'daily_template_usages' and grouped it by ‘day_id’ and ‘template_brand’.
The SQL below shows the aggregate query.
Source: Canva Engineering Blog The main steps in the transformation process were as follows:
Extraction of source event data in JSON format from DynamoDB, which was not optimal for data warehouse query processing. Therefore, some JSON properties were projected into separate table columns within Snowflake for optimization.
Deduplicate and aggregate the usage events.
A key aspect of this new architecture was the elimination of intermediary outputs. Instead of persisting data at various pipeline stages, Canva materialized intermediate transformation outputs as SQL Views.
The Advantage of OLAP Database
The separation of storage and computing in an OLAP database like Snowflake was a game-changer for Canva. It enabled them to scale computational resources independently.
As a result, they could now aggregate billions of usage records within minutes, a task that previously took over a day. This improvement was largely due to most of the computation being done in memory, which is several orders of magnitude faster than the database round trips required in their previous architecture.
There were several improvements such as:
The pipeline latency was reduced from over a day to under an hour.
Incident handling became more manageable. Most issues could be resolved by simply re-running the pipeline end-to-end, without manual database interventions.
Reduction in over 50% of the stored data and elimination of thousands of lines of deduplication and aggregation calculation code. The logic rewritten in SQL was simpler compared to the previous code.
The number of incidents dropped to once every few months or fewer.
Challenges of the New Solution
Despite the advantages, the solution also introduced new challenges such as:
Transformation Complexity: The data transformation jobs, written in an SQL-like language and deployed as a standalone service, introduced new deployment and compatibility considerations.
Data Unloading: Canva needed to build a reliable pipeline to unload data from Snowflake into databases used by other services. This involved using S3 as intermediary storage and integrating with SQS for durability. Optimizing the ingestion query and carefully tuning rate limits was crucial to prevent service database throttling.
Infrastructure Complexity: The new architecture increased infrastructure complexity due to data replication integration and standalone services running transformation jobs. This required additional effort to maintain observability across different toolsets.
Conclusion
Canva’s journey of implementing the counting service for the Creators Program is full of learning for software developers and architects.
Some of the key points to take away are as follows:
Simplicity is crucial for designing reliable services. In other words, reducing code and data complexity is the key. For example, minimizing intermediary output in various counting pipeline stages using OLAP and ELT helped simplify the system.
It’s good to start small and be pragmatic. The initial MySQL-based design served its purpose for the first two years and allowed timely delivery of the functionality to the users. Once scalability became a problem, they looked at alternative solutions.
Comprehensive observability, while requiring extra overhead, helps identify and resolve problems. Close monitoring from day one of every part of the pipeline was crucial to make the right decisions.
Recognizing when solutions are no longer adequate and being open to change is important. In Canva’s case, this was demonstrated by the willingness to pivot from familiar MySQL to a more suitable solution using OLAP database.
Big changes happen in increments. For example, the journey from MySQL to DynamoDB and finally to an OLAP solution showed the power of iterative improvements.
Reliable infrastructure is important to build a stable platform.
References:
SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com
© 2024 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:35 - 13 Aug 2024 -
Proposing a Link-Building Partnership
Hello,
I'm an outreach link-building assistant and guest post service provider.
I have 10000+ high-quality sites.??
Should I send you my full list with the price? or if you want you can share your requirements.
I will provide you with sites accordingly.
Your post will be permanent and the link will be do-follow. Your post will be indexed in Google.
If you have any questions, please don't hesitate to contact us.
I am waiting to hear back from you soon!
Thanks!Sourav Bauri
Blogger Outreach at Digital Link BuildingT: +91 8436922159
M: +919547021731
sourav.digitallinkbuilding@gmail.com - digitallinkbuilding.com
A: Jorda, Indpur, West Bengal , India
We’ve been voted one of the best SEO companies for 2010, and for good reason! Our team of knowledgeable experts can grow your business online by improving rankings, and increasing traffic and overall visibility.
by "Sourav Bauri" <sourav.digitallinkbuilding@gmail.com> - 11:11 - 13 Aug 2024
