Archives
-
High-quality aluminum products supply to meet your diverse needs
Dear info,
Hello! We are a manufacturer specializing in the production of aluminum plates, aluminum bars and aluminum products with a history of 13 years. We are committed to providing high-quality aluminum materials to customers around the world and have established a good reputation in the industry.
The aluminum plates and aluminum bars we provide are suitable for multiple industries such as aviation, automobiles, electronics and construction. The products meet international standards and have reliable quality. Whether you need custom specifications or standard products, we can meet your needs.
We are very much looking forward to working with your company to promote business development together. If you have any needs or questions, please feel free to contact me.
Thank you for your valuable time!
Best regards,
Suzhou Gebizhen Metal Materials Co., Ltd.
by "Lapsb Fakv" <fakvlapsb@gmail.com> - 01:13 - 7 May 2025 -
Elevate Your Offerings with Premium Plastic Solutions
Dear info,
We are Taizhou Hongshuo Plastic Industry Co., Ltd, your trusted partner in crafting high-quality plastic lifestyle products. Our range includes plastic straw cups, folding storage boxes, color-changing cups, rice storage containers, socks, and more—all designed for durability, innovation, and everyday convenience.
With advanced manufacturing techniques and a focus on eco-friendly materials, we ensure every product meets rigorous global standards. Whether you’re sourcing for retail, hospitality, or custom OEM/ODM projects, our team provides tailored solutions to align with your brand vision and market demands.
Let’s discuss how we can enhance your product line! Reply today to request samples, catalogs, or a personalized quote. We’re here to deliver excellence at every step.
Warm regards,
Haining Mengli Knitting Co., Ltd
by "Macco Kammerer" <maccokammerer@gmail.com> - 12:27 - 7 May 2025 -
Strengthening the US healthcare system: What leaders can do
On McKinsey Perspectives
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 11:05 - 6 May 2025 -
Feeling stuck on your next digital move?
Feeling stuck on your next digital move?
Hi MD Abul,
If you're trying to figure out what to do next with your digital strategy, maybe it's new software, maybe it's AI, maybe it's just untangling the chaos, we've got something that might help.
We just launched a set of free assessments that give you a quick look at where your business stands today and what you could be doing better. These aren’t just automated results either. After you complete any of them, one of our actual consultants will follow up to talk through your results and get to know your business a bit more.
Here’s what you can check out:
Small Business Digital Assessment
Perfect for growing companies looking to get organized and future-readyDigital Transformation Readiness Assessment
Helpful if you're thinking big and want to understand how prepared you really areOrganizational Change Readiness Assessment
Focuses on the people side of change so your team doesn't get left behindThey're a great starting point if you're overwhelmed by options or just unsure what your next step should be. No sales pitch, no pressure, just some real guidance from someone who’s been there.
You can try any of them here: Explore the Assessments
Let me know if you want to chat after. We’d love to hear what you’re working on.
Mission: AI-Possible is taking off this August at Digital Stratosphere 2025.
Join us in Denver!Use the code "earlybird" for 20% off your ticket(s)!
Best regards,
Eric Kimberling
by "Eric Kimberling" <eric.kimberling@thirdstage-consulting.com> - 06:06 - 6 May 2025 -
Elevate Your Brand with Our Premium Saw Blades
Dear info,
I am Elizabeth from Wenzhou Yichuan Tools. Our company specializes in hardware tools, including jigsaw blades, reciprocating saw blades, saber saw blades, and multi-tool blades.
With two decades of industry experience and state-of-the-art production facilities, we deliver high-quality products at competitive prices. Our monthly capacity exceeds 5 million pieces.
We also provide OEM services to customize products for your brand. Would you like a quotation and samples for evaluation?
Best regards,
Elizabeth
Wenzhou Yichuan Tools Co., Ltd.
by "Odelette Hoyle" <robbie169850@gmail.com> - 05:02 - 6 May 2025 -
Greetings from Hangzhen Energy
Dear info,
I hope you are doing well. I'm with Zhejiang Hangzhen Energy Technology Company Limited, where we specialize in providing advanced solutions for the chemical, petrochemical, auxiliary machinery, and metallurgical industries.
Our offerings include steam jet vacuum pumps, Roots vacuum pump units, automatic exhaust valves, VD/VOD/RH refining furnaces, and more. , designed to optimize your processes and ensure reliable performance in demanding environments.
I'd love to discuss how our products can benefit your operations. Please let me know if we can arrange a time to talk further.
Best regards,
Xu Shangyong/Marketing Department
Zhejiang Hangzhen Energy Technology Company Limited.
Yuexiu Xinghui Middle, No. 723 Dayuan Road, Qingshanhu Street, Lin'an District, Hangzhou City, Zhejiang Province, China
www.hzvacuum.com
by "Robert Pht" <robertpht09@gmail.com> - 01:04 - 6 May 2025 -
How Halo on Xbox Scaled to 10+ Million Players using the Saga Pattern
How Halo on Xbox Scaled to 10+ Million Players using the Saga Pattern
Forwarded this email? Subscribe here for moreThe 6 Core Competencies of Mature DevSecOps Orgs (Sponsored)
Understand the core competencies that define mature DevSecOps organizations. This whitepaper offers a clear framework to assess your organization's current capabilities, define where you want to be, and outline practical steps to advance in your journey. Evaluate and strengthen your DevSecOps practices with Datadog's maturity model.
Disclaimer: The details in this post have been derived from the articles/videos shared online by the Halo engineering team. All credit for the technical details goes to the Halo/343 Engineering Team. The links to the original articles and videos are present in the references section at the end of the post. We’ve attempted to analyze the details and provide our input about them. If you find any inaccuracies or omissions, please leave a comment, and we will do our best to fix them.
In today's world of massive-scale applications, whether it’s gaming, social media, or online shopping, building reliable systems is a difficult task. As applications grow, they often move from using a single centralized database to being spread across many smaller services and storage systems. This change, while necessary for handling more users and data, brings a whole new set of challenges, especially around consistency and transaction handling.
In traditional systems, if we wanted to update multiple pieces of data (say, saving a new customer order and reducing inventory), we could easily rely on a transaction. A transaction would guarantee that either all updates succeed together or none of them happen at all.
However, in distributed systems, there is no longer just one database to talk to. We might have different services, each managing its data in different locations. Each one might be running on different servers, cloud providers, or even different continents. Suddenly, getting all of them to agree at the same time becomes much harder. Network failures, service crashes, and inconsistencies are now of everyday situations.
This creates a huge problem: how do we maintain correct and reliable behavior when we can't rely on traditional transactions anymore? If we’re booking a trip, we don’t want to end up with a hotel reservation but no flight. If we’re updating a player's stats in a game, we can't afford for some stats to update and others to disappear.
Engineers must find new ways to coordinate operations across multiple independent systems to tackle these issues. One powerful pattern for solving this problem is the Saga Pattern, a technique originally proposed in the late 1980s but increasingly relevant today. In this article, we’ll look at how the Halo Engineering Team at 343 Game Studio (now Halo Studios) used the Saga pattern to improve the player experience.
Build AI products and grow your impact (Sponsored)
Two critical themes define today's engineering landscape: building AI applications and growing your influence in an increasingly automated world. So, we teamed up with Maven to curate 6 hands-on courses on these topics, led by experienced practitioners.
Building Agentic AI Applications with Aishwarya Reganti (AWS)
Multi-Agent Applications with Amir Feizpour (AI engineer)
AI Evals with Hamel Husain (AI engineer, ex-Github, Airbnb)
Break Through to Executive with Ethan Evans (Amazon VP)
Elite Product Leadership with Amit Fulay (Microsoft VP)
Eng Management System with Gilad Naor (ex-Meta, Amazon)
Our friends at Maven said to use code BYTEBYTEGO to get $100 off any of these selected courses.
ACID Transactions
When engineers design systems that store and update data, they often rely on a set of guarantees called ACID properties. These properties make sure that when something changes in the system, like saving a purchase or updating a player's stats, it happens safely and predictably.
Here’s a quick look at each property.
Atomicity: Atomicity means "all or nothing". Either every step of a transaction happens successfully, or none of it does. If something goes wrong halfway, the system cancels everything so that it is not stuck with half-completed work. For example, either book both a flight and a hotel together, or neither.
Consistency: Consistency guarantees that the system’s data moves from one valid state to another valid state. After a transaction finishes, the data must still follow all the rules and constraints it’s supposed to. For example, after buying a ticket, the system shouldn’t show negative seat availability. The rules about what’s valid are always respected.
Isolation: Isolation ensures that transactions don’t interfere with each other even if they happen at the same time. It should feel like each transaction happens one after another, even when they’re happening in parallel. For example, if two people are trying to buy the last ticket at the same time, only one transaction succeeds.
Durability: Durability means that once a transaction is committed, it’s permanent. Even if the system crashes right after, the result stays safe and won't be lost. For example, if a user bought a ticket and the app crashes immediately after, the ticket is still booked and not lost in the system.
Single Database Model
In older system architectures, the typical way to build applications was to have a single, large SQL database that acted as the central source of truth.
Every part of the application, whether it was a game like Halo, an e-commerce site, or a banking app, would send all of its data to this one place.
Here’s how it worked:
Applications (like a game server or a website) would talk to a stateless service.
That stateless service would then communicate directly with the single SQL database.
The database would store everything: player statistics, game progress, inventory, financial transactions, all in one consistent, centralized location.
Some advantages of the single database model were strong guarantees enforced by the ACID properties, simplicity of development, and vertical scaling.
Scalability Crisis of Halo 4
During the development of Halo 4, the engineering team faced unprecedented scale challenges that had not been encountered in earlier titles of the franchise.
Halo 4 experienced an overwhelming level of engagement:
Over 1.5 billion games were played.
More than 11.6 million unique players connected and competed online.
Every match generated detailed telemetry data for each player: kills, assists, deaths, weapon usage, medals, and various other game-related statistics. This information needed to be ingested, processed, stored, and made accessible across multiple services, both for real-time feedback in the game itself and for external analytics platforms like Halo Waypoint.
The complexity further increased because a single match could involve anywhere from 1 to 32 players. For each game session, statistics needed to be reliably updated across multiple player records simultaneously, preserving data accuracy and consistency.
Inadequacy of a Single SQL Database
Before Halo 4, earlier installments in the series relied heavily on a centralized database model.
A single large SQL Server instance operated as the canonical source of truth. Application services would interact with this centralized database to read and write all gameplay, player, and match data, relying on the built-in ACID guarantees to ensure data integrity.
However, the scale required by Halo 4 quickly revealed serious limitations in this model:
Vertical Scaling Limits: While the centralized database could be scaled vertically (by adding more powerful hardware), there were inherent physical and operational limits. Beyond a certain threshold, no amount of memory, CPU power, or storage optimization could compensate for the growing volume of concurrent reads and writes.
Single Point of Failure: Relying on one database instance introduced a critical operational risk. Any downtime, data corruption, or resource saturation in that instance could bring down essential services for the entire player base.
Contention and Locking Issues: With millions of users interacting with the database simultaneously, contention for locks and indexes became a bottleneck.
Operational Complexity of Partitioning: The original centralized system was not designed for partitioned workloads. Retroactively introducing sharding or partitioning into a monolithic SQL structure would have required major rewrites and complex operational procedures, creating risks of inconsistency and service instability.
Mismatch with Cloud-native Architecture: Halo 4’s backend migrated to Azure’s cloud infrastructure, specifically using Azure Table Storage. Azure Table Storage is a NoSQL system that inherently relies on partitioned storage and offers eventual consistency. The old model of transactional consistency across a single database did not align with this partitioned, distributed environment.
Given these challenges, the engineering team recognized that continuing to rely on a monolithic SQL database would limit scalability and expose the system to unacceptable levels of risk and downtime. A transition to a distributed architecture was necessary.
Introduction to Saga Pattern
The Saga Pattern originated through a research paper published in 1987 by Hector Garcia-Molina and Kenneth Salem at Princeton University. The research addressed a critical problem: how to handle long-lived transactions in database systems.
At the time, traditional transactions were designed to be short-lived operations, locking resources for a minimal duration to maintain ACID guarantees. However, some operations, such as generating a complex bank statement, processing large historical datasets, or reconciling multi-step financial workflows, require holding locks for extended periods. These long-running transactions created bottlenecks by tying up resources, reducing system concurrency, and increasing the risk of failure.
The Saga Pattern solves these issues in the following ways:
A single logical operation is split into a sequence of smaller sub-transactions.
Each sub-transaction is executed independently and commits its changes immediately.
If all sub-transactions succeed, the Saga as a whole is considered successful.
If any sub-transaction fails, compensating transactions are executed in reverse order to undo the changes of previously completed sub-transactions.
See the diagram below that shows an example of this pattern:
Key technical points about Sagas are as follows:
Sub-transactions must be independent: The successful execution of one sub-transaction should not directly depend on the outcome of another. This independence allows for better concurrency and avoids cascading failures.
Compensating transactions are required: For every sub-transaction, a corresponding compensating action must be defined. These compensating transactions semantically "undo" the effect of their associated operations. However, the system may not always be able to return to the exact previous state; instead, it aims for a semantically consistent recovery.
Atomicity is weakened: Unlike traditional transactions, where partial updates are never visible, in a Saga, intermediate states are visible to other parts of the system. Partial results may exist temporarily until either the full sequence completes or a failure triggers rollback through compensation.
Consistency is preserved through business logic: Instead of relying on database-level transactional guarantees, Sagas maintain application-level consistency by ensuring that after all sub-transactions and compensations, the system is left in a valid and coherent state.
Failure management is built in: Sagas treat failures as an expected part of the system's operation. The pattern provides a structured way to handle errors and maintain resilience without assuming perfect reliability.
Saga Execution Models
The main aspects of the Saga execution model are as follows:
Single Database Execution
When the Saga Pattern was first introduced, it was designed to operate within a single database system. In this environment, executing a saga requires two main components:
1 - Saga Execution Coordinator (SEC)
The Saga Execution Coordinator is a process that orchestrates the execution of all the sub-transactions in the correct sequence. It is responsible for:
Starting the saga
Executing each sub-transaction one after another
Monitoring the success or failure of each sub-transaction
Triggering compensating transactions if something goes wrong
The SEC ensures that the saga progresses correctly without needing distributed coordination because everything is happening within the same database system.
2 - Saga Log
The Saga Log acts as a durable record of everything that happens during the execution of a saga. Every major event, starting a saga, beginning a sub-transaction, completing a sub-transaction, beginning a compensating transaction, completing a compensating transaction, ending a saga, is written to the log.
The Saga Log guarantees that even if the SEC crashes during execution, the system can recover by replaying the events recorded in the log. This provides durability and recovery without relying on traditional transaction locking across the entire saga.
Failure Handling
Handling failures in a single database saga relies on a strategy called backward recovery.
This means that if any sub-transaction fails during the saga’s execution, the system must roll back by executing compensating transactions for all the sub-transactions that had already completed successfully.
Here’s how the process works:
Detection of Failure: The SEC detects that a sub-transaction has failed because the database operation either returns an error or violates a business rule.
Recording the Abort: The SEC writes an abort event to the Saga Log, marking that the forward execution path has been abandoned.
Starting Compensations: The SEC reads the Saga Log to determine which sub-transactions had been completed. It then begins executing the corresponding compensating transactions in the reverse order.
Completing Rollback: Each compensating transaction is logged in the Saga Log as it begins and completes.
After all necessary compensations have been successfully applied, the saga is formally marked as aborted in the log.
Halo 4 Stats Service
Here are the key components of how Halo used the Saga Pattern:
Service Architecture
The Halo 4 statistics service was built to handle large volumes of player data generated during and after every game. The architecture used a combination of cloud-based storage and actor-based programming models to manage this complexity effectively.
The service architecture included the following major components:
Azure Table Storage: All persistent player data was stored in Azure Table Storage, a NoSQL key-value store. Each player's data was assigned to a separate partition, allowing for highly parallel reads and writes without a single centralized bottleneck.
Orleans Actor Model: The team adopted Microsoft's Orleans framework, which is based on the actor model. Actors (referred to as "grains" in Orleans) represented different logical units in the system.
Game Grains handled the aggregation of statistics for a game session.
Player Grains were responsible for persisting each player’s statistics. In Halo 4’s backend system, a Player Grain is a logical unit that represents a single player’s data and behavior inside the server-side application.
Azure Service Bus: Azure Service Bus was used as a message queue and a distributed, durable log. When a new game is completed, a message containing the statistics payload is published to the Service Bus. This acted as the start of a saga.
Stateless Frontend Services: These services accepted raw statistics from Xbox clients, published them to the Service Bus, and triggered the saga processing pipelines.
The separation into game and player grains, combined with distributed cloud storage, provided a scalable foundation that could process thousands of simultaneous games and millions of concurrent player updates.
Saga Application
The team applied the Saga Pattern to manage the complex updates needed for player statistics across multiple partitions.
The typical sequence was:
Aggregation: After a game session ended, the Game Grain aggregated statistics from the participating players.
Saga Initiation: A message was logged in the Azure Service Bus indicating the start of a saga for updating statistics for that game.
Sub-requests to Player Grains: For each player in the game (up to 32), the Game Grain sent individual update requests to their corresponding Player Grains. Each Player Grain then updated its player's statistics in Azure Table Storage.
Logging Progress: Successful updates and any errors were recorded through the durable messaging system, ensuring that no state was lost even if a process crashed.
Completion or Failure: If all player updates succeeded, the saga was considered complete. If any player update failed (for example, due to a temporary Azure storage issue), the saga would not be rolled back but would move into a forward recovery phase.
Through this structure, the team could ensure that updates were processed independently per player without relying on traditional ACID transactions across all player partitions.
Forward Recovery Strategy
Rather than using traditional backward recovery (rolling back completed sub-transactions), the Halo 4 team implemented forward recovery for their statistics sagas.
The main reasons for choosing forward recovery are as follows:
User Experience: Players who had their stats updated successfully should not see those stats suddenly disappear if a rollback occurred. Rolling back visible, successfully processed data would create a confusing and poor experience.
Operational Efficiency: Retrying only the failed player updates was more efficient than undoing successful writes and restarting the entire game processing.
Here’s how forward recovery works:
If a Player Grain failed to update its statistics (for example, due to storage partition unavailability or quota exhaustion), the system recorded the failure but did not undo any successful updates already completed for other players.
The failed update was queued for a retry using a back-off strategy. This allowed time for temporary issues to resolve without overwhelming the storage system with aggressive retries.
Retried updates were required to be idempotent. That is, repeating the update operation would not result in duplicated statistics or corruption. This was achieved by relying on database operations that safely applied incremental changes or overwrote fields as necessary.
Successful retries eventually brought all player records to a consistent state, even if it took minutes or hours to do so after the original game session ended.
By using forward recovery and designing for idempotency, the Halo 4 backend was able to achieve high availability.
Conclusion
As systems grow to support millions of users, traditional database models that rely on centralized transactions and strong ACID guarantees begin to break down.
The Halo 4 engineering team’s experience highlighted the urgent need for a new approach: one that could handle enormous scale, tolerate failures gracefully, and still maintain consistency across distributed data stores.
The Saga Pattern provided an elegant and practical solution to these challenges. By decomposing long-lived operations into sequences of sub-transactions and compensating actions, the team was able to build a system that prioritized availability, resilience, and operational correctness without relying on expensive distributed locking or rigid coordination protocols.
The lessons learned from this system apply broadly, not only to gaming infrastructure but to any domain where distributed operations must maintain reliability at massive scale.
References:
SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com.
Like
Comment
Restack
© 2025 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:36 - 6 May 2025 -
[Save the Date] Spend Connect at SAP Connect
October 6–8, 2025 | Las Vegas, NV USA
Introducing:
Spend Connect at SAP Connect
As a spend and finance management professional, you have learned to fuel your transformation with solutions to strategically manage spend across all categories.
This year, you’ll have an opportunity to join us at a brand new event as we expand our business suite to bring out the best in your business. Experience live demos and real-world case studies tailored to finance and spend management applications, and get an inside look at how SAP helps you work smarter and adapt faster. In addition, you will see how SAP Business Suite brings AI, data, and applications together to deliver real results across your industry and organization.
Spend Connect at SAP Connect
Mark your calendar for Spend Connect at SAP Connect, taking place October 6–8, 2025 in Las Vegas, Nevada.
It’s part of SAP Connect, our new annual event designed for business leaders and practitioners looking to maximize impact and resilience across their business in an unpredictable world. You’ll connect with peers and partners, experience live demos tailored to your priorities, and discover how SAP empowers you to move forward with confidence — no matter what lies ahead.
Registration for Spend Connect at SAP Connect opens June 2025.
Sign up now to be notified about our event!
SAP (Legal Disclosure | SAP)
This e-mail may contain trade secrets or privileged, undisclosed, or otherwise confidential information. If you have received this e-mail in error, you are hereby notified that any review, copying, or distribution of it is strictly prohibited. Please inform us immediately and destroy the original transmittal. Thank you for your cooperation.You are receiving this e-mail for one or more of the following reasons: you are an SAP customer, you were an SAP customer, SAP was asked to contact you by one of your colleagues, you expressed interest in one or more of our products or services, or you participated in or expressed interest to participate in a webinar, seminar, or event. SAP Privacy Statement
This e-mail was sent to you on behalf of the SAP Group with which you have a business relationship. If you would like to have more information about your Data Controller(s) please click here to contact webmaster@sap.com.
This promotional e-mail was sent to you by SAP Global Marketing and provides information on SAP's products and services that may be of interest to you. If you would prefer not to receive such e-mails from SAP in the future, please click on the Unsubscribe link.
To ensure you continue to receive SAP related information properly please add sap@mailsap.com to your address book or safe senders list.
by "SAP Connect" <sap@mailsap.com> - 10:05 - 6 May 2025 -
4 new product releases. Live demo each one.
New Relic
Tired of tracking down essential software information?
Join us tomorrow at 10am BST for our Product Expert Webinar: Intelligent Observability for Developer Productivity.
Experience a live demo and Q&A session on service architecture intelligence, discovering how it saves you time by automatically creating a searchable catalog of critical software knowledge from observability data.
Demo how to:- Streamline software development: Build a central knowledge base and auto-discover services.
- Speed up problem solving: Understand dependencies and visualise your landscape.
- Gain team performance insights: Enhance reliability, security, and compliance.
Walk through innovations that free information from team, tool, and systems silos. Discover how observability can be an engine for faster, more satisfying software development.Save your spot now 
by "New Relic" <emeamarketing@newrelic.com> - 05:10 - 6 May 2025 -
Data centers: How companies can scale their investments
Only McKinsey Perspectives
Brought to you by Alex Panas, global leader of industries, & Axel Karlsson, global leader of functional practices and growth platforms
Welcome to the latest edition of Only McKinsey Perspectives. We hope you find our insights useful. Let us know what you think at Alex_Panas@McKinsey.com and Axel_Karlsson@McKinsey.com.
—Alex and Axel
•
The investment imperative. As AI usage continues to surge, compute power—that is, the hardware, processors, memory, storage, and energy to operate data centers—is becoming one of the decade’s most important resources. In their analysis, McKinsey Partner Jesse Noffsinger, Senior Partner Mark Patel, and coauthors find that by 2030, data centers could require $6.7 trillion in global investment. Of that, $5.2 trillion would be used specifically for AI workloads, with the rest supporting traditional IT applications that use less compute power.
—Edited by Belinda Yu, editor, Atlanta
This email contains information about McKinsey's research, insights, services, or events. By opening our emails or clicking on links, you agree to our use of cookies and web tracking technology. For more information on how we use and protect your information, please review our privacy policy.
You received this email because you subscribed to the Only McKinsey Perspectives newsletter, formerly known as Only McKinsey.
Copyright © 2025 | McKinsey & Company, 3 World Trade Center, 175 Greenwich Street, New York, NY 10007
by "Only McKinsey Perspectives" <publishing@email.mckinsey.com> - 01:26 - 6 May 2025 -
Innovative Petrochemical Equipmrnt Solutions for Enhanced Operational Efficiency
Dear info，
We are a manufacturing and trading combo for petrochemical special equipment in China, with more than 11 years full experience and a profession sales team.
If you need any product, please reply me.We will provide professional services and appropriate prices.
Best regards !
Jason
Senior Sales Manager
Lianyungang Gongding Petrochemical Machinery Manufacturing Co., Ltd.
No. 9, Keji Road 4, Songtiao Development Zone, Lianyungang City and No. 20-2, Jinqiao Road, Dapu Industrial Park, Lianyungang City
Website:https://www.gdpetrochemical.com/
by "Jason" <Jason@gd-petrochemical.com> - 10:51 - 5 May 2025 -
The Executive Mastermind Comes to London – Will You Be There?
The Executive Mastermind Comes to London – Will You Be There?
Hi MD Abul,
I’d like to personally invite you to join us for an upcoming in-person session of The Executive Mastermind, happening the 28th of this month in London.
This isn’t your typical event. It’s a hands-on, executive-level conversation with leaders who are navigating ERP and digital transformations, just like you. Our focus is on Phase 0: the foundational stage that sets the tone for the entire journey.
The truth is, most digital transformations don’t fail because teams aren’t working hard. They fail because the early warning signs were missed or ignored. In this session, we’ll dig into what’s working, what’s not, and how organizations are preparing themselves for success before selecting a system or launching implementation.
Whether you're a CIO, project leader, or transformation executive navigating the complexities of an ERP or digital transformation, this is your opportunity to connect with peers who are facing similar challenges, exchange real-world insights, and walk away with practical strategies you can apply immediately.
Register here: Mastermind
Best regards,
Eric Kimberling
by "Eric Kimberling" <eric.kimberling@thirdstage-consulting.com> - 08:15 - 5 May 2025 -
high-quality electrical products
Hiinfo,
Good day.
I’m Luca Lau from Hangzhou Liyi Electrical Equipment Co., Ltd. We specialize in high-quality electrical products including transformers, switch status indicators, and temperature & humidity controllers. Our solutions are designed to improve efficiency and reliability in modern electrical systems.
I would welcome the opportunity to discuss your requirements and explore potential collaboration. Please let me know a convenient time for a call.
Best regards,
Luca
Hangzhou Liyi Electrical Equipment Co., Ltd
Email: luca@liyiep.com
Whatsapp / Tel: +86 13386526575
Web: www.liyiep.com
by "L-james" <L-james@liyipower.com> - 07:04 - 5 May 2025 -
Reliable Busbar Solutions for Industrial & Commercial Power Distribution
Dear info ,
I hope this message finds you well. My name is [Your Name] from Hunan Renyun Busway Co., Ltd., a trusted manufacturer of electrical power distribution systems since 2008.
Our expertise lies in designing and producing high-performance copper and aluminum busbars (busways), cable trays, and related solutions tailored for industrial plants, commercial complexes, data centers, and infrastructure projects. With compliance to ISO, UL, and IEC standards, our product portfolio includes low-voltage, medium-voltage, and high-current busbars engineered for durability, safety, and energy efficiency.
Many of our clients appreciate the reduced maintenance and long-term reliability our systems provide. Should you have upcoming projects requiring robust power distribution components, we’d be glad to review your specifications and share tailored recommendations.
Kindly let us know how we can assist.
Best regards,
HUNAN RENYUN BUSBAR CO.,LTD
Add:No. 147, East 11th Road, Langli Industrial Park, Economic Development Zone, Changsha City, Hunan Province,410100,China
Email:willy@rybusbar.com
Mob/Whatsapp:+86 13924302650
Web:www.rybusway.com
by "Sanjines Radeland" <radelandsanjines@gmail.com> - 06:31 - 5 May 2025 -
Custom Cable Trays | Engineered for Your Unique Needs
Dear info,
Complex projects require precision. Shanghai Qiongkai offers fully customized Cable Trays backed by:
** In-House Engineering Team: 20+ experts with 10+ years in structural design.
**3D Modeling & Prototyping: Validate designs before production.
** Seamless Integration: Compatible with Light Steel Keels and Seismic Brackets.Or request a product sample at any time.
Feel free to reply for further discussion your requirements.Best regards,
customerSales Manager | Shanghai Qiongkai
by "customer" <customer@qkindustry.com> - 05:45 - 5 May 2025 -
grow room dehumidifier with Microchannel High EER 4.0 ETL certificate
Hi info
We are dehumidifier factory in China.
USD175 dehumidifier in 50L/D
grow room dehumidifier with Microchannel High EER 4.0 with ETL certificate. USD 1288/unit for 506 pints(26.7℃/60%)
Here are some models which we can supply customization.
Thanks
by "Winford Gatica" <gaticawinford@gmail.com> - 02:26 - 5 May 2025 -
How Canva Collects 25 Billion Events a Day
How Canva Collects 25 Billion Events a Day
Forwarded this email? Subscribe here for moreACI.dev: The Only MCP Server Your AI Agents Need (Sponsored)
ACI.dev’s Unified MCP Server provides every API your AI agents will need through just one MCP server and two functions. One connection unlocks 600+ integrations with built-in multi-tenant auth and natural-language permission scopes.
Plug & Play – Framework-agnostic, works with any architecture.
Secure – Tenant isolation for your agent’s end users.
Smart – Dynamic intent search finds the right function for each task.
Reliable – Sub-API permission boundaries to improve agent reliability.
Fully Open Source – Backend, dev portal, library, MCP server implementation.
Skip months of infra plumbing; ship the agent features that matter.
Disclaimer: The details in this post have been derived from the articles written by the Canva engineering team. All credit for the technical details goes to the Canva Engineering Team. The links to the original articles and videos are present in the references section at the end of the post. We’ve attempted to analyze the details and provide our input about them. If you find any inaccuracies or omissions, please leave a comment, and we will do our best to fix them.
Every product team wants data. Not just numbers, but sharp, trustworthy, real-time answers to questions like: Did this new feature improve engagement? Are users abandoning the funnel? What’s trending right now?
However, collecting meaningful analytics at scale is less about dashboards and more about plumbing.
At Canva, analytics isn’t just a tool for dashboards but a part of the core infrastructure. Every design viewed, button clicked, or page loaded gets translated into an event. Multiply that across hundreds of features and millions of users, and it becomes a firehose: 25 billion events every day, flowing with five nines of uptime.
Source: Canva Engineering Blog Achieving that kind of scale requires deliberate design choices: strict schema governance, batch compression, fallback queues, and a router architecture that separates ingestion from delivery.
This article walks through how Canva structures, collects, and distributes billions of events daily, without drowning in tech debt and increasing cloud bills.
Their system is organized into three core stages:
Structure: Define strict schemas
Collect: Ingest and enrich events
Distribute: Route events to appropriate destinations
Let’s each look at each stage in detail.
Structure
Most analytics pipelines start with implementation speed in mind, resulting in undocumented types and incompatible formats. It works until someone asks why this metric dropped, and there is no satisfactory answer.
Canva avoided that trap by locking down its analytics schema from day one. Every event, from a page view to a template click, flows through a strictly defined Protobuf schema.
Instead of treating schemas as an afterthought, Canva treats them like long-term contracts. Every analytics event must conform to a Protobuf schema that guarantees full transitive compatibility:
Forward-compatible: New consumers must handle events created by old clients.
Backward-compatible: Old consumers must handle events from new clients.
Breaking changes like removing a required field or changing types aren’t allowed. If something needs to change fundamentally, engineers ship an entirely new schema version. This keeps years of historical data accessible and analytics queries future-proof.
To enforce these schema rules automatically, Canva built Datumgen: a layer on top of protoc that goes beyond standard code generation.
Datumgen handles various components like:
TypeScript definitions for frontends, ensuring events are type-checked at compile time.
Java definitions for backend services that produce or consume analytics.
SQL schemas for Snowflake, so the data warehouse always knows the shape of incoming data.
A live Event Catalog UI that anyone at Canva can browse to see what events exist, what fields they contain, and where they’re routed.
Every event schema must list two human owners:
A Technical Owner: Usually the engineer who wrote the event logic.
A Business Owner: Often a data scientist who knows how the event maps to product behavior.
Fields must also include clear, human-written comments that explain what they mean and why they matter. These aren’t just helpful for teammates. They directly power the documentation shown in Snowflake and the Event Catalog.
Collect
The biggest challenge with analytics pipelines isn’t collecting one event, but collecting billions, across browsers, devices, and flaky networks, without turning the ingestion service into a bottleneck or a brittle mess of platform-specific hacks.
Canva’s ingestion layer solves this by betting on two things: a unified client and an asynchronous, AWS Kinesis-backed enrichment pipeline. Rather than building (and maintaining) separate analytics SDKs for iOS, Android, and web, Canva went the other way: every frontend platform uses the same TypeScript analytics client, running inside a WebView shell.
Only a thin native layer is used to grab platform-specific metadata like device type or OS version. Everything else, from event structure to queueing to retries, is handled in one shared codebase.
This pays off in a few key ways:
Engineers don’t have to fix bugs in three places.
Schema definitions stay consistent across platforms.
Feature instrumentation stays unified, reducing duplication and drift.
Once events leave the client, they land at a central ingestion endpoint.
Before anything else happens, each event is checked against the expected schema. If it doesn’t match (for example, if a field is missing, malformed, or just plain wrong) it’s dropped immediately. This upfront validation acts as a firewall against bad data.
Valid events are then pushed asynchronously into Amazon Kinesis Data Streams (KDS), which acts as the ingestion buffer for the rest of the pipeline.
The key move here is the decoupling: the ingestion endpoint doesn’t block on enrichment or downstream delivery. It validates fast, queues fast, and moves on. That keeps response times low and isolates ingest latency from downstream complexity.
The Ingest Worker pulls events from the initial KDS stream and handles all the heavy lifting that the client can’t or shouldn’t do, such as:
Geolocation enrichment based on IP.
Device fingerprinting from available metadata.
Timestamp correction to fix clock drift or stale client buffers.
Once events are enriched, they’re forwarded to a second KDS stream that acts as the handoff to the routing and distribution layer.
This staging model brings two major benefits:
It keeps enrichment logic separate from the ingestion path, preventing slow lookups or third-party calls from impacting front-end latencies.
It isolates faults. If enrichment fails or lags, it doesn’t block new events from entering the pipeline.
Deliver
A common failure mode in analytics pipelines isn’t losing data but delivering it too slowly. When personalization engines lag, dashboards go blank, or real-time triggers stall, it usually traces back to one culprit: tight coupling between ingestion and delivery.
Canva avoids this trap by splitting the pipeline cleanly. Once events are enriched, they flow into a decoupled router service.
The router service sits between enrichment and consumption. Its job is simple in theory but crucial in practice: get each event to the right place, without letting any consumer slow down the others.
Here’s how it works:
Pulls enriched events from the second Kinesis Data Stream (KDS).
Matches each event against the routing configuration defined in code.
Delivers each event to the set of downstream consumers that subscribe to its type.
Why decouple routing from the ingest worker? Because coupling them would mean:
A slow consumer blocks all others.
A schema mismatch in one system causes cascading retries.
Scaling becomes painful, especially when some consumers want real-time delivery and others batch once an hour.
Canva delivers analytics events to a few key destinations, each optimized for a different use case:
Snowflake (via Snowpipe Streaming): This is where dashboards, metrics, and A/B test results come from. Latency isn’t critical. Freshness within a few minutes is enough. However, reliability and schema stability matter deeply.
Kinesis: Used for real-time backend systems related to personalization, recommendations, or usage tracking services. Kinesis shines here because it supports high-throughput parallel reads, stateful stream processing, and replay.
SQS Queues: Ideal for services that only care about a handful of event types. SQS is low-maintenance and simple to integrate with.
This multi-destination setup lets each consumer pick the trade-off it cares about: speed, volume, simplicity, or cost.
The platform guarantees “at-least-once” delivery. In other words, an event may be delivered more than once, but never silently dropped. That means each consumer is responsible for deduplication, whether by using idempotent writes, event IDs, or windowing logic.
This trade-off favors durability over purity. In large-scale systems, it’s cheaper and safer to over-deliver than to risk permanent data loss due to transient failures.
Infrastructure Cost Optimization
Here’s how the team brought infrastructure costs down by over 20x, without sacrificing reliability or velocity.
SQS + SNS
The MVP version of Canva’s event delivery pipeline leaned on AWS SQS and SNS:
Easy to set up.
Scaled automatically.
Integrated smoothly with existing services.
But convenience came at a cost. Over time, SQS and SNS accounted for 80% of the platform’s operating expenses.
That kicked off a debate between streaming solutions:
Amazon MSK (Managed Kafka) offered a 40% cost reduction but came with significant operational overhead: brokers, partitions, storage tuning, and JVM babysitting.
Kinesis Data Streams (KDS) wasn’t the fastest, but it won on simplicity, scalability, and price.
The numbers made the decision easy: KDS delivered an 85% cost reduction compared to the SQS/SNS stack, with only a modest latency penalty (10–20ms increase). The team made the switch and cut costs by a factor of 20.
Compress First: Then Ship
Kinesis charges by volume, not message count. That makes compression a prime lever for cost savings. Instead of firing events one by one, Canva performs some key optimizations such as:
Batch collecting hundreds of events at a time.
Compressing them using ZSTD: a fast, high-ratio compression algorithm.
Pushing compressed blobs into KDS.
This tiny shift delivered a big impact. Some stats are as follows:
10x compression ratio on typical analytics data (which tends to be repetitive).
~100ms compression/decompression overhead per batch: a rounding error in stream processing.
$600,000 in annual savings, with no visible trade-off in speed or accuracy.
KDS Tail Latency
Kinesis isn’t perfect. While average latency stays around 7ms, tail latency can spike over 500ms, especially when shards approach their 1MB/sec write limits.
This poses a threat to frontend response times. Waiting on KDS means users wait too. That’s a no-go.
The fix was A fallback to SQS whenever KDS misbehaves:
If a write is throttled or delayed, the ingestion service writes to SQS instead.
This keeps p99 response times under 20ms, even under shard pressure.
It costs less than $100/month to maintain this overflow buffer.
This fallback also acts as a disaster recovery mechanism. If KDS ever suffers a full outage, the system can redirect the full event stream to SQS with no downtime.
Conclusion
Canva’s event collection pipeline is a great case of fundamentals done right: strict schemas, decoupled services, typed clients, smart batching, and infrastructure that fails gracefully. Nothing in the architecture is wildly experimental, and that’s the point.
Real systems break when they’re over-engineered for edge cases or under-designed for scale. Canva’s approach shows what it looks like to walk the line: enough abstraction to stay flexible, enough discipline to stay safe, and enough simplicity to keep engineers productive.
For any team thinking about scaling their analytics, the lesson would be to build for reliability, cost, and long-term clarity. That’s what turns billions of events into usable insight.
References:
SPONSOR US
Get your product in front of more than 1,000,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing sponsorship@bytebytego.com.
© 2025 ByteByteGo
548 Market Street PMB 72296, San Francisco, CA 94104
Unsubscribe
by "ByteByteGo" <bytebytego@substack.com> - 11:36 - 5 May 2025
