What Happened
On November 28, 2025, we experienced a service disruption affecting merchants using the Liquid email link snippet in their order notification emails. End customers clicking these "Edit Order" email links received 404 errors. The issue was introduced during a customization modification on or about November 21, 2025, reported on November 28 at 3:20 PM EST, and resolved by 5:20 PM EST (approximately 2 hours from report to resolution).
Timeline
- ~November 21, 2025: Issue introduced during customization modification
- November 28, 2025 3:20 PM EST (8:20 PM UTC): Issue reported - email links returning 404 errors
- November 28, 2025 5:20 PM EST (10:20 PM UTC): Fix deployed and service restored
- Detection to Resolution: ~2 hours
- Issue Present: ~7 days (undetected)
Impact
Who Was Affected:
- Merchants using the Liquid email link snippet in their order notification emails
- End customers clicking "Edit Order" email links received 404 errors
- Not Affected: Customers using the "View Order" status page link continued to work normally
- Not Affected: Merchants not using the Liquid email link snippet
- Not Affected: Enterprise customers on dedicated infrastructure
What Was Affected:
- Email-based customer portal access via Liquid snippet only
- Alternative access methods remained fully operational throughout the incident
Root Cause
During implementation of a customization for a specific merchant, an internal routing parameter was inadvertently changed without updating the corresponding authentication logic. This created a mismatch between the URL structure in email links and what our authentication system expected, affecting all merchants using the Liquid email link snippet.
Technical Details:
Our email links embed the order ID in the URL path (e.g., /orders/12345/edit/d). The authentication code was updated to look for a different parameter name, causing it to not find the order ID and reject valid authentication attempts.
Resolution
We deployed a backward-compatible fix that accepts both the old and new parameter names, ensuring email links work regardless of which format they use. This approach:
- Immediately restored service for all affected customers
- Maintains compatibility with any emails already sent
- Prevents similar issues if parameter names change in the future
What We're Doing to Prevent This
We've taken the following actions to prevent similar incidents:
Immediate Actions (Completed)
- ✅ Added comprehensive end-to-end testing for the email link authentication flow
- ✅ Implemented backward compatibility to handle both parameter formats
- ✅ Enhanced monitoring to detect authentication failures more quickly
Short-term Improvements (In Progress)
- Expanded test coverage for all customer portal access methods
- Improved change review process to identify impacts on shared authentication code
- Enhanced error monitoring to alert our team of unusual 404 patterns
Long-term Improvements (Planned)
- Standardized routing conventions across the customer portal
- Additional automated testing for critical user authentication flows
Our Commitment
This incident highlighted a gap in our testing coverage for a critical customer-facing feature. While the impact was limited and quickly resolved, we take any service disruption seriously.
We're committed to:
- Transparency: Sharing what went wrong and how we fixed it
- Continuous improvement: Learning from incidents to strengthen our systems
- Reliability: Investing in testing and monitoring to prevent future issues
If you experienced issues during this incident or have questions about this report, please contact our support team.
Technical Appendix
For customers interested in the technical details:
Authentication Flow
The customer portal email link authentication uses HMAC-SHA256 signatures to verify that links are legitimate and haven't been tampered with. The signature is computed over the customer ID and order ID, ensuring each link is unique and secure.
What Changed
- Before: Authentication code read order ID from the
id URL parameter
- Incident: Code was updated to read from
order_id parameter, but URL structure didn't change
- Fix: Code now accepts both
id and order_id parameters for maximum compatibility
Detection Delay
The issue went undetected for approximately 7 days between deployment (~November 21) and the first customer report (November 28). This delay highlights the importance of:
- Proactive monitoring - We have since enhanced error monitoring to detect similar authentication failures
- Usage analytics - The affected feature (Liquid email link snippet) is used by a subset of merchants, which contributed to the delayed detection
- E2E testing - Post-incident, we added comprehensive end-to-end tests for this authentication flow
Response Time
Once reported, the issue was resolved within 2 hours. Our team opted for a forward fix (backward compatibility) rather than a rollback, as it provided better long-term resilience and ensured any emails already sent would continue to work.
Alternative Access Methods
The customer portal can be accessed through multiple methods:
- Liquid email link snippet - Direct links with embedded authentication in email templates (affected by this incident)
- Status page "View Order" links - Links from order status pages (unaffected)
- Custom email integrations - Merchants using custom email implementations (unaffected)
This redundancy helped minimize impact during the incident, as only merchants using the specific Liquid email link snippet were affected.