Industry
Standards Organizations
Challenge
The client faced frequent timeouts, slow ingestion, legacy code limitations, lack of real-time observability, and poor scalability—hindering content delivery and platform reliability.
Solution and Results
Datavid helped ASTM modernize its ingestion architecture without disrupting daily operations—upgrading the platform, streamlining workflows, and stabilizing performance. The result was a faster, more reliable, and fully automated ingestion process with real-time visibility and a scalable foundation for future growth.
Progress MarkLogic, Node.js, AWS, python, Java, ReactJS
About ASTM
ASTM International is a globally recognized standards organization with over 30,000 members across 140+ countries. It has published over 12,000 voluntary consensus standards that support innovation and quality across industries.
Since 2024, Datavid has been working across three departments within ASTM—Publishing Services, Core, and Editorial—to modernize and optimize critical content workflows.
Setting the Scene
As ASTM scaled up its digital publishing operations, it faced a growing challenge: ageing infrastructure and complex ingestion processes made content delivery to subscribers and downstream systems unreliable and inefficient.
The goal was clear: enable 100K ASTM and 3rd party content assets (books, journals, standards, etc.) to be ingested within 24 hours—a critical capability to support revenue growth and operational resilience.
The Challenges
The client was facing multiple operational challenges.
The ingestion process was taking more than 48 hours to handle over 70,000 assets, which significantly delayed publishing content to customer-facing systems. Additionally, the platform was running on over 8+ years of legacy code, making updates and improvements slow.
Timeout errors and job failures were frequent, requiring manual intervention to maintain continuity. There was no real-time observability, which made it hard to monitor or debug issues when they occurred.
The system was not scaling well as content volume and frequency increased.
Still relying on outdated systems to publish standards at scale?
ASTM streamlined ingestion across 3 departments—improving reliability and cutting delivery times by 70%.
The Solution
To address these challenges, ASTM partnered with Datavid to modernize its ingestion architecture without disrupting day-to-day publishing operations.
The solution involved upgrading their infrastructure to MarkLogic 11, re-architecting ingestion workflows for scale, and optimizing processing logic to reduce system load.
The team also decoupled binary processing from the ingestion pipeline and stabilized the Node server to eliminate timeouts.
All work was delivered using an agile Scrum methodology, with close collaboration between Datavid and ASTM’s Editorial, Core, and Publishing teams.
The Outcomes
Post-implementation, the results were immediate and significant. Ingestion time dropped from 48 hours to just 14 hours for large datasets—an increase of over 70% in speed.
Timeout errors were completely eliminated, and the entire process became fully hands-off, requiring no manual intervention.
With improved visibility, real-time alerts, and predictive analytics in place, the team could proactively monitor system health and resolve potential issues before they became blockers.
Most importantly, this upgrade gave ASTM a scalable and resilient ingestion pipeline to support long-term growth and innovation in digital standards delivery.
Curious what’s possible in your environment?
