Benchmark data processing
Aviation
Business Intelligence
Market Analysis
Java
Spring Boot
Spring Integration
Docker
MariaDB
Development
QA
About this project
Benchmark data processing enables agile data-driven decisions about flight routes, pricing strategies, and market opportunities through processing and analyzing massive volumes of aviation industry data daily.
WRD partnered with Lufthansa Systems to supercharge the data processing pipeline by replacing their legacy system with a modern solution for flight connections and market share analysis.
Our streamlined approach achieved a 72x processing speed improvement, analyzing several gigabytes of data within just a few minutes, enabling Lufthansa to make quick decisions in the rapidly changing aviation market.
Client
Participants
Lufthansa Systems [Main contractor, Business analysis]
WRD [Development, QA]
Business value
- Comprehensive market analytics: The system handles both IATA DDS ticket data and estimate files, as well as various internal input files from Lufthansa's systems, providing verified market share data for all city pairs and aircraft cabin classes, enabling informed decision-making.
- Accelerated data processing: Our optimized system architecture achieves a 72x faster processing speed, enabling analysis of several gigabytes of data within minutes instead of hours, supporting rapid business decisions in a dynamic market.
- Reflecting real-world business relationships: We implemented a sophisticated carrier grouping system that enables partner airlines to be analyzed together, reflecting real-world business relationships.
- Automated routing calculations: We implemented a system that performs routing calculations automatically whenever the input data becomes available.
- Development performance boost: We enhanced Lufthansa Systems' delivery capabilities as an external team, bringing specialized expertise and fresh perspectives that accelerated development and improved overall project outcomes.
Challenges
- Processing massive data volumes: We created a system capable of handling ~150 million records and ~10GB of data from several input files in just a few minutes while maintaining processing consistency.
- Multi-step calculation logic: We engineered sophisticated algorithms that determine optimal routing and perform market share analysis. These algorithms incorporate complex aviation business rules while accounting for partner relationships and competitive positioning.
- Diverse data source integration: We built a unified processing framework that transforms and standardizes data from multiple heterogeneous carrier systems, IATA files, and internal Lufthansa formats into a consistent analytical model.
- Validating complex business logic: We implemented automated tests covering the entire decision space, with both step-by-step component testing and end-to-end tests to ensure accuracy across all scenarios. Our system design preserves inputs and outputs from each calculation step, making them available for future analysis and troubleshooting.
market combinations
analyzed
input records
per run
processing speed
improvement
Solution
- Integrated Data Processing Workflow: We implemented a complete data pipeline that processes flight schedule data through six modular steps to determine optimal connections, then incorporates IATA DDS ticket information for market share analysis.
- Specialized Data Handling: We implemented specialized parsers for each data source type, with transformation rules to standardize inputs from the diverse data sources, ensuring consistent processing methodologies.
- Simplicity-driven Performance: We focused on straightforward, efficient design principles rather than complex architectures, creating a system that excels at meeting performance challenges while remaining highly maintainable and adaptable to new requirements.
- Business-driven Approach: By deeply understanding Lufthansa's business needs and industry challenges, we were able to identify the right technical solution. This business-first mindset enabled us to create effective solutions precisely adapted to the airline market analysis domain, avoiding unnecessary complexity while providing flexible and rapid responses to the requirements.
- Effective Team Collaboration: As an external partner, we assigned specialists with the right expertise to the project, establishing an excellent day-to-day working relationship with the Lufthansa Systems team. This collaborative approach ensured smooth communication, efficient knowledge transfer, and ultimately contributed to the project's success.
Technology
Java
Spring Boot
Spring Integration
Docker
MariaDB
Let’s talk!
We’re here to help with your software development needs. Whether you have a question or want to discuss a project, feel free to get in touch. We look forward to connecting.