Jobs by Fastorial
Back to all jobs

Senior Market Data Engineer

6 views
BHFTDubaiRemote

5+ years

Posted 4 days ago

  • Airflow
  • AWS
  • Azure
  • Docker
  • GCP
  • Kafka
  • Python
  • Spark

Historical Data Capture and Storage

Design, develop, and maintain systems for the acquisition, storage, and retrieval of historical market data from multiple financial exchanges, brokers, and market data vendors.

Data Integrity and Accuracy

Ensure the integrity and accuracy of historical market data including implementing data validation, cleansing, and normalization processes.

Data Architecture Development

Build and optimize data storage solutions ensuring they are scalable, high-performance, and capable of managing large volumes of time-series data.

Versioning and Reconciliation

Develop systems for data versioning and reconciliation to ensure that changes in exchange formats or corrections to past data are properly handled.

Data Source Integration

Implement robust integrations with various market data providers, exchanges, and proprietary data sources to continuously collect and store historical data.

Data Access Tools

Build internal tools to provide easy access to historical data for research and analysis ensuring performance, ease of use, and data integrity.

Collaboration with Trading and Research Teams

Work closely with quantitative researchers and traders to understand their data requirements and optimize systems for data retrieval and analysis for backtesting and strategy development.

Performance and Scalability

Develop scalable solutions to handle growing volumes of historical market data including ensuring efficient queries and data retrieval for research and backtesting needs.

Required Skills and Experience

  • Commercial experience of financial instruments and markets (equities, futures, options, forex, etc.), particularly understanding how historical data is used for algorithmic trading.
  • Familiarity with market data formats (MDP, ITCH, FIX, proprietary exchange APIs) and market data providers.
  • Strong programming skills in Python (Go or Rust is a nice to have).
  • Familiarity with ETL processes (or other data pipeline architecture) and tools to clean, normalize, and validate large datasets.
  • Commercial experience in building and maintaining large-scale time-series or historical market data in the financial services industry.
  • Strong SQL proficiency: aggregations, joins, subqueries, window functions, indexes, query planning, and optimization.
  • Bachelor's degree in Computer Science, Engineering, or related field.

Preferred Qualifications

  • Experience in a proprietary trading firm or buy-side environment working with historical market data and vendors.
  • Experience with distributed data systems and tools such as Hadoop, Kafka, Spark, or similar technologies.
  • Proficiency in containerization orchestration: Docker, Airflow, SLURM tools.
  • Linux/Unix expertise, particularly in managing and optimizing systems for data storage and processing.
  • Experience with cloud-based storage solutions such as AWS S3, Google Cloud Storage, or Azure.