Scientific & IT engineer - technical coordinator of a COPERNICUS operational service hosted on a cloud - Earth Observation M/F

Context

Since 2020, Magellium has been producing water, snow and ice cover products for Europe. These products, derived from satellite observations, indicate the presence of snow on the continental surface and the presence of ice on lakes and rivers, and are delivered by the Copernicus terrestrial service. They are generated as soon as Sentinel-1 and Sentinel-2 observations are made available by the European Space Agency (ESA).

Magellium coordinates the entire project, around which a consortium has been formed. Within the consortium, Magellium is the privileged contact with the customer. The consortium is made up of European laboratories and private companies responsible for the design, maintenance and upgrading of scientific snow and ice detection algorithms.

For its part, the Magellium team provides customer support, promoting the products to users on the one hand, and is also and above all responsible for the operational production of snow and ice data in near-real time (a few hours). This team is also responsible for developing and maintaining the scientific building blocks that process ice data. Sentinel-1 and Sentinel-2 satellite products, which serve as the basis for the creation of snow and ice products, are processed as soon as they are received on a European cloud, called DIAS. DIAS provides direct access to all Sentinel data, as well as a large-scale computing infrastructure and data distribution capabilities.

Job description

You will join a team consisting of a project manager, an operations manager and 2 thematic engineers, where you will be responsible for technical developments within the project. Your tasks will be as follows:

• Coordinate the technical development of activities within the team and with project partners: specify and plan technical tasks to be carried out, take part in project meetings with the customer.
• Evolution of the linked production system, to integrate new scientific algorithms or implement new functionalities for partners, customers, users and the team itself.
• Monitor product production on a daily basis in support of the operations manager
• Maintain the production system in operational conditions on DIAS, including data distribution.

The position will be filled in two stages. Firstly, you will need to understand the system architecture and the triggering mechanisms for product generation, so as to be able to detect and correct any anomalies, suggest optimizations and implement new functionalities.

Production monitoring and maintenance will be carried out primarily by the Operations Manager, but support will also be required, particularly during vacation periods.

Monitoring operations consists of :

• Detect any anomalies or delays in the day's production using the system dashboard or tickets opened by scientific partners (MANTIS tool), who are responsible for reporting any anomalies in the generation of their products.
• Identify errors and their causes on the basis of existing documentation. If the anomaly is new, document it in the project Git and in the operations manual, understand it, remedy it and, if necessary, patch it in anticipation of a possible next occurrence.
• Check that product access interfaces are working properly.

In the event of a system failure, either detected by the operations manager or by yourself, it may be necessary to consult the database of everything that is happening and has happened in the system, in order to identify the cause of the failure. If necessary, the faulty process or the entire system can be restarted. DIAS may also be responsible for disruptions, so it will be necessary to interface with their support team to inform them of the problems encountered and work with them to find a solution. Depending on the severity of the situation, communication with the customer may be required to inform partners and users of the system's operational status.

Some anomalies are blocking or recurring, and need to be corrected. The next step is to develop a patch to resolve the problem - either by acting at source or by automating error management - test the new version of the system on a test environment, and then deploy it on the production environment once its operability has been verified.

Therefore, when updating the algorithms, it will be necessary to deploy the corresponding new Docker image(s) and run simulations on the test environment prior to deployment on the production environment.

These manipulations require regular checks, and therefore rigor and organization. You need to be able to prioritize the correction of errors/anomalies according to their severity, frequency of occurrence, and the loss of resources they entail, and to carry out these corrections.

Occasionally, it may be necessary to run the calculation of certain products on virtual machines on DIAS (non-real-time, annual or monthly products, for example, or as part of a reprocessing campaign or the processing of an archive of satellite products). These will then be published on the dedicated user interfaces with the help of DIAS support. The publication of products on the catalog will be verified by consulting the API.

In view of the extension of the project and the introduction of new Copernicus products, you will have to specify how to integrate new scientific algorithms in close collaboration with the rest of the team and partners.

Generally speaking, you'll be asked to rely on existing documentation and tools, so that you can quickly become autonomous.

Writing skills will also be needed to help maintain this technical documentation, as well as to summarize events and actions in progress at weekly team meetings.

Effective communication with project partners and the DIAS support team will be a priority (in English only).

Technologies used

OpenStack, Terraform (nomad, vault, consul, packer and s3), Docker, Python, PostgreSQL( SQL/PLSQL), Java, View, Git, Kubernetes, RESTful APIs, bash

Medium-term development

Depending on requirements and the success of the position described above, the position could be extended in the medium term to include technical coordination of two or three geophysical processing chains for spatial data deployed on the cloud, with fairly similar characteristics.