Información actualizada sobre la situación en Estrasburgo

Roubaix, Francia | El 18-03-2021

Información actualizada sobre la situación en Estrasburgo

Strasbourg datacentre: latest information

Roubaix, France | 18 March 2021 - 16:30 (CET)

Customer information:

1. Service Status

  • In the OVHcloud Control Panel, each affected customer can view the status of their services located in Strasbourg (SBG) - as detailed in the FAQ.
  • The OVHcloud Community platform is now available to interact with community members and support you.
  • By following the security protocol defined for on-premises operations: the restart will be carried out room by room, aisle by aisle and rack by rack.
  • Procedures for restarting services will be communicated to affected customers as details become available in the coming days.
  • Real-time monitoring of the restart of SBG by room and planned rack is available at: status.ovh.com/vms/sbg.

2. Commercial actions

  • At the beginning of the week, all customers concerned received a communication on the commercial measures that will be applied to their situation. For customers who have not received the email, a procedure has been indicated in the FAQ.

3. Plan of action

Priority 1: restoring services to SBG1, SBG3 and SBG4

  • Status summary for each datacentre

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

Datacentre operational.

 

Accomplished

Accomplished

First racks tested with success.

Handover to the customer today with visibility of the status in their OVHcloud Customer Control Panel.

SBG-2

Building non-operational.

Site secured.

 

 

Replacement of infrastructure in other datacentres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Datacentre operational.

 

Servers undamaged.

 

 

 

Accomplished

 

Accomplished

Test today on one rack and if conclusive provisional estimate for Friday 19 March for a progressive restart of the services.

SBG-4

Datacentre operational.

 

All floors and servers undamaged.

Accomplished

 

 

Accomplished

 

Provisional ETA: Monday, 22 March for gradual restart of services.

 

  • 136 people are mobilised on premises today
    • The team rotation schedule for the coming weekend has been established.
  • Cleaning of site:
    • Cleaning teams are active in SBG1 and SBG3 in all affected areas.
    • Some of the servers in SBG3 require specific cleaning prior to their return to service. We have started this process.

Priority 2: Provisioning of infrastructure for our customers in other datacentres

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), Londres (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.
  • Update on quantities delivered since 10th March:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

4519

 

NAS HA

TB

246

Public Cloud

VPS

VM

12353

Hosted Private Cloud

Host

Servers

485

 

Datastore

Number of zpool

522

 

 

Volume of TB

3067

  • 400 servers will be produced today.

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.

On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres. 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have set up a support plan for customers who have not yet activated their DRP.

Communications

  • The OVHcloud Community platform is now available to interact with community members and support you.
  • An FAQ is available here.
  • To follow updates about this situation in real time:
    • Our main communication tool remains "Task Status"
    • Status videos are available here:

- FR : https://twitter.com/olesovhcom/status/1371908744775360518

- EN : https://twitter.com/olesovhcom/status/1371934994751500288

Our mission remains to provide our customers with the highest quality of service, to support their online business.  and we know how important this is to them. We sincerely apologise any inconveniences related  this situation An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

------------

17 March 2021 - 16:30 (CET)

Customer information:

1. Service Status

  • Octave Klaba, founder of OVHcloud, has released a new video on the situation in Strasbourg (SBG).
  • In the Control Panel, each affected customer can view the status of their services located in Strasbourg (SBG) - as detailed in the FAQ.
  • The OVHcloud Community platform is now available to interact with community members and support you.

By following the security protocol defined for on-premises operations: restarting will be carried out room by room, aisle by aisle and rack by rack.

  • Procedures for restarting services will be communicated to affected customers as details become available in the coming days.
  • Real-time monitoring of SBG restart by room and planned rack via: status.ovh.com/vms.
  • Progressive updates to the status page for services and backups in Strasbourg (SBG), from today onwards, accessible via the FAQ.
  • Updating and tracking the percentage of services restarted per datacentre scheduled in the “Task Status” interface in the coming days.
  • A new order form has been set up (for Bare Metal Cloud) to offer concerned customers services similar to those they had at the Strasbourg (SBG) site.

 

2. Commercial actions

  • All concerned customers received, at the beginning of the week, a communication on the commercial measures that will be applied to their situation. For customers who have not received the email, a procedure has been indicated in the FAQ.
  • For each customer concerned:
    • A billing suspension notification for the month of March has been added in the Customer Control Panel.
    • Commercial actions have been indicated, the details of which will be communicated this week.

3. Plan of action

Priority 1: restoring services to SBG1, SBG3 and SBG4

  • Status summary for each datacentre

 

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

4 of 12 rooms were damaged

 

- Rooms 61AD/62AD: servers not affected

- Rooms 61E/62D: cleaning of servers underway

- Restored power to datacentre

 

- Backbone: reconnected

- Redeployment of the local network (Cabling finalised this Wednesday) and configuration today

 

 

A rack was tested today and if conclusive a provisional estimate of Friday, 19 March for a gradual restart of services after cleaning and drying servers.

 

SBG-2

Building out of use Site secured

 

 

Replacement of infrastructure in other data centres: Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Servers undamaged

 

 

 

- Restored power to datacentre

 

 

- New network power restored

- Redeployment of the local network this Wednesday 17th March (Cabling finalised).

 

Test tomorrow on a rack and if conclusive a provisional estimate of Friday, 19 March for a gradual restart of services.

SBG-4

All floors and servers were spared

- Restored power to datacentre

 

Redeployment of the local network carried out.

Provisional ETA: Monday, 22 March for gradual restart of services.

 

  • 116 people are mobilised at the premises today
  • Network redeployment: 
    • The SBG1, SBG3 and SBG4 datacentres will be connected to the network tonight.
  • Watercooling:
    • The entire system is operational in the SBG1/SBG3/SBG4 datacentres.
  • Cleaning of site:
    • Intensive mobilisation (24/24).
    • SBG1: finalisation of cleaning Wednesday, 17th March.

Priority 2: providing infrastructures in other datacentres for our affected customers

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres in Gravelines (GRA), Roubaix (RBX), Londres (LON), Warsaw (WAW), and Frankfurt (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks. Update on quantities delivered since 10th March:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

4200

 

NAS HA

TB

276

Public Cloud

VPS

VM

11998

Hosted Private Cloud

Host

Servers

434

 

Datastore

Number of zpool

466

 

 

Volume of TB

2950

 

  • 400 servers will be produced today.

Due to the current demand, delivery times on our Bare Metal services may be longer than usual. Our teams are fully mobilised, and we are working hard to deliver to our customers as quickly as possible, especially all affected customers.

On the other hand, Public Cloud, Web Cloud and VPS services are being delivered as normal in all datacentres. 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have set up a support plan for customers who have not yet activated their DRP.

Communications

- FR: https://twitter.com/olesovhcom/status/1370045226929299461

- EN: https://twitter.com/olesovhcom/status/1370045891806175232

Our mission remains to provide our customers with the highest quality of service, to support their online business.  and we know how important this is to them. We sincerely apologise any inconveniences related  this situation An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

--------

16 March 2021 - 16:00 (CET)

Customer information:

1.  Service Status

  • Progressive updates to the status page for services and backups in Strasbourg (SBG), from today onwards, accessible via the FAQ.
  • Monitoring of the % of service restarts per datacentre to be displayed in the "Travaux" interface in the coming days. Restarts will be carried out room by room, aisle by aisle and rack by rack.
  • Finalized procedures for restarting services will soon be communicated to customers.

2. Commercial actions

  • For each customer concerned:
    • A billing suspension notification for the month of March has been added in the Customer Control Panel.
    • Commercial actions have been indicated, the details of which will be communicated this week.

3. Plan of action

Priority 1: restoring services to SBG1, SBG3 and SBG4

  • Status summary for each datacentre

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

4 of 12 rooms were damaged

 

- Rooms 61AD/62AD: servers not affected

- Rooms 61E/62D: cleaning of servers underway

- Power restored

- Damaged UPS have been checked and are being replaced

- Backbone: temporary connection to SBG on 15th March

- LAN redeployment and configuration today

 

 

Provisional ETA: Monday, 22 March for gradual restart

SBG-2

Building out of use. Structural expertise carried out. Site secured.

 

 

 

 

Infrastructure replacement in other data centres Gravelines (GRA), Roubaix (RBX), London (LON), Warsaw (WAW), Frankfurt (FRA)

SBG-3

Servers undamaged

 

 

- Power restored today

- All inverters are operational

 

 

- New network room powered up today

- Redeployment of the local network on 17th March

Provisional ETA: Monday, 22 March for gradual restart

 

 

SBG-4

All floors and servers were spared

Power restored today

 

Redeployment of the local network today

Provisional ETA: Monday, 22 March for gradual restart

 

  • 101 people are mobilised and on premises
  • Network redeployment:
    • SBG connection to the backbone today (alternative routes to Paris and Frankfurt underway)
    • Reconnection of buildings and rooms:
      • Fibre optic redeployment in progress 
      • Configuration underway
  • Watercooling:
    • The entire system is operational.
  • Cleaning of site:
    • Intensive mobilisation (24/24).

Priority 2: Provisioning of infrastructure for our customers in other datacentres

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our other datacentres: Gravelines (GRA), Roubaix (RBX), Londres (LON), Warsaw (WAW), Francfort (FRA).
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks. Update on quantities delivered since 10th March:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

3939

 

NAS HA

TB

242

Public Cloud

VPS

VM

10731

Hosted Private Cloud

Host

Servers

372

 

Datastore

Number of zpool

393

 

 

Volume of TB

2704

 

  • 350 servers will be produced today.

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers and have set up a support plan for customers who have not yet activated their DRP.

Communications

- FR: https://twitter.com/olesovhcom/status/1370045226929299461

- EN: https://twitter.com/olesovhcom/status/1370045891806175232

Our mission remains to provide our customers with the highest quality of service, to support their online business and we know how important this is to them. We sincerely apologise for any inconveniences related to this situation. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

-------

15 March 2021 - 17:00 (CET)

Customer information:

1 - Server status

  • The status page for services and backups in Strasbourg (SBG) is now available in the FAQ.
  • For VPS service, a question about “zone vs datacentre (SBG)” has been added for clarification in the FAQ.

2 - Commercial actions

  • For each customer concerned:
    • A billing suspension notification for the month of March has been added in the Customer Control Panel.
    • Commercial actions have been indicated, the details of which will be communicated this week.

3 - Action plan

Priority 1: restoring services to SBG1, SBG3 and SBG4

  • Status summary for each datacentre

 

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

4 of 12 rooms were damaged

Temporarily repowered on 13th March and will be restored permanently on 16th March

- Backbone: temporary connection to SBG on 15th March

- Internal network to be redeployed on 16th March

Provisional ETA: Monday, 22 March

for gradual restart

SBG-2

Building out of use

Structural expertise carried out.  Site secured

 

 

Replacement of infrastructure in other datacentres

SBG-3

Servers undamaged

Temporarily repowered on 12th March and will be restored permanently on 16th March

- New network room to be deployed and powered on 16th March

- Internal network to be redeployed on 17th March

Provisional ETA: Monday, 22 March

for gradual restart

SBG-4

Servers undamaged

Power to be restored on 16th March

- Internal network to be redeployed on 16th March

Provisional ETA: Monday, 22 March

for gradual restart

 

  • 111 people are dedicated on-site to connecting network and electricity units (in rotation, 24/24), for: 
  • Power redeployment:
    • The electrical installations required to restart SBG1/SBG4 have been verified.
    • The high-voltage power supply for SBG3 is deployed and operational.
    • Inverter re-commissioning protocol underway today (verification of all UPS).

 

  • Network redeployment:
    • The SBG1 network room is being tested.
    • The new network room is being completed and will be supplied with electricity as soon as the inverters are verified.
    • The installation of the network equipment will be finalised this evening.
    • The connection of the backbone is scheduled for tomorrow.
    • The local network is being redeployed.

 

  • The cleaning of any soot in the SBG3 server rooms will begin on Tuesday, 16 March. The clean-up teams have arrived on site and are beginning deployed.

 

Priority 2: providing infrastructures in other datacentres for our affected customers

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our Roubaix (RBX) and Gravelines (GRA) datacentres.
  • We are committed to providing nearly 15,000 new servers to our customers in the coming weeks.
  • 500 servers will be produced today.

 

  • Update on quantities delivered since 10th March:

CLOUD UNIVERSE

PRODUCTS

 

QUANTITY

Bare metal

Bare metal

Servers

3658

 

HA-NAS

TB

220

Public Cloud

VPS

VM

8903

Hosted Private Cloud

Host

Servers

328

 

Datastore

Number of zpool

361

 

 

Volume of TB

2598

 

Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

  • Our Support teams and Technical Account Managers are in contact with our customers, and have set up a support plan for customers who have not yet activated their DRP.

Communications

- FR: https://twitter.com/olesovhcom/status/1370045226929299461

- EN: https://twitter.com/olesovhcom/status/1370045891806175232

 

Our mission remains to provide our customers with the highest quality of services to support their online activities, and we know how important this is to them. We sincerely apologise for the problems caused by this incident. An investigation into the cause is underway. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

-----------

14 March 2021 - 15:00 (CET)
 

Customer information:

1 - Server status

  • A page has been created listing the backups of services hosted in Strasbourg (SBG) managed by the customer or by OVHcloud along with their status, which will be available in the FAQ within the next few hours.
  • For Public Cloud and VPS customers, a question about “zone vs datacentre (SBG)” has been clarified in the FAQ today.

2 - Commercial actions

  • All affected customers will be notified about their suspended March bill in their Control Panel.
  • Details on how these measures will be implemented will be sent next week, and will also apply to March bills for impacted services that were already issued before 10th March.

3 - Action plan:

Priority 1: restoring services to SBG1, SBG3 and SBG4

  • Status summary for each datacentre

 

Datacentre

Situation

Electrical restart

Network restart

Server restart

SBG-1

4 of 12 rooms were damaged

Temporarily repowered on 13th March and will be restored permanently on 17th March

- Backbone: temporary connection to SBG on 15th March

- Internal network to be redeployed on 17th March

Provisional ETA: Monday, 22 March

for gradual restart

SBG-2

Building out of use

 

 

Replacing infrastructures in other datacentres

SBG-3

Servers undamaged

Temporarily repowered on 12th March and will be restored permanently on 16th March

- New network room to be deployed and powered on 16th March

- Internal network to be redeployed on 17th March

Provisional ETA: Monday, 22 March

for gradual restart

SBG-4

Servers undamaged

Power will return permanently on 17th March

Internal network to be redeployed on 17th March

Provisional ETA: Monday, 22 March

for gradual restart

 

  • A team of 60 people is working on site to connect mobile network and electricity units (working in rotation, 24 hours a day)
    • Cabling in progress.
    • Connection planned for tonight.
  • The drying out of the SBG3 buildings will be finished tonight.
  • The cleaning of any soot in the SBG3 server rooms will begin on Tuesday, 16 March.
  • The cooling system is operational :
    • Water cooling is protected from the cold weather (forecast locally on Thursday, 18 March and Friday, 19 March).
    • The pumps are operational.
  • Priority 2: providing infrastructures in other datacentres for our affected customers
     
  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our Roubaix (RBX) and Gravelines (GRA) datacentres.
  • We are committed to providing around 15,000 new servers in the coming weeks.
  • Update on quantities delivered since 10th March:

 

Statement

 

  • Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers
    • Our Support teams and Technical Account Managers are in contact with our customers, and have set up a support plan for customers who have not yet activated their DRP.

Communications

- FR: https://twitter.com/olesovhcom/status/1370045226929299461

- EN: https://twitter.com/olesovhcom/status/1370045891806175232

 

Our mission remains to provide our customers with the highest quality of services to support their online activities, and we know how important this is to them. We sincerely apologise for the problems caused by this fire. We will continue to communicate with the utmost transparency about the cause of the fire and its consequences.

An investigation is ongoing. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

------------

13 March 2021 - 16:00 (CET)
 

Customer information:

1 - Server Status

  • We have sent emails to all our customers informing them of the status of their services.

2 - Commercial actions

  • Billing has been suspended for Strasbourg (SBG) datacentre customers from the date of the incident.
  • Commercial measures have been set out to each affected customer.
  • Details on how these measures will be implemented will be sent next week, and will apply in particular to March bills for impacted services that were already issued before 10th March.
  • We have updated the FAQs.

 Our three priorities are as follows:

  • Priority 1: restoring services to SBG1, SBG3, and SBG4
  • Priority 2: providing infrastructures in other datacentres for our affected customers
  • Priority 3: Implementing all DRP (Disaster Recovery Plan) mechanisms with our customers

Action plan:

Priority #1:

  • Status summary for each datacentre

Datacentre

Situation

Electrical restart

Server restart

SBG-1

4 of 12 rooms were damaged

From Monday, 15 March

Progressive restoration of undamaged servers by Monday, 22 March

SBG-2

Building out of use

Audit and inventory of premises

Replacing infrastructures in other datacentres

SBG-3

Servers undamaged

Tests for the high voltage supply will be carried out this weekend.  Provisional restart on 15th March

A progressive restart of all services is estimated from Monday, 22 March.

 

SBG-4

Servers undamaged

Scheduled for Monday, 15 March

A progressive restart of all services is estimated from Monday, 22 March.

 

 

  • Mobile high-voltage network and electricity units arrived at the Strasbourg site today at 12pm.
  • 60 experts are currently working on site (in rotation over the weekend) in four teams, with the following missions:
  1. Team 1 (network) is tasked with connecting the mobile network unit and testing the network room in SBG1
  2. Team 2 (electricity) is working on high-voltage electricity connections
  3. Team 3 (electricity) is working on low-voltage electricity connections
  4. Team 4 is working on protecting and restoring the cooling systems

The aim is to reconnect, repower and restart the backbone access network from Monday, 15 March (following the authorisation of the ESR (Électricité de Strasbourg Réseaux)).

To repower the site:

  • Two generators have been rented since 11th March to power the temporary office.
  • One additional generator was rented to recharge the batteries of our generators following their activation during the incident.

 

  • Several service providers are currently working on site (30 people)
    • The water on the floor of the SBG3 building has been cleaned up
    • The SBG3 building is currently being dried out, which should be completed by Sunday evening
    • The cleaning of any soot in the server rooms of SBG3 will begin on Monday.
       
  • Repair of the damaged cooling system:
    • A crane has arrived on site.

Priority #2:

  • For customers who have been impacted, we are offering replacement infrastructures (Bare Metal, Hosted Private Cloud and Public Cloud) in our Roubaix (RBX) and Gravelines (GRA) datacentres.
     
  • We are committed to providing around 15,000 new servers in the coming weeks.
     
  • Update on quantities delivered since 10th March:
•	Update on quantities delivered since 10/03

Priority #3:

  • Backup mechanisms:
    • An overview of all backup mechanisms will be published in the FAQ today.
    • For Public Cloud and VPS customers, questions about “zone vs datacentre (SBG)” will be clarified in the FAQ today.
    • An ad hoc mechanism is in place to help our customers set up their DRP system.

Communications

- FR: https://twitter.com/olesovhcom/status/1370045226929299461

- EN: https://twitter.com/olesovhcom/status/1370045891806175232

Our mission remains to provide our customers with the highest quality of service, to support their online business. And we know how important this is to them. We sincerely apologise for the problems caused by this fire. We will continue to communicate with the utmost transparency about the cause of the fire and its consequences.

An investigation is ongoing. Whatever the cause of the incident, OVHcloud is currently evaluating potential technical and operational measures to bring solutions to all affected customers.

All of our communication channels, including our incident tracking platform, can be accessed so that you can stay informed of developments in real time.

---------

A 12 de marzo de 2021 - 14:00 (CET)

Información al cliente:

  • Octave Klaba, fundador de OVHcloud, ha publicado un vídeo online disponible en inglés y francés sobre la situación actual:

- FR: https://twitter.com/olesovhcom/status/1370045226929299461

- EN: https://twitter.com/olesovhcom/status/1370045891806175232

  • Hoy informaremos a nuestros clientes por correo electrónico sobre el estado de sus servicios. Asimismo, hemos creado un equipo de asistencia disponible las 24 horas, los 7 días de la semana, para prestar asistencia a todos los clientes afectados.

Queremos recordar que, en este momento, las tres prioridades de OVHcloud son las siguientes:

  • Prioridad 1: Restablecer los servicios en SBG1, SBG3 y SBG4
  • Prioridad 2: Poner a disposición de nuestros clientes infraestructuras en nuestros datacenters no afectados
  • Prioridad 3: Implementar en colaboración con nuestros clientes todos los mecanismos de DRP (planes de recuperación ante desastres)

Prioridad 1: Restablecer los servicios en SBG1, SBG3 y SBG4

  • Diagnóstico e inventario en progreso.
  • Limpieza y reparación de los edificios dañados en progreso para garantizar la seguridad de nuestros equipos.
  • Limpieza de la espuma y aspiración del agua en la vía pública finalizadas. Purga del tanque de suministro del circuito hidráulico finalizada.
  • Todas las conexiones de fibra de larga distancia están operativas.
  • SBG1 ha sufrido daños parciales (4 salas afectadas de las 12 existentes)
    • Ayer se realizaron las comprobaciones en la sala de red. Este fin de semana tenemos previsto realizar una prueba de funcionamiento eléctrico aislada, sin conectividad. Estimación del restablecimiento ordenado de la conectividad: a principios de la próxima semana.
    • Está previsto que el suministro energético se restablezca provisionalmente el lunes 15 de marzo.
    • Los servidores serán auditados y reactivados uno tras otro.
  • SBG3 no se ha visto afectado por el incendio
    • Al final de esta semana se instalará una sala de red que deberá estar operativa a principios de la semana del 15 de marzo.
    • Durante este fin de semana se realizarán pruebas para la alimentación de alta tensión. Estimación provisional para el restablecimiento de la alimentación: lunes 15 de marzo.
    • La reactivación progresiva de todos los servicios está prevista a partir del lunes 22 de marzo.
  • SBG4 no se ha visto afectado por el incendio
    • El plan para la reactivación de los servicios se ultimará a finales de esta semana.

Todos los servicios Email Pro afectados se restablecieron el jueves 11 de marzo, desde las 16:55. No se ha perdido ningún email.

Prioridad 2: Poner a disposición de nuestros clientes infraestructuras en nuestros datacenters no afectados

  • Estamos ofreciendo a nuestros clientes afectados infraestructuras de sustitución (Bare Metal, Hosted Private Cloud y Public Cloud) en los datacenters de Roubaix (RBX) y Gravelines (GRA).
  • Hasta el momento ya hemos entregado las siguientes soluciones a los clientes afectados:
    • Miércoles 10 de marzo: 2100 nuevos servidores
    • Jueves 11 de marzo: 630 nuevos servidores
  • La nueva organización de nuestras cadenas de montaje está operativa desde este viernes 12 de marzo por la mañana, y nos permitirá triplicar nuestra capacidad de producción (3000 a 4000 servidores por semana).
  • Así pues, nos comprometemos a producir cerca de 15 000 nuevos servidores en las próximas semanas.
  • Comunicaremos los plazos de entrega de los nuevos servicios lo antes posible.

Prioridad 3: Implementar en colaboración con nuestros clientes todos los mecanismos de DRP (planes de recuperación ante desastres)

  • Nuestros equipos de Soporte y nuestros Technical Account Managers elaborarán este fin de semana un plan de asistencia para aquellos clientes que todavía no hayan activado su DRP.

Comunicación

- FR: https://twitter.com/olesovhcom/status/1370045226929299461

- EN: https://twitter.com/olesovhcom/status/1370045891806175232

Nuestra misión consiste en ofrecer a nuestros clientes la más alta calidad de servicio para apoyar su actividad online. Así pues, somos plenamente conscientes de lo importante que esto es para todo ellos. Por ello, queremos transmitir nuestras más sinceras disculpas por los inconvenientes que esta incidencia haya podido causarles. Asimismo, nos comprometemos a proporcionar información lo más clara posible sobre las causas y el alcance de este suceso.

Ya hay en marcha una investigación. Independientemente de la causa del incidente, OVHcloud está analizando las posibles medidas técnicas y operativas para ofrecer una solución a los clientes afectados.

Puede consultar todos nuestros canales de comunicación, incluyendo nuestra plataforma para el seguimiento de incidencias, donde publicaremos en tiempo real información sobre la evolución de la situación.

--------------

A 11 de marzo de 2021 - 15:30 (CET)
 

Equipos:

  • El equipo de Recursos Humanos se ha desplazado hasta Estrasburgo para apoyar a los empleados que están trabajando en la zona afectada.
  • Se ha instalado una oficina temporal en la zona.
  • Asimismo, los equipos industriales y técnicos de OVHcloud también se han desplazado hasta el lugar para apoyar y ofrecer su experiencia.
  • Hemos reforzado el personal en la planta de producción francesa (Croix) con ayuda de voluntarios internos para poder desarrollar el plan de producción actual y hacer frente a la carga de trabajo adicional provocada por la incidencia.
  • La producción de servidores se realizará en horario ampliado de trabajo y durante el fin de semana.

Producción y datacenters:

  • Hemos empezado a ofrecer a nuestros clientes infraestructuras de sustitución (servidores dedicados, servicios de cloud privado y de cloud público) en los datacenters de Roubaix y Gravelines.
  • Para dar respuesta a la demanda, tenemos previsto desplegar cadenas de montaje adicionales en las próximas 48 horas para triplicar nuestra capacidad de producción.

En el recinto de Estrasburgo:

  • Diagnóstico e inventario en progreso durante el día de hoy.
  • Limpieza y reparación de los edificios dañados en progreso para garantizar la seguridad de las instalaciones para nuestros equipos.
  • Limpieza de la espuma y aspiración del agua en la calle finalizadas. Aspiración del depósito de agua prevista para hoy.
  • SBG2 deberá ser reconstruido casi por completo.
  • SBG1 ha sufrido graves daños.
    • Sala de red de SBG1 - Sala conservada tras la verificación preliminar. ETA: a principios de la próxima semana.
    • ETA provisional para la electricidad: lunes 15 de marzo.
    • En los días siguientes los servidores se irán activando pieza por pieza tras la auditoría.
  • En SBG4 las comprobaciones preliminares no han revelado ningún daño. El objetivo ahora es restablecer el suministro de electricidad durante la semana 12 y reactivar progresivamente todos los servicios.
  • SBG3 no se ha visto afectado por el incendio. El objetivo es restablecer el suministro de electricidad y de red durante la semana 12 y reactivar progresivamente todos los servicios.

Las tres prioridades de OVHcloud:

  1. Reservar infraestructuras en otros datacenters de OVHcloud para los clientes afectados. Disponemos de un stock de nuevos servidores, listos para ser entregados a la mayoría de los clientes afectados, en nuestros datacenters de Roubaix y Gravelines. En las próximas semanas reforzaremos aún más la disponibilidad en estos datacenters con la producción de cerca de 10 000 nuevos servidores. Ofreceremos más información sobre este procedimiento a los clientes afectados lo antes posible.
  2. Proteger las instalaciones. OVHcloud ya tiene acceso al recinto, por lo que se está procediendo a su limpieza y, posteriormente, al restablecimiento del suministro energético y de red en los tres datacenters restantes.
  3. Seguir evaluando junto con nuestros clientes el impacto para los servicios alojados en los datacenters afectados para encontrar la mejor solución en cada caso.
     

Acciones adicionales emprendidas por OVHcloud:

Red:

  • Todas las conexiones de fibra entrantes al datacenter funcionan correctamente.
  • Se han realizado todos los pedidos necesarios a los diferentes proveedores. ETA in situ: 11 de marzo.

Comunicación:

Nuestra misión consiste en ofrecer a los clientes la más alta calidad del servicio para apoyar su actividad online. Así pues, somos plenamente conscientes de la importancia de estos servicios para todos ellos. Por ello, queremos transmitir nuestras más sinceras disculpas por los inconvenientes que este incendio haya podido causarles o les esté causando. Asimismo, nos comprometemos a proporcionar información lo más clara posible sobre las causas y el alcance de este suceso.

Ya hay en marcha una investigación. Independientemente de la causa de esta incidencia, OVHcloud está analizando las posibles medidas técnicas y operativas para ofrecer una solución a los clientes afectados.

Puede consultar todos nuestros canales de comunicación, incluyendo nuestra plataforma para el seguimiento de incidencias, donde publicaremos en tiempo real información sobre la evolución de la situación.

--------

A 10 de marzo de 2021 – 21:00 (CET)

Antecedentes:

  • El miércoles 10 de marzo de 2021, a las 00:47, se declaró un incendio en una sala de SBG2, uno de los cuatro datacenters de OVHcloud situados en Estrasburgo. 
  • El incendio se dio por extinguido a primera hora de la mañana.
  • Afortunadamente, no ha habido que lamentar ningún herido.
  • El incendio destruyó principalmente el datacenter SBG2 y produjo daños parciales en el datacenter SBG1, donde cuatro de las doce salas han sido destruidas por el fuego. Los otros dos datacenters de OVHcloud en Estrasburgo, SBG3 y SBG4, no se han visto afectados por el incendio, aunque permanecen apagados por el momento.
  • Las instalaciones en Estrasburgo no están afectadas por la normativa Seveso. 
  • El origen del incendio ya está siendo analizado, y las autoridades competentes han iniciado un procedimiento de investigación para determinar las causas.

Acciones emprendidas por OVHcloud:

  • Nuestros equipos técnicos y comerciales en todo el mundo trabajan desde ayer para informar a nuestros clientes y paliar la indisponibilidad de nuestros centros de datos en Estrasburgo.
  • Octave Klaba, fundador de la empresa, se desplazó hasta el lugar de los hechos, junto con los equipos industriales y técnicos de la empresa.
  • Para consultar en tiempo real la información sobre la evolución de la situación:
  • Tarea asociada: http://travaux.ovh.net/?do=details&id=49484
  • Twitter: https://twitter.com/OVHcloud_ES
  • En las próximas horas publicaremos una nueva comunicación para nuestros clientes, así como un conjunto de FAQ para dar respuesta a las preguntas más comunes.
  • Para facilitar el tratamiento de sus solicitudes, le recomendamos que cree un tíquet de asistencia desde su área de cliente o consulte nuestro centro de ayuda.

Las tres prioridades de OVHcloud:

  1. Reservar infraestructuras en otros datacenters de OVHcloud para los clientes afectados. Disponemos de un stock de nuevos servidores, listos para ser entregados a la mayoría de los clientes afectados, en nuestros datacenters de Roubaix y Gravelines. En las próximas semanas reforzaremos aún más la disponibilidad en estos datacenters con la producción de cerca de 10 000 nuevos servidores. Ofreceremos más información sobre este procedimiento a los clientes afectados lo antes posible.
  2. Proteger las instalaciones en Estrasburgo. OVHcloud ya tiene acceso al recinto, por lo que se procederá a su limpieza y, posteriormente, al restablecimiento del suministro energético y de red en los tres datacenters restantes.
  3. Seguir evaluando junto con nuestros clientes el impacto para los servicios alojados en los datacenters afectados para encontrar la mejor solución en cada caso.

Desde OVHcloud, estamos haciendo todo lo posible para garantizar la continuidad del servicio a nuestros clientes:

  • Estamos trabajando en un plan para reanudar la actividad y restablecer la red en los tres datacenters no afectados (SBG3 y SBG4) o parcialmente afectados (SBG1), lo antes posible.
  • Le recomendamos que preste especial atención a los mensajes de correo electrónico recibidos, ya que, en tiempos de crisis, las actividades maliciosas (phishing, spam, solicitudes, etc.) se multiplican. Debemos estar más atentos que nunca para detectar estas acciones fraudulentas. 

Impacto para nuestras actividades:

  • Seguimos evaluando el impacto de esta incidencia, sobre todo para aquellos clientes cuyos datos estaban alojados en el datacenter destruido por el incendio.
  • Los servicios Web VOIP en Francia no se han visto afectados.
  • Todos nuestros servicios alojados en el resto de datacenters en Francia y otros países del mundo (15 de ellos en Europa) permanecen totalmente operativos. 

Nuestra misión consiste en ofrecer a los clientes una calidad del servicio óptima para apoyar su actividad online. Así pues, somos plenamente conscientes de la vital importancia de estos servicios para todos ellos. Por ello, queremos transmitir nuestras más sinceras disculpas por los inconvenientes que este incendio haya podido causarles o les esté causando. Asimismo, nos comprometemos a proporcionar información lo más clara posible y de forma regular sobre las causas y el alcance de este suceso.

Por último, también estamos analizando el impacto medioambiental y, en colaboración con las autoridades competentes, estamos siguiendo los procedimientos pertinentes para confirmar la ausencia de contaminación. En este momento, podemos confirmar que los residentes locales no corren ningún riesgo.

Seguimos evaluando permanentemente el impacto de esta incidencia. Ofreceremos información lo antes posible y de la forma más transparente posible sobre el progreso de nuestra investigación, así como sobre las soluciones que vayamos poniendo a disposición de nuestros clientes.

Puede consultar todos nuestros canales de comunicación, incluyendo nuestra plataforma para el seguimiento de incidencias, donde publicaremos en tiempo real información sobre la evolución de la situación.

(Intervención: http://travaux.ovh.net/?do=details&id=49484)

--------------------------------

A 10 de marzo de 2021 - 11:00 (CET)

Este miércoles 10 de marzo de 2021, a las 00:47, se declaró un incendio en SBG2, uno de nuestros cuatro datacenters situados en Estrasburgo. Queremos precisar que este establecimiento no está afectado por la normativa Seveso.

Los bomberos se desplazaron inmediatamente hasta las instalaciones para evacuar a los equipos y limitar la propagación del incendio. Desde las 02:54, procedieron al aislamiento total del recinto y de su perímetro. A las 04:09, el fuego ya había destruido SBG2 y seguía presentando riesgos para los datacenters vecinos. Finalmente, los bomberos consiguieron controlar el incendio. Desde las 05:30, el acceso a las instalaciones está prohibido por motivos obvios de seguridad, y el recinto permanece bajo control de la prefectura. En este momento, el incendio ya ha sido extinguido por completo. Afortunadamente, no tenemos que lamentar ningún herido, ni entre nuestros equipos ni entre los bomberos y los trabajadores de la prefectura, a los que queremos agradecer su ejemplar labor.

OVHcloud cuenta con un parque operativo de quince datacenters en Europa, por lo que nuestros equipos técnicos y comerciales están haciendo todo lo posible para ofrecer una solución a los clientes afectados, implementando servicios que permitan paliar la indisponibilidad de nuestros centros de datos en Estrasburgo. Nuestra misión consiste en ofrecer a los clientes una calidad del servicio óptima para apoyar su actividad online. Así pues, somos plenamente conscientes de la vital importancia de estos servicios para todos ellos. Por ello, queremos transmitir nuestras más sinceras disculpas por los inconvenientes que este incendio haya podido causarles o les esté causando. Asimismo, nos comprometemos a proporcionar información lo más clara posible sobre las causas y el alcance de este suceso.

En este momento estamos evaluando el impacto de esta incidencia. Ofreceremos información lo antes posible y de la forma más transparente posible sobre el progreso de nuestra investigación, así como sobre las soluciones que pongamos a disposición de nuestros clientes.

Puede consultar todos nuestros canales de comunicación, incluyendo nuestra plataforma para el seguimiento de incidencias (travaux.ovh.com, y la tarea asociada), donde publicaremos en tiempo real información sobre la evolución de la situación.

Service Presse | OVHcloud

press@ovhcloud.com

Área de clienteContactar con el equipo comercialWebmail OVHcloud Blog

¡Bienvenido/a a OVHcloud!

Identifíquese para contratar una solución, gestionar sus productos y servicios, y consultar sus pedidos

Conectar