tag:status.sonix.network,2005:/historySONIX Status - Incident History2024-03-28T05:15:15+01:00SONIXtag:status.sonix.network,2005:Incident/198697262024-01-31T23:58:26+01:002024-01-31T23:58:26+01:00OS upgrade on ixp-kg-sw1<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>23:58</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>23:38</var> CET</small><br><strong>Verifying</strong> - Verification is currently underway for the maintenance items.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>22:01</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>13:53</var> CET</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>20:54</var> CET</small><br><strong>Scheduled</strong> - The mentioned switch(es) will be undergoing maintenance at this time. Traffic will be culled according to RFC 8327 before data plane disruption to avoid impact for multi-homed members.<br /><br />During this time the route server for SONIX Stockholm will be unavailable.<br /><br />Traffic between IXN and KN7 will also be affected.</p>tag:status.sonix.network,2005:Incident/198754262024-01-31T10:00:00+01:002024-01-31T13:48:07+01:00Dataplane crash on KN7 site<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>10:00</var> CET</small><br><strong>Resolved</strong> - The data plane on ixp-kn7-sw1 crashed due to a currently unknown issue at 08:57 UTC. We were able to remotely power-cycle the switch and it was up and serving traffic again at 09:46 UTC.<br /><br />The switch was planned to receive maintenance that would it had happened would had prevented the crash. Sadly, this incident occurred before the maintenance event.<br /><br />Sorry for any inconvenience caused by this incident.</p>tag:status.sonix.network,2005:Incident/195368352023-12-26T19:50:38+01:002023-12-26T19:50:38+01:00Firewall upgrade<p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>19:50</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>19:01</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>26</var>, <var data-var='time'>18:42</var> CET</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.sonix.network,2005:Incident/192819582023-12-02T16:00:00+01:002023-12-02T21:11:04+01:00IXN-KG site link down<p><small>Dec <var data-var='date'> 2</var>, <var data-var='time'>16:00</var> CET</small><br><strong>Resolved</strong> - The active link between IXN and KG has gone down. It has been replaced by a stand-by link.</p>tag:status.sonix.network,2005:Incident/190275702023-11-05T20:56:45+01:002023-11-05T20:56:45+01:00IXP Manager software upgrade<p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>20:56</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>19:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>18:15</var> CET</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.sonix.network,2005:Incident/186822092023-10-04T06:37:04+02:002023-10-04T06:37:04+02:00Switch outage in Interxion STO1<p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>06:37</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>00:28</var> CEST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>00:27</var> CEST</small><br><strong>Update</strong> - Switch is operational again.</p><p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>23:54</var> CEST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.sonix.network,2005:Incident/179019222023-07-19T22:47:46+02:002023-07-19T22:47:46+02:00OS upgrade on RS1<p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>22:47</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>22:00</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>18:06</var> CEST</small><br><strong>Scheduled</strong> - The mentioned service(es) will be undergoing maintenance at this time.</p>tag:status.sonix.network,2005:Incident/179005822023-07-19T22:25:09+02:002023-07-19T22:25:09+02:00Graphs are not updating<p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>22:25</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>19</var>, <var data-var='time'>16:13</var> CEST</small><br><strong>Investigating</strong> - There's a internal network issue that is affecting metric collection for Grafana and IXP Manager.<br />We're currently looking into the issue.</p>tag:status.sonix.network,2005:Incident/173333182023-05-22T23:45:14+02:002023-05-22T23:45:14+02:00Capacity upgrade for ixp-ixn-sw1<p><small>May <var data-var='date'>22</var>, <var data-var='time'>23:45</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>22:45</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>22</var>, <var data-var='time'>22:36</var> CEST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.sonix.network,2005:Incident/171113342023-05-01T20:22:18+02:002023-05-01T20:22:18+02:00Switch software upgrade<p><small>May <var data-var='date'> 1</var>, <var data-var='time'>20:22</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>19:30</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>19:28</var> CEST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.sonix.network,2005:Incident/171089772023-05-01T16:19:27+02:002023-05-01T16:19:27+02:00Route server connectivity maintenance<p><small>May <var data-var='date'> 1</var>, <var data-var='time'>16:19</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>15:26</var> CEST</small><br><strong>Update</strong> - Maintenance is completed, but we're taking some extra time to verify future features while we're at it</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>14:30</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'> 1</var>, <var data-var='time'>14:17</var> CEST</small><br><strong>Scheduled</strong> - A dependent switch will be undergoing maintenance during this time.</p>tag:status.sonix.network,2005:Incident/170036472023-04-29T16:55:28+02:002023-04-29T16:55:28+02:00Route server connectivity maintenance<p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>16:55</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>16:50</var> CEST</small><br><strong>Update</strong> - Post-upgrade checks failed, rolling back to earlier version.</p><p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>16:16</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>16:07</var> CEST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.sonix.network,2005:Incident/170036722023-04-29T16:49:23+02:002023-04-29T16:49:23+02:00Switch software upgrade<p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>16:49</var> CEST</small><br><strong>Completed</strong> - Pre-checks failed, we will post-pone this upgrade to another day after investigations complete.</p><p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>16:45</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>29</var>, <var data-var='time'>16:11</var> CEST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.sonix.network,2005:Incident/169325752023-04-21T20:52:24+02:002023-04-21T20:52:24+02:00OS upgrade of ixp-ixn-sw1<p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>20:52</var> CEST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>20:51</var> CEST</small><br><strong>Update</strong> - Scheduled maintenance is still in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>20:30</var> CEST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>20:14</var> CEST</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.sonix.network,2005:Incident/98276632022-05-19T18:21:11+02:002022-05-19T18:21:11+02:00Switch outage in Kista Gate<p><small>May <var data-var='date'>19</var>, <var data-var='time'>18:21</var> CEST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>16</var>, <var data-var='time'>21:42</var> CEST</small><br><strong>Update</strong> - New information: The fiber between Kista Gate and KN7 has suffered a cut and is expected to be repaired 02:00 CEST. Hopefully this will restore data plane service.</p><p><small>May <var data-var='date'>16</var>, <var data-var='time'>21:39</var> CEST</small><br><strong>Update</strong> - Today at 12:42 PM UTC (14:42 PM CEST) both links between Kista Gate and KN7 went down. Plan of action is being discussed.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>15:34</var> CEST</small><br><strong>Investigating</strong> - We discovered that the management plane on the IXP switch located in Kista Gate has stopped sending logs and metrics to our collection daemons. We are investigating. At this time it seems the problem does not affect any data plane traffic.</p>tag:status.sonix.network,2005:Incident/91776272022-01-30T12:54:47+01:002022-01-30T12:54:47+01:00OS upgrade on ixp-kn7-sw1 and ixp-kg-sw1<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>12:54</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>11:01</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>17:37</var> CET</small><br><strong>Scheduled</strong> - The mentioned switch(es) will be undergoing maintenance at this time. Traffic will be culled according to RFC 8327 before data plane disruption to avoid impact for multi-homed members.</p>tag:status.sonix.network,2005:Incident/91468652022-01-23T16:19:06+01:002022-01-23T16:19:06+01:00Update and reboot of IXP Manager<p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>16:19</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>16:16</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>16:13</var> CET</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.sonix.network,2005:Incident/91060162022-01-18T20:41:02+01:002022-01-18T20:41:02+01:00Firmware upgrade on ixp-kn7-sw1<p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>20:41</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>18:00</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>22:24</var> CET</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>tag:status.sonix.network,2005:Incident/91056822022-01-18T20:29:33+01:002022-01-18T20:29:33+01:00L2 forwarding degraded<p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>20:29</var> CET</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>18</var>, <var data-var='time'>20:29</var> CET</small><br><strong>Update</strong> - After the firmware update things appear stable for now</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>21:25</var> CET</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.sonix.network,2005:Incident/91027542022-01-17T14:13:46+01:002022-01-17T14:13:46+01:00IXP Manager Upgrade<p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>14:13</var> CET</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>14:05</var> CET</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>17</var>, <var data-var='time'>14:04</var> CET</small><br><strong>Scheduled</strong> - We will be undergoing scheduled maintenance during this time.</p>