Runware - Service degradation on some models – Incident details

All systems operational

Service degradation on some models

Resolved
Partial outage 30 %
Started 4 days agoLasted about 1 hour

Affected

Models

Partial outage from 2:10 PM to 2:53 PM, Degraded performance from 2:53 PM to 2:59 PM, Operational from 2:59 PM to 3:07 PM

Official Models

Partial outage from 2:10 PM to 2:53 PM, Degraded performance from 2:53 PM to 2:59 PM, Operational from 2:59 PM to 3:07 PM

Updates
  • Resolved
    Resolved
    This incident has been resolved.
  • Update
    Update

    Provider issue has been resolved and service is resuming as normal. We will continue to monitor.

  • Monitoring
    Monitoring

    Re-routing is happening and queues are starting to come down on model inference requests.

  • Update
    Update

    Issue has been observed by the provider and the root cause is being identified. We are working with them to resolve this and bring the capacity back online. We continue to re-route requests to minimise impact.

  • Update
    Update

    We are working to distribute inference requests for these models to other GPU providers to minimise impact until the networking issue has been resolved.

  • Identified
    Identified

    We have a GPU vendor networking issue that is impacting Z-image, Flux.2 and some other models.