![]() This means, for example, that the server can send a real-time ETA of how long your prediction will take to complete. (2) They allow the server to send multiple updates to the frontend. This can be a problem if your inference function takes longer than 1 minute to run or if many people are trying out your demo at the same time, resulting in increased latency. ![]() (1) They do not time out - most browsers raise a timeout error if they do not get a response to a POST request after a short period of time (e.g. The SSE protocol has several advantages over simply using HTTP POST requests: When the user’s request has finished processing, the Gradio server returns the result back to the user using server-side events (SSE). submits an input to your function), Gradio adds the request to the queue, and requests are processed in order, generally speaking (this is not exactly true, as discussed below). When a user of your app submits a request (i.e. Overview of Gradio’s Queueing Systemīy default, every Gradio demo includes a built-in queuing system that scales to thousands of requests. Most of the information in this Guide is relevant whether you are hosting your demo on Hugging Face Spaces or on your own server. This is an advanced guide, so make sure you know the basics of Gradio already, such as how to create and launch a Gradio Interface. queue() method as well as some other related parameters, and discuss how to set these parameters in a way that allows you to serve lots of users simultaneously with minimal latency. How can you configure your Gradio demo to handle the most traffic? In this Guide, we dive into some of the parameters of Gradio’s. Let’s say that your Gradio demo goes viral on social media - you have lots of users trying it out simultaneously, and you want to provide your users with the best possible experience or, in other words, minimize the amount of time that each user has to wait in the queue to see their prediction. Setting Up a Demo for Maximum Performance
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |