Glossary: LLMs in Production

asyncio.Queue

Also written: Queue

A first-in, first-out line where one task puts items and another takes them. The streaming endpoint passes text pieces from the gateway to the response through one.