Inexplicable 'incomplete generation' error

[mwm5945]

System Info

Sagemaker Realtime Inference endpoints
TGI Version 2.4.1
p4d: 4 A100, 96 CPU, 1152 GB mem

MAX_INPUT_LENGTH: '16128'
MAX_TOTAL_TOKENS: '16384'

Information

• Docker
• The CLI directly

Tasks

• An officially supported command
• My own modifications

Reproduction

Unsure, as it happens sporadically, with no clear correlation to payload/prompts, traffic, or anything else. After the incomplete generation takes place, all subsequent requests timeout, and the only way to recover is to restart the endpoint. This has mainly occurred with Llama3.1 70b,

Example Logs:

Success

{
    "timestamp": "2025-02-21T01:15:43.569760Z",
    "level": "INFO",
    "message": "Success",
    "target": "text_generation_router::server",
    "filename": "router/src/server.rs",
    "line_number": 407,
    "span": {
        "inference_time": "286.497811ms",
        "parameters": "GenerateParameters { best_of: None, temperature: None, repetition_penalty: None, frequency_penalty: None, top_k: None, top_p: None, typical_p: None, do_sample: false, max_new_tokens: Some(1), return_full_text: None, stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None }",
        "queue_time": "1.276254517s",
        "seed": "None",
        "time_per_token": "286.497811ms",
        "total_time": "1.563136309s",
        "validation_time": "384.15µs",
        "name": "generate"
    },
    "spans": [
        {
            "inference_time": "286.497811ms",
            "parameters": "GenerateParameters { best_of: None, temperature: None, repetition_penalty: None, frequency_penalty: None, top_k: None, top_p: None, typical_p: None, do_sample: false, max_new_tokens: Some(1), return_full_text: None, stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None }",
            "queue_time": "1.276254517s",
            "seed": "None",
            "time_per_token": "286.497811ms",
            "total_time": "1.563136309s",
            "validation_time": "384.15µs",
            "name": "generate"
        }
    ]
}

Custom Attributes

(printed out headers returned from TGI from calling app)

2025/02/21 01:15:43 custom attributes: {
    "X-Compute-Characters": [
        "11"
    ],
    "X-Compute-Time": [
        "1.563136309"
    ],
    "X-Compute-Type": [
        "8-nvidia-a100-sxm4-40gb"
    ],
    "X-Generated-Tokens": [
        "1"
    ],
    "X-Inference-Time": [
        "286"
    ],
    "X-Prompt-Tokens": [
        "3"
    ],
    "X-Queue-Time": [
        "1276"
    ],
    "X-Time-Per-Token": [
        "286"
    ],
    "X-Total-Time": [
        "1563"
    ],
    "X-Validation-Time": [
        "0"
    ]
}

Incomplete generation

{
    "timestamp": "2025-02-21T01:15:43.600876Z",
    "level": "ERROR",
    "message": "Incomplete generation",
    "target": "text_generation_router::infer",
    "filename": "router/src/infer/mod.rs",
    "line_number": 246,
    "span": {
        "name": "generate"
    },
    "spans": [
        {
            "parameters": "GenerateParameters { best_of: None, temperature: None, repetition_penalty: None, frequency_penalty: None, top_k: None, top_p: None, typical_p: None, do_sample: false, max_new_tokens: Some(800), return_full_text: None, stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None }",
            "name": "generate"
        },
        {
            "name": "generate"
        }
    ]
}

Incomplete Generation

{
    "timestamp": "2025-02-21T01:15:43.600876Z",
    "level": "ERROR",
    "message": "Incomplete generation",
    "target": "text_generation_router::infer",
    "filename": "router/src/infer/mod.rs",
    "line_number": 246,
    "span": {
        "name": "generate"
    },
    "spans": [
        {
            "parameters": "GenerateParameters { best_of: None, temperature: None, repetition_penalty: None, frequency_penalty: None, top_k: None, top_p: None, typical_p: None, do_sample: false, max_new_tokens: Some(800), return_full_text: None, stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None }",
            "name": "generate"
        },
        {
            "name": "generate"
        }
    ]
}

Expected behavior

incomplete generations do not block the entire model

[Himank17Gupta]

+1, facing exactly same issue. @mwm5945 did you get any lead further?

Here are the last retrieved logs after which requests started to timeout.

#033[2m2025-03-12T09:36:06.501161Z#033[0m #033[31mERROR#033[0m #033[1msagemaker_compatibility#033[0m#033[2m:#033[0m#033[1mcompat_generate#033[0m#033[1m{#033[0m#033[3mdefault_return_full_text#033[0m#033[2m=#033[0mtrue #033[3mcompute_type#033[0m#033[2m=#033[0mExtension(ComputeType("4-nvidia-a10g"))#033[1m}#033[0m#033[2m:#033[0m#033[1mgenerate#033[0m#033[1m{#033[0m#033[3mparameters#033[0m#033[2m=#033[0mGenerateParameters { best_of: None, temperature: None, repetition_penalty: None, frequency_penalty: None, top_k: None, top_p: None, typical_p: None, do_sample: true, max_new_tokens: Some(100), return_full_text: Some(true), stop: [], truncate: None, watermark: false, details: false, decoder_input_details: false, seed: None, top_n_tokens: None, grammar: None, adapter_id: None }#033[1m}#033[0m#033[2m:#033[0m#033[1mgenerate#033[0m#033[2m:#033[0m #033[2mtext_generation_router::infer#033[0m#033[2m:#033[0m #033[2mrouter/src/infer/mod.rs#033[0m#033[2m:#033[0m#033[2m246:#033[0m Incomplete generation

[mwm5945]

we didn't really find the root cause. We had a inference call every minute or so that was meant to serve as a health check of sorts, and turning that off seemed fix the issue...