EN ▾

Drop-in uncensored LLM proxyhttps://api.llmproxyapi.com/v1

LangChain with Local LLM or a Hosted Endpoint

To run LangChain with local LLM style code, you only need a chat class that accepts a custom base URL. If you have no GPU, or do not want to babysit one, the same class can point at a hosted OpenAI-compatible endpoint instead, and your chains do not change.

Isometric desktop and cloud platform joined by a chain of glowing links, linking local and hosted models

Local or hosted: pick by the constraint that bites

People searching for LangChain with a local model usually want one of three things: privacy, no per-token bill, or independence from a vendor. A model on your own machine, served by a tool such as Ollama or llama.cpp, delivers all three if you have hardware that can run it at a usable speed. The trade is setup time, memory limits and the speed of whatever GPU or CPU you own.

A hosted endpoint trades those for a small usage cost. You send prompts over the network, and the model runs on someone else's hardware. Here that means an open-weight model, tuned to answer without content refusals for lawful adult use, served through an OpenAI-compatible API. Prompts are not used for training, but they do leave your machine, so a hosted service is the wrong answer if your rule is that data must never leave it.

The useful part for a LangChain developer is that the code barely differs. Local servers like Ollama and llama.cpp expose OpenAI-style endpoints, and a hosted endpoint speaks the same protocol. If you write your app against ChatOpenAI with a configurable base URL, you can develop locally and switch to a hosted model with an environment variable. That gives you a practical hybrid: iterate on a small local model, and use the hosted one when you want a larger model's behavior without owning the GPU.

Connect ChatOpenAI to the endpoint

Install the integration package, then create the model object with the three settings that matter.

pip install langchain-openai langchain-core
import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="uncensored",
    base_url="https://api.llmproxyapi.com/v1",
    api_key=os.environ["LLMPROXY_API_KEY"],
    temperature=0.7,
)

print(llm.invoke("Give me three names for a fantasy tavern.").content)

If this runs, everything else works the same way. Get your key from the API key page; new users can claim $0.50 of trial credit with one click there after signing up, no card needed. Pass the base URL explicitly in code rather than relying on an environment variable, because different library versions read different variable names and a silent fallback to the default OpenAI host would send your key to the wrong place.

The model name is the one place people trip. The only ID here is uncensored. A name like llama3 or gpt-4o will not be recognized.

A chain with a prompt template

The standard pipeline of prompt, model and output parser runs unchanged. This example summarizes text in a chosen tone.

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser

prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a writing assistant. Answer in a {tone} tone."),
    ("human", "Summarize in two sentences:\n\n{text}"),
])

chain = prompt | llm | StrOutputParser()

print(chain.invoke({"tone": "dry", "text": "Long article text goes here..."}))

For many inputs, use batch and cap the parallelism. A full key allows 8 concurrent requests and 300 requests per minute, so keep the setting at 8 or lower:

results = chain.batch(
    [{"tone": "dry", "text": t} for t in documents],
    config={"max_concurrency": 6},
)

A trial key allows only 2 concurrent requests and 60 per minute, so use max_concurrency of 2 while testing.

Streaming

The API streams over Server-Sent Events, and LangChain exposes this through stream. Use it for chat interfaces where a person watches the answer form.

for chunk in llm.stream("Write a short scene set in a lighthouse."):
    print(chunk.content, end="", flush=True)

One practical detail: a streamed request keeps its concurrency slot until the stream finishes. If your UI abandons a stream halfway, make sure the generator is closed, or your concurrency headroom shrinks. Async code uses astream in the same way.

Tools and agents

Tool calling is supported: the API accepts tools and tool_choice, streams tool_calls and takes results back as role: "tool" messages. LangChain wraps this in bind_tools.

from langchain_core.tools import tool

@tool
def dice(sides: int) -> int:
    "Roll a die with the given number of sides."
    import random
    return random.randint(1, sides)

llm_tools = llm.bind_tools([dice])
msg = llm_tools.invoke("Roll a twenty-sided die for the goblin attack.")
print(msg.tool_calls)

The result lists the calls the model wants, with names and arguments. Run them, append the results as tool messages and call the model again, or let an agent framework such as LangGraph run that loop. Agents make many calls per user request, so they hit the per-minute limit sooner than simple chains. Count calls per run before choosing a concurrency level.

The context window is 100,000 tokens for prompt plus completion, which is generous for conversation memory. Even so, trim old turns: you pay for the whole history on every call at $0.15 per 1M input tokens.

JSON output

JSON mode is available through response_format. In LangChain you can pass it with model_kwargs or use the structured output helper in JSON mode, and describe the fields in your prompt.

from pydantic import BaseModel

class Character(BaseModel):
    name: str
    role: str
    secret: str

structured = llm.with_structured_output(Character, method="json_mode")
out = structured.invoke(
    "Invent a character for a mystery story. Reply in JSON with keys name, role, secret."
)
print(out)

With method="json_mode" the prompt must tell the model which keys to produce; the Pydantic class then validates the parsed result. Keep a retry around it for the rare malformed reply.

What does not carry over

Two LangChain staples need a different source. The API has no embeddings endpoint, so OpenAIEmbeddings pointed here will fail. For retrieval over your documents, use a local embedding model or another embedding provider, and use this endpoint only for the chat step. It also has no image input or generation, no audio and no fine-tuning, so vision chains and multimodal loaders are out.

Error handling is mostly about 429 responses. The code rate_limited means you exceeded the per-minute cap, concurrency means too many parallel calls, and a 429 with Retry-After: 5 marks temporary overload. Set max_retries on ChatOpenAI so the client waits and retries. Failed and refused requests are not charged.

The response headers help with monitoring: X-Balance-USD shows your remaining credit and X-Request-Id identifies a call for support.

Switching between local and hosted with one variable

If you want the hybrid setup, put the three settings in environment variables and build the model in one function. Your chains import that function and never mention a vendor.

import os
from langchain_openai import ChatOpenAI

def make_llm():
    if os.environ.get("LLM_MODE") == "local":
        return ChatOpenAI(
            model=os.environ["LOCAL_MODEL"],
            base_url="http://localhost:11434/v1",
            api_key="not-needed",
        )
    return ChatOpenAI(
        model="uncensored",
        base_url="https://api.llmproxyapi.com/v1",
        api_key=os.environ["LLMPROXY_API_KEY"],
    )

The local branch assumes an Ollama server on its default port, which exposes an OpenAI-style route under /v1; check your own server's documentation for the right address and model name. Behavior will differ between models, so rerun your prompts after a switch. A prompt tuned for one model's habits can read differently on another, and tool calling quality varies with the model, so test those paths first.

A sensible workflow is to develop prompts against the hosted model while you have no GPU free, then decide which parts of the app must stay on your own machine. Sensitive documents can use the local branch, and public-facing creative features can use the hosted one.

Cost and next steps

Pricing is $0.15 per 1M input tokens and $0.50 per 1M output tokens, from prepaid credit with no subscription. A chain call with 2,000 input tokens and 500 output tokens costs 2,000 x $0.15 / 1M + 500 x $0.50 / 1M = $0.0003 + $0.00025 = $0.00055. A thousand such calls cost $0.55. Top-ups run from $20 to $500 by crypto (USDT on TRC20, USDC on Base), with a bonus of 5% from $50, 10% from $100 and 15% from $200. Details are on the pricing page.

For a plain-HTTP view of the same API, read the quickstart. The site also has a short overview of the pay-as-you-go service.

Questions and answers

Can LangChain use a hosted API instead of a local model?

Yes. ChatOpenAI accepts a custom base URL, so any OpenAI-compatible endpoint works. Set base_url to https://api.llmproxyapi.com/v1 and the model to uncensored.

Is this the same as running a model locally?

No. The model runs on a provider's servers, so prompts travel over the network, although they are not used for training. If data must stay on your machine, run a local server instead.

Does it work with LangChain embeddings?

No. There is no embeddings endpoint. Use a separate embedding model for retrieval and this API for chat generation.

How do I stop LangChain from hitting 429 errors?

Keep max_concurrency at 8 or lower on a full key, 2 on a trial key, and set max_retries on the client so it waits and retries.

Your key is one form away

Create an account, copy the key, change the base URL. That is the whole setup.

Get API key