The extraction job had been running for three weeks without a single failure. Same model, same prompts, same type of documents.

 

Then one Tuesday, a routine request simply never finished. The GPU sat at full load. No error, no timeout, no exception in the logs. When we looked at the raw output stream, the model was printing the same JSON fragment over and over, and had been doing so for several minutes.

 

Nothing had changed in our code. Nothing had changed in the model. And yet, there it was.

 

This is what working with large language models in production actually looks like. And it is the part that almost nobody talks about.

The Myth of „Seamless” AI Integration

 

Every AI vendor promises the same thing: plug and play integration. Connect an API key or spin up an open source model, and your business problems start solving themselves.

 

The demo usually supports this story. In a clean, controlled environment, with carefully chosen examples, the model performs beautifully.

 

Then it meets production. Real documents, real users, real edge cases, and a technology that is fundamentally non-deterministic at its core.

 

The truth is simple: deployment is the easy part. The real work begins the day after, when you discover that an LLM is not a library you install, but a system you have to continuously observe, diagnose, and tune.

 

In this post, we want to share, honestly, the four problems we run into most often when integrating local LLMs into business systems. In Part 2, we will show how we solve them.

Problem 1: The Infinite Loop Trap

 

Let’s go back to that frozen Tuesday request.

 

In production systems, you rarely want an LLM to write free form prose. You need structured data, usually JSON, that the rest of your pipeline can parse and act on. So you do what every guide recommends: enforce a strict JSON schema and lower the temperature, typically to somewhere between 0.1 and 0.3, to make the output as consistent and repeatable as possible.

 

Here is the paradox: the very settings that make output reliable also make it possible for the model to trap itself.

 

At low temperature, generation becomes nearly deterministic, at every step, the model almost always picks the single most probable next token. As long as that path leads somewhere sensible, everything is fine. But if the model slips into a repetitive pattern, continuing that pattern is now always the most probable next step. The closing bracket that would end the JSON never becomes the top candidate. The model cannot escape its own statistics, and a routine API call turns into an endless resource drain.

 

We have seen this across model generations and sizes:

• qwen3:14b (fine tuned with QLoRA) fell into these internal loops repeatedly during testing.
• qwen3:32b handled the same workloads noticeably better, but still looped from time to time.
• gemma4:31b, the model we currently work with, is remarkably stable. And recently, it too got stuck in a loop.

 

That last point matters most. This is not a flaw of one weak model that a bigger or newer one fixes. It is a structural property of how these systems generate text under strict constraints. No model is immune, which means your architecture has to assume it will happen, and be ready for it.

 

For a business, the takeaway is uncomfortable but important: „it worked in the demo” tells you very little. Stability in production is not something a model has; it is something your system around the model has to provide.

Problem 2: The Gap Between Papers and Production

 

The internet is full of confident advice on how to work with LLMs. Academic papers, LinkedIn guides, and Twitter threads regularly report 95%+ accuracy thanks to the latest prompting technique or agentic framework.

 

Our experience: most of these methods do not deliver anywhere near the advertised results when applied to real business problems.

 

This is not unique to AI, it is a version of the reproducibility problem the scientific world has struggled with for years. A technique that shines on a curated benchmark, with clean inputs and a narrow task definition, often falls apart when it meets messy documents, ambiguous questions, and domain specific logic.

 

The practical consequence is that there are no shortcuts. The only way to know whether a technique works for your case is to benchmark it yourself, on your data, against your requirements. That means building evaluation sets, running comparisons, and spending days, sometimes weeks, proving or disproving claims that sounded settled online.

 

If you are planning an LLM project, this changes how you should budget. Evaluation and testing are not a final QA step. They are a core, ongoing part of the development work, often the largest part.

Problem 3: Building on Shifting Sand

 

Suppose you do everything right. You test the approaches, discard the ones that don’t hold up, and carefully tune a solution to your needs.

 

By the time you finish, there is a good chance someone has published a better approach, or a new model has been released that makes your carefully engineered workaround simply unnecessary.

 

Context windows are the classic example. You can spend two months building a sophisticated pipeline that chunks large documents, attaches metadata, and stitches partial answers together, all to fit your material into a model’s limited context. Then a new model ships with a context window large enough to take all of those documents at once, and your entire custom architecture becomes obsolete overnight.

 

This creates a genuine dilemma that every engineering team in this space faces:

• Build now, solve today’s limitation with custom engineering, knowing it may be thrown away in three months.
• Wait, hold off and hope the next generation of models makes the problem disappear, knowing it might not.

 

We will be honest: there is no universal answer. It is a judgment call that depends on how urgent the business need is, how expensive the workaround is, and how likely the limitation is to fall. What is universal is the mindset: treat every architectural decision in this ecosystem as temporary, and design so that components can be replaced without rebuilding everything around them.

Problem 4: The RAG Misconception, Why Naive Vector Search Fails (Especially in Polish)

 

Whenever a company wants to connect its internal documents to an LLM, the default answer is always the same: „Just build a RAG system.”

 

Retrieval Augmented Generation is a sound idea. But the basic, out of the box version of it is far more fragile than most articles admit.

 

A naive RAG pipeline works like this: your documents are cut into fragments, each fragment is converted into a vector (an embedding), and when a user asks a question, the system searches the database for fragments whose vectors are „closest” to the question, usually by cosine similarity, and hands them to the model as context.

 

The catch is in what „closest” means. Cosine similarity measures statistical similarity of the text, overlapping vocabulary and phrasing, not genuine understanding of what the question is about. It finds fragments that look like the question, which is not the same as fragments that answer it.

 

In Polish, this weakness is dramatically amplified. Polish is a highly inflected language: the meaning of a sentence is carried by word endings, grammar, and context to a degree that English simply doesn’t match. A retrieval system tuned to surface level word similarity will happily return fragments that share vocabulary with the question yet are completely irrelevant to what was actually asked.

 

The result, from the user’s perspective, is the worst possible failure mode: the model receives the wrong context and confidently produces a bad or hallucinated answer, with a straight face.

 

If your documents are in Polish and someone proposes a „standard RAG setup,” this is the question to ask: how will retrieval understand the question, and not just match its words?

Instability Is Not Failure, Pretending Otherwise Is

 

Everything above might sound like an argument against using LLMs in business. It is the opposite.

 

These models are genuinely capable of things that were out of reach a few years ago. But they are also unstable, non-deterministic, and sensitive to conditions in ways traditional software is not. Admitting that is not engineering failure, it is the prerequisite for building systems that actually hold up.

 

If you already run an LLM in production, here are the signs that your integration has left the honeymoon phase:

• Occasional requests that never complete, with no error to show for it.
• Techniques that scored brilliantly in an article but underperform on your own data.
• Architecture decisions invalidated by a model release before they paid for themselves.
• Retrieval that returns text similar to the question, but answers a different one.

 

If any of these sound familiar: this is normal. This is what the technology is like today. The difference between fragile and resilient AI systems is not avoiding these problems, it is expecting them and engineering for them.

What’s Next

 

In Part 2, we move from diagnosis to treatment. We will share the specific mechanisms we use every day: how we detect and break infinite loops, how we handle context limits without betting the whole architecture on them, how we make retrieval understand Polish instead of merely matching it, and how we force our AI agents to verify their own logic before an answer ever reaches a user.

 

Because a system that works „out of the box” is a myth. A system that works reliably after honest engineering is not.