The demo worked on a Tuesday. By Friday it was answering customers with fabricated policy citations, ignoring role permissions, and costing more per request than the support agents it was supposed to assist. Nobody had changed the prompt. Real users had simply arrived.
This is the gap between an AI prototype and AI development services. The prototype answers controlled questions against clean data with no latency budget and no permission model. The service ships, operates, and maintains a software capability that uses models, and that capability is judged by the same standards as the rest of your product. We have inherited enough of these systems to know exactly where they break. Below, we’ll unpack where these services sit in an engineering org, what deliverables actually predict success, when to buy off the shelf versus build custom, and how to evaluate a partner before you sign anything.
Reading time: 10 min
Key points
- AI development services cover the full arc from product definition through operations, and are judged by the same standards as the rest of your product.
- The deliverables that predict success are architecture decisions, evaluation assets, and operational runbooks, not model or framework selection.
- Most AI projects slip on data readiness, not on models, so data constraints should be explicit and testable before model work begins.
- Agent workflows fail on permissions and approval gates, not on reasoning, and integration work dominates the timeline.
Table of contents
AI development services are product engineering, not a model demo
AI development services cover the full arc of shipping a model-powered capability: product definition, data preparation, integration, evaluation, deployment, and operations. “Adding AI” to a roadmap is not a scope. It is a wish.
Demos mislead because they remove everything hard. The prompts are controlled. The data is clean. There are no permission boundaries, no latency budgets, no real edge cases. An internal support assistant that works looks different from one that demos well: it respects user roles, cites sources, logs every action, and degrades gracefully when the model is uncertain. That behavior is engineered, not prompted into existence by artificial intelligence on its own initiative.
The security lens is the same one you already apply. The OWASP Top 10 remains the baseline for software that includes AI components, because injection, broken access control, and logging failures do not disappear when a model joins the stack (OWASP Foundation).
Prototype success followed by production failure is the most common outcome in this space. The feature ships, real data arrives, and someone quietly turns it off three weeks later.
Where AI development services sit in a modern engineering org
The service boundary is cross-functional by nature. It touches product, backend, data engineering, ML, security, and DevOps, and a good partner makes ownership explicit so your internal team is not left coordinating the gaps between vendors.
A RACI matrix settles the questions that otherwise fester. Model changes, prompt changes, and retrieval index updates each need a named owner.
| Decision | Responsible | Accountable | Consulted |
|---|---|---|---|
| Model version change | ML Engineer | Tech Lead | Product Owner, DevOps |
| Prompt change | ML Engineer | Tech Lead | Product Owner |
| Retrieval index update | DevOps | Tech Lead | ML Engineer |
| Data access approval | Tech Lead | Product Owner | Security |
| Incident rollback | DevOps | Tech Lead | ML Engineer |
Without this, a broken retrieval pipeline at 2 AM becomes a finger-pointing exercise between your vendor and your platform team. With it, the on-call rotation knows who pings whom. If you already run strong platform and ML teams, you may need only a narrow build squad for one feature rather than a fully dedicated software engineering team, and a partner who recommends the smaller engagement is telling you something about their honesty.
The deliverables that matter more than a model choice
Ask a prospective partner which LLM they would use and you learn nothing. Ask for their artifacts and you learn everything. The deliverables that predict success are architecture decisions, evaluation assets, and operational runbooks, not framework selection.
A production-grade engagement produces a system architecture diagram, a threat model, a data flow diagram, an evaluation plan, golden test sets, monitoring dashboards, a rollback plan, and handover documentation. The most valuable single document is a written definition of done that specifies latency SLOs, cost-per-request ceilings, failure-handling behavior, and citation correctness thresholds. If those numbers are not written down before the build starts, they will be negotiated after launch, and you will lose that negotiation.
This is the line that separates production ML solutions from experimentation. A partner who cannot produce these artifacts is delivering a prototype regardless of what the statement of work says. The work product is the proof, not the proposal.
Choosing between custom build and off the shelf AI
Off-the-shelf wins when the workflow is standard and you can adapt your process to the tool. Generic meeting transcription, basic ticket summarization, and standard chatbot widgets are solved problems. Buying them is the correct call.
Custom build wins when you need deep integration, differentiated UX, strict permissions, or domain-specific behavior. Product-embedded copilots, domain search over proprietary documents, agent workflows that call internal APIs, and edge or embedded constraints all disqualify the SaaS option. One decision criterion settles most debates: if the feature needs to call three internal APIs with role-based access control, off-the-shelf is out.
Custom is also the wrong choice when your team cannot maintain the system post-launch or when the data sources are unstable. A configured tool that ships this quarter beats a bespoke platform that ships next year and dies the quarter after. When custom is genuinely the right path, treat it as end-to-end product development, not as a bolt-on experiment.
When AI consulting is enough and when you need engineering
Consulting is sufficient when you need use case selection, feasibility assessment, and a build plan. You need engineering when the work includes integrations, reliability, security controls, and an operational lifecycle.
The outputs differ sharply. AI consulting produces an opportunity map, a data readiness assessment, a target architecture, a build-versus-buy recommendation, and a risk register. Engineering produces a working service with CI/CD, monitoring, an on-call plan, incident playbooks, an evaluation harness, and documentation. Strategy documents do not page anyone at night.
The transition point is easy to name. When the recommendation says “build a RAG pipeline over your support docs,” the consulting phase has ended and engineering has begun. Teams that need a shipped feature within a quarter are poorly served by consulting-only engagements, because the gap between strategy and production is where most delays live. Plenty of strategy decks sit on shelves precisely because nobody scoped the implementation behind them.
Data readiness is the real critical path
Most AI projects slip on data, not on models. Access, quality, and semantics are unclear at kickoff and stay unclear until someone forces the issue. AI development services should make data constraints explicit and testable before any model work begins.
The artifacts are unglamorous and decisive: a data inventory document, source-of-truth decisions, PII classification, retention rules, and access approval workflows. The typical scenario is customer support knowledge scattered across tickets, docs, and Slack threads, with conflicting answers and no canonical version. Retrieval built on that mess ships confident wrong answers, which erode user trust faster than no AI at all.
Here is the test. If data access takes longer than model selection, data readiness is your critical path. Talent pressure makes this worse: Eurostat reports that in 2024 more than 10 million people were employed as ICT specialists across the EU, representing 5.0% of all employed people, and every one of those teams is competing for the same data engineering capacity (Eurostat).
Building with RAG when your knowledge changes weekly
RAG is the default when answers must be grounded in your current documents and systems. The engineering work sits in retrieval quality, permissions, and evaluation. Prompt styling is the smallest part of the system.
At a buyer level, the mechanics that matter are the chunking strategy, the indexing cadence, metadata enrichment, and permission-aware retrieval that filters results by user role before the model ever sees them. A “citations required” UX closes the loop: every answer includes a link back to the source document in your own system, with a confidence indicator users can act on.
The failure mode is hallucination that looks plausible because the model is fluent. Users who verify and find a fabricated answer do not come back for a second try.
What makes retrieval fail in production
Stale indexes, missing metadata, poor chunk boundaries, and no query rewrite strategy account for most production failures. Each one produces confident wrong answers, which are harder to detect than outright refusals because nothing looks broken on the dashboard. The system keeps responding. It keeps being believed. It keeps being wrong.
Fine tuning and custom models when behavior must be consistent
Fine tuning or custom training earns its cost when you need consistent formatting, domain language, classification accuracy, or latency control that prompting cannot reliably deliver. Prompting a general model into stable behavior is possible. Keeping it stable across a thousand requests a day is not.
The strong use cases are document classification, structured extraction from unstructured text, ranking, anomaly detection, and predictive scoring on tabular data. Each one needs a labeled dataset with a documented acceptance threshold per class, a labeling workflow, and a retraining cadence. Without those three artifacts, you are not buying an ML solution, you are buying a one-time training run.
Custom models are the wrong choice when you cannot sustain labeling, retraining, and monitoring. Total cost of ownership includes the pipeline, not the training run. The predictable failure is a fine-tuned model that drifts within weeks because nobody planned retraining triggers or data refreshes, and nobody notices until accuracy quietly collapses.
Agent workflows fail on permissions, not on reasoning
Agents become risky the moment they can take actions. The hard engineering is permissioning, approval gates, audit logs, and safe tool design. The planning loop is the easy part, and vendors spend far too much of your evaluation time demonstrating it.
Consider an agent that drafts a refund request, updates a CRM record, and opens a support ticket. Done well, every action passes a mandatory human approval gate before it executes. The supporting infrastructure is scoped service accounts with least-privilege access, tool allowlists, an action simulation mode for testing, and a full audit log of every proposed action. The NIST Secure Software Development Framework is a useful baseline here, because it treats secure development as a lifecycle rather than a checkpoint (NIST).
The failure mode is vivid. An agent that “helpfully” cancels subscriptions, modifies records, and sends emails because no approval gate existed. If a partner cannot describe their permission model in detail, the agent design is incomplete regardless of how impressive the demo looked.
Integration work is where timelines are won or lost
AI features rarely live alone. The schedule is dominated by identity, data sources, eventing, and UI integration into the product your users already know. Budget integration as the largest single block of work, not as a final sprint task.
The concrete list is long and unglamorous: SSO and RBAC mapping, audit logging, connectors to ticketing systems, CRM, the data warehouse, and internal APIs, plus webhook subscriptions and their retry semantics. A typical scenario plays out in a single week. An internal API changes its response schema, the connector keeps returning 200s, and retrieval silently degrades because nobody monitored the payload. Users see worse answers. Nobody sees an error.
The result is a great model behind a feature nobody can access safely, because identity, permissions, and data pipelines were treated as afterthoughts. Partners who deliver end-to-end software solutions scope this work in week one, because they have been burned by it before.
Evaluation that survives real users
“Looks good in a demo” is not a metric. You need an evaluation harness that measures quality on your data and your tasks, with regression testing before every release.
The harness has known components: a golden dataset, an adversarial prompt set, red-team prompts, tool-call correctness tests, retrieval relevance checks, and automated regression runs in CI. Acceptance criteria must be numeric and task-specific: task success rate, citation correctness, refusal correctness, latency SLOs, and cost per request. Refusal correctness deserves emphasis. The system should decline when it does not know, and a system that never refuses is a system that fabricates.
The failure this prevents is quality drift. A prompt tweak or a model version update passes manual review and breaks one specific user workflow, and nobody catches it for a month because no regression suite existed.
What to evaluate for RAG, agents, and classic ML
RAG systems need groundedness, citation accuracy, and retrieval precision measured separately, because a fluent answer over wrong retrieved context is the worst outcome. Agents need action correctness, safe refusal rate, and approval-gate compliance. Classic ML needs precision, recall, calibration, and drift detection. One suite rarely covers all three, and a partner who proposes a single generic score has not thought about your system.
MLOps and LLMOps are part of the service, not an add on
If a partner cannot describe monitoring, rollback, and incident response in concrete terms, you are buying a prototype. Production AI needs observability across prompts, retrieval, tool calls, and model versions, because a regression can enter through any of them.
The concrete artifacts are a prompt and version registry, feature flags for gradual rollout, model routing with fallback behavior, drift alerts, retraining triggers, and incident runbooks. A healthy incident looks like this: a monitoring dashboard surfaces a regression, an engineer diagnoses it through prompt logs, and a feature flag rolls the change back within an hour. DORA’s software delivery performance metrics, including lead time and change fail rate, are the operational baseline AI systems must meet, not an aspiration (DORA).
The failure is silent degradation that customer support notices first, weeks after the actual regression. By then the logs you need are gone.
Security and privacy decisions you must force early
Security posture is set by data flows, tenancy boundaries, and logging choices. These decisions belong before the UX, not after it. Once a system ships with a leaky logging pipeline, fixing it means re-architecting, not patching.
Force the concrete questions early: data minimization principles, encryption in transit and at rest, secrets management, audit logs, retention policies, PII redaction pipelines, and sandboxing of tool calls. A representative example is keeping sensitive customer data out of prompts sent to third-party model providers while still enabling useful retrieval, through redaction and access-controlled indexes. It is a solvable problem. It is rarely solved retroactively.
The OWASP Top 10 applies to AI-enabled systems the same way it applies to any web application, and treating the model as a special exempt component is how confidential content ends up in logs, provider caches, and third-party observability platforms (OWASP Foundation). If security decisions are deferred to “after we get it working,” the system will ship with gaps that require expensive rework.
The engagement model that fits startups and scaleups
For fast-moving product teams, the best engagement behaves like an embedded engineering team with clear interfaces, shared rituals, and ownership transfer. A ticket-based vendor queue does not survive requirements that change weekly.
The mechanics matter more than the label: weekly planning with your PM or tech lead, a shared repository, a shared CI pipeline, shared on-call expectations, documented handover milestones, and pair programming sessions. The strongest configuration is a senior engineer on the partner side acting as a technical lead counterpart to your internal tech lead, with a defined escalation path. Sentice operates this way as a boutique tech partner, with engineers working from one office and following your practices from week one, because context that lives in your team’s head cannot be transferred through tickets.
The counter-case is real. A throw-it-over-the-wall model is fine for stable, well-specified work. It fails when requirements evolve weekly and the context is tacit, and it fails expensively: code with no tests, no documentation, and a reverse-engineering project for your own team.
Scope your AI feature with production constraints from day one
Tell us about the workflow you want to automate, and we’ll help you define the evaluation plan, permission model, and rollout sequence before any code exists.
What a good discovery phase produces in two weeks
A strong discovery phase ends with a buildable specification, not a slide deck. Target users, success metrics, architecture, data access plan, evaluation plan, and a phased rollout strategy, all written down.
The document set is specific: a PRD-lite, a system architecture diagram, a data inventory, a risk register, a milestone plan with dates, and a written definition of done. A sensible rollout sequence runs an internal alpha with logging for two weeks, a limited beta with real users and guardrails for two weeks, then general availability with full monitoring. Each phase has exit criteria. Each exit criterion is measurable.
Skipping discovery to save two weeks is a false economy. Brooks’s Law from “The Mythical Man-Month” states that adding manpower to a late software project makes it later, and the teams that skip discovery are the ones adding bodies in month three (Carnegie Mellon University). If discovery does not produce a document your tech lead can hand to engineers, it was a conversation, not a phase.
Behaviors that predict you will get a demo, not a system
Certain vendor behaviors predict failure with high reliability. Skipping data work, avoiding evaluation detail, hand-waving security, and promising “human-level” performance without constraints are the four most reliable signals.
Push for evidence. Ask how they test quality and watch for a specific answer. Ask to see a monitoring dashboard from a past project. Ask them to define a rollback procedure and describe post-launch support. The clearest warning is a plan that amounts to “we will just prompt engineer it,” with no evaluation harness, no data pipeline, and no operational lifecycle. Prompt engineering is a technique inside a system. It is not a system.
Treat any partner who cannot show production artifacts from prior work as a risk, regardless of pitch quality. Scope blowups arrive in month three, when everyone discovers there is no evaluation, no monitoring, and no documentation. One more signal deserves its own sentence. A partner who refuses to discuss handover or ownership transfer is planning retention, not delivery.
A buyer side checklist for selecting AI development services
Use a single scorecard that forces evidence across four dimensions: production readiness, integration depth, evaluation rigor, and ownership transfer. Compare vendors on artifacts, not narratives. Slides are cheap. Runbooks are not.
| Dimension | Ask for | Fails when |
|---|---|---|
| Production readiness | Sample runbook, monitoring dashboard | Answer is a promise, not a document |
| Integration depth | Architecture diagram from a past project | Identity and RBAC never mentioned |
| Evaluation rigor | Sample evaluation report, golden set design | Quality is judged by manual review |
| Ownership transfer | Proposed RACI for your team | Handover is not in the contract |
Add these questions to every vendor call. Who is on call after launch? How are model changes reviewed and deployed? How are permissions enforced on tool calls? How is data isolated between tenants? How does rollback work? A partner who answers with specifics and shows real documents passes. A partner who answers with slides does not.
Frequently asked questions
How long does an AI development project take from kickoff to production?
Plan on a two-week discovery, four to eight weeks for a pilot with real integrations, and ongoing iteration after launch. Integration and evaluation dominate the timeline, not model selection. Any proposal that promises production in two weeks is scoping a demo and calling it production.
What data do we need to provide on day one?
You need a data inventory with sources, access patterns, quality notes, PII classification, and known gaps. You do not need perfect data. You need an honest map of what exists, what is missing, and who can approve access to it.
Can we build AI features without sending data to third-party model providers?
Yes, through on-premise or private-cloud model deployment, data redaction before inference, and local embedding models for retrieval. The tradeoffs are latency, cost, and model capability, so the decision belongs in discovery, weighed against your actual data sensitivity rather than a blanket policy.
How do we measure success for an LLM feature?
Define task-specific metrics: citation correctness for RAG, action accuracy for agents, and refusal correctness for safety. Build a golden dataset before launch and run regression tests against it before every release, in CI, automatically.
Who should own the system after launch?
Your internal team should own the runbook, the evaluation harness, and the on-call rotation. The partner should transfer documentation, access, and knowledge during a defined handover phase, with the handover written into the contract as a milestone with acceptance criteria.
What is the difference between RAG and fine tuning in practice?
RAG grounds answers in your current documents and updates when your data changes. Fine tuning shapes model behavior for consistent formatting or classification. RAG is the default for knowledge retrieval, and fine tuning is for behavior control you cannot get reliably from prompting.
The next step is a scoped pilot with production constraints
Treat the first engagement as a pilot that includes real integrations, real permissions, and real evaluation, not a sandbox demo. Pick one workflow where success is measurable and failure is survivable, and write the definition of done before any code exists. Require monitoring, rollback, and documentation from day one, and make handover a contract milestone. If a partner resists those constraints, keep looking.
Let’s scope your pilot
Bring one workflow, and we’ll pressure-test the data plan, permission model, and evaluation criteria in a free consultation.
About Sentice
Sentice is a boutique software engineering partner founded in 2013 by Roni Levi and Martin Petkovic, now headquartered in Skopje, North Macedonia. Rather than supplying individual developers, we build embedded teams that blend with a client’s culture, tech stack and goals, working together from a single office under our own technical leadership. We deliver dedicated software engineering teams, end-to-end software solutions, product development, and system and embedded engineering, and we act as technical advisors across the full development lifecycle, from specification and architecture through development, testing and support. We use AI coding tools across every project, and we help clients integrate AI into their own products through chatbots, MCP services, connected application layers and workflow automation. Some of the clients we started with more than a decade ago are still building with us today.