The board revision arrived on a Tuesday. Firmware had been written against the datasheet, the datasheet turned out to describe revision A, and the silicon on the bench was revision B. Three weeks of work sat in the trash, and the hardware team learned about it from a bug report. That sequence, or something close to it, plays out in most embedded programs we have joined, inherited, or been asked to rescue. The code was rarely the problem. The decisions that surrounded the code were.
Embedded delivery punishes ambiguity more than any other kind of software engineering. You cannot patch a PCB overnight, you cannot hotfix a device that bricked itself during an update, and you cannot refactor a power budget. This article walks through the decisions that determine whether your program ships in months or stalls for quarters. It assumes you already know what a microcontroller is. It focuses on what goes wrong around it.
Reading time: 12 min
Key points
- Most embedded delays come from ambiguous requirements and deferred decisions, not slow coding.
- A layered architecture with a hardware abstraction layer makes board revisions survivable instead of triggering a rewrite.
- OTA updates, security baselines, and observability must be designed before the first device ships, not after the first field failure.
- A two to three week technical discovery that outputs a requirements baseline, interface contracts, and a risk register prevents the late surprises that stall programs.
Table of contents
What embedded systems development covers in real product teams
Embedded systems development is the coordinated delivery of hardware, firmware, and the surrounding software needed to ship and operate a device. It is not writing code for a microcontroller. The work spans board-level constraints, drivers, an RTOS or embedded Linux build, application logic, connectivity, and support for manufacturing. When a team treats it as isolated coding, hardware and software drift apart and integrate late, at the worst possible moment.
As a CTO, you should expect specific artifacts from this work, not just commits. A system architecture document. Interface contracts between subsystems. A test strategy that says what runs on the host and what runs on the target. A bring-up plan with a defined validation sequence. Teams delivering integrated embedded solutions produce these as a matter of course, because cross-discipline coordination is the actual job. Firmware development without them is a prototype, not a product.
Where embedded projects fail before the first line of firmware
Most embedded delays start with ambiguous requirements, not slow coding. We have reviewed enough stalled programs to see the pattern. Timing budgets never written down. Power modes undefined, so nobody knows whether the device must survive a year on a coin cell or an afternoon. Sensor accuracy assumed rather than specified. No error handling policy, which means every failure path gets invented by whoever happens to be holding the debugger at 11 p.m.
Two documents prevent most of it. A one-page requirements baseline stating what the device must do and measure. An interface map showing every connection between the MCU, sensors, radio, and cloud, with ownership assigned for each side. Without early interface definitions, firmware becomes the integration layer for every unresolved hardware and product question. Every ambiguity deferred shows up later as a firmware bug, because firmware is where all the deferred decisions finally collide.
Translating product behavior into engineering requirements that survive hardware changes
Requirements that survive board revisions are written as observable behaviors and constraints, then traced to interfaces and tests. Write what the device must do. Boot in under 800 milliseconds. Sample the accelerometer at 200 Hz with latency under 10 milliseconds. Run 14 months on the target battery. Queue telemetry offline for 24 hours and flush on reconnect. Each of those statements survives a chip swap. None of them mentions a register.
The counter-case is instructive. A team that writes requirements as specific register configurations has tied its product logic to one chip, and a silicon shortage or a board revision forces a complete rewrite. The fix is mechanical. Maintain a requirements-to-test traceability table, one row per behavior, and a definition of done for firmware features that includes a passing test on target hardware. If a requirement cannot name an observable behavior, it is not a requirement. It is an implementation note wearing a disguise, and it will cost you a sprint when the hardware changes.
Architecture decisions that lock in delivery speed
A small set of early architecture choices determines whether your team iterates weekly or gets stuck in long debug cycles. The task model. The interrupt strategy. How logging and telemetry reach the outside world. The update strategy, decided before the first device ships rather than after the first field failure. The configuration model, meaning how parameters live on the device and how they change.
The shape that keeps teams fast is a layered architecture. A hardware abstraction layer at the bottom, drivers above it, middleware for things like storage and protocol handling, application logic on top, and diagnostics threaded through all of it. The failure mode is familiar to anyone who has inherited an embedded codebase. Application logic calls vendor SDK functions directly, so every board change becomes a full rewrite. We have walked into codebases where a pin remapping on revision C touched 40 percent of the source files. That is not an engineering problem. That is an architecture decision made by omission, and it was made on day one.
MCU versus MPU choices that shape the whole stack
Microcontrollers optimize for deterministic control and low power. Microprocessors optimize for richer OS capabilities. The choice reshapes your entire stack, your hiring profile, and your update strategy, so it deserves more than a hallway conversation.
| Signal in your product | Points toward MCU | Points toward MPU |
|---|---|---|
| Timing behavior | Hard real-time, microsecond determinism | Soft deadlines, best effort |
| User interface | LEDs, small display, no UI | Rich UI, touchscreen, browser stack |
| Power source | Battery, months to years of life | Mains or large battery |
| Networking | One protocol, low throughput | Multiple stacks, TLS, local processing |
| Update model | Dual-bank firmware images | Package and container updates |
Our verdict for early-stage products is simple. Choose the simplest compute that meets your timing and update needs, with headroom for the features you can actually name. Choosing Linux because your backend team knows Linux is the most expensive form of comfort. When determinism and power dominate, an MCU wins even when the MPU would make development feel easier.
Bare metal versus RTOS tradeoffs under real deadlines
Bare metal can be faster for simple control loops. A single superloop, one timer, one ADC, done. An RTOS pays off when concurrency, timing isolation, and maintainability matter, and the signals are easy to spot. Multiple time-sensitive tasks with different periods. A networking stack that blocks. Complex peripheral interactions. A feature roadmap that grows past what one loop can sequence.
Delay the decision past those signals and you get ad-hoc scheduling. Flags set in interrupts, polled in the main loop, timing dependent on whatever code ran last. Shared-state bugs appear only under load and refuse to reproduce on the bench. Firmware development on that footing burns weeks per bug. The artifact that prevents it is a scheduling model document showing task priorities, inter-process communication mechanisms, and an explicit deadlock prevention rule. On FreeRTOS or Zephyr, that document takes an afternoon to write and saves a month of debugging.
Embedded Linux in products with operational complexity
Embedded Linux is a product decision as much as a technical one, because it changes your update strategy, your security posture, and your observability expectations the day you commit to it. Devices that justify it include gateways aggregating multiple protocols, edge compute nodes doing local inference or heavy preprocessing, products that need containers for third-party components, and teams that want faster iteration on user-space code without touching the kernel.
What most teams skip is ownership. Someone must own the kernel configuration. Someone must own the board support package and the build system, whether that is Yocto or Buildroot. Someone must own the CVE patch workflow, because a Linux device in the field is a small server with an attack surface. Assign those three owners by name before the first image ships. The counter-case matters just as much. Embedded Linux on a battery-powered device with strict timing requirements introduces boot times measured in tens of seconds and a power draw that kills the product, and no amount of development convenience outweighs a device that cannot survive a night off its charger.
The first two weeks of board bring-up set the tone for the whole program
Bring-up is a structured validation of power, clocks, memory, and peripherals, and it should produce repeatable evidence. Not a working demo. Evidence. The sequence is well established. Power rails measured against the schematic, with ripple checked on a scope. Reset behavior verified across power cycling and brownout. The clock tree confirmed against the intended frequencies. JTAG or SWD access established early, because losing debug access on a new board turns every later step into guesswork. Memory tests, then peripheral smoke tests one by one.
Three artifacts make this repeatable. A bring-up checklist. A lab notebook format, shared and timestamped, so the second board benefits from the first board’s findings. A minimal diagnostic firmware image that exercises each subsystem independently. Skip the evidence collection and something worse than delay happens. Firmware gets blamed for hardware faults, hardware gets blamed for firmware faults, and trust between the two teams erodes for the rest of the program.
Driver strategy that prevents vendor lock and rewrite cycles
A disciplined driver and hardware abstraction strategy isolates vendor SDK churn and makes hardware revisions survivable. The rule is a boundary. The vendor SDK gets wrapped behind stable interfaces that you own, and register-level assumptions never leak above that boundary into application code. Vendor SDKs change, chipsets get discontinued, and the vendor’s priorities are not your priorities. Your interface layer is the only thing standing between their roadmap and your product.
Concretely, define an interface contract for SPI, I2C, UART, ADC, GPIO, and timing services, with mockable interfaces so your application logic can be unit tested on the host without hardware in the loop. The failure mode is the team that calls the vendor SDK directly from business logic, then faces a complete rewrite when the vendor ships a breaking SDK update or the chipset goes end of life. We have seen a mid-size product lose a full quarter to exactly that. The interface layer costs days. Its absence costs quarters.
Debugging and observability without slowing the device to a crawl
Embedded observability is about designing for diagnosis, using lightweight logging, tracing, and fault capture that works under production constraints. Printf debugging over UART at 115200 baud changes your timing enough to hide the bug you are hunting. The techniques that work in the field are cheap and well proven. Ring-buffer logs in RAM, dumped on demand. Crash dumps capturing register state and the faulting call path. Watchdog reset reasons persisted across reboots. Event counters for anomalies that never quite crash anything. On-device health metrics reported alongside telemetry.
Define log levels and retention policies up front, and write a field-debug playbook so support engineers can pull diagnostics without a firmware engineer on the call. The trap to avoid is debug builds that change timing, because they hide the exact intermittent bug causing field failures and produce the unrepeatable “works on the bench” verdict. If your diagnostics only exist in debug builds, you do not have diagnostics in production. You have a lab instrument.
Testing strategy that does not wait for perfect hardware
A practical test pyramid for embedded starts on the host and moves toward the target. Host-based tests cover business logic, protocol parsers, and state machines, running in milliseconds on every commit with no hardware attached. Target tests cover drivers, timing behavior, memory, and power modes, running on real silicon in your continuous integration setup. System validation comes last, on assembled devices, because it is the slowest and least diagnostic layer.
Wire this into CI early. A GitHub Actions or Jenkins pipeline runs host tests on every commit, and dedicated hardware nodes in the lab run target tests nightly or per merge. Rely only on end-to-end testing on real hardware and you get late discovery, slow iteration, and bug reports that say “sometimes the device stops” with nothing else to go on.
When hardware in the loop earns its keep
Hardware in the loop testing earns its cost in three situations. Control systems, where the loop between sensor and actuator is the product. Timing-critical behavior, where you need to prove deadlines hold under load. And regression on sensor-actuator loops, where a firmware change six months in silently breaks a physical interaction no host test can see. Outside those cases, a well-mocked target test suite gives you more coverage per hour invested.
Integration between firmware, cloud, and mobile that avoids API ping pong
Device-cloud integration succeeds when identity, telemetry, and command models are defined early and versioned like any other interface. The failure is predictable. Firmware and backend teams test in isolation against different assumptions, ship incompatible changes in the same month, and spend the next sprint in API ping pong, each side convinced the other broke it.
Prevent it with contracts. Define message schemas for telemetry and commands, with a schema registry or at minimum a versioned protocol specification document. Define retry behavior, offline buffering limits, clock sync assumptions, and idempotency rules for commands, because a device that retries a command after a network timeout will execute it twice unless idempotency was designed in. Put a versioning policy in the same document. Teams delivering end-to-end software solutions handle this by treating the device protocol as a first-class API contract, owned jointly by firmware and backend leads, reviewed by both before any change ships. A one-page contract beats a month of reconciliation calls.
OTA updates and fleet safety as a core engineering requirement
Over-the-air updates are a reliability feature, not a convenience. Designing them as a feature means the device survives a failed update, a power loss mid-write, and a bad image that passes your tests anyway. The mechanics are established practice. A/B partitions or dual-bank firmware, so the device boots the previous image when the new one fails. Signed images, verified before boot. An update state machine that tracks download, verification, install, and confirmation. Staged rollouts with canary devices before the fleet.
Define your success metrics and your rollback trigger policy before the first device ships, not after the first incident review. What boot success rate counts as a failed rollout? What reverts automatically, and who decides? Without update state management, you get bricked devices, inconsistent fleet versions, and field incidents nobody can reconstruct, because nobody recorded which image was on which device when. A fleet you cannot update safely is a fleet you will eventually have to recall.
Security work that embedded teams underestimate early
Connected devices need a security baseline from day one, because retrofitting identity, secure boot, and update signing into shipped hardware is disruptive in a way web teams rarely appreciate. Some of it is baked into silicon and cannot be added later at all. Start with a threat model, even a rough one. Who would attack this device, what would they gain, and what does the blast radius look like if one device is compromised and its credentials are used against your cloud?
From there, establish a secure boot chain, key management, least-privilege services on the device, and a secure debug access policy that locks JTAG and SWD in production images. Anchor your review process to recognized practice. The OWASP Top 10 covers the common vulnerability categories, including broken access control and security misconfiguration, both of which show up on device backends constantly. The NIST Secure Software Development Framework defines practices for secure build, provenance, and vulnerability response that map directly onto embedded release engineering. Tie your security checklist to release gates rather than running a one-time audit. The failure mode is always the same shipment. Default credentials, an unsigned update path, and exposed debug ports on a device now sitting in thousands of homes.
Documentation that makes handover possible without a rewrite
Good embedded documentation is operational. It lets a new engineer reproduce builds, flash devices, run tests, and diagnose failures without archaeology. The must-have list is short and unforgiving. Build instructions that work on a clean machine. Toolchain versions, pinned exactly, because a GCC minor version bump changes code size and timing. Board revision history with pin mappings per revision. Protocol specifications. Test procedures with expected outputs.
Two artifacts make handover real. A first-day onboarding checklist that a new engineer completes unaided, from clone to flashing a working image. And a minimal set of architecture decision records explaining why the system is shaped the way it is, so the next engineer does not relitigate settled decisions. The failure mode is quiet and cumulative. Knowledge trapped in one senior engineer’s memory becomes a single point of failure for the entire product line, and you find out the week that person takes a vacation or an offer elsewhere.
Team structure that keeps hardware and firmware moving together
Embedded delivery improves when ownership is explicit across hardware, firmware, backend, QA, and manufacturing test, with one technical integrator accountable for system behavior. Name the roles. A system architect who owns the interfaces. A firmware lead and a hardware lead who each own their discipline but share the bring-up plan. A test engineer who owns the pipeline, not just manual verification. A backend lead for the device protocol. A manufacturing test owner, which is the role most often left unassigned.
A RACI matrix covering bring-up, protocol changes, release approvals, and incident response closes most of the gaps. The gaps it does not close are the ones nobody wrote down. Fixtures, production programming, and field diagnostics sit exactly at the boundaries between teams, and boundary work is where unowned work hides. Companies structuring their engineering capacity through dedicated engineering teams should insist on the same explicit ownership, whether the engineers sit in their office or yours. One integrator, accountable for the whole system, is worth more than any org chart symmetry.
Choosing an embedded partner without getting stuck with a body shop
Evaluate for senior ownership, evidence-driven engineering, and the ability to integrate across disciplines. Not headcount, not buzzwords on a capabilities deck. The questions that separate partners from body shops are specific. Show me a test strategy you wrote. Walk me through a bring-up you ran. What is your approach to update safety? What documentation does a client receive at the end of a milestone?
Ask for sample deliverables, a requirements baseline, an interface contract, a bring-up checklist, and judge the engineering culture they reveal. Run a short technical interview loop and define a paid trial milestone before committing to a program. The warning signs of a staff augmentation mindset are vague timelines, no plan for manufacturing support, and no approach to field failures. Sentice delivers embedded software as engineering ownership rather than capacity, and the distinction shows up in the artifacts a client can inspect within the first month. A partner who cannot show you their last bring-up checklist is telling you exactly what you will receive.
Evaluate a partner on evidence, not decks
If you are weighing an embedded program, bring us your board revision, your requirements, or your stalled bring-up, and we will walk through what a discovery would produce.
Frequently asked questions
How long does it take to reach a stable MVP firmware baseline?
Plan on eight to twelve weeks for a device of moderate complexity, assuming hardware is stable and requirements are baselined. The first four weeks go to bring-up and the abstraction layer. Stability comes from the test pipeline, so build that alongside the features rather than after them.
What artifacts should exist before committing to a PCB revision?
A validated bring-up report, a pin mapping review signed by the firmware lead, a power budget measured on the bench, and a list of known silicon issues with workarounds. If the firmware lead has not reviewed the schematic, the revision is premature. That review catches the pin conflicts that cost a respin.
Can we start firmware before hardware is finalized?
Yes, and you should. With a hardware abstraction layer and mockable peripheral interfaces, application logic, protocol handling, and state machines run on the host from week one. Target-specific work starts when the first board arrives. Budget roughly 60 percent of firmware as hardware-independent and it pays for itself at every revision.
What does production readiness mean for embedded software?
It means the firmware survives manufacturing, not just the lab. That requires a production test mode, programming and provisioning scripts the factory can run, traceable firmware versions per device, and a golden sample unit validating each production batch. Release it as a signed image with recorded checksums, and gate it on your security checklist.
How do we handle third-party modules and vendor SDK updates?
Pin the SDK version in your build and update it as a deliberate, scheduled task, not by accident. Wrap it behind your own interfaces, review the vendor changelog for breaking changes and security fixes, and run your full host and target test suite against the new version before merging. Never let vendor code updates ride along in an unrelated feature branch.
What is the minimum viable approach to device telemetry?
Device identity, firmware version, connection events, reset reasons, and a small set of health counters covering battery, temperature, and error rates. Ship that from the first field unit. Everything else, high-frequency sensor streams, diagnostics dumps, can be added later once the schema is versioned and the backend contract is stable.
Start with a technical discovery that produces evidence
Before you commit a program, run a two to three week technical discovery that outputs three documents. A requirements baseline written as observable behaviors. Interface contracts for every boundary, chip to driver, device to cloud, team to team. A risk register naming the decisions you have deferred and what each one costs per month of delay. This is the direct counter to the Tuesday-morning board revision that opened this article. Force the early decisions into documents and tests, and the late surprises stop being surprises. Your team can run this internally with the checklists above, or bring in a partner who has run it before. Either way, insist on evidence, not reassurance.
Scope your embedded program with us
Tell us where your program stands, and we will map out what a discovery would cover for your device, your stack, and your timeline.
About Sentice
Sentice is a boutique software engineering partner founded in 2013 by Roni Levi and Martin Petkovic, now headquartered in Skopje, North Macedonia. Rather than supplying individual developers, we build embedded teams that blend with a client’s culture, tech stack and goals, working together from a single office under our own technical leadership. We deliver dedicated software engineering teams, end-to-end software solutions, product development, and system and embedded engineering, and we act as technical advisors across the full development lifecycle, from specification and architecture through development, testing and support. We use AI coding tools across every project, and we help clients integrate AI into their own products through chatbots, MCP services, connected application layers and workflow automation. Some of the clients we started with more than a decade ago are still building with us today.