This decision gets made badly in two directions. Teams keep classic CAN because\n“it has always worked” and then spend a year fighting bus-load and 8-byte frames.\nOr they jump to industrial Ethernet everywhere and discover it costs more per\nnode, needs managed switches, and still does not give them determinism unless\nthey configured it for it.
\nHere is how the three actually differ, and what each is bad at.
\nWhat it is: a multi-drop differential bus, up to 1 Mbit/s, 8 data bytes per\nframe, non-destructive bitwise arbitration by identifier, strong error detection\n(15-bit CRC, bit monitoring, form/stuff checks) with automatic retransmission.
\nWhat it is genuinely good at:
\nWhat it is bad at — state these before choosing it:
\nWhat it changes: up to 64 data bytes per frame, and a second, faster\nbit rate for the data phase (commonly 2–5 Mbit/s, 8 Mbit/s achievable with\ncare) while the arbitration phase stays at the classic rate. Better CRC (17- or\n21-bit) for the larger payload.
\nWhy it is often the right upgrade:
\nWhat it is bad at / what to watch:
\nNot one thing. “Industrial Ethernet” covers protocols with very different\ndeterminism stories, all on 100 Mbit/s or 1 Gbit/s PHYs.
\nWhat Ethernet buys you:
\nWhat it is bad at:
\nGreenfield-everything-Ethernet is rare. The common shape is CAN or CAN FD at the\nmachine edge for sensors and drives, aggregated by a gateway that speaks the\nfieldbus on one side and OPC UA / MQTT over Ethernet on the other, doing protocol\ntranslation, buffering across link outages, and store-and-forward. That gateway is\nalso the right place to put the security boundary, the data-model mapping, and the\nfirmware-update path — rather than pushing all of that cost into every edge node.
\nDesigning that seam well — which data is published at what rate, what is buffered,\nhow backpressure and reconnection behave — is usually more of the engineering\neffort than picking the physical layer.
\n", "image": "https://exubits.com/engineering/og/can-canfd-ethernet-industrial-links.png", "date_published": "2026-09-03T00:00:00.000Z", "date_modified": "2026-09-03T00:00:00.000Z", "authors": [ { "name": "Exubits Engineering", "url": "https://exubits.com/authors/exubits-engineering" } ], "tags": [ "Reference", "can-fd", "ethercat", "tsn", "profinet" ] }, { "id": "https://exubits.com/engineering/production-handover-pack", "url": "https://exubits.com/engineering/production-handover-pack", "title": "What a production hand-over pack has to contain", "summary": "It lets someone else build, test, certify and change the product without the original team: design data, a reproducible firmware build, factory tests with limits, the compliance file, an SBOM, and known issues.", "content_html": "A hand-over pack has one job: let a competent engineer who was not on the project\nbuild the product, prove it works, keep it compliant, support it in the field, and\nchange it — without phoning the original team. Anything less is a dependency, not\na delivery.
\nThis is the contents list we work to. It spans hardware, firmware, test, and\ncompliance, because a product that splits those across separate incomplete\nhand-overs has no owner for the seams.
\nNot “the latest files” — a release, tagged, with a revision and a date.
\nHardware
\nFirmware and software
\nWhatever regimes apply (EMC, safety, radio, environmental, sector-specific):
\nEvery project has one. A hand-over without it means the client discovers the list\none incident at a time.
\nRun the test that matters: give the pack to an engineer who was not on the\nproject and have them, using only what is in the box, (a) build the firmware and\nget a matching binary, (b) build and pass one unit through the factory test, and\n(c) answer “what change would invalidate the CE marking”. If any of the three\nneeds a phone call, the pack has a hole, and the hole is cheaper to fill now.
\nAssembling this properly is weeks of work, concentrated at the end of a project\nwhen budget and patience are thinnest, and most of it produces no visible feature.\nIt is genuinely tempting to ship the design files and a README and call the rest\n“support”.
\nThe cost of that shortcut is entirely borne later and by someone else: the first\nCM build that fails because the test limits were in someone’s head, the CVE\nresponse that takes three weeks because there is no SBOM, the field failure with\nno service procedure, the change that quietly invalidates a certification because\nnobody wrote down what the certification depended on. The pack is insurance, and\nlike insurance its value is invisible right up until the moment it is the only\nthing that helps.
\n", "image": "https://exubits.com/engineering/og/production-handover-pack.png", "date_published": "2026-09-03T00:00:00.000Z", "date_modified": "2026-09-03T00:00:00.000Z", "authors": [ { "name": "Exubits Engineering", "url": "https://exubits.com/authors/exubits-engineering" } ], "tags": [ "Reference", "v-and-v", "traceability", "certification" ] }, { "id": "https://exubits.com/engineering/schematic-review-before-layout", "url": "https://exubits.com/engineering/schematic-review-before-layout", "title": "What a schematic review should catch before layout starts", "summary": "A schematic review catches the errors that get far more expensive after layout: power sequencing and budgets, reset and strapping, every net's return path, connector pinout and ESD, test access, and BOM risk.", "content_html": "The schematic review is the last cheap checkpoint. A missed pull-up found here is\na two-minute edit. Found after layout it is a re-route; found after fab it is a\nbodge wire and a respin; found in the field it is a recall. The review’s purpose\nis to spend an hour now to not spend a month later.
\nThis is the checklist we run schematics against before releasing them to layout.\nIt assumes the design is functionally “done” — the point is to find what “done”\nmissed.
\n(Vin − Vout) × I for every linear regulator. An LDO dropping\n3.3 V at 500 mA is dissipating 1.65 W and needs a heatsinking copper plan that\nlayout has to know about now.Signal integrity is decided in the schematic by what you connect to what, before\na single trace exists.
\nConnectors are where field failures enter.
\nA proper schematic review on a medium-complexity board is the better part of a\nday for two engineers, plus the fix cycle — call it two to three engineer-days\nbefore layout even starts. It feels like a delay when the schematic “looks\nfinished”.
\nWhat it does not do: it will not catch problems that only exist in the physical\nlayout — coupling, plane resonance, actual trace impedance, thermal reality,\nmechanical fit. Those need a layout review and, ultimately, measurement on\nhardware. The schematic review is necessary and cheap; it is not sufficient, and\ntreating a passed schematic review as “the hard part is over” is its own failure\nmode.
\n", "image": "https://exubits.com/engineering/og/schematic-review-before-layout.png", "date_published": "2026-09-03T00:00:00.000Z", "date_modified": "2026-09-03T00:00:00.000Z", "authors": [ { "name": "Exubits Engineering", "url": "https://exubits.com/authors/exubits-engineering" } ], "tags": [ "Reference", "signal-integrity", "power-design", "dfm" ] }, { "id": "https://exubits.com/engineering/sensor-to-display-latency", "url": "https://exubits.com/engineering/sensor-to-display-latency", "title": "Where the milliseconds go between sensor and display", "summary": "Glass-to-glass latency is exposure + sensor readout + CSI-2 + ISP + buffer and queue waits + composition + one or two display refreshes. Most of it is exposure and buffering, not code; shrink queues and sync to vsync.", "content_html": "“The camera feels laggy” is a latency-budget problem, and it is almost always\nlost in places people do not instrument: exposure time, the number of buffers in\neach queue, and the wait for the next display refresh. Optimising the application\ncode is usually the smallest available win.
\nThis page walks the whole path from photons to photons and gives the order of\nmagnitude of each stage, so you know which term to attack.
\nThe sensor integrates light for the exposure time. The photons that form a\nframe are spread across that whole window, so the effective latency contribution\nis roughly half the exposure time for motion (the centroid of the exposure),\nbut the frame is not readable until the full exposure ends, so for end-to-end\ntiming you count the full exposure time.
\nLever: cap the maximum auto-exposure time and accept more gain (noise) if latency\nmatters more than SNR. This is an ISP/3A tuning decision, not a code change.
\nThe pixel array is read out row by row and streamed. Readout takes on the order\nof one frame period at the sensor’s current mode — a rolling shutter sensor\nat 60 fps takes ~16 ms to clock the whole frame out, and the bottom of the frame\nis genuinely ~16 ms “younger” than the top.
\nSerialised over the D-PHY/C-PHY lanes into the SoC’s CSI receiver. At any sane\nlane rate this is sub-millisecond per frame for typical resolutions — it is\nalmost never the problem. It shows up only if lanes are marginal and you are\ngetting retimed/corrupted lines that force retry or drop.
\nThe CSI receiver writes frames into a ring of buffers in DRAM. Two things here:
\n(buffers − 1) × frame_period before anything consumes it. Four buffers at\n30 fps is up to 100 ms of potential sit time. This is the classic hidden lag.Debayer, black level, lens shading, denoise, sharpening, tone mapping, colour\nconversion, scaling. On a hardware ISP this is a few milliseconds and often\npipelined with readout. In software (libcamera soft ISP, OpenCV) it can be tens\nof milliseconds and steals CPU from everything else.
\nThe frame becomes a dmabuf handed to whatever draws it — a Qt QVideoSink, a\nGStreamer appsink/glimagesink, an LVGL canvas, a Weston/Wayland client, a\ndirect DRM/KMS plane.
depth × frame_period of potential latency.\nGStreamer’s default queue sizes, a v4l2src with many buffers, a compositor\nthat triple-buffers — each is a place frames wait.dmabuf import all the way to the display plane\navoids it.If the UI draws the video into a scene (overlays, HUD, controls), the GPU\ncomposites. One frame period at the render rate, plus the GPU’s own latency.\nUsing a hardware overlay plane for the video and only compositing the UI chrome\navoids putting the video through this stage.
\nPAGE_FLIP synced to vsync with exactly the buffers you need (often\ndouble, not triple), and pick a panel/timing controller with a documented,\nlow, fixed latency. Disable panel-side processing.A representative indoor 30 fps preview, no tuning:
\n| Stage | \nTypical | \n
|---|---|
| Exposure | \n15–33 ms | \n
| Readout | \n~16–33 ms | \n
| CSI-2 + DMA | \n1–3 ms | \n
| Buffer ring wait (4 deep) | \n0–100 ms | \n
| ISP (hardware) | \n2–6 ms | \n
| Pipeline queues (GStreamer defaults) | \n30–100 ms | \n
| GPU composition | \n~16–33 ms | \n
| Vsync wait + buffering | \n16–50 ms | \n
| Panel | \n8–33 ms | \n
That easily reaches 150–250 ms — the “laggy” complaint — and only ~5 ms of it\nis anything a code profiler would show you.
\nThe same pipeline, tuned:
\nleaky=downstream max-size-buffers=1.lands in the 40–70 ms range, and most of what remains is exposure, readout,\nand one refresh period — the irreducible physics.
\nDo not trust per-stage estimates; measure glass-to-glass:
\nv4l2_buffer.timestamp),\nGST_DEBUG with GST_TRACERS=latency, DRM/KMS PAGE_FLIP event timestamps,\nand a GPIO toggle at each app-level handoff. Line them up against the\nglass-to-glass number to find the fat stage.Most real-time control problems are argued in adjectives — “fast enough”,\n“low jitter”, “hard real-time” — and adjectives cannot be verified. A determinism\nbudget replaces them with four numbers agreed before implementation starts. After\nthat, every architectural question has a right answer: does option A fit inside\nthe budget or not.
\nThis page is about writing that budget. It is protocol- and silicon-agnostic on\npurpose; the numbers change per project, the structure does not.
\nf_s)The rate at which the loop reads its inputs and updates its output. Set it from\nthe plant, not from what the CPU can do:
\nf_s between 10× and 20× the closed-loop bandwidth you\nneed. Below 10× the phase lag from sampling eats your phase margin; far above\n20× you are burning CPU and amplifying sensor noise through the derivative term\nfor no dynamic benefit.Δt_s)The allowed variation in when the input is actually sampled, relative to the\nideal period 1/f_s. This is the number most specs omit and most loops get bitten\nby.
Jitter matters because a control law assumes a fixed Δt. If the real interval\nwanders by ±15% and the code still divides by the nominal Δt in the derivative\nand integral terms, you have injected a disturbance proportional to the jitter and\nthe signal slew rate. Effects:
I term accumulates the wrong area.D term produces spikes on jittered intervals.Budget it as a percentage of the period and as an absolute time. “≤ 2% of period\nor ≤ 5 µs, whichever is larger” is a typical starting point for a mid-rate loop.\nFor a current loop synchronised to PWM, the sampling instant is usually locked to\na timer/PWM trigger and an ADC hardware trigger, and the jitter budget there is\ntens of nanoseconds — a software-timer-driven sample cannot meet it and the\nbudget is what tells you that up front.
\nTwo mitigations worth stating in the budget itself:
\nΔt into the loop maths, so\njitter degrades gracefully instead of injecting noise.L_wc)The time from “the sampling instant” to “the new actuator command is in effect” —\nend to end, worst case, not typical. Its components:
\nL_wc = t_acq ADC conversion + settling\n + t_dispatch interrupt latency + scheduler to the loop task\n + t_compute the control law, worst-case path\n + t_output DAC/PWM update, or the bus transaction to a remote drive\n + t_actuator the actuator's own transport lag (often the biggest term)Rules for filling it in:
\nt_compute with the branch that runs the anti-windup and the fault\nchecks, bus latency including retransmission if the link allows it.L_wc shows up in the loop as pure dead time. Dead time destroys phase margin\nfast: as a guide, keep L_wc under about 1/10 of the loop period, and treat\nanything above 1/4 of the period as a redesign trigger.Define what a missed deadline is and what the system does about it. “It should\nnot happen” is not a policy.
\nOnce the four numbers exist, these stop being opinions:
\n| Question | \nDecided by | \n
|---|---|
Bare-metal, RTOS, or Linux with PREEMPT_RT? | \nCan its worst-case interrupt-to-task latency + scheduler jitter fit inside Δt_s and the t_dispatch share of L_wc, measured, under load? | \n
| Loop in an ISR or a task? | \nIf t_compute is far below the period and the jitter budget is tight, use the ISR. If t_compute is a meaningful fraction of the period, use a task so it can be preempted by faster ISRs. | \n
| Control local to the actuator or central? | \nDoes the bus transport lag fit inside L_wc? | \n
| Which fieldbus? | \nIts cycle time and cycle-time jitter versus Δt_s and L_wc. | \n
| Is this core allowed to run anything else? | \nOnly if the other work’s worst-case interference still leaves the budget intact — usually meaning a shielded/isolated core or a dedicated MCU. | \n
A budget you cannot measure is a wish. The standard instrumentation:
\nDWT->CYCCNT on Cortex-M, PMCCNTR / perf on\nCortex-A) around the compute path, logging min/max/histogram, not just mean.Hold the measurement for long enough to see the tail. Latency distributions in\nreal systems have long tails driven by rare cache/TLB/bus-contention events, and\nthe 99.99th percentile is often 3–10× the median. The budget is about the tail,\nso the measurement has to reach it.
\nWriting this budget costs a day or two of analysis and a negotiation with whoever\nowns the plant requirements, before any loop code exists. On a schedule under\npressure that day is tempting to skip, and the loop will usually “work” on the\nbench without it.
\nWhat you lose by skipping it is the ability to say why it works, and the ability\nto catch — at design time rather than during integration — the cases where the\nchosen bus or OS cannot meet the timing. Those cases are expensive exactly in\nproportion to how late they are found. The budget moves that discovery to the\ncheapest possible point.
\nA tighter budget is not free either. Driving jitter to nanoseconds means hardware\ntriggering, a shielded core, and no shared bus — real BOM and integration cost.\nThe budget should be as loose as the plant genuinely allows, and no looser.
\n", "image": "https://exubits.com/engineering/og/stating-a-determinism-budget.png", "date_published": "2026-09-03T00:00:00.000Z", "date_modified": "2026-09-03T00:00:00.000Z", "authors": [ { "name": "Exubits Engineering", "url": "https://exubits.com/authors/exubits-engineering" } ], "tags": [ "Reference", "determinism", "real-time", "closed-loop" ] }, { "id": "https://exubits.com/engineering/yocto-bsp-handover", "url": "https://exubits.com/engineering/yocto-bsp-handover", "title": "What a Yocto BSP hand-over should actually contain", "summary": "A usable Yocto BSP hand-over: a pinned manifest, vendored layers, a documented DISTRO/MACHINE, a reproducible build, a signed image with its SPDX SBOM, and the rationale for every kernel and U-Boot config choice.", "content_html": "A Yocto BSP hand-over fails in a predictable way. Six months after the last\ninvoice, someone runs the build on a fresh machine and it breaks: a layer moved,\nmeta-openembedded advanced a branch, an SRC_URI 404s, the host GCC is now too\nnew for a fetched tarball. The image that shipped is fine. The ability to\nrebuild it is gone. That is the thing a hand-over has to protect, and most\nhand-overs do not.
This page is the checklist we hold our own BSP deliveries to. It is deliberately\nabout artefacts and their properties, not about a particular board.
\nEverything below is in service of a single test. Take the delivery, put it on a\nmachine that has never seen the project, follow the written steps, and get an\nimage whose package manifest matches the one that shipped. If that test passes,\nthe BSP is maintainable. If it does not, you have bought a binary with source\nattached, not a BSP.
\nYocto gives you the machinery for this — BB_HASHSERVE, hash-equivalence,\nbuildhistory, reproducible-builds class — but none of it is on by a default\nthat survives a vendor change. It has to be configured, and the configuration is\npart of the deliverable.
repo manifest or a kas file that pins every layer to a commit SHA,\nnot a branch. honister, kirkstone, scarthgap — a branch name is a moving\ntarget. kirkstone today is not kirkstone from the release date.conf/bblayers.conf BBLAYERS order documented, because layer priority\nchanges which .bbappend wins.The test: kas checkout (or repo sync) on an air-gapped mirror reproduces the\nexact tree. No “then update meta-freescale to the tip”.
There should be exactly one layer that contains the project’s work —\nmeta-<project> — and it should be the only layer with local commits. Every\nchange to a BSP or upstream recipe lives there as a .bbappend or a versioned\nrecipe copy, never as an edit to meta-ti, meta-freescale, or poky.
Why this is a hand-over issue and not a style preference: when the client later\nmoves from kirkstone to the next LTS, the migration work is reviewing one\nlayer. If changes are smeared across five vendor layers, the migration is\narchaeology, and the estimate for it triples.
What belongs in meta-<project>:
.conf (or a .bbappend to the vendor’s) with every deviation\ncommented — why PREFERRED_VERSION_linux-* is pinned, why a MACHINE_FEATURE\nwas removed.debug-tweaks, openssh-sftp,\ngdbserver — and production must be verified not to).DISTRO. It usually should:\ninheriting poky and overriding twelve variables in local.conf means the\nconfig only exists on the machine that has that local.conf.local.conf is not part of the delivery — the distro config islocal.conf is per-developer scratch. Anything load-bearing that lives there\nwill be lost. The hand-over must move every meaningful setting into version\ncontrol:
DISTRO_FEATURES / DISTRO_FEATURES_remove — particularly systemd vs\nsysvinit, wayland/x11, pam, usrmerge.IMAGE_FSTYPES and how they map to what the factory actually flashes\n(wic.gz, wic.bmap, a .swu for SWUpdate).EXTRA_IMAGE_FEATURES, IMAGE_INSTALL:append.PACKAGE_CLASSES choice (package_rpm/ipk/deb) — this affects the\non-target update mechanism and cannot be changed casually later.PREMIRRORS / SSTATE_MIRRORS pointing at internal infrastructure.A kernel handed over as a 6,000-line .config is not maintainable, because\nnobody can tell an essential setting from an accident of make oldconfig.
defconfig plus fragments, wired through KERNEL_CONFIG_FRAGMENTS or a\nlinux-*.bbappend. The fragment is small and every line is intentional.CONFIG_ enables the\nEthernet PHY, which sets the RT behaviour, which was needed for a USB gadget\nmode, which disables an unused subsystem to cut attack surface and boot time.The device tree is where “the BSP works” and “the BSP is correct” diverge. A\nnode can be missing and the board still boots.
\n.dts in meta-<project>, including via .dtsi, not patched into\nthe vendor’s file in place.MX8MM_IOMUXC_SD2_CD_B_GPIO2_IO12 to the actual card-detect net saves the next\nengineer an afternoon with a multimeter.regulator-always-on, regulator-boot-on and\nthe supply chain to each peripheral. Half of “the peripheral randomly does not\nenumerate” bugs are a missing or lazy regulator description.boot.cmd/boot.scr or the extlinux\nconfig, in version control.buildhistory output committed for that build: the package list with\nversions, image size, dependency graph, and the diff against the previous\nrelease. This is what makes “what changed between v1.2 and v1.3” answerable in\nminutes.create-spdx — covering every\npackage in the image with its version and licence. Increasingly this is a\ncontractual and regulatory requirement (the EU Cyber Resilience Act among\nthem), and it is far cheaper to generate at build time than to reconstruct.license.manifest, deploy/licenses/) and the\nsources for anything under a copyleft licence, or a working bitbake -c archiver configuration that regenerates them.“Works on my machine” is not a hand-over. Pin the host too:
\nbitbake invocation, target names, and any BB_ENV_PASSTHROUGH\nadditions.TMPDIR and SSTATE_DIR, RAM, rough wall\ntime on a stated machine — so the client can size CI.DL_DIR archive is\nthe single most valuable non-obvious artefact in the box.Not a wiki dump. Five documents, each answering a question someone will actually\nask:
\nProducing this is real work — on the order of one to two engineer-weeks on top of\nthe BSP itself, most of it in items 1, 7, and 8. It is worth being explicit about\nthat in a statement of work rather than discovering it at the end. The payoff is\nentirely deferred: it shows up the first time the client rebuilds without the\noriginal team, and it is the difference between an afternoon and a re-engagement.
\nThe trade-off we make deliberately: we do not hand over a build that tracks\nupstream branches, even though that would look more “up to date” on delivery day.\nA pinned build is frozen and will drift out of security currency until someone\ndeliberately bumps it — that is the cost. It is the right cost, because a build\nthat silently changes under the client is not a build they own.
\n", "image": "https://exubits.com/engineering/og/yocto-bsp-handover.png", "date_published": "2026-09-03T00:00:00.000Z", "date_modified": "2026-09-03T00:00:00.000Z", "authors": [ { "name": "Exubits Engineering", "url": "https://exubits.com/authors/exubits-engineering" } ], "tags": [ "Reference", "yocto", "bitbake", "kernel", "device-tree" ] } ] }