Course overview
Learn to define, measure, and enforce data quality across batch and streaming systems. Design practical controls for schema, completeness, validity, uniqueness, freshness, and lineage.
Who this is for
You know the fundamentals and want structured practice that turns knowledge into production habits.
Prerequisites
No previous OSEKOO ACADEMY course is required. Use the optional diagnostic to confirm your starting point.
Syllabus
Each unit combines explanation, guided practice, a checkpoint, and practical evidence. Open a unit to review the exact sequence.
1. Quality Foundations9 lessons
Prerequisites: You should be able to read basic Python functions, lists, dictionaries, dataclasses, and assertions; calculate ratios; and explain primary-key and SQL-style aggregation concepts. All exercises use synthetic data.
Unit outcomes
- Frame data quality as fitness for a named consumer decision and failure consequence.
- Declare source, population, time boundary, grain, key, and version before interpreting quality evidence.
- Prioritize quality risks using consequence, exposure, detectability, proven controls, confidence, and reversibility.
- Use completeness, validity, uniqueness, consistency, accuracy, and freshness as distinct diagnostic questions.
- Produce structural and value profiles that are reproducible, privacy-minimizing, and explicit about limitations.
- Investigate anomalies with comparable baselines and contextual evidence instead of treating unusual values as defects.
- Define indicators, objectives, hard gates, ownership, and breach responses at the consumer boundary.
- Defend a publication decision using identity reconciliation, balances, diagnostics, consumer impact, and retained evidence.
Unit project
Create a reproducible quality-foundation decision packet for a synthetic operational dataset. Define the consumer action, boundary, grain, keys, risks, dimension evidence, structural and privacy-governed value profiles, anomaly investigation, reconciliation, service objective, hard gates, ownership, publication state, and limitations. Include at least eight adversarial fixtures and calculate every expected result independently.
Guided practice
- Create a decision brief and six-row risk register for a synthetic consumer workflow.
- Declare two different boundaries and grains over the same event history and calculate their expected populations.
- Build structural and privacy-aware value profiles with malformed, rare-category, and empty-batch fixtures.
- Classify seasonal, launch, missing-partition, duplicate-replay, and backfill signals using corroborating evidence.
- Write a service-objective pack with exact denominator policy, hard gates, ownership, and communication.
- Complete and defend the Northstar publication packet under adversarial boundary and reconciliation changes.
Lesson sequence
- Quality Begins with a Consumer Decision · Free preview
- Define the Dataset Boundary and Grain
- Turn Failure Consequences into Risk
- Use Quality Dimensions as Diagnostic Lenses
- Profile Structure Before Values
- Profile Values Without Leaking Data
- Separate Anomalies from Defects
- Set Indicators, Objectives, and Ownership
- Foundation Case Review
12 checkpoint questions
2. Executable Quality Contracts11 lessons
Prerequisites: Complete Quality Foundations. You should be able to read Python dataclasses, sets, dictionaries, exceptions, JSON, assertions, and simple relational constraints. Great Expectations appears only in the adapter lesson; all other examples use the Python standard library.
Unit outcomes
- Distinguish schema syntax, semantic rules, operational obligations, ownership, and evolution clauses in a data contract.
- Express keys, domains, cross-field invariants, dependencies, forbidden states, referential rules, aggregates, and temporal requirements precisely.
- Evaluate contract changes against concrete consumer parsers and semantic assumptions instead of version labels alone.
- Implement deterministic framework-independent validation with stable rule identity, prerequisites, bounded evidence, and four non-collapsing outcomes.
- Map selected rules through a pinned Great Expectations adapter while detecting differences in scope, counting units, and evidence meaning.
- Separate validation findings from consumer-specific publish, warning, quarantine, and stop policy decisions.
- Create privacy-minimizing, content-addressed publication and quarantine evidence with verified lineage and membership conservation.
- Test rule semantics, adapter parity, policy coverage, evidence schemas, determinism, mutation resistance, and compatibility in CI.
- Release a versioned one-command quality gate with documented exit semantics and reproducible machine-readable evidence.
Unit project
Build a versioned Northstar quality-gate package for an immutable synthetic capture batch. Define semantic contracts and compatibility policy, implement independently tested row and cross-boundary rules, adapt selected intent through Great Expectations, apply consumer-specific publication actions, preserve content-addressed quarantine and lineage evidence, and emit a deterministic report with explicit exit semantics. Include malformed, boundary, conflicting, mutation, incompatible-consumer, and evidence-tampering fixtures. Demonstrate that no stop or error path exposes partial output.
Guided practice
- Translate a Northstar requirement into separate structural, semantic, operational, ownership, and evolution clauses.
- Implement row, dependency, referential, aggregate, and temporal rules with independently calculated expected results.
- Build a producer-consumer compatibility matrix for additive, required-field, domain, and semantic changes.
- Compare a framework-independent rule with its Great Expectations adapter and document any evidence mismatch.
- Write policy entries for analytics, finance, and support consumers, including missing-coverage and unsafe-containment behavior.
- Construct and tamper-test a content-addressed quarantine bundle with disjoint membership and lineage verification.
- Kill plausible semantic mutants and reject incomplete release-gate evidence in CI.
- Ship and defend the cumulative versioned quality gate against malformed input and incompatible consumers.
Lesson sequence
- From Requirement to Contract
- Keys, Domains, and Cross-field Invariants
- Dependencies and Forbidden States
- Referential, Aggregate, and Temporal Rules
- Contract Versioning and Compatibility
- A Framework-independent Validation Core
- Great Expectations as One Adapter
- Severity and Publication Policy
- Evidence, Quarantine, and Lineage
- Testing Quality Controls in CI
- Module Studio: Versioned Quality Gate
12 checkpoint questions
3. Production Observability, Incidents, and Improvement12 lessons
Prerequisites: Complete Quality Foundations and Executable Quality Contracts. You should be able to read Python dataclasses, immutable collections, timezone-aware datetime code, deterministic serialization, hashes, graph traversal, and assertions. All operational identities and incident records are synthetic.
Unit outcomes
- Convert durable quality-gate evidence into bounded, actionable operational signals without leaking record or personal identifiers.
- Define partition-aware freshness, volume, distribution, missing-evidence, and baseline behavior with explicit consumer consequences.
- Trace artifact- and field-level lineage plus consumer-read evidence to make calibrated blast-radius claims.
- Design alert identity, lifecycle, routing, suppression, ownership, runbook, acknowledgement, and escalation contracts.
- Triage incidents using timestamped observations, competing hypotheses, discriminating tests, and immutable decisions.
- Select and execute authorized, reversible containment while preserving evidence and protecting clean scope.
- Repair and replay from immutable identities, prove bounded idempotency, and reconcile membership, values, totals, lineage, and reproducibility before promotion.
- Publish audience-specific incident updates from one factual core, including uncertainty, action, workaround limits, cadence, acknowledgement, and corrections.
- Produce evidence-timed causal reviews and measurable prevention, detection, and containment improvements.
- Prioritize operational controls using domain maturity, mandatory obligations, bounded cost, risk-reduction ranges, and scenario sensitivity.
- Integrate contracts, policy, artifacts, telemetry, lineage, alerts, incidents, recovery, communication, and review into one source-bound repository.
- Operate the frozen repository through a sealed incident and defend safe contemporaneous decisions without hindsight rewriting.
Unit project
Build and defend a source-bound Northstar operational quality-gate repository. Integrate immutable contract and publication evidence, bounded metrics, partition monitoring, exact lineage, alert lifecycle, incident ledger, scoped containment, idempotent replay, reconciliation, consumer communication, post-incident review, and prioritized improvement. Pass twelve acceptance gates and fourteen adversarial fixtures, then operate the frozen commit through a sealed timed-inject incident. Submit the append-only ledger, recovery and communication evidence, deterministic score bundle, architecture decisions, explicit limitations, and one measurable improvement action.
Guided practice
- Compile bounded metrics and diagnostic events from immutable publish, quarantine, stop, and error reports.
- Evaluate missing, zero-row, insufficient-peer, zero-MAD, and distribution-shift partitions under a versioned monitoring policy.
- Trace one affected field through exact runtime lineage and classify confirmed reads, possible exposure, and outside-path consumers.
- Simulate pending, firing, retained, inhibited, silenced, and resolved alert states with stable fingerprints and accountable routes.
- Build an append-only incident ledger with competing hypotheses, refutation conditions, decisions, and evidence-time boundaries.
- Compare halt, isolation, rollback, degraded-service, and communicate-only containment under authority and reversibility gates.
- Replay an immutable repair plan twice, reject operation-key conflicts, and prove exact candidate reconciliation before promotion.
- Issue finance, API, fraud, and leadership updates from one snapshot; test missed cadence, acknowledgement, and append-only correction.
- Construct a causal graph and action portfolio that rejects person-shaped endpoints and premature effectiveness claims.
- Select a constrained improvement portfolio under obligation, consumer-protection, recurrence, and evidence-deficit scenarios.
- Pass the twelve-gate capstone acceptance contract against fourteen adversarial cross-component fixtures.
- Respond to timed incident injects and defend impact, containment, recovery, communication, closure, and remaining limitations.
Lesson sequence
- From Validation Output to Operational Signals
- Freshness, Volume, and Distribution Monitoring
- Lineage and Blast-radius Analysis
- Alert Design and Ownership
- Incident Triage under Uncertainty
- Containment Decisions
- Repair, Replay, and Reconciliation
- Consumer Communication
- Post-incident Learning
- Maturity, Cost, and Improvement Prioritization
- Capstone Build
- Capstone Incident Exercise and Defense
12 checkpoint questions