Season 5 Ep 7 — Through Ep 1–6, data kept growing and getting more complex. Ep 7 is the opposite axis — "how do we govern it". If you do not manage your data, your data will rule your company.
- Prologue — "How many places in our company hold a customer email?"
- Chapter 1 · Defining Data Governance
- Chapter 2 · OpenLineage — The Standard for Lineage
- Chapter 3 · The Four Major Data Catalogs
- Chapter 4 · Technical Catalogs — Unity/Polaris/Glue
- Chapter 5 · PII — Definition and Detection
- Chapter 6 · Protecting PII — Masking, Tokenization, Encryption
- Chapter 7 · Regulation — GDPR, Korean PIPA, AI Act
- Chapter 8 · Data Subject Request (DSR)
- Chapter 9 · Extending Governance into the AI Era
- Chapter 10 · Practical Architecture — Governance Integration
- Chapter 11 · The Reality of Governance at Korean Companies
- Chapter 12 · Eight Failure Stories
- 12.1 PII Exposed on a BI Dashboard
- 12.2 Incident Response Without Lineage
- 12.3 A Deletion Request That Took a Month
- 12.4 Thousands of Tables With No Known Owner
- 12.5 A Catalog That Exists but Is Empty
- 12.6 Fragmented Access Permissions
- 12.7 Contractors With Access to All PII
- 12.8 Unknown Provenance for AI Training Data
- Chapter 13 · Ten Anti-patterns
- 13.1 "Governance Later"
- 13.2 A Catalog With No Owners
- 13.3 A Few Lines of Regex for PII Detection
- 13.4 Indifference to Regulation for Backups
- 13.5 Lineage as a Hand-written Document
- 13.6 Changing Upstream Freely Without a Data Contract
- 13.7 No Control Over External Sharing
- 13.8 Short Retention for Audit Logs
- 13.9 Running Several Catalog Tools in Parallel
- 13.10 Leaving AI and ML Outside Governance
- Chapter 14 · Checklist — Twelve Points for Data Governance
- Chapter 15 · Next Up — Season 5 Ep 8: "Observability 2025 (Logs, Metrics, Traces + LLM)"
Prologue — "How many places in our company hold a customer email?"
The hardest questions for a CTO to answer in 2025:
- How many tables and systems hold customer email addresses?
- How many of those are PII-masked?
- When a user says "delete my data", where do you start deleting?
- Which source data did the numbers on this dashboard come from?
If the answer to all four is "I do not know", that company carries regulatory risk and product quality risk at the same time. This post lays out the tools and processes that make those four questions answerable.
Chapter 1 · Defining Data Governance
1.1 The Four Axes of Governance
- Catalog: what data do we have
- Lineage: where does it come from and where does it go
- Quality: is this data trustworthy
- Security and regulation (PII, RBAC): who can see it and how long do we keep it
1.2 Why It Matters Now
- Fragmentation of the data stack (Ep 1–6) makes management complexity explode
- Tightening of the EU AI Act, GDPR and the Korean Personal Information Protection Act
- AI models train on and infer from data, raising copyright and PII issues
- Customers and employees exercise their data rights more often
1.3 The Governance Spectrum
- Catalog: read-oriented, discovery by employees
- Active governance: enforcing policy, blocking violations
- Federation: unified management across multiple clouds and multiple databases
Chapter 2 · OpenLineage — The Standard for Lineage
2.1 Identity
- Open sourced in 2020, under the LF AI & Data Foundation
- The industry standard spec for data lineage
- Reference implementations in Python, Java and OpenAPI
2.2 Structure
- Job: a unit of execution (Airflow DAG, dbt run, Flink job)
- Run: an execution instance of a Job
- Dataset: input and output datasets
- Event: START/COMPLETE/FAIL events
2.3 Integrations
- Supports Airflow, dbt, Spark, Flink, Dagster and Prefect
- Marquez (the reference metadata store)
- Datakin, then Astronomer, provides SaaS support
- DataHub, Atlan and Collibra receive OpenLineage events
2.4 Example Event
{
"eventType": "COMPLETE",
"job": {"name": "dbt.fact_orders"},
"run": {"runId": "..."},
"inputs": [{"name": "raw.orders"}, {"name": "raw.customers"}],
"outputs": [{"name": "analytics.fact_orders"}]
}
2.5 Value
- Automatic lineage collection reduces the documentation burden on people
- Portable between tools
- Incident analysis and impact assessment
Chapter 3 · The Four Major Data Catalogs
3.1 Collibra
- 2008, Belgium — the strongest player in enterprise governance
- Specialized in finance, public sector and pharmaceuticals
- Strong at business glossary and policy management
3.2 Atlan
- 2018, India — friendly to the modern data stack
- Outstanding integration with dbt, Snowflake and Databricks
- Excellent collaboration and UX
3.3 DataHub
- 2020, LinkedIn — then Apache
- Open source, can be self-hosted
- Backed by Acryl Data (commercial SaaS)
3.4 Alation
- 2012 — an enterprise collaboration catalog
- Positioned as a Data Intelligence Platform
3.5 Others
- Secoda, Castor(→ CoreView), OpenMetadata, Amundsen(Lyft)
- Informatica Data Governance, IBM Watson Knowledge Catalog
3.6 Comparison
| Tool | Strength | Customers | Model |
|---|---|---|---|
| Collibra | Regulation, enterprise | Finance, public sector | SaaS/self |
| Atlan | Modern stack, UX | SaaS, startups, mid-size | SaaS |
| DataHub | Open, flexible | Engineering teams | OSS + Acryl |
| Alation | Collaboration, business users | Mid-size, enterprise | SaaS |
| OpenMetadata | Open, lightweight | Self-hosted | OSS |
Chapter 4 · Technical Catalogs — Unity/Polaris/Glue
4.1 Unity Catalog
- The governance layer of Databricks
- Open sourced in 2024 (Unity Catalog OSS)
- Unifies Table, Volume, Model and Function
- Supports both Delta and Iceberg
4.2 Polaris
- The open Iceberg REST catalog from Snowflake
- Open sourced in 2024
- Accessible from external engines
- Combines governance with the catalog
4.3 AWS Glue Data Catalog
- The default catalog on AWS
- Supports Iceberg, Delta and Hudi
- Combined with Lake Formation for permissions and auditing
4.4 Nessie / Gravitino
- Nessie: Git-like branches and tags
- Gravitino: metadata federation (unifying several catalogs)
4.5 Business vs Technical Catalog
- Technical catalog: the metadata engines and queries consume
- Business catalog: the meaning, owner and SLA that people consume
- In 2025 linking the two (OpenLineage, OpenMetadata) is the standard
Chapter 5 · PII — Definition and Detection
5.1 Defining PII
- Personally Identifiable Information
- Direct identifiers (name, resident registration number, card number)
- Indirect identifiers (date of birth plus region, IP, cookie ID)
- Sensitive information (health, religion, orientation)
5.2 Automatic Detection Tools
- AWS Macie, Google DLP, Azure Purview: cloud DLP
- Privacera, BigID, OneTrust: enterprise specialists
- Presidio (Microsoft): open source
- Snowflake Data Classification: built into the DW
5.3 Detection Methods
- Regular expressions (card numbers, resident registration numbers)
- Dictionary based (names, addresses)
- ML classifiers (context included)
- Column sampling plus entropy analysis
5.4 Classification Grades
- Public / Internal / Confidential / PII / Sensitive PII / Regulated
- Access, log retention and encryption policies differ by grade
Chapter 6 · Protecting PII — Masking, Tokenization, Encryption
6.1 Masking
- Email:
john@***.com - Card:
**** **** **** 1234 - Name:
J*hn D*eorJohn OO - Dynamic Data Masking: shows different values per user at query time
6.2 Tokenization
- Replace PII with meaningless tokens and keep the mapping in a separate vault
- Preserves analytical usefulness while minimizing the impact of a leak
- Vault: HashiCorp Vault, AWS KMS plus an in-house service
6.3 Hashing and Anonymization
- One-way hashes such as SHA-256: analysis is possible, the original cannot be recovered
- k-anonymity, l-diversity, differential privacy
- Used for healthcare and public sector data
6.4 Encryption
- At rest: S3 SSE, disk encryption
- In transit: TLS
- Column-level: Parquet Modular Encryption
- Field-level: encrypt in the application before storing
6.5 Dynamic Masking Example (Snowflake)
CREATE MASKING POLICY mask_email AS (val STRING)
RETURNS STRING ->
CASE WHEN CURRENT_ROLE() IN ('ADMIN') THEN val
ELSE REGEXP_REPLACE(val, '.+@', '***@')
END;
ALTER TABLE customers MODIFY COLUMN email
SET MASKING POLICY mask_email;
Chapter 7 · Regulation — GDPR, Korean PIPA, AI Act
7.1 GDPR (EU, 2018–)
- Data subject rights (access, erasure, portability, rectification, objection to processing)
- A legal basis (consent, contract and so on) must be stated
- DPIA (impact assessment), DPO (data protection officer)
- Violations draw up to 4% of global turnover or 20 million euros
7.2 The Korean Personal Information Protection Act (PIPA)
- Enacted in 2011, amended in 2020 and 2023
- Consent for processing personal information and a stated purpose are required
- The 2023 amendment strengthened data subject rights (access, erasure, portability)
- Introduced an administrative fine capped at 3% of revenue
- Supervised by the Personal Information Protection Commission
7.3 The Korean AI Framework Act (2024–)
- Prior impact assessment for high-risk AI
- Guidelines for responsible AI development and operation
- Reporting obligations for operators
7.4 Other Regulations
- HIPAA, GLBA and CPRA (California) in the United States
- PDPA in Singapore, APPI in Japan
- By sector: the Financial Supervisory Service, the Medical Service Act, the Regulation on Supervision of Electronic Financial Transactions
7.5 Common Principles
- Purpose limitation: no use beyond the purpose of collection
- Minimal collection: only as much as necessary
- Retention period: destroy once the purpose is achieved
- Subject rights: access, rectification, erasure, portability
- Transparency: disclose how processing is carried out
Chapter 8 · Data Subject Request (DSR)
8.1 Request Types
- Access: view my data
- Rectification: correction
- Erasure (right to be forgotten): deletion
- Portability: move to another service
- Object: refuse processing
8.2 Implementation Difficulty
- Deleting that user rows in the warehouse
- Row-level delete in Lakehouse Iceberg
- Handling backups and logs
- When the data is part of ML training data (retrain?)
- Data shared with third parties
8.3 Practical Patterns
- Data Map: a list of the tables and locations that hold PII
- Unique ID: search every system with a per-user unique ID
- DSR workflow tools: OneTrust, TrustArc, Osano
- Log and backup policy: automatic destruction after the statutory period
8.4 AI and Training Data
- Deleting an individual from training data is difficult in principle
- Remedies: pseudonymization, anonymization, synthetic data
- Machine unlearning research is under way
Chapter 9 · Extending Governance into the AI Era
9.1 Model Catalog
- The list of models the company uses
- Version, data, performance, license, audit
- MLflow, Databricks Model Registry, Weights & Biases, HuggingFace Hub
9.2 Prompt and Agent Catalog
- System prompt versions
- Agent tools and permissions
- An extension of Ep 11 (LLMOps)
9.3 Tracking the Provenance of Training Data
- Documenting copyright and licenses
- Dataset Cards
- Presenting the original evidence when an audit asks
9.4 Auditing AI Output
- Response logs, citations, the basis for decisions
- Reproducibility for postmortems
9.5 The Link to Regulation
- EU AI Act: quality management and documentation for high-risk models
- The Korean AI Framework Act: impact assessment
Chapter 10 · Practical Architecture — Governance Integration
10.1 Layers
- Ingestion: Kafka, Fivetran, Airbyte emit OpenLineage events
- Transformation: dbt and Spark emit OpenLineage automatically
- Storage: Iceberg and Delta plus a catalog (Unity/Polaris/Glue)
- Governance: DataHub/Atlan/Collibra receive and consolidate
- Policy: Unity/Lake Formation/Ranger/OPA
- PII: Macie/Privacera/DLP
10.2 Policy Example
- Raw: original data, restricted access, retained 90 days
- Silver: masking plus tokenization, 1 year
- Gold: aggregates, permission-based masking, 3 years or more
- Backups: encrypted, destroyed automatically after the legal period
10.3 Auditing
- Logs of every query and DDL
- Access anomaly detection (unusual hour, IP, volume)
- A quarterly audit report
10.4 Automation
- A governance check in CI when a PR merges
- A new table triggers automatic classification and owner assignment
- Access requests go through an approval workflow
Chapter 11 · The Reality of Governance at Korean Companies
11.1 Current State
- Finance and public sector: in-house builds mixed with Collibra/Informatica
- Large enterprises: DataHub open source, then an internal fork
- Mid-size companies and startups: growing adoption of Atlan and Secoda
- Dedicated observability and PII tools are still at an early stage
11.2 Regulatory Response
- Growing numbers of audits and administrative fines from the Personal Information Protection Commission
- Guidelines from the Financial Security Institute
- The three data laws for public data (Personal Information Protection Act, Credit Information Act, Information and Communications Network Act)
11.3 Challenges
- Building a catalog inside a network-separated environment
- Support for Korean-language metadata
- Automatic collection of history from legacy DWs is hard
- Talent shortage: dedicated governance staff are scarce
11.4 Reference Cases
- Toss: DSR and catalog integrated into the internal data platform
- Kakao: an automatic PII classification pipeline
- Naver: a governance standards team plus an in-house catalog
- Samsung SDS and LG CNS: governance SI packages for enterprise customers
Chapter 12 · Eight Failure Stories
12.1 PII Exposed on a BI Dashboard
Real names shown without dynamic masking, flagged in an audit.
12.2 Incident Response Without Lineage
Spent half a day hunting for "why this metric is wrong".
12.3 A Deletion Request That Took a Month
Manual work, and simultaneous deletion across several systems failed.
12.4 Thousands of Tables With No Known Owner
Nobody knows who to ask.
12.5 A Catalog That Exists but Is Empty
Built once and never maintained, so it drifted away from reality.
12.6 Fragmented Access Permissions
RBAC configured separately in ten databases, central management failed.
12.7 Contractors With Access to All PII
An audit violation.
12.8 Unknown Provenance for AI Training Data
Helpless in a copyright dispute or a regulatory investigation.
Chapter 13 · Ten Anti-patterns
13.1 "Governance Later"
The larger you grow, the harder it is to tear up and redo. Lay the foundations from the start.
13.2 A Catalog With No Owners
There is documentation, but no accountability.
13.3 A Few Lines of Regex for PII Detection
You need ML classifiers and sampling alongside it.
13.4 Indifference to Regulation for Backups
The "the destruction deadline passed but it is still in the backup" incident.
13.5 Lineage as a Hand-written Document
Adopt automatic collection (OpenLineage).
13.6 Changing Upstream Freely Without a Data Contract
Every consumer breaks.
13.7 No Control Over External Sharing
Public S3 buckets, email attachments.
13.8 Short Retention for Audit Logs
Fails to meet regulatory requirements.
13.9 Running Several Catalog Tools in Parallel
Duplicated management with no consolidation.
13.10 Leaving AI and ML Outside Governance
In 2025 this is the biggest risk of all.
Chapter 14 · Checklist — Twelve Points for Data Governance
- Data catalog (business plus technical) consolidated
- Automatic lineage collection based on OpenLineage
- Owners and SLAs stated for tables and models
- Automatic PII detection and classification pipeline
- Masking, tokenization and encryption policy
- DSR workflow and response deadlines
- Permissions and audit logs centralized
- Data contracts and a change process
- Legal retention and destruction policy
- Governance for AI and ML (models, data provenance)
- Regulatory mapping (GDPR, PIPA, AI Act, sector regulation)
- Employee training and reporting to executives
Chapter 15 · Next Up — Season 5 Ep 8: "Observability 2025 (Logs, Metrics, Traces + LLM)"
If governance is "how do we manage data", observability is "how are the systems and the data actually running".
- The maturing of OpenTelemetry
- Unifying the three signals: metrics, logs and traces
- Grafana Cloud, Datadog, New Relic, Splunk, Honeycomb, SigNoz
- Tempo, Loki, Mimir, Jaeger, Zipkin
- LLM observability (LangFuse, LangSmith, Phoenix, Helicone, an extension of Ep 6)
- SLOs, SLIs and error budgets
- Chaos engineering and resilience
- The observability stack at Korean companies
- "Observability is not insurance, it is product quality"
"No observability, no operations" — the default of infrastructure in 2025.
See you in the next post.
Summary: Data governance in 2025 rests on four axes — catalog, lineage, quality, and PII and regulation. OpenLineage has become the standard for lineage, Collibra, Atlan, DataHub and Alation are the four catalog options, and Unity, Polaris and Glue handle the technical catalog. PII is managed in five stages of detection, classification, masking, tokenization and encryption, while GDPR, the Korean PIPA and the AI Act form the regulatory triangle. Governance in the AI era now extends to models, prompts and training data, and data subject requests (DSR) demand an automated workflow. For Korean companies, network separation, Korean-language metadata and legacy integration are the challenges, and securing dedicated governance staff is the next competitive edge. "If you do not manage it, it manages you" — the 2025 law of data governance.