HomeArticlesSlideshowsAsk BobLogin or Register
NewsCustom SearchContactLinks
Metadata driven ETL approach
ETL PATTERNS
10-Sep-2008
It could be named "Dream about metadata driven ETL". The schema depicted all elements what should participate in the ETL generating process separately to keep the system high manageable.


Following sources should be considered for ETL generating:
  • Repository
    • Data model, Data dictionary (metadata – static part)
    • ETL mappings (metadata – dynamic part)
  • ETL Approach
    • Design stereotypes
    • Functional patterns
  • Environment
    • Platform specific templates
    • Metadata layer and Generator interfaces


There are following success factors of metadata system:
  • Layered understanding – Business meaning of ETL mapping metadata has to be strict separated from technical implementation of mapping.
  • Reduce redundancy – such part of mapping coming from data model, design standards and ETL stereotypes (audit columns, surrogate keys... ) should not be explicitly defined in logical metadata.
  • Consistency – Consistency between metadata on various layers and physical implementation has to be assured (on both process and technical level)
  • Team collaboration and version control


Templates:
  • Pattern - Functional definition of generated job.
  • Algorithm – Technical implementation definition.
  • Standards – Implicit behavior definition - design standards.


Further requirements to metadata system:
  • Completeness – ability to store and maintain all required attributes
  • Extensibility – ability to put user defined extensions
  • Open interfaces, APIs
  • User friendly customizable interfaces
  • Impact analysis
  • Team collaboration support
  • Versioning

See: Presentation 
See: Simple generator sample
See: Other patterns 
Ludek Bob Jankovsky
All Right Reserved © 2007, Designed by Bob Jankovsky