Oracle_Golden_Gate_介绍

数据同步与 GoldenGate 概要 • 数据同步应用场景及需求 • GoldenGate介绍 • 成功案例分享 • 问题与交流 数据同步应用场景 高可用性与容灾 • 双业务中心 • 消除计划内停机 • 容灾与应急备份 实时数据集成 • 数据仓库实时数据供给 • 实时报表 影响系统高可用性的三种状态 Unplanned outage 移植 升级 系统停机 数据错误 #1: 活动状态 #2: 计划停机 维护 #3: 非计划停机 性能问题 高可用性: 双业务中心（Active-Active) 收益: • 两端数据库同时操作，提高计算能力 • 负载均衡，提高系统整体性能 • 业务永不停机，快速的应急接管 • 长距离容灾 – RAC无法解决的问题 需求: 防止死循环 DML操作冲突解决 实时性 减小网络带宽要求 Active-Active for Continuous Availability 是灾备建设的一种趋势 • 最大化利用灾备中心的IT设施 • 加倍的计算能力 • 提升系统整体性能 • 双中心负载均衡 • 改善地域分布，提高系统及网络响应 • 持续的IT服务 • 异地部署，没有距离限制 • 快速接管，RTO→0 • 双活的数据库 • 实现双向复制，同时两端数据库都是活动的，可以对两端的数据库同 时进行操作，并保证两端数据库的同步 Active-Active双业务中心需求 • 防止循环复制 • GoldenGate能够区分由应用或复制软件发起的操作 • 避免“插入”操作的冲突 • 利用数据库主键进行区隔 • 解决“删、改”操作的冲突 • 将特殊的应用放在一端开展 • 利用时间戳解决冲突 • 复杂逻辑可以通过存储过程实现 • 人工干预 高可用性: 减少计划内停机 收益: 保障业务零或者近似零停机 滚动升级 升级失败时保证回退 降低业务中断带来的损失 需求: 数据延时小 足够的可靠性 高可用性: 容灾与应急备份 收益: 快速恢复和切换 最小化数据损失 在非计划性停机时保持业务连续 需求: 数据延时小 应急切换与恢复时间 对源系统影响小（保证业务优先） 数据同步应用场景 高可用性与容灾 • 容灾与应急备份 • 消除计划内停机 • 双业务中心 实时数据集成 • 数据仓库实时数据供给 • 实时报表 数据仓库实时供给 收益: 消除数据抽取的时间窗口 数据仓库数据实时得到更新 提升商务智能和信息分析能力 需求: 对源系统低影响 适应各种异构环境 满足各种复杂场景要求，如单表对 多表、多表对单表、数据转换等 实时报表和GRC 收益: 将实时报表从主系统剥离，可根据客户 需求定制 查询offloading 显著提高主系统性能 需求: 数据延时小 支持一定的数据转换功能 数据同步需求 总结 初级经济法重点总结下载党员个人总结TXt高中句型全总结.doc高中句型全总结.doc理论力学知识点总结pdf 数据同步需求  实时性  异构环境支持  数据一致性  对源系统影响小 GoldenGate特性:  高性能，低延时 能够以低资源消耗完成每秒数千交易的复制 支持高达每天1,000G的日志处理量  高可扩展性 开放的结构使客户适应各种异构数据平台  高可靠性 基于交易的复制机制  对源系统影响 绿色软件，无需安装 资源占用少 概要 • 数据同步应用场景及需求 • GoldenGate介绍 • 成功案例分享 • 问题与交流 Source Database(s) Target Database(s) LAN / WAN / Internet (TCP/IP) Capture: 实时读取交易日志,捕捉数据变化,并可实现过滤. Capture 队列文件: 暂存数据变化. Source Trail Delivery:执行所需的数据变化，然后将数据 变化提交到目的库. Target Trail Delivery 双向 Source TrailTarget Trail Delivery Capture Pump Pump Oracle GoldenGate 整体架构 传输: 数据经过压缩和加密传送到目的地. Pump: 将数据分散路由到不同的目的端 Read/Write Workload Read/Write Workload On-disk Components Architecture Components (Processes) • Log based • VAM based • Local/Remote Queuing • Filtering • Parallel Coordination • Local/Remote* • Native database API • ODBC • Batched Operations • Filtering • Transaction/Operations Grouping • Parallel Coordination • Local/Remote* • Local/Remote • Filtering • TCP/IP • Compression • Encryption  Queue (Trail) Management  Process Management (Dynamic)  Monitoring and Administration (Lag reports)  Reporting (Events, Errors, Thresholds)  Receive/Route requests from UI  Trail Files  Checkpoint Files - Capture, Pump, Delivery  Data Definition Files (SOURCEDEFS) - Utility to generate definitions  Configuration Files  Output Files  Report/Log Files  Binaries CAPTURE DELIVERY PUMP MANAGER Oracle GoldenGate Capture Capture Memory Manager Database Access VAM API/Redo API (log reader interface) Metadata Manager SQL Interface FilteringFormatting Transformation and Mapping Checkpoint Manager Transformation Engine Serialization and Routing Messaging Subsystem Transaction Log Reader Transaction Logs GoldenGate Trail Files GoldenGate Event Log GoldenGate Checkpoint File Parser GoldenGate Parameter File Command and Control GoldenGate Command Interpreter Metadata Manager 17 Source Database Oracle GoldenGate Delivery Delivery Memory Manager Database Access Read API SQL Interface FilteringFormatting Transformation and Mapping Checkpoint Manager Transformation Engine SQL Generation Messaging Subsystem GoldenGate Trail Reader GoldenGate Event Log GoldenGate Checkpoint File Parser GoldenGate Parameter File GoldenGate Command Interpreter Command and Control Metadata Manager 18 GoldenGate Trail Files Target Database GoldenGate的工作特点 • 实时数据复制 • 支持异构环境 • 可靠性：断点续传，不影响系统连续运行。 • 高性能，对生产系统影响小 • 事务完整性 • 整合：ETL tools, Message Service • 灵活拓扑结构 • 复制冲突 检测 工程第三方检测合同工程防雷检测合同植筋拉拔检测方案传感器技术课后答案检测机构通用要求培训 和解决 • 事件标记基础架构(Event marker infrastructure) • 路由和压缩：TCP/IP，LAN，WAN • 加密：128-位，SSL • 自定义延时应用修改 • 根据事务大小和数量自动管理内存 基于日志的实时数据复制 • 亚秒级复制 • 不依赖源数据库的触发器和规则，对源数据库影响小。 数据库日志 网络 Capture Source Trail Target Trail Delivery Data Pump JMS Message Bus 事务完整性-事务级粒度 • 只复制成功提交的事务 • 防止源和目标的不一致性 网络 Capture Source Trail Target Trail Delivery Insert … Update …. Delete … Commit Databases O/S and Platforms Capture:  Oracle  DB2  Microsoft SQL Server  Sybase ASE  Teradata  Enscribe  SQL/MP  SQL/MX Delivery - all listed above, plus:  HP Neoview, Netezza, Greenplum, and any ODBC compatible databases  ETL products  JMS message queues  MySQL  TimesTen • Windows 2000, 2003, XP • Linux • Sun Solaris • HP NonStop • HP-UX • HP TRU64 • HP OpenVMS • IBM AIX • IBM z/OS • IBM AS400 GoldenGate 支持的异构环境 灵活的拓扑结构 单向复制 备份、查询 双向复制 灾备、双业务中心 点对点 多业务中心 广播复制 数据分发 集中复制 数据仓库、N+1灾备 多级复制 层次化企业数据 GoldenGate – Scaling for Performance - Capture / Extract - Delivery / Replicat - Trail GoldenGate – Data Processing • Data Mapping & Manipulation • Selecting records and columns • Selecting and converting operations • Mapping dissimilar columns • Testing and transforming data • Creating transaction history • Using tokens • Custom Processing • Stored procedure • SQL query • RETVAL for use in DML, FILTER, COLMAP statements • C User Exit  Processing Methods  Online  Batch - Interval based run  Task - Direct mode (Capture -> Delivery over TCP/IP)  Formatting  SQL  ASCII  XML  Bulk Load Utilities - Oracle SQL*Loader - Microsoft BCP/DTS - IBM LOADUTIL Page 25 Management Pack for GoldenGate • Define, configure, manage, and report on GoldenGate TDM processes deployed across your business • Utilizes GoldenGate Management Services architecture • Key features: • Centralized management of any number of GG processes • Rich-client and Web-based interfaces for increased visibility • Alert notifications, integration with 3rd-party monitoring products • Real-time feedback on GG processes • Zero-impact implementation without affecting performance GoldenGate Veridata™ Primary Backup Node BNode A High Availability & Disaster Tolerance Solaris Linux Migrating to New DB, O/S Version = = • High-speed data comparison solution • Audits, identifies, and reports data discrepancies between in-use databases • Benefits: • Reduce risk exposure and act on data discrepancies sooner • Speed and simplify comparison • No disruption to business systems • Key features: • Support for large data volumes • Comparisons ran while data sources are kept online • Unparalleled speed and efficiency • Selective comparison options • Unattended execution • Flexible reporting 28 Active Data Guard Direct Memory Access Read On-Disk Logs SQL Direct Write to Logs Oracle Products for Data Movement Comparing How They Work Redo in Memory Buffer GoldenGate Data Integrator D E C R E A S I N G L A T E N C Y I N C R E A S I N G T R A N S F O R M A T I O N Retrieve Data Load Data Disaster Recovery & Data Protection Real Time Data Integration & High Availability SQL Query Multi-step, Complex SQL Data Integration for Data Warehouse & SOA 概要 • 数据同步应用场景及需求 • GoldenGate介绍 • 成功案例分享 • 问题与交流 Oracle GoldenGate 使用场景 企业级的实时数据解决 方案 气瓶 现场处置方案 .pdf气瓶 现场处置方案 .doc见习基地管理方案.doc关于群访事件的化解方案建筑工地扬尘治理专项方案下载 基于日志的实时 变化数据捕捉 异构的多数据源系统 EDWODS EDW 灾备,数据保护 应用或系统在线升级 操作型报表 实时商务智能 Standby (Open & Active) Reporting Database OGG ETL ETL 查询卸载 数据分发 • Standardize on Single Technology for Multiple Needs • Deploy for Continuous Availability and Real-time Data Access for Reporting / BI • Highly Flexible • Fast Deployments • Lower TCO & Improved ROI AutoZone Boston Consulting Group Cendant/Travelport Duke Energy Federal Aviation Administration Fiserv Solutions Genuine Auto Parts Company Hertz MGM Grand Office of Military Manpower Admin. Orbitz Overstock.com Party City Publix Super Markets Allied Irish Banks ANZ Bank Banca Sella Banco de Credito e Inversion (BCI) Banco Internacional, S.A. - BITAL Banco Nacional de Mexico Banco Popular de Puerto Rico Banco Santander Mexicano Bank Gospodarki Zywnosciowej Bank Mandiri Bank Negara Bank of America Bank of Central Asia Bank of Nova Scotia Bank UTI Bankart d.o.o. BiK Borica China Trust Commercial Bank Citigroup OTP Bank PKO Bank Polski Postbank Rabobank Shanghai Commercial & Savings Siam City Bank State Bank of India State Bank of Mauritius Taitung Business Bank Toronto Dominion Bank Transbank S.A. Twinsoft for Deutsche Bahn Ulster Bank U.S. Bank Unibanco Wachovia Washington Mutual Bank Wells Fargo World Savings Zions Bank Corporation Comerica Commercial Bank of Greece Consorzio Triveneto Spa Emirates Bank International Eurobank Fleet Financial Services FSPA Foereningssparbanken HDFC Bank HSBC Hsin Chu Bank (HC Bank) Huntington Services Corp. ICB Bank ING Postbank JP Morgan Chase Kasaiorn Bank Knet Korean Exchange Bank Muzo National Australia Group National Bank Of Greece Allscripts Antelope Valley Hospital Baptist Health Bronx Lebanon Continuum Health Partners Group Health Cooperative Health Alliance of Cincinnati Huntsville Hospital INOVA Health Systems Iowa Health System Kettering Medical Center Legacy Health System Mayo Clinic Foundation Montefiore Medical Center Nebraska Health Systems North Carolina Baptist Health Oakwood Healthcare Systems OSF Healthcare Park Nicollet Health Services Parkview Health Systems Peace Health Schering-Plough Sharp Healthcare Stanford Medical St. Francis Hospital St. Joseph's Hospital Thomas Jefferson University University of Chicago Hospitals University of Calif- San Fran PreHoldings (ITC Financial) Promisant Pulse EFT Retail Decisions Raymond James Regionsykehuset i Tromsoe Samsung Card Samsung Securities Samsung Winners Speedway Superamerica T-Chek Systems Total Systems Travelers Express UBS VISA Western Union Woori Securities ABN-AMRO ADP Affinitas (Consortia) Alliance & Leicester Assurant Inc. ATH Costa Rica Austrian Payment Systems Bankserv Bolsa de Valores de Columbia Bombay Stock Exchange Bursatec (Mexican Stock Exchange) Capital Securities CashCard CEKAB Certegy CIBC Comdata Federated Investors Fidelity Information Services First Data Corp. First Data Slovakia First Tennessee Girobankard Hyundai ITS Inc JCB International Co. Jih Sun Securities KeyCorp Kookmin Credit LG Card (LGC) Link-UK Market Axess Corporation MasterLink Securities Merrill Lynch National Credit Card Center National Processing Company Pay Point Pay by Touch Polaris Polcard Adelphia Cable AMD BellSouth Cable One Cablevision Systems Corp. Cingular Wireless Comcast Communications Inc. Copperweld Continental Datagraphics Dell USA DirecTV Emergis Inc. Gentronics Hyundai Motor America Postini Procomp Singapore Computer Systems SIS/ADP Sprint Syscom T-Com Texas Electronic Information Comp. Tokyo Information Systems West Corp Westcon Ross Royal Instit. Chartered Surveyors Sabre Holdings Safeway Sandia National Labs – U.S. DoE Shell Oil – Equiva Services Southwest.com State of Pennsylvania U.S. Air Force U.S. Department of Defense US Foodservice Valero Energy Corporation Wal-mart Mexico 客户列表（按照行业划分） Banking Financial and Insurance Services Healthcare Cable, Telco, and Manufacturing eBusiness, Retail, Public/Govt., Services 部分典型客户 美 国 银 行 超过五千八百家分行 及一万六千多部自动 柜员机 热 备 份 数 据 中 心 从 贯 穿 北 美 的 四 台 主 机 (HP NSK) 抓 取 数 据 保 障 ATM 7X24小时 可用 热 备 份 点 主 数 据 中 心 客 户 客 户 客 户 客 户 双向数据复制 GoldenGate has helped the Bank of America accommodate growth. Michele Schwappach, Vice President and Senior Systems Manager 中华人民共和国海关总署 远距离容灾 低带宽 低延时 可随时接管的双业务 中心 快速业务切换 Oracle 9i RAC Oracle 9i RAC Oracle 9i RAC 决策支持库 GoldenGate 单向复制 GoldenGate 单向复制 H2000生产库 Oracle 9i 全国各地41个口岸H2000应用 GoldenGate 双向复制 广州北京 H2000生产库 决策支持库 网络 国家体育总局体彩中心  N+1模式容灾 各省市集中备份 远距离 低带宽 IBM P570h Hp Superdome Hp Superdome IBM P570h HA Oracle 10g R2 RAC Oracle 10g R1 IBM P570h IBM P570h HA Oracle 10g R1 IBM P570h IBM P570h HA Oracle 10g R1 IBM P570h IBM P570h HAOracle 10g R1 IBM P570h IBM P570h HA Oracle 10g R1 IBM P570h IBM P570h HAOracle 10g R1 IBM P570hIBM P570h HA Oracle 10g R1 IBM P570hIBM P570h HA Oracle 10g R1 IBM P570hIBM P570h HA Oracle 10g R1 IBM P570hIBM P570h HA Oracle 10g R1 IBM P570hIBM P570h HA Oracle 10g R1 IBM P570hIBM P570h HA Oracle 10g R1 天津分中心 成都分中心 武汉分中心 广州分中心 北京备份中心 GoldenGate GoldenGate GoldenGate GoldenGate 国家税务总局 集中容灾 远距离 低带宽 问 答