1 YEAR UPGRADE BUYER PROTECTION PLAN
Building
SANs Brocade with
Fa b r i c S w i t c h e s How to Design, Implement, and Maintain Storage Area Networks (SANs) with Brocade Fabric Switches • Step-by-step instructions for establishing your SAN requirements—such as high availability, performance, and cost savings—and translating those requirements into an effective SAN design • Detailed examples to guide you through the process of installing and troubleshooting your Brocade SAN • Practical discussions about SAN components and popular SAN configurations such as storage consolidation, disaster tolerance, and LAN-free backup Chris Beauchamp Author Josh Judd Author Benjamin Kuo Contributor
140_SANs_FM
8/14/01
3:40 PM
Page i
[email protected] With more than 1,500,000 copies of our MCSE, MCSD, CompTIA, and Cisco study guides in print, we continue to look for ways we can better serve the information needs of our readers. One way we do that is by listening. Readers like yourself have been telling us they want an Internet-based service that would extend and enhance the value of our books. Based on reader feedback and our own strategic plan, we have created a Web site that we hope will exceed your expectations.
[email protected] is an interactive treasure trove of useful information focusing on our book topics and related technologies. The site offers the following features: ■ One-year warranty against content obsolescence due to vendor product upgrades. You can access online updates for any affected chapters. ■ “Ask the Author”™ customer query forms that enable you to post questions to our authors and editors. ■ Exclusive monthly mailings in which our experts provide answers to reader queries and clear explanations of complex material. ■ Regularly updated links to sites specially selected by our editors for readers desiring additional reliable information on key topics. Best of all, the book you’re now holding is your key to this amazing site. Just go to www.syngress.com/solutions, and keep this book handy when you register to verify your purchase. Thank you for giving us the opportunity to serve your needs. And be sure to let us know if there’s anything else we can do to help you get the maximum value from your investment. We’re listening.
www.syngress.com/solutions
140_SANs_FM
8/14/01
3:40 PM
Page ii
140_SANs_FM
8/14/01
3:41 PM
Page iii
1 YEAR UPGRADE BUYER PROTECTION PLAN
Building
SANs Brocade with
Fa b r i c S w i t c h e s
Chris Beauchamp Author Josh Judd Author Benjamin Kuo Contributor
140_SANs_FM
8/14/01
3:41 PM
Page iv
Syngress Publishing, Inc., the author(s), and any person or firm involved in the writing, editing, or production (collectively “Makers”) of this book (“the Work”) do not guarantee or warrant the results to be obtained from the Work. There is no guarantee of any kind, expressed or implied, regarding the Work or its contents.The Work is sold AS IS and WITHOUT WARRANTY.You may have other legal rights, which vary from state to state. In no event will Makers be liable to you for damages, including any loss of profits, lost savings, or other incidental or consequential damages arising out from the Work or its contents. Because some states do not allow the exclusion or limitation of liability for consequential or incidental damages, the above limitation may not apply to you. You should always use reasonable case, including backup and other appropriate precautions, when working with computers, networks, data, and files. Syngress Media®, Syngress®, and “Career Advancement Through Skill Enhancement®,”are registered trademarks of Syngress Media, Inc. “Ask the Author™,”“Ask the Author UPDATE™,”“Mission Critical™,” “Hack Proofing™,” and “The Only Way to Stop a Hacker is to Think Like One™” are trademarks of Syngress Publishing, Inc. Brands and product names mentioned in this book are trademarks or service marks of their respective companies.“Brocade®,” “SilkWorm®,” and the Brocade logo are registerd trademarks of Brocade Communications Systems, Inc., in the United States and/or any other countries. KEY 001 002 003 004 005 006 007 008 009 010
SERIAL NUMBER Q3G4T9U2F5 6YHQ94MLE4 VMERKJ6C4N XD7Y4B39UN 8SRT9U6N7H 3W7YRNTEP4 LHB65TR46T 4DB9R5LZMR N835M4KBAZ QT6Y4RTWFC
PUBLISHED BY Syngress Publishing, Inc. 800 Hingham Street Rockland, MA 02370 Building SANs with Brocade Fabric Switches
Copyright © 2001 by Syngress Publishing, Inc. All rights reserved. Printed in the United States of America. Except as permitted under the Copyright Act of 1976, no part of this publication may be reproduced or distributed in any form or by any means, or stored in a database or retrieval system, without the prior written permission of the publisher, with the exception that the program listings may be entered, stored, and executed in a computer system, but they may not be reproduced for publication. Printed in the United States of America 1 2 3 4 5 6 7 8 9 0 ISBN: 1-928994-30-X Technical Editors: Chris Beauchamp, Josh Judd, Benjamin Kuo Acquisitions Editor: Catherine B. Nolan Developmental Editor: Kate Glennon Copy Editor: Beth A. Roberts
Freelance Editorial Manager: Maribeth Corona-Evans Cover Designer: Michael Kavish Page Layout and Art by: Shannon Tozier Indexer: Jennifer Coker
140_SANs_FM
8/14/01
3:41 PM
Page v
Syngress Acknowledgments We would like to acknowledge the following people for their kindness and support in making this book possible. Greg Reyes, Jack Cuthbert, Doug Wesolek, Maggie Conroy, Julie Chiu, Elaine Tite, Jeff Seltzer, and Chris Mingrone at Brocade, for championing the idea of a Brocade SANs book. Also special thanks to Viet Dao, John Gareri, Mark Murphy, Jay Rafati, Ron Totah, Ezio Valdevit, John Bae, James Carpignano, Steve Daheb, Derek Granath, Jay Kidd, Omy Shani, James Bleess, Owen Higginson, Leo Kappeler, Chris M. Nguyen, Mark Peluso, and Henry Robinson for their help in making this book a reality. Ralph Troupe of Callisma for his invaluable insight and guidance. Ralph’s expertise in SAN architecture and design solutions for next-generation storage networking implementations helped define our vision for this book. Richard Kristof and Duncan Anderson of Global Knowledge, for their generous access to the IT industry’s best courses, instructors, and training facilities. Karen Cross, Lance Tilford, Meaghan Cunningham, Kim Wylie, Harry Kirchner, Kevin Votel, Kent Anderson, and Frida Yara of Publishers Group West for sharing their incredible marketing experience and expertise. Mary Ging, Caroline Hird, Simon Beale, Caroline Wheeler,Victoria Fuller, Jonathan Bunkell, and Klaus Beran of Harcourt International for making certain that our vision remains worldwide in scope. Anneke Baeten and Annabel Dent of Harcourt Australia for all their help. David Buckland,Wendi Wong, Daniel Loh, Marie Chieng, Lucy Chong, Leslie Lim, Audrey Gan, and Joseph Chan of Transquest Publishers for the enthusiasm with which they receive our books. Kwon Sung June at Acorn Publishing for his support. Ethan Atkin at Cranbury International for his help in expanding the Syngress program. v
140_SANs_FM
8/14/01
3:41 PM
Page vi
This book was designed and written to provide information about storage area networking architectures. Every effort has been made to make this book as complete and accurate as possible. However, the information in this book is provided to you “AS IS,” without warranty of any kind, including, without limitation, any implied warranty of merchantability or fitness for a particular purpose. The authors and Brocade Communications Systems, Inc., shall have no liability or responsibility to any person or entity with respect to any loss, cost, liability, or damages arising from the information contained in this book or the computer programs that accompany it, and specifically disclaim any implied.
vi
140_SANs_FM
8/14/01
3:41 PM
Page vii
Brocade Acknowledgments This book truly represents a complete Brocade team effort.We would like to acknowledge several people in particular.Without their help, dedication, and knowledge, this book would not have been possible.The thorough technical review by Viet Dao, John Gareri, Mark Murphy, Jay Rafati, Ron Totah, and Ezio Valdevit shaped our manuscripts into a book that Brocade can be proud of. John Bae, James Carpignano, Steve Daheb, Derek Granath, Jay Kidd, and Omy Shani provided several timely contributions to the content.We also incorporated material written by others within Brocade: James Bleess, Owen Higginson, Leo Kappeler, Chris M. Nguyen, Mark Peluso, and Henry Robinson.We would also like to thank Maggie Conroy and Doug Wesolek for their guidance and help with the publication process.
—Josh Judd and Chris Beauchamp
vii
140_SANs_FM
8/14/01
3:41 PM
Page viii
140_SANs_FM
8/14/01
3:41 PM
Page ix
Authors Chris Beauchamp is a Senior SAN Architect for Brocade Communications Systems, Inc. Chris moved to Brocade in 1998 as a Systems Engineer supporting several strategic customers with the application and qualification of SilkWorm fabric switches. Chris now focuses on SAN design and architecture, with an emphasis on scalability and troubleshooting. His specialties include Sun servers, storage performance analysis and capacity planning, Fibre Channel trace analysis, scripting in various languages, and SAN administration. Chris holds a Master of Science in Computer Engineering from Villanova University and a Bachelor of Science in Computer Science degree from West Chester University. Chris’s background includes positions as a Systems Administrator at General Electric and a Systems Engineer at Sun Microsystems. Chris currently resides outside of San Jose in the Santa Cruz Mountains with his wife Sarah and daughter Meagan. Josh Judd is a Senior SAN Architect with Brocade Communications Systems, Inc. In addition to writing technical literature, he provides senior-level strategic support for major OEMs and end-users of Brocade storage network products worldwide.When he first went to work for Brocade, he was the company’s Senior IT Specialist, responsible for escalations in every area of the company’s network, server, and desktop infrastructure. Josh’s career as an IT consultant has given him a diverse skill set, which includes senior-level expertise in several UNIX variants,Windows 9x/NT/2k administration, RAID configuration and optimization, storage virtualization and clustering software (such as that produced by VERITAS Software), and network engineering with many vendors, including Cisco, Foundry, Lucent, and 3com. Before joining Brocade four years ago, Josh worked at IBM Global Services, LSI Logic, and Taos Mountain Consulting. He lives in San Jose, California.
ix
140_SANs_FM
8/14/01
3:41 PM
Page x
140_SANs_FM
8/14/01
3:41 PM
Page xi
Special Contributor Benjamin F. Kuo is a Software Development Manager at TROIKA Networks. Headquartered in Westlake Village, CA,TROIKA Networks is a developer of Fibre Channel Host Bus Adapters, dynamic multipathing, and management software for Storage Area Networks. Ben manages development of network management software at TROIKA and is also active as the chair of the HBA API subgroup of the Storage Networking Industry Association (SNIA), where he spearheads efforts to develop interoperable standards in storage networking. Ben’s background includes positions at Paracel Inc. (now Celera Genomics), IBM, Micropolis, and Echelon Corp. Ben also runs socalTECH.com, a Web site and daily newsletter focused on high tech in Southern California. Ben holds a Bachelor’s degree in Electrical Engineering from the University of Southern California and is a member of the IEEE. Ben lives in Moorpark, California, with his wife Jennifer and son Jonathan.
Contributor Alex Neefus is the Lead Interoperability Test Engineer at Lamprey Networks, Inc. Lamprey Networks offers certification testing services and test tool development to the Fibre Channel industry. Alex has worked on developing testing tools for the SANmark program hosted by the FCIA. This program certifies Fibre Channel devices for conformance and interoperability. Alex has also co-authored and written a number of conformance test suites for both the FCIA and the University of New Hampshire Interoperability Lab. Alex’s background also includes working for the UNH Interoperability Lab in the Fibre Channel Consortium for over a year and a half. At the lab his primary work is in developing procedures and tools for testing Fibre Channel products, and working with members of the industry to solve interoperability problems with devices on the market. Alex resides in Durham, New Hampshire. xi
140_SANs_FM
8/14/01
3:41 PM
Page xii
140_SANs_toc
8/14/01
3:45 PM
Page xiii
Contents
Foreword Learn When to Deploy a SAN
Things to consider when deciding whether a SAN is the right solution: ■ The primary application that needs to be solved ■ Speed, bandwidth, and distance requirements ■ The amount of data sharing or consolidation needed ■ The cost of the SAN infrastructure required, such as switches, cables, and HBAs
Chapter 1 Introduction to SANs Introduction Overview of SANs Taming the Storage Monster Benefits of Building a SAN Ensuring High Availability Consolidating Storage Reducing Network Congestion from Backup Accelerating Backup Cycles Speeding Up Data Access Increasing Server Cycles Ensuring Disaster Tolerance When to Deploy a SAN Designing Around the Application Assessing Speed, Bandwidth, and Distance Requirements Data Sharing and Consolidation Needs Resource Sharing Volume-Level Sharing File-Level Sharing Steps to a Successful SAN Deployment Summary Solutions Fast Track Frequently Asked Questions
xxv 1 2 2 8 10 10 11 13 14 14 14 15 16 16 17 18 19 19 19 20 25 26 28
xiii
140_SANs_toc
xiv
8/14/01
3:45 PM
Page xiv
Contents
Master Fabric Services
Fabric services provide information to nodes in a switched fabric topology. Services can be distributed across all switches, creating the appearance of single-service type servers. In this chapter, we discuss a number of different fabric services, including: Login Server Name Server Fabric/Switch Controller Management Server Time Server
Chapter 2 Fibre Channel Basics Introduction The Architecture of SANs Fibre Channel Protocol Classes of Service Storage Network Topologies Fabric Services Fibre Channel Protocol Basics Fibre Channel Levels ULPs Classes of Service Class 1 Class 2 Class 3 Class 4 Class F Storage Network Topologies Point-to-Point Topology Fibre Channel Arbitrated Loop (FC-AL) Topology Switched Fabric Topology Fabric Services Login Server Name Server Fabric/Switch Controller Management Server Time Server Other Services Summary Solutions Fast Track Frequently Asked Questions Chapter 3 SAN Components and Equipment Introduction Overview of Fibre Channel Equipment Cabling and Media
29 30 30 37 37 37 38 38 40 42 43 43 44 44 44 45 45 45 47 48 49 50 50 51 51 52 52 53 54 57
59 60 61 61
140_SANs_toc
8/14/01
3:45 PM
Page xv
Contents
Understand Fibre Channel Equipment
WARNING Any single-mode or multimode laser can damage your eyes if it is transmitted at 1300 nm. The 1300 nm wavelength is not in the visible spectrum, so you will not see a laser being transmitted like in 850 nm fiber. A 1300 nm laser is dangerous, because it can cause severe retina damage.
GBICs and Connectors Hubs Switches Storage Host Bus Adapters Routers and Bridges Cabling and GBICs Copper Versus Optical: Selecting Your Media Copper Cabling Multimode Optical Cabling Single-Mode Optical Cabling Connecting with Connectors The DB-9 Copper Connector The HSSDC Copper Connector The SC Optical Connector High-Density Fiber-Optic Connectors Comparing GBICs to Fixed Media Using a GBIC Pros and Cons of Using GBICs GBIC Ports on Equipment Serialized Versus Nonserialized Common Problems with GBICs Media Interface Adapters Using Hubs Simple Electrical Hubs Managed Hubs LIP Service: Fibre Channel LIPs, Problems, and Solutions Getting Out of the Loop: Migrating to Switched Fabric Using Switches and Fibre Channel Fabrics Basic Switch Types Entry-Level Switches Scalable Fabric Switches Core Fabric Switches
xv
61 63 63 64 64 64 65 65 65 66 68 69 69 70 71 72 73 73 74 74 74 75 75 76 76 76 78 79 80 80 81 81 81
140_SANs_toc
xvi
8/14/01
3:45 PM
Page xvi
Contents
Features of Fibre Channel Switches Zoning Classes of Service Fabric Services Redundancy Buffer Credits per Port Self-Configuring Ports Auto-Negotiating Speeds IP over Fibre Channel Broadcasting Firmware Upgrade Methods Loop Operation: Making Your Switch Act Like a Hub FSPF Compliance Management Interfaces Serial Port Telnet SNMP Web-Based Management Application-Based Management SCSI Enclosure Services Connecting Your Servers with Host Bus Adapters Connecting Hosts to the Fabric HBA Types Speeds Ports Combination Adapters Fabric-Capable Versus Loop Adapters HBA-Based LUN Masking Persistent Binding Default LUN Access Permissions Upper-Level Protocol Access Permissions Dynamic Versus Static Discovery Configuration Management Software HBA API Support Remote Boot across the SAN Hot-Plug Support
82 83 84 85 86 86 87 88 88 89 90 90 91 91 91 91 93 94 94 95 95 95 97 98 98 98 99 99 100 100 101 101 101 103 104
140_SANs_toc
8/14/01
3:45 PM
Page xvii
Contents
Connecting Legacy Devices into Your SAN Basic Features of Routers Number of SCSI Buses Types of SCSI Ports,Termination Selective LUN Presentation Extended Copy Support Management Interfaces Bridging and Routing to IP Networks and Beyond Fibre Channel to DWDM Fibre Channel across IP Networks IP over Fibre Channel to Gigabit Ethernet Fibre Channel Storage Individual Disk Drives and JBODs High-End Storage Arrays Selective LUN Presentation LUN Export across Multiple Ports Snapshot Backup Volumes Summary Solutions Fast Track Frequently Asked Questions Simplify SAN fabric management with Brocade WEB TOOLS
Brocade WEB TOOLS is a software utility that enables you to manage and monitor your fabric through a Web browser interface and Java plug-in. Using WEB TOOLS, you can view all switches in the SAN from a single interface from any workstation in your enterprise—even at a remote location.
Chapter 4 Overview of Brocade SilkWorm Switches and Features Introduction Selecting the Right Switch Entry-Level Switches SilkWorm 2010 (8 Ports) and 2210 (16 Ports) SilkWorm 2040 (8 Ports) and 2240 (16 Ports) SilkWorm 2050 (8 Ports) and 2250 (16 Ports) Scalable Fabric Switches SilkWorm 2400 SilkWorm 2800
xvii
106 106 107 108 108 108 109 109 109 110 110 111 111 113 113 113 114 115 116 121
123 124 124 126 127 127 128 128 129 129
140_SANs_toc
xviii
8/14/01
3:45 PM
Page xviii
Contents
SilkWorm 6400 Integrated Fabric SilkWorm 12000 Core Fabric Switch Understanding the Brocade Fabric OS Fabric OS Core Functions Fibre Channel Services for Reconfiguration Dynamic Routing Services Facilities for End-to-End SAN Management Brocade Command Line Interface Using Optional Brocade Features Brocade Zoning Extended Fabrics Fabric Watch Understanding Loop Support, QuickLoop, and Fabric Assist Brocade WEB TOOLS Future Capabilities in the Brocade Intelligent Fabric Services Architecture Brocade ISL Trunking Brocade Frame Filtering More Robust Hardware-Enforced Zoning Enhanced End-to-End Performance Analysis Secure Fabric OS Summary Solutions Fast Track Frequently Asked Questions
Chapter 5 The SAN Design Process Introduction Looking at the Overall Lifecycle of a SAN Data Collection Data Analysis Architecture Development Prototype and Testing Transition Release to Production
130 131 132 133 133 134 135 135 135 136 136 138 138 139 140 140 142 142 143 143 144 144 146
149 150 151 153 153 153 153 154 154
140_SANs_toc
8/14/01
3:45 PM
Page xix
Contents
Master the seven phases of the SAN design lifecycle:
1. Data Collection 2. Data Analysis 3. Architecture Development 4. Prototype and Test 5. Transition 6. Release to Production 7. Maintenance
Maintenance Conducting Data Collection Creating an Interview Plan Conducting the Interviews What Overall Business Problem Are You Trying to Solve? What Are the Business Requirements of the Solution? What Is Known about the Nodes that Will Attach to the SAN? Which SAN-Enabled Applications Do You Have in Mind? Which Components of the Solution Already Exist? Which Components Are Already in Production? Which Elements of the Solution Need to Be Prototyped and Tested? What Equipment Will Be Available for Testing? How and When Are Backups to Be Done? What Will Be the Traffic Patterns in the Solution? What Do We Know about Current Performance Characteristics? What Do We Know about Future Performance Characteristics? How Much Downtime Is Acceptable to Production Components During Implementation? How Much Downtime Is Acceptable for Routine Maintenance? How Much Downtime Is Acceptable for Upgrades and Architectural Changes? When Do You Need Each Piece of the Solution to Be Complete?
xix
155 156 156 157 158 158 160 165 165 166 166 166 167 167 168 172
174
174 175
140_SANs_toc
xx
8/14/01
3:45 PM
Page xx
Contents
Answer Your Questions about SAN Applications and Configurations
Q: I would like to cluster my databases for better performance. What databases can I use?
A: Most major databases now support fabric switch-based clustering, including Oracle Parallel Server, IBM DB2 Parallel Edition, and Microsoft SQL Server.
Q: I would like to have my Exchange Mail Server highly available. What should I do?
A: Brocade has developed HA solutions for the Exchange Server that can be used in setting up your desired SAN configuration.
Summary List of Questions Conduct a Physical Assessment Analyzing the Collected Data Processing What You Have Collected Establishing Port Requirements Simple Case Moderate Case Complex Case Preparing an ROI Analysis The Return On Investment Proposition The Rest of the Process and the Repetition of the Cycle Summary Solutions Fast Track Frequently Asked Questions
Chapter 6 SAN Applications and Configurations Introduction Configuring a High-Availability Cluster Typical HA Application or Database Server Microsoft Cluster Server Using a SAN for Storage Consolidation Shared Storage Using a Web Farm Storage Partitioning Using Switch Zoning Switch Zoning Configuration for Departmental SANs Storage Partitioning Using Storage LUN Masking Storage Partitioning Using HBA LUN Masking Partitioning with Software LAN-Free Backup Configuration SAN Server-Free Backup SAN-Based Third-Party Copy Data Movers Making Your Enterprise Disaster Tolerant Data Replication and Remote Backup
176 176 177 177 182 183 185 186 187 188 190 191 192 193
195 196 196 198 200 203 206 208 208 210 210 211 212 213 215 216 218
140_SANs_toc
8/14/01
3:45 PM
Page xxi
Contents
Metropolitan Area Network Solutions Summary Solutions Fast Track Frequently Asked Questions
Develop a SAN Architecture
xxi
219 222 222 226
Chapter 7 Developing a SAN Architecture Introduction Identifying Fabric Topologies and SAN Architectures Useful Topologies Scalability Cascade Topology Ring Topology Mesh Topologies Core/Edge or Star Topologies Topologies at a Glance Complex Topologies Working with the Core/Edge Topology Scaling without Downtime Adding an Edge Switch Upgrading the Core Determining Levels of Availability Configuring Traffic Patterns Leveraging Tiers Exploiting Locality Using Any-to-Any Connectivity Evaluating Performance Considerations When Is Over-Subscription Bad? Considerations Outside the Fabric Summary Solutions Fast Track Frequently Asked Questions
227 228
Chapter 8 SAN Troubleshooting Introduction The Troubleshooting Approach:The SAN Is a Virtual Cable
277 278
229 235 236 236 237 238 242 246 246 246 248 248 250 256 261 261 266 268 269 270 270 272 273 275
278
140_SANs_toc
xxii
8/14/01
3:45 PM
Page xxii
Contents
SAN Troubleshooting
When you start the troubleshooting process, determine whether the issue is fabric-related or device-related. A fabricrelated issue impacts many devices, while a device-related issue affects only a few devices.
A Typical Scenario: “I Cannot See My Disks” Where to Start and What Data to Gather Take a Snapshot: Describe the Problem and Gather Information Troubleshooting Tools Using the Switch LEDs Switch Diagnostics Helpful Commands SAN Profile What Data Can a Host Provide? When to Use portLog and Other Advanced Tools Troubleshooting the Fabric What to Look for in a Malfunctioning Fabric Host Behavior SAN Profile Switch LEDs The errShow Command The switchShow Command The topologyShow Command The nsShow and nsAllShow Commands Now that You Suspect a SAN Issue: Digging Deeper Timeout of Edge Devices During Fabric Bring Up Port Configuration Conflict or Missing Fabric License Segmented Fabrics Troubleshooting Devices that Cannot Be Seen What to Look for in the Fabric Are the Host and Storage Visible via switchShow on Their Respective Switches? Do the Devices Show Up in the Name Server? Rule Out Zoning Issues Edge Device Not in the Name Server
279 283 284 287 287 289 290 308 312 314 316 317 317 317 318 318 318 320 320 321 321 322 323 327 329 329 332 333 334
140_SANs_toc
8/14/01
3:45 PM
Page xxiii
Contents
Troubleshooting Marginal Links Marginal Point-to-Point/Fabric Device Links Marginal Loop Connections Nx_Port (Host/Storage) Behavior with a Marginal Port in the Loop Marginal GBIC/Cable Connected Device Fault Isolation How the Switch Can Help: Fabric Watch and QuickLoop Zoning Overview of SilkWorm Port Error Statistics Troubleshooting I/O Pauses Summary Solutions Fast Track Frequently Asked Questions
Chapter 9 SAN Implementation, Maintenance, and Management Introduction Installation Considerations Use licenseShow to How to Cable Your SAN for Ease of Determine What Operation Licenses Are Installed Racking Considerations on Your Switch In-Band or Out-of-Band Management? IPFC In-Band Guidelines core1:admin> licenseShow Setting Switch Parameters SRzy9Sz9zeTS0zAG: What Fabric OS Version Should I Use? Licenses Web license Automating Switch Administration Activities bbSz9eQb9zccT0AQ: Fabric OS APIs Zoning license Expect Scripting RdzdSRcSyzSe0eTn: A Switch Management Wrapper QuickLoop license Using Expect Brocade Zoning Considerations cSczRScd9RdTd0SY: Where to Zone? Fabric license Hard Zoning or Soft Zoning?
xxiii
335 335 337 338 338 339 339 339 341 342 344 345 347
349 350 351 351 354 356 357 358 361 366 367 367 369 369 372 373 375
140_SANs_toc
xxiv
8/14/01
3:45 PM
Page xxiv
Contents
Hard Zoning and Soft Zoning Differences Zone Management Scripting Zoning Operations Zoning Tips Validating Your Fabric Baseline Your SAN Profile Fault Injection Running an I/O Load Types of Load I/O Generators SAN Maintenance The Configuration Log: Key Information to Gather and Maintain about Your SAN Backing Up and Restoring a Switch Configuration Bringing Up a Fabric Expanding a Fabric: Merging Fabrics, Adding a Switch, or Replacing a Switch Upgrading Your Fabric Issues Applicable to Both Hot and Cold Upgrades Performing a Hot Fabric Upgrade Performing a Cold Fabric Upgrade How to Automate firmwareDownload Replacing or Adding an Edge Device in the Fabric Summary Solutions Fast Track Frequently Asked Questions
378 378 379 381 382 382 384 385 386 387 391 391 393 394 395 398 398 399 400 400 401 403 405 408
Appendix Building SANs with Brocade Fabric Switches Fast Track
409
Index
431
140_SANs_fore
8/14/01
3:44 PM
Page xxv
Foreword
Why Write a Book about SANs? During the last few years, Storage Area Networks (SANs) have fundamentally changed the way organizations design, build, and manage their enterprise networks. As a superior alternative to direct-attached storage models, SANs have enabled a wide range of new configurations and applications. In turn, those applications have generated a variety of benefits for the organizations that have implemented them.These advantages include superior scalability, simplified storage management, optimized resource sharing, higher availability, and tremendous cost savings to name just a few. As a primary facilitator of the networked storage model, Brocade Communications Systems actively seeks out new opportunities to raise industry awareness about the value of SANs. One of our primary goals at Brocade is to help educate all kinds of organizations about the advantages a networked storage environment can offer. Based on feedback from our customers and business partners, we realized that there was no self-contained, effective guidebook for implementing Fibre Channel SANs.To help fill that void, we have joined with Syngress Publishing to bring you Building SANs with Brocade Fabric Switches. This book details the design, installation, configuration, and troubleshooting of Brocade-based SANs —basically everything you need to know before beginning your own SAN implementation.
Who Should Read This Book? Building SANs with Brocade Fabric Switches is written for anyone who plans to design, build, and manage SANs using Brocade switches and software. In particular, this book provides a “how to” reference that describes what you can do today, given the technologies currently available. xxv
140_SANs_fore
xxvi
8/14/01
3:44 PM
Page xxvi
Foreword
By necessity, the focus is Brocade-centric and features the theory of operation behind Brocade SilkWorm switches and Fabric OS. However, this book is not intended to be a comprehensive guide for every configuration and scenario possible. After all, with the rapid expansion of the SAN marketplace, there will undoubtedly be other technologies available in the not-so-distant future.
What Does the Book Contain? In addition to providing an overview of current technology, tools, products, and design topologies, this book should serve as a guideline for actual SAN implementation. For instance, the book begins with a detailed analysis of technology requirements and the benefits of implementing a SAN. Next, you can learn about Fibre Channel concepts and definitions as well as the full range of SAN components. We then introduce you to the Brocade SilkWorm series of Fibre Channel switches, including guidelines for integrating these switches into your existing IT environment.The book concludes with examples of design processes, popular SAN applications, and detailed troubleshooting and maintenance tips. In addition, each chapter features a high-level summary and FAQs for anyone who needs a quick overview of the SAN basics. Our goal is to make this book a valuable tool for implementing your own SAN infrastructure and teach how a well-designed SAN can deliver a competitive advantage for your organization.We welcome your feedback on our efforts. If you have any comments or suggestions about this book, please let us know at www.syngress.com/solutions.
—Kumar Malavalli Vice President,Technology Brocade Communications Systems
www.syngress.com
140_SANs_01
8/14/01
1:32 PM
Page 1
Chapter 1
Introduction to SANs
Solutions in this chapter: ■
Overview of SANs
■
Taming the Storage Monster
■
Benefits of Building a SAN
■
When to Deploy a SAN
■
Steps to a Successful SAN Deployment
; Summary ; Solutions Fast Track ; Frequently Asked Questions
1
140_SANs_01
2
8/14/01
1:32 PM
Page 2
Chapter 1 • Introduction to SANs
Introduction In the early 1980s, direct-attach disk storage through interconnects such as Small Computer Systems Interface (SCSI) was the standard way to connect to data.This worked well for the amount of data typically handled at the time, and became the standard way to connect high-speed, high-performance storage to computer systems. However, as computer systems increased in speed and data storage needs increased, the parallel bus architecture of SCSI soon started hitting performance and distance limits. In response to these needs, Fibre Channel was developed to provide gigabit-speed serial networking capabilities for storage. Fibre Channel includes support for the SCSI protocols for storage, the Internet Protocol (IP) for networking, and the Virtual Interface (VI) protocol for clustering, which are mapped onto a network architecture.The Fibre Channel standard combines long distances of up to 10 km, simplified serial cabling over multiple media types, gigabit speeds, and the ability to simultaneously use more than a single protocol over the same wire.These features won adoption for Fibre Channel throughout the 1990s as a replacement for parallel SCSI, and Fibre Channel is now used for most high-capacity, high-end direct storage devices. With the advent and market acceptance of Fibre Channel as a point-to-point replacement for parallel bus SCSI, a new technique has emerged that combines pure storage usage with networking—the Fibre Channel Storage Area Network (SAN). A SAN is a network of storage and system components, all communicating on a Fibre Channel network, that can be used to consolidate and share storage, provide high-performance links to data devices, add redundant links to storage systems, speed up data backup, and support high-availability clustered systems. This chapter provides an overview of what a SAN is, what types of problems are best solved with a SAN, and some steps to make a SAN deployment successful. After reading this chapter, you should be able to determine if you should deploy a SAN, identify the types of applications best solved by SAN technology, and be ready to create a deployment plan for your SAN.
Overview of SANs Throughout the 1980s, the standard way of connecting hosts to storage devices was point-to-point, direct-attach storage through interfaces such as Integrated Drive Electronics (IDE) and parallel SCSI (Figure 1.1). Parallel SCSI offered
140_SANs_01
8/14/01
1:32 PM
Page 3
Introduction to SANs • Chapter 1
relatively fast (5 or 10 Mbit/sec) access to SCSI-enabled disks, and several disks could be connected at once to the same computer through the same interface. This worked well for the time, with fairly reliable, fast-speed connections allowing administrators to connect internal and external storage through just simple ribbon cabling or multiconductor external cables. However, as storage subsystems became larger and larger and computers faster and faster, a new problem emerged—external storage (which at one time was just a simple disk drive on the desk next to a machine) started to get bigger.Tape libraries, Redundant Array of Inexpensive Disks (RAID) arrays, and other SCSI devices began to require more and more space—requiring the parallel SCSI connection to be stretched farther and farther away from the host. Input/Output (I/O) rates also increased, pushing on the physics of keeping signal integrity in a large bundle of wires (32- and 64-bit data bus widths). Simple parallel SCSI variants were devised to enable longer distances and to address the signal integrity issues. However, they all eventually ran up against the difficulties of high-speed signals across the parallel SCSI bus architecture. Figure 1.1 Parallel SCSI Bus Connection SCSI ID 1
SCSI ID 2
SCSI ID 3
Host
Parallel SCSI Bus
The solution to all of this was slow in coming, but eventually the storage industry settled on using a serial protocol with high-speed transceivers—offering good noise immunity, ease of cabling, and plentiful bandwidth. Different specifications (Serial Storage Architecture [SSA] and Fibre Channel as well as more advanced parallel SCSI technologies) competed for adoption, and companies began experimenting with different serial communications media. New highspeed circuits made serial transfers (using a simple pair of wires to transmit bits serially, in order, rather than a large number of wires to transfer several bytes or
3
140_SANs_01
4
8/14/01
1:32 PM
Page 4
Chapter 1 • Introduction to SANs
words of data at a time) the most practical solution to the signal problems.The high speed of the circuits enabled the data rates for Fibre Channel to offer up to 100 Mbit/sec transfers, versus the slower 10 to 20 Mbit/sec parallel limitations. When Fibre Channel was first applied to the area of storage connections, the primary reason for the technology was for the extended distances and simplified cabling that the technology offered.This extension of direct-attach operation basically replaced the old parallel SCSI attachments with a high-speed serial line (Figure 1.2).The new Fibre Channel connections offered a much faster interface and simplified cabling (four copper wire connections through DB-9 connectors, as well as optical cabling), and could be used to distribute storage as far as 10 km away from a host computer, or 30 km away with optical extenders. Figure 1.2 Using Fibre Channel to Extend Distances from Storage Host Storage Array
Fibre Channel Link Up to 10 km
The connections to disks at this time began using the Fibre Channel Arbitrated Loop (FC-AL) protocol, which enabled disks to negotiate their addresses and traffic on a loop topology with a host (Figure 1.3). Because of the combined ability to easily cable and distribute storage, users were now able to add separate racks of equipment to attach to hosts. A new component, the Fibre Channel hub, began to be used to make it easier to plug in devices.The hub, a purely electrical piece of equipment that simply connected pieces of a Fibre Channel loop together, made it possible to dynamically add and remove storage from the network without requiring a complete reconfiguration. As these components began to be used in increasingly more complex environments, manufacturers began to add “intelligence” to these Fibre Channel hubs, enabling them to independently deal with such issues as failures in the network and noise in the network from loops being added and removed. An alternative to the hub came in the form of the Fibre Channel switch, which, unlike a hub, was not just connecting pieces of a loop, but instead offered the packet-switching ability of traditional switches.
140_SANs_01
8/14/01
1:32 PM
Page 5
Introduction to SANs • Chapter 1
Figure 1.3 Arbitrated Loop Disk Configuration Attached to a Single Host Disk
Disk
Disk
Fibre Channel Arbitrated Loop
Host
Disk
Disk Disk
Because there was now a Fibre Channel network available, other hosts (not storage) were added to take advantage of the same network.With the addition of SAN-aware software, it was suddenly possible to share storage between two different devices on the network. Storage sharing was the first realization of the modern SAN, with companies in the multimedia and video production areas paving the way by using the Fibre Channel network to share enormous data files between workstations, distribute jobs for rendering, and make fully digital production possible (Figure 1.4). The next big step in Fibre Channel evolution came with the increased reliability and manageability of a Fibre Channel switched fabric. Early implementations of FC-AL were sometimes difficult to manage, unstable, and prone to interoperability problems between components. Because the FC-AL protocol was quite complex, what sometimes would happen would be an inability for anything to communicate and stay operational on a loop.The solution to this was a move to a switched fabric architecture, which not only enhanced the manageability and reliability of the connection, but provided switched, high-speed connections between all nodes of a network instead of a shared loop. As a result, each port on a switch now provides a full 1 Gbit/sec of available bandwidth rather than just a portion of the total 1 Gbit/sec of bandwidth shared between all the devices connected to the loop. Fabrics now make up the majority of Fibre Channel installations. A typical Fibre Channel switched fabric installation (Figure 1.5) has multiple hosts and storage units all connected into the same Fibre Channel network cloud through one or more Fibre Channel switches.
5
140_SANs_01
6
8/14/01
1:32 PM
Page 6
Chapter 1 • Introduction to SANs
Figure 1.4 Multiple Host Arbitrated Loop for Storage Sharing Disk
Disk
RAID
Fibre Channel Arbitrated Loop
Host
Disk
Host Host
Figure 1.5 Switched Fabric, Multiple Host, and Storage Unit Configuration JBOD JBOD
RAID
Fibre Channel Switch
Tape
JBOD Host
Host
Host
140_SANs_01
8/14/01
1:32 PM
Page 7
Introduction to SANs • Chapter 1
Today, the modern SAN looks much like any other modern computer network. Network infrastructures such as switches, hubs, bridges, and routers help transport frame-level information across the network. Network interface cards interface computer systems to the same network (called HBAs in the SAN world, as they replaced SCSI Host Bus Adapters). Figure 1.6 shows an example of how these components could be used in conjunction with Fibre Channel switches. Figure 1.6 Typical Deployed SAN Configuration with Multiple Hosts, Storage, and Tape Devices Legacy Parallel SCSI Storage RAID Array
Web Server
Database Server Fibre Channel-toSCSI Router HBA Remote SAN
Fibre Channel Switch HBA Fibre Channel Switch
Fibre Channel Switch ISL (Inter-Switch Link)
Fibre Channel-toDWDM Bridge
HBA
Fibre Channel Cloud Host
Fibre Channel Switch
Fibre Channel Hub
Storage Array
HBA Fibre Channel-toSCSI Router
Host Tape Array
JBOD
7
140_SANs_01
8
8/14/01
1:32 PM
Page 8
Chapter 1 • Introduction to SANs
Resources for SAN Information Rather than relying on just the equipment vendor, an effective way to learn and become an expert on the technology is to track the industry and attend conferences, meetings, and tutorial sessions about the subject. Additional resources for learning more about SAN technology are the industry organizations devoted to this area. The Storage Networking Industry Association (SNIA) offers white papers and educational resources, holds technical tutorial sessions and Storage Networking World conferences, and supports both the users and vendors involved in the storage networking field. More information can be found at www.snia.org. The Fibre Channel Industry Association (FCIA) provides resources for users and vendors, conducts the SANmark suite of Fibre Channel interoperability tests, and holds conferences and meetings to help promote Fibre Channel technology. Their site can be found at www.fibrechannel.org.
Taming the Storage Monster The advent of SANs has been driven by today’s insatiable appetite for storage. The Internet, e-mail, multimedia, and the increasing digital nature of society have resulted in an ever-increasing demand for ways to store, retrieve, and back up that data. For example, e-mail has been on a staggering growth path in the last few years, as more and more people have gone online and businesses have made e-mail a critical part of their communications infrastructure. According to the Year-End 2000 Mailbox Report, there are over 891 million e-mail mailboxes now in existence. Corporate mail usage grew 34 percent in 2000, bringing with it a huge increase in the need for data storage to save all of that e-mail. Multimedia attachments, the movement of business processes to e-mail, and just the sheer volume of e-mail have made the storage and backup of e-mail one of the most pressing requirements of IT departments.
140_SANs_01
8/14/01
1:32 PM
Page 9
Introduction to SANs • Chapter 1
The Internet has also affected the need for storage, with increasing numbers of Web servers and storage required to support those Web servers. As information is increasingly digitized and published on the Web, there is an insatiable appetite for storage to contain that information. Music and full-motion video, even with compression, take an immense amount of disk space, and the movement of studios and companies to run a “full digital” shop has resulted in an enormous demand for storage capacity. Databases, which used to be considered big if they were gigabytes in size, are now well beyond a terabyte—with companies talking about eventually having to manage petabytes of database storage. In addition, with caching servers,Web load balancing, and Web farms built to distribute the processing load for Web traffic, the data being presented on Web sites has to be duplicated 10, 20, and even 100 times to serve those distributed hosts with information.With the increased connectivity of the Internet, information and content are being generated and distributed faster than ever before in history—so much, in fact, that the University of California at Berkeley recently released a study that claims that more data will be created in the next two years than was produced in the history of mankind. All of this data has to go somewhere, and it has exceeded the space available and beyond what can practically be managed on local, direct-attached storage to hosts. Because local storage is relatively fixed and difficult to expand, and because its local nature is difficult to manage, organizations have started to look for a better way to manage this data.The solution has come in the form of very large storage arrays, capable of storing terabytes and terabytes of data, and farms of inexpensive Just A Bunch Of Disks (JBOD). All of this needs to be connected, and the logical way to connect high-speed, block-oriented traffic is through a Fibre Channel SAN. Increased manageability, the ability to centrally manage storage, and consolidation of storage space have made the SAN a necessity in any growing enterprise. Data growth is increasing at such a rapid pace that IBM recently reported that storage sales now exceed server sales at a 70:30 ratio.The requirements to store data are increasing at a greater rate than the requirement for CPU cycles, and the entire industry is changing as a result.This gain has meant that data is now managed separately from the machines that consume that data, making SANs an ideal choice to break the dependency of hosts from the storage, and increasing the manageability and usability of a corporation’s investment in data storage.
9
140_SANs_01
10
8/14/01
1:32 PM
Page 10
Chapter 1 • Introduction to SANs
Implementing a SAN is an ideal technique for taming the storage requirement monster that has resulted from the growth of the Internet and increased connectivity of our electronic age.
Benefits of Building a SAN A number of practical, real-world uses for SANs have emerged in recent years. Knowledgeable administrators have figured out the types of problems that SAN technology best solves. SANs are typically used for the most business-critical, technically challenging problems a company faces. Critical, high-availability systems used for e-mail, database, and file servers have been the first to switch to SANs. A need to consolidate storage and centrally manage volumes has resulted in a trend toward using SANs for storage consolidation.With the increase in data growth, backups have also become a problem, with companies looking to accelerate backup cycles. Protocols such as IP available on Fibre Channel also make SANs attractive for some general networking applications, and VI clustering support allows installations to leverage their SAN infrastructure for VI-enabled clustering applications. Finally, the distance capabilities of Fibre Channel and bridges to Metropolitan Area Networks (MANs) and even Wide Area Networks (WANs) have enabled a new level of disaster tolerance for storage resources.
Ensuring High Availability As the Internet and digital data have grown exponentially,Web caching techniques,Web load balancers and distributed server clusters, and other techniques have been used to handle the demands of serving up Web requests for static pages. Images, files, and Web pages that do not change often can be copied across a bank of hosts, all of which can service a request from a user. However, these techniques cannot be applied in many critical applications. For example, an e-mail server requires one single, consistent image for e-mail storage. Back-end databases of e-commerce applications require combining live, real-time inventory data with live pricing data to service requests correctly. None of these can be cached across a Web server due to the real-time, non-cacheable nature of the information.This dependency on a consistent, single image of data cannot be solved by just replicating data or sharing across a cluster.The result is a new, critical point of failure in the e-mail server or database. Especially with the growth in data, more and more vital data is being trusted to those single points of
140_SANs_01
8/14/01
1:32 PM
Page 11
Introduction to SANs • Chapter 1
failure, raising the stakes and potential losses if those services go down.The concern over these critical points of failure has resulted in a renewed focus on highly available (fault tolerant) solutions, particularly in the storage area. In combination with failover software packages such as Microsoft Cluster Server or VERITAS Cluster Server, high-availability hardware and software has come to the forefront in ensuring the performance and availability of these critical systems. For example, one area where the use of SANs is ideal has been the use of high-availability solutions for managing and running very large Microsoft Exchange databases.With the immense increase in data stored in Exchange servers all over the world, there has been an increase in the amount of back-end storage required for serving those Exchange installations. Because of the nondistributed nature of Exchange mail databases, there has been a concentration of data tied to single hosts and storage units—a single point of failure that could cripple many businesses.The natural solution has been to use application clustering techniques combined with a robust, fully redundant high-availability SAN to support those clusters and share redundant storage between hosts. High-availability systems are now regularly used for ensuring fault-tolerant access to storage. A focus on eliminating single points of failure has stimulated demand for fault-tolerant equipment configurations, specific fault-tolerant network equipment, and techniques for ensuring high availability. SAN technology is ideal for these types of solutions. It allows host-to-host connectivity for heartbeat, equipment status, and network communications, as well as for sharing critical storage between alternate and backup servers. The availability of SAN connections has solved one of the big problems with high-availability, clustered installations: access to the same data across a network. In combination with high-availability features in storage arrays and other equipment, the SAN allows for multiple redundant paths to be made from multiple redundant hosts, dramatically increasing the reliability of critical systems. In addition, with flexible SAN interconnections, the large amount of data that needs to be accessed can more easily be managed separately, rather than being captive to a potential failure in a host.
Consolidating Storage As data needs have increased, it has become increasingly difficult to manage the hundreds of hosts and local disks attached to those hosts. In order to manage this growth, administrators have begun to centralize their storage resources. Large storage arrays and pooled storage are much more efficient and infinitely more manageable than local storage.
11
140_SANs_01
12
8/14/01
1:32 PM
Page 12
Chapter 1 • Introduction to SANs
As storage needs have increased, the model of attaching local storage to hosts has broken down. Administrators figured out that, even though a company as a whole might own enough storage for all of its needs, that storage was not necessarily in the right place. For example, a Web server might be running out of space, with no more space available on local disks and not enough SCSI connections to add more external storage, while the database server next to it has gigabytes free. In the old model of local storage, there was no way to take advantage of that fact. You ended up purchasing much more storage than you needed, because you had a very low rate of utilization—yet you never had enough capacity.You also ended up purchasing more servers than you needed, because you did not need more CPU cycles, but rather, more storage slots. With the advent of the Fibre Channel network, the ability of both clients and storage to coexist and share storage has spawned a new crop of solutions that take advantage of that sharing. Sharing of storage, which previously was limited to vertical markets such as video editing and multimedia, has become a general technique used anywhere that storage is more easily managed in a pool, such as in Internet Service Provider (ISP) and Application Service Provider (ASP) installations. Indeed, most corporate IT environments can take advantage of this technique. Through software such as VERITAS Volume Manager,Tivoli SANergy, and DataCore SANsymphony, users are now able to allocate and share storage among multiple hosts. By using the SAN infrastructure, large centralized pools of disks can be divided between hosts, and new volumes allocated as needed from the general pool.This results in a huge increase in efficiency in use of storage, eliminating the pools of expensive, local, unusable storage. Instead, one large, easily managed virtual storage pool can be centrally administered, and storage costs and administration centralized and consolidated. Sharing is accomplished through this high-level software, which discovers and manages all of the storage on the network. Drivers and software in the host manage which machines do and do not get access to a specific part of a storage device. In general, a central administrator is able to allocate arbitrary pieces of storage to specific hosts, all while the network and all hosts are running in real time. A typical example of this is an ISP with a large number of user Web homepage accounts. Extensive pools of clustered and independent Web servers help to ease the traffic load and provide redundancy on the Internet, while being tied into a single- or dual-redundant SAN. Storage allocation and sharing software is run on all of these hosts, and the different Web homepage accounts are allocated to different Web servers.When a failure on a host or storage device occurs, either an
140_SANs_01
8/14/01
1:32 PM
Page 13
Introduction to SANs • Chapter 1
automated process or manual intervention will re-allocate those user volumes to another Web server, or fail over to another storage device, resulting in uninterrupted service and no dependency of specific users on a local disk. In some cases, multiple Web servers can access the same, read-only data on the SAN, providing a high-bandwidth pipe and eliminating the need for expensive, redundant copies of the same data.
Reducing Network Congestion from Backup A typical problem any administrator faces is that of data backup. Because of the huge growth in data, even on local disks, and the increasing criticality of the data stored on networks, backup has become very important. Software packages such as VERITAS NetBackup, Legato NetWorker, and other packages have long relied on agents that transport data over IP connections to a central backup host.The result has been a noticeable slowdown due to the vast amount of data being transported across these IP packets over Ethernet connections—and not just late at night.The backup window for many enterprises has extended from overnight to include hours of peak system operation, simply because there is too much data to fit into the more traditional and convenient backup windows. Anecdotal stories from system administrators illustrate how entire corporate networks have become swamped with daily backups over IP, slowing not only e-mail, but critical file servers, print servers, and Web access. Some shops have gone as far as to install separate, high-speed Ethernet networks in an attempt to offload this problem. SANs lend themselves to several techniques that directly help the backup problem. One of these techniques is the use of IP over Fibre Channel to offload the network congestion on the Ethernet network. IP, when transported over Fibre Channel, is identical in form and function to IP over Ethernet and other networks.Taking advantage of the fact that there are already Fibre Channel connections into a network for access to shared data, administrators have installed IP drivers into their servers and entirely offloaded the backup function onto the Fibre Channel network.This frees the corporate Ethernet from the immense job of transporting IP backup traffic, and takes advantage of the increased bandwidth efficiency that is characteristic of Fibre Channel. Due to the connection-oriented protocols built on Fibre Channel, IP traffic impacts the Fibre Channel network less and helps administrators gain better usage out of their networks. In addition, an increasing number of applications can perform shared backups over a SAN using the backup devices’ native SCSI protocol, which greatly increases the efficiency of the backup process.
13
140_SANs_01
14
8/14/01
1:32 PM
Page 14
Chapter 1 • Introduction to SANs
Accelerating Backup Cycles Another reason for SAN implementation, which attacks the problem of overall backup cycles, has been the development of the technique of third-party copy. Taking the advantages of the Fibre Channel network a step further, specialized hardware called data movers work in conjunction with next-generation backup software to skip the IP transport of backup data entirely; they directly move storage that needs to be backed up from storage devices on the network to tape backup devices on the same network. Because the transfer is direct, it is very fast (no copy to server memory), and drastically reduces the CPU processing power needed for backup.The time spent copying data to and from storage to local host memory frees up valuable CPU cycles for something else: for example, running the applications that the host was installed to run. Companies such as Chaparral Network Storage and Crossroads Systems have been developing these third-party copy devices as part of their Fibre Channel-to-SCSI bridges, in conjunction with different backup vendors, who are now able to move data across a Fibre Channel network without the intervention of hosts.
Speeding Up Data Access The keyword to SANs is speed, speed, and more speed. As a block-level protocol, SCSI over Fibre Channel Protocol (FCP) is the fastest and most efficient networking technology available to transport block-type data from storage to hosts. Companies that previously were using TCP/IP-based networking technology over Ethernet and attempted to migrate that to Gigabit Ethernet have found that, despite the similar wire speeds of the technology, the efficiency and protocols available don’t allow for the use of bandwidth that Fibre Channel does. By using Fibre Channel, companies have found that they can speed up data access between hosts and storage. In addition, the more efficient usage of IP over Fibre Channel has advantages in network utilization over Gigabit Ethernet, with a shared network making standard TCP/IP networking over SANs an attractive solution.
Increasing Server Cycles A growing problem has emerged with high-speed networks based on IP. Companies have been using clustering techniques (running many, coordinated servers in tandem to distribute processing) to attempt to get past the problems of limited CPU speeds and server scalability. However, clustering techniques rely on
140_SANs_01
8/14/01
1:32 PM
Page 15
Introduction to SANs • Chapter 1
the latency of a network to determine what types of scaling processing are available.With most Ethernet networks comes a negative scaling effect by adding clustered servers—adding more servers allows less and less processing power, due to the increased dedication of CPU cycles required just to coordinate that cluster. The Virtual Interface (VI) Architecture protocol, a standard proposed by Intel, Microsoft, and Compaq for reducing the use of the CPU for network transfers, has emerged as the leading protocol for network communications for clustered environments. By providing a simplified model for direct hardware access to clustered machines,VI eliminates the complex IP stack in favor of a hardware-based, Direct Memory Access (DMA) approach to transferring data across a network. The FC-VI standard maps the VI protocol (also available on Ethernet and proprietary interconnects) to the Fibre Channel protocol, and makes low-latency, direct access available to clustering applications. The primary areas in which the VI protocol is being used today include clustered databases such as Oracle Parallel Server and IBM DB2, both of which natively support the VI protocol over Fibre Channel and other networks. A significant base of researchers and other developers are also using the VI protocol for scientific computing and distributing computational tasks across large networks of machines. SANs are beginning to be used in this area to take advantage of the FC-VI protocol. Businesses are using the VI protocol to free server cycles on their database servers, and recent third-party copy records have been set using VI-capable hardware. An administrator running a clustered database such as the Oracle Parallel Server or IBM DB2 should consider taking advantage of the SAN infrastructure and installing an FC-VI-capable HBA to further accelerate the database cluster.
Ensuring Disaster Tolerance One of the major advantages of SAN technology is its high-performance, longdistance capability. Initially, SAN technology was mostly used to extend to larger distances within a building or campus. However, recently this has been applied to the problem of disaster tolerance: being able to keep an operation up and running even if catastrophe strikes. For example, data center managers are now using Fibre Channel technology bridged through MANs to make their installations more disaster tolerant. A typical example of using MANs for disaster tolerance is brokerage houses located on Wall Street. A common scenario is a large data center that supports
15
140_SANs_01
16
8/14/01
1:32 PM
Page 16
Chapter 1 • Introduction to SANs
customer operations and the trading floor in Manhattan, which needs to have a live backup site to handle the possibility of a power or telecommunications outage, natural disaster, or other major catastrophe. Brokerages are now locating live, connected SANs through Fibre Channel to Dense Wavelength Division Multiplexing (DWDM) and other types of MAN technology directly to SANs in New Jersey.The two data centers continually share data and replicate (mirror) data between the independently running sites, so that if a catastrophe strikes, all information is up to date and still available.This technique is also heavily used in continental Europe, where operations can be spread between countries through these metropolitan connections or dark fiber, an optical communications technology that allows the transport of high bandwidth data.
When to Deploy a SAN Before deciding to deploy a SAN, consider whether implementing a SAN is the right thing for the situation at hand. Frequently in technology, people decide to implement something before they have evaluated whether the technology is actually the best for their needs.The result is often disappointing for the user and for the vendors involved when, after lots of money and time is spent, there was little chance that the solution would have solved the overall problem in the first place. On the other hand, if the problem is first considered and matched to the best technology, the odds for success are much greater.Things to consider when deciding whether a SAN is the right solution: ■
The primary application that needs to be solved
■
Speed, bandwidth, and distance requirements
■
The amount of data sharing or consolidation needed
■
The cost of the SAN infrastructure required, such as switches, cables, and HBAs
Designing Around the Application The most important part of determining whether to deploy a SAN is to focus on the actual business application that will be served with the SAN deployment. Unlike Ethernet networking technology, Fibre Channel SAN technology really should be applied on a network application-by-application basis. Equipment and software deployment is entirely driven by what types of applications need to be served, as opposed to being just an interconnect to plug in all of the desktops in
140_SANs_01
8/14/01
1:32 PM
Page 17
Introduction to SANs • Chapter 1
an organization.Typical applications are storage consolidation,Web hosting farms, database and business-critical transaction servers, and workgroup data sharing.The types of issues to consider are the use of those applications: data sharing, faster backup, or the disaster tolerance aspects of the network. An understanding of the benefits of deploying a SAN is vital to driving the design, software, and hardware used to deploy a network. One thing to consider when starting SAN deployment is to carefully rank and prioritize the goals of the project. Equipment, software, and solutions are all geared toward specific types of applications, and understanding what is needed is very important in determining if a new feature that a vendor is trying to sell is critical to making an application deployment successful.
Assessing Speed, Bandwidth, and Distance Requirements Key factors in deciding to use a SAN are speed and bandwidth requirements. For the most demanding applications, a SAN might be the only option. One Gbit/sec (100 MB/sec) components are widely distributed now; 2 Gbit/sec (200 MB/sec) switches, storage, and HBAs are starting to hit the market; and Fibre Channel standards of up to 10 Gbit/sec are already in development. If unimpeded access to storage is required, Fibre Channel exceeds the speeds available from legacy techniques such as parallel SCSI, and simplified cabling and connections make it far more reliable. Compared with other technologies, such as IPbased file sharing and Network Attached Storage (NAS), the Fibre Channel protocol provides for more usable bandwidth and faster data transfer. Distance is another factor in deciding to use SAN infrastructure. If data needs to be distributed across a building, campus, or city, a storage network is perfect. Long cable lengths, multiple cabling options, and robust components make a SAN a perfect fit for distributing data. For example, many companies have solved the problem of not having enough space for all of the data and hosts they need in a single server room by running Fibre Channel from different parts of the building together. Large storage arrays, which infrequently need access by administrators, can be housed in a “lights out” facility separate from production servers, which often need connection to monitors, keyboards, and administration. Deployment of
17
140_SANs_01
18
8/14/01
1:32 PM
Page 18
Chapter 1 • Introduction to SANs
these solutions is also relatively easy: optical fiber is connected on either end to devices or switch ports, and full bandwidth operation is seamlessly available.
SANs Versus NAS: What’s the Difference? First-time data administrators are often confused as to whether they should be using SANs or NAS. Both techniques are useful, but for different types of applications. NAS uses common client networks such as Ethernet to connect client computers to a host file server. Unlike SANs, the client does not directly communicate with the storage. Instead, the client computer uses a high-level networking file system such as Network File System (NFS), which runs the TCP/IP protocol over Ethernet. Data exchange occurs at the file level, unlike a SAN where data is operated at the block level over Fibre Channel. In general, NAS techniques are best used for client-to-host connections, and SANs are better suited to high-speed file sharing and host-to-storage connections. NAS connections are typically easier to deploy over existing infrastructures, albeit much slower. SANs are typically used where bandwidth and speed are most important, and block-level, direct connections are required. Both techniques can coexist in the same installation. In fact, some NAS systems require a back-end SAN to support their operation.
Data Sharing and Consolidation Needs Determining whether data will be shared on a SAN is important. As one of the major motivations behind moving to SAN technology, it is critical to understand exactly how data will be shared across the SAN. Areas to consider when assessing data sharing and consolidation needs include: ■
Will information be shared at a file level or volume level?
■
Are resources such as storage arrays (static shares) shared, or is file sharing required as part of the workflow (dynamic shares)?
140_SANs_01
8/14/01
1:32 PM
Page 19
Introduction to SANs • Chapter 1
Resource Sharing The simplest form of storage network data sharing and consolidation is simple resource sharing. In this case, a large storage array or storage farm is shared among many machines, but access to each array or disk is statically allocated. Each machine on the network is assigned storage, and does not change that ownership very often, if at all.This is typically used to partition a large storage array among many hosts and can be a very convenient way to manage storage resources.This resource partitioning can be done in a variety of ways, including zoning at the switch level, storage-based Logical Unit Number (LUN) masking, HBA-based LUN masking, or by a virtualization device that presents “virtual LUNs” to the hosts. Resources are generally allocated once and are infrequently changed or modified, and rarely are resources switched between hosts.
Volume-Level Sharing Volume-level sharing is the sharing of resources at the volume level: for example, sharing a LUN between two systems for a clustering application, or moving volumes between machines as part of a digital media workflow.This generally requires the intervention of software that can mount and unmount volumes as part of an operating system, and also might require translators between different operating system formats.
File-Level Sharing File-level sharing is sharing of resources at a file level.This means writing and reading files on a single volume between different machines.This is typically done in SAN configurations with one machine that has write permissions to a volume, and many machines with read permissions to the files on that volume. However, if per-user security (only certain users have access to certain files) is desired, or if many machines need to write to the same volume, it will require more advanced file systems to achieve this functionality. Some techniques (such as global file systems) can be used in this case, but often a better fit is NAS technology and more traditional network file-sharing techniques such as NFS. A SAN is an ideal solution for: ■
Block-level access to shared storage
■
High bandwidth requirements
■
Need for expandability
19
140_SANs_01
20
8/14/01
1:32 PM
Page 20
Chapter 1 • Introduction to SANs ■
Required access to very large centralized storage arrays
■
Need for redundant, highly available access to storage
■
Clustered server configurations
■
Distributed applications
■
Need for disaster tolerance
■
Backing up large amounts of data nightly
■
Running clustered databases (Oracle Parallel Server, IBM DB2)
■
Need for a highly scalable infrastructure
■
Centralized storage management
A SAN is not appropriate for: ■
A small amount of storage with no sharing required
■
File-level, client access to volumes only
■
No storage consolidation required
Steps to a Successful SAN Deployment As with any advanced technology, the most difficult part of working with the technology is the actual deployment of the hardware and software.This section outlines at a high level some recommended steps to take to help ensure success in SAN deployment, and explains the overall process of deploying a SAN. Later chapters discuss the SAN design process in more detail, breaking the process into seven steps: data collection, data analysis, architecture development, testing a prototype, transitioning existing hardware, release to production, and maintenance. The first step to successful SAN deployment is to evaluate the intended goals of the deployment. A firm ranking of the top items to achieve with the deployment is key to evaluating hardware and software options as well as determining topology and design layout.The channel characteristics of storage devices make topology selection and overall architectural design critical in SANs. For example, it is critical for high availability. If this is the primary goal, consider dual-redundant SAN fabrics, fault-tolerant components, and a topology that allows for redundant HBAs and storage ports. On the other hand, if data consolidation and
140_SANs_01
8/14/01
1:32 PM
Page 21
Introduction to SANs • Chapter 1
cost reduction is the goal, the design should be a topology without the redundancy and separate fabrics.The same considerations go into other applications such as backups, databases, or disaster tolerance. Defining the goals and developing a detailed technical plan is an important factor in success. Future chapters cover the data collection and analysis phases in detail. As with any technology solution, a great deal of a SAN deployment should be the investigation of the different software and hardware that will be part of the solution. Because hardware and software are constantly changing, and new innovations and equipment are available every day, speaking with various vendors, attending trade shows, and talking to current users will help in determining the best options for a SAN. Although Fibre Channel equipment, for the most part, is now fully interoperable and will work together, it is still a good idea to get a sense of the different options available. Storage arrays, the foundation of a SAN and the most important part of the solution, should be a primary focus when designing a network. However, because everything in a network has to work together, the infrastructure and HBAs are also a critical part of the formula. The next step is installing a SAN prototype and testing the install, as covered in future chapters. Networking is a complex area, and SANs are no exception. An important deployment risk reduction item is the actual installation of a SAN testbed to prototype the installation. Creating a SAN prototype allows for testing the ultimate installation and working out any issues that might be encountered with both the software and hardware being used. Unless an outside party is doing the installation, consider it a necessity to personally install the setup, to make sure that all of the components work together. It is usually best to set up a lab with all the required power, cabling, and hardware needed to install and test equipment.This includes enough host machines to run the application and hosts to install and test HBAs and software, along with racks and benches for the storage, switches, and other hardware that will be used. In fact, for mission-critical SANs, maintaining a testbed in parallel with the production SAN should be seriously investigated before rolling it out to the actual production network.The testbed allows for pretesting configuration changes and new components, debugging production problems, validating changes, and verifying that new versions of network and system software and firmware behave as expected.
21
140_SANs_01
22
8/14/01
1:32 PM
Page 22
Chapter 1 • Introduction to SANs
SANmark and Other Interoperability Programs The Fibre Channel Industry Association runs a program called SANmark, which certifies equipment against Fibre Channel interoperability suites defined by the industry. Run in conjunction with the University of New Hampshire, the SANmark tests define how different kinds of equipment need to interoperate with other types of Fibre Channel hardware. Several levels of SANmark certification exist, and the standards are constantly evolving to make sure that the latest hardware is tested against existing standards. Because of the importance of interoperability in the Fibre Channel area, many companies publish compatibility matrices that describe which components and versions of software and hardware have been tested against common configurations.
The best way to select hardware and software is to start with the information available from different vendors on suggested configurations and interoperable hardware.Vendor “interoperability labs” and certifications give excellent points on how to pick the right products that will work together. Interoperability labs are now a standard part of almost every vendor’s support structure.These labs, where vendors extensively test and qualify equipment and certify configurations, are set up to make sure that all of the equipment they provide is compatible with other vendors. Extensive testing by vendors ensures that the hardware and software work together as expected. Stress testing, configuration testing, and negative testing under various loads and in different configurations flush out problems before a piece of equipment is shipped. Most vendors make all of this information available to users and are happy to share what configurations they have certified for use with their equipment. Researching the interoperability information from the software and hardware vendors should provide a good idea of what works together.
140_SANs_01
8/14/01
1:32 PM
Page 23
Introduction to SANs • Chapter 1
After investigating the options, installing a SAN testbed, and selecting what seems to be the right software and hardware, it’s testing time. Even with preconfigured and specified configurations, it’s best to set up a real-world configuration to help test the deployment on a small scale.This configuration should include a representative sample of everything that will be deployed on the SAN, with reallife applications and representative data and with sufficient load generated to replicate a limited deployment. No configuration or combination of hardware and software is foolproof, so testing a real-life configuration in a controlled environment before rollout can help to flush out the last issues and major showstoppers that could derail the deployment. Key areas to cover in testing include: ■
Installing all of the major hardware vendors that have been selected
■
Testing for interoperability of components with the versions of software and firmware that will actually be deployed
■
Testing all crucial functionality with the software and applications running
■
Testing for error handling and tolerance
Simple testing includes plugging components in and out of the network, powering down components to see how they recover, and moving cables. More complex testing involves running heavy traffic to components, setting up the application, and running simulated loads.The most important part of this testing is running a simulated or actual load on the application that is being deployed and making sure that even under real-life conditions, everything works as expected, with no problems. The final step in a successful SAN deployment is staging the actual deployment into the enterprise. Staging the deployment helps to minimize risk and maximize the probability of success. Rather than moving to a solution in one fell swoop, it is better to deploy on a limited basis in certain areas and expand that deployment once everything is up and running smoothly. A technique that is frequently used to minimize deployment risk is in-place staging of SAN deployment. In this technique, the equipment and software are set up and tested where the network will permanently be installed. All the testing in the previous step is done where the SAN deployment will eventually be installed, so no equipment is moved, damaged, or inadvertently reconfigured. Instead, when the time comes to
23
140_SANs_01
24
8/14/01
1:32 PM
Page 24
Chapter 1 • Introduction to SANs
deploy, a cable is connected and the SAN configuration is instantly live and available. Using the advantage of long-distance Fibre Channel cabling to enable “live” installation of new storage, large enterprises use this technique whenever they add new SAN hardware.The new storage is tested, run through diagnostics and stress tests, and then added onto the production network with the addition of cabling and modifications to zoning—all without moving the equipment or reconfiguring the setup. Carefully staging the deployment, applying changes on a limited basis, and then rolling it out gradually will minimize any risk and ensure that everything operates smoothly. Future chapters discuss the steps of the SAN design process in more detail, how to analyze options and the underlying hardware and software, how to design a network, and how to best take advantage of the tools available.
140_SANs_01
8/14/01
1:32 PM
Page 25
Introduction to SANs • Chapter 1
Summary The need for more data storage is constantly growing, with the Internet, e-mail, multimedia, and the sheer generation of data demanding more and more storage. With predictions calling for more data to be created, stored, and managed in the next two years than was produced in the history of mankind, it is very important to address those storage needs in a scalable and reliable way. In this chapter, we covered a bit of the history behind the SAN: its roots in parallel SCSI connections, its evolution from just a SCSI replacement to more sophisticated, loop-based storage sharing, to its current incarnation in highly scalable, reliable switched fabric networks. Fibre Channel networks are now being deployed for the most business-critical and important areas in enterprises.Through this evolution, SAN technology now offers a robust platform to establish and support the most important business applications. The benefits of building a SAN include ensuring high-availability access to data, consolidating storage resources and management, reducing backup windows and traffic, freeing host CPU cycles for other important tasks, and ensuring data availability through disaster tolerance techniques. Building a Fibre Channel SAN enables a more reliable, highly scalable, large bandwidth access to data. A SAN is typically used for the most business-critical, technically challenging problems a company faces. Deploying a SAN takes some planning. It is important to consider the application in use, speed and bandwidth requirements, and whether data sharing and consolidation offer any benefit. It is also important to consider the budget for the project.The keys to a successful SAN deployment are evaluating the goals for the technology; fully investigating the software and hardware to purchase; taking the time and resources to install a testbed; working with vendors to select the right combination of software and hardware; and testing the configuration thoroughly. Finally, stage the deployment so that problems can be solved on a limited scale first, before rolling it out on a larger basis. SAN technology has the ability to meet the most demanding business needs and is the only technology currently available that meets the distance, bandwidth, and reliability requirements of critical applications.With the explosive growth of data storage requirements, this technology enables the efficient use and management of data resources. By following proven techniques and carefully planning deployments, Fibre Channel SANs can help solve the most difficult data storage problems.
25
140_SANs_01
26
8/14/01
1:32 PM
Page 26
Chapter 1 • Introduction to SANs
Solutions Fast Track Overview of SANs ; SAN technology evolved from direct-attach interconnects like Small
Computer Systems Interface (SCSI). ; Fibre Channel supports SCSI, Internet Protocol (IP), and the Fibre
Channel Virtual Interface (FC-VI) Protocol. ; The distance between Fibre Channel nodes can be as much as 10 km. ; Fibre Channel supports copper, multimode optical, and single-mode
optical media. ; SAN technology has moved from Fibre Channel Arbitrated Loop to full
Fibre Channel switch fabric.
Taming the Storage Monster ; Data storage needs are increasing rapidly. ; Requirements due to databases, e-mail, multimedia, and the Internet
have dramatically increased the required amount of storage for data. ; Disk farms, storage arrays, and storage consolidation are the keys to
solving the storage problem.
Benefits of Building a SAN ; Fibre Channel is ideal for supporting high-availability configurations and
business-critical back-end operations, due to the ability to set up redundant networks and clusters. ; SAN technology allows for storage consolidation and data pooling for
more efficient use of storage resources. ; Backup windows are shrinking, and backup traffic on the LAN can be
easily reduced by using a SAN to reduce network congestion due to backup. ; Block-level, high-speed access through SCSI-Fibre Channel Protocol
(FCP) can accelerate data access between storage and hosts, and can
140_SANs_01
8/14/01
1:32 PM
Page 27
Introduction to SANs • Chapter 1
free up host resources that would be occupied serving files and data through IP. ; Cluster protocol access through FC-VI frees up CPU cycles in hosts
and enables clustered database operations. ; One of the major advantages of SAN technology is its long-distance
capability for disaster tolerance.
When to Deploy a SAN ; The most important part of determining whether to deploy a SAN is to
focus on the actual business application that will be served with the SAN deployment. ; Speed and bandwidth requirements determine if the technology is right
for the application. Compared with other technologies, such as IP-based file sharing and Network Attached Storage (NAS), the Fibre Channel protocol provides for more usable bandwidth and faster data transfer. ; A SAN is ideal for block-level access to shared storage. ; Fibre Channel works well for centralized access to storage arrays,
redundant connections, clustered configurations, and disaster tolerance.
Steps to a Successful SAN Deployment ; Data collection Evaluate the goals of the deployment to determine
options in achieving high availability, redundancy, fault tolerance, data consolidation, cost reduction, and so forth. ; Data analysis Investigate the hardware and software options that
support those goals. ; Architecture development Design and install a SAN testbed to set up
configuration and components. Select the software and hardware carefully to avoid any interoperability problems. ; Testing the prototype Test the configuration for interoperability,
functionality, error handling, and fault tolerance. ; Transition existing hardware in a controlled release to production
Stage the deployment by rolling out the setup gradually, making changes on a limited basis to minimize risk.
27
140_SANs_01
28
8/14/01
1:32 PM
Page 28
Chapter 1 • Introduction to SANs
Frequently Asked Questions The following Frequently Asked Questions, answered by the authors of this book, are designed to both measure your understanding of the concepts presented in this chapter and to assist you with real-life implementation of these concepts. To have your questions about this chapter answered by the author, browse to www.syngress.com/solutions and click on the “Ask the Author” form.
Q: Is Fibre Channel more expensive to deploy than Gigabit Ethernet? A: Cabling, GBICs, and transceivers are physically identical to Gigabit Ethernet. Costs of Fibre Channel switches and other equipment are very close to those of Gigabit Ethernet components.
Q: Is there any way to preserve the investment in legacy SCSI storage in an enterprise?
A: Yes, through the use of Fibre Channel-to-SCSI bridges. Q: Where can I get expert help in setting up a SAN? A: There are numerous system integrators and resellers who can help. Check with the equipment or software vendors.
Q: Is interoperability a problem with Fibre Channel? A: No, the earlier problems with interoperability in Fibre Channel were mostly due to Fibre Channel Arbitrated Loop (FC-AL) technology. Switched fabric technology eliminates these problems and provides very reliable performance. However, as with any technology, it is still a good idea to check for equipment compatibility with the respective vendors.
140_SANs_02
8/14/01
1:33 PM
Page 29
Chapter 2
Fibre Channel Basics
Solutions in this chapter: ■
The Architecture of SANs
■
Fibre Channel Protocol Basics
■
Classes of Service
■
Storage Network Topologies
■
Fabric Services
; Summary ; Solutions Fast Track ; Frequently Asked Questions
29
140_SANs_02
30
8/14/01
1:33 PM
Page 30
Chapter 2 • Fibre Channel Basics
Introduction Storage Area Network (SAN) infrastructures are built using new technologies that, although related to and derived from other technologies such as SCSI and IP networking, has its own set of terminology and concepts. Like standard computer system networking, Fibre Channel also has its own stack of protocol levels, ranging from the physical connectors and media (FC-0) to upper-level protocols (FC-4). Each of these levels defines a different and separate part of how Fibre Channel equipment communicates. An understanding of these protocol levels, although not required, helps in understanding the equipment and how to debug and monitor the equipment.The different FC-4 protocols (FCP, IP,Virtual Interface [VI], and others) are tied directly to the different kinds of applications (storage, networking, and clustering), and enable Fibre Channel to support a robust set of uses. This chapter introduces some of the basics of Fibre Channel and reviews the underlying architecture of Storage Area Networks (SANs).You will discover the major parts of the Fibre Channel protocol, the primary physical components involved, and how they relate to the software and applications running on a SAN. At the end of this chapter you will be able to determine the kinds of protocols you need to run in your network, and better understand the various SAN topologies and terminology.
The Architecture of SANs SANs provide a topology for connecting a number of hosts to storage devices. An exciting part of Information Technology (IT), SANs allow more users access to more data at faster rates.The concept of a SAN is to provide an infrastructure over which large amounts of data can be transferred robustly between servers and storage devices such as Just a Bunch of Disks (JBODs), tape drives, and Redundant Array of Independent Disks (RAID) systems. SANs also enable the sharing of storage devices such as tape silos and RAID systems. Although there are some efforts in the industry directed to using Gigabit Ethernet and InfiniBand technologies to implement SANs, the primary SAN infrastructure available today is Fibre Channel based. SAN storage is useful for business, because the high level of connectivity allows you to consolidate all your storage needs in a SAN, which is easily expanded, as you require more space. SANs are also accessible to everyone on the network, which makes it easy to share large projects. Another advantage of using a SAN as a means to distribute data across your network is speed.The most common protocol used to implement a SAN is the Fibre Channel protocol,
140_SANs_02
8/14/01
1:33 PM
Page 31
Fibre Channel Basics • Chapter 2
which most commonly operates at 1 Gbit/sec (a data transfer rate of 100 MB/sec). There are also 2 Gbit/sec devices that just came to market and plans for 10 Gigabit devices.These high speeds mean that data is less susceptible to bottlenecks. A Fibre Channel SAN also provides the advantage of increased reliability.The Fibre Channel protocol uses both buffer-to-buffer and end-to-end flow control, and it calculates a Cyclical Redundancy Check (CRC) on every transmitted frame. Reliability can also be increased through redundancy by developing fallback connections over a large geographic area. Fibre Channel is designed to transmit distances that exceed the epicenter of an earthquake.This means that a SAN can stay fully operational if it is designed with redundant links at remote points. Another advantage of a SAN is scalability. SANs provide storage that is not server attached, which improves performance by avoiding bottlenecks on the connection to one machine. SANs also provide affordable scalability, because a storage device can be directly attached to the SAN. Since disks are detached from direct host attachment, multiple devices can allocate the same storage area without performance limitations (Figure 2.1). Server-detached storage can offer a more cost-effective storage solution, since a server is no longer necessary in order to distribute a file system over a multihost network. A SAN implemented using the Fibre Channel protocol incorporates the benefits of a channeled connection and a network. A channel is a high-speed information conduit but, unlike a network, it is hardware-intensive. Channels specialize in streaming data between two devices, such as your computer and a storage subsystem. Some examples of channel protocols are Small Computer System Interface (SCSI) and High-Performance Parallel Interface (HiPPI). A network, on the other hand, specializes in connectivity, allowing flexibility to add and remove nodes from the environment. Examples of network protocols are Token Ring, Ethernet, and Asynchronous Transfer Mode (ATM). Fibre Channel incorporates the flexibility of a network with the high speed and reliability of a channel—essentially allowing you to connect a large number of devices without degrading performance. When we talk about a SAN, we generally think of transporting SCSI data over Fibre Channel. Although this is what is most commonly used in a SAN, Fibre Channel supports many other protocols. Some other protocols that can be transported over Fibre Channel are HiPPI, Internet Protocol (IP), Fiber Distributed Data Interface (FDDI), and ATM, although IP, SCSI, and Virtual Interface (VI) are the predominate protocols transported on Fibre Channel today.
31
32
8/14/01
1:33 PM
Page 32
Chapter 2 • Fibre Channel Basics
Figure 2.1 Storage Server Versus SAN Architecture Storage Server
RAID
Bottle
Client
Neck
Server Client
TAPE
Client
SAN
Server
TAPE
Client
Switch
Ethernet
140_SANs_02
Client RAID Server Client
A SAN is constructed from three primary types of elements: target devices, initiating devices, and interconnecting devices. A target device is usually a storage device on a SAN.There are many different types of storage devices, including tape drives, JBODs, RAIDs, and IP targets. A
140_SANs_02
8/14/01
1:33 PM
Page 33
Fibre Channel Basics • Chapter 2
tape drive is commonly used for backup of other storage devices which might be a database or critical file system. Fibre Channel tape technology is an emerging technology, and testing procedures have just recently been developed.We discuss Fibre Channel components in further detail in Chapter 3, “SAN Components and Equipment.” Fibre Channel storage disks and Fibre Channel-capable tape drives are the most common types of target devices. Fibre Channel disks have a Fibre Channel controller on them. In general, Fibre Channel disks are contained in a JBOD. In a JBOD, each disk is visible to the SAN, each is assigned an address, and each is treated as an autonomous device even though the physical disks are located in the same enclosure. SCSI disks might also be contained in a RAID, in which case the RAID controller will make the array of disks appear to be one disk to the SAN. The RAID is one disk in the sense that it will take a single address on the SAN. Another type of target might be an IP target. Since IP is a protocol commonly used over Fibre Channel, you might see devices communicate by passing IP packets back-and-forth. In this case, there is no distinction between a target device and an initiating device, since both devices can initiate exchanges of Fibre Channel and IP frames. An initiating device is a device that actively seeks out and interacts with target devices on the SAN. Examples are a server or a workstation, and they are often called hosts. A Host Bus Adapter (HBA) is a Peripheral Component Interconnect (PCI) or bus-type adapter that resides in a host machine.That machine can be a server, a workstation, or other device that would request information from a group of disks or storage. It could conceivably be an automated tape backup system.The distinction between a target and an initiator is that an initiator actively searches for a target with which to initiate a transfer, while a target is a passive device.There is often a fine line between the two, since some devices (such as IP devices or bridge devices) might read and write to each other.When a device opens an exchange, it acts as an initiator. From an infrastructure perspective, the most important components in a SAN are the interconnecting devices, namely as switches. Switches create the foundation of a Fibre Channel SAN and provide a high-speed interconnect for routing frames from one device to another. Switches provide fabric services, additional ports for scalability, and the linking capability of the SAN over a wide distance. Although a Fibre Channel SAN can technically exist without any switches using arbitrated loop topology (discussed later in this chapter), a loop-only topology does have its challenges. Arbitrated loop topologies can be subject to performance issues, which can be avoided by connecting the SAN in a switched topology. Switches
33
140_SANs_02
34
8/14/01
1:33 PM
Page 34
Chapter 2 • Fibre Channel Basics
are responsible for correctly routing frames from one node to another over the entire network, with a group of one or more interconnected switches called a fabric (Figure 2.2). Figure 2.2 Interconnected Switches Make Up a Fabric RAID
RAID
RAID
JBOD RAID
FABRIC Switch
Switch
Switch
Switch
Switch
Switch Workstation
Host Host
Host
Host FC/SCSI Bridge
SCSI Tape Library
There are two other common devices encountered in a SAN architecture: hubs and routers. A Fibre Channel hub provides similar function to an Ethernet hub. A hub is a box with a number of ports to which devices can be attached, which simplifies device interconnection.The bandwidth on the hub, which is 1 Gbit/sec, is shared among all the connected devices.There are two types of hubs, managed and unmanaged. An unmanaged hub simply provides a physical wiring
140_SANs_02
8/14/01
1:33 PM
Page 35
Fibre Channel Basics • Chapter 2
between all the connected devices. It does not do any signal processing. A device that is transmitting data, regardless of what the data is, will be connected to the other devices on an unmanaged hub. A managed hub, on the other hand, will wait to connect a device to the other devices on a hub until it sees valid transmission data from the device.The disadvantage of unmanaged hubs is that they increase disruption time during the insertion of a new device, and they will also allow a device that is no longer functioning properly to continue transmitting bad data over the link.This can cause a Fibre Channel loop to remain in an unusable state, and stop traffic between other devices on the hub. For these reasons, most hubs are managed hubs. The terms router and bridge are interchangeable in Fibre Channel terminology. A router generally connects two different protocols, such as Fibre Channel and Ethernet, or Fibre Channel and SCSI. A router is usually a one-to-many connector or a many-to-many connector, whereas a bridge generally connects in a one-to-one manner. The American National Standards Institute (ANSI) began work on Fibre Channel in 1988, and since then the X3T11 Task Group has developed over 20 standards. Fibre Channel’s complexity is not without reward, however: Fibre Channel presently transmits at 1.0625 Gbit/sec over all types of physical media. Recently, many companies have increased that number to speeds of 2.125 Gbit/sec and specifications have recently been published on 10 Gbit/sec Fibre Channel as well. Since Fibre Channel bytes are encoded in 10-bit blocks, this provides a transfer rate of approximately 100 Mb/sec at 1.0625 Gbit/sec.
NOTE Right now there are over 20 Fibre Channel standards projects with many more to come. The main organization involved in the standards process is T11 (www.t11.org). Copies of all the current standards are available at the T11 Web site. The Fibre Channel Industry Association (FCIA, at www.fibrechannel.com) has also started the SANmark program to test the conformance of Fibre Channel devices to those standards, based on a sample set of interoperability tests. Devices can be certified to a number of SANmark Conformance Documents (SCDs). A device’s ability to pass these tests is an indication of its ability to interoperate with other Fibre Channel devices.
35
140_SANs_02
36
8/14/01
1:33 PM
Page 36
Chapter 2 • Fibre Channel Basics
Fibre Channel is most easily understood if it is broken down into layers. There are five Fibre Channel layers, labeled FC-0 to FC-4.The layered breakdown makes Fibre Channel easier to study and understand.We can break it down further by thinking of FC-0 and FC-1 as the physical and signaling layers. FC-2 is a link, or protocol layer.The FC-3 layer specifies common services such as the Name Server, which provide services for all nodes on a Fibre Channel network. The FC-4 layer specifies the mapping of Upper-Level Protocols (ULPs) with the Fibre Channel protocol. The physical media is the FC-0 layer. Although it is called Fibre Channel, it can be carried over either fiber-optic cables or copper twisted-pair type cables. There are two common types of fiber-optic cable: single-mode and multimode. Single-mode cable has the ability to transmit longer distances (100 km) than multimode fiber (500 m). Fibre Channel transmits in 8b/10b-encoded characters, and the signaling interface is the FC-1 layer.This means that for each 10 bits of information transmitted, you actually receive 8 bits of information, which is encoded into a character.The 8b/10b encoding of characters provides a low level of error detection, because if bits are lost or inadvertently changed, invalid characters will be received. Four transmission characters make a transmission word. Certain transmission words are then used as the primitives in the Fibre Channel protocol for signaling purposes. We consider primitives and transmission words to occur at the FC-2 level. Primitives control the flow of frames on a Fibre Channel link. Frames are sets of transmission words that contain routing headers and a payload.The payload is where ULP information is stored, such as SCSI commands or data.The mapping of SCSI commands or data into Fibre Channel frames is an ULP activity that occurs in the FC-4 layer. Devices in a SAN are generally interconnected with a switch. A single switch or a group of all interconnected switches is commonly referred to as a fabric, which provides certain services to the nodes attached to it.The services provided are part of the FC-3 layer and include a Name Server,Time Server, Alias Server, and so on.The Name Server is a distributed database that registers all devices on a fabric and responds to requests for address information. On a fabric, all services are conceptually distributed, meaning that the same server provides service to all nodes independent of direct switch attachment.
140_SANs_02
8/14/01
1:33 PM
Page 37
Fibre Channel Basics • Chapter 2
Fibre Channel Protocol This chapter explains the concepts of the Fibre Channel protocol.The goal is to gain a high-level understanding of the mechanisms of the protocol, such as arbitration, arbitrated loop address selection, and frame generation and transfers. We abstract the Fibre Channel protocol by dividing it into five layers and analyze how each layer interacts with the other layers.We will further analyze the FC-4 layer in detail, because it is the Fibre Channel layer that controls the mapping of ULPs that can be transported over Fibre Channel.We discuss SCSI and IP primarily, but also consider HIPPI, ATM, and IPI-3.
Classes of Service Classes of service are different semantics used to transfer frames using various verification and buffering mechanisms.The classes of service section later in this chapter describes the different types of classes of service and their uses: ■
Class 1 Acknowledged connection-oriented service
■
Class 2 Acknowledged connectionless service
■
Class 3 Unacknowledged connectionless service
■
Class 4 Connection-oriented fractional bandwidth
■
Class F Inter-switch communication format
Storage Network Topologies In the storage network technologies section later in this chapter, we will look at different topologies and discuss how differences in architecture can affect data flow over your SAN.There are three primary topologies, and the goal in this section is to understand how different functions can be achieved in a SAN by using a single topology or a combination of topology models.We look at examples of topologies and define the terminology for referring to nodes in the topology: ■
Point-to-point topology
■
Arbitrated loop topology
■
Switched fabric topology
37
140_SANs_02
38
8/14/01
1:33 PM
Page 38
Chapter 2 • Fibre Channel Basics
Fabric Services Fabric services provide information to nodes in a switched fabric topology. Services can be distributed across all switches, creating the appearance of singleservice type servers.The services provided by the different servers on a fabric make the interconnection of hundreds to thousands of devices seamless.They provide addressing, device-type, and connection-type information to requesting nodes. In this chapter, we discuss a number of different fabric services, including: ■
Login Server
■
Name Server
■
Fabric/Switch Controller
■
Management Server
■
Time Server
Fibre Channel Protocol Basics Fibre Channel was developed to combine the benefits of channel and network technologies. Channels are directly connected devices that do not require large amounts of logic to be incorporated. Channels are hardware-intensive because they are designed for fast transfer of large amounts of data between buffers. Examples of channels are HiPPI and the serial connection made between serial ports on two computers. Networks, on the other hand, are capable of handling very large numbers of nodes. Networks used to be software-intensive because packets needed to be routed to one of many devices on a network. Most of today’s networks use hardware-forwarding. Networks also have to adapt “on the fly” to devices being added and removed. Fibre Channel was developed to incorporate the best features of both. Fibre Channel allows data to be transferred at faster speeds.The base speed of Fibre Channel is 1 Gbit/sec. Many devices, however, are running at double speed right now, and the 10 Gbit/sec specification is presently in draft form. Another advantage of Fibre Channel is that it incorporates the ability to dynamically connect large numbers of nodes over a very wide area. Using single-mode fiber, elaborate SANs can span many kilometers.This adds the benefit of being able to incorporate redundancy for mission-critical applications. Fibre Channel was designed to produce redundant dynamically reconfigurable SANs that would provide storage even in the event of a natural disaster after a large portion of the infrastructure was damaged.
140_SANs_02
8/14/01
1:33 PM
Page 39
Fibre Channel Basics • Chapter 2
Fibre Channel is primarily used to transport the SCSI and IP protocols.The benefits of using Fibre Channel for the mapping of these protocols is increased speed connectivity and longer connection distances.There are three primary topologies for Fibre Channel devices.The first topology is point-to-point.This topology is used between two devices. In point-to-point topology, there is no addressing, since all frames (Fibre Channel packets) are intended for “the other device.” Device connections to switches are sometimes called point-to-point connections. The second topology is arbitrated loop. In this topology, devices are connected together in a loop, with the receive fiber coming from an upstream device and the transmit device going to a downstream device. An 8-bit Arbitrated Loop Physical Address (AL_PA) identifies devices on that loop. For specifics regarding FC-AL, you can consult the state diagrams in FC-AL-2 (www.t11.org).This section provides the basics on how the arbitrated loop protocol works. It is not important that as a system administrator or end user you understand the protocol in detail. However, it is important to understand the concepts, because it will make diagnosing problems in your SAN faster and easier. The third topology is the switched fabric configuration, which enables you to connect a large number of devices. A switched fabric topology is sometimes referred to as a point-to-point topology as well.The switched fabric topology is easily scalable, allowing devices to be added and removed with little disruption to the rest of the attached nodes. A switched topology allows more efficient use of bandwidth by using circuits in the switches to route paths between nodes, as opposed to arbitrated loop where there is one path between a set of nodes on the loop. Information is transferred in frames, which contain a header and a payload.The header contains routing information. It specifies where the frame came from, what kind of frame it is, and where it is going. Frames start with a primitive Start Of Frame (SOF), which indicates the class of service the frame is being transmitted in, specifying the connection type.The class of service, discussed in detail later in the chapter, is a set of universal rules for nodes handling the frame of that type. Classes of service handle tasks such as frame acknowledgment and transfer verification. Information transfer in Fibre Channel is analogous to writing a paper. A total message or idea is broken down into parts. Like a paragraph made up of sentences, it is a collection of related information composed of sequences. A sequence (sentence) is a collection of frames (words) that fit together logically. In Fibre Channel there is punctuation around the “words” as well. Frames start with
39
140_SANs_02
40
8/14/01
1:33 PM
Page 40
Chapter 2 • Fibre Channel Basics
an SOF and end with an End Of Frame (EOF).There are multiple fields in a frame—the first six words, fields zero through five, are mandatory and are called a frame header.The frame header specifies the source of the frame, the destination of the frame, and what type of frame it is.The payload contains the data from the ULP frame.
Fibre Channel Levels When discussing Fibre Channel, it is usually easiest to break down the technology into a number of levels.You can also use the level approach when trying to debug a situation where something is broken.You can first determine the failure level, and work with the components of that level. In this section, we abstract the Fibre Channel protocol by dividing it into five layers and analyzing how each layer interacts with the other layers.There are five Fibre Channel layers, designated FC-0 to FC-4.The layers include all aspects of the technology, from the physical media to the ULPs that are transported on Fibre Channel. Figure 2.3 is a commonly used visual aid to help envision how the different layers interact. Figure 2.3 Fibre Channel Layers FC-4 Fibre Channel Upper Level Protocol (ULP) Mappings FC-3 Fibre Channel Common Services FC-2 Fibre Channel Framing and Flow Control FC-1 Fibre Channel Encode and Decode FC-0 Fibre Channel Physical Media
The FC-0 layer is the lowest-level layer; we have already seen most of the components of this layer by looking at the different media types and connectors involved in creating a Fibre Channel connection.The FC-0 layer specifies how light is transmitted over fiber and how transmitters and receivers work for all media types.This layer deals with the physics of transmitting and receiving a signal at different transfer rates.When there is a problem with a GBIC or fiber line, you know you have an FC-0 level problem. Most of the work being done at the FC-0 level is electrical engineering work in designing transmitter and receiver components.
140_SANs_02
8/14/01
1:33 PM
Page 41
Fibre Channel Basics • Chapter 2
The FC-1 layer is the signal encoding and decoding layer.When you consider the FC-0 and FC-1 layers together, they are generally referred to as the signaling interface.This layer is responsible for taking the serialized signal and encoding it into data characters you can use.The FC-1 layer uses 8b/10b encoding.This means that for each 10 bits sent, you get 8 bits of actual data—the other bits are parity bits.The bits are encoded into two kinds of characters: K characters and D characters. In Fibre Channel, all primitives (LIP, SOF, OPN, CLS, IDLE, and so on) are delimited by a K character, which is often referred to as a special character. Data characters are used to provide all the other 8-bit values. The FC-2 layer is the Fibre Channel protocol level, which is responsible for framing and flow control.The protocol level operates on primitives that are encoded from the FC-1 level as a special character followed by three data characters (D characters). Primitives drive the state machines that control things like arbitration, loop initialization, and data-carrying frames.The FC-2 layer is where the firmware embedded on chips in your Fibre Channel devices is active.The FC-2 layer controls the flow of data by sending the correct primitives to initiate transfers. Although the FC-2 layer sends the frames, the payload of the frames is not part of the FC-2 layer.The FC-2 layer is responsible for correctly filling in the frame headers that are responsible for routing the frames. The FC-3 layer is the Fibre Channel common services layer. An example of common services is the Name Server, which provides to requestors the addresses of other fabric-connected devices. Fabric servers are necessary to provide centralized resources to all attached nodes. Although there might be server agents on each individual switch, all the Name Server agents will share their information through a switch protocol, which makes the Name Server on each switch identical to every other Name Server.This creates the illusion of a single Name Server.This concept is called distributed fabric services, and the same theory is applied to all servers, like the Time Server, which synchronizes with all of the other switches as well.The behavior of all servers that can be implemented in a distributed fabric is specified in FC-GS-3, a Fibre Channel standard that specifies generic service. FC-4 is the Fibre Channel ULP mappings layer.This layer specifies how ULPs like SCSI, IP, HiPPI, IPI-3, and ATM can be carried over a Fibre Channel conduit.The most commonly transported protocol is SCSI. SCSI Fibre Channel Protocol (SCSI-FCP) is the standard that specifies how to encapsulate SCSI frames in the Fibre Channel protocol.The FC-4 layer is responsible for making sure that the ULP data or commands get broken down appropriately and packaged correctly in the Fibre Channel frames.The frame is then passed down to the
41
140_SANs_02
42
8/14/01
1:33 PM
Page 42
Chapter 2 • Fibre Channel Basics
FC-2 layer where the node might query an FC-3 server to obtain the destination address for the frame.The FC-2 layer then adds this information into a header on the frame and sends the frame to the FC-1 decoder, which breaks the frame into bits that can be sent over the physical wire at the FC-0 level.
ULPs The FC-4 layer specifies the mapping of different ULPs to Fibre Channel.To recap, ULPs are the protocols that can be transferred over Fibre Channel. A wide variety of protocols can be transported over Fibre Channel.The advantage for most network protocols of mapping over Fibre Channel is increased speed. For channel protocols, the advantage is added scalability and dynamic reconfiguration. The following are some specifics on protocols and their mappings on Fibre Channel: ■
Small Computer System Interface (SCSI) The most widely used ULP on Fibre Channel networks. SCSI is a parallel interface standard capable of speeds up to 80 MB/sec. SCSI devices can be chained together to create a channel with multiple nodes. SCSI gains speed from Fibre Channel since Fibre Channel operates at a base speed of 100 MB/sec. FCP is the name of the FC-4 protocol for SCSI.
■
Internet Protocol (IP) IP is a standard networking protocol. IP over Fibre Channel is used for different reasons than Gigabit Ethernet, and has many uses, such as offloading backup traffic, in-band access for managing devices, and so on. Fibre Channel allows faster transfer speeds than most Ethernet technology.
■
Virtual Interface (VI) VI is a standard protocol defined for low-level clustering communications over Fibre Channel.This protocol is used by distributed databases, file systems, and other clustering applications to efficiently transfer cluster information over a network between hosts.
■
Intelligent Peripheral Interface (IPI) IPI is an ANSI-defined standardized protocol for controlling peripherals from a host computer.The IPI-3 is the level-three part of IPI that deals with packetized communication between a host and a peripheral device.
■
High-Performance Parallel Interface (HiPPI) HiPPI is a channel used to transfer large amounts of data at 800 Mb/sec or more to supercomputers that have the processing power to use that much data at such
140_SANs_02
8/14/01
1:33 PM
Page 43
Fibre Channel Basics • Chapter 2
a fast rate.The data can be read either between a file system and processor, or between memories on separate systems to create parallel machines. Fibre Channel is a good conduit in the point-to-point topology since it can provide the speed at the FC-0 and FC-1 levels to achieve the goals of HiPPI. ■
Fiber Distributed Data Interface (FDDI) FDDI is one of the first protocols developed for fiber-optic technology. FDDI networks use token passing and support transfer rates of up to 100 Mbit/sec. FDDI networks were typically used as backbones for WANs, but are now being commonly replaced by Fibre Channel and other high-speed Ethernet technologies.
Classes of Service Classes of service specify what mechanisms will be used for the transmission of data. Different classes of service are used for different types of data. Flow control is one mechanism that is specified by class of service. End-to-end flow control is when a receiving port transfers a frame back to the sender to verify receipt of the transmitted frame.When the transmitter receives the acknowledge frame (ACK) back, it is allowed to adjust its credit by one, so it can send another frame. Buffer-to-buffer flow control is used between fabric ports and node ports, or two node ports, to indicate the maximum number of frames the device can receive. A Receiver Ready (R_RDY) primitive signal is sent on the link to indicate that the device can receive a frame. If a certain number of R_RDYs are sent, it indicates that the device has enough buffer space to accept that number of frames. In addition to flow control, classes of service also specify whether the connection is dedicated. In a connectiontype transfer, you cannot send frames that are not addressed to the dedicated receiver. In addition, you cannot send frames in a class other than the connection class.This guarantees that the connection can utilize the full bandwidth.
Class 1 Class 1 service is a dedicated connection class between a transmitter and a receiver. Class 1 connections emulate the features of channel protocols. All packets sent are acknowledged, meaning that an ACK frame is sent back for every frame transmitted.The connection is dedicated, which means that the communicating device uses the full bandwidth of the connection. No other devices can communicate with the connected devices as long as the Class 1 connection is open.
43
140_SANs_02
44
8/14/01
1:33 PM
Page 44
Chapter 2 • Fibre Channel Basics
In a Class 1 connection, since the connection is dedicated, you are assured that frames will arrive in the order in which they were sent. Only end-to-end flow control is used in a Class 1 connection. Class 1 connections are used for timecritical applications and for transferring streaming data such as sound or video. Intermix is an optional service of devices that support Class 1 connections. Intermix allows the transmitter to send Class 2 and Class 3 frames when there are no Class 1 frames to be transmitted, allowing the device to use the bandwidth more efficiently.The Class 2 or Class 3 frames cannot be sent to the same device with which the Class 1 connection is established.
Class 2 Class 2 service provides a connectionless conduit between two ports.This means that the frame is transferred to the switch, and the switch is responsible for attempting delivery at the switch’s earliest convenience. Class 2 allows devices to share all the available bandwidth. Class 2 frames use both buffer-to-buffer and end-to-end flow control, so the transmitter will receive either a positive or negative acknowledgment of receiving a frame.This is an acknowledged class of service.
Class 3 Class 3 service is similar to Class 2, except there is no end–to-end confirmation of the data transfer.This is the preferred class of service for SCSI, and therefore is the class of service most often used in transfers over a SAN. Class 3 uses bufferto-buffer flow control, which is controlled on an FC-2 level using the R_RDY primitive. Class 3 allows devices to share the bandwidth of the SAN. Class 3 allows devices to operate at full speed when there is little traffic, but causes the bandwidth to be shared when there is heavy traffic. It is ideal for distributed storage solutions like SANs.
Class 4 Class 4 service is a less common service but most similar to Class 1. In Class 4 service, the bandwidth is divided into Virtual Circuits (VCs). For this reason, Class 4 is known as a fractional bandwidth class of service.Within a VC, the bandwidth allocated is guaranteed. A node can divide the bandwidth into a number of VCs that share the connection.VCs can be established with a number of other ports. In a Class 4 connection, both buffer-to-buffer and end-to-end flow control are used. Frame ordering within VCs is guaranteed. For Class 4, intermix is required with Class 2 and Class 3 frames.
140_SANs_02
8/14/01
1:33 PM
Page 45
Fibre Channel Basics • Chapter 2
Class F Class F service is used for internal control and coordination of the fabric. Class F frames can be sent only between switches, so all devices are instructed to ignore them. Switches use Class F frames to coordinate services like the Name Server and resolve transmission hierarchy.
Storage Network Topologies As mentioned earlier, there are three primary types of topologies in Fibre Channel: point-to-point, arbitrated loop, and switched fabric. Each topology is useful for different purposes, and the topologies can be combined to form a SAN specific to the solution you need. Point-to-point connections limit you to two devices, so you will generally use a point-to-point connection only when you have two systems that need to talk to only each other at high transfer rates, or if one of the devices is acting as a switch or bridge.The arbitrated loop topology is useful for connecting many devices, but because only one device can arbitrate at a time, it severely limits your bandwidth. In any large SAN, you will need to use a switched topology as the backbone of your connection.There might be one or many switches that form your fabric. However, just because your SAN incorporates a fabric topology does not mean that the other topologies cannot be integrated into the SAN as well. A fabric port can easily be connected as part of an arbitrated loop. Point-to-point connections are established when a single device is plugged into a switch port. Figure 2.4 is a diagram of a fictional SAN that incorporates all the different topology features.
Point-to-Point Topology Point-to-point connections in Fibre Channel are limited to a few specific situations.The primary use of the point-to-point topology is to connect devices directly to switches or other bridge devices. Rarely would a target device and initiating device be connected in a point-to-point topology.This is generally a waste of resources; since Fibre Channel components are faster than all disks manufactured today, it is not likely that the disk they would be attaching could fully utilize the bandwidth provided to it. Furthermore, rarely would a single host system require data streamed at that rate or be able to process it.There are exceptions, particularly if Fibre Channel is being used to build parallel systems, like in the case where HiPPI would be the ULP used for memory sharing.
45
140_SANs_02
46
8/14/01
1:33 PM
Page 46
Chapter 2 • Fibre Channel Basics
Figure 2.4 Fibre Channel Topologies
RAID
RAID
WAN
JBOD
IP / Fibre Channel Bridge
FABRIC Switch
Switch
Switch
Switch
Inter-Switch Link (ISL)
Workstation
FC/SCSI Bridge
Arbitrated Loop
Host SCSI Tape Library
JBOD
Host
In a point-to-point topology there is no addressing, since any data transmitted is intended for the other device. A point-to-point topology can be set up by connecting device A’s transmit fiber into device B’s receive connector, and vice versa (Figure 2.5). Point-to-point connections have a very simple initialization routine,
140_SANs_02
8/14/01
1:33 PM
Page 47
Fibre Channel Basics • Chapter 2
since no address assignment needs to be resolved. Point-to-point connections also have the advantage of allowing the two devices the entire bandwidth of the line at all times. Figure 2.5 Point-to-Point Topology Fibre Channel Device
Transmit (tx)
Receive (rx)
Receive (rx)
Transmit (tx)
Storage Device
Fibre Channel Arbitrated Loop (FC-AL) Topology The arbitrated loop topology is a configuration used to connect up to 127 devices without a switch. Devices in an arbitrated loop are connected in a ring formation.The transmit fiber of the upstream device goes into the receive port of the downstream device.This is repeated around the loop until the first device receives the transmit fiber of the last device (Figure 2.6). Figure 2.6 Arbitrated Loop Topology FC-AL Disk FC-AL Disk
FC-AL Disk
FC-AL Disk
Host FC-AL Disk FC-AL Disk
FC-AL Disk
47
140_SANs_02
48
8/14/01
1:33 PM
Page 48
Chapter 2 • Fibre Channel Basics
The arbitrated loop initialization process discussed earlier is complicated.The complexity is necessary in order to fairly assign all devices in the loop an AL_PA. The initialization state machines are difficult to implement and because of this, interoperability among different vendors’ products is a major issue in arbitrated loop topology. By arbitrating for control of the loop, the arbitrated loop configuration allows all devices connected on the loop to share the bandwidth of a single line. For this reason it is important to put devices on a loop that can afford to suffer the performance degradation that sharing bandwidth entails.You might consider a looped topology if the devices on the loop would rarely be accessed concurrently. It is generally best to place a small group of storage devices on a loop rather than hosts, which constantly require access to the loop. A loop configuration is also good for archiving across many drives that will be accessed rarely, but then need to dump large quantities of data across the network, such as automated backup systems.
Switched Fabric Topology Fabrics allow you to expand your SAN as need dictates, and they allow thousands of devices to be interconnected.The switched fabric topology is easily scalable, allowing devices to be added and removed with little disruption to the rest of the attached nodes.This is a distinct advantage over an arbitrated loop topology, which requires a reinitialization of all nodes every time a node is added. Imagine how unstable a network with hundreds of nodes would be if all devices were reset every time a device was inserted. Switches also allow more efficient use of bandwidth by using circuits in the switches to route paths between nodes. In this way, many transfers can occur at once using full bandwidth.This is also an advantage over the arbitrated loop topology. Switches have two types of ports: F_Ports and FL_Ports. FL_Ports are fabric loop ports—arbitrated loops are attached to these ports. F_Ports are fabric ports to which point-to-point connections are established. Switches also have E_Ports, which are used to connect to other switches. E_Ports communicate in Class F frames to distribute information about the different servers and to set up circuits for the passing of frames to the appropriate nodes over the fabric. A fabric provides a way for devices to communicate with each other over long distances. In order to find a port, the fabric needs an address for identification. Nodes attached to a fabric receive a 24-bit address.The address has the format XXYYZZ and is carried in the Destination ID (D_ID) of frames intended for the device and in the Source ID (S_ID) of frames sent from the device. XX is the domain.This two-digit hexadecimal number refers to the physical
140_SANs_02
8/14/01
1:33 PM
Page 49
Fibre Channel Basics • Chapter 2
switch itself.YY is the area, which corresponds to the port on the switch to which the device is attached. ZZ is the AL_PA, the address assigned to the device during the arbitrated loop initialization process.This ZZ value is set to 0x00 for point-to-point connections between the switch and the edge device. See Figure 2.7 for an illustration of switched fabric addressing. Brocade addressing is discussed further in Chapter 7. Figure 2.7 Switched Fabric Addressing
Storage Array 081600
F_Port
Host 011e00
F_Port
E_Port
FL_Port
Host 0a1602
FC-AL Disk 0514ef
Fabric Services The switches that form a fabric save information about the devices connected directly to them in databases.The switches also provide services for notifying devices of changes on the fabric that affect the way the device functions.
49
140_SANs_02
50
8/14/01
1:33 PM
Page 50
Chapter 2 • Fibre Channel Basics
Switches distribute this information among themselves through Class F service frames. Switches exchange information in their servers so that all individual switch servers contain the same information.This creates a singular fabric entity and makes it appear that there is only one of each type of server. For instance, the Name Server information is shared among all Name Servers on all attached switches.This creates a distributed Name Server that has information about all devices on the fabric. By distributing the servers, the switch structure becomes transparent to attached nodes. There are a number of fabric services defined in FC-GS-3 (Generic Services).The Alias Server manages aliases for multicast groups and hunt groups. The Time Server distributes time information for setting timers and expiration times.The Key Distribution Server provides encryption keys for secure connections between two nodes. In this section, we cover in detail the Login Server, Name Server, Fabric/Switch Controller, Management Server, and Time Server. Like nodes, fabric services have addresses, but the address of a fabric service is a fixed value called a well-known address.Well-known addresses are reserved by the standard.
Login Server The fabric port is at well-known address FFFFFE. It is sometimes called the Login Server because a device is required to send a Fabric Login (FLOGI) frame to this port before it can communicate with the rest of the fabric. A port that needs to connect to the fabric must log in with this server.The node sends a FLOGI frame with the S_ID field filled in only for its AL_PA value.The Login Server then sends a response with the D_ID field filled in with the device’s AL_PA and newly assigned domain and area values (see the Switched Fabric Topology section earlier in this chapter).
Name Server Directory services can be accessed at well-known address FFFFFC.The Name Server is the primary feature of directory services. It is a database used to store information about devices attached to a fabric.The Name Server gets information from a device through the Port Login (PLOGI) frame at initialization and through subsequent registration frames.The Name Server acts like a database— entries can be looked up, added, or deleted. Nodes transmit request frames to the Name Server and receive a response containing the information requested or confirmation of the action requested. One of the most common requests is a Request For Transfer (RFT_ID) frame. An RFT_ID is a request to Register
140_SANs_02
8/14/01
1:33 PM
Page 51
Fibre Channel Basics • Chapter 2
FC-4 types (ULPs). A device does this so that the Name Server has a record of what type of device it is. Often, a host computer will send a request to return the address of all devices that support a certain type of ULP, such as FCP.This way, a host can find all the SCSI disks on the fabric. Another common request is a Get All Next (GA_NXT).This request obtains all information about the next highest node in the Name Server at the specified address.This command is useful for devices that are trying to map the fabric, such as a fabric management utility, or for devices that are trying to find appropriate hosts with which to begin transfers.
Fabric/Switch Controller The Fabric Controller, at well-known address 0xFFFFFD, provides a state change notification service to registered nodes, which notifies any device registered to receive the service when a change in fabric topology occurs. Devices that use this service are generally hosts that want to keep track of a number of storage targets. A device registers for state change notification by transmitting a State Change Registration frame (SCR) to the well-known address.When there is a change in fabric topology, the Switch Controller transmits a Registered State Change Notification (RSCN) frame to the device.The RSCN frame is simply a notification to the device that there has been a change. It is up to the device to query the Name Server to assess the state of the fabric at this time.
Management Server The Management Server provides information about the fabric without stipulation as to zone. A zone is a collection of nodes defined to reside in a zone set. Multiple zones can be defined. Nodes within a zone are aware of other nodes within that zone, but not of nodes outside their zone. For instance, a Name Server query will not return information for nodes outside the requestor’s zone. The Management Server provides a single access point for managing the fabric as well as three services. First is the Fabric Configuration Server, which provides information to management entities trying to discover the fabric topology.The second service is the Unzoned Name Server, which provides access to Name Server information for nodes within all zones.The final service is the Fabric Zone Server, which allows management entities to control zone participation and access present zone information.
51
140_SANs_02
52
8/14/01
1:33 PM
Page 52
Chapter 2 • Fibre Channel Basics
Time Server The Time Server is provided so devices can maintain system time with each other.The Time Server is accessed at well-known address identifier FFFFFB. A client will send a Get_Time frame to the Time Server, which then responds with a Get_Time_Response frame containing the time offset, in seconds.
Other Services Switch manufacturers often provide many other common services, such as the Alias Server, which acts like a Name Server to handle the aliases for multicast groups. A multicast group is a group of nodes that receives data destined for a multicast address.The Alias Server keeps a registry of all nodes belonging to a multicast address, and also handles registration and deregistration of nodes from multicast groups.The Alias Service is not involved in the routing of frames for any group.
140_SANs_02
8/14/01
1:33 PM
Page 53
Fibre Channel Basics • Chapter 2
Summary An exciting part of IT, SANs are allowing more users access to more data at faster rates.The purpose of a SAN is to provide an infrastructure over which large amounts of data can be transferred robustly between servers and storage devices such as JBODs and RAID systems. SANs provide three key advantages: speed, reliability, and scalability. Fibre Channel is the primary SAN technology. Currently, the most popular protocols used over Fibre Channel are FCP and IP. A SAN implemented using the Fibre Channel protocol incorporates the benefits of a channeled connection and a network. A Fibre Channel SAN is constructed from initiating devices, switches, target devices, hubs, repeaters, and bridges. A target device is a storage device on a SAN, and there are many different types of storage devices, including tape drives, JBODs, RAIDs, and IP targets. An initiating device is a device that actively seeks out and interacts with target devices on the SAN. Examples are a server or workstation, and they are often called hosts. Switches create the foundation of your Fibre Channel SAN and provide a high-speed interconnect for routing frames from one device to another. Switches provide the linking capability of a SAN over a wide distance, as well as additional ports for scalability. Fibre Channel is most easily understood if it is broken down into its five layers, which are labeled FC-0 to FC-4.The physical media is the FC-0 layer. Fibre Channel transmits in 8b/10b-encoded characters.The signaling interface is the FC-1 layer.This means that for each 10 bits of information transmitted, 8 bits of information are received, which are encoded into a character. Four transmission characters make a transmission word. Primitives and transmission words are at the FC-2 level. Primitives control the flow of frames on a Fibre Channel link. A fabric provides certain services to the nodes attached to it—the services provided are part of the FC-3 layer, and include a Name Server,Time Server, Login Server, and others. On a fabric, all services are conceptually distributed, meaning that the same server provides service to all nodes, independent of direct switch attachment. SCSI data mapped into Fibre Channel frames is the ULP mapping referred to as the FC-4 layer. There are three topology types for a SAN: point-to-point, arbitrated loop, and switched fabric. Most often, your SAN will contain examples of all three topologies. Switched fabric (also called point-to-point) is used to connect single nodes to a switch F_Port. Arbitrated loop is a topology used to connect a number of
53
140_SANs_02
54
8/14/01
1:33 PM
Page 54
Chapter 2 • Fibre Channel Basics
devices and can be connected to a switch through an FL_Port. Devices on an arbitrated loop share the bandwidth of one line. One or more interconnected switches are called a fabric. Switches distribute data about devices connected to them among the entire fabric to provide distributed services.The switches that form a fabric save information about the devices connected directly to them in databases.They also provide services for notifying devices of changes on the fabric that affect the way the device functions. Classes of service specify what mechanisms will be used for the transmission of data. Different classes of service are used for different types of data. Class 1 service provides a dedicated connection using end-to-end flow control. Class 2 service is connectionless and uses end-to-end flow control. Class 3 is used for SCSI. It uses buffer-to-buffer flow control and is connectionless. Class 4 service provides fractional bandwidth connections.
Solutions Fast Track The Architecture of SANs ; A Fibre Channel SAN provides the advantages of increased speed, relia-
bility, and scalability. ; Fibre Channel presently transmits at 1.0625 Gbit/sec over single- and
multimode optical and copper cabling. ; A SAN implemented using the Fibre Channel protocol incorporates the
benefits of a channeled connection and a network. ; A SAN is constructed from three primary types of elements: initiating
devices, switches, and target devices. ; A target device is a storage device on a SAN. Device enclosures like
tapes, JBODs, or RAIDs are the most common type of target device. ; An initiating device is a device that actively seeks out and interacts with
target devices on the SAN. ; Switches create the foundation of the Fibre Channel SAN. A group of
interconnected switches is called a fabric.
140_SANs_02
8/14/01
1:33 PM
Page 55
Fibre Channel Basics • Chapter 2
Fibre Channel Protocol Basics ; Fibre Channel is primarily used to transport the SCSI and IP protocols. ; Devices are identified by an 8-bit Arbitrated Loop Physical Address
(AL_PA) in an arbitrated loop topology, and a 24-bit address for switched fabric connections. ; Frames start with a primitive Start Of Frame (SOF) and end with an
End Of Frame (EOF) primitive. ; There are five Fibre Channel layers, designated FC-0 through FC-4. ; The FC-0 layer is the physical media layer and includes the media
selection and connectors. ; The FC-1 layer is the signal encoding and decoding layer.The FC-1
layer uses 8b/10b encoding. ; The FC-2 layer is the Fibre Channel protocol layer. ; The FC-3 layer is the Fibre Channel common services layer.The ser-
vices are servers in a Fibre Channel fabric that manage connections between devices connected remotely through the switched fabric. ; The FC-4 layer is the Fibre Channel ULP mappings layer.
Classes of Service ; Classes of service specify what mechanisms are required for transmission
of different types of data. ; Class 1—Acknowledged connection-oriented service. ; Class 2—Acknowledged connectionless service. ; Class 3—Unacknowledged connectionless service. ; Class 4—Fractional bandwidth connection-oriented service. ; Class F—Used for inter-switch communication.
55
140_SANs_02
56
8/14/01
1:33 PM
Page 56
Chapter 2 • Fibre Channel Basics
Storage Network Topologies ; There are three primary types of topologies in Fibre Channel: point-to-
point, arbitrated loop, and switched fabric (also called point-to-point). ; The primary use of the point-to-point topology is to connect devices
directly to switches or other bridge devices. ; The arbitrated loop topology allows up to 127 devices in a ring
formation to share the bandwidth of a single line without a switch. ; Fabrics allow thousands of devices to be interconnected. ; Switches have three types of ports. FL_Ports are fabric loop ports that
attach arbitrated loops to the fabric. F_Ports are fabric ports that connect single devices to the fabric in a point-to-point topology. E_Ports connect a switch to another switch. ; Fabric-attached devices have a three-part address.The first segment indi-
cates the physical switch, the second part indicates the physical port, and the last part is the arbitrated loop address in a loop device or 0x00 for a fabric device.
Fabric Services ; Switches exchange information in their servers so that all individual
switch servers contain the same information.This creates distributed servers. ; The fabric port is used to log a device into the fabric.The response
frame from login assigns the device its 24-bit address. ; The Name Server is used as a database to register and store information
about all devices on the fabric. ; The Fabric Controller at well-known address 0xFFFFFD provides state
change notification service to registered nodes. State change notification is a service that notifies devices when a change in fabric topology occurs. ; The Management Server provides information about the fabric without
stipulation as to zone.
140_SANs_02
8/14/01
1:33 PM
Page 57
Fibre Channel Basics • Chapter 2
Frequently Asked Questions The following Frequently Asked Questions, answered by the authors of this book, are designed to both measure your understanding of the concepts presented in this chapter and to assist you with real-life implementation of these concepts. To have your questions about this chapter answered by the author, browse to www.syngress.com/solutions and click on the “Ask the Author” form.
Q: When should I use a hub rather than a switch? A: Hubs can be used on small SANs to interconnect devices in an arbitrated loop topology or connect two devices directly. Hubs should not be used when there is more than one active host, since there will be more competition for the limited bandwidth of the loop.The more hosts that are added to the loop, the less efficient it is, because a larger percentage of time is spent arbitrating for control of the loop than actually transmitting data. Remember that a switch creates circuits to maximize bandwidth, while all devices plugged into a hub share the bandwidth of one line.
Q: How does Fibre Channel compare to SCSI in terms of performance? A: SCSI LVD (wide) has a maximum transfer rate of 80 MB/sec, as opposed to Fibre Channel’s 100 MB/sec.The advantage of using Fibre Channel over SCSI is not entirely speed, however. Fibre Channel allows you the unique opportunity to create a switched network with Fibre Channel devices. Not only can you attach more devices together, but the performance is actually increased as well. Fibre Channel also allows you to use fiber-optic cable as a media type, which extends the area you can connect the devices by 10 km per cable length.
Q: What are some other SAN technologies? A: Right now, Fibre Channel is by far the most common protocol in the SAN marketplace. No other technology has the ability to incorporate the aspects of networks and channels in the way Fibre Channel does. In the past, FDDI was a popular technology that used a loop configuration similar to Fibre Channel’s arbitrated loop. Some emerging technologies in the SAN industry are InfiniBand and Gigabit Ethernet.
57
140_SANs_02
58
8/14/01
1:33 PM
Page 58
Chapter 2 • Fibre Channel Basics
Q: How do I know if my new device will interoperate properly with my existing SAN?
A: It is always difficult to know. However, devices are getting dramatically better at working across multivendor switched fabrics.The Fibre Channel Industry Association (FCIA) has also started an initiative to document common interoperability problems and develop testing specification documents to determine whether a device contains interoperability bugs.The SANmark Program has been active for a little over a year, and devices can now be certified as SANmark-compliant. Devices that pass these tests are probably the most interoperable devices.
140_SANs_03
8/14/01
1:35 PM
Page 59
Chapter 3
SAN Components and Equipment
Solutions in this chapter: ■
Overview of Fibre Channel Equipment
■
Cabling and GBICs
■
Using Hubs
■
Using Switches and Fibre Channel Fabrics
■
Connecting Legacy Devices Into Your SAN
■
Bridging and Routing to IP Networks and Beyond
■
Fibre Channel Storage
; Summary ; Solutions Fast Track ; Frequently Asked Questions 59
140_SANs_03
60
8/14/01
1:35 PM
Page 60
Chapter 3 • SAN Components and Equipment
Introduction Whether you are building a small or large network, one aspect of a robust Fibre Channel deployment is the SAN components used to build your solution. By understanding the different components, features of those components, and how they are best used, you can plan and deploy a reliable, scalable network. Upfront qualification, testing, and selection of equipment are important pieces of making your SAN deployment work. Understanding which features you will be using in your equipment will help guide your testing process.This chapter discusses the different components of a SAN and their major features, and guides you in selecting the right equipment for the job. Fibre Channel has its own various types of connectors and media, including both optical and copper interfaces, and varied ways to connect between the different kinds of media. Fibre Channel uses fiber-optic and high-speed copper media to bring together the speed and reliability of a channeled technology, with the scalability of networking technologies.This is the perfect medium for transporting large amounts of data quickly to many different nodes across a network. The standardized use of Gigabit Interface Connectors (GBICs) has made switching between media types simple and easy, and mixed-media networks are standard.This chapter explains the different connector and cabling options, and how to select the right one for your application. It also covers the kinds of network topologies you can implement and why. Hubs, switches, Host Bus Adapters (HBAs), and storage make up the components of a SAN. Hubs serve as the center of simple Fibre Channel-Arbitrated Loop (FC-AL) configurations, and range from simple unmanaged hubs to more intelligent managed hubs capable of switching frames between ports but not acting as switched fabrics. For more reliable, manageable, and scalable networks, Fibre Channel switches are used instead of hubs. Switches scale between as few as eight ports to 64 or more ports, and form the core of a switched fabric. HBAs serve as the entry point into the SAN from your servers and hosts, providing translation of Small Computer Systems Interface (SCSI) information from the operating system to Fibre Channel addresses on the network. High-capacity storage systems can contain petabytes of data and form the core of the data storage infrastructure of your storage network. Finally, routers and bridges enable you to move data between legacy SCSI components and Fibre Channel, as well as to networks based on Gigabit Ethernet, Asynchronous Transfer Mode (ATM), and Dense Wave Division Multiplexing (DWDM). This chapter reviews the hardware components that make up a SAN, explains the major features and functionality of each, and describes the tools and techniques
140_SANs_03
8/14/01
1:35 PM
Page 61
SAN Components and Equipment • Chapter 3
available to manage each piece.We will guide you through the features to look for on each component and how to use these features.
Overview of Fibre Channel Equipment Fibre Channel shares much of the same terminology as Ethernet networking with hubs, switches, network interface cards, and routers all representing a part of the network infrastructure. However, although the names are often the same, the way they work is quite different. In the development of the Fibre Channel industry, many of the terms were borrowed from more familiar networking terminology, even though in actual practice the functionality has changed. For example, in Ethernet a hub exists mostly as an unmanaged electrical device that allows multiple Ethernet connections to connect to a single point, with all connections seeing the same network traffic. In Fibre Channel, a hub connects each port to the one next to it in a circuit. It is important not to confuse the Ethernet use of the terminology with the Fibre Channel terminology and usage. Figure 3.1 identifies the components of a SAN.
Cabling and Media Many characteristics of your SAN are determined by the physical wiring plan of your network.The type of media you select impacts the scalability and functionality of your SAN.This chapter discusses the options for choosing a physical media, including the advantages and disadvantages of different types of fiber-optic cabling and the choice between fiber and copper cables. Types of media we discuss in this chapter include: ■
Copper (Shielded Twisted Pair [STP])
■
Multimode optical
■
Single-mode optical
GBICs and Connectors The cabling and GBICs section in this chapter is dedicated to familiarizing you with the different types of physical connectors used to connect devices and terminate cable.Your selection of a fixed connector versus a GBIC affects the capability of your SAN to adapt to new devices and support legacy connector types. The choice of connector type can also affect your ability to add to your SAN in the future. Different types of connectors have different considerations and might
61
140_SANs_03
62
8/14/01
1:35 PM
Page 62
Chapter 3 • SAN Components and Equipment
require special handling to use correctly.Types of connectors we discuss in this section include: ■
DB9 (copper)
■
HSSDC (copper)
■
SC (optical)
■
High-density optical connectors (Small Form Factor Pluggable [SFP]): —MT-RJ —LC
Figure 3.1 Different SAN Components in a Network JBOD
RAID
RAID
Remote SAN
Hub DWDM
Remote
Host Switch
Switch
Switch
Switch
SAN
FC-to-ATM Bridge
Host
FC/SCSI Bridge
Host Host
HBA
HBA SCSI Tape Library
140_SANs_03
8/14/01
1:35 PM
Page 63
SAN Components and Equipment • Chapter 3
Hubs In Fibre Channel, hubs serve at a very basic level as electrical connections between the different ports and are used only in FC-AL configurations. Hubs originally started out as simple electrical devices, which, if a cable was attached to a port, completed the electrical loop between ports. If a signal is detected, a hub will complete the circuit and pass traffic through the attached wire. A simple resilientloop circuit is used to make sure that connections are maintained through unused ports. A hub in this sense is a simplification of cabling that reduces the need for separate transmit and receive wires to all devices in a system (one of the original ways to connect Fibre Channel equipment). Intelligent managed hubs provide basic functionality but add more sophisticated error and fault detection, switched frames, and additional features for managing loop environments. Hubs support the use of Fibre Channel to connect up to 127 devices in a loop. Due to the complexity of looped environments and available bandwidth, the number of devices is generally significantly less than the maximum of 127. Also, while loops of 127 devices are theoretically possible, but impractical, networks larger than 127 devices could not be built due to address space limitations.
Switches Fibre Channel switches, unlike hubs, are primarily used in Fibre Channel switched fabric installations. Instead of a loop, where traffic is passed between all nodes (a shared bandwidth and error segment architecture), the Fibre Channel fabric instead routes frames directly from initiators and targets across a full-bandwidth fabric.This means that each connection across a fabric can exist independently of every other connection. Switches, which can range from as few as eight ports to 64 ports or more, contain sophisticated switching hardware used to route frames from any port to any other port. In addition, switches can also be cascaded through Inter-Switch Links (E_Ports), which allow fabrics to extend to thousands of nodes and up to a current limit of 239 switches. Switches add the intelligence of fabric services such as name services and management services, and provide a more robust protocol set for connecting devices. Switches are used in almost all environments to provide a reliable mechanism for connecting hosts to storage and are a necessity for environments with multiple initiators or more than just a few devices. Fibre Channel switches are the foundation upon which the rest of the SAN infrastructure is built.
63
140_SANs_03
64
8/14/01
1:35 PM
Page 64
Chapter 3 • SAN Components and Equipment
Storage Fibre Channel storage is a key component of a Fibre Channel network, providing the shared storage resources that can be accessed through your SAN. Storage is usually the area of focus for your SAN, except in the case where the network is being used exclusively for IP or Virtual Interface (VI) traffic. Fibre Channel storage ranges from single disk drives that support dual-port Fibre Channel connections, to banks of disk drives called a Just A Bunch Of Disks (JBOD) wired together into a cabinet, to more sophisticated Redundant Array of Independent Disks (RAID) storage devices with hundreds of gigabytes of capacity, and finally to enterprise-level storage subsystems that contain a terabyte or more of data.
Host Bus Adapters HBAs are used to connect servers and hosts to the Fibre Channel network. Analogous to Network Interface Cards (NICs), the term host bus adapter comes from their use of connecting servers to the SCSI bus. HBAs consist of hardware and drivers, which interface with operating systems to represent Fibre Channel storage as devices in the operating system. HBAs are the gateway to accessing your SAN. HBAs typically plug into a host’s bus (for example, PCI or Sbus), although some HBAs might be embedded on the motherboard and translate signals on the local computer to frames on a Fibre Channel network. A key part of this process is the driver, which controls your Fibre Channel HBA and determines how the device behaves with the operating system as well as the general Fibre Channel network. Unlike typical NICs, Fibre Channel HBAs tend to be far more intelligent than the standard network card, providing for negotiation with switches and tracking devices that are attached to the network. Robust software and hardware functionality enable these components to offload I/O processing from the host, monitor network configurations, and support load balancing and failover capabilities.
Routers and Bridges Routers in the Fibre Channel sense do not serve the same purpose as routers in the networking world. Instead, Fibre Channel routers act as bridges, translating Fibre Channel frames to other types of transports.The most common routers translate between legacy SCSI connections, representing a SCSI bus as a number
140_SANs_03
8/14/01
1:35 PM
Page 65
SAN Components and Equipment • Chapter 3
of individual Logical Unit Numbers (LUNs) behind a Fibre Channel port. SCSIto-Fibre Channel routers are frequently used to connect SCSI tape devices to Fibre Channel. It is possible that true Fibre Channel routers will emerge in the future, which might cause some confusion. Other types of bridges include Fibre Channel-to-Gigabit Ethernet bridges, which typically bridge IP frames from Gigabit Ethernet to Fibre Channel, and Fibre Channel to DWDM or ATM bridges, which transport full Fibre Channel frames across DWDM or ATM technology for extended fabrics that span from several kilometers to several hundred kilometers.
Cabling and GBICs The most basic layer of your SAN is the physical layer, which includes your media and connector choices.These choices depend on the primary purpose of your SAN. Like most technologies, improvements are happening every day, so we will highlight the connectors and components that are most popular today. You must consider a number of factors as you put together the physical layer of your SAN. One factor is distance, or how far you must connect two points of interest. Next, you need to consider your existing architecture: if there is already fiber or copper that can be used, determine if it is compatible with the components you would like to add.You also need to consider scalability, so your SAN will be easily upgradeable to allow devices to be added and removed with a minimal amount of added materials.The final consideration is cost.What components are the least expensive, and what are their advantages and disadvantages? This section provides information on specific media and connectors, so that you can assemble a cost-effective SAN that is efficient, reliable, and scalable.
Copper Versus Optical: Selecting Your Media In selecting the type of media to use for your SAN, you have two primary choices: copper and optical.The distinct advantage of copper is that it is inexpensive compared to all types of optical.The advantage of optical fiber is that it provides a reliable signal over a longer distance than copper.The choice between the two types of optical fiber (multimode and single mode) is also one of distance and cost.There are no speed differences between any of the media types.
Copper Cabling Copper has the advantage of being the least expensive media by which to connect components of your SAN. Copper is generally 150-ohm shielded twisted pair,
65
140_SANs_03
66
8/14/01
1:35 PM
Page 66
Chapter 3 • SAN Components and Equipment
although 75-ohm video cable and mini-coaxial cable are also used. Copper cabling, when used at a rate of 100 MB/sec, has an effective range of 0 to 25 meters without sacrificing quality.Transmitting at half speed and quarter speed increases the effective distance of transmission. However, few companies manufacture half- or quarter-speed products. Copper is usually terminated with either an HSSDC or DB-9 male connector (on the DB-9 connector, the end with the pins is male). Although at this time DB-9 is a more common connector, HSSDC is quickly becoming more popular. See Table 3.1 for a comparison of media specification. Table 3.1 Copper Media Type Comparison Chart Media Type Shielded Twisted Shielded Twisted Video Cable Shielded Twisted (Active) Video Cable Shielded Twisted (Active) Video Cable
Speed (MB/sec) Optimal Distance (Meters) Pair (Active) Pair (Passive)
100 100 100
0–30 0–15 0–25
200 200
0–10 0–10
50 50
0–40 0–40
Pair
Pair
Values in table are estimated lengths based on optimal signal strength.
Copper is highly durable and easy to store, which makes it useful for a lab or area where devices are commonly plugged and unplugged, or when you are constantly connecting and disconnecting a device over a short distance to a number of different hosts. An advantage to using the DB-9 and HSSDC copper connectors is that there is only one way they fit into the complementary connector, which means it is impossible to cross a transmit and receive line, a common mistake for even experienced individuals dealing with fiber optics. Optical cabling is harder to terminate and can be susceptible to scratches. In addition, copper is a better choice in the cabinet short-length connection. For lengths longer than five meters, single or multimode optical fiber might be a better choice.
Multimode Optical Cabling Multimode optical cable is available in 50 micrometers (µm) and 62.5 µm sizes. These measurements correspond to the diameter of the fiber—there is no speed difference between the two that affects Fibre Channel. Multimode optical cable is
140_SANs_03
8/14/01
1:35 PM
Page 67
SAN Components and Equipment • Chapter 3
available in 850 nanometer (nm) and 1300 nm wavelengths.The 850 nm wavelength is within the visible spectrum and is not harmful to your eyes.This is not true of 1300 nm wavelength lasers, which are not visible but could severely damage your retinas. Multimode optical fiber is terminated using a variety of optical connectors, including SC, LC, and MT-RJ (we discuss connector types later in the chapter). A 50 µm multimode fiber has an effective range between 0 and 500 meters at a 1 Gbit/sec rate (see Table 3.2 for specifications on other multimode fibers).The 62.5 µm fiber has about half the range of 50 µm fiber. Multimode fiber is the more common media type and is inexpensive compared to single-mode fiber, although the two are coming closer together in price as the demand for single mode rises. Multimode transmitting and receiving components are also much less expensive, because multimode generally uses a concentrated LED rather than an actual laser.This is because multimode fiber is much wider in diameter than single-mode fiber. Many multimode fibers have a feature called Open Fiber Control (OFC), which is a feature of the transmitter and receiver pairs. In OFC, the transmitter periodically transmits short bursts of light.When the receiver detects this light, it begins to transmit regularly and causes the other transmitter to go out of OFC mode.The OFC mechanism was designed to avoid the potential hazards of having unconnected lasers transmitting in a work environment. OFC is becoming a less common feature, since most multimode transmitters use LEDs rather than lasers and there is no associated safety risk. Table 3.2 Multimode Optical Media Comparison Media Type
Laser/ LED Type (nm)
Speed (MB/sec)
Optimal Distance (Meters)
50 µm multimode 62.5 µm multimode 50 µm multimode 62.5 µm multimode 50 µm multimode 62.5 µm multimode
850 850 850 850 850 850
100 100 200 200 50 50
2–500 2–300 2–300 2–90 2–1000 2–400
Values in table are estimated lengths based on optimal signal strength.
67
140_SANs_03
68
8/14/01
1:35 PM
Page 68
Chapter 3 • SAN Components and Equipment
Single-Mode Optical Cabling Single-mode optical fiber (Figure 3.2) is the most expensive media type, but preferable for long distances. It most often comes in 1300 nm wavelength, which is not visible and can be harmful to your eyes. Figure 3.2 Single-Mode Fiber with SC Terminators
Single-mode optical fiber is approximately 9 µm in diameter.The small diameter makes light waves less likely to be altered over long distances, so for longdistance SANs, single-mode fiber is the best solution. Because of its small diameter, it also theoretically has the highest transmission speed potential (the theoretical limit is around 25 Tbit/sec, as opposed to multimode, which is around 10 Gbit/sec). Single mode is the ideal media to use for long interconnects. Single-mode fiber itself is not significantly more expensive than multimode fiber or even copper—the added price is in the transmitting components, which use lasers rather than LEDs. Since the fiber has such a small diameter, it takes added precision to align the laser in the transmitter with the fiber. See Table 3.3 for specifications on single-mode fibers.
140_SANs_03
8/14/01
1:35 PM
Page 69
SAN Components and Equipment • Chapter 3
Table 3.3 Single-Mode Optical Media Comparison Media Type
Laser/LED Type (nm)
Speed (Mb/sec)
Optimal Distance (Meters)
9 µm single mode 9 µm single mode 9 µm single mode
1300 1300 1300
100 50 200
2–10,000 2–10,000 2–2,000
Values in table are estimated lengths based on optimal signal strength.
WARNING Any single-mode or multimode laser can damage your eyes if it is transmitted at 1300 nm. The 1300 nm wavelength is not in the visible spectrum, so you will not see a laser being transmitted like in 850 nm fiber. A 1300 nm laser is dangerous, because it can cause severe retina damage.
Connecting with Connectors There are many different types of connectors, and no particular connector makes a difference in performance as long as the connection is clean. Some connectors are bonded, which means that the transmit and receive fibers are physically mounted in the same piece of plastic.This is usually acceptable, but for some less orthodox wiring systems it might be preferable to select connectors that have loose transmit and receive fibers.This section covers the most well-known types of connectors. You should try to minimize the total number of connections and patches when building your SAN. As discussed earlier, light is reflected by poor connections and patches in the path between devices, so minimizing the number of patches between devices makes your SAN less susceptible to loss-of-signal errors.
The DB-9 Copper Connector DB-9 is the standard copper connector, although more organizations are switching to HSSDC because of its improved reliability and smaller size. DB-9 has the same appearance as DB-9 serial cabling, so it is important to understand that they are not the same (Figure 3.3). DB-9 connectors have a metal D-shaped connector rim
69
140_SANs_03
70
8/14/01
1:35 PM
Page 70
Chapter 3 • SAN Components and Equipment
with 9-pin sockets on the female end and either four or eight pins on the male end, rather than all nine used in serial cabling. Currently, DB-9 terminated cable is less expensive than HSSDC terminated cable. Figure 3.3 DB-9 Copper Connector
Copper cabling is available in two types: passive copper and active copper. Passive DB-9 has four pins (two for transmitting and two for receiving) and, like HSSDC, is used to terminate shielded twisted pair. In active copper cabling, four pins of a DB-9 connector are used to transmit power in addition to the two pairs that are used for transmit and receive. Active copper lines get twice the distance of passive copper lines. Both active and passive type DB-9 connectors are equally priced. Again, it is important when purchasing DB-9 cables not to confuse the connector with DB-9 serial cabling.The resistance between the two is not the same and can severely damage your equipment.
The HSSDC Copper Connector The HSSDC, shown in Figure 3.4, is starting to replace DB-9 connectors on some HBAs.The most probable reason is that they are smaller than the DB-9 connectors, so more can fit on a single interface card.The HSSDC connector uses a single plastic squeeze lock, so it is easy to insert and remove.The HSSDC connector was specifically designed as a Gigabit copper connector, by improving density and performance over the DB-9 style connector.
140_SANs_03
8/14/01
1:35 PM
Page 71
SAN Components and Equipment • Chapter 3
Figure 3.4 HSSDC Copper Connector
Looking Forward The InfiniBand architecture, a widely supported and fast-moving effort, uses a type of copper connector called HSSDC-2, as well as the same types of cabling media as Fibre Channel. The InfiniBand protocol is designed as a replacement for the PCI bus on server systems that need greater width for I/O. The goal is to avoid the impact on the bus of a server by directing data to the appropriate channel in the server. InfiniBand will be able to encapsulate many protocols, including FCP, in order to transfer Fibre Channel or other ULPs to the appropriate adapters; however, this should be transparent to the SAN architecture. With devices already coming to market, the ability to include InfiniBand servers on your SAN might be a consideration for expansion.
The SC Optical Connector The SC connector, shown in Figure 3.5, is probably the most widely used optical connector.The SC connector has been commonly used to replace the ST connector, which at one time was widely used with legacy fiber technologies.The SC connector is a square plastic block containing a glass housing for the fiber. The plastic fits snugly in the connector slot on the board or GBIC. SC connectors are used to terminate single or multimode fiber. SC connectors can be either bonded or unbonded, and come in either single or duplex quantities. A single-quantity SC patch cord is a piece of single or multimode fiber terminated at both ends with one SC connector. A duplex quantity is a pair of fibers, one for transmitting and one for receiving.The plastic insulation on the fiber is molded together to provide a transmit/receive pair. There are a total of four SC connectors, two at each end of the fiber.The SC
71
140_SANs_03
72
8/14/01
1:35 PM
Page 72
Chapter 3 • SAN Components and Equipment
connectors on duplex fiber might be bonded together, meaning that the SC connector pairs are made of one piece of (or bonded) plastic. Bonded connectors have the advantage of reducing your ability to plug the transmit fiber into the transmit slot on the GBIC or port. Figure 3.5 Unbonded and Bonded SC Optical Connectors
High-Density Fiber-Optic Connectors High-density connectors represent the next generation of fiber-optic connectors. High-density connectors are designed to be small to allow more connections in a small space, which might be the back of a PCI adapter card or the faceplate of a hub or switch. As the Fibre Channel protocol develops, you can connect more nodes reliably, and you will need to have space to make those physical connections.The most popular types of high-density connectors are the LC and MT-RJ connectors. Neither type uses any new optical technology. In fact, the connectors use the same multimode and single-mode fiber that SC connectors use.The difference is entirely in the piece of plastic in which the fibers are housed. HSSDC copper connectors, discussed earlier in this section, are a high-density-type connector.The HSSDC connectors are designed to accomodate more copper ports on HBAs and switch and hub faceplates. LC connectors are bonded pairs of miniature connectors.The design of the plastic pieces looks like a small, elongated version of the SC connector. However, the LC pair is comparable to the width of a single SC connector.The MT-RJ uses a single terminator for pairs of fibers.The plastic design is similar to a miniaturized HSSDC without the copper contacts on top of the housing. Instead, the
140_SANs_03
8/14/01
1:35 PM
Page 73
SAN Components and Equipment • Chapter 3
fiber transmits and receives out of two pinholes in the end of the plastic housing. MT-RJ connectors are also about the width of a single SC connector.
Comparing GBICs to Fixed Media GBICs are removable transceivers used in all types of Fibre Channel devices, including switches, hubs, and HBAs. They are used widely in Fibre Channel and other network technologies. GBICs offer the option of interfacing with almost all types of connectors. A GBIC fits into a GBIC port on a device. A large percentage of Fibre Channel devices have a GBIC slot rather than a fixed media type port. GBICs convert the electrical signal generated from the device into the appropriate signal for transmission, depending on the type of connection the GBIC was designed to make. GBICs can convert the device’s electrical signal to a signal that is appropriate for single-mode fiber, copper (either HSSDC or DB-9), and multimode fiber. GBICs have a variety of connector types (Figure 3.6) and can be used to make your SAN connection types homogeneous.
Figure 3.6 SC, HSSDC, and DB-9 GBICs
Using a GBIC Many devices have GBIC slots.Vendors provide them to make devices easier to connect to a variety of media.The GBIC should slide easily into the GBIC slot.
73
140_SANs_03
74
8/14/01
1:35 PM
Page 74
Chapter 3 • SAN Components and Equipment
GBIC slots generally have a trap door, which flips up on insertion of the GBIC. The GBIC should be inserted with the single socket pointed in and the connection end facing out. (If you meet with any resistance, do not force the GBIC, since it is most likely upside down.) GBICS come in a variety of types, including multimode, single mode, HSSDC, and DB-9. It is important to remember to connect the right type of media. For instance, multimode fiber will not work in a single-mode GBIC.You will most likely need to use GBICs in a switch or hub.
Pros and Cons of Using GBICs The advantage to using GBICs is the versatility they give devices to work within a topology.With a range of GBICs, a device can be attached to any SAN. Using GBICs, however, breaks a cardinal rule of networking, which is to minimize the number of connections. Although GBICs, when working properly, do not degrade signal, including a GBIC in the connection introduces another element that can malfuntion. Although the newer GBICs are highly dependable, they tend to be used over and over in different slots. Since connectors go through so many insertions and removals, they tend to break more quickly than a fixed connection could.When GBICs are used frequently over a long period of time, they become less dependable. GBICs are also expensive—second only to fiber in being the most often replaced piece of a SAN.
GBIC Ports on Equipment Equipment might or might not have a GBIC port.Without a GBIC port, you are limited to the type of connection on board, whereas if you have a GBIC port, you have additional options. Switches and hubs almost always have GBIC ports. HBAs and storage devices often have fixed media. GBIC ports are becoming more common on devices now.This allows you to choose your media type based on the location of your device and other device specifications.With fixed media, however, if the vendor decided that a particular type of port is appropriate, it might limit your options as far as the distance and speed at which the device can be connected. Another drawback to a fixed port is that a failure in that port requires the unit to be replaced.
Serialized Versus Nonserialized Serial ID GBICs provide serial number, model numbers, and diagnostic data through embedded Electrically Erasable Programmable Read-Only Memory
140_SANs_03
8/14/01
1:35 PM
Page 75
SAN Components and Equipment • Chapter 3
(EEPROM).This allows for better asset tracking and diagnosis of GBIC problems.This feature is used by SAN management tools and is the only way to see if you have a mixture of GBICs in your fabric. Serial ID is a very important feature if you want to use Brocade Fabric Watch to monitor GBICs. See Chapter 4, “Overview of Brocade SilkWorm Switches and Features,” for further discussion about Fabric Watch.
Common Problems with GBICs GBICs are highly reliable devices but, as mentioned earlier, their pluggable functionality causes them to break more frequently than other components. If you are careful and never force a GBIC, it should last a long time.When trying to diagnose a connection problem with a device, start by making sure your transmit and receive wires are not crossed before deciding to replace the GBIC. Another common problem with GBICs is finding appropriate GBICs for the media type you are using. Use only single-mode GBICs with single-mode devices, and only multimode GBICs with multimode devices. It is a common mistake to plug a fiber into an already-inserted GBIC and assume it is the correct mode for the type of fiber you are connecting. Also, be careful when using various GBICs between equipment from different vendors. Although GBICs can be used between devices, vendors often ship GBICs that have been fully qualified and tested specifically with their equipment. It is usually best to stick with the GBICs that are shipped with a product or provided by the manufacturer, since this reduces the possibility of support issues with your equipment. Although GBICs have their issues, it would be almost impossible to develop a scalable SAN without them.
Media Interface Adapters Media Interface Adapters (MIAs) convert copper signals to optical signals by sitting between a copper port and generating a laser from the copper signal. MIAs convert DB-9 copper connectors to optical SC connectors and are most commonly used when a device with a fixed media copper port needs to be connected optically to the rest of the SAN. Since the maximum range of an active copper line is 30 meters, using MIAs extends your connection range to 500 meters, the maximum distance for multimode fiber. MIAs are most commonly used in this manner to connect legacy devices with fixed copper media. You should carefully consider using MIAs. Using an MIA as an adapter adds a connection, which significantly reduces signal quality.
75
140_SANs_03
76
8/14/01
1:35 PM
Page 76
Chapter 3 • SAN Components and Equipment
Using Hubs Fibre Channel hubs are used to connect simple FC-AL environments. Hubs were the original interconnect mechanism used for Fibre Channel, and provide connectivity between nodes in a loop. A hub connects ports, sending frames between individual ports but not routing them to other ports. Simple hubs do this electrically, and more intelligent hubs might also switch frames through the loop. Switched hubs do not implement a switched fabric protocol, but they still maintain the FC-AL environment. As a result, they have most of the same reliability, scalability, performance, and manageability limits of unmanaged hubs. For this reason, switched fabric is becoming the dominant SAN technology. However, for low-end installations, hubs can offer a less-expensive alternative to switches. If you need to scale your SAN at some point in the future, consider buying an entry-level switch instead of a hub. This section briefly describes the different kinds of hubs and their major features and how hubs are best used in your network.
Simple Electrical Hubs Simple electrical hubs consist of a simple series of circuits that detect whether a connection has been plugged into a port on the hub.Think of a hub as being a “feature-rich” wire. A resilient-loop circuit simply completes a connection to ensure that a loop is continuous throughout the hub. Simple hubs generally support only copper and do not include any software functionality. Although these hubs are still available, they are used only in the simplest of configurations due to their lack of fault tolerance, operational difficulty, shared bandwidth, and difficulty in maintaining a stable loop when there is more than one initiator.
Managed Hubs As Fibre Channel has evolved, manufacturers have found that just having simple hubs does not address problems of stability and manageability in a network. Managed hubs were designed in response to these issues. Managed hubs, unlike their simple electrical predecessors, do not just connect wires from port to port. Instead, they add more sophisticated functionality such as fault detection on ports, settable port modes, and isolated loop operation. More advanced hubs enhance performance or usability with frame switching capabilities between ports, privately routed frames between initiators and targets (rather than having all nodes pass traffic through the entire loop), and advanced diagnostic capabilities.
140_SANs_03
8/14/01
1:35 PM
Page 77
SAN Components and Equipment • Chapter 3
A typical managed hub operates much like a simple electrical hub, by connecting adjacent ports in a continuous electrical circuit. However, managed hubs add a fair amount of intelligence to monitoring the port, and might also contain circuits that can interpret and modify frames received on the port. Instead of blindly connecting two electrical paths, a managed hub might actually receive the frames on a port and decide whether to transmit them further down the loop. Another capability of these hubs is to filter out unwanted frames. For example, when a marginal connection is sending invalid primitives into the line, a managed hub will discard this frame. Managed hubs can also interpret a frame and send it directly to a downstream port—essentially “switching” the packet and avoiding the need for every node on the loop to see the frame. Managed hubs can also segment a loop through software (loop zoning), and usually include intelligence to handle the difficulties of managing Loop Initialization Primitive (LIP) conditions and loop bring up. LIPs are part of a loop initialization process and are expected in a healthy and normal loop. However, certain conditions inherent to loops create scenarios where LIPs cause an interruption to I/O or prevent devices from effectively communicating on the loop. Advanced managed hubs also avoid the problems of simple electrical hubs by isolating initiators. Initiators on the loop can be configured to see and communicate only with specific storage devices and can be screened from other initiators. This prevents some problems that can occur when initiators try to reset or otherwise communicate with each other. The following is a list of the typical capabilities provided by managed hubs. Each item is discussed in more detail in the sections that follow: ■
LIP isolation
■
Automatic port bypass
■
Signal retiming
■
Loop zoning
■
Web interface
■
Telnet
■
Port-event logging
■
SNMP support
LIP isolation is the capability of a hub to prevent LIPs from being transmitted to all nodes in a loop. LIPs used to be the primary source of instability in an
77
140_SANs_03
78
8/14/01
1:35 PM
Page 78
Chapter 3 • SAN Components and Equipment
FC-AL configuration, due to the complexity of the FC-AL protocol.With LIP isolation, LIPs are isolated from all other parts of the loop, preventing the disruption of traffic and avoiding many LIP protocol-related problems. Automatic port bypass is the ability of a hub to automatically bypass a port if too many errors have been detected. Software keeps track of the number and rate of errors coming from devices, and if the error rate exceeds a certain, user-defined threshold, it will automatically bypass a port.This helps to prevent the whole loop from becoming unusable due to a single device having problems.When a device in a loop experiences partial failure and is not bypassed, it normally will constantly LIP.This essentially eliminates communication between other devices on a loop— similar to somebody’s cell phone continually ringing during a meeting. Signal retiming is the ability of hub hardware to clean up the signal received from a device. Instead of just electrically passing a signal through the port (including potential errors or noise on the line), a retimed port will take the signal and re-encode it on the wire. Any errors will be removed from the signal and noise removed from the line. Managed hubs, unlike unmanaged hubs, also provide for manageability functions such as telnet, SNMP, serial ports, and port logging to monitor the ports. This makes it easier to configure devices, diagnose problems, and view activity in your FC-AL configuration. Due to inherent limitations in FC-AL, hubs are being used less frequently in Fibre Channel installations and are generally found only on low-end, low–portcount installations with only a few initiators. Early problems with instability in Fibre Channel were due to the difficulties in implementing the FC-AL protocol. Although managed hubs have reduced the problems, the inherent limitations of the early technology have driven installations to migrate to fully switched fabric switches, which are more reliable and provide a higher level of performance and manageability.
LIP Service: Fibre Channel LIPs, Problems, and Solutions With hub technology and FC-AL, one of the hardest parts of maintaining a network is managing the LIP process. Because of the complexity of this process and various early incompatibilities between equipment, the LIP process often resulted in the instability of Fibre Channel installations. Note that LIPs were designed as, and continue to be, a healthy aspect of FC-AL. Unfortunately, LIPs can also be the cause of loop instability.
140_SANs_03
8/14/01
1:35 PM
Page 79
SAN Components and Equipment • Chapter 3
A typical problem occurs when nodes are added or removed from the loop, either intentionally by an administrator, when any cable is either plugged or unplugged, or when a device is powered up or down.When a change to an FCAL loop occurs, the LIP process starts and all of the devices on a loop stop whatever they are doing and renegotiate for addresses in the loop.When a node is added or removed from a loop, nodes go into the LIP process. Each node passes a frame through the loop with information used to determine which address to use to send Fibre Channel frames to it—its device ID. As you can imagine, this is a problem if traffic is being sent across the Fibre Channel loop—any data that has been sent must be re-sent, drivers and software must time out, and all transactions must be retried. Even worse, because of early hardware incompatibilities and the complexity of the LIP process, sometimes the process can take minutes for a loop to quiesce, and in rare cases the loop might never settle and prevent transactions from continuing. The term LIP storm was coined to describe what happens in these situations, and an entire generation of managed hubs were designed to minimize or eliminate this problem.Today, with the newest hubs and switches operating in loop mode, these problems are minimized.
Getting Out of the Loop: Migrating to Switched Fabric Because of the problems described in FC-AL environments, many installations have been migrating out of loop environments into switched fabric. Loops scale to only 127 devices, while switched fabrics support hundreds or even thousands of devices. Because Fibre Channel switches inherently support loop devices, and in fact implement a superset of the Fibre Channel loop protocol, migration is fairly straightforward. By migrating to a switched fabric environment, the reliability problems, manageability problems, and bandwidth limitations of loops can be eliminated fairly easily. Brocade switches provide features called QuickLoop and Fabric Assist that make it easy to migrate from a loop environment directly to switched fabric. Note that it might be necessary to purchase a separate license for these features. Hubs can be entirely replaced by switches and, in fact, some low-end Brocade switches are positioned to directly replace hubs as a component (for instance, the SilkWorm 2010 switch). Devices that cannot take advantage of switched fabrics can still operate in private-loop mode, fabric-aware devices can operate in fabric mode, and operation and capabilities can be maintained with only a simple equipment swap.
79
140_SANs_03
80
8/14/01
1:35 PM
Page 80
Chapter 3 • SAN Components and Equipment
QuickLoop operates by setting up a virtual loop through switched ports. Each of the ports in this case operates as if it were a hub port, but takes advantage of the capabilities of the Brocade switch, including switching capabilities.
Using Switches and Fibre Channel Fabrics A Fibre Channel switch is logically positioned in the center of a SAN and is connected to hosts, storage, or other switches.The fabric infrastructure can be viewed as the foundation upon which the rest of the SAN is built.When a frame arrives from a device, a switch accepts and then routes that frame to the proper destination device. In fact, using the Brocade cut-through routing approach, a frame can begin to be forwarded even before it has been completely received. A fabric switch also contains a great deal of intelligence, providing services for locating other nodes in a network (the Simple Name Server [SNS]), automatically establishing routes between other switches in the fabric, compartmentalizing devices into zones (zoning), as well as monitoring and handling errors (basic Brocade Fabric OS functions and Fabric Watch).We discuss fabric services further in Chapter 2, “Fibre Channel Basics.” Brocade switches also provide functionality that allows private loop devices to participate in a fabric and translate the communication between fabric devices and older private devices. In fact, the translative mode of operation for a port on a Brocade switch will automatically allow any private target node (such as a private loop JBOD) to function fully as part of a fabric.This feature is a core piece of the Fabric OS and does not require a license. Making this work for a private HBA, on the other hand, requires QuickLoop and/or Fabric Assist options.
Basic Switch Types Fibre Channel switches are often classified into different categories, depending on capabilities and features. In many cases, the hardware might be based on the same underlying architecture or Application-Specific Integrated Circuit (ASIC), but the software features are variable and priced accordingly to meet the requirements of that class of switch.The exception is highly redundant “core-class” switches, which tend to be developed on their own fault-tolerant hardware platforms.This section covers the major categories of switches and explains the differences between each kind of switch.
140_SANs_03
8/14/01
1:35 PM
Page 81
SAN Components and Equipment • Chapter 3
Entry-Level Switches Entry-level switches are focused on small workgroups of eight to 16 ports, are geared toward low cost, and deliver limited scalability and management.They tend to be used to replace hubs and offer higher bandwidth and reliability. Entry-level switches are often integrated into complete storage solutions rather than purchased separately. Entry-level switches offer limited levels for cascading of switch ports. Brocade entry-level switches can be upgraded with a license to handle more scalability or to add functionality such as zoning or Web management.
Scalable Fabric Switches Fabric switches provide the ability to cascade switches together to create a larger fabric. By connecting one or more ports between two switches, all of the ports connected to either switch see one single image of the network, with any nodes on the switches available to other nodes in the fabric. Essentially, by connecting the switches together, you can create one large, virtual switch that also has the advantage of being distributed—even over large distances. Fabrics built with fabric switches work as a single fabric, with all ports connected into the network able to view and access any other node on the network as if it were on the local switch. A unified Name Server and management services allow you to view and modify fabric information for an entire fabric through single interfaces. An important factor in creating a distributed fabric is understanding the bandwidth availability of the ISLs. It is important to remember that the speed available between any two ports can be impacted by the lack of available bandwidth on an ISL and that you might need to employ multiple ISLs to maintain the necessary bandwidth.We discuss this topic further in Chapter 5, “The SAN Design Process,” and Chapter 7, “Developing a SAN Architecture.”
Core Fabric Switches Core fabric switches are designed to reside in the middle of a large SAN, interconnecting multiple edge switches to form multihundred-port SANs. Core fabric switches can also function as standalone or edge switches, of course, but their robust feature set and internal architecture is designed to allow them to work in carrier-class environments as well. Other attributes of core fabric switches are the ability to support protocols other than Fibre Channel, such as InfiniBand, 2 Gbit/sec support, and advanced fabric services like security, trunking, and frame filtering.
81
140_SANs_03
82
8/14/01
1:35 PM
Page 82
Chapter 3 • SAN Components and Equipment
Core fabric switches generally provide a high port count, from 64 to 128 ports, and employ extensive internal interconnects to route frames at full speed. These switches are built for scalability and bandwidth, and are designed to route as many ports as quickly as possible with the least amount of delay to a frame. In addition, they tend to be blade-based: you can add and remove switch blades in a chassis to add functionality as needed, to facilitate hot sparing or online repairs, and to “pay as you grow.” Some enterprise switches do not support arbitrated loop operation or other loop devices directly, instead focusing on core switching capability. Brocade highend enterprise switches provide all of the functionality of Brocade mid- and entry-level switches, including support for loops. For environments in which availability is most important, and you are willing to pay a premium for redundancy, highly redundant switches provide fully redundant components throughout the switch, remove single points of failure, and provide extremely high uptime. A premium is paid for highly available backplanes, power supplies, redundant circuitry, and software to maintain availability.These types of switches include a great deal of logic and circuitry to deal with hardware failures within the switch. Beyond redundancy, core fabric switches support nonservice—interrupting software upgrades, virtually eliminating the need to schedule maintenance windows. An alternate approach that provides a level of redundancy in the network is deploying a resilient, dual fabric. A resilient, dual fabric allows you to remove single points of failure and protect against the unlikely event of an entire fabric going down due to a software or hardware error, fire, natural disaster, or operator error. For the most highly available networks, you should deploy a dual fabric built with core fabric switches.
Features of Fibre Channel Switches Fibre Channel switches provide many different features, including support for GBICs, redundant fans and power supplies, zoning, loop operation, and multiple interfaces for management. Each of these features adds to the overall operation of your switched network and understanding the benefits and advantages of each can help you design a robust and scalable SAN.This section covers the major features of Fibre Channel switches, describes what you need to know about each of the features, and how to best use these features.The capabilities of Fibre Channel switches are listed here: ■
Self-configuring ports
■
Loop mode operation
140_SANs_03
8/14/01
1:35 PM
Page 83
SAN Components and Equipment • Chapter 3 ■
Switch cascading
■
Auto-sensing speed detection
■
Configurable frame buffers
■
Zoning (physical port- and WWN-based)
■
IP over Fibre Channel broadcasting
■
Telnet
■
Web-based management
■
Simple Network Management Protocol (SNMP)
■
SCSI Enclosure Services (SES)
Zoning Control over which nodes in a network can view and access each other has become a necessary part of configuring your SAN. Depending on which Fibre Channel switch you buy, zoning is implemented in different ways and also might support different kinds of zoning. The simplest kind of zoning is port-based zoning, or zoning by a physical port on the switch. A port zoning entry could be translated something like, “Only allow devices on switch 1, port 1 to talk to devices on switch 3, port 2.” WWN-based zoning provides the capability to restrict devices, specified by a port or node WWN, into zones.This is much more flexible, since it allows nodes anywhere in a network to maintain the zones they are restricted to. However, it does have its disadvantages. For example, if you replace a device, the WWN might change, while the port address stays the same. Zoning is classified into two types: hard and soft zoning. Soft zoning uses only software to enforce zones—usually through selective information presented to end nodes through the fabric SNS. Nodes in a zone are informed of each other only through names services in soft zoning. However, frames are not barred from being transmitted between nodes that are not in the same zone.This works fairly well, but does suffer if zones change, if hardware caches Name Server tables, or if you want to guarantee that no frames (intentional or accidental) are sent to devices. Hard zoning uses hardware, examining each frame that comes across the fabric and ensuring that it is allowed to pass through to a node. Hard zoning behaves exactly like soft zoning and is usually used in conjunction with it.
83
140_SANs_03
84
8/14/01
1:35 PM
Page 84
Chapter 3 • SAN Components and Equipment
However, no frames, accidental or intentional, can pass through to nodes where permission is not given. Newer hardware is starting to extend the features of zoning further into the network, to the level of storage LUNs. As hardware advances and the ability to filter traffic beyond the port level all the way down to the LUN level becomes available, zoning will enable finer granularity control over the specific logical units on a storage unit that a specific initiator on the Fibre Channel fabric can access.This will enable better control and allocation of storage through zoning on a network. If you are sharing storage on your SAN, if the network is large, or if you want to closely control access to data and information on your SAN, zoning is a necessary feature of your switch and should be considered a requirement. On the other hand, if the size of your SAN is limited and devices attached to it are very well controlled, then zoning might not be as much of a necessity.
Classes of Service The Fibre Channel protocol supports different classes of service.The class of service determines the level of error control for transfers. For communication between nodes to be successful, a switch has to support different modes of operation.The important part of support for classes of service is making sure that all of the equipment that you are running supports the same classes of operation. Otherwise, they will be unable to communicate with each other. Most Fibre Channel switches and other hardware devices support Class 3 operation, which is a connectionless conduit without confirmation of transfers across the SAN—the ideal case for SCSI transfers.This is because the upper-layer protocol on top of Fibre Channel is already doing the error control. Doing it in Fibre Channel as well would just add overhead.The majority of components you can buy for Fibre Channel are Class 3 devices and are fully interoperable. In some cases, the confirmation of transfers (acknowledgement frame [ACK]) between nodes is desired for better error detection, in which case, Class 2 would be used. However, Class 2 is not widely available on all hardware, although most Fibre Channel switches support it. Another class of service is Class F, which is used for internal control and coordination of the fabric. Switches are exclusive users of Class F. Finally, there is Class 1, which is rarely implemented in switches, but is supported by some older equipment. See Chapter 2, “Fibre Channel Basics,” for further detail on the various classes of service defined for Fibre Channel.
140_SANs_03
8/14/01
1:35 PM
Page 85
SAN Components and Equipment • Chapter 3
Fabric Services Fabric services are the set of internal services available to devices in a SAN. Fabric services determine the level of manageability and interoperability of your switch fabric.These services are used by devices when they first attach to your network and allow different devices to locate others on the network.This section discusses fabric services relative to switch and SAN implementations. Name Server support is a part of the Fibre Channel standard for switched fabrics and provides devices with a directory of other devices on the fabric.There is one Name Server service for an entire fabric, whether it is a single switch or dozens of switches. However, that service is distributed across every switch in the fabric and provides a high degree of resiliency. Any node querying the Name Server will receive the same answer regardless of location—all of the switches participating in the Name Server service cooperate and present a unified picture of the fabric. All switches support this functionality since it is a basic part of switched fabric operation. The Management Service is an in-band fabric service that provides basic management data about the network. Included in this data is topology information: what is connected to where on a switched fabric and basic information about attached nodes. In addition, the Management Server provides unzoned access to the Name Server.This is necessary when you have a management station that needs to know about everything in the network, but does not need to have access to the storage or hosts in a zone.The Management Server is used by SAN management software that needs additional information about the fabric, topology (physical layout) of the network, and other management information. Registered State Change Notification (RSCN) is a service of the fabric that notifies nodes of changes in state of other attached nodes: for example, if a node is reset, removed, or otherwise undergoes a significant change in status. Most switches support RSCN, which is critical for operation of your SAN.This is particularly important for detecting error conditions and informing nodes about problems.When the state of the node changes, devices using that node are immediately informed and can react properly, rather than trying requests and timing out due to errors.This feature is required for hosts that add storage “on the fly,” since the RSCN is the mechanism by which the host finds out about the newly available storage. RSCN events are generated by devices and by the fabric itself for any sort of physical change to the topology of the fabric. For example, a device is added or removed from the SAN, a switch is added to a fabric, or a device has been internally reset and has dropped off and comes back onto the
85
140_SANs_03
86
8/14/01
1:35 PM
Page 86
Chapter 3 • SAN Components and Equipment
SAN. RSCNs reduce the need for hardware to repeatedly check for changes in equipment condition, called polling, and thus reduce the amount of nondata traffic necessary to keep your network up and running. There are additional services defined in the Fibre Channel standards that are not necessarily supported by all, or even any, switches. For example, the Time Server is not yet supported on any switching platform that we know of, but it is defined as a standard. If you or the management software you buy requires the additional services, you should ensure that the switch you buy supports or can enable those services. For example,VERITAS SANPoint Control uses the Management Server, but many switch vendors do not support that service. Brocade does support the Management Server.
Redundancy Because SANs are usually involved in the critical parts of your business, and because, unlike regular network traffic, data traffic on a SAN must not be lost or corrupted, the need for equipment protection through redundancy is important. Redundancy in its most basic capability takes the form of redundant power supplies and fans. In the field, power supplies and fans are the most likely components to fail. Fan bearings, which are the most mechanical pieces of any equipment, receive the most use, and because they are relatively inexpensive, these components tend to have a shorter life span than an integrated circuit. Redundant and hot-swappable power supplies help alleviate the problem of wear and tear on power supplies and the fans that cool them.With a redundant power supply, if one of the power supplies fails, circuitry can detect it and shut down the offending supply and issue a warning—either through software or through an indicator light or buzzer—all while allowing the equipment to continue running. A technician can swap in a new replacement unit for the power supply in real time, without affecting operation. Similarly, other components of a switch can also be made redundant, including back planes, circuit boards, memory, and CPUs—albeit at a much higher cost.
Buffer Credits per Port An important aspect of Fibre Channel throughput is the amount of buffer credits that are available on each switch port.The number of buffer credits available on a switch port is an important factor, particularly for long-distance applications. Although optical networks are fast, light still has a definite speed, which is
140_SANs_03
8/14/01
1:35 PM
Page 87
SAN Components and Equipment • Chapter 3
approximately 5 km/microsecond.This is slow enough that over long distances, the amount of buffers required to keep operations running is very important. If there are not enough buffers available on the far end of a long run of optical cable, the hardware will run out of buffers to receive them, throttling the actual amount of data that can be sent over that cable. By ensuring that you have enough buffers to support this sort of operation, you can ensure smooth and maximum throughput across your optical link. In shorter cable length configurations, buffer credits are less important. Most Fibre Channel switches are configured with plenty of buffer credits per port when dealing with distances up to 30 km. However, it is worth knowing what your switch can support so that you can ensure optimal operation, especially when you intend to use your switch across long distances. At 1 Gbit/sec, it takes five 2 KB frame buffers to provide enough buffering to ensure full-bandwidth performance at 10 km.To ensure full-bandwidth performance at 100 km, you need approximately 50 buffer credits for each switch port.
Self-Configuring Ports Fibre Channel has many different modes of operation for ports: loop (FL_Port), switched fabric (F_Port), and ISLs (E_Port). Even within loop ports there can be different modes, depending on whether the attached devices are public or private loop, and if they are initiators or targets. Also, the emerging 2 Gbit/sec and higher Fibre Channel standards will create even more modes of operation. Self-configuring ports are able to detect what kind of mode the other side of the link is operating on and automatically configure themselves to support that mode of operation.This is particularly important in the case of fabric-supporting devices, which operate much better and with more reliability if they are operated in fabric mode (also called point-to-point when a device is connected to a switch). A self-configuring port analyzes the primitives on the wire to properly configure operation to match the connected Nx_Port hardware.The term Nx_Port is used to identify either an N_Port (point-to-point connection) or an NL_port (loop connection) for the connecting device.This also supports dynamic reconfiguration of a network: for example, changing the placement of an ISL should happen automatically, rather than requiring an administrator to telnet or log in to a Web interface to control the configuration of a particular port. Some switch vendors have specific ports that support only certain operations, requiring that ISLs be connected on only a few specified ports. All Brocade switches can support self-configuration on all ports. Certain entry-level products might require an additional software license to enable this support, but the capability is present in the hardware. Self-configuring ports make it much easier
87
140_SANs_03
88
8/14/01
1:35 PM
Page 88
Chapter 3 • SAN Components and Equipment
to manage your fabric, eliminating the need for dedicating certain ports for certain functions. Another feature of switch hardware and software is the ability to manually set the configuration of these ports. Sometimes, equipment is not able to properly auto-configure a port. A device that supports switched fabrics might not be recognized as a fabric device, and a port might be configured as a loop device. In order to ensure that those devices operate in the best mode, you might need switch software to force configuration of a particular port. Being able to manually set the type of port indicates that there is a conflict between the connecting device and the switch. If a port is locked to a certain type, it limits the functionality and can cause potential problems if other devices are plugged into that port. Once a port is locked, it does not become “plug-and-play” anymore.
Auto-Negotiating Speeds As Fibre Channel moves from 1 Gbit/sec to 2 Gbit/sec and beyond, support for auto-negotiation of speeds becomes necessary to support mixed-speed networks. Auto-negotiation uses communication with devices attached to a switch to determine if they are running at 1 Gbit/sec or 2 Gbit/sec and automatically selects the correct speed.
IP over Fibre Channel Broadcasting The use of IP over Fibre Channel (IPFC) is, for the most part, identical to any other IP network. Fibre Channel, as a communications medium, does not inherently support broadcasting frames to all nodes on a fabric identically to Ethernet or other IP networks. Fibre Channel broadcast is a function of switches that will automatically resend broadcast frames to all attached ports on the Fibre Channel network, effectively emulating the broadcast properties of Ethernet networks.This helps to fully support file sharing, such as NFS, bootp, ARP, ping, and other protocols on top of IP that are dependent on broadcast and that are usually not aware of the behavior of IPFC. This is a necessary part of fabric operation if you intend to send any IP frames across your network. Some HBAs do not react well to IP broadcasts, so you might need to use switch zoning to allow them to coexist with other HBAs that are running IP.
140_SANs_03
8/14/01
1:35 PM
Page 89
SAN Components and Equipment • Chapter 3
Firmware Upgrade Methods Although many users will buy a switch directly out of a box and never look at the firmware installed on it again, sometimes it is necessary to upgrade the firmware to fix a bug with newly introduced hardware, add a new feature, or enable management through a third-party software package. Also, if you are building a large fabric, it might be desirable to have a unified firmware version throughout that fabric.This will ensure a consistent feature set and the most reliable operation in a large heterogeneous fabric. Software upgrades can be accomplished in different ways. For fast upgrades of firmware, look for the capability to download firmware to the switch through Ethernet.The most basic of these downloads is through the serial port, which requires an RS232 connection and a PC or other machine that sends the firmware to a switch through a slow serial link. Brocade switches do not support the download of firmware through a serial connection and instead use Ethernet for downloading firmware. However, it is possible to manage all Brocade switches except the SilkWorm 2800 with a serial connection. Another consideration for firmware upgrades is how much impact this will have on your network.The ideal operation is “hot upgrades,” firmware upgrades that can be done while equipment is running and can be “rolled in” to production. Few pieces of equipment currently support this, but equipment that does can keep downtime to a minimum. Next is upgrade-on-reboot, where firmware upgrades can be done, but the new firmware does not take effect until the switch is booted. Operation can continue until a reboot is triggered. The worst option is offline upgrading.This is required when a component must be offline to upgrade, or even worse, when all equipment must be upgraded at the same time. Many pieces of hardware are eliminating this, but you should still be aware what kind of work is required when you need to upgrade switch firmware. The good news with firmware upgrades is that, in a dual-fabric SAN, you can upgrade one fabric at a time.This will enable a firmware upgrade to take place with no disruption to your environment. Using dual fabrics might require additional software on your host, such as VERITAS DMP, a multipathing HBA driver such as the TROIKA driver, or multipathing RAID drivers. Since dual fabrics are always advisable in uptime-sensitive environments, the firmware upgrade disruption question is moot for real-world applications. See Chapter 7 for more information on SAN availability models.
89
140_SANs_03
90
8/14/01
1:35 PM
Page 90
Chapter 3 • SAN Components and Equipment
Loop Operation: Making Your Switch Act Like a Hub A convenient feature of almost all Fibre Channel switches is their capability to act similarly to a Fibre Channel hub. By behaving like a hub, a switch can work with FC-AL devices that do not support fabric operation. Because the Fibre Channel standard began with simple FC-AL operation, and much of the storage hardware and even some of the HBAs available might be only FC-AL and not switched fabric-compatible, the ability to make a switch act like a hub can ensure that older equipment will still work in your network. In general, a set of switch ports or an entire switch might be configured for loop operation, specifying which ports are running in loop mode. Some low-end switches are actually forced to operate in loop mode, with the ability to be license-upgraded to support full fabric. The capability for loop operation is a must if you are directly attaching storage devices that support only the older FC-AL protocol to your switch. Refer to Chapter 4 for further detail regarding how QuickLoop and Fabric Assist enable private-loop devices to fully participate in the SAN.
FSPF Compliance Fibre Channel switches from different vendors started out fully compatible with end nodes in a network. However, until fairly recently they were not able to pass frames between each other (inter-vendor switch frame compatibility). Several different routing algorithms for inter-switch routing existed. Brocade switches all used a protocol developed at Brocade called Fabric Shortest Path First (FSPF). Recent efforts in compatibility have standardized on the FSPF routing protocol algorithm for routing Fibre Channel frames between switches, and now all vendors are required to support this protocol in order to be Fibre Channel standardscompliant. FSPF, which was originally used by all Brocade switches, forms the basic protocol for exchanging and routing frames between switches in a Fibre Channel fabric. FSPF compliance is most important if you are trying to mix and match Fibre Channel switch hardware. Because all switches must follow this standard fully to interoperate, you must make sure all switches in your network support the standard. In addition, because the standard is very new, it is important to make sure that all pieces of hardware you expect to work together have actually been tested together. Also check to see what advanced features will be lost when interconnecting switches from other vendors into a Brocade fabric. At this time, no switch vendor supports the complete feature set that Brocade switches implement, and it
140_SANs_03
8/14/01
1:35 PM
Page 91
SAN Components and Equipment • Chapter 3
is possible that the needed functionality might be lost by introducing other vendors’ switches. Since Brocade implements a superset of most other vendors’ feature sets, it might be more practical to introduce a Brocade switch into another vendor’s existing fabric than vice versa. Firmware versions and topology are still very important in these mixed environments, which are still undergoing testing at the publication of this book. The trend toward full compatibility in inter-vendor switch compatibility is an important basis for the future and promises not only to reduce cost, but also more important, to permit the interchangeability of hardware in your network.
Management Interfaces Fibre Channel switches support many different ways to manage a switch.These different interfaces allow you to choose how you want to configure a switch, and they also take into consideration how you intend to manage your switch, as well as any other tools you might already have deployed in your network.
Serial Port The most basic management interface for Fibre Channel switches and other equipment is the serial port. A standard, RS232-based port is generally available on Fibre Channel switch equipment that allows command-line interaction with different configuration options.
Telnet Telnet is the standard IP networking ability to log in to a piece of equipment through a telnet interface from any host server attached on Ethernet, or even in-band through Fibre Channel itself (Figure 3.7).You typically log in to a telnet interface and execute command-line commands.Telnet has the advantage of being convenient to run remotely or through a slow connection, and can also be scripted for automatically configuring switch settings through a nightly script or for difficult operations.The disadvantage is that command-line interfaces tend to be difficult to use, especially for complex operations like zoning and viewing lots of information at once, in which a GUI interface such as WEB TOOLS is more practical.
SNMP Simple Network Management Protocol (SNMP) is an IP-based protocol for managing any kind of network equipment, including Fibre Channel switches. SNMP provides mostly read operation of switch functionality and configuration, as well as
91
140_SANs_03
92
8/14/01
1:35 PM
Page 92
Chapter 3 • SAN Components and Equipment
critical error counters and statistics. Almost all Fibre Channel switches (as well as hubs, routers, and even some storage arrays) provide SNMP Management Information Base (MIB) information.This is mostly used in conjunction with traditional network monitoring applications like HP OpenView, CA Unicenter, and Tivoli NetView, but also is used by Fibre Channel management software to provide information from network hardware. Figure 3.7 Telnet Session with a Brocade Switch
Standards for SNMP Management Information SNMP defines only the basic protocol that transports management information. The actual information transported across the network is defined on top of the SNMP protocol through Management Information Base (MIB) definitions. It is important to make sure that the equipment you buy supports the correct MIBs that enable software to interpret and use the information available from your switch. The FibreAlliance MIB, now under consideration in the Internet Engineering Task Force (IETF) standards organization, is supported by most Fibre Channel network hardware. Defined by the FibreAlliance organization, which was started by storage manufacturer EMC, the MIB provides common information for discovery of topology and equipment capabilities in a SAN. The MIB provides information about how many Continued
140_SANs_03
8/14/01
1:35 PM
Page 93
SAN Components and Equipment • Chapter 3
ports exist on a piece of equipment; what is connected on each port; and even detailed information such as error counters and frame counts. In addition, basic asset tracking information such as manufacturer strings, model numbers, and other information is presented in this MIB definition. The Fibre Channel Management MIB, which is supported by some earlier switches, is a different MIB that provides much of the same information. This MIB preceded the FibreAlliance MIB and is required by some software for accessing parameters not exposed by the FibreAlliance MIB. It is best to check with your software vendor to understand if this information is required for operation. Most hardware also presents equipment- or vendor-specific MIBs, defined by the manufacturer. Often times, these MIB definitions are also made available to customers and can expose things that are not industry-standard: for example, special features of a switch, information that is specific to the hardware, or special functions. You should check with your manufacturer or with software shipped with your hardware for equipment-specific MIB definitions.
Web-Based Management Web-based management interfaces provide a graphical,Web-based way of accessing and modifying switch settings. In general, most Web-based management tools provide a page that you can access in your browser, view switch status, and set most switch settings. In many cases, the Web-based GUI can help make complex tasks such as zoning much easier, and also provide a visual indicator of switch function. However, not all switches export all functionality through a GUI and they instead might require a telnet or serial port session to access some tasks. Brocade switches allow practically all management through the Web interface, which greatly simplifies the management by more casual users. There are two types of Web-based GUIs available: pure HTML-based and Java-based Web pages.Web-based applications usually present simple HTML pages to access all switch functionality, where Java-based Web pages have embedded Java applets running more like a standard application.There are advantages and disadvantages to each, but you are generally stuck with the type of interface that ships with your switch.The advantage of pure HTML pages is the speed of loading a specific page. A Java Virtual Machine (JVM) does not have to be loaded, which sometimes can take quite a while across a slow link. Moreover, because of compatibility issues between different Web browsers, you might
93
140_SANs_03
94
8/14/01
1:35 PM
Page 94
Chapter 3 • SAN Components and Equipment
encounter some difficulties if your browser is not exactly the same as the version for which the switch GUI developers qualified their Java software.The advantage of a Java-based applet is it offers user-friendly management of complex tasks. When you select the switches you intend to use in your network, you should compare what common tasks you would like to do against the Web-based GUI tools available, and make sure that you purchase the licenses for Web-based management.Web-based tools are very helpful for accomplishing day-to-day tasks. In addition,Web-based tools make management of your switches easy even from remote locations or offsite.
Application-Based Management Some switches on the market also support application-based management. Instead of an embedded Web server or Java application, a separate, externally run program manages your switch.These applications are usually based on Java as well, but need to be installed on a server. Managing your switch through an application sometimes can be faster than running an application from a Web interface, and some hardware offers identical interfaces between the Web and an application. Brocade Fabric Manager is an example of an externally based management program. Application-based management works best when you have a permanent network management station where you can have software installed and used normally. It is more difficult if you need to move from place to place and do not want to have to reinstall the software on whatever machine you happen to be using that day. Application-based management hosts are a management single point of failure.
SCSI Enclosure Services SCSI Enclosure Services (SES) is a SCSI protocol-based method of obtaining management information. SES support gives some information on the status of SCSI equipment on the network. If your software supports the SES standard, you can use this feature of Fibre Channel switches to also monitor the basic health and well-being of your switch through the same SCSI-based management software. SES is generally used by operating systems so that they can incorporate certain management functionality into their environment. End users rarely use it, and its usage by operating systems is waning due to the advent of sophisticated and powerful alternatives such as the Management Server.
140_SANs_03
8/14/01
1:35 PM
Page 95
SAN Components and Equipment • Chapter 3
Connecting Your Servers with Host Bus Adapters HBAs connect hosts to your Fibre Channel network. An HBA translates operating system SCSI commands to the proper Fibre Channel frames and protocols on the wire when plugging into a bus such as PCI or SBus. Unlike Ethernet network adapters, Fibre Channel adapters are actually much more intelligent and frequently contain embedded processors and embedded firmware to negotiate the Fibre Channel protocol. Fibre Channel HBAs provide advanced functionality such as persistent binding and HBA-based LUN masking, which are used in conjunction with switch zoning and storage LUN masking to control and allocate storage in your SAN. This section covers the major types of HBAs and details the features available on most HBAs. In addition, a discussion of specific issues about using HBAs and how to implement these features in your own SAN will help you understand how to best use these components in your network.
Connecting Hosts to the Fabric HBAs operate by plugging into the internal bus of your host machine (for instance, PCI or SBus). Loaded with device driver software, the HBAs appear to operating systems as a SCSI adapter. In most cases, an HBA is indistinguishable from other storage adapters, such as SCSI adapters, to the operating system. HBAs even emulate the way the legacy SCSI adapters communicate with the operating system.The HBA will map devices seen on the network to SCSI bus, target, and LUN addresses associated with a SCSI adapter. An operating system treats an HBA exactly like it does a SCSI adapter, down to the exact same SCSI commands and packets.The HBA takes these packets and translates them into the Fibre Channel protocol, adding network headers and error handling. It transmits the packets across the network, makes sure of the response from the storage, and returns the information and status back to the operating system—all as if it were a SCSI adapter. Other advanced HBAs also do this for the Internet Protocol (IP) and Virtual Interface (VI) protocol, providing network and clustering adapters to the operating system and software.
HBA Types HBAs range from low-cost, embedded chips to high-end, dual-channel multipathing adapters.The most basic HBAs support only small FC-AL loops with a few devices and contain minimal buffering memory or intelligence. On the high
95
140_SANs_03
96
8/14/01
1:35 PM
Page 96
Chapter 3 • SAN Components and Equipment
end, adapters might include additional buffer memory for better performance and throughput, intelligence to handle large fabric deployments, and high-end features such as HBA-based LUN masking and failover capability.
A Plethora of Protocols Fibre Channel networks, although primarily used for storage using the SCSI FCP protocol, also can be used for other protocols such as IP for standard networking and VI for clustering. Different HBAs can support different protocols and at a minimum, support SCSI FCP. It is becoming standard for adapters to support SCSI FCP and IP, and newer adapters now support the VI protocol as well. If you are designing a SAN and think you might want to use it for routing IP frames, backup, or other IP traffic, it is well worth checking to see if your HBA hardware supports IP or VI, or can support these protocols through a software or firmware upgrade.
The FCP/SCSI Protocol FCP/SCSI is the primary protocol used to transfer data over the Fibre Channel network. Fibre Channel Protocol (FCP) encapsulates standard SCSI commands, which are identical to the old SCSI bus commands. Instead of signals, however, the Fibre Channel standard transmits the commands as a set of frames containing the usual command and data phases of the SCSI protocol. In fact, a SCSI application running on top of Fibre Channel is identical to running on the SCSI bus, with no modification. HBAs take responsibility for translating requests to a SCSI bus, target, and LUN, and redirecting that to a specific Fibre Channel address and LUN address. Applications and operating systems written for SCSI can run on top of Fibre Channel unmodified.
The IP Protocol The IP protocol, the standard for the Internet, runs on top of Fibre Channel by following the IPFC protocol. Using the same concepts of IP address and mask, IPcapable adapters generally look and behave identically to Ethernet adapters—only much faster. By installing the appropriate drivers, you can add a network adapter and a set of IP addresses, which instead of being transmitted via Ethernet or another network can be sent by Fibre Channel. Although not typically used to replace Ethernet, IPFC is useful for managing in-band Fibre Channel devices, offloading backup traffic, or connecting machines over the same long-distance links as your storage and taking advantage of the cost savings of not having to run a different network over a different wire. In installations where server slot space is at a premium, IPFC can save an additional slot and network infrastructure.
140_SANs_03
8/14/01
1:35 PM
Page 97
SAN Components and Equipment • Chapter 3
The one area to note is that unlike Ethernet, Fibre Channel is not well suited to broadcasts and multicast operation. Some Fibre Channel networks do not inherently support broadcast packets and rely on software and switch support to properly broadcast frames to all nodes on the network. Brocade supports hardware forwarding of broadcast and multicast frames and the software necessary to support both in a multiswitch fabric.
The VI Protocol VI is a specification that was developed by Intel for low-latency server clustering. In standard clustering environments, Ethernet or other IP protocols have been used to pass data from machine to machine across a network. Unfortunately, because of the software overhead of IP stacks and the many software layers that data must pass through to send IP frames, clustering has struggled to reach its full potential.To solve this problem,VI technology removes the traditional IP networking stack and instead provides a method of directly sending data across a wire to another computer’s memory. Applications, especially cluster-aware applications like databases, have started to use this protocol for cluster communications.VI over Fibre Channel provides the ability to leverage the Fibre Channel infrastructure to also pass VI traffic. Fully clustered databases like Oracle Parallel Server and IBM DB2 both support the VI protocol in their inter-node communications. By configuring these databases to use VI on a Fibre Channel card, these applications run faster and with significantly less CPU usage.
Speeds All Fibre Channel adapters today support the 1 Gbit/sec (100 MB/sec) speeds that all Fibre Channel equipment supports. As network infrastructure, such as switches, moves towards the new 2 Gbit/sec standards, so do HBAs and storage.This new standard provides for double the speed of current adapters, and standards are firming up that allow for auto-negotiation between the 2 Gbit/sec and 1 Gbit/sec speeds and protocol differences.The next-generation 10 Gbit/sec standards are now in development and should allow for 10 times the speed of operation of current-generation SAN components in the future.With most Fibre Channel storage, it is rare that you will even approach the 100 MB/sec performance numbers (200 MB/sec full duplex) that current adapters allow you to reach at 1 Gbit/sec.
97
140_SANs_03
98
8/14/01
1:35 PM
Page 98
Chapter 3 • SAN Components and Equipment
Ports The number of ports available on a Fibre Channel adapter can range from a single port to dual-port adapters with the capability to act as two individual HBAs on a single card. Dual-port adapters add a significant cost reduction by enabling two separate connections into a single card and can be useful where there is a need to connect to two separate fabrics from a single system. One limitation to be aware of with dual-port cards is the overall bandwidth you can achieve when using these adapters.This is not due to the HBAs, but because of the computer system buses to which they are attached. Different architectures such as PCI (33 MHz, 32 bit) cannot handle much more than the 100 MB/sec speeds available on a single Fibre Channel port today.You will probably not be able to get twice the performance with the two ports available on a single card. Moreover, if the HBA were to fail and it contained several ports, they would all fail.
Combination Adapters A recent innovation in Fibre Channel adapters is the appearance of combination adapters, which combine the functionality of Fibre Channel with other network interfaces. For example, combination Gigabit Ethernet and Fibre Channel cards exist on the market.These tend to be used where slot space is at a premium, or in embedded applications where there is a need to support multiple interfaces in a small space.
Fabric-Capable Versus Loop Adapters There are two different classes of HBA capability. Legacy HBAs usually support only loop operation to connect to a network and do not support connection to the fabric.These adapters are termed private HBAs.They can generally connect only to other private FC-AL devices. Some low-end HBAs still support only loop operation. However, it is possible to upgrade the drivers on these HBAs to support fabric attachment. Sun and HP HBAs originally were capable of private mode only and are examples of legacy HBAs. Fabric-capable HBAs support both loop and fabric and can address thousands of nodes connected into a switch.They use the fabric Name Server to access different fabric devices. In addition, fabric switches are aware of the different fabric protocols used to monitor and find other nodes in the network and do not require special modes in a switch to operate in fabric. In general, it is best to find
140_SANs_03
8/14/01
1:35 PM
Page 99
SAN Components and Equipment • Chapter 3
a fabric-capable adapter for your switched fabric SAN, since they are much more advanced and able to deal with the complexities of a switched fabric operation.
HBA-Based LUN Masking HBA-based LUN masking is the capability of an HBA to selectively hide— “mask”—storage devices on the network from a host. By masking specific LUNs from a host, you can control which storage a host maps into the operating system.This is important when you have mixed operating systems on the same network, since you can prevent corruption of data because of ownership contention. LUN masking also provides a method for dividing storage capacity in your network. LUN masking is very important in the context of operating systems such as Windows NT and Windows 2000.Windows operating systems, which are not natively Fibre Channel-aware, do not expect storage volumes to ever be shared with any other hosts. If a storage volume is exposed to more than a single host, the operating system might not be able to mount the file system located on this disk. As a response, the operating system might write a signature on a disk it does not own and will most likely corrupt any data that is on that volume already. By masking the LUNs to only the volumes a host is permitted to see and own, you can avoid these problems entirely. LUN masking is also very important when you are mixing operating systems in a network. Because the way file systems are written varies for different operating systems, if a LUN is formatted for one operating system, the other operating system will not recognize that it is in use. If LUN masking is not used, the second operating system could assume that, because it does not recognize an operating system, it can write its own data on the identical disk—corrupting data that is already there.
Persistent Binding Persistent binding, sometimes referred to as LUN mapping, is the mapping of a Fibre Channel device into an operating system at a specific device location. Persistent binding is particularly important for applications that use the operating system SCSI address to address a device: for example, a fixed device node in Solaris or a raw volume used by an Oracle database. In both of these applications, the ID must be persistent and fixed from reboot to reboot. In some implementations of HBAs, persistent binding and LUN masking are the same. Some vendors use persistent binding to enforce LUN masking: only
99
140_SANs_03
100
8/14/01
1:35 PM
Page 100
Chapter 3 • SAN Components and Equipment
devices that have been persistently bound by hand or through software are allowed access into the system. Other implementations do not couple persistent binding and LUN masking, and instead automatically bind devices as they are discovered—as long as LUN masking allows those new devices to show up to the operating system.This allows for more flexibility, since devices do not manually have to be configured for masking settings.
Default LUN Access Permissions Default LUN access permissions are used by HBA software to determine whether a disk device should be mounted and accessible to a host operating system. HBA drivers can usually be configured to always allow access (automatic mapping of devices to an operating system), or to never allow access (manual mapping of devices to an operating system). For example, it is important for very large networks to never allow automatic access and require manual mapping of a device to the operating system. For example, you might have 20 LUNs exported from a storage array, but only the first two LUNs should be accessible to your host. If you set access permissions to default to allow automatic access, potentially all of those 20 LUNs would be claimed and probed by the operating system. By setting the default access to deny, only the intervention of an administrator will allow the operating system access to the disks (through LUN masking).This keeps hosts from trampling on the data already written to LUNs in a network. Typically, for very small networks the HBA drivers are configured to always allow access to new devices in the network. For large networks, it is a requirement to never allow access unless an administrator specifically grants access.
Upper-Level Protocol Access Permissions As HBAs have begun to add IP and VI capabilities to their cards, an important option that is beginning to appear is the control over IP and VI access permissions. Like LUN access permissions, this allows you to control which devices are allowed to receive IP or VI frames.The major use of this currently is to prevent IP or VI frames from being sent to hosts or storage that do not understand those protocols. In some cases, receiving these frames causes errors or software to crash on these storage devices, even though they should not recognize the frames and should ignore them.The ability to set these permissions adds the ability to control this functionality and prevent these types of errors.
140_SANs_03
8/14/01
1:35 PM
Page 101
SAN Components and Equipment • Chapter 3
Dynamic Versus Static Discovery Fibre Channel is a networking protocol, and unlike a parallel SCSI bus, devices in a network are not usually powered up and down at the same time. In fact, devices can be added or removed at any time, and the whole network continues to stay up. Older parallel SCSI devices used to require a reboot to discover any new storage devices attached to a host, because of the static nature of the parallel SCSI bus. Dynamic discovery of devices is the capability of an HBA to discover new devices on a network without rebooting.This allows the most flexibility, and by rescanning drives with operating system software a new storage volume can be added or removed without rebooting the system. Static discovery occurs when an HBA requires a reboot to discover new devices, which is still the case for some hardware. Understanding if your hardware supports dynamic or static discovery is important. If you want to run a Fibre Channel network, dynamic discovery is necessary for 24x7 network operation.
Configuration Management Software The suite of software available in most network adapters has usually been limited to very basic command-line utilities or a few simple configuration pages attached to the device driver. However, in the Fibre Channel HBA world, the features and capabilities of cards are more advanced than a few simple configuration settings. The advent of sophisticated management software has made it possible to not just change card settings, but also monitor what a Fibre Channel card sees in the network; monitor the status of connections; identify externally connected nodes; and run diagnostics. Figure 3.8 shows the TROIKA SAN Command utility, and Figure 3.9 shows an example of Emulex’s configuration utility. Different vendors provide varying amounts of configuration management software, ranging from simple command-line utilities to sophisticated GUI applications.
HBA API Support The HBA API is a C-level API supported by Fibre Channel HBA manufacturers to enable the collection and management of information available from HBAs. This API is used by SAN management software to collect information such as model numbers, vendor names, hardware and software version numbers, port speeds and settings, as well as ports attached to a Fibre Channel HBA. Support
101
140_SANs_03
102
8/14/01
1:35 PM
Page 102
Chapter 3 • SAN Components and Equipment
for the HBA API is widespread and is a requirement for managing your Fibre Channel HBAs through most management software. Figure 3.8 HBA Configuration Management Software (TROIKA SAN Command)
Figure 3.9 Emulex Configuration Software
140_SANs_03
8/14/01
1:35 PM
Page 103
SAN Components and Equipment • Chapter 3
The HBA API consists of two major parts: a shared, common library that is loaded onto your operating system and accessible to any applications, and a vendor-specific library that supports the HBA that you are loading onto your system.The common library allows applications to generically call specific functions from the HBA API and is dynamically linked with applications such as VERITAS SANPoint Control, SANavigator, and other SAN management software.This library is installed into a common location (such as C:/WINNT/ SYSTEM32 on Windows NT and \usr\lib on UNIX).The vendor-specific library is provided by the HBA manufacturer and is usually installed in a manufacturer’s install location. The HBA API has an advantage over the previous technique used for managing HBAs, which tended to be vendor-specific I/O Controls (IOCTLs).The API, which was developed as part of the efforts of the Storage Networking Industry Association (SNIA) to increase interoperability and manageability of Fibre Channel networks, has the advantage of running with any HBA vendor’s hardware across different operating systems and allowing different vendors’ hardware to be addressed in the same box.
Remote Boot across the SAN Remote boot is the capability of an operating system to use a Fibre Channel HBA to access and mount the boot volume for a system. Unlike parallel bus SCSI, volumes in a SAN are not limited to a local bus, and additional logic is necessary to boot an operating system from the network. A boot binding between a specific volume in the network—WWN and LUN—is required for remote boot to start. This choice is usually made through software accessible before boot and startup. Remote boot allows for an interesting use of the Fibre Channel SAN. Because boot volumes can be on a network and be available to practically any device connected to the SAN, they make it possible to dynamically change which physical hardware a system is booting up from by changing the remote boot binding. For example, you could have the operating system image of your Web server stored on the network on a disk, and if the hardware that was running that Web server fails, all you would need to do is reassign the boot volume to a new server.This makes it possible to easily reallocate functions of your servers across your network when hardware failure occurs, without requiring moving, reinstalling, restoring, or any other typically cumbersome and lengthy processes associated with local storage utilization.This is also used to enable
103
140_SANs_03
104
8/14/01
1:35 PM
Page 104
Chapter 3 • SAN Components and Equipment
advanced disaster tolerance, so that you can boot the image of a host if using different hardware. The difficulty of remote booting is that when you use network volumes to run your operating system, if there is a network error or failure in your network, you will not be able to access anything on the boot operating system. Errors will not be logged, and in many cases, everything will come to a grinding halt until the network error is fixed. A simple case of someone unplugging the Fibre Channel connection on the back of a machine could be catastrophic, versus the normal situation where an internal disk is unlikely to be removed without powering down a machine and opening the case.This situation can be remedied by using a dual-fabric configuration.
Hot-Plug Support With newer operating systems and hardware, the capability to swap failed equipment while your computer and operating system are still running has begun to become possible. Peripheral Component Interconnect (PCI) hot-plug systems allow the removal and insertion of hardware, even while I/O is occurring.This makes it possible to fix problems without taking down critical systems or rebooting systems. In redundant configurations, this ensures that operation can continue even through single hardware failures. The typical sequence of events in a hot-plug situation is this: an HBA fails or is otherwise showing signs of problems.Through system error logs, an administrator determines that a certain piece of hardware is at fault, and either indicates through software or through buttons or levers on the system that he or she would like to swap this HBA.This signals to driver software and to the hardware bus to stop I/O to a card and isolate the connections of that card from the remainder of the system. Usually, a light or other indicator will show that it is acceptable to remove the offending card, and the administrator will swap it with an identical piece of hardware. By pressing another button or lever, or through running software, the administrator tells the operating system that a new piece of hardware has been swapped in. Finally, that signals drivers and the operating system to start using that hardware again. This generally works great. However, there are several Fibre Channel-specific complications that need to be addressed with PCI hot-plug.These are Fibre Channel settings, storage LUN masking, and zoning settings, which are usually tied to a very specific piece of hardware, and which will not understand that the newly swapped card is identical to the other piece of hardware it replaces. Fibre Channel settings are the specific settings for how a piece of hardware should behave on the Fibre Channel network. Because there are different modes
140_SANs_03
8/14/01
1:35 PM
Page 105
SAN Components and Equipment • Chapter 3
of operation, and specific compatibility modes and settings needed to operate with different Fibre Channel hardware, these settings are important to maintain. Often, these settings are stored in Nonvolatile RAM (NVRAM), which is a physical piece of hardware attached to an HBA.When you swap the HBA, these values are no longer remembered, and instead will use the settings that are written into the new card.Two things need to be done in this case to properly set up these Fibre Channel settings. First, if possible, replacement cards and spares should be configured ahead of time identically to the running system. Different timeout values, modes of operation, and settings should be logged and set on all spare hardware so that they do not need to be reset on a hot-swap card. In addition, it is best to force settings in your driver software (typically done through registry entries or configuration management software in Windows systems and through .conf files in UNIX), so that the driver forces the setting no matter what is programmed into the card. Storage LUN masking is the capability of Fibre Channel storage to enforce a level of security on which devices are allowed to access specific volumes on the storage. High-end RAID arrays are typically the only pieces of hardware that contain this functionality. A piece of storage will contain settings within its software and memory that allow only specific devices (HBAs) access to certain LUNs.This is usually defined either with a port WWN or node WWN of the HBA that is installed into your host system. If you swap this HBA, the WWNs will change, because they are globally unique to that specific HBA.You will have to reconfigure your storage to accept the new port WWN or node WWN of the HBA that you have hot-swapped into a system, or no I/O operations will be allowed to the LUN.To some extent, you can avoid this by knowing in advance the port and node WWNs of the hot swap and preprogramming your storage to accept the hot swap’s WWNs. If this is not possible, you will have to ensure that as part of your hot-plug procedures that the reprogramming of your storage array is included. The last Fibre Channel-specific setting that is important in hot-plug situations is switch zoning. Switch zoning operates much like storage LUN masking, since it also relies on the WWNs programmed on an HBA to perform the zoning operation (unless you are using port zoning, in which case this does not apply).When you insert a new HBA, the WWNs programmed into different zones will no longer match, meaning that any storage that your host previously had access to will not be available if switch zoning is in effect.There are different things you can do to minimize the amount of work needed to do a hot-plug swap. One is to use an alias in your switch to define the host, so that it is easy to
105
140_SANs_03
106
8/14/01
1:35 PM
Page 106
Chapter 3 • SAN Components and Equipment
change the actual port or node WWN of the alias and have all other zone settings change automatically. Another is to do as in storage LUN masking, and preprogram, if possible, the WWNs of your spares into the proper zones so that they are automatically included as a part of the zone. Finally, some HBAs allow you to force a node WWN to apply to an entire set of HBAs in a system and use that node WWN as the zoning member, which makes it possible to swap in new HBAs without requiring reconfiguration of your switch zoning. If you use this method, perform all zoning by using the node WWN of each HBA, rather than the port WWN.
Connecting Legacy Devices into Your SAN When Fibre Channel networks were first put into operation, many storage devices were not able to natively communicate using the Fibre Channel protocol. However, the parallel SCSI bus was widely supported and many different kinds of devices were implemented with parallel SCSI support. Because Fibre Channel provides the same SCSI protocol over a different kind of medium, devices called Fibre Channel-to-SCSI routers were developed to translate Fibre Channel frames to the appropriate parallel SCSI commands.These routers include one or more Fibre Channel ports on one side, and one or more parallel SCSI bus connections on the other. Devices on the parallel SCSI bus side are presented to the Fibre Channel network as any other Fibre Channel native storage devices, as LUNs available from a Fibre Channel port. Fibre Channel routers make it possible to use legacy parallel SCSI devices by simply plugging a box into the network on one side and the legacy parallel bus on the other side. In particular, tape libraries have relied on Fibre Channel routers to help enable direct backup of storage on the SAN.
Basic Features of Routers Fibre Channel routers should more accurately be called bridges, as they bridge legacy SCSI devices and Fibre Channel, translating Fibre Channel SCSI-FCP transactions and parallel SCSI bus transactions. A Fibre Channel router plugs into the Fibre Channel network on one side and a SCSI bus on the other.To the SCSI bus, a router looks like an initiator such as a host. It issues SCSI commands such as resets and inquiries, and determines what SCSI devices exist on the SCSI bus.To the Fibre Channel network, a router looks like any other storage node on
140_SANs_03
8/14/01
1:35 PM
Page 107
SAN Components and Equipment • Chapter 3
the network. Presenting each of the SCSI devices it has found on the SCSI bus as a LUN connected to a port, the router takes any frames sent to the SCSI devices and translates them from SCSI-FCP to parallel bus SCSI. Likewise, any responses received from the SCSI devices are changed to SCSI-FCP, tagged with network headers, and sent back to the initiating device. Routers support any type of parallel SCSI device and are used for everything from legacy RAID arrays and disk storage to non-Fibre Channel-capable tape drives. Most routers act as Class 3 devices, because they are geared toward the transport of storage SCSI traffic (which operates best with Class 3).They also tend to support loop-only operation, rather than full Fibre Channel fabric protocols, due to the fact that they are target and not initiating devices. In addition, these routers support additional levels of error recovery on top of normal SCSI error recovery, including all error recovery in FCP and newer FCP-2 error recovery procedures.The capabilities of Fibre Channel-to-SCSI routers are as follows: ■
Number of SCSI buses
■
Types of SCSI buses
■
Internal or external SCSI termination
■
Selective LUN presentation
■
Extended copy support
■
SNMP
■
Telnet
■
Ethernet ports
■
Serial ports
Number of SCSI Buses The most basic routers have one SCSI bus and one Fibre Channel port connected to the Fibre Channel network. More capable routers have more than one bus and multiple Fibre Channel ports.The advantages of having more than one bus include available bandwidth and isolation of error conditions on a bus.With high-bandwidth SCSI devices and the architecture of parallel (shared bus) SCSI, speeds are limited by the amount of available bandwidth that can be shared on a SCSI bus. By providing more than a single SCSI bus, routers with multiple SCSI ports allow you to reduce the contention for resources on those buses for high-speed devices.
107
140_SANs_03
108
8/14/01
1:35 PM
Page 108
Chapter 3 • SAN Components and Equipment
Types of SCSI Ports,Termination Another consideration in routers is the types of SCSI ports available.With different kinds of connections available for SCSI, you need to make sure that the type of SCSI ports available on your router matches the device you are running. The different types of SCSI ports that might be supported by your device range from narrow, single-ended SCSI to fast, wide, ultra-wide, ultra2, and Low Voltage Differential (LVD) devices. One area to look at is the type of SCSI termination used by your router. Parallel SCSI termination can be either internal or external, depending on the manufacturer. In the case of internal termination, flipping a switch can turn on termination, enabling the device to be the end of a SCSI chain.With external termination, you either need to buy an external terminator or the device needs to be in the middle and not at the end of a SCSI chain. Internal termination makes it easier to manage this aspect of parallel SCSI operation and also means one less component you need to track.
Selective LUN Presentation More advanced routers provide the capability to selectively filter which hosts are allowed to access a specific SCSI target. Like HBA-based LUN masking and switch zoning, this can be used to help enforce which devices are allowed to access a specific storage volume.To configure this LUN presentation, management software is used to specify which hosts are allowable for specific LUNs.This is usually specified with a port WWN of the HBA. Selective LUN presentation can be used to limit access to a specific SCSI device on one side of a router to only one Fibre Channel device, such as to allow only a backup server access to your tape drive. It can also be used to partition data between hosts, in order to assign different SCSI targets to different hosts.
Extended Copy Support Extended copy support is the capability of a Fibre Channel router to support the Extended SCSI Copy command, which is used for server-free backup on your SAN. In conjunction with special Fibre Channel SAN-aware backup software, routers help form a part of a solution that can offload the backup traffic from the hosts on your network to the Fibre Channel router, enabling storage to be backed up directly to a tape drive connected to the router instead of requiring the intervention of different hosts in the network.
140_SANs_03
8/14/01
1:35 PM
Page 109
SAN Components and Equipment • Chapter 3
Extended copy appears as a feature of a Fibre Channel router, which interprets SCSI commands received from backup software to fetch and store data from storage arrays.
Management Interfaces Like switches, routers also support different management interfaces, including serial ports, Ethernet ports, SNMP, FTP,Web interfaces, and also emerging standards like Sun’s Jiro management standard. Most Fibre Channel routers also support the FibreAlliance MIB through their SNMP interface, which allows most management software to derive basic information from the equipment.
Bridging and Routing to IP Networks and Beyond As Fibre Channel expands from just a few machines and storage to much larger networks, interoperability and connectivity are becoming much more important. Routers and bridges are starting to be used to take Fibre Channel traffic and send it across regional and wide area networks, as well as for remote backup and IP traffic.
Fibre Channel to DWDM Fibre Channel-to-DWDM technology is beginning to be used to help extend the distances of SAN operation. Fibre Channel-to-DWDM equipment operates by multiplexing Fibre Channel optical signals onto a higher wavelength fiber. Typically, a single switch E_Port is connected to DWDM equipment, which multiplexes the Fibre Channel signal to a remote DWDM port. On the other side, a DWDM multiplexer is connected to a remote SAN where the frame is sent out through a switch E_Port back into the network.To the Fibre Channel switch, the existence of a DWDM link is almost transparent—no change occurs to the protocol, information is received at full speed, and there is no indication that a frame went over a long-distance link. However, because of distance and the speed of packets of light, switches do need to be configured for more available frame buffers between the two switch links connected by the DWDM equipment. Fibre Channel-to-DWDM equipment is used primarily for regional transport, because of the distance limitation of Metropolitan Area Networks (MANs). Using DWDM has many advantages for Fibre Channel.The first is extending the distance limitation of Fibre Channel beyond 10 km. DWDM technology can
109
140_SANs_03
110
8/14/01
1:35 PM
Page 110
Chapter 3 • SAN Components and Equipment
extend Fibre Channel to MAN distances up to 100 km. Second is the ability to multiplex a large number of Fibre Channel connections to fewer fiber links, reducing the amount of optical fiber needed between facilities and simplifying long-distance cabling.
Fibre Channel across IP Networks Developing standards allow for the encapsulation of Fibre Channel onto an IP frame (FC_IP) for transport across any IP-capable network. Like Fibre Channel across DWDM, this is being targeted at extending the distance of SANs across regional and wide area distances.The FC_IP protocol encapsulates Fibre Channel frames within IP packets.This works through special equipment that connects into a Fibre Channel network and encapsulates Fibre Channel frames and transmits them on IP networks.This allows any IP-capable network—including Gigabit Ethernet, ATM, and any other technology—to transport Fibre Channel frames. Transporting Fibre Channel over IP networks can help extend SANs well beyond the campus networks and regional networks now in use.With suitable bandwidth available,WAN distances can be possible for SANs with this emerging technology. The major limitation of running FC_IP is the speed and latency of the IP network.The IP network must be able to handle the amount of bandwidth generated by Fibre Channel, or otherwise risk being bottlenecked in the IP transport. In addition, different vendors are not compatible with each other using FC_IP, since the standard is still in a draft phase. However, as the standard progresses, expect that vendors will begin to interoperate.
IP over Fibre Channel to Gigabit Ethernet Fibre Channel to Gigabit Ethernet routers allow for the transport of IP frames generated on either the Fibre Channel network or Gigabit Ethernet network to appear transparently on the other network.This can be done through dedicated hardware, or even through a host computer set up to route IP frames between different network interfaces. Standard IP protocol services such as Web pages, FTP, and telnet can be seamlessly run across your Fibre Channel network and routed across to an IP network like Gigabit Ethernet. This is now usually done primarily as a way to retrieve management information from network equipment through in-band management. In particular, with expensive DWDM or remote links being used to connect remote sites,
140_SANs_03
8/14/01
1:35 PM
Page 111
SAN Components and Equipment • Chapter 3
using IPFC and bridging that information to other general-purpose networks makes it more cost-efficient to manage the remote equipment. However, as IPFC becomes more prevalent, these bridges and routers can make it possible to merge traffic between both IP and Fibre Channel networks.
Fibre Channel Storage Selecting Fibre Channel storage depends on what kinds of applications and, most importantly, how much data and protection you need. Fibre Channel storage ranges from individual disk drives that support the Fibre Channel protocol, to devices with dozens of connections and different kinds of storage connection ports, such as Fibre Channel, SCSI, and Enterprise System Connection (ESCON).This section attempts to briefly describe the Fibre Channel aspects of this storage, but does not cover the very complex task of evaluating storage systems.
Individual Disk Drives and JBODs Individual disk drives, although they support the Fibre Channel protocol, are rarely deployed alone in a SAN. In general, these individual disk drives are added to JBOD enclosures that hold four or more single disks in a single, loop-ready configuration.These drives are wired together into a miniature FC-AL loop with one or two ports to connect to the drives.The first port of a JBOD is generally wired to one of the dual ports on a disk drive, and the secondary port of the disk drives is wired on a secondary loop.To an HBA and system on the other side, there is no difference between a disk drive and a JBOD. In fact, you cannot tell that individual disks are tied together into a JBOD electronically or through software. Different JBOD systems differ in the number of ports, physical enclosure features, and individual disk drives used in the systems. Most JBOD systems do not add any actual Fibre Channel functionality and just physically connect the internal Fibre Channel disk drives to the Fibre Channel network. Differences in JBODs include the number of disk drives included in an enclosure; rack-mount and standalone options; amount of cooling and the power supplies; temperature sensors and alarms; and the ability to hot-swap and replace faulty components. Fibre Channel RAID systems can also be connected into a Fibre Channel network with one or more ports, and range from low-end systems with only several gigabytes of capacity and little cache to higher end, hundreds of gigabyte capacity arrays with extensive cache.They provide varying levels of redundancy
111
140_SANs_03
112
8/14/01
1:35 PM
Page 112
Chapter 3 • SAN Components and Equipment
and performance, and generally can be configured into different types of RAID levels for differing protection. An important Fibre Channel consideration when you are selecting a RAID system is the ease of configuration of the RAID. For example, some systems might require specific operating systems and HBAs in order for configuration software to work.You should make sure that the RAID system you select supports the other components in your system and has been qualified with switches in your system.The capability to configure a RAID system from the serial port or from an Ethernet port can ease management of that system. RAID systems also are available in different rack-mount and standalone configurations and provide redundant cooling and power component options.
RAID Levels RAID arrays generally support different levels of protection and redundancy for your data. By selecting different RAID levels, you can trade off speed of operation and recovery against the amount of protection for your data. The following is a brief description of the most common RAID levels available: ■
RAID 0: Striping Provides very rapid access to data by striping information across different disks. By distributing data across different spindles (disks), the data can be retrieved very rapidly and at a rate greater than an individual disk drive can support. However, RAID 0 provides no redundancy or protection for data if any disks fail.
■
RAID 1: Mirroring Duplicates data across disks on a one-toone basis. This provides 100 percent protection of data across disks and instant access to data if one disk fails. A duplicate copy of all data is simultaneously written to a mirrored disk, which is made available if the primary disk fails. However, this is fairly expensive since it requires purchasing twice as much capacity.
■
RAID 3: Striping with Parity Data is striped as in RAID 0, but with the addition of a parity disk. This provides speed and fault tolerance.
■
RAID 5: Striping with Distributed Parity Similar to RAID 3, but parity is instead distributed across all disks, allowing for better read performance and fault tolerance.
140_SANs_03
8/14/01
1:35 PM
Page 113
SAN Components and Equipment • Chapter 3
High-End Storage Arrays High-end storage arrays generally support multiple terabytes of data and often include the capability to support Fibre Channel connections as well as interfaces such as SCSI, FICON, and other specialized storage interconnects. Built to support dozens or even hundreds of storage users, the arrays can range from refrigerator-sized to half a room. In addition to many storage connections, these devices provide large amounts of memory cache to accelerate disk accesses, as well as advanced capabilities like LUN masking through selective LUN presentation, snapshot backup volumes, redundant controllers, and failover and replication capabilities.
Selective LUN Presentation Selective LUN presentation is the capability of a storage device to filter or mask which hosts are allowed to see a LUN. For example, storage can be configured to show LUN A to host X and LUN B to host Y, but not vice versa, making it possible to partition the storage entirely by hosts on the network.This has many advantages, including the capability to allocate the storage in the box within the network at a single interface, to guarantee that users do not accidentally mount an incorrect volume and corrupt data, and to better control how hosts see the storage in the network. Selective LUN presentation works through hardware and software that examines frames coming into a storage subsystem from the network. By examining the frames and comparing the source of those frames with an administratorconfigured list of allowable hosts, the storage array can allow or deny access to specific LUNs in the array. In addition, arrays can also enforce LUN numbers that a host sees, even if those are not the actual physical LUN numbers of the internal volumes.This is typically used for operating systems like Windows NT, which requires that every storage volume have a LUN 0—which is fine for simple RAID arrays but impractical on a storage array that might have a hundred LUNs allocated by dozens of hosts.Through selective LUN presentation, every host can see a different LUN 0, which is physically different but can be addressed through the same LUN 0 address.
LUN Export across Multiple Ports With Fibre Channel, the capability to make data highly available through a network has resulted in software support for highly available transport of data from
113
140_SANs_03
114
8/14/01
1:35 PM
Page 114
Chapter 3 • SAN Components and Equipment
storage.This software can identify volumes that are the same, even if they are seen at different points of a network. High-end storage arrays have a feature that allows a single logical LUN to be exported across multiple Fibre Channel ports on an array. For example, a single volume for an e-mail database can be exported across two different storage ports, across two separate fabrics. If a storage port’s hardware fails or a switch fabric is disrupted, the other port can still be accessed across a different path.This ability to export LUNs across multiple ports enables software and dynamic multipathing software and drivers to intelligently access, either on an active-passive or activeactive basis, the same LUN. Array management software typically must be configured to allow for this multiple LUN export. Exporting a LUN across multiple interfaces without the addition of dynamic multipathing or volume management software can result in data corruption or collisions due to duplicates in the operating system image. In addition, the arrays need to identify that these different exported LUNs are actually and logically the exact same volume images.This is referred to as Page 83h information, named after the SCSI mode page that describes the identification of storage volumes on a storage device such as a RAID.
Snapshot Backup Volumes Snapshot backup volumes are a special feature of high-end arrays that take a “snapshot” of an operational LUN’s data at a point in time and copy that data to another volume.This snapshot, which is made instantaneously while traffic is running and sometimes in coordination with a halt in traffic to a storage LUN, enables an easy way to back up a very busy storage array. Because high-end storage arrays are typically highly utilized and attached to business-critical systems that must be available 24x7, backup is a very challenging task. Usually, backup has to be done to a static system, or a system that is not being written to during the entirety of the backup.With these critical systems, this never happens—and backups still have to be done. In fact, backups are probably even more important for these systems. Snapshot backup volumes solve this problem by providing a copy of the data at a point in time, which can be backed up without the problem of data constantly being changed on a volume.These backup volumes are exported to different hosts or backup hardware on the SAN and at predetermined periods will refresh their information from the “live” data and allow for the next backup.
140_SANs_03
8/14/01
1:35 PM
Page 115
SAN Components and Equipment • Chapter 3
Summary The keys to robust Fibre Channel deployment are the components that you use to build your network. By understanding these components, you can better design a robust, scalable network. Understanding the features and capabilities of your hardware will help you select and qualify the best equipment for your needs. The most basic layer of your SAN is the physical layer, which includes your media and connector choices.There are a number of choices that are dependant on the primary purpose of your SAN. In selecting the type of media to use for your SAN, you have two choices: copper and optical.The distinct advantage of copper is that it is inexpensive compared to all types of optical fiber.The advantage of optical fiber is that it provides a reliable signal over a longer distance than copper.There are two types of fiber: single mode and multimode.The difference in the fibers is the diameter of the fiber. Smaller-diameter single mode can transmit at a far greater distance. The cardinal rule for connecting a SAN is to minimize the total number of connections and patches. SC is the standard optical connector, but high-density connectors are becoming more popular as more devices are connected to SANs. Examples of high-density connectors are LC, MT-RJ, and HSSDC. HSSDC and DB-9 are the copper connectors available. In most SANs, you will need to use some types of GBICs. GBICs provide an easy way to adapt devices to whatever connection type you prefer.This lets you customize your SAN based on distance and speeds required between devices. Hubs serve as the most rudimentary support for Fibre Channel, providing basic FC-AL connectivity to smaller networks. Simple hubs provide only basic electrical connectivity, shared bandwidth, and no intelligence. Intelligent hubs provide much better error recovery and management, especially in multi-initiator fabrics. Key features to look at in hubs are the different management interfaces available, error recovery, and LIP isolation. Switches form the core of a switched Fibre Channel fabric, and not only switch frames between nodes but also contain intelligent services to locate and manage nodes in the fabric. In addition, the capability to work in loop mode also makes Fibre Channel switches a good replacement for hub deployments. Performance and functionality such as zoning and the ability to cascade switches together make switches a key to controlling access in the SAN and scaling your installations. HBAs connect hosts to the Fibre Channel network. In conjunction with device drivers, these devices translate SCSI commands and protocols from operating
115
140_SANs_03
116
8/14/01
1:35 PM
Page 116
Chapter 3 • SAN Components and Equipment
systems and send them across the Fibre Channel network. Advanced functionality such as persistent binding and LUN masking add to your ability to control how storage is allocated in your network. HBAs are also capable of non-SCSI traffic such as IP frames and VI clustering protocols, which are enabling new applications for your SAN. Fibre Channel-to-SCSI routers provide an important bridge from legacy parallel bus SCSI devices to new Fibre Channel networks. By translating between Fibre Channel SCSI and older parallel bus SCSI protocols, older non-Fibre Channel SCSI devices can seamlessly be used in your network. Advanced features such as selective LUN presentation and support for extended copy can also help support applications on your SAN. Companies are starting to extend their SAN infrastructure over long distances, adding disaster tolerance and remote backup to their data centers.Technologies such as Fibre Channel-to-DWDM equipment and encapsulation of Fibre Channel over IP networks are gaining ground as a way to extend SANs to MANs and even WANs. Bridging IP from Fibre Channel is also helping to drive the interoperability between standard corporate IP networks and specialized data SANs. Fibre Channel-capable storage forms the core of a SAN and is present as low-cost JBOD disk drive cabinets, midrange RAID arrays, and high-end storage subsystems. JBOD arrays present individual Fibre Channel-capable disks drives to the network. RAID arrays add error tolerance, cache, and management ability to the network, with high-end storage subsystems providing much higher functionality and capacity to Fibre Channel and other storage interconnects. High-end features such as selective LUN presentation, multiple export of LUNs, and snapshot backup are some of the features that can help better manage the allocation of storage on your SAN, add redundancy to your networks, and help to keep your data available 24x7. These components together form the SAN. By studying them and understanding their major features and benefits, you can better select components and design your network.
Solutions Fast Track Overview of Fibre Channel Equipment ; Understanding the features of your Fibre Channel equipment is key
when building a robust infrastructure.
140_SANs_03
8/14/01
1:35 PM
Page 117
SAN Components and Equipment • Chapter 3
; A Fibre Channel network is comprised of cabling, GBICs, hubs,
switches, HBAs, and routers. ; Fibre Channel shares much of the same terminology as Ethernet
networking, but the functionality of similarly named equipment is not necessarily identical.
Cabling and GBICs ; Copper cabling is almost always terminated with either an HSSDC or
DB-9 male connector. ; Multimode optical fiber is terminated using a variety of optical
connectors, including SC, LC, and MT-RJ. ; Single-mode fiber is the most expensive media type, but preferable for
long distances. ; Single-mode fiber, because of its small diameter (9 µm), has the highest
transmission speed potential. ; Copper cabling is available in two types: active and passive. Active copper
lines provide twice the distance of passive copper lines. ; The HSSDC connector was specifically designed as a Gigabit copper
connector, improving density and performance over the DB-9 style connector. ; GBICs are removable transceivers used in all types of Fibre Channel
devices, including switches, hubs, and HBAs. ; GBICs offer the option of interfacing with almost all types
of connectors. ; A Media Interface Adapter (MIA) converts DB-9 copper connectors to
optical SC connectors.
Using Hubs ; Hubs serve as a very basic level for connecting different ports in a
network together. ; Hubs can connect up to 127 devices together in an FC-AL loop.
117
140_SANs_03
118
8/14/01
1:35 PM
Page 118
Chapter 3 • SAN Components and Equipment
; Simple hubs contain no intelligence, just electrical connections. ; Managed hubs provide a level of error tolerance and
management features. ; Managed hubs provide LIP isolation, automatic port bypass, signal
retiming, and management interfaces. ; Fibre Channel LIPs can be a major source of problems in arbitrated loop
configurations. ; To avoid an earlier generation of problems due to loop architecture,
most people are moving to switched fabric devices instead.
Using Switches and Fibre Channel Fabrics ; Switches are classified into three categories: entry-level, scalable fabric,
and core fabric switches. ; Entry-level switches are focused on small workgroups of 8 to 16 ports,
usually are geared toward low cost, and deliver limited scalability and management. Fabric switches provide the capability to cascade switches together to create larger fabrics. ; A core fabric switch is designed for interconnecting multiple edge
switches to form multihundred-port SANs. ; HBAs are used to connect servers to the network.They map SCSI
commands in the operating system to Fibre Channel frames on the network. HBAs range from low-end, loop-only devices to high-end, fabric multipathing adapters. ; Major protocols supported by HBAs are SCSI-FCP for storage, IPFC for
networking, and VIFC for clustering. ; HBAs either support 1 Gbit/sec or 2 Gbit/sec speeds, with current gen-
eration cards supporting 1 Gbit/sec, and emerging cards supporting both. ; HBAs can be found in single one-port configurations or dual-port
adapters for higher density. ; LUN masking enables control of access to devices in the network from
the HBA.
140_SANs_03
8/14/01
1:35 PM
Page 119
SAN Components and Equipment • Chapter 3
; Persistent binding is the mapping of a Fibre Channel device into an
operating system at a specific device location. ; Dynamic discovery is the capability to dynamically add and remove
drives from your system without reboot. ; HBA API support is an important feature that allows management of
your HBA by SAN management software. ; Remote booting is the use of an HBA to boot an operating system
image across the SAN and is used to dynamically change hosts and enable ease of disaster recovery.
Connecting Legacy Devices into Your SAN ; Α Fibre Channel router, which is also known as a bridge, allows legacy
parallel SCSI devices to attach to your Fibre Channel SAN. ; A Fibre Channel router plugs into Fibre Channel on one side and
a SCSI bus on the other. ; Frames are translated from SCSI-FCP to parallel SCSI bus signals
through routers. ; Routers provide many different features, including different numbers
of SCSI buses and different support for parallel SCSI protocols and termination. ; Advanced features include selective LUN presentation, extended copy
support, and various management interfaces. ; Selective LUN presentation is the capability of a router to mask the
presence of devices to different hosts in the network and allow for better security and control over resources. ; Extended copy support (third-party copy) allows software to directly
back up data on the SAN, saving CPU and network traffic. ; Available management interfaces include telnet, SNMP, Ethernet, and
serial ports.
119
140_SANs_03
120
8/14/01
1:35 PM
Page 120
Chapter 3 • SAN Components and Equipment
Bridging and Routing to IP Networks and Beyond ; Fibre Channel-to-DWDM technology multiplexes Fibre Channel signals
onto higher bandwidth fiber for transmission over MAN distances (up to 100 km). ; Use of DWDM is transparent to Fibre Channel switches, except for
buffer settings. ; It is necessary to increase buffer credit settings to handle the long
distances/delays involved in MANs. ; Fibre Channel can also be transported across IP networks like ATM and
Gigabit Ethernet. ; FC_IP (not to be confused with IPFC) encapsulates Fibre Channel
frames in the IP protocol and can be used for remote backup and extending SAN distances.
Fibre Channel Storage ; Fibre Channel storage is important as the core of the data storage on
your network. ; Fibre Channel storage ranges from simple JBOD devices to multi-
terabyte storage arrays. ; A JBOD is a cabinet of independent disks, all connected into the Fibre
Channel network in a loop. ; Hosts individually address disks in a JBOD. ; RAID arrays provide additional protection and performance to
your storage. ; Different RAID levels are appropriate for different applications. ; High-end storage arrays add support for multiple terabytes of data.
Other types of connections include parallel SCSI, ESCON, and FICON. ; High-end arrays also generally include a large amount of cache, which is
used to speed up data access.
140_SANs_03
8/14/01
1:35 PM
Page 121
SAN Components and Equipment • Chapter 3
; Selective LUN presentation is the ability of high-end storage to control
access by hosts to data and to ensure data integrity. ; LUN export across multiple ports is used for redundancy and high avail-
ability, but requires dynamic multipathing software or drivers to work. ; Snapshot backup volumes are used to enable backup on live databases
and data images.
Frequently Asked Questions The following Frequently Asked Questions, answered by the authors of this book, are designed to both measure your understanding of the concepts presented in this chapter and to assist you with real-life implementation of these concepts. To have your questions about this chapter answered by the author, browse to www.syngress.com/solutions and click on the “Ask the Author” form.
Q: Should I buy a switch that supports GBICs or fixed media? A: This depends on your application. GBICs offer much more flexibility in the configuration and media in your network, allowing you to mix copper and optical media and different copper and optical connectors. However, GBICs do add some cost to your installation and are another possible point of failure. Using fixed media generally lowers the cost of your switches and is more reliable. However, a failure of a transceiver in fixed media means a swap of the entire switch and not just a GBIC.
Q: I notice that many Fibre Channel devices support a common SNMP MIB. How do I get this to work with my network management software?
A: You can get MIB files from the manufacturer of your hardware, usually from a distribution disk or through a download from their Web site. Using an MIB browser in your network management software, you can browse this data. However, a better way to use this is through Fibre Channel management software that understands how to interpret these MIBs and show (for example) a visual topology of the network from this data.
121
140_SANs_03
122
8/14/01
1:35 PM
Page 122
Chapter 3 • SAN Components and Equipment
Q: Should I be buying 1 Gbit/sec or 2 Gbit/sec components for my network? A: You should consider the bandwidth requirements of your application. 1 Gbit/sec networks are the most proven and mature technology. However, as 2 Gbit/sec equipment starts to become available, you can expect to see all parts of the network (storage, switches, and HBAs) supporting 2 Gbit/sec standards.The higher bandwidth will provide significant performance improvements when used on ISLs, though.
Q: It looks like most Fibre Channel components provide some sort of masking ability to control access to either ports or LUNs.Which one of these methods should I use in my network?
A: Even though different components all offer some sort of zoning, LUN masking, or LUN presentation techniques, you will probably need to use all of them in one way or another to control your network. For example, switch zoning is invaluable for isolating specific SAN segments and control, but you will have to use either HBA-based LUN masking or storage-based LUN presentation to more finely partition your volumes.
140_SANs_04
8/14/01
3:34 PM
Page 123
Chapter 4
Overview of Brocade SilkWorm Switches and Features
Solutions in this chapter: ■
Selecting the Right Switch
■
Understanding the Brocade Fabric OS
■
Using Optional Brocade Features
■
Future Capabilities in the Brocade Intelligent Fabric Services Architecture
Summary Solutions Fast Track Frequently Asked Questions
123
140_SANs_04
124
8/14/01
3:34 PM
Page 124
Chapter 4 • Overview of Brocade SilkWorm Switches and Features
Introduction This chapter covers the Brocade switch models and describes their differences and similarities. It also describes how to select the most effective switch for specific requirements or SAN applications, a critical part of the overall process of implementing and managing a SAN. A discussion of the Brocade Fabric OS helps to explain the core aspects of switch functionality, and how that functionality relates to your own installation. In addition, this chapter gives you an overview of the features of Fabric OS and how they control the behavior of Brocade switched-fabric SANs. Finally, a discussion about the future capabilities of SANs assists you with developing your SAN vision.The information in this chapter presents an overview of the Brocade product family to provide context for the other chapters in the book and to enable this book to be a self-contained reference on Brocade SANs. For the latest and most detailed information about Brocade products, visit the Brocade Web site at www.brocade.com.
Selecting the Right Switch By choosing among the wide variety of switches available today, you can deploy them in simple, standalone configurations or by networking them with other switches to build larger or more resilient fabrics. Selecting the right switch for your application is extremely important, since the switch is the building block of the infrastructure for your SAN.The right SAN infrastructure can significantly improve information management and allow you to address some of your most challenging business requirements. As with other types of technology implementations, selecting a switch is a strategic decision.You must understand your current IT infrastructure and requirements, as well as how you might need the switch to fit into your strategic direction. As a result, you need to understand key factors such as scalability, compatibility, and interoperability with other hardware resources. After all, your SAN is hardly a static platform, and you should expect to expand it or integrate it with other components as new technologies emerge.This is how a SAN provides you with investment protection by enabling you to scale and adapt it to meet future requirements. At a very high level, you should be aware of a few key characteristics when selecting a switch, including hardware redundancy features, which reflect the level of reliability and availability you require; port count, or the overall size of your installation and the number of devices you need to connect; function, that is, the capabilities you need for certain types of applications or environments; and finally, cost, in planning the overall budget to achieve your SAN goals.You can use these
140_SANs_04
8/14/01
3:34 PM
Page 125
Overview of Brocade SilkWorm Switches and Features • Chapter 4
characteristics to differentiate the various members of the Brocade SilkWorm family of switches: ■
Hardware Redundancy The low-cost SilkWorm 2000/2200 entrylevel models have a single fixed power supply and cooling mechanism. The enterprise-class SilkWorm 2400/2800 products have dual hotswappable power supplies and hot-swappable cooling fans. All except the lowest-end SilkWorm 2000 series of switches offer pluggable Gigabit Interface Converters (GBICs) to enable fast replacement of optical transceivers.
■
Port Count Brocade offers 8- and 16-port switches, a 64-port integrated fabric and a dual 64-port core fabric switch (in the future).You can use these switches alone or together to form fabrics consisting of hundreds of ports.
■
Function The Brocade Fabric OS software installed on all Brocade switches is inherently capable of supporting all key features. However, software license keys might be required to activate some of these features. Each switch comes bundled with certain software keys, and if you need features that are not bundled with the product, you will need to budget for the additional licenses.
■
Cost Brocade has a wide range of switch models, from low-cost entrylevel departmental switches all the way up to robust, enterprise-class fabric switches, integrated fabrics, and core fabric switches.
One of the first steps in selecting a switch is to decide how many servers and storage devices you want to connect.There are some general guidelines you can follow when deciding what type of switch best meets your needs. For networks of less than eight devices with no growth plans, an 8-port switch will suffice. If you plan to have more than eight devices at some point in time, you will probably want a 16-port switch. If you have more than 16 devices, you will want a network built from a number of 16-port switches. If you have over 50 devices, you will want to either build a network from 16-port switches or consider a more advanced solution such as the SilkWorm 6400 Integrated Fabric, which provides a preconfigured network in a convenient package. If you have business-critical applications that must be continuously available, you will need to connect them in a resilient manner.You might even want to use a dual-fabric configuration.We discuss this in more detail in Chapter 5, “The SAN Design Process,” and Chapter 7, “Developing a SAN Architecture.” Since
125
140_SANs_04
126
8/14/01
3:34 PM
Page 126
Chapter 4 • Overview of Brocade SilkWorm Switches and Features
redundancy lies within the fabric, it might be possible to utilize the less expensive switches with a single fixed power supply. If one of the switches fails, multipathing software on the host and storage array will route traffic over the redundant fabric until the failed switch is replaced. For applications that can tolerate occasional downtime for periodic maintenance, a single fabric is less costly and can be built upon switches with redundant, hot-swappable power supplies and fans—so the fabric can withstand the loss of one of the components without downtime.We discuss the topics of redundancy and resilience in detail in Chapter 7. Most Brocade switches support hot-pluggable GBICs that provide both flexibility—you can choose optical or copper media—and high availability. If a single GBIC fails, you can replace it without disturbing the other ports on the switch. However, the lowest priced 8-port entry model supports one GBIC and seven fixed optical ports. The Brocade family of fabric switches includes a wide variety of cost options, ranging from the least expensive 8-port SilkWorm 2010 switch with fixed media to the highly available 16-port SilkWorm 2800 and the redundantly configured SilkWorm 6400 Integrated Fabric.The high end of the family is the SilkWorm 12000 Core Fabric Switch, which will provide 128 ports in a single enclosure when it is released. All Brocade switches are fully interoperable with each other and can be mixed in fabrics to provide the optimal balance of cost-effectiveness, expansion capability, and high availability.
Entry-Level Switches Brocade entry-level switches are designed for environments with smaller storage growth requirements where cost considerations outweigh the need for high availability.The 8-port SilkWorm 2010, 2040, and 2050 are the same physical 1Uhigh switch with different feature sets enabled (Figure 4.1).The 16-port SilkWorm 2210, 2240, and 2250 are the same physical 1.5U-high switch with different features enabled (Figure 4.2). Figure 4.1 SilkWorm 2000 Series Switch
140_SANs_04
8/14/01
3:34 PM
Page 127
Overview of Brocade SilkWorm Switches and Features • Chapter 4
Figure 4.2 SilkWorm 2200 Series Switch
SilkWorm 2010 (8 Ports) and 2210 (16 Ports) The SilkWorm 2x10 are arbitrated loop-only switches and are a viable alternative to a managed hub-based solution.The SilkWorm 2010 and 2210 series deliver superior performance compared to a hub by providing simultaneous, full-bandwidth data transfers and enhanced availability through fault isolation and intelligent Loop Initialization Primitive (LIP) management.The SilkWorm 2010 features seven fixed optical ports and one pluggable GBIC slot, while the 2210 offers 16 pluggable GBIC slots.The fixed media solution in the SilkWorm 2010 reduces cost by eliminating the need to purchase GBICs for the remaining slots. A consideration to this configuration is that, if one fixed optical component fails, you need to replace the entire switch rather than a single component. The SilkWorm 2x10 are bundled with Brocade Zoning, Brocade WEB TOOLS, QuickLoop, and the Simple Name Server (single switch) enabled.The 2x10 can be upgraded to a full-fabric switch with a license key.This capability is key to investment protection: you might need only a loop-switch now but decide in the future that you need fabric capabilities as well.With the 2x10, you do not need to perform a forklift upgrade of your SAN in order to add that functionality.
SilkWorm 2040 (8 Ports) and 2240 (16 Ports) The SilkWorm 2040 and 2240 are the same physical switches as the SilkWorm 2010 and 2210, respectively, with additional support for entry-level fabrics.You can use these switches in dual-switch configurations to support fabrics or loop environments of up to 30 ports. Alternatively, you can have one E_Port connection per switch to connect the switch to a larger Brocade fabric. The SilkWorm 2040 is bundled with Brocade Zoning, Brocade WEB TOOLS, and the Distributed Name Server enabled. It can be upgraded to a full-fabric switch (SilkWorm 2x50) with a license key. However, you must then also purchase Brocade QuickLoop to connect Fibre-Channel Arbitrated Loop (FC-AL) hosts.
127
140_SANs_04
128
8/14/01
3:34 PM
Page 128
Chapter 4 • Overview of Brocade SilkWorm Switches and Features
SilkWorm 2050 (8 Ports) and 2250 (16 Ports) The SilkWorm 2050 and 2250 are the same physical switches as the SilkWorm 2010 and 2210, respectively, but they feature a full-fabric configuration that enables multiple E_Port connections and networked fabrics of multiple switches (equivalent to what is provided by the SilkWorm 2400 and 2800 switches).This design enables redundant paths between switches. In a networked configuration, any device can connect to any other device, with I/O taking the shortest available path to the target device.The SilkWorm 2x50 is bundled with Brocade Zoning, Brocade WEB TOOLS, and the Distributed Name Server enabled. However, optional Brocade QuickLoop software is necessary to connect private FC-AL hosts. While the SilkWorm 2400 and 2800 switches offer more high-availability features (such as redundant, hot-swappable power and cooling), the highest availability environment requires that dual fabrics be used with connections from each server and storage to both fabrics. Operator errors, natural disasters, and other catastrophic events can cause an entire fabric to become inoperable. In that respect, a single fabric is a single point of failure. Path failover software—such as EMC PowerPath,VERITAS Dynamic Multipathing software, Compaq SecurePath, or other solutions—allows traffic to flow over either fabric. In these environments, the SilkWorm 2050 and 2250 switches can provide cost-effective solutions when availability of an individual switch is not the highest requirement.
NOTE By enabling data routing, rerouting, self-healing, and high scalability, Brocade full-fabric products enable switch-to-switch networking to produce a resilient multiswitch fabric. If you think you might eventually upgrade to a full fabric, this feature set is highly useful.
Scalable Fabric Switches The Brocade SilkWorm family of switches is designed to work with other popular storage hardware and servers to enable a best-of-breed open systems environment. By integrating with heterogeneous IT infrastructures, Brocade switches help leverage existing storage investments while providing a strategic path to manage continued data growth.
140_SANs_04
8/14/01
3:34 PM
Page 129
Overview of Brocade SilkWorm Switches and Features • Chapter 4
The SilkWorm 2400 and 2800 are the most widely deployed members of the Brocade SilkWorm family. Both provide a fault-tolerant solution with dual power supplies and redundant cooling fans.To further increase availability, both the power supplies and cooling fans can be replaced without taking the switch offline.These switches are bundled with Brocade Fabric OS, including Distributed Name Server, FSPF routing, automatic discovery, and advanced diagnostic and management functions. Most Brocade switches are sold bundled with Brocade Zoning and Brocade WEB TOOLS optional software packages.
SilkWorm 2400 The SilkWorm 2400 is an 8-port full-fabric switch (Figure 4.3). It is 1U high and has two slim power supplies that can be removed and replaced while the switch is online.These power supplies are the same power supplies used for the SilkWorm 2800 and therefore are interchangeable with the SilkWorm 2800.The SilkWorm 2400 supports pluggable optical or copper GBICs on all ports and offers an Ethernet connection and a serial connection for management. All of the ports are capable of supporting and automatically detecting fabric connection (F_Port), loop device connection (FL_Port), or connections to other fabric switches (E_Port). Figure 4.3 SilkWorm 2400 Fabric Switch
SilkWorm 2800 The SilkWorm 2800 is a 16-port full-fabric switch (Figure 4.4). It is 2U high and has dual power supplies (the same power supplies are used for the 2400) that can be removed and replaced while the switch is online.The SilkWorm 2800 supports pluggable optical or copper GBICs on all ports and offers an Ethernet connection, a built-in two-line LCD display, and a four-key keypad for management. This management capability enables you to configure a switch without additional equipment (such as an ASCII terminal). All ports are capable of automatically
129
140_SANs_04
130
8/14/01
3:34 PM
Page 130
Chapter 4 • Overview of Brocade SilkWorm Switches and Features
detecting and supporting either fabric connection (F_Port), loop device connection (FL_Port), or connections to other fabric switches (E_Port). All ports can be used in any mode. Figure 4.4 SilkWorm 2800 Fabric Switch
SilkWorm 6400 Integrated Fabric The SilkWorm 6400 is a 64-port integrated fabric designed for data center environments (Figure 4.5). It harnesses the strong networking capability of Brocade switches to create an integrated 64-port solution that is typically half the cost of director-class products. In addition, it enables interconnection of a large number of hosts and storage devices for enterprise-wide distributed applications.The SilkWorm 6400 is simple to install, cable, and configure.The high-port-count solution features a modular design of six SilkWorm 2250 switches in a custom housing.The six switches are preconfigured and preconnected to form a highly available fabric with 64 user ports and no single point of failure. Brocade Fabric Manager software enables consolidated management of the switch modules. The SilkWorm 6400 is interoperable with other SilkWorm switches, and it supports private loop Host Bus Adapter (HBA) environments through Brocade QuickLoop.The switch is bundled with Fabric OS, Brocade Zoning, Brocade WEB TOOLS, and Brocade Fabric Watch. The internal topology of the SilkWorm 6400 is designed to provide a costeffective high-availability solution.Two of the switch modules operate as the core of the fabric and use eight ports to provide dual connections to each of the four edge switch modules.The remaining eight ports are used for device connections. Each edge switch module provides two connections to each of the core switch modules, leaving 12 ports for device connections.The use of dual core switch modules and dual connections from each edge switch ensures that no single failure can bring down the entire integrated fabric.
140_SANs_04
8/14/01
3:34 PM
Page 131
Overview of Brocade SilkWorm Switches and Features • Chapter 4
Figure 4.5 SilkWorm 6400 Integrated Fabric
Fabric Manager By providing a centralized view of the SilkWorm 6400 switch modules, Brocade Fabric Manager simplifies SAN administration and maintenance of the SilkWorm 6400 Integrated Fabric. A portable, Java-based management application that is easy to install on a Windows management station, Fabric Manager makes it easy to view the status of all switch modules, drill down to individual switch modules, and access Brocade WEB TOOLS.
SilkWorm 12000 Core Fabric Switch To further support enterprise-level SAN deployment, Brocade has developed the SilkWorm 12000 Core Fabric Switch, which will provide up to 128 ports of connectivity in a single enclosure (Figure 4.6).The switch will be the first model based on a third-generation ASIC that enables auto-sensed link speeds of 1 and 2 Gbit/sec.With its high-performance and high-reliability characteristics, the SilkWorm 12000 will provide the same capabilities as director-class switches, but with improved intelligence, scalability, and interoperability.With superior built-in intelligence, the SilkWorm 12000 will enable a centrally managed core-to-edge network model based on proven core backbone networking methodology. The SilkWorm 12000 will also feature a protocol-independent backplane that supports 2 Gbit/sec Fibre Channel blades on release, and in the future, 10 Gbit/sec Fibre Channel blades.The protocol-independent design will also support emerging storage protocols such as Small Computer Systems Interface over IP (iSCSI), Fibre Channel over IP (FC_IP), and InfiniBand. In addition to
131
140_SANs_04
132
8/14/01
3:34 PM
Page 132
Chapter 4 • Overview of Brocade SilkWorm Switches and Features
Fibre Channel, InfiniBand, and IP, the SilkWorm 12000 will support an optional Application Platform blade that enables the deployment of high-performance fabric services such as storage virtualization and third-party copy.With the Application Platform integrated into the switch, higher data rates will be possible and management between switches and applications will be much easier. Figure 4.6 SilkWorm 12000 Core Fabric Switch
NOTE At the time of printing, the SilkWorm 12000 represents an unreleased product. This section is, therefore, a visionary statement of future SAN design capabilities.
Understanding the Brocade Fabric OS The Brocade Fabric OS enables you to easily configure, manage, and maintain a SAN for your specific needs.The de facto industry standard, Fabric OS simplifies management for both FC-AL and switched-fabric SANs. Fabric OS allows you to discover the network of connected storage and host devices and automatically determines the available data paths through the switches and fabric. In addition, Fabric OS enables you to customize your fabric via telnet commands or a Webbased Graphical User Interface (GUI). Figure 4.7 shows the high-level functions provided by Fabric OS.
140_SANs_04
8/14/01
3:34 PM
Page 133
Overview of Brocade SilkWorm Switches and Features • Chapter 4
Figure 4.7 Fabric OS Functions Optional Fabric Software Base Services
BROCADE SES
BROCADE WEB TOOLS
BROCADE QUICKLOOP
FABRIC SERVICES Distributed Simple Name Server Alias Server (Multicase) Routing Services, FSPF Universal Port
EXTENDED FABRICS
MANAGEMENT SERVICES Management Server SNMP Agent Telnet / Serial (Front Panel) Web Server Fabric Watch
REMOTE SWITCH
Fabric OS
Fibre Channel Platform
BROCADE ZONING
SILKWORM SWITCH FAMILY
Fabric OS Core Functions Fabric OS provides core functions such as: ■
Automatic discovery of devices Fabric devices log in to the Simple Name Server (SNS).Translative mode is automatically set to allow fabric initiators to communicate with private loop targets.
■
Universal port support Fabric OS identifies port types and automatically initializes each connection specific to the attached Fibre Channel system, whether it is another switch, host, private loop, or fabric-aware target system.
■
Continuous monitoring of ports for exception conditions Fabric OS disables data transfer to ports when they fail. Ports are automatically enabled after the exception condition is corrected.
Fibre Channel Services for Reconfiguration Fabric OS provides a standard set of Fibre Channel services that provide fault tolerance and automatic reconfiguration when a new switch is introduced to the fabric.These services include: ■
Management Server Supports in-band discovery of fabric elements and topology.
■
Simple Name Server (SNS) Incorporates the latest Fibre Channel standards and registers information about SAN hosts and storage devices.
133
140_SANs_04
134
8/14/01
3:34 PM
Page 134
Chapter 4 • Overview of Brocade SilkWorm Switches and Features
It also provides a Registered State Change Notification (RSCN) when a device state changes or a new device is introduced. ■
Alias Server Supports the multicast service that broadcasts data to all members of a group.
Dynamic Routing Services Fabric OS provides dynamic routing services for high availability and maximum performance.These routing services include: ■
Dynamic path selection via link-state protocols Uses Fabric Shortest Path First (FSPF) to select the most efficient route for transferring data in a multiswitch environment.
■
Load sharing to maximize throughput through Inter-Switch Links (ISLs) Supports high throughput by using multiple ISLs between switches.
■
Automatic path failover Automatically reconfigures alternate paths when a link fails. Fabric OS distributes the new configuration fabricwide and reroutes traffic without manual intervention.
■
In-order frame delivery Guarantees that frames arrive in order.
■
Automatic rerouting of frames when a fault occurs Reroutes traffic to alternative paths in the fabric without interruption of service or loss of data.
■
Routing support for link costs Enables network managers to manually configure the link costs of individual ISLs to create custom FSPF function that supports unique network management objectives.
■
Support for high-priority protocol frames (useful for clustering applications) Ensures that frames identified as priority frames receive priority routing to minimize latency.
■
Static routing support Allows network managers to configure fixed routes for some data traffic and ensure resiliency during a link failure.
■
Automatic reconfiguration Automatically reroutes data traffic onto new ISLs when they are added to the SAN fabric.
140_SANs_04
8/14/01
3:34 PM
Page 135
Overview of Brocade SilkWorm Switches and Features • Chapter 4
Facilities for End-to-End SAN Management Fabric OS includes an extensive set of facilities for end-to-end SAN management, including: ■
Management Server based on FC-GS-3 Permits in-band access to fabric discovery.
■
An SNMP agent and a series of comprehensive Management Information Bases (MIBs) —Assists with monitoring and configuring the switches. —Provides an extensive set of trap conditions. —Immediately alerts administrators about critical exception conditions.
■
In-band (through IP or over a Fibre Channel link) or external Ethernet interface Gathers SNMP information and provides access to all the switches in the fabric through a single fabric connection.
■
Syslog daemon interface Directs exception messages to up to six recipients for comprehensive integration into a host-based management infrastructure.
■
Switch beaconing Identifies an individual switch among a group of remotely managed fabric elements.
Brocade Command Line Interface The command line interface can be an excellent tool for managing your switch. You can log in to the command line interface through two methods: you can telnet into the switch or, on some models, you can connect an ASCII console to the DB9 serial port and log directly in.
Using Optional Brocade Features Brocade offers a wide variety of optional features designed to simplify the deployment, management, and administration of SAN fabric environments.These optional features are designed to help you fully leverage your SAN resources to ensure a fast Return On Investment (ROI).
135
140_SANs_04
136
8/14/01
3:34 PM
Page 136
Chapter 4 • Overview of Brocade SilkWorm Switches and Features
NOTE SilkWorm switch features are enabled through a licensing system. When you purchase additional feature products, you simply enter a license key into the switch to enable the feature.
Brocade Zoning Brocade Zoning, now bundled with almost every switch, provides advanced SAN management capabilities. It enables the separation of a fabric into smaller, isolated subfabrics to address closed user group requirements. As a result, zoning is an excellent way to enhance security within a fabric. With zoning, fabric-connected devices are arranged into logical groups of devices over a single physical configuration, enabling you to segregate certain devices from other devices.This can be helpful when you have devices that do not interoperate well, or when you want to separate a development environment from a production environment without purchasing additional switches. Devices in a zone only “see” other devices in that zone and can access only those members. Any device not included in a given zone is not available to the devices in that zone. Brocade Zoning is available in both software and hardware formats, and you can intermix both formats within a fabric. In general, software-enforced zoning provides more flexibility, while hardware-enforced zoning provides the highest level of security. Brocade Zoning involves zone specification, enforcement, and management. You can use a set of telnet commands (either in-band or out-of-band) to create, delete, and display zones; to add or remove zone members; and to configure a set of zones. Further information about Brocade Zoning appears in Chapter 6, “SAN Applications and Configurations,” and Chapter 9, “SAN Implementation, Maintenance, and Management.”
Extended Fabrics Because you might need to connect multiple data centers over a long distance, Brocade offers Extended Fabrics support. Extended Fabrics reconfigures the switch to support the rigors of transmitting I/O over long distances in conjunction with technologies such as Dense Wave Division Multiplexing (DWDM).
140_SANs_04
8/14/01
3:34 PM
Page 137
Overview of Brocade SilkWorm Switches and Features • Chapter 4
This feature extends all of the scalability, reliability, and performance benefits of Fibre Channel SANs beyond the native 10 km distance specified by the Fibre Channel standard. Moreover, it enables the use of full-performance applications over extended distances, including disaster recovery, remote backup, extended storage consolidation, remote mirroring, and tape consolidation. Extended Fabrics can be especially useful if you want to connect remote locations—such as a disaster recovery facility—with the high performance and reliability associated with a Fibre Channel SAN. Using Extended Fabrics, you can leverage existing high-speed public and private networks to connect your Fibre Channel SANs over Metropolitan Area Networks (MANs) and Wide Area Networks (WANs).
Brocade Fabric Access Layer API As the proliferation of storage networking increases, the need for storage applications to directly access and control fabric resources has become a critical requirement. Fabric Access, the Brocade Fabric OS Access Layer API, provides a flexible way for applications to access a variety of SAN information. Through Fabric Access, applications can control the fabric for functions such as discovery, access (zoning) management, performance, and switch control. Fabric Access consists of a host-based library that interfaces the application to switches in the fabric through an out-of-band TCP/IP connection or an in-band IP-capable HBA. You can develop your own SAN management applications from the API or take advantage of applications from third-party software developers, such as EMC and VERITAS. Key benefits of Fabric Access include: ■
Single point of access to the fabric
■
Secure access control
■
Multifabric access
■
Transaction-based management
■
Object-oriented XML interface
■
Multiplatform support
■
Conformity to industry standards
137
140_SANs_04
138
8/14/01
3:34 PM
Page 138
Chapter 4 • Overview of Brocade SilkWorm Switches and Features
Fabric Watch An optional SAN monitor for Brocade SilkWorm switches running Fabric OS 2.2 or higher, Fabric Watch enables each switch to constantly watch fabric elements for potential faults—and automatically alert you about problems before they become costly failures. Fabric Watch tracks a variety of SAN fabric elements, events, and counters—including fabric-wide events, ports, GBICs, and environmental parameters. Using Fabric Watch, you can quickly identify and isolate faults while optimizing fabric-wide performance. In addition, Fabric Watch integrates easily with standard enterprise systems management tools—sending traps as opposed to requiring polls to report exceptions. Compared to other types of management protocols—such as SNMP—Fabric Watch provides a more robust solution that enables proactive management of your SAN environment.
Understanding Loop Support, QuickLoop, and Fabric Assist Brocade switches can replace intelligent hubs by creating virtual loops.This is done by logically connecting ports on one or two SilkWorm switches, with one or more private loop devices connected to each of the ports. Each switch port and the devices attached to it form a “looplet” that can independently transfer data at 100 MB/sec. Unlike hub-based environments, bandwidth is not shared across looplets, so full-bandwidth data transfers can occur simultaneously on all switch ports. Multiple hosts can simultaneously transfer data to different looplets in parallel. Also unlike hubs, Brocade switches provide superior fault isolation capabilities, preventing errors on a single device from disrupting the entire SAN. The Brocade QuickLoop software product provides a cost-effective solution for using private loop hosts with your switched-fabric-based SAN. QuickLoop is an alternative to various hub-based solutions, and since you are connecting these devices to a switch as opposed to a hub, private loop environments should exhibit significant performance and reliability improvements. As a result, QuickLoop can help protect your existing technology investment in hub solutions while integrating them into a higher performance environment. If you want to connect private loop HBAs to your Brocade Fabric, you will need QuickLoop or QuickLoop with Fabric Assist. QuickLoop operates as a translator between the legacy loop initiators and the fabric target devices.When setting up QuickLoop, you specify which ports are connected to loop initiators and which fabric targets will be available to them.
140_SANs_04
8/14/01
3:34 PM
Page 139
Overview of Brocade SilkWorm Switches and Features • Chapter 4
NOTE Only private loop initiators (HBAs) require the QuickLoop/Fabric Assist products. Public loop devices, and even private loop targets (storage) use the Fabric OS built-in translative mode feature instead.
QuickLoop offers two modes of private loop host attachment: Hub Emulation and Fabric Assist.With Hub Emulation, QuickLoop actually builds private loops across a set of switch ports on one or two switches. A combination of QuickLoop functions and Brocade Zoning, Fabric Assist creates virtual loops capable of spanning the entire fabric while allowing private loop hosts to function as if they were attached to a physical loop switch. Brocade Fabric OS assigns a “phantom” fabric address to each private loop target device, enabling each device to be registered transparently in the fabric.To migrate a private loop host to the fabric, a QuickLoop port is reconfigured to operate in Fabric Loop Attach (FLA) mode. Next, a single private loop host is attached to the port (which cannot connect to any other devices). A Fabric Assist zone is then configured to include the private loop host and all attached storage with which it needs to communicate.The storage can be private loop, public loop, or fabric aware.The private loop host appears to reside on a dedicated private loop with all of the storage in the Fabric Assist zone.
Brocade WEB TOOLS To simplify SAN fabric management, Brocade WEB TOOLS is a software utility that enables you to manage and monitor your fabric through a Web browser interface and Java plug-in. Using WEB TOOLS, you can view all switches in the SAN from a single interface from any workstation in your enterprise—even at a remote location.You can perform administration and configuration tasks for the entire SAN fabric, fabric switches, and individual ports.The utility presents a graphical representation for each switch licensed for WEB TOOLS, and you can manage and manipulate each switch through the GUI. In addition to showing a graphical representation of each switch, the WEB TOOLS screen indicates the status of the switch.When a switch has a warning, you can click on that switch to obtain a detailed view to see power supply status, GBIC/link status, and activity indicators.This approach enables you
139
140_SANs_04
140
8/14/01
3:34 PM
Page 140
Chapter 4 • Overview of Brocade SilkWorm Switches and Features
to glance at a display and know immediately if there is a problem with the fabric so you can take corrective action.
Future Capabilities in the Brocade Intelligent Fabric Services Architecture To provide a powerful yet flexible framework for addressing critical SAN requirements, Brocade has developed the Intelligent Fabric Services Architecture, which will provide both the basic switching functions and the advanced services that improve manageability, availability, security, and scalability (Figure 4.8).This architecture, which will help transform the network into an intelligent SAN fabric, consists of the following building blocks: ■
The SilkWorm family of fabric switches
■
Advanced Fabric Services
■
Open Fabric Management tools
■
Enterprise-class security products
NOTE At the time of printing, components of the Brocade Intelligent Fabric Services Architecture represent unreleased products or product functions. This section is, therefore, a visionary statement of future SAN design capabilities.
Beginning with the SilkWorm 12000, the Intelligent Fabric Services Architecture will enable a wide range of advanced switch fabric services, including Brocade ISL Trunking, Brocade Frame Filtering, more robust hardwareenforced zoning, more comprehensive performance monitoring, and enhanced security with Brocade Secure Fabric OS.
Brocade ISL Trunking Brocade ISL Trunking enables as many as four Fibre Channel links between switches to be combined to form a single logical ISL with an aggregate speed of up to 8 Gbit/sec.These high-speed trunks simplify network design, optimize bandwidth utilization, and ensure that server-to-storage performance remains balanced under heavy network loads (Figure 4.9).
140_SANs_04
8/14/01
3:34 PM
Page 141
Overview of Brocade SilkWorm Switches and Features • Chapter 4
Figure 4.8 The Brocade Intelligent Fabric Services Architecture
SAN Management and Administration, Storage Resource Management, Storage Administration, Storage Virtualization, LAN-Free Backup
Enterprise-Class Security Fabric OS
Advanced Fabric Services 2000
2200
2400
Open Fabric Management 12000
6400
2800
Intelligent Switching Platform Fibre Channel
IP
InfiniBand
Multiprotocol Support
Figure 4.9 Brocade ISL Trunking Relieves Congestion and Enables High-Speed Data Traffic Optimal bandwidth utilization using load balancing on up to four 2 Gbit/sec links
Without Trunking
1G
With Trunking
1.5 G 0.5 G 1G 1G
2G
1.5 G 0.5 G 1G 2G
Congestion 2G 1.5 G 0.5 G 1G 2G
2G 1.5 G 0.5 G
1G 2G
ASIC Preserves In-Order Delivery
ISL Trunking is an optional software product available with all Brocade 2 Gbit/sec Fibre Channel fabric switches.This new technology is ideal for optimizing
141
140_SANs_04
142
8/14/01
3:34 PM
Page 142
Chapter 4 • Overview of Brocade SilkWorm Switches and Features
the performance and simplifying the management of a multiswitch SAN fabric containing Brocade 2 Gbit/sec switches.When two, three, or four adjacent ISLs are used to connect two switches, the switches automatically group the ISLs into a single logical ISL, or a “trunk.”The throughput of the resulting trunk is 4, 6, or 8 Gbit/sec.
Brocade Frame Filtering Another advanced feature incorporated into Brocade 2 Gbit/sec switch hardware is the ability to filter Fibre Channel frames to increment counters or to perform other actions such as blocking the frame itself. Overall, Frame Filtering enables a variety of new capabilities for monitoring and managing SAN fabrics, including the ability to: ■
Increase zoning capabilities and security
■
Facilitate the deployment of new SAN management applications that improve visibility into, and control over, the fabric
■
Enable new fabric services for clustering, virtualization, and shared file systems
Wire-speed filtering of each frame based on the content of several fields in both the header and the payload enables fabric zoning based on Logical Unit Number (LUN), network protocol, or I/O request type.This approach enables fabric-wide heterogeneous LUN masking managed from a central point. Brocade Frame Filtering combines the security of hardware zoning with the cabling flexibility of software zoning.When an administrator moves a cable from one port to another, the Frame Filtering capabilities can monitor the unique address of the device, change the zone, and block inappropriate data from communicating with it.This critical zoning improvement helps ensure security, while minimizing the time and effort required to manage the SAN fabric and its associated zones.
More Robust Hardware-Enforced Zoning Hardware-enforced zoning by World-Wide Name (WWN), port ID, or Arbitrated Loop Physical Address (AL_PA) simplifies administration while providing the highest level of secure control over data access.This capability provides administrators with much more flexibility in how they partition storage and servers to secure the overall fabric.
140_SANs_04
8/14/01
3:34 PM
Page 143
Overview of Brocade SilkWorm Switches and Features • Chapter 4
Enhanced End-to-End Performance Analysis Enhanced end-to-end performance analysis enables more effective tracking of resource utilization on a fabric-wide basis. Administrators can capture I/O performance levels associated with specific initiator and target device IDs anywhere in the fabric, independent of fabric topology. In addition to reducing management cost through more proactive capacity planning, this capability enables reporting at a level required to demonstrate adherence to service level agreements.
Secure Fabric OS Within the Intelligent Fabric Services Architecture, Brocade provides Secure Fabric OS, the most comprehensive SAN security architecture available. Based on state-of-the-art networking security technologies, Secure Fabric OS addresses vulnerabilities in the SAN fabric and supports authentication methods at the following access points: ■
User access to the management interface
■
Management console access to the fabric
■
Server access to the fabric
■
Switch access to an existing fabric
To prevent unauthorized configuration or management changes, Fabric OS employs policies with multilevel passwords, extensive use of Access Control Lists (ACLs), and centralization of fabric configuration changes on “trusted” switches. Fabric OS prevents WWN spoofing—the practice of assuming the WWN of another server to gain access to storage in its zone—at both the HBA and server level by locking certain WWNs to certain ports.With Secure Fabric OS, new switches are assigned digital certificates, enabling an existing fabric to authenticate any switch that joins the fabric.While Secure Fabric OS prevents unauthorized access to the fabric from the outside, Brocade Zoning ensures that devices can access only their authorized storage resources.
143
140_SANs_04
144
8/14/01
3:34 PM
Page 144
Chapter 4 • Overview of Brocade SilkWorm Switches and Features
Summary Before selecting what switches you should use to build your SAN fabric configuration, you need to consider a wide variety of variables. As with all resource planning, you should identify and prioritize your key requirements—both current and future—before comparing the available products in the marketplace. Brocade offers a full range of Fibre Channel switches—from hub alternatives to highlyavailable fabric switches, integrated fabrics, and core fabric switches with 64 ports of connectivity. Each Brocade SilkWorm switch provides unique characteristics designed for entry-level SANs, medium-sized data centers, or very large enterprises. Along with these switches, Brocade provides Fabric OS, a real-time operating system designed to deliver all the key functions for managing your SAN environments. Fabric OS includes a wide range of basic functions, including Fibre Channel services and rerouting services. It also features an API that you can use to write your own SAN management applications (or you can take advantage of applications developed by third-party software vendors). To simplify the deployment, management, and ongoing administration of fabric-related tasks, Brocade offers a comprehensive set of software products. Products such as Brocade QuickLoop, Zoning, and WEB TOOLS can help you fully leverage your existing hardware investments and help position you for continued growth. In addition, Brocade provides optional features—such as Brocade Extended Fabrics and Fabric Watch—designed for specific types of SAN environments or functions. By taking advantage of the full range of these products, you can significantly increase the overall return on your SAN investment. Brocade will also provide strategic software functions as part of its Intelligent Fabric Services Architecture. Functions such as ISL Trunking and Frame Filtering will provide high-performance and high-reliability characteristics to meet the most demanding enterprise-level requirements. In addition, enhanced hardware zoning functions and Secure Fabric OS will greatly improve security within SAN environments, enabling SANs to grow in a safe, controlled manner.
Solutions Fast Track Selecting the Right Switch Identify your requirements for availability, port density, functionality,
and cost.
140_SANs_04
8/14/01
3:34 PM
Page 145
Overview of Brocade SilkWorm Switches and Features • Chapter 4
Decide whether you need an arbitrated loop or full-fabric environment. Learn which switch functions best satisfy your requirements. Consider what strategic direction you want to take, and whether your
current switches will scale easily to meet your needs.
Understanding the Brocade Fabric OS Fabric OS is the operating system for all Brocade SilkWorm switches. Key functions include auto-discovery, in-order frame delivery, zoning,
and others. Fabric OS provides the capability to work with other storage
management applications.
Using Optional Brocade Features You can use Brocade Zoning to isolate devices into separate,
virtual SANs. Zoning is ideal for multiple customer environments where data security
is critical. Extended Fabrics enables the benefits of Fibre Channel technology at
distances up to 100 km. Fabric Watch tracks switch and fabric events to help you optimize
fabric-wide performance and proactively identify problems before they happen. QuickLoop integrates private loop-based devices into switched fabric
environments. QuickLoop helps support legacy devices to protect existing investments
while also providing performance and reliability advantages. WEB TOOLS is an advanced monitoring tool that sends alerts about
fabric events to help prevent downtime. You can use a Web browser interface and Java plug-in to monitor
switched-fabric SANs from any workstation in your enterprise.
145
140_SANs_04
146
8/14/01
3:34 PM
Page 146
Chapter 4 • Overview of Brocade SilkWorm Switches and Features
Future Capabilities in the Brocade Intelligent Fabric Services Architecture The Brocade Intelligent Fabric Services Architecture includes the
SilkWorm family of fabric switches, advanced fabric services, open fabric management tools, and enterprise-class security products. ISL Trunking is an optional product ideal for optimizing πperformance
of Brocade 2 Gbit/sec Fibre Channel fabric switches. Frame Filtering enables a variety of new capabilities for monitoring and
managing SAN fabrics and enhancing both security and reliability. Secure Fabric OS is the most comprehensive SAN security architecture
available, addressing vulnerabilities in the SAN fabric and supporting multiple authentication methods.
Frequently Asked Questions The following Frequently Asked Questions, answered by the authors of this book, are designed to both measure your understanding of the concepts presented in this chapter and to assist you with real-life implementation of these concepts. To have your questions about this chapter answered by the author, browse to www.syngress.com/solutions and click on the “Ask the Author” form.
Q: Can I use Brocade switches in an arbitrated loop environment? A: Yes. Brocade offers switches that are viable alternatives to managed hubs. In addition, the Brocade QuickLoop product enables the integration of private loop-based devices into a switched fabric.
Q: What switch is most capable of providing high availability at a low cost? A: For a relatively inexpensive switch that provides high-availability characteristics, try the SilkWorm 2400 and 2800. Both switches have redundant, hotswappable components at a cost-effective price point.
Q: What is the most reliable way to keep certain hosts from interacting with other hosts or storage devices?
140_SANs_04
8/14/01
3:34 PM
Page 147
Overview of Brocade SilkWorm Switches and Features • Chapter 4
A: Brocade Zoning enables you to configure distinct zones to restrict interoperability between various devices.
Q: How can I perform fabric administration and management? A: Brocade offers several tools for simplifying these types of tasks. Fabric Watch, Fabric Access, and WEB TOOLS all provide timesaving functions that help reduce SAN management costs.
Q: What kind of switch is most suited for a very large enterprise SAN? A: The Brocade SilkWorm 6400 provides 64 ports of connectivity with highavailability characteristics, making it ideal for data center environments and enterprise SAN implementations.
147
140_SANs_04
8/14/01
3:34 PM
Page 148
140_SANs_05
8/14/01
3:35 PM
Page 149
Chapter 5
The SAN Design Process
Solutions in this chapter: ■
Looking at the Overall Lifecycle of a SAN
■
Conducting Data Collection
■
Analyzing the Collected Data
Summary Solutions Fast Track Frequently Asked Questions
149
140_SANs_05
150
8/14/01
3:35 PM
Page 150
Chapter 5 • The SAN Design Process
Introduction We intend this book to allow you to effectively design, implement, and maintain storage networks. Doing so requires an understanding of the processes in each of the seven phases of a SAN’s lifecycle, and their relationships with each other. Without taking a moment to review the process from the highest level, it is easy to get lost in the details of SAN hardware. In this chapter, we provide that high-level view.We show how the SAN design process is really an ongoing lifecycle.We take you through the process from the moment the decision is made to deploy a SAN, through releasing the SAN to production.Then we explain the extent to which the process should be repeated when upgrades and architectural changes are needed.We also provide detail on the first two parts of the lifecycle. The processes presented here are derived from other areas of Information Technology (IT) and they are normal parts of any large-scale IT project. For example, when implementing a SAN, you should interview people who will have a key interest in the finished product—the same is true when putting in a Local Area Network (LAN) or Wide Ares Network (WAN). Much of this material should be second nature to any IT network architect, Database Administrator (DBA), or senior systems administrator. For the more advanced users to whom these techniques are well understood in general, this chapter will serve as reference material showing how these processes are applied to SANs in particular.We have attempted in this book to provide material that will allow both the beginner and the expert alike to successfully design a SAN. It is true that more attention must be paid to SAN design than to most other networking technologies.This is because SANs typically have more stringent availability and performance requirements than other networks. A SAN is similar to a traditional network in its requirements, but is also somewhat like a channel (for example, a CPU/RAM interconnect mechanism, or a PCI bus). Channels require very high performance, and are almost assumed to be 100 percent reliable.This is in stark contrast to the traditional Ethernet LAN, where things like five-nines uptime for all node connections, in-order packet delivery, and tuned approaches to bandwidth management are rare indeed. Fortunately, SANs provide the tools necessary to achieve these performance and availability goals. For example, it is commonplace in a Fibre Channel SAN to use a dual-fabric approach to SAN architecture.This means having two completely separate networks for data to travel over, and potentially using both networks as active paths.While it is certainly possible to do this sort of thing using IP/Ethernet networks, it is substantially more difficult, since Fibre Channel was
140_SANs_05
8/14/01
3:35 PM
Page 151
The SAN Design Process • Chapter 5
designed with this in mind, and Ethernet was not.The SAN designer must provide for higher availability and spend some time thinking about performance, but will know going into the process that these goals are entirely achievable. We should also note here that the process outlined in this chapter is designed to make a complex SAN design successful.With less complex designs (that is, the majority of SAN deployments to date), it is perfectly acceptable to skip over much of the process. For example, if you are deploying a SAN with only three servers and two storage arrays, spending much time on architectural analysis is unnecessary.The complexity is presented here so that users with complex requirements will have it available to them; users with simpler scenarios can use their judgment about which bits to incorporate into their design process. The seven phases of the lifecycle of a SAN at the very highest level can be broken down into three broad categories: design, implementation, and maintenance.The first of these, designing the SAN, includes the collection and the analysis of data, which defines the requirements of the network.We will go into detail on these first two phases of the design process in this chapter.These phases provide a solid launch pad for your journey through the remainder of the SAN’s lifecycle. The third and fourth phases of the SAN lifecycle—architecture development and prototype testing—complete the design process. Implementing the SAN encompasses the transition phase and the release to production phase, the fifth and sixth phases of the lifecycle.These phases are discussed in Chapters 6 and 7 of this book. Chapters 8 and 9 cover the troubleshooting, maintenance, and management—the final phases of the lifecycle model. When you are finished reading this chapter, you should have a solid understanding of the design processes, and have a valuable reference tool to enable project planning on any future SAN deployments.
Looking at the Overall Lifecycle of a SAN Any SAN will go through certain phases over the course of its life. Depending on the size and complexity of the SAN, some phases might take months to complete, and some might be only glanced over. For example, a single-switch SAN does not require much in the way of network design. However, if the solution involves hundreds of devices, including storage components from many different vendors that were not already pretested and determined to be interoperable, it could require extensive testing or validation.
151
140_SANs_05
152
8/14/01
3:35 PM
Page 152
Chapter 5 • The SAN Design Process
When an existing SAN must undergo a fundamental change, be it at the architectural level or simply the introduction of a new type of storage array, you should cycle back through the phases of SAN development.This will ensure that the critical applications running on the SAN are not unexpectedly disrupted by changes. However, when the change is fundamental but small (like adding a new type of storage array) it is possible to take a fast track through this process. The SAN’s lifecycle, which can be described at a high level as design, implementation, and maintenance, translates directly into action-oriented phases on the part of the SAN designer: data collection, data analysis, architecture development, prototype and testing, transition, release to production, and maintenance. See Figure 5.1 for a flowchart of these phases and their relationships to each other. Figure 5.1 An Overview of the Lifecycle of a SAN Upgrade / Architectural Change
Design
Implementation
Data Collection
Transition
Data Analysis
Release to Production
Architecture Development
Prototype and Test
Maintenance Add / Change / Remove / Management / Troubleshooting
140_SANs_05
8/14/01
3:35 PM
Page 153
The SAN Design Process • Chapter 5
Data Collection You must define the requirements of the SAN before building it.What business problem is being solved by the SAN? What are the overall goals of the project? To determine the requirements, you should interview all affected parties, to find out what they all hope to achieve (in other words, their goals and objectives), and develop both a detailed technical requirements document and a timeline for the project.
Data Analysis Once you have gathered input from all parties, you must analyze it and put it into a meaningful format.The first two phases together will allow you to start with the business goals that are driving the project, and determine at a high level the necessary technical properties required of the SAN. Once this phase is completed, all business requirements should be translated into technical requirements. The technical requirements document will be created during the collection phase, and completed during the analysis phase.You will also have created a working document for a Return On Investment (ROI) proposition to justify the expense of the project.
Architecture Development Now that you have a list of technical requirements, you will develop a SAN architecture that meets those requirements.This process will involve balancing many factors. For example, there might be a tradeoff between performance considerations and cost. It might be necessary for you to cycle back to the data collection and analysis phases to gather more input to make compromises with input from all affected parties.When finished, you will have a detailed architecture of the SAN that you intend to build. A SAN architecture includes the fabric topologies of all related fabrics, the storage vendors involved, the SAN-enabled applications being used, and other considerations that affect the overall SAN solution.This step is the most likely to be skipped over quickly when the SAN requirements are small.
Prototype and Testing SANs deal directly with the mission-critical data of today’s enterprises.When building any mission-critical solution, you must test it before releasing it to production. In this phase, you will build a prototype of the SAN solution and test it
153
140_SANs_05
154
8/14/01
3:35 PM
Page 154
Chapter 5 • The SAN Design Process
to ensure that it will function properly when released.This should be done using nonproduction systems. It might be necessary to cycle back to the architecture development phase if problems are found. Wherever possible, build a test bed identical to the solution you are implementing.This will provide the greatest assurance of success in production. However, budgetary concerns, limits on time and space, and other factors will usually prevent this from being practical. Imagine a 200-port SAN. Now imagine 200 hosts and storage arrays plugged into it. Now imagine asking the CFO to buy another 200 devices to test with, and to provide administrators, space, power, and cooling for all of it. Because of this, the test phase will be a balance of conducting your own testing, and leveraging other organizations’ test results. Finding a document that says “vendor X already tested or certified this configuration” might be as good or better than testing it yourself. Even if the components of a solution have been tested by you and/or others to your satisfaction, you must test all aspects of the complete system prior to releasing it to production.This is due to the fundamental nature of a large networked system where interactions, timing, and other factors can produce different results from devices tested individually.The actual final test will occur during the release to production phase, but creation of the test plan should occur in this phase. At the end of this phase, all parties with an interest in the outcome of the project will approve it, and the transition to production will begin.
Transition Now that you have a working prototype, and all interested parties have signed off on it, you will begin to transition your existing hardware onto the new SAN. If a SAN is already in place, this phase might be as simple as adding a new node to the SAN, or changing the Inter-Switch Link (ISL) architecture. If the SAN is completely new, it might involve a long migration process consisting of moving one production system at a time. In any case, there might be a need to cycle between this phase and the release-to-production phase repeatedly. Once a component has completed the transition onto the SAN, release to production can occur for that component.
Release to Production Once a component has been transitioned onto the new SAN, it must be tested again and then approved before becoming a part of the enterprise’s production environment. Since there might be many components that must be transitioned
140_SANs_05
8/14/01
3:35 PM
Page 155
The SAN Design Process • Chapter 5
and released, it might be necessary to cycle between the transition and release-toproduction phases repeatedly until all components have entered production. After this phase is complete, the SAN will enter the maintenance phase.
Maintenance This is the useful life of the SAN. All of the benefits that prompted the SAN designer to implement the SAN in the first place are found in this phase. It is therefore desirable to have a SAN spend as much time as possible in this phase, and as little as possible in the other phases.The goal of this phase is to keep the SAN running as well as possible for as much of the time as possible, and to expand its capabilities only according to the original, tested, and approved parameters.This phase includes adding, changing, or removing components, as well as managing, monitoring, and troubleshooting existing components. During the maintenance phase, no changes should be made to the SAN that fall outside of the original blueprint that was established in the previous phases. Any such change necessitates a repetition of the entire lifecycle. For example, if the SAN were originally built using vendor X storage arrays, an additional vendor X array could be added as part of maintenance, but an array from vendor Y would require thought and testing before its introduction. It might not require much thought and testing, but it must, in any case, be looked into.
NOTE Any fundamental change to the SAN requires a repetition of the entire lifecycle.
In summary, the seven phases of the SAN design lifecycle are: 1. Data Collection 2. Data Analysis 3. Architecture Development 4. Prototype and Test 5. Transition 6. Release to Production 7. Maintenance
155
140_SANs_05
156
8/14/01
3:35 PM
Page 156
Chapter 5 • The SAN Design Process
Conducting Data Collection The data collection phase of SAN design is the foundation upon which the SAN will be built. It is vital that the information collected in this phase be both complete and accurate. If the SAN requirements are poorly defined, it is guaranteed that the resulting SAN will meet business objectives poorly.You should therefore take your time with this phase. Some of the information you will collect is generic to any major IT project. If you already have an established data collection process in your company, integrate the SAN-specific material from this section into that process. Data collection consists of determining which people you will need to interview, interviewing them, and conducting a physical assessment of existing equipment and facilities.When this process is complete, you will have a technical requirements document consisting of a list of the business problems that the SAN will solve, the business requirements for the SAN, characteristics of all devices that will be attached to it, and detailed information about all relevant facilities. You will also have a timeline for implementation.
Creating an Interview Plan Who has a stake in the SAN solution? Well, you could argue that every person who uses a system attached to the SAN has a stake in it.While true, this is not useful for creating an interview list, because there would be too many people involved. Similarly, you could argue that only the person who initiated and “owns” the project should be consulted. Again, this is not useful, because it leaves out people who have a strong interest in the project, and might have knowledge that is critical to its success. A balanced approach to creating an interview list is critical.You can view the people on this list as a SAN solution “core team.”Think about having all of these people together in a room, and trying to solve the SAN solution problem together.Try to include everyone needed to solve the problem, but nobody else. Typically, a core team might include: ■
At least one systems administrator
■
At least one storage administrator
■
A network administrator
■
A DBA, if a database server will be involved
140_SANs_05
8/14/01
3:35 PM
Page 157
The SAN Design Process • Chapter 5 ■
At least one application specialist associated with each application that will run on the SAN
■
At least one manager who can act as an overall “owner” of the project
It is probable that you will be one of these people, in addition to being the SAN designer. Unless you are an external consultant, this is typically the case. Once you have a list of the desired members of the core team, you must contact them and ask them to take time to help with the project. Ensure that each team member has allocated the necessary time and that their management appreciates the demands of participating in this team. As the SAN design goal of the team might require a long-term process, getting this buy-in initially will minimize disruption to the team later. Often in the past, SAN design teams did not include network administrators, as the focus was on the storage side. Experience has shown that SANs are networks, and should be coordinated with the traditional IP network groups to ensure that proper networking experience is at hand. Whenever possible, schedule an interview as a face-to-face, one-on-one meeting.This format will allow you to communicate the questions and understand the answers most effectively.You should also have a group meeting with the entire core team after conducting individual interviews.This will allow you to resolve any differences before analyzing the data, and review the analysis as a team.
Conducting the Interviews Now that you know who to interview and have a schedule of when you will interview them, you need to know what questions to ask, and what format to put the collected data into.This section contains a suggested set of questions that you should ask, and some detail on what each question is about. It is followed with a summary that could be used to create an interview form.
NOTE Not every person you interview will be able to answer every question. Between the members of the core team, the expertise necessary to answer all of these questions should be completely represented. Some members might provide conflicting answers. You will be in a key position to resolve these differences, and achieve a compromise. It is vital that all affected parties agree with the deployment strategy before implementation begins.
157
140_SANs_05
158
8/14/01
3:35 PM
Page 158
Chapter 5 • The SAN Design Process
What Overall Business Problem Are You Trying to Solve? A business problem that would initiate a SAN design might be something like: ■
“We need to keep our business running in case of a disaster like an earthquake or fire.”
■
“Our backups take so long to finish that they are impacting our ability to process customer orders.”
■
“We need to save money on storage by utilizing free space more efficiently.”
Chapter 6 discusses some of the more common business problems that SANs can solve. Brocade maintains a series of documents that detail specific SAN solutions.These documents are known as Brocade SOLUTIONware configuration guidelines and are available on the Brocade Web site at www.brocade.com/SAN.
NOTE A SAN might be intended to solve multiple business problems. In this case, you should separate each business problem into a different set of questions and answers. You will correlate these during the analysis phase.
What Are the Business Requirements of the Solution? Once you know the business problem that you need to solve, it should be easy to figure out what the business requirements of the solution must be.This is simply a matter of rephrasing the previous answers, with more specific criteria: ■
“The SAN must allow all functionality of all business-critical servers at site X to resume within Y minutes at site Z.”
■
“The SAN must allow the following list of servers to complete backups within X minutes: …”
■
“The SAN must allow the following list of servers access to the corresponding list of storage arrays: …”
140_SANs_05
8/14/01
3:35 PM
Page 159
The SAN Design Process • Chapter 5
This is useful because it acts as a migratory step toward turning the business problem into a matching technical solution.
Moving from Business Requirements to Technical Requirements You should not deploy a SAN simply for the sake of adopting the “hot new technology.” SANs are hot because they solve important business problems and allow companies to make more money.This could be fairly direct—for example, a matter of saving more money on IT than the project cost, since SANs are very efficient at providing a clear ROI. ROI is often achieved by management efficiencies, resource efficiencies, or better utilization of resources. On the other hand, it could be indirect—by making IT systems more efficient, thus increasing users’ productivity. The first key to a successful SAN deployment is the accurate and complete statement of what business problem(s) you intend for the SAN to solve. Unfortunately, you cannot turn a business problem into a technical solution without work. There is no silver bullet to make your backups run faster so that your users will not have to work on a slow system. However, there are tape libraries that run fast, and can be shared by many devices.This, when combined with an appropriate Fibre Channel fabric, and a SAN-enabled backup application, could amount to the same thing as the silver bullet. In order to know which hardware and software will solve your business problem, you have to define in a technical way what you need to accomplish. This is a necessary intermediate step between the business problem and the purchase of specific technical solutions. It is fairly straightforward to change a sentence like, “We need to keep our business running in case of a disaster like an earthquake or fire” into a sentence like, “The SAN must allow all functionality of all business-critical servers at site X to resume within Y minutes at site Z.” Once you have done this, you will have the business requirements of the solution.You know that you have a business requirements statement when you could phrase it like this, and still have it make sense: “Our business will run better if we have a SAN that can allow all functionality of all business-critical servers at site X to resume within Y minutes at site Z.”The components of the business requirements statement are “our business will run better” (or something to that effect) followed by a reasonably specific statement about what the SAN must do to make that happen. However, you will still not have the technical requirements detailed.This is not something that you, the SAN designer, can simply ask in an interview.This is a large part of what you will bring to the table as the SAN designer once you have
159
140_SANs_05
160
8/14/01
3:35 PM
Page 160
Chapter 5 • The SAN Design Process
gathered the data and then analyzed it in the next phase. A technical requirements document set should list, in detail: ■
All of the devices that are to be attached to the SAN
■
Their locations
■
The communication patterns between them (random I/O, streaming access such as video, I/O-intensive database access)
■
Their performance characteristics (reads, writes, max/min/typical throughputs)
■
What software will run on them relative to the SAN (for example, a LAN-free backup application, or anything SAN-specific)
■
How all of this is expected to change over time (storage growth, server growth)
The technical requirement statement would be, “The SAN needed to meet the business requirements outlined must have the following characteristics: …” This would be followed by the body of the technical requirements document. The rest of the questions to ask in the interview process will provide you with the body of this document.
What Is Known about the Nodes that Will Attach to the SAN? You should try to get a list of all information possible about every node attached to the SAN. For each node, the relevant information can include questions about each host, storage device, facilities where hosts and storage will be located, and questions about the SAN itself. Questions about each host could include the following: ■
What operating system is installed? What patch or service pack level?
■
Are fabric HBA/controller drivers available? Are they well tested?
■
What type of connection is supported (private loop, public loop, or fabric)?
■
Which applications will run on this host (databases, e-mail, data replication, file sharing)?
140_SANs_05
8/14/01
3:35 PM
Page 161
The SAN Design Process • Chapter 5 ■
How much storage does it require?
■
How will its storage requirements change over time?
■
Physically, what are its dimensions? How heavy is it?
■
Does it rack mount? Does it have a rack kit? Will it set on a shelf?
■
If there is a management console, what type is it? (Is it a traditional keyboard/video/mouse combo [KVM], or is it a serial connection, like a TTY?) Does it need to be permanently attached? (For example, a Sun SPARC server could have a keyboard, mouse, and monitor permanently attached, or it could be managed through a serial port attached to a modem.)
■
How many HBAs will it have?
■
If it has more than one HBA, what software will be used to provide failover or performance enhancements of multiple paths?
■
Do these interfaces exist, or do they need to be purchased? (You should keep track of every piece of equipment that you need to buy for the project, for budgeting and ROI analysis.)
■
If they exist, what are the make, model, and version information?
■
If not, what kind will be purchased to meet the objective?
■
How many Ethernet interfaces will it have?
■
In what temperature range will it operate?
■
Will it need a telephone line for management?
■
Where will the node be physically located?
These questions could be used to create an interview form for each host, which might look like the following:
161
140_SANs_05
162
8/14/01
3:35 PM
Page 162
Chapter 5 • The SAN Design Process
OS HBA Drivers
fabric
PTP
private loop
public loop
New
make
model
version
Existing
make
model
version
HBA Count DMP / Failover Support HBA New or Existing
Application List Initial Storage Requirements Projected Storage Requirements Dimensions Weight Mounting
rack mount
Console Type
KVM
rack shelf switched KVM
floor TTY
table top
terminal server
modem
Console Location Ethernet Interface List Operating Temperature Power Requirements
Voltage
Amperage
Connector Type
Need Telephone Line for Management? Physical Location
Questions about each storage device could include the following: ■
What are the make, model, and version information?
■
What type of connection is supported (private loop, public loop, fabric, SCSI, SSA)?
■
How many hosts can this device serve?
■
If it is a multiport device, does it have limits on how many hosts can access it through each port?
■
Physically, what are its dimensions? How heavy is it?
■
What is its capacity in gigabytes?
140_SANs_05
8/14/01
3:35 PM
Page 163
The SAN Design Process • Chapter 5 ■
Does it rack mount? Does it have a rack kit? Will it sit on a shelf?
■
If there is a management console, what type is it? Does it need to be permanently attached?
■
How many Fibre Channel interfaces will it have?
■
Do these interfaces exist, or do they need to be purchased?
■
If they exist, what are the make and model? If not, what kind will be purchased?
■
How many Ethernet interfaces will it have?
■
In what temperature range will it operate?
NOTE Obviously, some of these questions do not relate directly to the SAN deployment. However, they are generally relevant whenever making a large architectural change in a data center. For example, it is necessary to know what temperature a server can operate at in case the server is in a location where temperature control is an issue. In this case, adding a large number of switches might increase the room temperature beyond operating levels. As always, use your judgement about which questions to include in your interview, and which to skip over.
■
Will it need a telephone line for management?
■
Where will the node be physically located?
■
What is the firmware level?
■
For tape libraries, what is the capacity of each cartridge, number of cartridges the library can hold, number and speed of drives, and number of transports?
■
SCSI or Fibre Channel interface? What type of SCSI (wide/narrow, differential/single ended)?
163
140_SANs_05
164
8/14/01
3:35 PM
Page 164
Chapter 5 • The SAN Design Process
NOTE While it is possible to manage an entire fabric through a single Ethernet connection, this is not the method that Brocade currently recommends. You should plan on one Ethernet connection per Brocade switch, in addition to planning connections for hosts and other SAN devices. It is also advisable for the highest availability plan to balance switches across multiple electrical circuits, even if an Uninterruptible Power Supply (UPS) protects them.
Questions about facilities where hosts and storage will be located could include the following: ■
Who is responsible for this facility?
■
Are there any existing optical cables, and what type?
■
Is there sufficient electrical power?
■
What about cooling?
■
Is there enough rack space?
■
What is the network infrastructure?
■
Physical access? If the location is on an upper floor, is there a freight elevator?
Answers to questions about the SAN itself must be considered preliminary. They will indicate preconceptions that members of the core team have, but all members should be prepared to be flexible on these preconceptions as the SAN design process progresses. Questions about the SAN itself could include the following: ■
Are there any distance considerations? (For example, long cable runs between floors of a building, campuswide networks, or MAN/WAN connections.)
■
How many hosts will attach to the SAN?
■
How many storage devices will attach to the SAN?
■
If known at this point, do they require any-to-any connectivity? Alternately, are there groups of devices that need to communicate only among themselves?
140_SANs_05
8/14/01
3:35 PM
Page 165
The SAN Design Process • Chapter 5
Which SAN-Enabled Applications Do You Have in Mind? Will the SAN use a serverless backup application? How about clustering software? How about volume management? This category of software requires special attention because of its close ties to the SAN hardware you choose to build the solution. For example, if you plan to use vendor X serverless backup software, you must make sure that your backup hardware (tape libraries, Fibre Channel/ SCSI gateways, etc.) is supported.
Which Components of the Solution Already Exist? Any hardware or software that is already in place and that must be included in the solution will create points for you to build around.You must find out as many details as possible about everything in this category.When you are finished with the interviews, and conduct the physical assessment, you should personally inspect every piece of hardware.This will prevent surprises later in the process. Make sure that you find out exactly where all hardware is located, and how to access it. You must pay special attention to devices that already exist and already have Fibre Channel interfaces. Find out which kinds of HBAs are installed in hosts, and which driver revisions are installed on them. Find out code levels for RAID arrays and Fibre Channel tape libraries. Find out if upgrades to driver/code levels are planned or at least allowed.
NOTE You must know if each device is public loop, private loop, or full fabric. Some devices might even be SCSI and require additional hardware to bridge between SCSI and Fibre Channel.
If possible, you should not use private loop drivers on initiators unless the device does not support fabric drivers or is not easy to upgrade. Private loop hosts require special licenses, typically Brocade QuickLoop and Zoning. Find out if the existing devices are configured as full-fabric devices. If not, find out if their drivers support full fabric, or if they can be upgraded to full fabric.This is not intended to discourage incorporation of private loop devices into a fabric:
165
140_SANs_05
166
8/14/01
3:35 PM
Page 166
Chapter 5 • The SAN Design Process
QuickLoop and Fabric Assist exist specifically to enable this to occur. However, if a device can support full fabric, then integration into the SAN will be easier if it does so.
Which Components Are Already in Production? Components that are in production require special attention in two areas: ■
Duplicate equipment might be desired for testing.
■
The transition phase is more complex.
It is vital to know as much as possible about production systems that are going to transition onto the SAN.Therefore, somebody intimately familiar with and responsible for every such system should be included on the core team.
Which Elements of the Solution Need to Be Prototyped and Tested? For relatively simple solutions that involve only components already certified to work together, it might be that you do not have to do any testing at all. For example, if you are implementing a SAN-based solution on a Brocade SOLUTIONware document, you might feel that you need only to do minimal validation.This is opposed to a solution where no documentation or testing information exists, which generally requires extensive validation. For more complex solutions involving a large number of devices that might be from many different vendors, you might feel that every single element needs to be tested in combination before release to production can occur.You should get input on this from every member of the core team. If any team member feels that you should conduct inhouse testing on a component, you should strongly consider doing so.
What Equipment Will Be Available for Testing? Any existing equipment that is not in production, and any equipment that is going to be purchased specifically for this project might be good material with which to build a test bed. Existing equipment that is in production is not good to test with. If existing equipment already in production will be transitioned onto the SAN, it might be beneficial to budget for a representative sample of duplicate, nonproduction systems with which to prototype the solution. It is generally a good idea to have such systems available for testing in any case. It may also be possible to borrow systems to test with. In any case, it's probably worth asking your vendors for such loans.
140_SANs_05
8/14/01
3:35 PM
Page 167
The SAN Design Process • Chapter 5
Whether or not test equipment is available, you should research what testing third-party vendors or third-party organizations have already done. In this way, you will avoid duplicating their efforts. If you cannot get representative test equipment for an element that needs to be prototyped, it might be acceptable— and necessary—to rely entirely upon the work done by others to validate the solution. Again, with many solutions, this is a perfectly acceptable way to go. If you do not feel that inhouse testing is warranted, then you can save time and money by skipping the prototype and test phase. Just make sure that you have documentation certifying the solution before you make this decision.
How and When Are Backups to Be Done? You need to get a list of everything that relates to the system’s backups: ■
What backup hardware will be used?
■
What backup software will be used for each host?
■
Which storage arrays will be backed up by which tape libraries?
■
When will these backups occur?
■
How long can they take?
■
How much data needs to be backed up?
■
Will snapshots be used? How do they work?
■
Will split mirrors be used? How do they work?
What Will Be the Traffic Patterns in the Solution? You should produce a matrix showing every initiator-to-target communication expected in the SAN.This is necessary to determine performance characteristics, and to set up zoning on the fabric: ■
Which hosts will use a specific storage array?
■
Which hosts in a cluster will talk directly to each other over the SAN?
■
Which backup devices will be performing serverless backups?
■
Which arrays will they be backing up?
Create a table listing every device on the SAN that can act as an initiator in one column.This will include every host, every storage virtualization product,
167
140_SANs_05
168
8/14/01
3:35 PM
Page 168
Chapter 5 • The SAN Design Process
and every serverless backup server. It might include storage arrays, if they have data replication capabilities.Then put a second column next to it with all of the targets that each initiator will communicate with (Table 5.1). Table 5.1 Initiator-to-Target Mapping SAN Traffic Patterns Initiators
Targets
host1 host2
array3 array1 array2 tape1 array1 array1 array2 array1 array3 array4 array3
host3 host4 tape1 array3 array4
Note that some devices on a SAN can act as both an initiator and a target. If so, they will appear in both columns. See array3 and array4 in Table 5.1. This is how you would indicate that array3 and array4 perform data replication between them. You will not necessarily be able to build this table by interviewing one person; it will likely be developed over the course of the interview process, changed as the implementation takes place, and maintained for the life of the SAN.
What Do We Know about Current Performance Characteristics? Any devices that currently exist, and will be transitioned onto the SAN, are candidates for empirical performance testing. Create a second set of columns next to the traffic pattern columns, as shown in Table 5.2.You will need entries for peak utilization and sustained utilization. Obviously, you will only be able to enter current data for initiators that already exist, and already communicate with the same targets they will talk to after the SAN is complete.
140_SANs_05
8/14/01
3:35 PM
Page 169
The SAN Design Process • Chapter 5
Table 5.2 Current Traffic SAN Traffic Patterns
Current Peak
Current Sustain
Initiators host1 host2
MB/sec 10
MB/sec 5
50
10
host3 host4 tape1 array3 array4
Targets array3 array1 array2 tape1 array1 array1 array2 array1 array3 array4 array3
In this example, host1 and host3 already exist, and are already talking to array3 and array1, respectively. All of the other devices are to be added, are not talking to the same targets that they will be after the SAN is up, or performance data might simply be unavailable. If the owner of a system has already done this kind of analysis, you will simply transfer the numbers to your table. If not, you should work with the owner to get the performance information, as this might have a substantial impact on your SAN design.
Gathering Performance Data On almost any kind of system, some facility exists for measuring performance. More often than not, there will be multiple options for gathering disk I/O performance information. For example, on a Windows NT system, you might use the diskmon feature. You have to install this from the Windows NT Resource Kit. If you do not install diskmon, standard Windows perfmon will not have a disk monitoring tool. Alternately, you could install a package like Intel’s Iometer, and use that to generate a simulated load and measure performance.This tool is presently available as a free download from Intel’s Web site. Under Sun’s Solaris operating system, performance can be measured using the iostat utility, the GUI utility perfmeter, or one of a number of third-party utilities like Extreme SCSI.There are similar tools in every UNIX variant.We are providing
169
140_SANs_05
170
8/14/01
3:35 PM
Page 170
Chapter 5 • The SAN Design Process
examples for Solaris only, since the details of these commands will vary between every flavor of UNIX, and providing examples for every variant is impractical. Refer to the man pages for your particular version of UNIX for the exact syntax. There are also a number of options for generating loads under Solaris, ranging from the dd command, to—again—a utility like Extreme SCSI.
NOTE Tools like Iometer, dd, and Extreme SCSI should be used with care. It is tempting to use them to generate maximum load. A more useful test to run is to generate a representative load. Try to determine what your application will actually be doing in terms of read/write ratio, and total bandwidth consumption, and use these tools to generate that kind of load on the system.
In cases where performance data cannot be collected empirically—such as when the system in question does not exist yet—there is still hope. Most hosts are not capable of generating sustained load at full wire speed.They are generally going to be limited by other factors.These could include: ■
CPU speed Although Fibre Channel has much lower overhead than the TCP/IP stack, it still takes a fast processor to get near to full performance on a 1 Gbit/sec Fibre Channel link, simply because the processor will be busy running whatever task is actually generating the I/O. While almost all hosts now shipping have sufficiently fast CPUs, you also need to estimate how much of that CPU resource is taken up by other tasks the host is performing that do not result in disk I/O (such as running a TCP/IP stack). Moreover, many data centers have older CPU servers that might not be capable of running at 1 Gbit/sec even without taking these tasks into consideration.
■
PCI bus speed Fibre Channel full duplex is 200 MB/sec. A 32-bit 33 MHz PCI bus can only sustain about 120 MB/sec. A 64-bit 33 MHz or 32-bit 66 MHz PCI bus can handle about 240 MB/sec, and a 64-bit 66 MHz bus can handle about 480 MB/sec. Even on the higher rate buses, you must bear in mind that it is a shared bus. If you put two Fibre Channel HBAs onto a bus that can handle 240 MB/sec, that will be the total possible full-duplex speed for both HBAs.Therefore, you would on
140_SANs_05
8/14/01
3:35 PM
Page 171
The SAN Design Process • Chapter 5
average get 120 MB/sec out of each interface. For example, this could— in a balanced read/write environment—mean that you get only 60 MB/sec of read performance out of each card. Also bear in mind that there may be other cards on the bus taking up some of that bandwidth. ■
HBA speed Although designed to work on a 1 Gbit/sec SAN, many HBAs cannot achieve or at least cannot sustain full 1 Gbit/sec transfers. Newer HBAs typically have better performance. Older HBAs might only be able to achieve 60 MB/sec, regardless of the other possible issues.
■
RAID controller speed Many RAID controllers cannot sustain 100 MB/sec per interface on all interfaces simultaneously. Some barely operate at 30 MB/sec per interface, which is more than acceptable for many applications! Finding out the limits of your RAID array should be as simple as calling the vendor’s support channel. Of course, you might also check third-party testing results such as those done by many industry magazines for an unbiased opinion.
■
RAM quantity and speed If your system is short on RAM, it might spend a lot of time paging. If it does, performance will be substantially degraded.
■
Disk seek time If your application does a lot of random I/O, the disk heads will have to jump all over the platform. Since disk seek time is an order of magnitude or more slower than a Fibre Channel link, you might have to allocate substantially less bandwidth for random I/O applications like a file server than for sequential I/O applications like a video server or decision support system.
■
Application overhead This ties into the CPU-limit factor. How much CPU do you have, and how much of it is free for handling I/O?
■
Write speed of tape device Most tape drives cannot come anywhere near 100 MB/sec. It is usually sufficient to ask a vendor for performance data in the case of tape drives, although optimistic compression ratios can inflate the performance numbers they provide.
In addition, if anything is known about the application that is running on the host, you might be able to make a good guess about how much load it will even try to place on the disk subsystem. For example, if you know that the host is an intranet Web server, and that it receives only 500 hits a day, you can safely guess that its I/O requirements will be minimal.
171
140_SANs_05
172
8/14/01
3:35 PM
Page 172
Chapter 5 • The SAN Design Process
Once you have collected your best empirical or estimated numbers for each factor, use the lowest common denominator approach to estimate the maximum bandwidth that the system could need.You can guarantee that the overall system will not outperform its weakest link. Also note that on systems with multiple HBAs, I/O load might be distributed across these HBAs. Achieving active-active distribution across HBAs might require third-party applications like the VERITAS Dynamic Multipathing software,Troika’s HBA driver, or one of the storage vendor’s dual-path products. If this is the case, you might estimate that each HBA will usually have a fraction of the total load. In a dual-fabric, active/active HBA architecture, each HBA normally has 50 percent of the total load. If a system is capable of sustaining 70 MB/sec, then each HBA will sustain 35 MB/sec. Note that this might change during system maintenance if you shut down one path, and the remaining path could then take on the full 70 MB/sec, so the design should incorporate the worst-case scenario. It is usually also good practice to add some padding to the top of this estimate (perhaps 10 percent) to allow for the unexpected.
NOTE Unlike physical-disk counter data, logical-disk counter data is not collected by the NT operating system by default. To obtain performance counter data for logical drives or storage volumes, you must type diskperf -yv at the command prompt. This will cause the disk performance statistics driver used for collecting disk performance data to report data for logical drives or storage volumes. By default, the NT operating system uses the diskperf -yd command to obtain only physical drive data. For more information about using the diskperf command, type diskperf -? at the command prompt.
What Do We Know about Future Performance Characteristics? Performance numbers change over time. Consider a customer database for a catalog retail company. Perhaps you will install the SAN in February, because this is your slow month of the year, and you can get the necessary downtime.You might know that the database host will start talking to its storage array(s) at a sustained rate of 5 MB/sec during the business day, with a peak of only 10 MB/sec.
140_SANs_05
8/14/01
3:35 PM
Page 173
The SAN Design Process • Chapter 5
However, when the Christmas season comes along and your business picks up, you might move to a 50 MB/sec sustained rate, peaking at 70 MB/sec. Because of the potential for substantial changes in performance requirements over time, it is essential to plan for both current and projected performance. Most of this might be educated guesswork, since many of the systems you are going to deploy might not yet exist. Again, you will need to come up with numbers for both sustained traffic and peak traffic for each communication. Also try to determine what days/times peak performance will occur.This will be added to your table (Table 5.3). Table 5.3 Adding Traffic Projections SAN Traffic Patterns
SAN Peak Performance
SAN Sustained Performance
Initiators host1
Targets array3
Initial 10
Expected 10
Initial 5
Expected 5
host2
array1 array2 tape1 array1
0 0 20 50
70 70 20 50
0 0 0 10
50 50 0 20
array1 array2 array1
0 0 0
90 90 20
0 0 0
50 50 0
array3
0
20
0
0
array4 array3
10 5
30 5
5 0
5 0
host3 host4 tape1
array3 array4
Peak Times Initial M–F 8a–5p
Expected same
M–F 8a–5p
+ Sa 10a–4p
Sa same 5p–9p Sa same 9p–11p
Again, you can only enter data for systems about which you can make an educated guess. If you know about what the peak traffic could be based only on the limitations of a system, you might not have any way of guessing when this would occur.You should also enter projected data for systems that you know that you will add later. In Table 5.3, host2 and the application it is running might not exist yet, so every piece of data about that system is pure guesswork. Let us say that host2 is a Return Merchandise Authorization (RMA) system, and your rapidly growing
173
140_SANs_05
174
8/14/01
3:35 PM
Page 174
Chapter 5 • The SAN Design Process
company has never had an RMA system before.You might not be able to reliably guess when customers are going to call in with RMA requests most often, or even how many RMAs you are going to get in a given day.The best you can do is determine what performance the hardware and software you are installing could reasonably run at, and design the SAN to support it all the time it could be in use.While this approach might result in over-engineering your network, this is better than the alternative. During future design phases, you can alter the SAN design to adjust or scale back the design accordingly, as well as incorporate other additions and changes. For backup devices, peak usage will always correspond with your backup schedule.This will usually not correspond with peak usage of the rest of the system.This is particularly useful knowledge when planning an ISL architecture, because you can often count on having low nonbackup-related utilization of ISLs during backup windows. An obvious exception to this is a SAN that is used solely for performing LAN-free backups.
How Much Downtime Is Acceptable to Production Components During Implementation? It will likely be necessary to shut down some existing production devices during implementation, to ensure a safe transition onto the SAN. For example, you might have to shut down a host to install an HBA. Determine how much downtime is acceptable for each host, and at what times this can occur. Generally, you should try to schedule more downtime than you think you need to ensure that any unforeseen issues that arise during the implementation can be handled within the downtime window.
How Much Downtime Is Acceptable for Routine Maintenance? How Much Downtime Is Acceptable for Upgrades and Architectural Changes? These two questions are intimately related, because—to an end user—there is really no difference between downtime to a production system for maintenance, and downtime for an upgrade. Once systems are in production, you will want to keep them running as much as possible. Many upgrades can be accomplished with zero downtime by using a doubleor triple-redundant fabric architecture. No matter how well you plan the upgrade and maintenance processes beforehand, you will need to shut down specific hosts
140_SANs_05
8/14/01
3:35 PM
Page 175
The SAN Design Process • Chapter 5
on occasion. For example, you might want to upgrade an HBA driver, which would typically require a reboot.
NOTE Wherever possible, a redundant fabric architecture should be used. This will ensure the best performance and reliability, and will simplify maintenance tasks. In a redundant fabric architecture, every host has at least two paths to every storage device it connects to, and these paths traverse two completely unconnected fabrics. While it might appear on the surface to be more expensive, if hosts are to be dual-attached anyway, it is actually less expensive to attach them to two separate fabrics than to use one larger fabric, or a director-class switch. This does not even include the downtime ROI calculation, which, in high-availability environments, will usually overshadow the entire cost of the SAN. More details about redundant and resilient fabrics are provided in Chapter 7.
You should therefore determine in advance when you will be able to schedule downtime for every host and storage array, and for the fabric itself.You might not have to use every scheduled outage, but having them available to you when you do need them is essential. One way to do this is to make a list of applications and services provided by the hosts on the SAN, and determine an owner for each.Take your list of SAN devices and map these devices to the applications and services they affect.This will provide a mapping of application/service owners, who are typically responsible for scheduling downtime, to devices that typically require downtime. Have each owner approve the downtime calendar for each device that affects his or her service. The mapping of owners to devices should be kept up to date as changes in personnel, applications, and/or SAN infrastructure occur.
When Do You Need Each Piece of the Solution to Be Complete? Once you have a table detailing which of the initiators communicate with which targets, you can begin to create a timeline for the project. Other members of the core team will tell you something like, “the customer database application must be online by mid-June.” It is your task to define which SAN components you
175
140_SANs_05
176
8/14/01
3:35 PM
Page 176
Chapter 5 • The SAN Design Process
need to accomplish this, and to develop a timeline for adding these components that meet their requirements.
Summary List of Questions This is a high-level list of some of the questions that should appear on a SAN design interview form: ■
What overall business problem are you trying to solve?
■
What are the business requirements of the solution?
■
What is known about the nodes that will attach to the SAN?
■
Which SAN-enabled application do you have in mind?
■
Which components of the solution already exist?
■
Which components are already in production?
■
Which elements of the solution need to be prototyped and tested?
■
What equipment will be available for testing?
■
How and when are backups to be done?
■
What will the traffic patterns in the solution be?
■
What do we know about current performance characteristics?
■
What do we know about future performance characteristics?
■
How much downtime is acceptable to production components during implementation?
■
How much downtime is acceptable for routine maintenance?
■
How much downtime is acceptable for upgrades and architectural changes?
■
When do you need each piece of the solution to be complete?
Conduct a Physical Assessment You should now have the location of every piece of hardware that currently exists. In addition, you should know where each piece of hardware in the eventual SAN will be located. Look at each piece of hardware. Make sure that it does exist, and has all necessary pieces to function.This could include things like power cords, keyboard,
140_SANs_05
8/14/01
3:35 PM
Page 177
The SAN Design Process • Chapter 5
mouse, monitor, Ethernet card, Ethernet cable, HBAs, and Fibre Channel cables. Note the physical dimensions of the hardware, and its power/cooling requirements. Does it rack mount? Does it have a network interface? How many Fibre Channel interfaces does it have? How much does it weigh? You should already have this information from the interview process, but you should verify that the information you were given is correct. Go to each location where SAN equipment or nodes will be installed, and again check to see that your information was correct. Notice how the equipment will fit into the space available. Notice how the equipment will enter the building.You should also have a meeting with the person in charge of the facility to discuss power, cooling, and equipment locations.
Analyzing the Collected Data Now that you have collected information from all key stakeholders in the project, and verified the accuracy of this information, you will analyze it to determine the characteristics of the required solution.When you have completed this process, you will have a list of technical requirements, and an ROI analysis to justify the project.
Processing What You Have Collected You have a matrix detailing communication between nodes. Attempt to group the nodes by communication patterns.The purpose of this is to determine the amount of known locality in the SAN. Locality of reference is a concept prevalent in many areas of computer science, from disk drive construction to LAN design. Locality is important in SAN design because if you can localize traffic into specific areas of a SAN, you directly improve the SAN’s performance and reliability.This will allow a more cost-effective SAN design as well, preventing overdesigning the network to handle nonexistent cross traffic. Locality is discussed in greater detail in Chapter 7. A SAN with a great deal of known locality might be constructed out of many separate fabrics, with no ISLs whatsoever. A SAN with little or no known locality might require a high-performance ISL architecture (Table 5.4).
177
140_SANs_05
178
8/14/01
3:35 PM
Page 178
Chapter 5 • The SAN Design Process
Table 5.4 Initiator–to-Target Mapping for Locality Example SAN Traffic Patterns Initiators
Targets
host1 host2
array3 array1 array2 tape1 array1 array1 array2 array1 array3 array4 array3
host3 host4 tape1 array3 array4
In Table 5.4, array3 would be grouped with host1, tape1, and array4. None of those devices will need to communicate with any of the other devices.They could be grouped onto a single switch, or even put onto a totally separate fabric. You might find it helpful to do the grouping in a diagram. For another example, look at Figure 5.2. Figure 5.2 SAN Diagram without Grouping
Serv
ge
Stora
ers
SAN
140_SANs_05
8/14/01
3:35 PM
Page 179
The SAN Design Process • Chapter 5
Nothing is known about the communication patterns in this SAN. Consequently, there is no way to optimize ISLs for performance. After grouping the initiators with their targets, the SAN diagram could look something like Figure 5.3. If you look carefully, you will notice that there are only 12 connections into this SAN. If there are fewer connections than there are ports in your switches, you do not really need to go through the grouping exercise because localization of traffic will happen automatically. It is only useful if you will be using ISLs; however, as most systems scale well past the size of the largest switches available, it will be a frequent exercise. For the purposes of making the examples more readable, we will just assume that they are all dealing with a subset of the devices that the SAN will support. Figure 5.3 SAN Diagram with Simple Grouping
SAN Group 1 Group 2 Group 3 Group 4
Making a diagram such as this will allow you to see at a glance what the communication patterns for your SAN are. This example is simplistic, and in large SANs, there will likely be conflicts. When you cannot effectively group all of the communication patterns, you should focus on grouping faster performing devices. For example, if you find that
179
140_SANs_05
180
8/14/01
3:35 PM
Page 180
Chapter 5 • The SAN Design Process
the bulk of traffic will be between host1, array3, and array4, these could be grouped separately from tape1 and host2 if necessary.This could happen if you find that there are so many interrelationships that you end up with very many devices, but very few very large groups.The grouping technique does not help for performance if you only have one big group. It could also happen if you have a few devices that are shared by a great many devices, such as a large RAID array in a storage consolidation solution. Another way to combat this “group growth” problem is to account for multiple interfaces on storage arrays. Let us say that you have a redundant fabric architecture.Your RAID array has eight interfaces, and each host will access only two of them—one interface on each fabric. List each interface on the array separately in your traffic pattern table.Then, you associate servers or groups of servers with specific interfaces.With the array listed as a single entity, a diagram of the communication could look something like Figure 5.4. Figure 5.4 SAN Grouping Diagram with Single-Entity Arrays Server Group
Server Group
SAN A
SAN B RAID Array1
Server Group
Server Group
If, however, you separate the interfaces, your diagram could look more like Figure 5.5. You can indicate that a device crosses groups but does not need much in the way of performance by varying the line color, weight, or pattern. Figure 5.6 shows that the tape robot crosses all groups, but does not need much bandwidth.
140_SANs_05
8/14/01
3:35 PM
Page 181
The SAN Design Process • Chapter 5
Figure 5.5 SAN Grouping Diagram with Separated Interfaces Server Group 1
Different array controllers attaching to different groups.
SAN A
Server Group 2
Group 1 Group 2 Group 3 Group 4
SAN B Group 1 RAID Array1
Group 2 Group 3 Group 4
Server Group 3
Server Group 4
Figure 5.6 SAN Grouping Diagram with Tape Robot Addition Server Group 1
RAID Array1
SAN A
Server Group 2
Group 1 Group 2 Group 3 Group 4
SAN B Group 1 Group 2 Group 3 Group 4
Server Group 3
Tape1 One Interface Going to Multiple Groups
Server Group 4
181
140_SANs_05
182
8/14/01
3:35 PM
Page 182
Chapter 5 • The SAN Design Process
If you are able to make relatively small performance groups, your SAN will benefit greatly from applying the principal of locality. For now, you simply need to be able to determine the category of architecture you will require: one that has lots of known locality (has well-defined performance groups), or one that does not.This will affect how many switch ports you need to allot for ISLs. If traffic is localized within an area of the SAN, it will obviously not need to make use of ISLs leaving that area. In this case, you will be able to get superior performance even with far fewer ISLs, resulting in more ports available for servers and storage.
Establishing Port Requirements Now you will determine how many switch ports you will need to purchase. (This is a general estimate for calculating ROI; it might be a bit more or less than your final estimate.) Take the ports you found out about during the interview process. Make sure that you account for all ports on each node. Some RAID arrays have many ports, and many hosts have at least two HBAs. Add up these ports to get the total number of exposed ports your SAN will require.You will then divide this by the number of different fabrics you will be using. If you have dual-redundant fabrics, you will divide by two. If you have triple-redundant fabrics, divide by three, and so on.This will give you the number of required exposed ports per fabric.The number of “overhead” ports you must allocate for ISLs and for unused ports will depend on several factors: ■
The total number of required ports per fabric.
■
The amount of known locality.
■
Your need to manage all switches as a single entity.
■
The physical layout of your SAN—any MAN/WAN connections, or intra-building campus connections, or intra-floor building connections— might dictate use of additional ISLs and less than perfect utilization of the ports on each switch.
■
Your applications’ expected performance characteristics.
■
The rate of expected growth in port count of the fabric.
■
Your maintenance policies regarding port usages on network devices. For example, you might require that a certain number of ports be left available for expansion or troubleshooting during the course of normal operation.
140_SANs_05
8/14/01
3:35 PM
Page 183
The SAN Design Process • Chapter 5
Simple Case If the number of required exposed ports is less than the number of ports on a single switch, you will generally need zero ports for ISLs. In this case, you will require one switch per fabric. However, as larger switches utilize more hardware internally to connect the higher number of user ports, a decision might need to be made between using a larger switch versus utilizing a network of smaller ones. The appropriate decision will depend on performance requirements, budget, and design factors. In addition, if you have made small performance groups that have no components in common, you might be able to localize traffic 100 percent, and require no ISLs.You would have many small, unconnected SAN islands if you follow this approach. One reason not to use isolated islands is that requirements change. Someday you might need access between islands at a moment’s notice. A robust architecture can achieve your immediate connectivity requirements, and give you the flexibility to handle change as well. You will require each fabric to be a network if this is not the case, or if you wish to design in flexibility to your configuration.You will have to reserve port count for these. Simple case requirements include the following: ■
Fewer ports required than exist on a single switch, or…
■
Each performance group is well defined and smaller than the number of ports on a single switch.
■
Future requirements for growth and change are minimal.
Assume that you have two 16-port arrays (32 storage ports total), 10 dualHBA servers (20 ports), and two single-port tape libraries (two ports).Your total port count is 54. However, assume further that you are using a dual-redundant SAN architecture.Your port count per fabric is 27.You are building the fabric out of 16-port switches. It is possible that some ISLs are required.You will need to determine how many are needed.
Variant A With a relatively small fabric like this and relatively high locality, you can assume that you will have about 14 free ports per switch.Two switches with two ISLs between them will yield 28 ports per fabric.You are using a dual-redundant architecture, so there will be two fabrics, for a total of four switches.Your grouping diagram will look like Figure 5.7.
183
140_SANs_05
184
8/14/01
3:35 PM
Page 184
Chapter 5 • The SAN Design Process
Figure 5.7 Determining ISL Requirements for Variant A RAID Array1
Server Group 1 (5 Hosts)
A1-1 A1-2
4
SAN A
A1-3
4
Group 1
A1-4 4
A2-1
4
5
Group 2
4
5 5
SAN B
4
Group 1
A2-2
4 4
A2-3 A2-4
Group 2
5 Server Group 2 (5 Hosts)
RAID Array2
This grouping would result in an actual implementation resembling Figure 5.8. Figure 5.8 Variant A Implementation RAID Array1
Server Group 1 (5 Hosts)
SAN A
A1-1
4
A1-2
5
4
A1-3 A1-4
4
5
4 4 A2-1 A2-2 A2-3 A2-4 RAID Array2
4
5
4 5
4 SAN B
Server Group 2 (5 Hosts)
Variant B If you decide that you cannot guarantee the localization of traffic for some reason, grouping will not help. Assuming also that you have a requirement for high performance between the switches, you would add two ISLs per switch to
140_SANs_05
8/14/01
3:35 PM
Page 185
The SAN Design Process • Chapter 5
the estimate, for a total of about four ISLs per switch.Your architecture might look Figure 5.9. Figure 5.9 Adding ISLs for High Performance in Variant B 36 Ports per Fabric (Balance storage and hosts across the 3 switches for best performance.) RAID Array1
Server Group 1 (5 Hosts)
SAN A
A1-1
4
A1-2
5
4
A1-3 A1-4
4
5
4 4 A2-1 A2-2 A2-3 A2-4 RAID Array2
4
5
4 5
4 SAN B
Server Group 2 (5 Hosts)
The same technique can be applied to any SAN, no matter how complex. In fact, the larger the SAN, the greater the benefits will be from grouping traffic.
Moderate Case If the required exposed port count is about double or triple the per-switch port count, and some locality is known, you will be able to use very few ISLs. In this case, estimate two ISLs per switch. Let us say that you need 26 ports, and you are using 16-port switches.Two ISLs per switch means that you actually get 14 ports per switch.Two switches will give you 28 ports, so you would budget for two switches per fabric, or four switches total. Moderate case requirements include the following: ■
No more than three times as many ports are required than are present on a single switch.
■
Performance groups are reasonably well defined. Some locality is known.
■
Future requirements for growth and change are minimal.
185
140_SANs_05
186
8/14/01
3:35 PM
Page 186
Chapter 5 • The SAN Design Process
NOTE The low port count/high locality/low ISL count configurations work well for either two or three switches. Two switches would be cascaded together with two ISLs, with 16-port switches yielding 28 ports. Three switches would be connected in a ring, supporting about 40 devices. If you are over that limit, a four-switch full mesh can support about 50 devices. The full-mesh architecture does not scale well beyond that point, and none of these work well if you have performance groups with more than 13 or 14 members. It is feasible to build ring or partial-mesh topology fabrics with higher port counts, but it is generally better to use a core/edge topology for higher port count solutions. These topologies are explained in detail in Chapter 7.
Complex Case If you need more ports than one of these configurations will handle, you will need to allocate about four ISLs per switch.You might use fewer than four ISLs on some switches, and perhaps nothing but ISLs will be present on other switches. In the complex case for port count estimates, the intent is to average the ISL requirements. Until a detailed architecture is developed, you will have to make general estimates for a few things. If you have any distance requirements, add two ISLs per switch. If you have very high-performance requirements, and very little known locality, add two ISLs per switch. Take the estimated number of ISLs per switch (I) and subtract it from the number of ports per switch (PS). Divide the total required ports per fabric (P) by this number and round up.This is the estimated number of switches (S) that you need to budget for. For estimating complex SAN switch counts, S=P/(PS – I). For example, if you have a need for 30 ports per fabric (P=30), are using 16port switches (PS=16), and each switch will use about two ISLs (I=2), then the number of switches you estimate needing per fabric is 30/(16–2).This is 2.14, which rounds up to 3. If you have a single fabric, this is the number of switches you should budget for. If you have a dual-fabric SAN, you should budget for six switches. Complex case requirements include the following:
140_SANs_05
8/14/01
3:35 PM
Page 187
The SAN Design Process • Chapter 5 ■
Any number of exposed ports might be required.
■
Performance groups might or might not be defined.
■
Future requirements for growth and change are significant.
Preparing an ROI Analysis In any business transaction, it is important to understand the economic benefits or the Return On Investment (ROI) that your company will receive. Preparing an ROI analysis for your SAN project will show how your company will not only return the capital investment, but also save additional money as well in time, management, and other efficiencies. During the interview process, you made a list of all of the equipment that you would need to purchase.To begin the ROI analysis of your SAN, determine which components are specific to the SAN project. For example, if your company will need to buy additional storage arrays whether or not a SAN is used, these would not be included on the expense side of the analysis. If the SAN is expected to prevent you from having to buy an array, this cost savings would go onto the benefit side of the analysis.You should include any hardware you intend to buy for testing that will not be used elsewhere. When accounting for staff time spent on the project, make sure that you only charge the project for time spent beyond what would be spent by not building the SAN. If you are expected to save staff time in the long run, apply this to the benefit side.Your ROI analysis will be a living document, and will be updated as the SAN project develops.
The Return On Investment Proposition Technical justifications for SAN infrastructure deployments can often be made more credible by adding an ROI analysis for the proposed implementation. Follow the guide in the following sections to produce an ROI analysis based on SAN solutions to particular problems.
Step One: Pick a Theme or Scenario Most implementations have a purpose.That purpose could be a server or storage consolidation to improve infrastructure usage and gain economies of scale, ensuring storage and server resources are utilized in the most cost-effective manner. High-availability clustering can improve the availability of mission-critical applications, thus ensuring business continuance and the cost saving associated
187
140_SANs_05
188
8/14/01
3:35 PM
Page 188
Chapter 5 • The SAN Design Process
with it. SAN-based backup deployments improve data integrity by performing backups and restores more efficiently and quickly, again saving in business continuance time and effort.
Step Two: Identify the Affected Infrastructure Components Most SAN deployments will focus on affected servers. Servers can be grouped according to the applications they run or the functional areas they support. Examples of application groupings include Web servers, file and print servers, messaging servers, database servers, and application servers. Functional support servers might include financial and personnel systems or engineering applications. Once the server groups are known, get the characteristics of servers in each group. For example, if your solution fits into a storage consolidation theme, you should consider factors such as: ■
Amount of attached disk storage
■
Storage growth rates
■
Storage space reserved for growth (headroom)
■
Availability requirements
■
Server downtime and an associated downtime cost
■
Server hardware and software costs
■
Maintenance costs
■
The administration effort required to keep the servers up and running
Step Three: Identify the SAN-Enabled Benefits The scenario approach allows you to focus more closely on the benefits. Server and storage consolidation, for example, will concentrate on benefits accrued from more efficient use of server and storage resources, improved staff productivity, lower platform costs, and better use of the infrastructure. Simply take the list of characteristics you developed in step two, and show how a SAN can provide benefits in those areas. Establishing specific cost savings is one of the two key elements in the ROI process, so be sure to look hard for every area of benefit.
Step Four: Identify the SAN-Related Costs Determining the costs associated with the scenario involves identifying the new components specifically required to build and maintain the SAN.These can
140_SANs_05
8/14/01
3:35 PM
Page 189
The SAN Design Process • Chapter 5
include software licenses, switches, Fibre Channel HBAs, optical cables, and any service costs associated with the deployment. Be careful to include only those items that relate directly to the SAN implementation.This is the second key element in the ROI process: if you do not correctly estimate expenses, the ROI might be substantially better or worse than your estimate.
Step Five: Calculate the ROI There are several standard ROI calculations in common use, such as net present value (in dollars), internal rate of return (as a percentage), and payback period (in months). Briefly, these can be defined as: ■
Net Present Value (NPV) A method used in evaluating investments where the net present value of all cash flows is calculated using a given discount rate.
■
Internal Rate of Return (IRR) A discount rate at which the present value of the future cash flows of an investment equal the costs of the investment.
■
Payback Period The length of time needed to recoup the cost of a capital investment on a nondiscount basis.
Detailed explanations of these techniques and how to use them can be found in most accounting textbooks. It is likely that your company has a preferred method for calculating ROI.You should determine which method this is, and if there are standard forms for presenting your analysis. Asking your accounting department might be a good first step. This approach to calculating ROI allows you to focus on a particular project or infrastructure-based problem. It allows you to reduce deployment risk by deploying SANs in phases by scenario. Deploying by scenario will keep investments limited to the solution at hand and create an investment base for future deployments.The initial investment will improve the ROI on other scenarios by reducing some of the investment required to deploy them.
189
140_SANs_05
190
8/14/01
3:35 PM
Page 190
Chapter 5 • The SAN Design Process
The Rest of the Process and the Repetition of the Cycle Now you have the following documents: ■
Detailed results from the interview process, which define what the SAN project needs to accomplish.This includes: ■
A technical requirements document
■
A timeline for accomplishing the tasks associated with implementing the SAN
■
A list of everything that you will need to buy to make the project work
■
A rough idea of how the SAN will be designed.
■
An ROI analysis to justify continuing with the project.
These will be used and maintained throughout the life of the SAN.The timeline will be the framework in which all activities in the SAN’s lifecycle will reside. In later chapters, you enter the architecture development phase and will use these documents to develop a detailed architecture for your SAN.This will in turn be used to develop a test plan.These documents will be used in the approval process for implementation, and will be kept up to date during the maintenance phase as part of the SAN’s documentation set. If any major changes to the SAN are needed, the lifecycle will be repeated and another set of documentation will be produced.
140_SANs_05
8/14/01
3:36 PM
Page 191
The SAN Design Process • Chapter 5
Summary The SAN design process consists of seven phases, which are cycled through as needed throughout the life of your SAN. Data collection and analysis together define the requirements of your SAN.These requirements feed into the architecture development process to produce a SAN design blueprint. After you have a plan in place for your SAN, you must test certain components to ensure that it is working the way you thought it would, before you can begin to transition and release it into production. Once the SAN has entered production, it falls into an ongoing maintenance phase, and continues in that phase until a change occurs that causes the cycle to repeat. The first two phases (data collection and analysis) are critical to the health of the SAN. Simply put, if the information on which the design is based is incomplete and/or inaccurate, the design will be incorrect. Data collection consists of a series of interviews, collecting the answers into a meaningful format (a technical requirements document), and verifying the accuracy of the collected data. It is imperative that all key stakeholders in the SAN project be included on the interview list. While listed as a separate phase, data analysis actually coincides with data collection.The objective of the analysis phase is to turn the raw data, which is generally in the form of business requirements, into a more technical format—the technical requirements document. Some of this occurs “on the fly” during the interview process. However, certain tasks are done after the interviews are complete. For example, detailed port count and performance requirements are generated “on the fly,” and an ROI proposition is created after the fact. Once the requirements of the SAN are well defined, the remaining phases can take place. These phases are covered in subsequent chapters.
Solutions Fast Track Looking at the Overall Lifecycle of a SAN The SAN design process is a cycle. This process consists of seven phases:
1. Data Collection 2. Data Analysis
191
140_SANs_05
192
8/14/01
3:36 PM
Page 192
Chapter 5 • The SAN Design Process
3. Architecture Development 4. Prototype and Test 5. Transition 6. Release to Production 7. Maintenance Whenever there is a fundamental change to the SAN, the cycle
should repeat.
Conducting Data Collection Data collection is the foundation on which a SAN is built. You should interview everybody who has an interest in the project. During the interview process, create a technical requirements document.
Analyzing the Collected Data There are several things that you need to get out of data analysis:
— The number of different fabrics that will make up the SAN solution — The port count and performance characteristics of each fabric — An estimate of the hardware required to meet these requirements You might be able to localize traffic for better performance if you can
create well-defined groups. Prepare an ROI proposition to justify your SAN project.
140_SANs_05
8/14/01
3:36 PM
Page 193
The SAN Design Process • Chapter 5
Frequently Asked Questions The following Frequently Asked Questions, answered by the authors of this book, are designed to both measure your understanding of the concepts presented in this chapter and to assist you with real-life implementation of these concepts. To have your questions about this chapter answered by the author, browse to www.syngress.com/solutions and click on the “Ask the Author” form.
Q: Once I have designed my SAN, shouldn’t it be done? I don’t want to have to keep reinventing the wheel!
A: Yes and no. After a SAN enters production, it is “done” until you want to change it in a fundamental way. As long as you are happy with leaving your SAN the way it is, there is no reason why you would have to repeat the design cycle. Simply adding a new storage array does not require a repetition of the cycle. Moreover, events that do cause the cycle repeat might cause it to repeat relatively quickly. For example, if you decide to go through the design process because you are adding a new type of storage array to the SAN, and want to validate that doing so won’t break anything, you will be able to take a fast track through most of the process. After all, adding this device will not by any stretch of the imagination require that you change your fabric topology, or affect much of your SAN architecture.
Q: Every end user in my company is a stakeholder in the SAN. Do I need to interview everybody?
A: No. It is true that everybody who uses a system is a stakeholder in that system. However, we mean something a little less broad.When we refer to a stakeholder, we mean somebody whose job revolves around taking care of one or more of the systems that will attach to the SAN.This can include systems, database, and storage administrators, as well as other technical people. It can also include people responsible for the data that resides on these systems. For example, a manager responsible for a call center at a phone-in catalog company might be a key stakeholder in the SAN, because he or she is responsible for the data entered into that company’s business system—which is attached to the SAN.Why is this person a key stakeholder? Because he or she might have something to say about the availability and performance requirements of the system.When in doubt, try to include anybody on the
193
140_SANs_05
194
8/14/01
3:36 PM
Page 194
Chapter 5 • The SAN Design Process
team who wants to be there. It is usually better to have more data than you need, rather than less.
Q: Do I need to wait until data collection is complete before beginning data analysis?
A: Actually, the data collection and analysis phases are most effective if there is some degree of overlap. If you have analyzed data from the first interview when you go into the second, you will be able to better understand the answers, and might also be able to direct the line of questioning along more useful lines. Be careful not to develop firm convictions too early on, though. Always approach SAN design scientifically. Never start an interview with a firm preconception of the outcome! Collection and analysis are divided into two phases because some of the analysis naturally occurs after all data collection is complete. For example, you can’t prepare an ROI proposition until you have a fairly complete picture of what the SAN will need to accomplish, and some idea of the technical infrastructure that will be involved.
140_SANs_06
8/14/01
3:37 PM
Page 195
Chapter 6
SAN Applications and Configurations
Solutions in this chapter: ■
Configuring a High-Availability Cluster
■
Using a SAN for Storage Consolidation
■
LAN-Free Backup Configuration
■
SAN Server-Free Backup
■
Making Your Enterprise Disaster Tolerant
Summary Solutions Fast Track Frequently Asked Questions
195
140_SANs_06
196
8/14/01
3:37 PM
Page 196
Chapter 6 • SAN Applications and Configurations
Introduction This chapter covers configurations for some of the most common Storage Area Network (SAN) applications.The surest route to SAN success is to base your installations on proven configurations—equipment layouts that have been found through practice and repeated refinement to be best suited for the desired application. In this chapter, we review the major SAN configuration types and discuss the advantages of each.This chapter does not go into specific detail (we do not give specific vendor names or driver revisions) in order to keep the material generally useful.We give you enough information to understand and set up your own cluster, but we do not identify a specific storage or Host Bus Adapter (HBA) vendor for use in that cluster; thus, the material will be useful to users of any storage or HBA. If you do desire low-level detail that identifies specific configuration information such as vendor, model, and revision level, Brocade SOLUTIONware might be helpful.
Using Brocade SOLUTIONware Brocade provides a number of pretested configurations on its Web site (www.brocade.com) for administrators who wish to configure their own SANs, and integrators who want to have a head start on developing solutions for general deployment. Brocade SOLUTIONware guides can be used to help define the basic configurations for your SAN and can be modified to fit your solution. SOLUTIONware guides offer very specific model and part numbers for configurations similar to the solutions given in this chapter. The solutions are specific as to the model of storage, switch, and HBAs, but can be extended to similar models. Thus, a SOLUTIONware can be used either as a “cookbook” for building a SAN identical to the one discussed in the paper, or as a “reference solution” from which other, similar solutions can be derived.
Configuring a High-Availability Cluster High-availability (HA) clusters are used to support critical business applications. They provide a redundant, fail-safe installation that can tolerate equipment,
140_SANs_06
8/14/01
3:37 PM
Page 197
SAN Applications and Configurations • Chapter 6
software, and/or network failures, and continue running with as little impact upon business as possible. HA clusters have been in use for some time now. However, until the advent of Fibre Channel, they were very limited in size and reliability.This is because clusters require shared storage, and sharing Small Computer Systems Interface (SCSI) storage subsystems is difficult and unreliable. In fact, sharing a SCSI device between more than two initiators is completely impractical due to SCSI cabling limitations, and SCSI’s poor support for multiple initiators.Thus, clustering technology has been greatly enhanced by the network architecture of SANs. SANs provide ease of connectivity, and the ability to interconnect an arbitrarily large number of devices. Because of this, SANs can support more than just dual failover, and can be easily extended to support many-to-one failover configurations. The advantages of an HA cluster fall into three categories: availability, manageability, and scalability. Availability is the capability of a cluster to be tolerant of hardware, network, or software errors. In short, it is the capability of a system to “stay up.” Clustering software automatically detects error conditions, and restarts or transfers applications from one server to another. Little or no downtime will result from these problems, as the HA software can be configured to automatically act to correct them. Manageability is the set of processes with which an administrator keeps a system running. HA clusters enhance manageability by allowing all servers in each cluster to be managed as a group. Moreover, when software and/or hardware needs to be upgraded, each node in a cluster can be upgraded separately without taking important applications or the system offline—a concept known as a rolling upgrade. Scalability is the ability of a system to grow. Factors that limit scalability might prevent you from adding servers to a data center. For example, you might not have enough rack space, power, network connections, or budget to add another server. Cluster-aware applications can take advantage of the distributed nature of the cluster to distribute processing, dynamically balancing load between servers.Thus, an HA cluster can provide better utilization of the server resources you already have, saving the data center and budgetary resources for scalability in other areas. Moreover, adding servers to the cluster can be easier with HA software. A new server can be added to the cluster while other servers are still online.Then, applications can be transferred to the new server to distribute the load evenly, in much the same way that a rolling upgrade is accomplished. The next section covers the configurations of an HA application or database server, and Microsoft Cluster Server (MSCS) on Windows NT/2000.
197
140_SANs_06
198
8/14/01
3:37 PM
Page 198
Chapter 6 • SAN Applications and Configurations
Typical HA Application or Database Server A typical HA application or database server consists of several components: ■
Two or more redundantly configured servers
■
One or more Fibre Channel SANs to enable the sharing of storage
■
At least one fault-tolerant/redundant storage volume
■
An interconnect for cluster messaging (which might also be Fibre Channel)
■
A software mechanism for providing failover operation
Through clustering software, the application server continually communicates with the clustered spare using network heartbeats to indicate to the other machines that everything is operating correctly.This heartbeat is typically carried over a dedicated network for clustering traffic. In case of a problem (for example, a software crash on the operational server or a hardware component failure), a heartbeat link will indicate to the other server that something has failed or is otherwise inoperable. If that heartbeat is lost, the spare server takes over the function provided by the application service. Depending on the clustering software, either the entire server or only specific services on the server can be failed over or failed back.
NOTE HA clusters can use either an active/active or an active/passive model of operation. In the active/passive model, the passive server does not provide any service until a failure condition causes it to assume control of the cluster and become the active server. Thus, the passive, or standby server is not utilized at all in normal operation. The active/active model, by contrast, allows each server to provide service—in other words, to be utilized—even during normal operation. The downside to active/active clustering is that when a failure does occur, performance will be impacted, since there will suddenly be fewer resources providing the same services.
HA clusters are also used to enable rolling upgrades—the upgrading of one of the machines in a cluster to new software or hardware. In a rolling upgrade, resources are manually transferred to a standby server from the operational server, and the operational server is taken offline for maintenance. Hardware or software is
140_SANs_06
8/14/01
3:37 PM
Page 199
SAN Applications and Configurations • Chapter 6
added or upgraded, tested, and then the operational server is brought back online and resources are transferred back to the server.Then, the standby server is similarly upgraded, without end users encountering a substantial interruption in service. This enables systems to be upgraded and maintained without affecting critical business operations.
NOTE Almost all HA servers in use today require a small interruption in service during failover. This is the time it takes the standby server to decide that the primary has actually failed, and then to start up the applications that that server had been running. The technology required to provide a true zero-downtime failover is understood. Some HA databases even implement zero downtime failover; however, this functionality is not in general use, due to its complexity and the need to have applications specifically written to take advantage of it.
Redundant components are used throughout a system to provide high availability. Eliminating all single points of failure is important to ensure that the failure of a single component does not bring down the entire cluster. Redundant HBAs are used to provide two paths to the cluster’s shared storage from each host. Redundant, noncoupled fabrics provide separate network paths to the storage.We discuss redundant fabrics further in Chapters 5 and 7. In addition to redundancy, fault-tolerant equipment designed with dual power supplies, internally redundant controllers, and circuitry is preferable in these environments. To eliminate single points of failure, a dual-fabric SAN architecture is used. These fabrics are separated intentionally to prevent loss of service due to operator error, the need for a rolling upgrade, or major software or hardware problems on one of the fabrics. By using separate redundant paths, physical cabling problems are minimized, and disruption of the network from operator error can be isolated to just one segment. Dual HBAs are used in the servers to connect to each fabric. Multipathing software, which is normally provided by your RAID vendor, is used to detect I/O errors and redirect traffic to the other HBA, thus avoiding unnecessary cluster node failover.Third-party software such as VERITAS’ Dynamic MultiPathing (DMP) product can also be used to provide this functionality. Finally, some HBA vendors (such as TROIKA) provide multipathing software support built into their HBA drivers.
199
140_SANs_06
200
8/14/01
3:37 PM
Page 200
Chapter 6 • SAN Applications and Configurations
NOTE The overriding design principal in an HA environment is to keep all failures at as low a level as possible. You want to minimize the chance of the HA software (the highest level of protection) having to actually perform a failover simply because a fan or power supply (the lowest level) fails. This is particularly critical in large environments, where it is statistically predictable that even components with high Mean Time Between Failures (MTBF) will fail frequently, simply because there are so many of them. Therefore, you should not assume that the presence of an HA environment eliminates the need for fault-tolerant components.
Critical to any HA application server is the use of at least one HA storage device, supporting either active/passive or active/active storage controllers. In the active/passive case, a single Logical Unit Number (LUN) is exported to two or more separate Fibre Channel ports. One port is used at a time, with either an automatic or manual failover causing the alternate port to become active. Only one port can be active at any given time. For active/active devices, this limitation does not apply. A LUN or target is exported to any number of paths.Traffic can flow across these multiple paths from any node to this LUN.
Microsoft Cluster Server Microsoft Cluster Server is the most common way that Windows administrators add HA capabilities to their critical IT systems. Figure 6.1 shows a typical MSCS configuration.The basics of how a generic MSCS configuration works and descriptions of the critical parts are explained further later in this section. An MSCS cluster consists of two (Windows NT,Windows 2000 Advanced Sever) or four (Windows 2000 Data Center) server nodes connected via redundant HBAs to a dual-ported storage subsystem via a redundant/resilient architecture SAN (see Chapter 7,“Developing a SAN Architecture,” for further explanation of a redundant/resilient architecture).The most common MSCS setup in use today is the two-node cluster.The two servers are configured as Active and Standby nodes in the cluster.This employs the active/passive model of HA.The Active node owns the cluster LUN(s), but the Standby has the right to take ownership when required. Having ownership rights does not imply sharing. MSCS uses a share-nothing
140_SANs_06
8/14/01
3:37 PM
Page 201
SAN Applications and Configurations • Chapter 6
architecture, which means that only one server can use a LUN at a time.The management software provided by the storage vendor must allow a LUN to be accessed by two or more hosts for use in a cluster environment. Figure 6.1 Microsoft Cluster Server Configuration Dual-Controller Storage Array
C1
C2
DualFabric SAN Architecture Fabric A
HBA 1
Fabric B
HBA 2
HBA 1
HBA 2
Heartbeat
Active Cluster Server
Ethernet LAN
Standby Cluster Server
201
140_SANs_06
202
8/14/01
3:37 PM
Page 202
Chapter 6 • SAN Applications and Configurations
The clustering software uses one or more heartbeat networks to ensure that the Standby server knows the status of the Active server at all times.These heartbeats can be sent over a dedicated (private) IP network (either Ethernet or IP over Fibre Channel), a public Local Area Network (LAN), or by using a shared disk volume. Generally, at least one dedicated connection is used to ensure that the heartbeat is not interrupted by outages to the public network. Absence of the heartbeat detected by the Standby server triggers the node failover function.The Standby node then takes ownership of the cluster LUN and continues serving the application’s clients. Once the primary node server has been fixed, a manual failback operation will restore the original configuration. Manual failback is recommended to avoid a ping-pong effect that can result from an automatic failback setting.The software can also be manually moved over to enable server maintenance or rolling upgrades. A dual fabric and dual HBAs are typically used in HA clusters to ensure that there are fully redundant hardware components in the system.The recommended storage is a dual-controller storage array to allow for redundant connections to the data. Microsoft Windows 2000 Data Center is a four-node cluster configuration that has been certified using fabric switches. Unlike MSCS, Data Center absolutely requires a SAN, since SCSI storage cannot support more than two initiators.
Microsoft WHQL Certification Microsoft Windows Hardware Quality Lab (WHQL) provides lists and references to hardware that has passed certification by Microsoft for certain applications. WHQL certification is often useful in determining if hardware and drivers for that hardware have been certified for working with Microsoft operating system software and advanced OS capabilities like MSCS. Special certifications are given to hardware (storage, HBAs, and network components) for running with MSCS, which ensures that Microsoft has tested all of the hardware for that application and certified that it works. Brocade switches are cluster-certified through Original Equipment Manufacturers (OEMs) in bundle configurations along with the OEM storage and qualified HBAs. It is also possible to build custom HA solutions with Brocade switches, in addition to buying pre-integrated bundles. Microsoft provides an online database of WHQL-qualified hardware, which can be searched by keyword or category of equipment (www.microsoft.com/hcl).
140_SANs_06
8/14/01
3:37 PM
Page 203
SAN Applications and Configurations • Chapter 6
Using a SAN for Storage Consolidation One of the major uses of SAN technology has been for storage consolidation. The availability of a storage network has enabled data center administrators to centralize where their storage resources are located, making better use of precious storage dollars through storage sharing and pooling. By having storage available in a central pool, usage of that data space is more efficient and the data is easily backed up, managed, and accessed.This section describes the major methods of using a SAN for storage consolidation, reviews several of the techniques used to accomplish this goal, and provides sample configurations. Before the use of storage networks, storage was dedicated to a specific host. This resulted in often underutilized or poor distribution of storage capacity. For example, you might have 100 GB available on one server that was being used for user’s home directories, but only 1 GB available on another server that was running a business-critical e-commerce database.With dedicated storage, it was impossible to reallocate those disks from the less critical, and less resourceconstrained, system to the more important one.You would have to buy additional storage for the critical database server, even though you already owned 100 GB of unused storage! With the use of a SAN, reallocation could be as simple as reassigning some of the disks associated with less important user files to the businesscritical database in a few minutes without any recabling or downtime. With dedicated storage, whether externally attached or internal to a server, adding storage capacity typically requires the full reconfiguration of a system: shutting down the host, connecting the new storage, and restarting the system. External arrays might have the capability to add disks “on the fly,” but what if you need to add a new cabinet? There are always fundamental limits to directattach storage that do not apply to storage attached via a SAN. With Fibre Channel SANs, making storage available to hosts in a network is less complex.You connect any Fibre Channel storage device to your switch, and the device is immediately available to any of the hosts that are connected to the fabric. Minimal configuration of the fabric is required for those storage resources to show up on the SAN, and minimal effort is required for that storage to become available to hosts in the network.You simply connect the storage to the network, and all of those resources become immediately available. Usually, all that is required is to configure the new device into the fabric’s zoning table, if necessary, and to import those newly available volumes into the operating system. On Windows 2000 and Solaris, this does not require a reboot, and can be accomplished entirely without downtime.
203
140_SANs_06
204
8/14/01
3:37 PM
Page 204
Chapter 6 • SAN Applications and Configurations
NOTE Some storage devices require specialized drivers, which might have different procedures for adding capacity. You should check with your storage vendor to find out how to add access to new storage arrays “on the fly.”
Storage consolidation is supported in almost any configuration of the fabric. The topology of a Fibre Channel network for storage consolidation and sharing really is the simplest case for SAN layout.You can use a “SAN islands” approach, with an interface on each multiport storage array connected to each island.You can also construct one large fabric.The route you take will depend on your performance and management goals.These topics are covered in greater detail in Chapters 5 and 7. With Fibre Channel, devices are dynamically added or removed at any time from the network.This is an advantage in terms of flexibility and control, but also a potential difficulty due to the way operating systems handle dynamic volumes. For example, some operating systems have been built to assume that they own all storage they are connected to, and will even try to overwrite data on any volume that they come across.This is particularly the case for Windows NT, which will write an operating system signature on anything that it discovers, even if it has previously been claimed by another system in the network.These problems have led to a number of hardware- and software-based approaches, like Brocade Zoning, for controlling which systems on your network have access to the devices you have added.To truly add storage “on the fly,” one or more layers of functionality might be required. For example, you might want to be able to dynamically resize logical volumes. Additional software and configuration might be required in order to facilitate this. For example,VERITAS Volume Manager and VERITAS File System provide this functionality. Operating systems currently available generally do not have native support for Fibre Channel volumes. Instead of directly representing Fibre Channel devices to an operating system, Fibre Channel HBA drivers abstract storage devices in a Fibre Channel network, and present them to the operating system as SCSI targets.This leads to an area where configuration is important.You need to understand how this mapping occurs, so that you can translate between the devices as they appear in the operating system, and the reality of where they are located in the SAN. Figure 6.2 depicts a simple Fibre Channel SAN.
140_SANs_06
8/14/01
3:37 PM
Page 205
SAN Applications and Configurations • Chapter 6
Figure 6.2 Simple Fibre Channel SAN for Storage Consolidation Server
Server
Server
Server
Fibre Channel Storage
Shared Access
The access management techniques currently in use fall into one of three categories: switch zoning, LUN masking, and software control. In any SAN where storage is expected to be shared between different hosts, it will be necessary to utilize at least one of these methods to control which device has access to which
205
140_SANs_06
206
8/14/01
3:37 PM
Page 206
Chapter 6 • SAN Applications and Configurations
storage volumes. Examples of the different kinds of storage partitioning techniques follow. Zoning is also discussed from an implementation perspective in Chapter 9, “SAN Implementation, Maintenance, and Management.”
Shared Storage Using a Web Farm A common use of HA clustering is for making Web farms and their data available at all times. Unlike most HA scenarios,Web farms do not necessarily require specialized HA software to be used as a group.This is because read-only file systems can be used to provide shared access to storage, and front-end IP load balancing switches provide the rest of the solution. In this use of Fibre Channel SANs, a read-only, centralized storage array is used to support a large number of Web servers.This approach helps to enormously reduce the costs of acquiring and managing storage. The traditional approach to designing a Web farm involves buying large amounts of local storage for all the servers.With a SAN, far less storage can be used, and it can be managed in a centralized way. Multiple Web servers are connected on a single Fibre Channel network, with shared access to the same pieces of storage. Static Web data is kept on this storage, which has been mounted readonly on all of the Web servers. Because of the high speeds of Fibre Channel versus the slower IP network connections to the Internet, many systems can access the same storage with no impact on performance. In a typical configuration for Web farm storage sharing, as shown in Figure 6.3, a large farm of Web servers would be connected to the Internet through an IP load balancer (Layer 4 switch).This allows traffic to be distributed to the least busy server, while making all servers in the farm appear as one logical entity to the clients on the Internet. All of the hosts would have access to the same volumes on a read-only basis. One “content master” host would need to have readwrite access. A shared file system is required to enable this configuration, unless some care is taken in the design. This technique drastically reduces the cost of a Web farm, because you gain a great amount of efficiency in the use of your storage resources. Instead of dozens of replicated environments with the same Web data, a single large storage array is used to support Web requests.This results in significant savings in data storage, and simplifies management of the data. Less manpower is required to manage fewer storage devices, use of floor space is minimized, maintenance contract costs are reduced, and electricity and cooling costs go down.
140_SANs_06
8/14/01
3:37 PM
Page 207
SAN Applications and Configurations • Chapter 6
Figure 6.3 Web Farm Using a SAN
Internet
Web Load Balancer Ethernet Backbone
Web Server
Web Server
Fibre Channel Storage
Web Server
Web Server
207
140_SANs_06
208
8/14/01
3:37 PM
Page 208
Chapter 6 • SAN Applications and Configurations
Storage Partitioning Using Switch Zoning Brocade Zoning, the use of the fabric to partition storage and servers into different accessible areas, can be used to partition storage into different pieces. For example, each disk within a Just A Bunch Of Disks (JBOD) could be assigned to a different host. By zoning different pieces of storage with different servers, sharing of the network can be used to allocate storage among hosts. Zoning within the fabric is particularly important in larger configurations, as it not only acts to provide partitions to control disk access, but also provides “broadcast containers” similar to IP Virtual LANs (VLANs).Thus, Brocade Zoning acts to increase the scalability and reliability of fabrics. To set up storage partitioning, administrators typically map out on paper what kind of storage distribution they would like across their SAN. Specific storage targets are assigned to specific servers and workstations.The administrator then uses a Graphical User Iinterface (GUI) (such as Brocade WEB TOOLS) or a Command Line Interface (CLI), (such as the Brocade telnet CLI), to create individual zones for all of the servers and workstations. If your SAN contains multi-LUN devices such as RAID storage arrays, current fabric zoning cannot partition individual LUNs to different hosts, although hardware offering this capability is on the horizon. Using the software supplied with such a RAID or using HBA-based LUN masking might be required in these cases.These should not be seen as a substitute for fabric zoning as they do not provide an equivalent security model, do not allow for centralized management, and do not act as Registered State Change Notification (RSCN) containers. Instead, these techniques are supplemental to fabric zoning.
Switch Zoning Configuration for Departmental SANs When an administrator wants to take advantage of storage consolidation and enhanced manageability, he or she might want to collect several departments onto one large storage network. In the example in Figure 6.4, three departments (Engineering, Finance, and Marketing) have been consolidated onto a single SAN. Each department has a dedicated storage array for its operations and a set of hosts. Hosts and storage throughout the company can be connected into a single Fibre Channel storage network. By using switch zoning, engineering hosts have access to only the engineering storage array, finance hosts have access to only the finance array, and marketing hosts have access to only marketing data on the marketing storage array. Using the Brocade Zoning tools, the SAN administrator would create three zones: Zone1, Zone2, and Zone3. Engineering Host A,
140_SANs_06
8/14/01
3:37 PM
Page 209
SAN Applications and Configurations • Chapter 6
Engineering Host B, and Engineering Storage Array would be added to Zone1, as shown in Figure 6.4. Finance Server would be added to Zone2 with Finance Storage Array. Marketing Host and Marketing Storage Array would make up Zone3. All of these zones would be added to a single zone configuration, which would be set as the active configuration for the fabric.The advantage of this partitioning includes the capability to install a single Fibre Channel-connected infrastructure in a building or campus that can support any desktop or server connection to storage, the ability to centrally manage access to storage by an administrator through the switches, and the ability to centrally back up all storage through the network. Figure 6.4 A Departmental SAN Partitioning Using Switch Zoning
Engineering Server
Engineering Server
Marketing Server
Finance Server
Zone2 Zone3
Zone1
Engineering Storage
Finance Storage
Marketing Storage
209
140_SANs_06
210
8/14/01
3:37 PM
Page 210
Chapter 6 • SAN Applications and Configurations
An administrator would set this up by first defining which departments had storage that could benefit from central connectivity. A Fibre Channel SAN would be wired throughout a building or campus, and storage for each department brought online within specific zones. Individual servers and hosts would be added, one at a time, to the appropriate zones to which they belong by identifying their World-Wide Names (WWNs) or port addresses and assigning them into a zone set.
Storage Partitioning Using Storage LUN Masking Storage partitioning is also accomplished using LUN masking on the storage. Storage arrays from all major vendors have added this feature to control host access to storage volumes.The storage administrator determines which hosts talk to which storage volumes. Usually, this is done by specifying the port or node WWNs of the HBAs (and thus, hosts) that are connected in the network, and which physical LUNs they are allowed to access. If a host that is not granted access attempts to access a volume, the storage array will prevent this and reject any commands to that device from the alien host. Hosts that are not allowed access to a LUN will simply not get access. Rogue hosts on a network, human error, and operating system vagaries will not jeopardize the integrity of your data. Storage LUN masking controls access where the data is being stored. However, in an environment where many storage arrays exist, this decentralized management model might be time consuming to manage. Storage array manufacturers might charge a substantial extra fee for the capability and software to manipulate storage LUN masking.You should make sure that you understand up front whether your array provides this capability, and if so, at what cost. Finally, if you use more than one manufacturer’s array in your SAN, you need to ensure that you have expertise in each LUN management application.
Storage Partitioning Using HBA LUN Masking The other end of your storage network is the HBA.This is another feasible point for controlling which hosts can or cannot access certain LUNs. HBAs also offer LUN masking functionality for this reason. An HBA can mask what devices the operating system accesses.This control is typically exercised through a console application, registry settings, text files, or third-party software. Host-based LUN masking produces much the same effect as storage LUN masking. It is typically
140_SANs_06
8/14/01
3:37 PM
Page 211
SAN Applications and Configurations • Chapter 6
less expensive, although it requires the active participation of every HBA on your network.The security model is substantially weaker, since one host—either intentionally, through operator error, or through software malfunctions—can compromise data integrity on your SAN. If even one host on your network does not have the LUN masking set up correctly, it could mean corrupted data. A rogue host could also gain access to your data without permission.This should not imply that HBA LUN masking should not be used, but rather that it should be used in conjunction with fabric zoning for maximum effectiveness.This type of multilayer security is common in traditional IP networks: any security consultant will tell you that the correct place to provide security is not just at the host or in the network, but everywhere. Some HBAs allow you to change LUN masking “on the fly,” meaning that changes you make to masking are reflected immediately on the network. Some software might require a reboot of the system for the changes to take effect. In addition, LUN access changes need to be propagated across a network to every host that is accessing common storage—not a trivial feat.The principal advantages of HBA-based LUN masking are cost and accessibility.When HBA-based LUN masking is available for an HBA, generally the LUN masking is included as a standard part of the cost of purchasing the hardware.
Partitioning with Software To tackle the problems of allocating volumes across devices, software companies have come up with a number of solutions to help you control which systems in your network have access to which devices.These solutions usually exist as drivers that are layered on top of file systems. Like HBA LUN masking drivers, this software must be loaded on every system in order to work.The existence of hosts that are accessing storage in the same zone as these machines but that are not running the software is almost guaranteed to result in data corruption.These software applications have one thing in common: they act as a filter for other drivers that control which LUNs are seen by a host, and selectively allow or disallow access to these devices depending on administrator or user requests. Some software packages actually allow multiple hosts to have read-write access to the same device at the same time.This is a shared volume/shared file system approach. Other software packages merely allow convenient and dynamic reallocation of resources between hosts, with only one host having control at a time. Data-sharing applications are loaded onto servers in a network, and generally all machines in a network are required to have cooperating servers, or must be zoned into special areas for volume sharing. Some software applications also
211
140_SANs_06
212
8/14/01
3:37 PM
Page 212
Chapter 6 • SAN Applications and Configurations
require the installation of a metadata server to coordinate access to volumes. Software applications like VERITAS Volume Manager,Tivoli SANergy, and HP LUN Manager allow you to manipulate which hosts in the network are allowed to see which volumes in a network.Through metadata passed between devices, information about which systems are using which volumes is exchanged with the drivers on all of the systems.When there is a request to share or unshare a volume, the software tells the hosts that are no longer allowed access to unmount a volume. It then tells the hosts that now have access to go ahead and mount and access the volume. All of this can happen automatically, without additional user intervention. These techniques are suited for very dynamic environments.With all of the packages available today, sharing of data can be changed “on the fly,” from minute to minute if necessary. A common use of this has been for rendering farms at Hollywood computer graphics firms, animation houses, and special effects companies.Volumes are shared to individual animators to store work in progress.The high speeds of the Fibre Channel network are used to transfer the many gigabytes of data generated by those animators.Volumes are shared between workstations and groups, and can be reallocated based on changing workload requirements. Shared file systems also can be used to facilitate the creation of Web farms. The disadvantage of using software is primarily one of security. All of the software available today requires that all of the hosts running in the SAN zone be loaded with the software. Accidentally attaching hosts that are not running the software could cause massive data corruption.The software relies on cooperating hosts to be loaded properly to control individual access to volumes. Because of this, these software packages are best used in tightly controlled environments, and in conjunction with fabric zoning.
LAN-Free Backup Configuration Traditional backup systems used SCSI direct-attached tape storage as a method to back up business-critical data accessed by application servers.This meant that each application server had its own tape storage, which backed up the data stored on locally attached disks. Server RAM, I/O bus, and CPU resources were used to drive the backup process.To combat management problems associated with coordinating the growing number of local tape drives and libraries, inefficient use of secondary storage resources, and ineffective use of personnel, companies implemented LAN-based backup using a server-client model.
140_SANs_06
8/14/01
3:37 PM
Page 213
SAN Applications and Configurations • Chapter 6
A central backup server would be installed on the LAN.The application servers and workstations were configured as clients of this backup server.The central backup server would accept requests from backup agents running on its clients, and transfer the data through the LAN to the locally attached tape resources it managed.This method provided a centralized, easy-to-manage backup scheme and allowed greater efficiency by sharing tape resources over the network. Unfortunately, the LAN-based solution has several shortcomings. Backup jobs require a large amount of block data movement, which in this scenario is carried across the LAN.With so much data being generated every day and backup windows extended into normal working hours, LAN connections become swamped with backup jobs. End users complain that they cannot access network resources, and systems administrators see network performance dwindle. Administrators can attempt to minimize the effect of the backups on the operation of the LAN by running backups after business hours. However, the amount of data being backed up continues to increase at nearly exponential rates.With 24x7 operations, the LAN-based solution has turned out to be unfeasible for most enterprise environments. LAN-free backups using storage networks have helped solve these problems. Because Fibre Channel is a high-bandwidth channel, and has been designed from inception as a separate network for bulk data movement, the bandwidth problems that appear in running backups over LANs disappear.This separate network, in addition to offloading traffic from the LAN, also performs its operations with less CPU overhead than the LAN approach requires.This is due to the fact that Fibre Channel connections do not need to go through the server’s TCP/IP stack, and because certain levels of error checking are accomplished in Fibre Channel hardware. It is necessary in many solutions to have a host to manage the shared storage and to store the backup database, which is used to locate and recover data.VERITAS NetBackup is an example of backup software that facilitates LAN-free backup.
SAN Server-Free Backup Server-free backup is the use of the SAN to remove backup traffic from the current Ethernet or other IP network without requiring a separate dedicated server. A SAN-based server-free backup is therefore also LAN-free. Because Fibre Channel storage networks are now used for data sharing or consolidation, it is natural to design and implement a server-free fabric-wide backup scheme.This
213
140_SANs_06
214
8/14/01
3:37 PM
Page 214
Chapter 6 • SAN Applications and Configurations
type of backup implementation is in contrast to legacy LAN-based approaches, where each server reads the data and using IP, sends it on the corporate LAN to another server with locally attached hardware or LAN-free schemes, where backup traffic is isolated to a separate Fibre Channel-based network. In contrast to one host backing up to another, whether it be on a LAN or SAN, a data mover, at the request of a host, reads from the disk and writes directly to SANbased shared tape resources, without the requesting host ever having to be directly in the data path. What can be a data mover? Just about anything. Old servers gathering dust can be recycled and turned into data movers. Native “smart” Fibre Channelattached tape drives might have embedded data mover functionality. Fibre Channel-to-SCSI bridges and routers, which allow legacy SCSI device attachment to the SAN, almost universally have this feature.Typically, data movers can be either Network Data Management Protocol (NDMP)-based or use the Extended Copy command, sometimes called third-party copy. NDMP is an open standard protocol for enterprise-wide backup of heterogeneous storage. NDMP clients and servers pass metadata about the backup job status as well as the data itself.Traditionally, NDMP was used in the networkattached storage model. Extended Copy is a SCSI protocol command that allows a remote block-level copy to occur.The reason this method is called third-party copy is that the host actually requesting the copy command does not send it directly to the devices in question. Instead, it sends the request to a third-party device, which then sends the command to the appropriate targets. In general, Fibre Channel-to-SCSI routers and native Fibre Channel tape drives use Extended Copy, while NDMP uses legacy hosts for data movement from disk to tape. Why do server-free backups? Typically, backup used to be done through a centralized console or backup server.This backup server would communicate with backup agents on all of the hosts or servers in the corporate Ethernet network at a convenient hour, and request certain files to be sent from storage to tape.This works well for installations with a few file servers and small data sets. Enterprise companies are now finding that all other IP traffic comes to a halt when backups are occurring, and server CPU cycles are being saturated from all of the IP traffic. Moreover, with the immense data growth occurring throughout the industry, the amount of time a backup takes has extended from a few hours each night to sometimes an entire day. In extreme cases, the times required to back up an enterprise data set, even incrementally, have gone beyond the Ethernet network’s capability to transport that data within a day.To make matters
140_SANs_06
8/14/01
3:37 PM
Page 215
SAN Applications and Configurations • Chapter 6
worse, the move toward 24x7, always-on Web-based business models, has made no time of the day available for loading the corporate network. Continuous customer access and nonstop transaction processing have become critical for business survival. Today, the best solution to these dilemmas is server-free backup. Existing or new Fibre Channel storage networks can take full advantage of server-free backup technology. For most new SAN installations using LAN-free only backups, the hardware functionality exists by default.What problems are solved in the data center? Server-free backup is done directly by each data mover writing to tape directly, without the need for agents or IP traffic, or any consumption of CPU time on other servers. A master backup server coordinates tape sharing between media servers. Other features such as snapshot copy, which exist as hardware- and software-bundled features for enterprise RAIDs, in addition to a backup software application option, enhance server-free backup implementations for database servers.The idea is that instead of the data being read from disk drives, into the memory of servers, and sent through an IP network to the backup server, the data is block-copied from disk to tape directly. Figure 6.5 shows an example of this, where three different hosts share a single tape drive across the SAN.
SAN-Based Third-Party Copy Data Movers Third-party copy backup systems are very similar to the LAN-free systems discussed previously. However, with this technique, specialized pieces of hardware and software called “data movers” are used to back up critical data from storage arrays in the SAN without the need for a dedicated server or server(s) to handle the data copying and movement. All data movers support the SCSI Extended Copy command.The third-party copy hardware actually moves the data from the disk to tape.The backup software controls this operation without the need for the servers in the network to get involved in the actual movement of data. Agents are not required to run on the server, and critical servers are not occupied backing up data. Third-party copy hardware, such as some Crossroads or Chaparral Fibre Channel/SCSI bridges, works in conjunction with third-party copy-enabled backup application software (from VERITAS, Legato, Computer Associates, and others).This backup software operates identically to normal backup software in terms of user interface and operation. Overall, this technique increases performance, greatly reduces the time needed for backup windows, and eliminates the task of backup job processing from the CPU of the server.
215
140_SANs_06
216
8/14/01
3:37 PM
Page 216
Chapter 6 • SAN Applications and Configurations
Figure 6.5 Using Storage Networks for Server-Free Backup Host A
Fibre Channel Storage
Host B
Fibre Channel Storage
Host C
Fibre Channel Tape
A typical third-party copy configuration with currently available hardware typically uses a Fibre Channel-to-SCSI router as a bridge to legacy SCSI tape drives, and combines that functionality with third-party copy support (Figure 6.6).
Making Your Enterprise Disaster Tolerant As computer systems become increasingly central to business operations, the integrity and availability of those systems has become one of the most important charters of IT organizations.The loss of access to computer systems and data today even for a few minutes can mean millions of dollars of lost revenue, and damage to reputation. Because of the criticality of those systems and data, service-level agreements and in some cases government regulations require that systems must be available to provide business continuance in the face of major disasters.
140_SANs_06
8/14/01
3:37 PM
Page 217
SAN Applications and Configurations • Chapter 6
Figure 6.6 Third-Party Data Mover Configuration Host A
Host C
Host B
Command to move data
er
ov
aM
t
Da
Fibre Channel/SCSI Bridge
Fibre Channel Storage
Fibre Channel Storage
Actual block data movement
Fibre Channel Tape
Enterprises that are implementing a SAN are finding that the ability to mirror and operate their devices spread across large distances helps to provide disaster tolerance to their critical installations.The presence of a SAN helps enhance the ability of a company to protect and recover data in the case of a disaster, and provides the tools to enable an administrator to design a disaster-tolerant system. Fibre Channel technology can be coupled with technologies like Dense Wave Division Multiplexing (DWDM) for Metropolitan Area Networking (MAN), and tunneling through existing high-speed Wide Area Networks (WANs).Thus, it is
217
140_SANs_06
218
8/14/01
3:37 PM
Page 218
Chapter 6 • SAN Applications and Configurations
now possible to separate key data sites across great distances, while still allowing them to share disk subsystems and backup devices.The following sections summarize the creation of a geographically separated, but fully connected SAN.They describe how Brocade switches are uniquely suited to handle these needs with features known as Remote Switch and Extended Fabrics.
Data Replication and Remote Backup Data replication is used to make enterprises disaster tolerant across long distances, particularly in the case where bandwidth is not readily available or is very expensive. Data replication is the technique of taking a snapshot of your operational storage images at a specific point in time, and moving across a network to a geographically separate storage facility.This data is then moved, on schedule, across a potentially slower network across large distances, and replicated at the backup facility. Data replication is typically done every day or at most every few hours, and it can be done across a city or across the globe. Because there is a delay between updates, data can sometimes be sent over a slow link and reassembled on the other side. Data replication can be done directly across Fibre Channel, enabling very fast replication and minimal mismatch between a replica and live data.This technique can also be used across existing IP and other network infrastructures to transport the data across large distances. For example, it is possible to tunnel Fibre Channel connections over an Asynchronous Transfer Mode (ATM) network using the Brocade Remote Switch product and an appropriate Fibre Channel-to-ATM gateway.This is an optionally licensed product available for all Brocade SilkWorm 2000 and higher switches. Remote backup is the use of a long-distance link to enable backup to a remote site. Normal backup techniques are used, with the difference being that backup tapes and media are stored far away from the servers being backed up. This helps ensure the safety of that data in the case of a geographically limited disaster. Like remote replication, the task can be done via a Fibre Channel-toATM or Fibre Channel-to-IP gateway. Figure 6.7 shows a typical remote backup configuration utilizing existing WAN infrastructure.
140_SANs_06
8/14/01
3:37 PM
Page 219
SAN Applications and Configurations • Chapter 6
Figure 6.7 Remote Backup over WANs Primary Site Server ATM
Recovery Site
Fibre Channel/ATM
Fibre Channel Storage
Fibre Channel/ATM
Fibre Channel Tape
Metropolitan Area Network Solutions A recent innovation in optical technology, DWDM hardware has enabled the transport of native Fibre Channel over greater distances up to 100 km. A DWDM allows for real-time, full-speed transport of Fibre Channel to match the very high bandwidth requirements of real-time, mission-critical applications.
219
140_SANs_06
220
8/14/01
3:37 PM
Page 220
Chapter 6 • SAN Applications and Configurations
This approach is used for creating disaster-tolerant solutions, by enabling remote mirroring of operations across large distances. Because these solutions can transmit full-speed Fibre Channel frames, two separate data centers can share the same data, and can create remote mirrors of data.This remotely mirrored data allows for a hot-standby system that can take over the operations of a failed system at a moment’s notice, with all data intact and no need for data recovery. In fact, DWDMs can be used in conjunction with HA software to allow this failover to occur both automatically and quickly. Brocade switches compensate for the signal delays that happen when transmitting frames over long distances with the use of extended amounts of buffering (buffer-to-buffer credits) available on Inter-Switch Links (ISLs).This delay is caused by the speed with which light travels through the glass in the fiber-optic cable. By configuring two switches to use extended buffer credits on the long-distance E_Ports, Brocade switches can handle this delay without losing bandwidth. A Brocade switch can be configured to handle extended distance fabrics by installing the Extended Fabrics software license, and then setting the long-distance fabric settings to “1” in the switch configure command. Individual switch E_Ports can either be set to handle distances of 50 km (mode 1) or 100 km (mode 2, 60). See the Brocade Extended Fabrics documentation for details on configuration. One of the most typical uses of Fibre Channel SANs over MANs is sharing data for disaster tolerance. Banks, brokerages, and other businesses in Manhattan are some of the biggest users of this technology.These organizations require realtime backup of data to a remote site. A combination of disaster tolerance requirements and the cost of real estate in Manhattan has resulted in a large number of organizations establishing disaster recovery sites in New Jersey for a secondary operations center. Figure 6.8 shows a typical disaster tolerance configuration used for a MAN topology.Two parts of the same SAN exist on either side of a MAN (DWDM), operating just as if they were not geographically separated. Data is mirrored between both sides of the SAN, and failover software can be used to provide high availability with backup servers on either side of the link.
140_SANs_06
8/14/01
3:37 PM
Page 221
SAN Applications and Configurations • Chapter 6
Figure 6.8 Metropolitan Area Network, Bridging Fibre Channel over DWDM Site B
Site A Server
Server
DWDM
DWDM
Data Mirror
Fibre Channel Storage
Fibre Channel Storage
221
140_SANs_06
222
8/14/01
3:37 PM
Page 222
Chapter 6 • SAN Applications and Configurations
Summary In this chapter, we discussed the most common overall applications of SANs, and some sample configurations for those applications. HA clusters are used with Fibre Channel networks to support mission-critical business applications utilizing redundant SAN components.These can include database clusters like Oracle Parallel Database Server, and specialized failover packages such as Microsoft Cluster Server. Storage consolidation is another typical application of Fibre Channel networks. SANs have made it possible for data center administrators to centralize their storage resources, making better use of the storage they have, and leveraging their budget for storage through more efficient allocation and capacity planning. With some attention paid to managing the allocation of storage through fabric zoning, LUN masking, or storage sharing software, the use of a SAN for storage consolidation can pay off handsomely in better use of IT budgets, enhanced manageability of data, and more reliable operation of your data center. The use of a SAN for LAN-free backup, server-free backup, and newer techniques like third-party copy, offers solutions to backing up the vastly growing amounts of storage in your enterprise. Because of the bandwidth and efficiency of Fibre Channel, SANs are fundamentally better suited for backup than LANs are. Server-free backup configurations offer even more efficient backup of storage resources, requiring more advanced software capable of directly backing up data without the intervention of servers. Finally, third-party copy provides even more efficient use of the network without requiring even a backup server. With the advent of DWDM equipment that can transmit high-speed Fibre Channel data across MANs, enterprises can add disaster tolerance to their data centers. By enabling remote mirroring and replication of data, clustering across long geographical distances enhances the ability of the enterprise to keep critical systems up and running in the most extreme conditions.
Solutions Fast Track Configuring a High-Availability Cluster HA clusters are used for redundant, fail-safe installations of mission-crit-
ical business applications. Clustering provides availability, manageability, and scalability.
140_SANs_06
8/14/01
3:37 PM
Page 223
SAN Applications and Configurations • Chapter 6
Availability is the capability of a cluster to tolerate hardware, network, or
software errors. The most common use of clustering is two servers configured to share
storage through Fibre Channel. Redundant HBAs and switches should be used to provide fault tolerance. The use of dynamic multipathing software, drivers, or HBAs can provide
higher levels of availability to your cluster.
Using a SAN for Storage Consolidation Storage consolidation enables administrators to centralize
storage resources. Consolidation provides more efficient use of storage, enhances
manageability, and improves accessibility. Almost any layout of a storage network can be used for
storage consolidation. Consolidation requires attention paid to how operating systems treat
shared volumes. In order to properly partition data in a consolidation environment, you
need to use fabric zoning, LUN masking on storage or the host, or software to control permissions. It is generally best to use fabric zoning even when also using another
access control product to achieve a more effective security model, and to provide a “broadcast container,” which can increase the scalability and reliability of a SAN. An example of a typical storage consolidation setup is a shared SAN
used to provide data storage for a Web farm, where many servers read the same disks to present data. Storage LUN masking is used to ensure that only specific hosts are
allowed access to specific logical units of a storage array.The advantage of storage LUN masking is that the storage guarantees which host is allowed access to any volume.
223
140_SANs_06
224
8/14/01
3:37 PM
Page 224
Chapter 6 • SAN Applications and Configurations
HBA LUN masking is also used to limit what storage a host can see, and
requires that every host in the network participate in the same masking scheme. Software partitioning provides another type of control over LUN
presentation, but it generally requires upper-level software and demands that every host in the network be loaded with that software. Switch zoning, available in Brocade switches, provides a convenient way
to allocate storage to hosts, and to consolidate different departments into a single company network. Switch zoning does not currently support control at the LUN level, only
at the port and WWN levels. Upcoming products will add this capability. For now, other access control techniques might need to be used in addition to switch zoning to provide access control at the LUN level. Storage LUN masking provides another way to control access to
volumes in a shared SAN. High-end storage arrays provide the capability to specify the port or
node WWN of a host HBA, and specify which volumes in the array will respond to requests. By using storage LUN masking, you can ensure that only hosts with
permission can read or write from a specified volume. Storage LUN masking requires the participation of the storage only to
enforce permissions. HBAs provide access control to volumes through LUN masking. LUN masking controls which volumes an operating system can see
through a particular HBA. HBA LUN masking requires the participation of all of the hosts in the
network to avoid contention for storage resources.
LAN-Free Backup Configuration Traditional backup systems used SCSI direct-attached tape storage.The
LAN-based client-server backup model, although an improvement, cannot account for ever-increasing amounts of data through the LAN connection. LAN-free backups using storage networks solve LAN-based problems by offloading traffic from the LAN and increasing bandwidth.
140_SANs_06
8/14/01
3:37 PM
Page 225
SAN Applications and Configurations • Chapter 6
SAN Server-Free Backup Server-free backup is the use of a SAN to remove backup traffic
from a LAN. Backup is done directly on the SAN for each device, rather than each
host being involved in data transfer. Third-party copy provides an even more efficient way to transfer data to
tape, freeing a backup server from needing to directly access disks and copy data to tape.
Making Your Enterprise Disaster Tolerant Fibre Channel SANs are ideal for mirroring and accessing data across
large distances. It is now possible to separate critical systems many miles apart. Brocade switches provide extended credits on ISLs to enable high
performance and reliable long-distance operation.
225
140_SANs_06
226
8/14/01
3:37 PM
Page 226
Chapter 6 • SAN Applications and Configurations
Frequently Asked Questions The following Frequently Asked Questions, answered by the authors of this book, are designed to both measure your understanding of the concepts presented in this chapter and to assist you with real-life implementation of these concepts. To have your questions about this chapter answered by the author, browse to www.syngress.com/solutions and click on the “Ask the Author” form.
Q: My configuration does not look exactly like any of these. Is this a problem? A: These examples represent typical configurations for applications. Many reallife configurations might be more complex.
Q: Where can I get more information on configurations? A: Interoperability programs through most Fibre Channel manufacturers provide example configurations, along with more detailed version numbers and specific model information for tested configurations. Brocade has an extensive set of SAN solutions called SOLUTIONware Guides available on their Web site at www.brocade.com/SAN.
Q: I am trying to control access to storage, and do not know what type of control I need: zoning, LUN masking, or software? What should I do?
A: The kind of control over your storage depends entirely on your application. Analyzing how dynamic your environment is will determine whether you can just use zoning or software. In most cases, you might actually use a combination of these techniques to achieve what you need.
Q: I would like to cluster my databases for better performance.What databases can I use?
A: Most major databases now support fabric switch-based clustering, including Oracle Parallel Server, IBM DB2 Parallel Edition, and Microsoft SQL Server.
Q: I would like to have my Exchange Mail Server highly available.What should I do?
A: Brocade has developed HA solutions for the Exchange Server that can be used in setting up your desired SAN configuration. For more information, visit the Brocde Web site: www.brocade.com/SAN.
140_SANs_07
8/14/01
3:38 PM
Page 227
Chapter 7
Developing a SAN Architecture
Solutions in this chapter: ■
Identifying Fabric Topologies and SAN Architectures
■
Working with the Core/Edge Topology
■
Determining Levels of Availability
■
Configuring Traffic Patterns
■
Evaluating Performance Considerations
Summary Solutions Fast Track Frequently Asked Questions
227
140_SANs_07
228
8/14/01
3:38 PM
Page 228
Chapter 7 • Developing a SAN Architecture
Introduction In Chapter 5, “The SAN Design Process,” you performed the requirements analysis to determine what your SAN needs to accomplish. In Chapter 6, “SAN Applications and Configurations,” you explored some of the solutions that could be built on your SAN. At this point in the book, you should know the following information about your SAN: ■
How many ports you need for hosts
■
How many ports you need for storage
■
What the traffic patterns will be
■
The network’s performance requirements
■
Where all of the equipment will be located
■
What, if any, MAN/WAN or campus distances will be involved
■
What type of solution you are building (such as, storage consolidation)
■
How all of this will likely change over time
In this chapter, you will take this information and determine the fabric topology or topologies that best suit your needs as part of your overall SAN architecture.We discuss the different categories of fabric topologies that you could apply and which topology is most appropriate in any given case. Furthermore, we describe how you could use multiple fabrics to form a highly reliable and scalable SAN architecture.We delve into detail on one particular topology, the core/edge fabric, also commonly known as a star topology network.There are subtle differences between “normal” star networks and a core/edge SAN that require using the new term. However, for the most part, if you think of a star network, you will not be far off base.
NOTE Do not view the focus on the star topology as an indictment of the other approaches you could take. We chose to highlight this approach because it is simple. One of the strongest features of Brocade Fabric OS is the robust implementation of Fabric Shortest Path First (FSPF), which allows arbitrarily complex networks to be possible. However, the simplicity principle of “Occam’s razor” tells us to use simpler solutions whenever we can, and the core/edge SAN is simple indeed. Thus, we felt it to be an appropriate design to highlight.
140_SANs_07
8/14/01
3:38 PM
Page 229
Developing a SAN Architecture • Chapter 7
Methods for providing the best redundancy in all topologies and a discussion of performance follow. Finally, we talk about how to deal with SANs that must span long distances.
Identifying Fabric Topologies and SAN Architectures Before embarking on a discussion of SAN architectures, we will review a set of working definitions for those SAN components particular to these architectures. The terminology for describing fabric topologies, SAN architectures, and their components is still evolving. Much of the current terminology is derived from other networking technologies, like Ethernet, frame relay, or even high-performance computing node interconnects in supercomputers. Since there are occasionally multiple terminologies that might apply to the same entity, the terms we use in this book are the ones that we consider the most useful in describing any given SAN. For example, the term ring topology fabric is quite self-explanatory, and is therefore useful in describing “a fabric in which the Inter-Switch Links (ISLs) form a logical ring.” ■
Fabric A fabric consists of one or more interconnected Fibre Channel switches.The term fabric can refer to the physical switches or to a set of global software components such as the routing tables, zoning configurations, and Name Servers.
■
SAN A SAN can consist of one or more related fabrics and connected edge devices. It is possible to build a SAN using various networking technologies. However, all SANs discussed in this book are Fibre Channel fabric-based. Several emerging technologies might complement Fibre Channel in the future, so it is important to make the distinction. For now, though, the vast majority of SANs in production—and all SANs based on Brocade technology—use Fibre Channel.
■
Fabric Topology Fabric topology is the arrangement of the switches that form a fabric.This term is used in the context of ISL interconnection and does not relate to the way in which nodes are connected to the fabric. Moving a storage device from one port to another does not change the fabric topology.
■
Resilient Core/Edge Fabric Topology Resilient core/edge fabric topology is when two or more switches act as a core to interconnect
229
3:38 PM
Page 230
Chapter 7 • Developing a SAN Architecture
multiple edge switches. Nodes are attached to these edge switches.We discuss this topology in greater detail later in the chapter.The convention for describing a simple core/edge fabric involves stating the number of edge switches, the number of core switches, and the number of ISLs used to interconnect each edge switch to each core switch. (For now, all switches are assumed to be 16 ports.) It is written as 16ex4cx1i, and is read as “A simple core/edge fabric consisting of sixteen edge switches, each of which is connected to four core switches by one ISL.” A shorter reading could be “A sixteen edge by four core by one ISL core/edge fabric.” Figure 7.1 illustrates this nomenclature. Figure 7.1 Core/Edge Fabric Nomenclature 16e x 4c x 1i
4c
x by
1i
x
4 co
L
re
by 1 IS
16 e dge
230
8/14/01
16 e
140_SANs_07
■
Node A node is any device—usually either a host or storage device— that attaches to a fabric.The terms node and edge device can be used interchangeably.
■
Node Count Node count is the number of nodes attached to a fabric. Each node might take up one or more ports in one or more fabrics.
■
Fabric Port Count Fabric port count is the number of ports available for connection by nodes in a fabric.The term port count used alone refers to a fabric port count.
■
SAN Port Count SAN port count is the number of ports available for connection by nodes in the entire SAN.This is the sum of the port counts in all fabrics that make up the SAN.
■
SAN Architecture SAN architecture is the overall design or structure of a storage network solution.This includes one or more related fabrics, each of which has a topology. It might also include other networks over which the SAN is bridged or tunneled—such as a Metropolitan Area Network (MAN). More broadly, SAN architecture can include software
140_SANs_07
8/14/01
3:38 PM
Page 231
Developing a SAN Architecture • Chapter 7
components—such as path failover or data backup software—and the nodes that are attached to the fabric(s). ■
Hop Count Hop count can be defined in several ways. For the purpose of evaluating SAN designs, the hop count is identical to the number of ISLs that a frame must traverse to reach its destination. If traffic is localized to a switch, the hop count is zero. If traffic has to cross one ISL, the hop count is one.
■
Latency Each hop takes a small amount of time.This time is referred to as the latency of the link. It is a very small amount of time (2 microseconds maximum across a switch, 1 microsecond typical), and will influence performance only when the hop is a long-distance link (as in a SAN/MAN or SAN/WAN).This number is usually so small when compared to disk access times that it normally can be treated as inconsequential and eliminated as a factor.
■
Over-Subscription Whenever more nodes could potentially contend for the use of a resource—such as an ISL—than that resource could simultaneously support, that resource is said to be over-subscribed. Oversubscription can be a desirable attribute in a fabric topology and is common in most networks as a cost/benefit trade-off, as long as it does not produce unacceptable levels of congestion. Although a design might have over-subscription, it does not necessarily mean that it will have congestion. Most nodes cannot sustain full Fibre Channel speeds, typically running at 50 to 80 percent of the maximum 100 MB/sec. For performance-limiting congestion to occur, several devices must not only all operate at their peak at the exact same time, but must also sustain simultaneous peak operation. As most traffic is bursty, as well as relatively random, the chance of significant congestion is usually reduced to an insignificantly small amount.The exception to this rule is traffic such as video streaming.This kind of application produces a long, constant stream of data. If designing for this type of traffic, you must ensure adequate bandwidth for these streaming sources or localize the traffic onto a switch.
■
ISL Over-Subscription Ratio The over-subscription ratio for an ISL is the number of different ports that could contend for the use of its bandwidth.This should be calculated for an edge switch in a core/edge SAN by making a ratio of the number of free ports (non-ISL) on that
231
140_SANs_07
232
8/14/01
3:38 PM
Page 232
Chapter 7 • Developing a SAN Architecture
switch to the number of ISLs. For example, in the 16ex4cx1i SAN, each edge switch has four ISLs (one ISL going to each of four cores). Since each edge switch has 16 ports, there are 12 left. 12:4 reduces to 3:1, so the over-subscription ratio is 3:1 for this SAN. Keep in mind that this just indicates the number of potential devices that could contend for access to an ISL.This does not necessarily mean that they actually are contending for access.
NOTE In discussing over-subscription ratios, assume that all ports operate at the same speed: for example, 1 Gbit/sec. In a network where some ports are different speeds (some are 1 Gbit/sec, while others are 2 Gbit/sec), the process of calculating over-subscription can be fairly complex.
The worst-case scenario of meaningful over-subscription for an ISL on a 16-port edge switch is 15:1.This ratio means that 15 devices could be contending for the use of one ISL.This is not a property of Brocade switches. It is a mathematical property of “networks built with 16-port switches where all ports operate at the same speed.” One could argue that more than 15 nodes outside a switch could contend for access to it. However, this is not a meaningful definition of ISL over-subscription, since the nodes would be subject to performance limitations of node over-subscription. If two hosts are trying to access one storage port, it does not matter how well the network is built—the over-subscribed storage port will be the limiting factor. Over-subscription of a storage port is a completely different performance metric. See the definitions for fan-in and fan-out. As switches get larger, over-subscription will continue to be a design factor. Networks and node counts will continue to grow faster than the ability to build larger switches. (In fact, the Fibre Channel addressing standards themselves limit the maximum size of a switch to 256 ports, while a fabric could be significantly larger than that.) Networking will still be required in SANs and will have the potential for congestion regardless of the vendor or the switch size.
140_SANs_07
8/14/01
3:38 PM
Page 233
Developing a SAN Architecture • Chapter 7 ■
Congestion Congestion is the realization of potential oversubscription. A congested link is one on which multiple devices are actually contending for bandwidth.These devices will be throttled down to consume the total bandwidth, assuming that all the devices are peaking and congesting the link at the same time. A frequent point of confusion is the difference between congestion and blocking. Blocking means that the data actually does not get to its destination, whereas congestion means that the data might simply have to wait a bit. As an analogy, consider the checkout line in a supermarket.When there are more customers in the store than there are cashiers, the checkout line is over-subscribed. However, if only one customer wants to check out at a time, the presence of other customers in the store is irrelevant to him or her. If more customers need to check out than there are cashiers, the checkout line is congested.This will have an impact on how quickly each checkout can occur. In a congested but nonblocking supermarket, each customer might have to wait in line a bit, but will eventually get to go through the checkout. If the supermarket took a blocking approach, excess customers would be turned away, and would not be allowed to wait in line. Some networks block; Brocade Fibre Channel networks do not.
■
Fabric Shortest Path First (FSPF) The FSPF protocol was developed by Brocade and subsequently adopted by the Fibre Channel standards community for allowing switches to discover the fabric topology, and route frames correctly. (As an interesting side note, Brocade proposed FSPF as a standard to ANSI one and a half years before its adoption. A number of competing protocols were proposed by other companies in the interim, but the T11 standards committee selected only FSPF.) FSPF also provides for load sharing between equalcost links.This capability is the key to eliminating congestion from a fabric. Fabric topologies that have many equal-cost links—such as the resilient core/edge topology—benefit greatly from FSPF load sharing features. Fabrics with many ISLs, but few equal-cost paths—like the full mesh—do not use FSPF as efficiently.
■
Single Point of Failure (SPOF) A single point of failure in a SAN is any component—either hardware or software—that could bring down an entire SAN solution.This could be a switch or an ISL in a nonresilient topology, or a host with no clustering software installed. In an
233
140_SANs_07
234
8/14/01
3:38 PM
Page 234
Chapter 7 • Developing a SAN Architecture
environment where uptime is critical, even distributed fabric services— such as the Name Server—can be viewed as single points of failure. This is why dual fabrics are always recommended in high-availability applications. ■
Fan-out This is the ratio of host ports to a single storage port. It is the view of the SAN from the storage port’s perspective: “How many different hosts will be trying to access me via this port?” If you have many hosts accessing a single storage port, you can be assured that all of them together will not use more than the 1 Gbit/sec the port has available. Thus, if you have a fan-out of 10:1 (10 hosts sharing access to one storage port), it is possible that on average each host will get only onetenth of the available bandwidth. Using high fan-out or ISL over-subscription ratios can be a perfectly acceptable way to build a SAN, as long as the performance characteristics of the applications involved are well understood.
■
Fan-in This is the ratio of storage ports to a single host port. It is the view of the SAN from the host port’s perspective:“How many different storage devices will I be trying to access from this HBA port?” Fan-in and fan-out are both useful for planning the aggregate bandwidth needed in the ISL matrix. If, for example, you have a fan-in of 3:1 (three storage ports for each host port), and your ISLs are 3:1 over-subscribed (three devices could potentially contend for access to one ISL), proper device placement will allow this SAN to run with an actual congestion ratio of 1:1.This is because you can guarantee that certain devices will simply never try to talk to each other.Thus, the SAN will not adversely affect performance, despite the ISLs being over-subscribed.
There are several categories of information for which we need to define terms.These categories cover such topics as how the switches are interconnected (the topology), what strategy is used for optimizing traffic patterns (locality or tiering), and what the distance requirements are (MAN/WAN). In order for distance to affect topology, the distances involved do not have to be long, they just have to affect your ability to cable components together. Even having SAN components located in different rooms or floors of the same building might affect the SAN topology you choose.You will need to know how many fabrics are involved, and whether they all have the same topology.The last two items are part of the SAN architecture, not the fabric topology.
140_SANs_07
8/14/01
3:38 PM
Page 235
Developing a SAN Architecture • Chapter 7
A SAN that attaches to a long-distance network such as a MAN or WAN is described as being “over” that technology. A SAN tunneled over a WAN is referred to as “SAN over WAN” or “SAN/WAN.” If a specific WAN technology is known to be in use, then you can refer to technologies in the same way. For example, if Fibre Channel is used on the SAN, and ATM is used on the WAN, you can refer to this as “FC over ATM” or “FC/ATM.” The following sections provide terms and definitions for the pieces that are not quite so easily defined. In particular, we provide terms for fabric topologies and SAN architectural strategies. Each of the topologies that we cover includes a description of the topology, a picture of a representative network built using that topology, and a summary list of its properties. All examples and properties assume that the networks are built using 16-port switches, unless otherwise noted. Size limits are based on current production-level support. Production level implies that Brocade has tested fabrics to these size limits, and/or has customers who have successfully done so.This does not mean that all vendors will support the maximum size listed, but rather that somebody does, and that there are no fundamental technical reasons why fabrics of that size cannot be deployed in production environments. Also note that sizes are based on the most symmetrical designs. It is usually possible to increase the size of a fabric by “hanging another switch off the edge.” This means that you would take a “pure” design like a core/edge fabric, and plug another switch or two into it anywhere you like to increase the port count.
Useful Topologies The topologies for which we provide definitions are not the only topologies possible.This list does not represent all of the standard topologies that can be applied to fabrics. Instead, we have attempted to compile a list of topologies that we have found to be the most useful in practical, real-world SANs. For example, we include the core/edge topology, but not the tree topology.This is because we have not seen any instance where the tree is the best way to solve a design problem, not because there is anything that would prevent you from using it. You can also combine these topologies to form complex networks, provided that you do not exceed the supported numbers of switches or exceed the Fibre Channel standards defined hop count limit. (There can be no more than seven hops between an initiator and a target.) These complex networks are desirable when building SAN/MAN or SAN/WAN solutions, but should be avoided to keep the network as simple as possible.
235
140_SANs_07
236
8/14/01
3:38 PM
Page 236
Chapter 7 • Developing a SAN Architecture
Scalability One of the properties we discuss for each topology is its ability to scale well. There are two metrics for measuring scalability: the size that the topology can scale to (in terms of port count and switch count), and the ease of performing this process. A number of scalability factors are involved in determining these metrics, including the fundamental “geometric” properties of the design, and the way that the channel-like fabric services perform in it. Certain topologies, such as the resilient core/edge topology, are well suited to taking a “start small and grow large” approach without needing much effort to add switches. However, some topologies, like the partial mesh, might scale to a large size, but require extensive recabling and possibly even downtime to do so.There is nothing fundamentally wrong with the latter categories of design, but they might not be well suited for dynamic environments that have high uptime requirements.
Cascade Topology A cascaded fabric, illustrated in Figure 7.2, is like a bus network: it is a line of switches with one connection between each switch and the switch downstream of it.The switches on the ends are not connected. Figure 7.2 A Cascaded Fabric
Cascaded fabrics are very inexpensive, easy to deploy, and easy to expand. However, they have the lowest reliability and limited scalability.They are most appropriate in situations where most if not all traffic can be localized onto individual switches, and the ISLs are used primarily for management traffic. There are cascade variations that use more than one ISL between switches. This will eliminate ISLs as a single point of failure and greatly increase the
140_SANs_07
8/14/01
3:38 PM
Page 237
Developing a SAN Architecture • Chapter 7
reliability of the solution. However, this also greatly increases the cost of the solution, and each switch can still be a single point of failure.Table 7.1 charts the properties of a cascade topology. Table 7.1 Properties of a Cascade Topology Limit of scalability (edge port count) 114 ports / 8 switches Properties Ease of scalability Performance Ease of deployment Reliability Cost (per edge port)
Ratings 3 1 3 1 3
Ratings indicate how well the topology meets the ideal requirements of that property (1 = Not well, 2 = Moderately well, 3 = Very well).
Ring Topology A ring is like a cascaded fabric, but with the ends connected (Figure 7.3). Figure 7.3 A Ring Topology
The ring has superior reliability to the cascade because traffic can route around an ISL failure or a switch failure. It does cost more than a cascade, but not significantly more.The ring is usually preferable to the cascade for this reason. Rings still do not perform well, or scale well.The ISLs are still very much over-subscribed and will perform acceptably only when significant localization of
237
140_SANs_07
238
8/14/01
3:38 PM
Page 238
Chapter 7 • Developing a SAN Architecture
traffic is possible. Of course, you can improve performance by using more than one ISL per switch interconnect, but—just as in a cascade—this substantially reduces the scalability and increases the cost of the fabric.To scale the fabric, you must disconnect at least one ISL, which might be disruptive to a production SAN. This design is fine if you plan for your SAN to start small and stay small. It can also be used when implementing SAN over MAN or WAN, where the topology of the MAN/WAN might dictate the topology of the Fibre Channel network. (Rings are common MAN/WAN topologies.) Finally, it is a good choice when the ISLs are mostly used for management—when performance over the ISLs is not a concern—but reliability of the ISL structure is still required, and cost is a driving factor. For example, this design might be well suited for an enterprise backup SAN with many backup libraries dedicated to server groups.(The ISLs do not carry much in the way of volume of traffic, but what they do carry is important traffic.) If your SAN performance requirements are low (as in many Windows NT environments), the ring architecture might also work for you.Table 7.2 charts the properties of a ring topology. Table 7.2 Properties of a Ring Topology Limit of scalability (edge port count) 112 ports / 8 switches Properties
Ratings
Ease of scalability Performance Ease of deployment Reliability Cost (per edge port)
2 1 3 3 3
Ratings indicate how well the topology meets the ideal requirements of that property (1 = Not well, 2 = Moderately well, 3 = Very well).
Mesh Topologies Technically, almost any topology can be described as some sort of mesh. Since this is not a very useful definition—and above all else, a definition must be useful—we will discuss and provide working definitions for two meshes: the full mesh and the partial mesh.
140_SANs_07
8/14/01
3:38 PM
Page 239
Developing a SAN Architecture • Chapter 7
Full-Mesh Topology In a full-mesh topology (Figure 7.4), every switch is connected directly to every other switch. Using 16-port switches, the largest useful full mesh consists of eight switches, each of which has nine available ports.This gives you a total of 72 available ports. Adding more than eight switches will actually reduce the number of available ports. Figure 7.5 shows the maximum size of a full mesh. Figure 7.4 A Full-Mesh Topology
Figure 7.5 Maximum Size of a Full-Mesh Topology
NOTE Scaling a full mesh can require unplugging edge devices. If you have a 4-switch full mesh (52 edge ports) and you use all the ports with edge devices, you will need to unplug one device from each switch in the mesh in order to add another switch. Make sure that you plan for this by either leaving some ports free on all switches, or making sure that there are some devices that can withstand downtime on each switch. Because of this, full meshes do not have a high rating for ease of scalability.
239
140_SANs_07
240
8/14/01
3:38 PM
Page 240
Chapter 7 • Developing a SAN Architecture
Full meshes are best used when you know that your fabric will not grow beyond four or five switches, since the cost of the ISLs becomes prohibitive after that. Also, networks that use six or more switches are very good candidates for a core/edge design, which will cost less, perform better, and scale much better than a full mesh.The good thing about performance on a full mesh is that there will never be more than one hop between switches unless there is a failure. However, since the ability to support load sharing is more of a performance enhancement than is keeping hop count low, this is not a good choice for most performancecritical applications.Table 7.3 charts the properties of a full-mesh topology.
NOTE Special cases: A 2-switch full mesh is identical to a 2-switch cascade. A 3-switch full mesh is identical to a 3-switch ring.
Table 7.3 Properties of a Full-Mesh Topology Limit of scalability (edge port count) 72 ports / 8 switches Properties
Ratings
Ease of scalability Performance Ease of deployment Reliability Cost (per edge port)
1 2 3 3 1
Ratings indicate how well the topology meets the ideal requirements of that property (1 = Not well, 2 = Moderately well, 3 = Very well).
Partial-Mesh Topology The common definition for a partial mesh (Figure 7.6) is broad enough to encompass almost all SANs that are not full meshes. A partial mesh is defined by Brocade as follows: “A partial mesh is similar to a full mesh, but with some of the ISLs removed. In most cases, this will be done in a structured pattern (for example, each switch will directly connect to its neighbor, and to every other
140_SANs_07
8/14/01
3:38 PM
Page 241
Developing a SAN Architecture • Chapter 7
switch across from it).”While this definition is not in general use outside of Brocade, it describes a desirable variant on the full mesh. Figure 7.6 A Partial-Mesh Topology
The network in Figure 7.6 might be useful if you knew that traffic would never flow between the two left-hand switches.The network still is fully resilient to failure, but you are not paying a price premium for an ISL that will not be used. Partial meshes also scale farther than full meshes. Figure 7.7 shows a partial mesh that has 176 free ports. (Remember that the largest full mesh has 72 ports.) Each switch is connected to its neighbor.Two switches are skipped before the next connection.The worst-case hop count between switches in the event of an ISL failure is three hops. Figure 7.7 Maximum Size of a Partial-Mesh Topology
241
140_SANs_07
242
8/14/01
3:38 PM
Page 242
Chapter 7 • Developing a SAN Architecture
While these networks can be scaled to produce a large number of edge ports, they still have performance characteristics that are less than ideal. None of the networks listed thus far will benefit much from FSPF load-sharing capabilities, for example. Since bandwidth is more frequently a concern than hop count (Fibre Channel latency being effectively a second-order derivative in most real-world performance calculations), the ability of a topology to load share across ISLs is key to its performance. Moreover, partial meshes can be difficult to scale without downtime.The procedure for moving from the full-mesh fabric in Figure 7.5 to the partial-mesh fabric in Figure 7.7 would require not only adding new switches and potentially disconnecting nodes, but actually disconnecting ISLs that were already in place. The same is true for scaling between many partial-mesh designs.This would be disruptive to many production SANs, especially if redundant fabrics were not used.Therefore, meshes—either full or partial—are recommended only for networks that will change infrequently.They might also be used as a static component of a network. For example, a full mesh could be used in an environment where the “SAN islands” architecture was employed, or as the core of a complex core/edge design (which we discuss later).Table 7.4 charts the properties of a partial-mesh topology. Table 7.4 Properties of a Partial-Mesh Topology Limit of scalability (edge port count)
176+ ports / 16+ switches
Properties
Ratings
Ease of scalability Performance Ease of deployment Reliability Cost (per edge port)
1 1 2 3 2 to 3
Ratings indicate how well the topology meets the ideal requirements of that property (1 = Not well, 2 = Moderately well, 3 = Very well).
Core/Edge or Star Topologies In a resilient core/edge fabric (Figure 7.8), two or more switches will reside in the center of the fabric (the core) and interconnect a number of other switches (the edge). Switches that reside in the middle of the fabric are referred to as core
140_SANs_07
8/14/01
3:38 PM
Page 243
Developing a SAN Architecture • Chapter 7
switches.The switches that are interconnected by the core switches are referred to as edge switches. Devices such as hosts and storage are attached to free ports on the edge switches.These ports are referred to as edge ports. Free ports (if any) on the core switches should be reserved for use as ISLs, in order to avoid limiting the fabric’s growth potential. Figure 7.8 A Core/Edge or Star Topology Edge
Core
We will focus on the core/edge fabric as being a solution for scalable fabrics for a number of reasons.The core/edge topology is: ■
Easy to grow without downtime (“pay as you grow”).
■
Easy to transition to future large core fabric switches, and good at providing investment protection.
■
Economical, with a good cost-to-performance ratio.
■
Simple and easy to understand.
■
Well tested and reliable.
■
Proven in traditional data networks such as Ethernet.
■
Capable of exhibiting stellar performance, with full utilization of FSPF load sharing and redundancy features.
■
Conducive to performance analysis. Performance in a core/edge SAN is deterministic, and you can easily determine how much bandwidth any given switch has to “get to” any other switch.
■
Scalable to hundreds of ports now, and thousands in the future.
■
Able to solve most design problems, and a good choice when design requirements are not well known.
243
140_SANs_07
244
8/14/01
3:38 PM
Page 244
Chapter 7 • Developing a SAN Architecture
Again, we must stress that the simple core/edge SAN is by no means the only way to build scalable networks with Brocade switches.There are limitless ways in which Brocade switches could be interconnected.We will focus on this particular topology in the hopes that it will be beneficial to the majority of designers, and we will leave the limitless variations on this and other topologies as exercises for the reader.
Simple Resilient Core/Edge Topology A simple resilient core/edge fabric has two or more core elements, each of which consists of a single switch. All of the core/edge fabrics depicted thus far are simple.Two core elements are recommended to maintain a high level of resilience and avoid a single point of failure (Figure 7.9).Table 7.5 charts the properties of a simple resilient core/edge topology. Figure 7.9 A Simple Resilient Core/Edge Topology
Two core elements ( and ) each of which is a single switch
Table 7.5 Properties of a Simple Resilient Core/Edge Topology Limit of scalability (edge port count)
224 ports / 20 switches
Properties
Ratings
Ease of scalability Performance Ease of deployment Reliability Cost (per edge port)
3 3 3 3 2
Ratings indicate how well the topology meets the ideal requirements of that property (1 = Not well, 2 = Moderately well, 3 = Very well).
Complex Core Resilient Core/Edge Topology A complex core resilient core/edge SAN (Figure 7.10) has two or more core elements, each of which consists of multiple switches.These designs are more
140_SANs_07
8/14/01
3:38 PM
Page 245
Developing a SAN Architecture • Chapter 7
complex, more expensive to build and maintain, and are generally not necessary. However, in certain cases, you might want to expand beyond 16 edge switches, and large-port-count core switches might not be available. In this case, you could build a larger fabric out of complex core elements.Table 7.6 charts the properties of a complex core/edge topology. Figure 7.10 A Complex Core Resilient Core/Edge Topology
Table 7.6 Properties of a Complex Core Resilient Core/Edge Topology Limit of scalability (edge port count)
300+ ports / 28+ switches
Properties
Ratings
Ease of scalability Performance Ease of deployment Reliability Cost (per edge port)
2 2 1 3 1
Ratings indicate how well the topology meets the ideal requirements of that property (1 = Not well, 2 = Moderately well, 3 = Very well).
Composite Resilient Core/Edge Topology A composite resilient core/edge SAN has two or more cores (Figure 7.11). Each core consists of two or more single-switch elements. It could be viewed as two
245
140_SANs_07
246
8/14/01
3:38 PM
Page 246
Chapter 7 • Developing a SAN Architecture
stars “glued together at the edge.”This is useful when using a tiered approach to performance optimization (we discuss tiered SANs later). Figure 7.11 A Composite Resilient Core/Edge Topology
Two cores ( and ) each of which has two elements, each of which is a single switch
Topologies at a Glance Table 7.7 on the following page provides a compilation of all of the characteristics of topologies listed thus far, so that you can compare them easily.
Complex Topologies It is possible to build arbitrarily complex architectures by connecting switches together in a seemingly random way. For example, a ring could be connected to the edge of a core/edge fabric, which could also have a cascade and a smaller core/edge fabric hanging off it.What would you call this SAN other than “complex?”While this approach might be desirable in certain cases, it is usually better to use a more structured approach to network design.This will ensure flexibility and the maintainability of the network. Even without deviating from the “simple” core/edge network—or, indeed, from a simple full mesh, or ring—it is possible to provide for variations in performance requirements. Figure 7.21 later in this chapter is an example of an asymmetrical core/edge network, which is still geometrically “simple.”
Working with the Core/Edge Topology Building one or more core/edge fabrics is, at the time of this writing, the best way to implement a general-purpose scalable SAN. Again, more experienced designers should feel free to implement whatever topology they see fit; we are simply advocating the core/edge approach because it can solve most problems easily, which is, after all, what most users want.This section provides you with some tips on how to use your core/edge fabric most effectively.
1 3
1 3
247
3 3
3 1
2 3
1
3 2 to 3
1 2
1
Partial Mesh Ratings
176+ ports / 16+ switches
3 2
3 3
3 1
2 1
2
Complex Core/Edge Ratings
Simple Core/Edge Ratings 3
300+ ports / 28+ switches
224 ports / 20 switches
Ratings indicate how well the topology meets the ideal requirements of that property (1 = Not well, 2 = Moderately well, 3 = Very well).
1 3
2
3
Ease of scalability Performance Ease of deployment Reliability Cost (per edge port)
Full Mesh Ratings
72 ports / 8 switches
3:38 PM
Ring Ratings
Cascade Ratings
Topology Properties
112 ports / 8 switches
114 ports / 8 switches
8/14/01
Limit of scalability (edge port count)
Table 7.7 Topology Properties Comparison
140_SANs_07 Page 247
140_SANs_07
248
8/14/01
3:38 PM
Page 248
Chapter 7 • Developing a SAN Architecture
Scaling without Downtime You can scale a resilient core/edge network without any downtime.This requires a small amount of planning and attention to detail, but it is well worth the effort. Here are two examples of how you can scale these networks easily, by adding an edge switch and upgrading the core. The procedures listed here are designed to be applicable in a large, real-world production environment, where a fine degree of control over the SAN is desired. Consequently, they are much more complex than the plug-and-play approach, which would be used in a less-demanding environment. For example, it is possible to add an edge switch by simply plugging it in, turning it on, and allowing the Brocade Fabric OS’s plug-and-play features to do the configuration for you. In larger production environments, however, it is usually desirable to control the process manually, to ensure that the new switch not only enters the fabric, but also does so in accordance with your site-specific administration policies. For example, Fabric OS can automatically pick a domain ID for you. Most large sites have a structured approach to domain ID assignment, and in these cases you would want to assign the domain ID manually before adding the new switch to the fabric.The following manual procedures are not necessary, but are recommended in these more tightly controlled environments.We leave it to you to decide what level of manual control you wish to take in this process, and how much control you want to give to the Fabric OS.
NOTE It is best to use dual fabrics to achieve the highest possible uptime. The use of resilient fabrics, like the mesh, ring, or core/edge, is part of a fully redundant—or High-Availability (HA)—SAN solution, but not the whole of it.
Adding an Edge Switch You might want to implement only part of the edge of your core/edge fabric at the beginning, and add the rest of the edge switches later. In other words, you can take the “pay-as-you-grow” route.This is easy to do without downtime, as long as you do not use the free ports in the core switches for edge devices, but rather reserve them for scalability.
140_SANs_07
8/14/01
3:38 PM
Page 249
Developing a SAN Architecture • Chapter 7
Let us say that your architecture (as shown in Figure 7.12, Step C) is a 10switch resilient core/edge fabric. Let us say further that you have only implemented half of the edge (Figure 7.12, Step A), and then decide that you want to add one more edge switch (Figure 7.12, Step B). Figure 7.12 Adding an Edge Switch
Step A 4ex2cx2i
Step B 5ex2cx2i
Step C 8ex2cx2i
Step One: Setting Up the New Switch Set the new switch up by itself in the location where it will attach to the SAN (bolt it to the rack), but do not attach it to the SAN at this point.
NOTE We discuss the process of adding a switch to the fabric in greater detail in Chapter 9, “SAN Implementation, Maintenance, and Management.”
Power on the switch, assign it a host name and an IP address, and then use either the telnet or Brocade WEB TOOLS interface to configure its domain ID, switch name, and any special variations of fabric parameters—such as ErrorDetect Time Out Value (E_D_TOV)—that you are using at your site. Ensure that no zoning configuration is in effect.
Step Two: Connecting the New Switch Issue the portDisable command on the new switch for each port that you intend to use as an ISL. For example, if you are using ports 0 through 3 as the ISL ports, you would execute the following commands: newswitch:admin> portDisable 0 newswitch:admin> portDisable 1 newswitch:admin> portDisable 2 newswitch:admin> portDisable 3
249
140_SANs_07
250
8/14/01
3:38 PM
Page 250
Chapter 7 • Developing a SAN Architecture
Attach cables to these ports and route them to the appropriate core switches. For example, ports 0 and 1 could be connected to one core switch, and 2 and 3 could be connected to the other core switch. Re-enable the first port (port 0). At this point, the new switch should join the fabric. After the fabric has reconfigured, you can issue the fabricShow command to see if it has indeed joined the fabric: newswitch:admin> portEnable 0 newswitch:admin> fabricShow
Once you have verified that the switch joined the fabric successfully, reenable the remaining ports.While not necessary, this procedure will help ensure minimal disruption to your fabric.
Upgrading the Core As technology progresses, inevitably there will be a “bigger, better, faster” product available that you might wish to purchase. It is quite likely that you will want this new product to reside in the core of your fabric, as this position will probably benefit the most from the features that caused you to buy the product in the first place.This approach is common in all areas of networking: when you build a LAN in a rapidly growing company, you might use several entry-level Ethernet switches as your backbone.When the environment grows, you might replace them with workgroup switches, and then perhaps a bladed chassis, but the entrylevel switches that you used at the start will still be included in the network; they will simply not form the core of the network anymore. For this example, we will say that you have a fully populated 18-switch resilient core/edge fabric (16ex2cx1i), as shown in Figure 7.13. Figure 7.13 A Fully Populated 18-Switch Resilient Core/Edge Fabric
140_SANs_07
8/14/01
3:38 PM
Page 251
Developing a SAN Architecture • Chapter 7
Since the core switches in this design are fully populated, you cannot add any more edge switches. However, you can change the core to a larger switch without downtime.This will allow you to add more edge switches and will even give you the first two new edge switches “for free” in the bargain!
Step One: Adding the New Core Switches As in the previous example, you should put the new core switches in their correct physical location, but not attach them to the fabric at this point.You should configure their IP addresses, domain IDs, and so on, and then clear the zoning configuration as before. However, you do not need to disable all of the ports that will be used for ISLs. Use the switchDisable command to disable all ports on that switch. Again, this is not necessary, and you can allow Fabric OS to do the work for you if your site does not require such fine control. Telnet into the first of the two existing core switches that you want to replace. Issue the switchDisable command: oldswitch:admin> switchDisable
Verify that all traffic is going over the remaining core switch. After you are sure that the SAN is functioning correctly on its one remaining core, you can power off the disabled core. Remove the ISL cables from the switch and the rack cable management structure.The switch should now be completely separated from the SAN (Figure 7.14). Cable the first new core switch—the one that you have configured and disabled—to one of the edge switches. Figure 7.14 Removing the First Core Switch
Step Two: Configuring the New Core Switches Issue the switchEnable command. As before, the new switch should join the fabric.You might see a fabric reconfigure message on the telnet session. After the fabric has reconfigured, you can issue the fabricShow command:
251
140_SANs_07
252
8/14/01
3:38 PM
Page 252
Chapter 7 • Developing a SAN Architecture newswitch:admin> switchEnable newswitch:admin> fabricShow
One at a time, cable in the remaining edge switches.When you are done, the SAN should look like Figure 7.15. Figure 7.15 Adding the New Core Switch
Repeat the procedure with the other new core switch, and you will have the SAN shown in Figure 7.16. Figure 7.16 Newly Installed Core Switches
You also have two extra switches (the former core switches that are sitting off to the side disconnected). Follow the procedure for adding an edge switch to bring these back into the fabric (Figure 7.17).This set of procedures allows you to future-proof your SAN, because it gives you an architecture that has a migration path to future technologies already built into it.
140_SANs_07
8/14/01
3:38 PM
Page 253
Developing a SAN Architecture • Chapter 7
Figure 7.17 Redeploying the Former Core Switches Install core fabric switches
Redeploy to the edge
Target Designs for a Core/Edge SAN If you have a rough idea of your port count and performance requirements, but do not want to spend much time designing an optimized SAN architecture, you can pick one of the following four designs and “be done with it.” You do not have to build the entire design at once. You can build toward the target design, and “pay as you grow.” Design One (the 10-switch core/edge), Design Two (the 20-switch core/edge), and Design Three (the 18-switch core/edge) are “pure” designs. They are simple, symmetrical, and easy to understand and deploy. You can “start small and grow large“ with them. Design Four (the 14-switch core/edge) is a combination of Designs One and Three. The approach used in Design Four is less symmetrical, but might be desirable if parts of your SAN have different performance requirements, or if you are expanding an existing SAN.
Design One The 10-switch core/edge design, shown as Design One in Figure 7.18, is ideal for SANs that require high performance, have little known locality, and will not immediately grow beyond 96 edge devices. Identification
8 edge by 2 core by 2 ISL (8ex2cx2i)
Edge ports ISL over-subscription Switch count
96 3:1 10 Continued
253
140_SANs_07
254
8/14/01
3:38 PM
Page 254
Chapter 7 • Developing a SAN Architecture
Figure 7.18 Design One (10-Switch Core/Edge)
Design Two Design Two, the 20-switch core/edge design shown in Figure 7.19, is very similar to Design One. It uses twice as many switches, produces twice as many ports, and performs exactly the same way. This design is appropriate for SANs that require high performance, have little known locality, and will not immediately grow beyond 192 edge devices. Identification
16 edge by 4 core by 1 ISL (16ex4cx1i)
Edge ports ISL over-subscription Switch count
192 3:1 20
Figure 7.19 Design Two (20-Switch Core/Edge)
Continued
140_SANs_07
8/14/01
3:38 PM
Page 255
Developing a SAN Architecture • Chapter 7
Design Three Design Three, the 18-switch core/edge design shown in Figure 7.20, produces more ports, and takes fewer switches to implement than the previous design. It is substantially more cost-effective. The tradeoff is that the performance is lower, so it is appropriate if high performance is not critical, or if some locality is known. Identification
16 edge by 2 core by 1 ISL (16ex2cx1i)
Edge ports ISL over-subscription Switch count
224 7:1 18
Figure 7.20 Design Three (18-Switch Core/Edge)
Design Four Design Four, the 14-switch core/edge design shown in Figure 7.21, balances the higher performance of Design One with the lower cost and higher port count of Design Three. This approach is appropriate if you have some devices that require high performance, and some that either do not require that degree of performance or have known locality. Many variants are possible that use this approach, and it is possible to yield anywhere between 96 ports and 224 ports. Identification
12 edge by 2 core by [1 or 2] ISL (12ex2cx[1/2]i)
Edge ports ISL over-subscription Switch count
160 3:1/7:1 14 Continued
255
140_SANs_07
256
8/14/01
3:38 PM
Page 256
Chapter 7 • Developing a SAN Architecture
Figure 7.21 Design Four (14-Switch Core/Edge)
Like the SilkWorm 6400
It is also useful to take this approach when expanding a SilkWorm 6400 Integrated Fabric. The internal ISL topology of the SilkWorm 6400 is like an incomplete Design One SAN. You can complete the SAN exactly like Design One by attaching four switches to the SilkWorm 6400’s outer modules (modules 1 and 6), attaching eight switches to get Design Four, or attaching some switches like Design One and some like Design Four. These four designs will solve most SAN design problems and will all migrate easily to incorporate high-port-count core fabric switches in the future.
Determining Levels of Availability Resiliency is the capability of a fabric topology to withstand failures.This is equivalent to the fault-tolerant level of availability seen in many high-uptime RAID arrays or enterprise-class servers.The core/edge, mesh, and ring topologies provide at least two internal fabric routes and are considered resilient because each topology can withstand a switch or ISL failure while the remaining switches remain operational.Thus, the fabric can “heal” an “injury” without any operator intervention.This self-healing capability is enabled by the Brocade-authored FSPF protocol. (FSPF is now the standard Fibre Channel routing protocol.Visit www.t11.com for details on this and other Fibre Channel standards.) Redundancy is the duplication of components up to and including the entire fabric to prevent the failure of the SAN solution. For example, an airplane hydraulic system is resilient to failures. However, most jumbo jets also have redundant hydraulic systems so that the jet will not crash even if the resiliency fails to keep the system up. Redundancy is equivalent to the HA model popular
140_SANs_07
8/14/01
3:38 PM
Page 257
Developing a SAN Architecture • Chapter 7
with critical server farms using products like VERITAS Cluster Server or Microsoft Cluster Server (MSCS). In a redundant SAN architecture, you must have at least two completely separate fabrics—just as an HA server solution requires at least two completely separate servers.This type of design not only accounts for software and hardware failures, but also for the human errors that can be the more common cause of system downtime. As the fabrics are independent, an incorrect operator-initiated change on one fabric (such as disabling an entire switch instead of just a port) would not affect the redundant SAN. Since the other SAN has a separate connection to the devices that have been disconnected from the first fabric, the services performed by those devices can continue. Because of this, even a larger (director-class) single switch with highly available characteristics is still a single point of failure, and should be used with caution in HA environments. There are four primary categories of availability in SAN architecture. In order of increasing availability, they are: ■
Single fabric, nonresilient All switches are connected to form a single fabric, which contains at least one single point of failure.The cascade topology is an example of this category of SAN.
■
Single fabric, resilient All switches are connected to form a single fabric, but there is no single point of failure that could cause the fabric to segment.The ring, mesh, and core/edge topologies are examples of single, resilient fabrics.
■
Dual fabric, nonresilient Half of the switches are connected to form one fabric, and the other half form an identical, separate fabric.Within each fabric, at least one single point of failure exists.This design can be used in combination with dual-attached hosts and storage devices to keep a solution running even if one fabric fails.
■
Dual fabric, resilient Half of the switches are connected to form one fabric, and the other half form an identical, separate fabric. Neither fabric has a single point of failure that could cause the fabric to segment. This design can be used in combination with dual-attached hosts and storage devices to keep an application running even if one entire fabric fails due to operator error, catastrophe, or quality issues.This is generally the best design approach for HA requirements. Another key benefit of a dual-SAN design is the capability to take half of the dual fabric offline for upgrades or maintenance without affecting production operations on the remaining fabric.
257
140_SANs_07
258
8/14/01
3:38 PM
Page 258
Chapter 7 • Developing a SAN Architecture
Figure 7.22 depicts the failure of a switch in a cascade topology. Switches A and B are unable to communicate with the remaining switches when the switch marked with the “X” fails, resulting in segmentation of the fabric into three smaller fabrics. Figure 7.22 Switch Failure in a Nonredundant Cascade Topology B A
C
Segmentation
D E
This fabric is neither resilient nor redundant.The uptime of this fabric solution could be improved by duplicating the fabric, as in Figure 7.23. In this example, even though one of the fabrics segmented, all devices are attached to another SAN that is still working. A switch failure in a ring, mesh, or core/edge topology SAN does not cause a loss of communication with the remaining switches, as shown in Figure 7.24, for the core/edge topology. If one core switch fails, the alternative core switch can still communicate with all edge switches.This will allow the SAN to continue to operate.The failover to alternate paths is performed by FSPF and is completely transparent to the users of the fabric.These topologies are therefore considered resilient. In order to ensure maximum uptime, you should use both resilience and redundancy. Figure 7.25 shows a resilient and redundant SAN solution. Redundant fabrics are desirable because they provide the most effective protection against hardware and software failures, and protect against user error. For example, if a SAN administrator were to telnet into SAN A and accidentally damage the zoning configuration—making Fabric A unusable—SAN B would not be affected.
140_SANs_07
8/14/01
3:38 PM
Page 259
Developing a SAN Architecture • Chapter 7
Figure 7.23 Switch Failure in a Redundant Cascade Topology
Devices
B A
C D E
Figure 7.24 Switch Failure in a Resilient Ring, Mesh, or Core/Edge Topology
Failed Core Switch
Alternate Path
259
140_SANs_07
260
8/14/01
3:38 PM
Page 260
Chapter 7 • Developing a SAN Architecture
Figure 7.25 A Resilient and Redundant SAN Solution
Hosts
SAN A
SAN B
Storage
NOTE Whenever you have a mission-critical application that requires the highest uptime possible, you should use resilient/redundant fabrics to build the SAN infrastructure, in conjunction with other HA software and components.
140_SANs_07
8/14/01
3:38 PM
Page 261
Developing a SAN Architecture • Chapter 7
Configuring Traffic Patterns There are a number of approaches possible for optimizing traffic patterns within a fabric, or an overall SAN.This section discusses two approaches: the tiered approach and the localized approach.
Leveraging Tiers In many SANs today, traffic predictably flows between hosts and storage devices, not between hosts and other hosts, or storage devices and other storage devices. If your SAN is used primarily to support hosts accessing storage, you can optimize traffic flow in your SAN by following the tiered approach described in this section.
NOTE The trend in leveraging tiers might change as IP over Fibre Channel, third-party copy, peer-to-peer copy, and cluster technologies (VI) are more widely adopted. These technologies require host-to-host or storage-to-storage data flow. However, these new technologies will represent a relatively small fraction of the traffic in a SAN, so even if these technologies are widely adopted, the tiered approach will still be valid.
For example, in Figure 7.26, you have a 4-switch full mesh with hosts attached to two of the switches and storage attached to the others. Since we can assume that the only traffic patterns we will see with any regularity are between hosts and storage, we can change the ISL topology as seen in Figure 7.27.
261
262
8/14/01
3:38 PM
Page 262
Chapter 7 • Developing a SAN Architecture
Figure 7.26 Analyzing Traffic Flow in a Full Mesh
Hosts
Traffic
140_SANs_07
No Traffic
Storage
Removing the unused ISLs increases the number of available ports without affecting resiliency.This improves the scalability and the cost/benefit ratio of the SAN, with very little effort.
8/14/01
3:38 PM
Page 263
Developing a SAN Architecture • Chapter 7
Figure 7.27 Optimizing Traffic Flow by Using a Two-Tier Approach
Hosts
Traffic
140_SANs_07
Storage
In the example in Figure 7.28, the switches on the right are the host tier switches and the switches on the left comprise the storage tier.This is a tiered SAN because of the way it is being used, not because of its topology.The initial topology was a full mesh, and the modified topology is a ring. It can also be viewed as a two-tier SAN from a performance standpoint.
263
140_SANs_07
264
8/14/01
3:38 PM
Page 264
Chapter 7 • Developing a SAN Architecture
Figure 7.28 Storage Tier and Host Tier in a Two-Tier SAN
Hosts
Host Tier
Storage Tier
Storage
NOTE Performance-related terms such as tiered SAN or localized traffic do not refer to a topology. However, the use of these performance techniques might influence the choice of topology for a SAN.
140_SANs_07
8/14/01
3:38 PM
Page 265
Developing a SAN Architecture • Chapter 7
The benefit of tiered SANs is that they do not need ISLs or bandwidth optimization between switches on the same tier. Figure 7.29 is an example of a three-tier SAN. In addition to a host and storage tier, it also has a core tier.This SAN is a variation on the core/edge architecture that exploits the performance characteristics of tiered SANs while maintaining the advantages of the core/edge SAN. It is actually a composite of two 18-switch resilient core/edge SAN components.Tiered SANs are effective for managing your SAN.When you need to add hosts, you also add a host switch.When you need to add storage, you add a storage switch if ports are not available.This organization makes it easier to manage your SAN and simplifies your job as a SAN administrator. Figure 7.29 A Three-Tier SAN
Hosts
Hosts
Core Tier
Storage
265
266
8/14/01
3:38 PM
Page 266
Chapter 7 • Developing a SAN Architecture
Exploiting Locality Locality is another performance optimization strategy. It is the opposite of a tiered approach and will always outperform the tiered SAN. However, it also takes more planning. You can attain the best performance in any network design by understanding the traffic patterns the network will transport. If these patterns are well understood, it might be possible to localize traffic by putting ports that need to communicate with each other close together.This concept is known as locality and is used not just in SAN design, but also throughout computer science. The amount of known locality in a SAN will combine with application performance requirements to determine the number of ISLs required to achieve the SAN design objectives.This in turn will affect the number of edge ports and the cost of the SAN infrastructure. In Figure 7.30, the servers attached to Switch A are using storage also attached to Switch A.The servers attached to Switch B are using storage also attached to Switch B. No data should cross the ISL connecting the two switches. In the SAN shown, traffic has been 100 percent localized. Figure 7.30 Localizing Traffic 100 Percent
Traffic Flow
Switch A
Switch B
ISL
140_SANs_07
Traffic Flow
140_SANs_07
8/14/01
3:38 PM
Page 267
Developing a SAN Architecture • Chapter 7
Sometimes it is impossible to determine anything at all about locality during the SAN design phase. In the case of Figure 7.31, locality is at zero percent.The single ISL in Figure 7.31 is over-subscribed, because many more devices could be trying to use it than it can simultaneously support. Over-subscription is not necessarily a bad thing and does not always affect SAN performance. Only congestion—the realization of the potential of over-subscription—can affect performance. Figure 7.31 Localizing Traffic Zero Percent
Traffic Flow
Traffic Flow Switch A
Switch B
Traffic Flow
Frequently, locality knowledge will be neither zero percent nor 100 percent. For example, it might be possible to localize all traffic between a group of hosts and a RAID array, but the hosts might be sharing access to a tape device that needs to be located on a different switch (Figure 7.32). While smaller SANs will not benefit as greatly from the use of locality as large SANs will, all SANs will benefit somewhat.The use of locality will reduce cost, increase scalability, increase reliability, and improve performance. It is always worth making some effort to use locality in a SAN design. However, for low-bandwidth applications, the management benefits of organizing your edge devices in a tiered fashion are significant, and zero-percent locality can be quite acceptable.
267
268
8/14/01
3:38 PM
Page 268
Chapter 7 • Developing a SAN Architecture
Figure 7.32 Using Locality in a SAN Design
90% of traffic
10% of traffic
140_SANs_07
Using Any-to-Any Connectivity A well-designed SAN should employ locality to minimize congestion as much as possible. However, congestion usually occurs only in unique traffic conditions, such as the following: ■
The SAN application is extremely bandwidth-intensive. For example, certain video applications use a large I/O size to the order of 64 KB or larger, and typically consume 80 MB/sec to 100 MB/sec of bandwidth. More common are low-bandwidth applications such as Online Transaction Processing (OLTP) or e-commerce, where the typical I/O size is approximately 2 KB to 8 KB, and bandwidth consumption peaks at 16 MB/sec.
■
The majority of I/O is streaming, as opposed to bursty at peak utilization. In a SAN environment, it is unlikely that all devices will concurrently require peak utilization, since most SAN traffic is bursty and in this respect similar to LAN traffic.
140_SANs_07
8/14/01
3:38 PM
Page 269
Developing a SAN Architecture • Chapter 7 ■
The majority of SAN devices support 100 MB/sec or 200 MB/sec throughput. However, most current storage and server devices are incapable of sustaining 100 MB/sec or 200 MB/sec throughput. For example, some of the fastest tape drives can deliver a maximum native throughput of only 14 MB/sec. In many cases, network congestion is only theoretically possible. Some newer SAN deployments might end up with a larger mix of higher throughput devices than existing SANs that have evolved gradually over time. Even when the hardware is capable of maximizing the HBA’s theoretical bandwidth, it is rare to have an application that needs to do so on a sustained basis. Due to the typical capabilities of the SAN edge devices and normal traffic patterns, congestion rarely occurs in most SANs.
If you cannot determine locality or host/storage locations in advance, or simply feel that your SAN does not have much potential for congestion and you do not need to use locality, you can plan for any-to-any connectivity.This could also be necessary if you are using a clustering application or a data replication application that requires hosts to talk to other hosts and storage to talk to other storage. The core/edge topology is the preferred way to build any-to-any connectivity SANs, since it is symmetrical and deterministic performance characteristics are well suited for it. It might be desirable to use a “thicker” ISL structure for an any-to-any SAN.This provides a lower degree of over-subscription. In a case where traffic patterns are totally unknown, high levels of over-subscription are likely to turn into congestion.The core/edge designs that are best suited for this are the 8ex2cx2i and 16ex4cx1i networks. Each of these SANs provides for a 3:1 over-subscription ratio, which is usually perfectly acceptable.
Evaluating Performance Considerations Building a fabric with very low ISL over-subscription can eliminate performance constraints within the fabric. However, this might not be desirable. If the devices attached to the fabric have performance limits greater than the limits imposed by the ISL topology of the fabric, the extra ISLs used will add nothing to the solution but cost.This section describes that principle, and will help you evaluate how much performance you need to build into your fabric.
269
140_SANs_07
270
8/14/01
3:38 PM
Page 270
Chapter 7 • Developing a SAN Architecture
When Is Over-Subscription Bad? Since over-subscription is only a potential for link contention, it is never a problem in and of itself. In fact, over-subscription is the normal state in almost all networks in use today. For example, over-subscription is deliberately designed into the Internet as a way of reducing cost. Congestion, which is the realization of the potential of over-subscription, is usually only a problem when the congestion is sustained. If you have a link that becomes congested for a total of five minutes per day and is underutilized the rest of the time, there is not really a justification for adding another link. If, however, the link is congested half of the day, you should re-evaluate the SAN’s performance optimization strategy.The crossover point is usually this: If the congestion is significantly affecting your application’s performance, you need to eliminate or at least reduce it. Fabric OS provides visibility into the fabric’s performance by providing accurate performance metrics for ports and switches, as well as methods for setting thresholds for proactive notification of overutilization. Refer to the Fabric Watch manual for information on thresholds and the Fabric OS manual on general performance measurements.
Considerations Outside the Fabric The fabric itself can affect performance by being congested or having many long-distance (for example, 50 to 100 km) and relatively latency-heavy links. Again, we must stress that “latency-heavy” is relative. Even such a long-distance link is typically at least one order of magnitude faster than the storage devices that it serves. Moreover, the higher degree of latency on these links is caused by the speed with which light travels through the glass fiber-optic cable—since there is as yet no way to make a signal travel faster than light, this latency must be considered acceptable.These considerations can create bottlenecks in a SAN, which will limit the performance of applications using it. However, much more frequently, devices outside the fabric create the bottleneck. Some things that do so include: ■
CPU speed
■
PCI bus speed
■
Resource sharing
■
Application I/O profile
■
File system block access profile
140_SANs_07
8/14/01
3:38 PM
Page 271
Developing a SAN Architecture • Chapter 7
See Chapter 5, “The SAN Design Process,” for a detailed discussion of these items. In short, if you know that you have performance limits outside the fabric, there is no reason to design the fabric itself to avoid over-subscription. If the devices attached to the fabric cannot or do not generate 1 Gbit/sec of sustained traffic (they very rarely do), it is not necessary for the SAN to support that.The built-in over-subscription will never become congestion in this case. It will simply save you money, since you will not have to use as many ports for ISLs. Of course, if you find subsequently that there is a need for more bandwidth, it is a simple matter of connecting more ports to achieve additional bandwidth.
271
140_SANs_07
272
8/14/01
3:38 PM
Page 272
Chapter 7 • Developing a SAN Architecture
Summary A fabric consists of one or more interconnected Fibre Channel switches. A SAN includes one or more related fabrics and everything attached to them. Some fabric topologies are better suited for general purpose use—such as the core/edge topology—while other topologies—such as the cascade, ring, full mesh, and partial mesh—might be useful in more limited or special-case deployments. Core/edge fabrics have many useful traits.They can scale to accommodate a large number of ports, and do so easily. A resilient core/edge fabric can be scaled “on the fly.” Switches can be added or replaced in the fabric without downtime. This includes the core switches, as long as the core is resilient. Core/edge fabrics also handle varying degrees of locality well.This topology is the most frequently recommended by Brocade and is the best general-purpose choice.There are four availability models in SAN architecture. In order of increasing availability, they are: 1. Single fabric, nonresilient 2. Single fabric, resilient 3. Dual fabric, nonresilient 4. Dual fabric, resilient The last of the models—dual fabric, resilient—is always recommended for applications where high uptime is a strong requirement.This kind of SAN consists of two usually identical, completely unconnected fabrics, neither of which contains a single point of failure. It is also possible to build SANs with more than two fabrics for even greater availability. (For example, the “triple fabric, resilient” topology consists of three fabrics.) While a tiered approach to SAN architecture can simplify management and storage resource planning, the most effective approach to performance tuning a SAN is to localize traffic within areas of the SAN. Applying the principle of locality takes a certain amount of work on the front end of a project and adds to the complexity of managing the growth of a fabric. However, when properly applied, locality greatly enhances performance and scalability. Using locality is one way to ensure that the ISLs in a fabric will not become over-subscribed. Over-subscription within a fabric is not necessarily a bad thing. A well-designed fabric can actually benefit from deliberate use of over-subscription, which can drive down cost, improve manageability, and decrease network complexity. Over-subscription affects performance only when it becomes congestion.
140_SANs_07
8/14/01
3:38 PM
Page 273
Developing a SAN Architecture • Chapter 7
You should look for performance limitations outside the fabric—such as host and storage limitations, or application I/O profiles—before building a low-oversubscription fabric.This will produce a topology that will provide the most appropriate performance characteristics for your environment. However, if you are unable to do this kind of in-depth analysis, there are a number of core/edge topologies presented in this chapter from which you can choose a general purpose approach to SAN design.These topologies are well tested, reliable, scalable, and perform well with most I/O profiles.
Solutions Fast Track Identifying Fabric Topologies and SAN Architectures A fabric consists of one or more interconnected Fibre Channel switches. A SAN includes one or more related fabrics and everything attached
to them. In a resilient core/edge fabric topology, two or more switches act as a
core to interconnect multiple edge switches.This is the best “generaluse” topology available, especially when combined with the dual-fabric approach to SAN architecture. In order to select the right topology, you must first decide the require-
ments for your SAN architecture.This includes redundancy and scalability in addition to port count. In general, the cascade, ring, full mesh, and partial mesh are best used
in architectures where the individual fabrics that comprise the SAN will not change much.This could be true in a static, low-growth environment, or in a “SAN islands” design. The resilient core/edge topology is the best choice for general use and
for situations where SAN requirements are either unknown or might change frequently. The resilient core/edge topology can be combined with dual fabrics to
achieve maximum performance, reliability, and scalability.
273
140_SANs_07
274
8/14/01
3:38 PM
Page 274
Chapter 7 • Developing a SAN Architecture
Working with the Core/Edge Topology The core/edge topology offers a number of key advantages over other
topologies. Core/edge fabrics are: —Easy to scale without downtime. —Capable of scaling to a large number of ports. —Flexible in terms of their cost-to-performance ratios. (You can add switches to the core with a clear knowledge of how doing so will affect both cost and performance.) —Easy to understand, manage, and performance-tune. —Well-tested and reliable. Several core/edge fabrics can be used as “cookie-cutter fabrics” when
design information is incomplete or might change frequently.
Determining Levels of Availability There are four levels of availability that a SAN architecture might
employ.The dual-fabric, resilient approach is the most reliable and the most frequently recommended. In most cases, this approach is not more expensive to implement than
the other three approaches, and it might be less expensive in some cases. This approach allows for the failure of anything up to and including an
entire fabric without application downtime.
Configuring Traffic Patterns Tiered fabrics allow simplified management and storage resource plan-
ning, but are the worst-case scenario from the standpoint of locality. Locality is the most effective approach to performance tuning in a SAN,
but it is frequently unattainable. You should view locality as a “moving target,” since network complexity
increases over time. However, it is worth getting as much locality as is practical into a SAN, since all SANs benefit in several ways from this technique.
140_SANs_07
8/14/01
3:38 PM
Page 275
Developing a SAN Architecture • Chapter 7
Evaluating Performance Considerations Over-subscription is never a bad thing in and of itself. It is only when
over-subscription becomes congestion that problems might arise. Latency is almost never a driving consideration in real-world SAN
performance, since fabric latency is at least one order of magnitude lower than typical disk subsystem latency. Exceptions to this rule include clustering software and some highly performance-sensitive applications. In almost all cases, considerations outside the fabric will limit perfor-
mance, such as CPU speed of hosts or the I/O profile of an application.
Frequently Asked Questions The following Frequently Asked Questions, answered by the authors of this book, are designed to both measure your understanding of the concepts presented in this chapter and to assist you with real-life implementation of these concepts. To have your questions about this chapter answered by the author, browse to www.syngress.com/solutions and click on the “Ask the Author” form.
Q: What is the difference between a fabric and a SAN? I have usually heard these terms used interchangeably.
A: A SAN is a storage area network.This could be comprised of any underlying technology, not just Fibre Channel fabrics.While certain traditional networking technologies are not fundamentally well suited to SAN construction (for example, Gigabit Ethernet), a number of emerging technologies can also be used to build enterprise-class SANs.Thus, a SAN is a fairly general term and does not limit itself to one specific approach. A fabric, on the other hand, is very specific. It is a set of interconnected Fibre Channel switches.The terms are frequently used interchangeably because Fibre Channel fabrics are, at this point, pretty much the only available production-level technology for SANs. Right now, nearly all SANs are Fibre Channel fabrics.
Q: All of this sounds very complicated. I just want to build the thing! Is there a fabric that I can just build?
275
140_SANs_07
276
8/14/01
3:38 PM
Page 276
Chapter 7 • Developing a SAN Architecture
A: Sure.You can build a dual-fabric, resilient core/edge SAN.You can pick one of the designs from the core/edge target design sidebar earlier in this chapter, and you will have a design that will probably do what you need. It might not be the least expensive way to solve your design problem, but it is certainly the approach that requires the least planning.We presented the more advanced design material because many users want to take more control of their SANs; not because you must apply all of it in order to get the SAN you need.
Q: I have heard that you should always try to minimize the number of hops between hosts and their storage. Is this always true?
A: The best performance is always obtained by localizing traffic within a switch. This is a zero-hop scenario. Once you go outside of a single switch and cross an ISL, you can get to your destination in one hop. In most cases, getting the best performance requires an additional hop.Two or more hop equal-cost path networks make better use of FSPF load-sharing capabilities. Since bandwidth usually makes much more of a difference to performance than latency, the added bandwidth from FSPF load sharing more than compensates for the extra hop. From a practical standpoint, when using a core/edge design, this means that you should always put the hosts and storage devices around the edge, rather than locating some devices directly on the core.
140_SANs_08
8/14/01
5:26 PM
Page 277
Chapter 8
SAN Troubleshooting
Solutions in this chapter: ■
The Troubleshooting Approach: The SAN Is a Virtual Cable
■
Troubleshooting the Fabric
■
Troubleshooting Devices that Cannot Be Seen
■
Troubleshooting Marginal Links
■
Troubleshooting I/O Pauses
; Summary ; Solutions Fast Track ; Frequently Asked Questions 277
140_SANs_08
278
8/14/01
5:26 PM
Page 278
Chapter 8 • SAN Troubleshooting
Introduction A SAN is a complex system that can consist of multiple switches, hosts, storage devices, routers, and hubs. A SAN can also be as simple as a single switch with attached storage and hosts. A breakdown of the individual components yields a range of subcomponents, from simple subcomponents, such as cables, to complex subcomponents, such as switches. At a macro level, the fabric itself is considered a component that might require troubleshooting. Switches are logically positioned in the middle of the network between hosts and storage, and have visibility to both storage and hosts.This visibility into both sides of the storage network enables you to use switches to determine the cause of any malfunction in the SAN.This chapter presents a structured process for identifying marginal or faulty SAN components by helping you figure out where to start and then to methodically home in on the problem. Specific areas of focus include troubleshooting the following symptoms and SAN components: ■
Fabric
■
“Missing” devices
■
Marginal links
■
Input/Output (I/O) interruptions
The context of your problem influences how to interpret the data output by the variety of commands available in Fabric OS. For example, focus on the port state information for switchShow output when you are troubleshooting a port issue, and the switch status information from the same command when investigating a fabric issue. We will cover the details of how to troubleshoot using Fabric OS commands such as switchShow, errShow, portStatsShow, and other commands. Understanding host behavior and interpreting host information is also an important part of the troubleshooting process we discuss in this chapter.
The Troubleshooting Approach: The SAN Is a Virtual Cable When first approaching troubleshooting, think of the SAN as a virtual cable. Storage traditionally involved connecting a Small Computer Systems Interface (SCSI) disk via a SCSI cable to a host; with this scenario, you focus on four components: the storage, the Host Bus Adapter (HBA), the host’s OS, and the cable/terminator.Troubleshooting a SAN is more challenging, but still has many
140_SANs_08
8/14/01
5:26 PM
Page 279
SAN Troubleshooting • Chapter 8
things in common with the traditional storage troubleshooting process.To the operating system, the SAN provides a link to a disk, just as a traditional SCSI connection would. You can apply the same “tried-and-true” process of elimination used to troubleshoot a direct-attach SCSI issue or Ethernet network issue to SAN troubleshooting. At a macro level, if you consider the SAN a virtual cable, the issue can reside in three possible areas: the host, the “cable,” or the storage. Troubleshooting can work like a binary search when you start investigating these areas. Start in the middle and determine whether you are “above” or “below” the problem, and then keep dividing the suspect path until you resolve the problem. When troubleshooting with a simple single-switch configuration, a single host, and a single storage device, you need to focus on the HBA, the Gigabit Interface Converter (GBIC), the host’s OS, the cable, the switch, and the storage. Brocade fabrics run a single-image distributed operating system known as Fabric OS. Fabric OS delivers functionality such as Name Server, Registered State Change Notification (RSCN), Zoning, and security.These functions are part of the SAN and are also variables in the troubleshooting equation. A large SAN can consist of dozens of switches and is capable of growing to thousands of ports. Knowing where in the SAN to initiate troubleshooting can be daunting.The next section uses a typical SAN troubleshooting scenario—a host unable to “see” its disks—to illustrate the method of resolving the problem by treating the SAN as a virtual cable and working with a process of elimination.
A Typical Scenario: “I Cannot See My Disks” We provide the scenario described in this section to introduce the troubleshooting process and to establish a framework with which you are familiar. Some terms, commands, and concepts may seem foreign.This is okay.We address everything discussed in this section in greater detail later in the chapter. When a host cannot see its disks, one thing to check is whether that device is logically connected to the switch by reviewing the output from the switchShow command. If the device is not logically connected (that is, it does not show up as an Nx_Port), you need to focus on the port initialization. Notice that port 15 in Figure 8.1 indicates a logically connected device, as this port is connected as an F_Port. Port 14 is an example of an unsuccessful device connection, as the device connected to port 14 is connected as a G_Port. A G_Port indicates an incomplete connection to the fabric. Initially knowing that the missing device is not logically connected eliminates the host and everything on that side of the data
279
140_SANs_08
280
8/14/01
5:26 PM
Page 280
Chapter 8 • SAN Troubleshooting
path from the suspect list, as depicted in Figure 8.2.This includes all aspects of the host’s OS, the HBA driver settings and binaries, the HBA Basic Input Output System (BIOS) settings, the HBA GBIC, the cable going from the switch to the host, the GBIC on the switch side of that cable, and all switch settings related to the host.That is quite a lot for one command! If the missing device is logically connected to the switch, you need to check to see if the device is present in the Simple Name Server (SNS). Figure 8.1 Example of a Successful and Unsuccessful Device Connection core2:admin> switchshow switchName:
core2
switchType:
2.4
switchState:
Online
switchRole:
Subordinate
switchDomain:
5
switchId:
fffc05
switchWwn:
10:00:00:60:69:10:9b:5b
switchBeacon:
OFF
port 0: sw
Online
E-Port
10:00:00:60:69:11:f9:f7 "edge1"
(upstream) port
1: sw
Online
E-Port
10:00:00:60:69:10:9b:52 "edge2"
port
2: sw
Online
E-Port
10:00:00:60:69:11:f9:f7 "edge1"
port
3: sw
Online
E-Port
10:00:00:60:69:10:9b:52 "edge2"
port
4: sw
Online
E-Port
10:00:00:60:69:12:f9:8c "edge3"
port
5: sw
Online
E-Port
10:00:00:60:69:12:f9:8c "edge3"
port
6: —
No_Module
port
7: —
No_Module
port
8: —
No_Module
port
9: —
No_Module
port 10: —
No_Module E-Port
10:00:00:60:69:12:f9:8c "edge3"
port 11: id
Online
port 12: —
No_Module
port 13: —
No_Module
port 14: cu
Online
G-Port //incomplete fabric connection
port 15: id
Online
F-Port
50:06:04:82:bc:01:9a:0c
140_SANs_08
8/14/01
5:26 PM
Page 281
SAN Troubleshooting • Chapter 8
Figure 8.2 The SAN Virtual Cable OK
Host
Storage
Problem Virtual SAN Cable
The SNS is a directory service provided by the fabric. Initiators query the Name Server much in the same way you would query a telephone directory looking for a particular person or service. If a device is not in the Name Server, it is essentially invisible to other devices in the fabric.When a device connects to the fabric, that device will register itself with the Name Server.This is similar to the situation in which you change neighborhoods and have your name listed in the new telephone directory.When an initiator, which is normally an HBA, enters the fabric, it queries the Name Server to identify all accessible devices and obtain the addresses of these devices, just like you might scan your telephone directory for a name. Some targets also will query the Name Server.Then the initiator starts the process of establishing a connection with those devices for which the Name Server provides addresses. Check the Name Server for the presence of your missing device by issuing the nsShow command on the switch to which the device is attached (see the sample output in Figure 8.3).This will list all of the nodes connected to that switch, allowing you to determine if a particular node is accessible on the network. An alternate method is to check the Name Server list in the WEB TOOLS Graphical User Interface (GUI) on any switch in the fabric, as it contains a consolidated list of all devices in the fabric. Note that we started the process in the middle of the virtual SAN cable, which is the fabric.This is the process we described earlier as being like a binary search algorithm.You start in the middle half of the data path, figure out if you are “above” the problem or “below”it and keep dividing the suspect path in half until you identify the problem.
281
140_SANs_08
282
8/14/01
5:26 PM
Page 282
Chapter 8 • SAN Troubleshooting
Figure 8.3 nsShow Sample Output ore2:admin> nsshow The Local Name Server has 9 entries { Type Pid *N
COS
021a00;
PortName
NodeName
TTL(sec)
2,3;20:00:00:e0:69:f0:07:c6;10:00:00:e0:69:f0:07:c6; 895
Fabric Port Name: 20:0a:00:60:69:10:8d:fd NL
051edc;
3;21:00:00:20:37:d9:77:96;20:00:00:20:37:d9:77:96; na
FC4s: FCP [SEAGATE ST318304FC
0005]
Fabric Port Name: 20:0e:00:60:69:10:9b:5b NL
051ee0;
3;21:00:00:20:37:d9:73:0f;20:00:00:20:37:d9:73:0f; na
FC4s: FCP [SEAGATE ST318304FC
0005]
Fabric Port Name: 20:0e:00:60:69:10:9b:5b NL
051ee1;
3;21:00:00:20:37:d9:76:b3;20:00:00:20:37:d9:76:b3; na
FC4s: FCP [SEAGATE ST318304FC
0005]
Fabric Port Name: 20:0e:00:60:69:10:9b:5b NL
051ee2;
3;21:00:00:20:37:d9:77:5a;20:00:00:20:37:d9:77:5a; na
FC4s: FCP [SEAGATE ST318304FC
0005]
Fabric Port Name: 20:0e:00:60:69:10:9b:5b NL
051ee4;
3;21:00:00:20:37:d9:74:d7;20:00:00:20:37:d9:74:d7; na
FC4s: FCP [SEAGATE ST318304FC
0005]
Fabric Port Name: 20:0e:00:60:69:10:9b:5b NL
051ee8;
3;21:00:00:20:37:d9:6f:eb;20:00:00:20:37:d9:6f:eb; na
FC4s: FCP [SEAGATE ST318304FC
0005]
Fabric Port Name: 20:0e:00:60:69:10:9b:5b NL
051eef;
3;21:00:00:20:37:d9:77:45;20:00:00:20:37:d9:77:45; na
FC4s: FCP [SEAGATE ST318304FC
0005]
Fabric Port Name: 20:0e:00:60:69:10:9b:5b N
051f00;
2,3;50:06:04:82:bc:01:9a:0c;50:06:04:82:bc:01:9a:0c; na
FC4s: FCP [EMC
SYMMETRIX
5267]
Fabric Port Name: 20:0f:00:60:69:10:9b:5b }
At this point, if the device is not present in the Name Server, you have narrowed your search along the virtual SAN cable to the Name Server interface
140_SANs_08
8/14/01
5:26 PM
Page 283
SAN Troubleshooting • Chapter 8
between the storage.The missing device process defined in this section is summarized in flowchart form in Figure 8.4. Note that Figure 8.4 is an excerpt from the complete missing-device troubleshooting process, which is shown in Figure 8.25. Remember that we will go deeper into this missing-device troubleshooting process and flowchart later in the chapter. Figure 8.4 Flowchart Excerpt of Troubleshooting a Missing Device (See Figure 8.25 for the Complete Flowchart.) Storage device not visible to host
Is the storage device present in switchShow?
No
Issue between storage device and switch. Not a host issue
No
Issue between storage device and switch. Not a host issue
Yes
Is storage device visible in name server?
Where to Start and What Data to Gather As stated in the previous section, SAN troubleshooting should begin in the center of the SAN and proceed outward. Once you know where to start troubleshooting, the next question is how to proceed. Start the troubleshooting process by gathering a preliminary set of data, and then analyze this data to identify where the problem resides: the host, the fabric, or the storage.Then gather additional data from the appropriate area and home in on the cause of the problem. A plethora of data is available from the switches, hosts, and storage. Knowing what data to look at and when to look at it is fundamental to the SAN troubleshooting process.
283
140_SANs_08
284
8/14/01
5:26 PM
Page 284
Chapter 8 • SAN Troubleshooting
Take a Snapshot: Describe the Problem and Gather Information Start with a general description of the problem and identify as much supporting detail as possible. At the very least, this should include a statement about what the “bad” behavior is, and a statement about what you are doing or have done to expose this behavior. Note that this is not the same as describing what you have done that causes the behavior.You might be doing something correctly, like plugging in a disk array and adding it to a zone, yet it might affect something else in the fabric if there is an underlying problem that is exposed whenever a zone change occurs. For example, an HBA responding incorrectly to an RSCN could fail when the new zone configuration is enabled. An RSCN is a fabric service for which an edge device optionally registers.When a device registers for an RSCN, it is asking the fabric to send that device a notice anytime something in the fabric changes. For example, when a new device is added to the fabric, any devices that registered for RSCNs will receive a notice.The registered device receiving the RSCN then checks the Name Server to see what has changed and takes appropriate action. For example, if the registered device is a host and a new disk drive is added to the SAN, the host might create the necessary device operating system structures so the new device is accessible to the user. This information will help you with the problem resolution, and might be necessary if you need to contact Brocade or any Brocade-authorized support channel. Some examples of a general problem description include: ■
When I enable a switch zoning configuration with cfgEnable, storage devices are no longer accessible to the host.
■
There are frequent pauses in I/O when I copy large files between arrays.
■
My edge device sometimes connects as an N_Port, and other times it connects as a Node Loop (NL)_Port when I power it up.
■
The fabric segments and the following error message is logged (provide error message in your description). It does this under normal operation, even when I do not touch any device on the SAN.
Include the answers to the following questions with your problem description: ■
Can you recreate the problem on demand? If so, how? (Go into detail.)
■
Is the problem intermittent? If so, how frequently does it occur?
140_SANs_08
8/14/01
5:26 PM
Page 285
SAN Troubleshooting • Chapter 8 ■
Has anything at all changed recently on the fabric? If so, what? (Provide a complete list.)
■
Is the problem localized or fabric-wide? For example, is the problem happening with other devices in the fabric, or just locally with a single device attached to the switch?
■
Is this an initial install and the device was never working, or was the device working and now it has stopped working?
Other information to record: ■
If there are any error messages, include them with the problem description.
■
Firmware and driver versions for the affected HBA and storage devices.
■
Firmware and operating system versions for affected hosts and all fabric switches.
■
External switch information, such as LED state.
■
External HBA and port information, such as LED state.
■
A diagram of the SAN configuration.
■
If long-distance links are present, include information about the length and quality of the lines, and the mechanism being used to achieve the distance (for example, “The line is 10 km long, and we are using Long Wavelength [LWL] GBICs,” or “It is 80 km long, and we are using a Dense Wave Division Multiplexor [DWDM] and the Extended Fabrics product”).
Finally, gather supportShow information from the switches.The supportShow command is a switch command used to gather information about the switch and the fabric; it can provide valuable clues about what is happening in your switch network. It is like a macro in that it executes a long list of switch commands, which Brocade identifies as important for the troubleshooting process. Note that the commands that supportShow executes vary between Fabric OS releases.The v2.4.1 supportShow command executes the following switch commands: ■
version
■
uptime
■
tempShow
285
140_SANs_08
286
8/14/01
5:26 PM
Page 286
Chapter 8 • SAN Troubleshooting ■
psShow
■
licenseShow
■
diagShow
■
errDump
■
switchShow
■
portFlagsShow
■
portErrShow
■
mqShow
■
portSemShow
■
portShow
■
portRegShow
■
portRouteShow
■
fabricShow
■
topologyShow
■
qlShow
■
faShow
■
portCfgLport
■
nsShow
■
nsAllShow
■
cfgShow
■
configShow
■
faultShow
■
traceShow
■
portLogDump
One benefit of supportShow is that you do not have to repeatedly retrieve various types of data, since most of the data you need is available from supportShow in one place. As this command rapidly streams in a telnet window, capture mode should be turned on prior to executing the command so that it can be captured to a text file for later review.
140_SANs_08
8/14/01
5:26 PM
Page 287
SAN Troubleshooting • Chapter 8
NOTE It is important to execute the supportShow command at the time the problem is occurring, rather than waiting until the fabric is functioning normally.
Due to the large volume of data created by supportShow, you might choose to gather the supportShow data once and then selectively issue a subset of its commands as part of your troubleshooting process.
Troubleshooting Tools Many tools are available to the SAN troubleshooter. Many of these tools are switch commands. Other tools involve viewing the switch LEDs, host information such as Solaris’ /var/adm/messages file, Fibre Channel analyzers, and diagnostics available on many storage arrays. Rarely is it possible to use a single tool to successfully troubleshoot a problem. It is more common to use several tools to attain a successful resolution of a problem.
Using the Switch LEDs A significant amount of information can be gathered just by looking at the switch LEDs. At a rudimentary level, it is possible to identify that a device has faulted or is not yet online by looking for a “fast yellow.” If the switch is located in another room, you can get a visual real-time LED status using the WEB TOOLS interface. Fast flickering green lights are a sign of a healthy SAN. By physically observing the switches that comprise a SAN, it is possible to detect patterns and identify a marginal or faulty component. For example, if you have a situation in which you are trying to identify a device that is repeatedly toggling online and offline, you can use the switch LEDs. While observing a functional fabric, you can easily identify a potentially disruptive device by scanning for a port that goes offline (no LED light), sends light (steady yellow), comes online (steady green), and then cycles through the same steps—blank, yellow, green.You also want to look for correlations or patterns, such as one device going offline followed by a group of devices going offline and back online again.This situation is common in QuickLoop configurations when the first device going offline is sending a Loop Initialization Primative (LIP), which then causes the other devices to LIP.
287
140_SANs_08
288
8/14/01
5:26 PM
Page 288
Chapter 8 • SAN Troubleshooting
How to Identify a Healthy SAN Using the LEDs A settled and healthy fabric should have solid green or fast flickering green lights. A solid green light indicates an active link, while a fast flickering green light indicates I/O activity.
How to Identify a SAN Problem Using the LEDs A yellow light or blinking yellow light indicates a problem with your SAN. An LED that transitions from yellow to green, however, is not a problem. A powered-off edge device, or edge device that is not yet online, might cause the switch LEDs to blink yellow.
Another helpful use for the LEDs is for fabric “bring up.”When bringing up a fabric, one sign to look for that indicates a fabric has reached convergence are steady green lights.When the fabric is coming up, the Inter-Switch Links (ISLs) go through initialization, which appear to the observer as flickering green and yellow lights prior to the fabric fully converging. Once the fabric is converged, the lights go to a steady green.Then, as I/O in the fabric begins, you will see flickering green lights on the ISL ports and the edge device ports. A slowly flashing switch power LED indicates that the switch failed the Power-On Self-Test (POST) and is not able to come online. Refer to the associated switch manual for the location of the power LED.Table 8.1 lists the port LEDs and their definitions (you can also find this table in the Brocade SilkWorm 2800 Hardware Reference Manual). Table 8.1 Front Panel LED Port Indicators Ports
LED Definition
No light showing
No light or signal carrier (no module, no cable) for media interface Receiving light or signal carrier, but not yet online
Steady yellow
Continued
140_SANs_08
8/14/01
5:26 PM
Page 289
SAN Troubleshooting • Chapter 8
Table 8.1 Front Panel LED Port Indicators Ports
LED Definition
Slow yellow (flashes two seconds) Fast yellow (flashes a half second) Steady green Slow green (flashes two seconds) Fast green (flashes a half second) Flickering green
Disabled (result of diagnostics, switchDisable, or portDisable command) Error, fault with port
Online (connected with external device over cable) Online, but segmented (loopback cable or incompatible fabric parameters) Internal loopback (diagnostic)
Online and frames flowing through port
Switch Diagnostics A robust set of switch diagnostics is available so you can validate the operational level of a SilkWorm switch. Several of these diagnostics, such as portLoopbackTest, are also helpful in the troubleshooting process. For example, if you suspect a bad GBIC or switch port, you can use portLoopbackTest to confirm your suspicion. Using portLoopbackTest for troubleshooting is discussed in the section “Troubleshooting Marginal Links” later in the chapter.The supportShow diagnostic command in particular, discussed in detail later in this chapter, is very helpful to the troubleshooting process.The Brocade Fabric OS manuals provide detailed description regarding the usage of diagnostic commands.To see what diagnostic commands are available online, issue the command diagHelp at the switch prompt.The following list of diagnostic commands is available in the V2.4.1 Fabric OS: ■
ramTest System DRAM diagnostic
■
portRegTest Port register diagnostic
■
centralMemoryTest Central memory diagnostic
■
cmiTest CMI bus connection diagnostic
■
camTest QuickLoop CAM diagnostic
289
140_SANs_08
290
8/14/01
5:26 PM
Page 290
Chapter 8 • SAN Troubleshooting ■
portLoopbackTest Port internal loopback diagnostic
■
sramRetentionTest SRAM Data Retention diagnostic
■
cmemRetentionTest Central Mem Data Retention diagnostic
■
crossPortTest Cross-connected port diagnostic
■
spinSilk Cross-connected line-speed exerciser
■
diagClearError Clear diag error on specified port
■
diagDisablePost Disable Power-On-Self-Test
■
diagEnablePost Enable Power-On-Self-Test
■
setGbicMode Enable tests only on ports with GBICs
■
setSplbMode Enable 0=Dual, 1=Single port LB mode
■
supportShow Print version, error, portLog, etc.
■
diagShow Print diagnostic status information
■
parityCheck Dram Parity 0=Disabled, 1=Enable
■
spinFab ISL link diagnostic
■
loopPortTest L_Port cable loopback diagnostic
Helpful Commands With dozens of switch commands at your disposal, it can be difficult to determine which command to use in a given situation. An annotated list of helpful commands follows in this section, with additional commands highlighted as they relate to specific issues discussed in following sections.This list of commands is a starting point for gathering data and initiating your troubleshooting process. While the information generated by these commands is also available in supportShow, you will want to use individual commands as you advance through the troubleshooting process. SupportShow creates a significant amount of data and is helpful when you want to perform the original snapshot of the configuration and environment (to report a problem to your switch supplier), or you are not sure what data to capture.
140_SANs_08
8/14/01
5:26 PM
Page 291
SAN Troubleshooting • Chapter 8
NOTE Although the switch commands are shown with various capitalization as originally coded in Fabric OS, the commands are no longer case-sensitive and can be entered with all lowercase if desired.
Entering the command help at the switch prompt generates a list of commands available to the user as shown in Figure 8.5. Entering the command help
generates a help page (similar to UNIX man pages) for that specific command. Many commands differ by the extension show or dump (for example, errShow and errDump). The difference is that show commands require you to type a return between entries, while the dump commands stream data to the screen without any pauses. Dump commands are used when you have a facility for logging command output to a file. It might be necessary to execute commands on more than one switch in the fabric, especially if the location of the problem is unclear.
NOTE As of Fabric OS 2.4.1, there is no time synchronization among the switches, which can make troubleshooting a challenge if the clocks between the switches are skewed. Before you begin troubleshooting your fabric, you should make a note of any time skew so that you can compensate for it when reading command outputs. You should also make an effort to keep switch clocks set correctly during normal operation to avoid this problem.
Figure 8.5 Use the help Command to See What Commands Are Available or Type the help Command for Help About a Specific Command dev172:admin> help
agtcfgSet
Set SNMP agent configuration
agtcfgShow
Print SNMP agent configuration
agtcfgDefault
Reset SNMP agent to factory default . Continued
291
140_SANs_08
292
8/14/01
5:26 PM
Page 292
Chapter 8 • SAN Troubleshooting
Figure 8.5 Continued . . qlHelp
Print quick loop help info
routeHelp
Print routing help info
trackChangesHelp
Print Track Changes help info
zoneHelp
Print zoning help info
dev172:admin> help errShow
NAME errShow - display the error log
SYNOPSIS errShow
AVAILABILITY all users
DESCRIPTION This command displays the error log, prompting the user to type return between each log entry. It is identical to errDump, except
. . . SEE ALSO errDump, uptime
The errShow Command The errShow command provides a listing of up to 64 logged errors and is helpful for identifying where a problem might reside. It sends messages to the console and to the error log. Note that the error log is cleared after a reboot or power cycle; if you want to maintain error logs that persist after reboots or power
140_SANs_08
8/14/01
5:26 PM
Page 293
SAN Troubleshooting • Chapter 8
cycles, consider using the syslog facilities of the switch to log errors to persistent storage. See syslogdIpAdd, syslogdIpRemove, and syslogdIpShow for further detail on how to set up persistent logging. When examining errShow data, which can be quite wordy, look for trends or patterns. For example, look for an excessive number of errors associated with a specific port. In addition, watch for high error-count values, which indicate a repeated error that has been logged many times. Logging error counts limits errors that occur multiple times from consuming the space provided for the error log. It is important to note that with every error, a severity level is associated. A warning (error level 3) is just that—a warning. An error (error level 2) or critical (error level 1) message is more severe and requires further attention. An excerpt from the errShow help entry is provided in Figure 8.6. Please refer to the help page or the Fabric OS manual for details on interpreting Diag Err#, as the list of codes is lengthy. A Diag Err# usually indicates a problem with hardware, so contact your switch supplier for further assistance. In addition to software errors, errShow logs environmental issues, such as over-temperature conditions, and equipment issues such as fan failures or power supply failures. A detailed list of error messages, descriptions, probable causes, and actions is maintained in the Fabric OS Reference Manual Version 2.4 (Publication Number 53-0001569-01). Figure 8.6 Excerpt from the errShow help Entry Each entry in the log has the same format:
Error Number —————— taskId (taskName): Time Stamp (count) Error Type, Error Level, Error Message Diag Err#
Error Number
Starting from one. If there are more error than the size of the log, only the most recent errors are shown.
Task Id & Name
The ID and name of the task recording the error.
Continued
293
140_SANs_08
294
8/14/01
5:26 PM
Page 294
Chapter 8 • SAN Troubleshooting
Figure 8.6 Continued Time Stamp
The date and time of the first occurrence of the error.
Error Count
For errors that occur multiple times, the repeat count is shown in parenthesis. The maximum count is 999.
Error Type
An uppercase string showing the firmware module and error type. The switch manual contains a detailed explanation of each error type.
Error Level
Error Message
0
panic (the switch reboots)
1
critical
2
error
3
warning
4
information
5
debug
Additional information about the error.
Figure 8.7 is an example of an errShow message. The fabric is segmented, meaning that the switch that generated this message is logically disconnected from the SAN, and any devices in the SAN that are not directly connected to this switch are inaccessible to this switch. Moreover, any devices located on this switch are unable to access other devices in the fabric. The error level is a warning (3). The task ID (0x10e2b7f0) can be cross-referenced by issuing the telnet command “i” to obtain additional information on the task in question. The Task Name is self-explanatory, and interpreting it is somewhat intuitive. For example, tTransmit is the transmit task. The Task Name can be helpful in identifying the nature of the problem. Finally, the error message indicates that there is a discrepancy between the zone information contained on this switch and the zone information contained in the rest of the fabric. When the switch tried to join the fabric with this conflicting information, the join request was denied; hence, the segmented fabric. The message even identifies the zone that is causing the
140_SANs_08
8/14/01
5:26 PM
Page 295
SAN Troubleshooting • Chapter 8
conflict; in this case, it is the “red” zone. This zone should be checked and compared to the rest of the fabric, and if the zone information is different, either correct or delete it. Figure 8.7 errShow Example Task Id
Task Name
Error 04 -------0x10f6f4f0 (tTransmit): May 15 09:38:57 (6) Error FABRIC -SEGMENTED, 3, port 0, zone conflict: content mismatch: red
Error Type
Warning
Port
Error Message
Level
The portErrShow Command The portErrShow command is an effective command for troubleshooting marginal ports. This command provides an error summary for all ports associated with the switch and provides a status of all ports from a link integrity perspective. The key to interpreting the statistics is looking for a very high number of errors relative to the frames transmitted and frames received. For example if 2,000,000 frames have been received and only three Cyclic Redundancy Check (CRC) errors have been logged, the CRC errors relative to the frames received is a very low ratio and the associated port is not suspected as being marginal. On the other hand, if 2,000,000 frames have been received and 10,000 CRC errors have been logged, the CRC errors relative to the frames received is a high ratio and the associated port should be examined further. A rough guideline is to look for errors in excess of 0.5 percent of the total number of frames transferred. Another important trend to watch is a steadily increasing number of errors. You can track increasing errors by sampling every five or ten seconds and monitoring the delta between the samples. Simple Network Management Protocol (SNMP) polling can be used to facilitate this. Also, the optionally licensed Fabric
295
140_SANs_08
296
8/14/01
5:26 PM
Page 296
Chapter 8 • SAN Troubleshooting
Watch product can be used to note changes in error rates over time and send out an SNMP trap or error log entry. Streaming errors is a high-order indicator and requires close monitoring—even if the error rate is less than one percent.While the error count relative to frames transmitted or received might be low, a steadily increasing number of errors indicates a marginal port. The portErrShow statistics shown in Figure 8.8 were gathered from a switch that had a marginal NL_Port (HBA), connected to port 6. It turned out that the Gigabit Link Module (or GLM, a connector similar to a GBIC) was failing and causing a degraded signal. Note how high the enc_in and CRC errors are! Figure 8.8 portErrShow Example frames tx
enc
rx
in
crc err
too
too
bad
shrt long
enc disc link loss loss frjt fbsy
eof
out
c3 fail sync
sig
—————————————————————————————————— port 0: 2.9g 1.7g
0
12
0
0
0
0
0
0
2
1
0
0
port 1: 305m 3.0g
0
0
0
0
0
0
0
0
1
0
0
0
port 2: 1.2g 892m
0
0
0
0
0
0
0
0
556
27
0
0
port 3: 1.1m
25m
0
0
0
0
0
82
0
4
9
4
0
0
port 4:
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
1.4k
1.4k
0
0
port 6: 668m 4.0g 6.0m 66m
0
0
236
51m
0
87
54
11
0
0
0
port 5: 9.5m 4.0g
The error statistics shown in boldface are the primary statistics on which to focus.The following listing explains relevant statistics and associated definitions: ■
enc_in Received data: the number of 8b/10b encoding errors that have occurred inside frame boundaries.This counter is generally a zero value, although occasional errors might occur on a normal link and give a nonzero result. (Minimum compliance with the link-bit error rate specification on a link continuously receiving frames would cause approximately one error every 20 minutes.) Reinitialization or reboots of the associated Nx_Port can also cause these errors, resulting in a low-count error count.
■
crc_err Received frames: the number of CRC errors detected. A CRC error indicates that the contents of a frame are no longer valid.
140_SANs_08
8/14/01
5:26 PM
Page 297
SAN Troubleshooting • Chapter 8
Reinitialization or reboots of the associated Nx_Port can also cause these errors, resulting in a low count. ■
too_long Received frames: the number of frames that were longer than the maximum Fibre Channel frame size (such as a header with more than a 2112-byte payload).
■
bad_eof The number of frames received with a badly formed end-of-frame.
■
enc_out Receive link: the number of 8b/10b encoding errors recorded outside frame boundaries.This number might become nonzero during link initialization, but it indicates a problem if it increments faster than the allowed link-bit error rate (approximately once every 20 minutes).
■
er_disc_c3 Receive link: the number of Class 3 frames discarded. Class 3 frames can be discarded due to timeouts or invalid or unreachable destinations.This quantity could increment at times during normal operation, but might be used for diagnosing problems in some situations.
NOTE Steadily increasing errors between samples is a very strong sign that the associated port is not functioning properly.
Marginal link troubleshooting and related troubleshooting commands are discussed in more detail in the “Troubleshooting Marginal Links” section later in this chapter.
The switchShow Command The switchShow command is another powerful command that has many uses for the troubleshooting process. An excerpt from the switchShow help entry is provided here. It is helpful for troubleshooting fabric as well as edge device connectivity issues.This command is likely to be one of the first commands you will execute as part of your troubleshooting process.The nature of the problem will dictate what switchShow data to focus on and how to interpret this data. As shown in Table 8.2 and Figure 8.9, switchShow data is loosely organized into three categories.
297
140_SANs_08
298
8/14/01
5:26 PM
Page 298
Chapter 8 • SAN Troubleshooting
Table 8.2 How switchShow Data Relates to the SAN Functional Areas Fabric-Related
Edge Device-Related
Miscellaneous
switchState switchRole switchDomain port state
port state
switchId switchBeacon switchType switchName
Figure 8.9 switchShow Definitions This switchShow command displays switch and port status information. Some information varies with the switch model, e.g. number of ports, and Domain ID values. The lines of the display show:
switchName
The switch's symbolic name.
switchType
The switch's model and revision numbers.
switchState
The switch's state: Online, Offline, Testing, Faulty.
switchRole
The switch's role: Principal, Subordinate, Disabled.
switchDomain
The switch's Domain ID: 0-31 or 1-239.
switchId
The switch's embedded port D_ID.
switchWwn
The switch's Worldwide Name.
switchBeacon
The switch's beaconing state (either ON or OFF).
The switch summary is followed by one line per port:
port number
The port number: 0-7 or 0-15.
module type
The port module type (GBIC or other): — - no module present sw - shortwave laser lw - longwave laser cu - copper id - serial ID
port state
The port's state: Continued
140_SANs_08
8/14/01
5:26 PM
Page 299
SAN Troubleshooting • Chapter 8
Figure 8.9 Continued No_Card
- no interface card present
No_Module - no module (GBIC or other) present No_Light
- the module is not receiving light
No_Sync
- receiving light but out of sync
In_Sync
- receiving light and in sync
Laser_Flt - module is signaling a laser fault
comment
Port_Flt
- port marked faulty
Diag_Flt
- port failed diagnostics
Lock_Ref
- locking to the reference signal
Testing
- running diagnostics
Online
- the port is up and running
The comment field may be blank, or may show: Disabled
- the port is disabled
Bypassed
- the port is bypassed (loop only)
Loopback
- the port is in loopback mode
E_Port
- fabric port, shows WWN of attached switch
F_Port
- pt-pt port, shows WWN of attached N_Port
G_Port
- pt-pt but not yet E_Port or F_Port
L_Port
- loop port, shows number of NL_Ports
if a port is configured as a long-distance port, the long distance level is shown in the format of "Lx", x being the long-distance level number. See portCfgLongDistance for the level description.
When troubleshooting issues involve the fabric services or a switch’s ability to participate in the fabric, the important parts of switchShow data to focus on are switchState, switchRole, and switchDomain. Port state is applicable from a fabric perspective for observing the state of expansion ports (E_Ports). E_Ports associated with ISLs are the ports used to
299
140_SANs_08
300
8/14/01
5:26 PM
Page 300
Chapter 8 • SAN Troubleshooting
connect multiple switches together forming a fabric. Port state is also useful for troubleshooting connectivity problems with end devices (F_Ports and FL_Ports). In a running fabric, the switchState should always be online. If not, access to and from the switch is not possible. It is possible that the switch may be in a transitory state as it comes online from a power cycle or reboot, so check again to make sure this is not the case. It is also possible that the switch has been manually disabled using the switchDisable command. A switch can be operating as a principal, subordinate, or disabled, which is indicated by the switchRole variable.There is only one principal switch in the fabric, and if the principal fails, another switch will assume this role.The principal switch facilitates the bring up of the fabric and assignment of domain IDs. A switch domain ID is an address that defines the switch in a fabric. Domain IDs are automatically assigned as part of the fabric initialization process by the principal switch. It is possible to manually assign a domain ID as well. SilkWorm 1000 series switches use the domains 0–31, and SilkWorm 2000 series switches and beyond use the domains 1–239. If a switch is not a principal, it operates in a subordinate switch role. If the switch role indicates disabled, access to and from the switch is not possible and it is likely that someone disabled the switch by typing switchDisable, or the switch was unable to obtain a domain ID.When a switch is disabled, a comment of “unconfirmed” accompanies the domain ID (Figure 8.10). Normally, a switch will be in disabled state after issuing the command switchDisable.The “unconfirmed” attribute could also be caused by a problem with the fabric, which causes a switch to be unable to confirm its domain ID even though the switch is enabled.When the switch is disabled, the LEDs will blink yellow every two seconds and the port state will indicate disabled. Figure 8.10 Switch Disabled and Unconfirmed Domain core1:admin> switchshow switchName:
core1
switchType:
2.4
switchState:
Offline
switchRole:
Disabled
switchDomain:
1 (unconfirmed)
switchId:
fffc01
switchWwn:
10:00:00:60:69:10:8d:fd
switchBeacon:
OFF Continued
140_SANs_08
8/14/01
5:26 PM
Page 301
SAN Troubleshooting • Chapter 8
Figure 8.10 Continued port
0: sw
Laser_Flt
Disabled
port
1: sw
In_Sync
Disabled
port
2: sw
In_Sync
Disabled
port
3: sw
In_Sync
Disabled
port
4: sw
In_Sync
Disabled
port
5: sw
In_Sync
Disabled
port
6: —
No_Module
Disabled
port
7: —
No_Module
Disabled
port
8: —
No_Module
Disabled
port
9: —
No_Module
Disabled
port 10: —
No_Module
Disabled
port 11: —
No_Module
Disabled
port 12: —
No_Module
Disabled
port 13: —
No_Module
Disabled
port 14: —
No_Module
Disabled
port 15: —
No_Module
Disabled
The SilkWorm 1000 series of switches uses the domain IDs 0–31, and the SilkWorm 2000 series and beyond switches use the domain IDs 1–239. Normally, a domain ID is automatically assigned when a switch joins the fabric; however, there are circumstances that can result in domain ID conflicts.This can happen when connecting two online switches that have already been assigned the same domain ID.When two switches in a fabric have the same domain ID, the fabric segments along an ISL that allows domain IDs to be unique in each segment. The port state information generated by switchShow is pertinent to fabricrelated issues if an ISL port is affected. One issue that relates to ISLs involves the port’s inability to fully initialize.While the port is online, it remains in a generic port (G_Port) state since it could not initialize as an E_Port. Another issue that affects ISLs occurs when the link is unable to initialize, resulting in the port not coming online at all.This could be caused by a marginal link, an offline switch connected to the other end of the ISL, or a fabric initialization issue. In either circumstance, it is incumbent upon the SAN administrator to establish that the port is an ISL port or an edge device that is not connected, as there is no way to tell the type of device connected until after the port initializes. Execute the commands
301
140_SANs_08
302
8/14/01
5:26 PM
Page 302
Chapter 8 • SAN Troubleshooting
portDisable and portEnable, providing the offending port number as an argument to try to reinitialize the port. The Switch Name is assigned by the user and does not have to be unique in the fabric. However, uniquely naming each switch can make your SAN administration easier.With some Fabric OS versions,WEB TOOLS might not function properly if the Switch Name does not match the switch’s actual host name.You assign a Switch Name with the switchName command. The switchId value is the switch’s 24-bit Destination ID (D_ID) address in the fabric.This is the Fibre Channel address that another switch would use to send the frame to the switch itself, rather than to a device connected to the switch.This value might appear in portLog data—for example, when the switch probes an edge device for Name Server information. Using the switchBeacon switch command, you can have the switch flash a back-and-forth pattern (from left to right, and right to left) in yellow to identify the switch.This is helpful if you are doing maintenance and need to identify a switch that is positioned in a rack with many other switches. Finally, the switchType information indicates the switch model and revision in the form model.revision, as shown in Table 8.3. Table 8.3 switchType Values and Associated Architecture SwitchType Value
Switch Model
1 2 3 4 5
1000 series 2800 2400 20x0 22x0
Information in the port state section includes the port state, the type of media, the World-Wide Name (WWN) of the connected device, and the switch name if the attached device is a switch, private, phantom, and upstream or downstream information. The port state will typically be online or offline; however, as shown in Figure 8.4, a laser fault is also indicated when encountered.The type of interface media is shown as well, indicating the type of GBIC used. SW is for shortwave GBICs, LW is for long wavelength GBICs (for longer distances), and ID is for serial ID GBICs. Serial ID GBICs are smart GBICs with serial number and state information.
140_SANs_08
8/14/01
5:26 PM
Page 303
SAN Troubleshooting • Chapter 8
A private device is normally a loop device that does not perform a Fabric Login (FLOGI) and uses an 8-bit address. A phantom address is a 24-bit translated address for an 8-bit device. A phantom is created for the embedded port so that services and other devices within the SAN can communicate with the devices on a private loop.The switch recognizes only device addresses of 24 bits in length. Therefore, services on the switch that need to communicate with the private devices need to have a 24-bit proxy for their 8-bit addresses. Each device that wants to communicate with devices on a private loop needs to be “represented” on the loop directly.This is done by creating a phantom device for each host that wants to communicate with devices on the private loop.This phantom is acting on behalf of each of the devices that wish to communicate to devices on the loop. The terms upstream and downstream designate that particular switch’s position in reference to the principal switch in the fabric.These paths are used in the process for assigning switch domain IDs. In Figure 8.11, notice that switch core1 is the principal switch, and all “stream” designators are downstream. For switch edge1, the path to the principal switch is upstream through port 2.There is also a downstream path from switch edge1.This path is used by switch core2 to access switch core1; hence, port 3 is designated as a downstream port.The principal switch has no upstream ports. The port state section of the switchShow output is very helpful in identifying edge-device connection issues. These issues can involve a range of problems, from missing devices to devices initializing with the wrong topology (for example, a loop-configured device initializing as point-to-point topology). The explanation of port states and associated comments is fairly straightforward. When in doubt, check to see that the port is online, assuming a device is attached, and that the topology is correct (F_Port or L_Port). If neither of these values is present, you will need to do further analysis.
The nsShow Command An excerpt from the nsShow help entry is provided in Figure 8.12. The most important thing about nsShow output is whether the device in which you are interested appears in the command output. If a device does not appear in the Name Server, other devices will not be able to access it. There are some instances where initiators bypass the Name Server and directly communicate with a device by using an earlier obtained address or doing a table scan of addresses. This behavior is considered suspect, as it is bypasses a standard methodology. Note that hard zoning prevents such activities from occurring, ensuring that all devices behave appropriately within the SAN.
303
140_SANs_08
304
8/14/01
5:26 PM
Page 304
Chapter 8 • SAN Troubleshooting
Figure 8.11 Upstream and Downstream Paths in Reference to switchShow Output switchName: core1 switchType: 2.4 switchState: Online switchRole: Principal switchDomain: 1 switchId: fffc01 switchWwn: 10:00:00:60:69:10:8d:fd switchBeacon: OFF port 0: sw Laser_Flt port 1: sw Online E-Port 10:00:00:60:69:10:9b:52 port 2: sw Online E-Port 10:00:00:60:69:11:f9:f7 port 3: sw Online E-Port 10:00:00:60:69:10:9b:52 port 4: sw Online E-Port 10:00:00:60:69:12:f9:8c port 5: sw Online E-Port 10:00:00:60:69:12:f9:8c
"edge2" (downstream) "edge1" (downstream) "edge2" "edge3" (downstream) "edge3"
switchName: edge1 switchType: 2.4 switchState: Online switchRole: Subordinate switchDomain: 2 switchId: fffc02 switchWwn: 10:00:00:60:69:11:f9:f7 switchBeacon: OFF port 0: sw No_Light port 1: sw Online E-Port 10:00:00:60:69:10:9b:5b "core2" port 2: sw Online E-Port 10:00:00:60:69:10:8d:fd "core1" (upstream) port 3: sw Online E-Port 10:00:00:60:69:10:9b:5b "core2" (downstream) port 4: — No_Module
Principal core1
edge1
core2
edge2
edge3
NOTE If the device is not in the Name Server, it is most likely invisible to the rest of the fabric and therefore inaccessible.
140_SANs_08
8/14/01
5:26 PM
Page 305
SAN Troubleshooting • Chapter 8
Figure 8.12 nsShow help Page NAME nsShow - display local Name Server information
SYNOPSIS nsShow
AVAILABILITY all users
DESCRIPTION This command displays local Name Server information, which includes information about devices connected to this switch, and cached information about devices connected to other switches in the fabric.
The message "There is no entry in the Local Name Server" is displayed if there is no information in this switch, but there still may be devices connected to other switches in the fabric. The command nsAllShow shows information from all switches.
Each line of output shows: *
an asterisk indicates a cached entry from another switch.
Type
U for unknown, N for N_Port, NL for NL_Port.
Pid
The 24-bit Fibre Channel address.
COS
A list of classes of service supported by the device.
PortName
The device's port Worldwide Name.
NodeName
The device's node Worldwide Name.
TTL
The time-to-live (in seconds) for cached entries, or 'na' (not-applicable) if the entry is local.
There may be additional lines if the device has registered any of the following information (the switch automatically registers SCSI inquiry data for FCP target devices): FC4s supported, Continued
305
140_SANs_08
306
8/14/01
5:26 PM
Page 306
Chapter 8 • SAN Troubleshooting
Figure 8.12 Continued (node) IP address, IPA, port and node symbolic names, fabric port name, hard address and/or port IP address.
Often, the returned SCSI inquiry data is meaningful and indicates telling information such as the vendor, model, and the firmware revision level of the attached device, as shown in Figure 8.13. For HBAs, SCSI inquiry data occasionally is not returned and the Name Server entry is a bit sparse, so it is harder to identify the device. Some vendors are starting to allow administrators to manually populate this field to allow the textual information to be site-specific, such as node names or locations. Figure 8.13 The nsShow Output Explained The Seagate disks support Class 3 service and the EMC supports Classes 2 & 3
Type Pid
An HBA—not much info
COS
PortName
NodeName
TTL(sec) N
0a1000;
2,3;20:00:00:e0:69:40:13:19;10:00:00:e0:69:40:13:19;
na
NL
0a19cb;
3;21:00:00:20:37:26:b0:6c;20:00:00:20:37:26:b0:6c;
na
FC4s: FCP [SEAGATE ST39102FCSUN9.0G0D29] NL
0a19cc;
3;21:00:00:20:37:26:84:22;20:00:00:20:37:26:84:22;
na
FC4s: FCP [SEAGATE ST39102FCSUN9.0G0D] N
0a1b21;
2,3;50:06:04:84:35:46:b5:4d;50:06:04:84:35:46:b5:4d;
FC4s: FCP [EMC N
0a1c21;
SYMMETRIX
2,3;50:06:04:84:3a:3b:1f:4d;50:06:04:84:3a:3b:1f:4d;
FC4s: FCP [EMC
SYMMETRIX
na
5265] na
5265]
An EMC storage device with 5265 firmware
FCP = SCSI over Fibre Channel
It can be confusing understanding the difference between a device node WWN and a port WWN. A device has only one node WWN and can potentially have one or more port WWN(s).This way, it is possible to uniquely identify
140_SANs_08
8/14/01
5:26 PM
Page 307
SAN Troubleshooting • Chapter 8
multiple paths or interfaces to the same device. For example, today’s Just a Bunch of Disks (JBOD) systems usually have two ports (A and B), and each port has an associated port WWN.This enables two paths to connect to the same disk. How do you know it is the same disk? The node WWN is the same for each path, with each path having a unique port WWN. In Figure 8.14, if the entry for Port ID (PID) 0a19cb is connected on both ports A and B, the node WWN stays the same (20:00:00:20:37:26:b0: 6c), the A port would have a WWN of 21:00:00:20:37:26:b0: 6c, and the B port would have a WWN of 22:00:00:20:37:26:b0: 6c. Figure 8.14 The Difference between Port WWN and Node WWN A Port WWN = 21:00:00:20:37:26:b0:6c
B Node WWN = 20:00:00:20:37:26:b0:6c
Port WWN = 22:00:00:20:37:26:b0:6c
The use of node WWNs and port WWNs is not always strictly followed, and the Fibre Channel specifications are not clear on their usage. A node WWN sometimes is used to represent an entire system and all ports (Port WWNs) associated with that system. The Name Server also provides information about a device’s PID. Knowing how to decode a PID is helpful in translating a device’s SAN logical address into a SAN physical location. If you know a device’s PID, you know the physical port that device is attached to, the domain ID of the switch that device is attached to, and whether that device is an N_Port or an NL_Port. Figure 8.15 explains this decoding process further.
The topologyShow Command The topologyShow command displays the fabric topology, as seen by the local switch. topologyShow output consists of a list of all domains that are part of the fabric, and for each of those domains, all the possible paths to reach these domains from the local switch. In addition, topologyShow displays the total number of switches in the fabric, and the domain ID of the local switch. It is also helpful to issue the switchShow command to identify directly connected switches. Look for E_Ports and the name of the switch located at the other end of the E_Port to create a SAN topology. Perform a switchShow for every switch
307
140_SANs_08
308
8/14/01
5:26 PM
Page 308
Chapter 8 • SAN Troubleshooting
in the fabric. First, write down the name of the switch on which the command is issued. For each E_Port on that switch, write down the name of the switch to which the E_Port connects.Then draw a line between the switch on which the command is being run and the switch that shows up on the other end of the E_Port.The data in Figure 8.16 indicates that switch edge3 is directly connected to switches core1 and core2.To identify direct-connect switches in the topologyShow output, look for domain entries with a hop count of one.To obtain additional information on the switches in the fabric, such as their IP address, use the fabricShow command. Figure 8.15 How to Interpret the Port Addressing Port Addressing 0x XX 1Y ZZ where: ■
XX is a value between 0x1 to 0xef inclusive and indicates the domain id of the switch to which the device is attached
■
The “1” will always be there in 2000 series switches
■
Y is the port number (0-F hex) that the device is attached to ZZ is the AL_PA for a loop device or 00 for an F_Port
■
An example: 021500 XX=02 Domain_ID of the switch Y=5 Port # ZZ=00 If 00, then F_Port. IF non-zero, then ALPA of the device on the FL_Port
SAN Profile It is recommended that you create a profile of your fabric when it is functioning normally so that you always have a baseline to compare the current state of your SAN.You will want to create a profile before making any changes to the SAN, such as firmware upgrades or additions or deletions of switches or edge devices. This information can be captured from a logging facility within telnet and stored as a text file.
140_SANs_08
8/14/01
5:26 PM
Page 309
SAN Troubleshooting • Chapter 8
Figure 8.16 Use topologyShow to Determine the Number of Online Switches in the SAN 5 domains in the fabric; Local Domain ID: 3 Domain Metric Hops Out Port In Ports Flags Name -------------------------------------------------------------------1 1000 1 0 0x00000002 D "core1" 2 0x00000008 D 2 2000 2 0 0x00000000 D "edge1" 2 0x00000000 D 3 0x00000000 D 1 0x00000000 D 4 2000 2 0 0x00000000 D "edge3" 2 0x00000000 D 3 0x00000000 D 1 0x00000000 D 5 1000 1 3 0x00000001 D "core2" 1 0x00000004 D
core1
edge1
core2
edge2
edge3
When you finish your maintenance or suspect a problem, take a new profile and compare the baseline profile to your current profile. Any discrepancies require further investigation. For troubleshooting purposes, a profile should consist of the following information extracted from a healthy SAN: ■
The number of domains in the fabric, which can be obtained from topologyShow outputs.
■
The overall topology of the fabric, again from topologyShow and switchShow outputs.
309
140_SANs_08
310
8/14/01
5:26 PM
Page 310
Chapter 8 • SAN Troubleshooting ■
The number of noncached Name Server entries for each switch in the fabric, which can be obtained by issuing the command nsShow.
■
The total number of Name Server entries, which can be determined by issuing the command nsAllShow.
You can also obtain this data by issuing the command supportShow for every switch and then pulling the required data out of log. Another option is to automate the acquisition of data and then parse out the necessary fields. Figure 8.17 and Table 8.4 are examples of the necessary data collection and what a SAN profile looks like.The data to collect is bolded in Figure 8.17 as well. Figure 8.17 Data to Collect When Establishing a SAN Profile BigSAN102:admin> nsShow The Local Name Server has 2 entries { Type Pid N
COS
PortName
NodeName
TTL(sec)
661600; 3;50:00:60:e8:02:76:b9:04;50:00:60:e8:02:76:b9:04; na FC4s: FCP [HITACHI OPEN-9
0112]
Fabric Port Name: 20:06:00:60:69:10:67:c4 N
661b00; 3;50:00:60:e8:02:76:b9:00;50:00:60:e8:02:76:b9:00; na FC4s: FCP [HITACHI OPEN-9
0112]
Fabric Port Name: 20:0b:00:60:69:10:67:c4 } BigSAN102:admin> nsAllShow 16 Nx_Ports in the Fabric { 641300 661600 661b00 6a1100 6b1000 6b1101 6b1600 6d1100 6d1200 6d1300 7215e1 761d01 761e00 771d00 771f00 781e00 } BigSAN102:admin> topologyShow
26 domains in the fabric; Local Domain ID: 102
Output truncated. Make sure you capture all domain Ids in the fabric.
140_SANs_08
8/14/01
5:26 PM
Page 311
SAN Troubleshooting • Chapter 8
Table 8.4 Formatted SAN Profile Switch
Local NS Entries
BigSAN100 BigSAN101 BigSAN102 BigSAN103 BigSAN104 BigSAN105 BigSAN106 BigSAN107 BigSAN108 BigSAN109 BigSAN110 BigSAN111 BigSAN112 BigSAN113 BigSAN114 BigSAN115 BigSAN116 BigSAN117 BigSAN118 BigSAN119 BigSAN120 BigSAN121 BigSAN122 BigSAN123 BigSAN124 BigSAN125 Total Nodes Total Switches
0 0 2 4 1 1 4 4 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 16 26
311
140_SANs_08
312
8/14/01
5:26 PM
Page 312
Chapter 8 • SAN Troubleshooting
What Data Can a Host Provide? A host can provide a significant amount of data to aid the SAN troubleshooting process.Think again of the SAN as a virtual cable. A working virtual SAN cable means that edge devices that are expected to communicate with each other are successfully connected as N_Port or NL_Port (verify this with switchShow), and that the devices are present in the Name Server (verify this with nsShow). Assuming that zoning is properly configured, these edge devices should be able to communicate with each other, just as if they are directly connected to each other with a cable. A host can indicate if devices are visible to that host. In a Windows environment, do this by running Disk Administrator; in a UNIX environment, do this by issuing the format command. Many tools from HBA vendors are GUI-based and allow for real-time, live viewing of connection status to storage devices. Some examples of these tools are TROIKA’s SAN Command and JNI’s EZ Fibre. If the devices do not show up at the host when these commands are issued, the next step is to see why these devices are not visible to the host.The key to this involves reviewing the host log files. For Solaris, the message log file is normally located in the file /var/adm/messages.You can watch the SAN HBA events in real time, by doing a tail –f /var/adm/messages. For a Microsoft environment, you can use the Event Viewer to see the HBA-related activity.You might need to change the verbosity levels or set the HBAs to debug mode to see detailed data in the message logs. An example of a log from a Solaris host is provided in Figure 8.18 to familiarize you with the data and how you might use it to assist in the troubleshooting process. In Figure 8.18, the HBA recognizes and has visibility to seven JBOD disks. At this point, you can conclude that the SAN virtual cable is working fine and that the host has visibility to the devices at a SAN level.The next step is to see if the devices are visible to the operating system. Based on the next set of error messages, the indications are that the devices are not visible to the operating system and that the HBA should be investigated further.The problem illustrated in Figure 8.18 occurred because the host HBA drivers were not configured to bind with any targets; hence, the disks were not presented to the operating system.To resolve this issue, it is necessary to follow the HBA directions for binding SAN targets.
313
The devices return
The link comes back up
The devices go away
—an HBA issue ?
Now there looks to be an issue with the bindings
May 16 21:30:45 sun1 jnic: [ID 594302 kern.notice] jnic0: Target15 Lun0: Initialization failed: No fibre channel bindings provided.
May 16 21:30:45 sun1 jnic: [ID 473743 kern.notice] jnic0: Target14 Lun0: Initialization failed: No fibre channel bindings provided.
May 16 21:30:45 sun1 jnic: [ID 353184 kern.notice] jnic0: Target13 Lun0: Initialization failed: No fibre channel bindings provided.
May 16 21:30:45 sun1 jnic: [ID 232625 kern.notice] jnic0: Target12 Lun0: Initialization failed: No fibre channel bindings provided.
May 16 20:08:35 sun1 jnic: [ID 549483 kern.notice] jnic0: Port 0214EF (WWN 2000002037D97745:2100002037D97745) available.
May 16 20:08:35 sun1 jnic: [ID 121592 kern.notice] jnic0: Port 0214E8 (WWN 2000002037D96FEB:2100002037D96FEB) available.
May 16 20:08:35 sun1 jnic: [ID 900463 kern.notice] jnic0: Port 0214E4 (WWN 2000002037D974D7:2100002037D974D7) available.
May 16 20:08:35 sun1 jnic: [ID 172876 kern.notice] jnic0: Port 0214E2 (WWN 2000002037D9775A:2100002037D9775A) available.
May 16 20:08:35 sun1 jnic: [ID 887895 kern.notice] jnic0: Port 0214E1 (WWN 2000002037D976B3:2100002037D976B3) available.
May 16 20:08:35 sun1 jnic: [ID 861423 kern.notice] jnic0: Port 0214E0 (WWN 2000002037D9730F:2100002037D9730F) available.
May 16 20:08:35 sun1 jnic: [ID 475247 kern.notice] jnic0: Port 0214DC (WWN 2000002037D97796:2100002037D97796) available.
May 16 20:08:35 sun1 jnic: [ID 184835 kern.notice] jnic0: Link Up
May 16 20:08:29 sun1 jnic: [ID 846798 kern.notice] jnic0: Port 0214EF (WWN 2000002037D97745:2100002037D97745) removed.
May 16 20:08:29 sun1 jnic: [ID 944579 kern.notice] jnic0: Port 0214E8 (WWN 2000002037D96FEB:2100002037D96FEB) removed.
May 16 20:08:29 sun1 jnic: [ID 682426 kern.notice] jnic0: Port 0214E4 (WWN 2000002037D974D7:2100002037D974D7) removed.
May 16 20:08:29 sun1 jnic: [ID 355286 kern.notice] jnic0: Port 0214E2 (WWN 2000002037D9775A:2100002037D9775A) removed.
May 16 20:08:29 sun1 jnic: [ID 515504 kern.notice] jnic0: Port 0214E1 (WWN 2000002037D976B3:2100002037D976B3) removed.
May 16 20:08:29 sun1 jnic: [ID 832114 kern.notice] jnic0: Port 0214E0 (WWN 2000002037D9730F:2100002037D9730F) removed.
May 16 20:08:29 sun1 jnic: [ID 957663 kern.notice] jnic0: Port 0214DC (WWN 2000002037D97796:2100002037D97796) removed.
May 16 20:08:29 sun1 jnic: [ID 229332 kern.notice] jnic0: Link Down
5:26 PM
May 16 20:08:29 sun1 jnic: [ID 619166 kern.notice] jnic0: Loss of sync detected
The link was reset
8/14/01
Solaris 2.8 host JNI HBA
Figure 8.18 Solaris Host SAN-Related Messages
140_SANs_08 Page 313
140_SANs_08
314
8/14/01
5:26 PM
Page 314
Chapter 8 • SAN Troubleshooting
When to Use portLog and Other Advanced Tools The portLog debugging tool is a low-level tool for debugging the SAN.The portLog facilities are available in two forms: portLogDump and portLogShow. The help page for portLogShow is reasonably detailed and helpful for decoding portLog entries.To effectively understand portLog data, you will need a solid background in Fibre Channel fundamentals.Training on decoding portLog data is available from Brocade (www.brocade.com/education_services). An annotated example of a portLog entry is shown in Figure 8.19 to provide some insight into how a portLog entry is decoded.You will most likely encounter portLog data when entering the supportShow command, which calls portLogDump, or if you are requested to obtain this data by Brocade support. Figure 8.19 portLog Entry Example Port 6 D_ID = name server ELS -> PLOGI
21:01:30.216 tReceive
Rx3
6
116
22fffffc,00011600,00f2ffff,03000000
Task Timestamp
Class 3 frame received
Frame payload size
S_ID = Domain 1 Port 6
For Fibre Channel developers and people who are intimately involved with SANs, a programmer’s guide is available (Fabric Programming Guide Revision 2.1 Publication number 53-0001561-01Rev. A 4/11/00).The guide is available from the Brocade Web site (www.brocade.com); however, a login and password are required. Instructions for obtaining a login and password are posted on the Web site. Another low-level debugging tool is a Fibre Channel analyzer. Companies such as Finisar (www.finisar.com) and Xyratex (www.xyratex.com) manufacture Fibre Channel analyzers. An analyzer is typically used in a development environment and rarely to debug production environments. Analyzers can generate a tremendous amount of data. An analyzer is usually inserted into the SAN between the switch and an edge device, or between two switches. Normally, a
140_SANs_08
8/14/01
5:26 PM
Page 315
SAN Troubleshooting • Chapter 8
detailed analysis and troubleshooting effort is required to identify where to insert the analyzer into the SAN and what data the analyzer should look for. Again, an extensive background in Fibre Channel is necessary to effectively use an analyzer.
In-Depth Troubleshooting with Fibre Channel Analyzers Although configuring SANs is getting easier with each new generation of equipment, it is often useful to have the appropriate tools for configuring and testing your SAN. As in standard Ethernet-based networks, and even in local parallel SCSI bus installations, network sniffers and bus analyzers are very handy tools to really understand what is going on. Fibre Channel cable testers can be purchased for nominal amounts, link activity analyzers for several hundred dollars, and full-blown protocol analyzers for several thousand dollars. Fibre Channel cable testers provide simple connectivity tests for a cable; in the case of copper cables, they test for connectivity between two ends of a cable. Similar optical tools are available for checking the amount of light that is transmitted through an optical cable, and they provide convenient diagnostic capabilities for cable integrity. An affordable alternative to full-blown protocol analyzers is a link activity analyzer. Link activity analyzers attach to Fibre Channel cables and analyze basic activity on the link. Basic functionality includes LEDs to indicate when traffic is being sent and received, as well as information such as MB/sec counters, online or offline information, error lights for CRC errors, and optical signal quality indicators. These types of link activity analyzers are ideal for isolating specific problem areas in a SAN, and identifying questionable links or devices. Finally, for the most information about what is happening on a SAN, protocol analyzers are the best tools available. These tools will record every bit of information that comes across a wire, and through user software can play back activity, show errors, highlight questionable transactions, and more. Ranging from simple two-channel analyzers embedded in a PC to multichannel testers that can test all of the ports of a Fibre Channel switch in a single box, these analyzers are invaluable if you really want to know what is going wrong with your network. These tools can be invaluable for debugging problems directly at the source, and are often bundled with training and classes to help you learn the basic protocol and debugging techniques. For many problems Continued
315
140_SANs_08
316
8/14/01
5:26 PM
Page 316
Chapter 8 • SAN Troubleshooting
you encounter in a development environment, a protocol analyzer is the only tool that will help you really see what is going on. However, in production environments it is unnecessary to invest in a full analyzer for day-to-day operation.
Troubleshooting the Fabric A problem with the fabric is a pervasive issue that often affects more than one device.When a fabric issue is experienced in a resilient SAN, it might have no impact on SAN functionality since the SAN redundancy compensates for the marginal situation.Table 8.5 provides a high-level review of problematic fabric symptoms and associated possible causes. Fabric issues are normally associated with heterogeneous storage and server environments in which all devices have not been tested as a system. Table 8.5 Symptoms Indicative of a Fabric Problem Symptom
Possible Causes
Multiple edge devices are inaccessible from multiple hosts
■ ■ ■
■ ■ ■ ■ ■ ■
Incompletely initialized ISLs: ISL port initializes as a G_Port or does not come online
■ ■
Fabric segmentation (zone conflict, mismatched fabric parameters) Switch failure Edge device timeout or communication conflict when accessing the Name Server (FFFFFC) or Fabric F_Port (FFFFFE) Unconfirmed domain Message Queue (MQ) issues Hosts and/or storage attempted to access the fabric prior to fabric convergence Domain ID conflict Port configuration conflict No fabric license installed Marginal link Fabric initialization error
The remainder of this section identifies what tools to use and data to analyze when a fabric issue is suspected. Symptoms are explained in further detail and specific issue traits are identified.Where possible, workarounds or corrective actions are specified.
140_SANs_08
8/14/01
5:26 PM
Page 317
SAN Troubleshooting • Chapter 8
What to Look for in a Malfunctioning Fabric If a switch is unable to join the fabric, all devices on that switch become inaccessible to the fabric and possibly to each other.When edge devices time out or are unable to properly communicate with fabric services, communication between numerous edge devices is interrupted and some devices become inaccessible.
NOTE When initially identifying a fabric issue, look for a large number of edge devices to be behaving marginally or not communicating at all. See if you can identify a pattern. Is the outage random throughout the fabric, or can you correlate the outage to a particular switch? Does the outage correlate to one particular host type or storage device?
Host Behavior Hosts that are involved with a fabric problem exhibit a variety of symptoms, one of which is that some or all edge devices become inaccessible.You can verify this situation for UNIX hosts using the command format to see if any devices have disappeared. For Microsoft Windows 2000, start up the Disk Management utility and check if any devices have disappeared.The Solaris /var/adm/messages file and the Microsoft Event Viewer might provide further insight into the issue. ISL initialization issues normally are invisible to the host, as the fabric will reroute around failed ISLs and ensure connectivity—unless the ISL failure results in the SAN becoming segmented, in which case edge devices will become inaccessible to the host. Another possible symptom on the server is reduced performance of the application. In the event of an ISL failure, the fabric will reroute the traffic as mentioned.When this happens, typically, the traffic will have to share ISLs with more devices than normal, possibly resulting in reduced performance due to congestion on the ISLs. Utilities supplied by your HBA vendor can also be helpful in identifying host SAN status.
SAN Profile If you suspect a SAN issue, create a new SAN profile and compare your baseline SAN profile to your newly created SAN profile. Any unexplained discrepancies
317
140_SANs_08
318
8/14/01
5:26 PM
Page 318
Chapter 8 • SAN Troubleshooting
require further investigation—whether one or more switches have dropped out, or if there are several missing Name Server entries.
Switch LEDs If you can observe the SAN switches while the problem is occurring, try to detect an LED pattern. Focus on the ISL ports first. Any yellow lights (blinking or steady) indicate that manual intervention is required. At this point, log in to the switch with yellow lights and issue the command supportShow to extract debugging information for further analysis. If the switch is disabled (all ports blinking a slow yellow), issuing a switchEnable command will bring the switch back into the SAN. If a port is yellow (blinking or steady), you can bring the device back online by issuing the command portDisable and then a portEnable on the yellow port. Issue the command switchShow to verify the port state or a disabled switch.
The errShow Command Start the troubleshooting process by reviewing errShow data for every switch in the fabric. Fabric segmentation and Message Queue (MQ) errors are indicative of an error that will cause the switch and its connected devices to become inaccessible to the fabric. Fabric segmentation is also caused by zone conflicts, incompatible fabric parameters, or domain conflict. Review the errShow as a starting point.
The switchShow Command When investigating fabric issues, you need to look at switchShow for port state information and for fabric-related information. Issue the switchShow command on every switch in the fabric. Examine the port state section of the switchShow data for incompletely initialized E_Ports, which will show up as G_Ports or as ports that are not online. If the port does not reinitialize itself, then manually reinitialize the ISL by executing the commands portDisable and portEnable, providing the offending port number as an argument. A fabric issue that has less impact involves incomplete ISL initialization. If ISL initialization issues occur, it is usually during fabric bring up. ISL initialization issues can also occur during a fabric reconfiguration, which is triggered when an ISL is added or removed or when a switch is added or removed. If the SAN is designed to be resilient, an incomplete ISL initialization minimally impacts the fabric, since there are multiple ISLs connecting the switches and edge devices are still able to communicate with each other. On the other hand, if the SAN is not
140_SANs_08
8/14/01
5:26 PM
Page 319
SAN Troubleshooting • Chapter 8
resilient, an ISL initialization problem may result in a segmentation of the SAN and many devices may lose communications with the SAN. Resilient topologies deliver at least two internal fabric routes and are considered more resilient because each topology is capable of sustaining a switch or ISL failure while the remaining switches and fabric remain operational.This selfhealing capability is enabled by Fabric Shortest Path First (FSPF) and is depicted in Figure 8.20. Figure 8.20 In a Resilient SAN, an ISL Failure Does Not Affect Communication
A
B failed ISL
A
B
C
Figure 8.20 also depicts the failure of an ISL in a cascade topology, which is the SAN located on the left. Note that switches A and B are unable to communicate with the remaining switches when the ISL marked with the “X” fails. However, a similar switch failure in a resilient topology SAN (located on the right) does not sever communications between the remaining switches. If the ISL fails, it is still possible for switch A to communicate with switch C, using several paths, such as the path highlighted in Figure 8.20. In a resilient topology, an ISL failure might go unnoticed unless some type of monitoring is used (such as Fabric Watch, a separately licensed product available from Brocade). Additionally, with the loss of an ISL, there may also be performance degradation due to a loss of overall available bandwidth. When reviewing the fabric-related information of switchShow, search for a switch that is disabled or has an unconfirmed domain. An unconfirmed domain indicates that the switch was unable to communicate with the principal switch in the fabric to obtain a domain ID.To resolve either situation, issue the command switchDisable followed by switchEnable to enable the switch to join the fabric.
319
140_SANs_08
320
8/14/01
5:26 PM
Page 320
Chapter 8 • SAN Troubleshooting
The topologyShow Command The topologyShow information is straightforward.You have to issue the topologyShow command on only one switch, unless that switch happens to be disabled or segmented. If this is the case, the topologyShow data will indicate the number of switches in your fabric as one, and you need to pick another switch to obtain the topologyShow information. The number of domains should equal the number of switches in the SAN.You can reference your SAN profile to establish the expected number of switches in the SAN. If there is an unexplained discrepancy, you most likely have a failed, segmented, or disabled switch.You can use switchShow data to identify a disabled or segmented switch. If a switch that is supposed to be part of the fabric does not show up in the topologyShow output (the previous SAN profile helps here), the administrator should identify the switch, log in to it, and try first a portDisable-portEnable sequence on any of the ports that should be an E_Port. If this does not work, try a switchDisable-switchEnable sequence.
The nsShow and nsAllShow Commands Issue the command nsAllShow on any switch in the fabric to obtain the total number of edge devices registered with the Name Server. Note that issuing the nsAllShow command on a switch that is segmented or disabled will return Name Server data for only the switch and not the entire fabric. If there is an unexplained discrepancy between this number and the number of Name Server entries recorded in your SAN profile, you will need to identify which switches are associated with the missing Name Server entries. First, check to see if there are a number of missing devices; if so, then it is likely that one of the switches has segmented or is offline.This should have been seen in the prior step. If you are unsure of what devices are missing, issue the command nsShow on each switch in the SAN and compare the number of Name Server entries to your SAN profile. Next, attempt to correlate the missing Name Server entries. Are the missing entries all associated with any particular switch or edge device? Once you rule out a segmented or disabled switch, determine if the port associated with the missing devices is online. If the port is not online, bring the port online by executing the commands portDisable and portEnable, supplying the questionable port number as an argument to these commands.This should refresh the Name Server with the missing port edge devices. If the missing Name Server device port comes online, and it still does not register with the Name Server, then it indicates that there is either a timeout or a conflict in communication between
140_SANs_08
8/14/01
5:26 PM
Page 321
SAN Troubleshooting • Chapter 8
the Name Server and the edge device in question. It is now time to work with your switch supplier and edge-device supplier to resolve this complex problem.
Now that You Suspect a SAN Issue: Digging Deeper Now that you suspect a SAN issue, you will need to investigate further to identify the root cause.The use and context of each command follows, relative to troubleshooting a SAN issue.Where possible, workarounds or corrective actions are identified. Several commands must be run on each switch; this is something that can be automated.The details for doing so are presented later in the book in Chapter 9.
Timeout of Edge Devices during Fabric Bring Up If the problem occurs after a SAN bring up or during reconfiguration, it is possible that the edge devices came online before the SAN is ready. If this is the case, you will see flickering green and possibly flickering yellow lights on the ISL ports as the SAN converges while the edge ports remain steady green.You will also see messages on the switch console as edge devices attempt to FLOGI and Port-to-Port Login (PLOGI). Normally this is acceptable; however, if the SAN requires an extended period of time for bring up, devices might time out. Be careful to differentiate between an edge device that successfully retries PLOGIs and FLOGIs while the fabric converges, and do not interpret these retries as failures.When the fabric is completely up, most devices that time out will try again; however, if they do not, a timeout failure is to be expected. If you suspect a PLOGI/FLOGI timeout failure during fabric convergence, you can confirm your suspicions by reviewing the host logs.You can determine the SAN state, by issuing a topologyShow command and verifying that the correct number of domains are in the fabric. If the edge devices are not tolerant of the time it takes the SAN to converge, they might time out their FLOGI or not successfully interact with the Name Server. In either case, that device will be inaccessible to the fabric. If you suspect this is happening with your SAN, investigate the edge device logs to conclusively determine that timeouts are occurring, the type of timeout, and how long these timeouts last. If timeouts are occurring, one resolution is to increase the timeout values in the fabric (Resource Allocation Time Out Value [R_A_TOV] or Error-Detect Time Out Value [E_D_TOV]) or with the edge devices.There might be other timeout values on the edge device that might help prevent this issue; however changing timeout
321
140_SANs_08
322
8/14/01
5:26 PM
Page 322
Chapter 8 • SAN Troubleshooting
values is a complex procedure and it is suggested that you work with your switch supplier and edge device supplier at that point.
Port Configuration Conflict or Missing Fabric License If your switch is not configured with a fabric license, it cannot join the fabric. The port state section of the switchShow will indicate that the E_Ports are unknown.When you issue the command licenseShow, you should see a fabric license. If the switchShow data indicates unknown E_Ports and you do not have a fabric license installed, you will not be able to join that switch into a fabric until you acquire a fabric license from your switch supplier.The SilkWorm 2010 and 2100 switches are entry-level switches and are not configured with a fabric license, but can be upgraded with a simple license key.These switches are designed for switched loop connectivity using Brocade QuickLoop.They can have a single E_Port for connecting another QuickLoop switch; however, if additional ISLs are connected, they will not come online.The SilkWorm 2240 and 2250 are entry fabric switches designed for small SANs or for the edge of a larger SAN.They can also only support a single E_Port unless you upgrade them to a full fabric license. Figure 8.21 provides an example of a properly installed fabric license. Figure 8.21 Example of a Properly Installed Fabric License core1:admin> licenseShow SRzy9Sz9zeTS0zAG: Web license bbSz9eQb9zccT0AQ: Zoning license RdzdSRcSyzSe0eTn: QuickLoop license cSczRScd9RdTd0SY: Fabric license
<------A Fabric license is properly installed
It is possible to prevent switches from connecting into a fabric by disabling E_Port functionality.You might want to do this for security purposes to prevent unauthorized switches from joining the SAN. If your E_Ports are unknowingly disabled, it will not be possible to join the switch into a fabric.To verify the status of the switch E_Ports, issue the command portCfgEport, as shown in Figure 8.22.
140_SANs_08
8/14/01
5:26 PM
Page 323
SAN Troubleshooting • Chapter 8
Note that switch port 0 E_Port capability is disabled and that you cannot use this port as an E_Port.To disable or enable E_Port support for a port, use the portcfgEport command.You might want to do this for security purposes, since turning off an E_Port prevents someone from attaching a switch to your fabric without first obtaining your approval. Figure 8.22 E_Port Configured as Disabled Example core1:admin> portcfgEport Ports:
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
----------------------------------------------------------NO
-
-
-
-
-
-
-
-
-
-
-
-
-
Segmented Fabrics A fabric can segment for a variety of reasons, including zone conflicts, incompatible fabric parameters, and domain ID conflicts.This section helps you identify whether you have fabric segmentation, and what type of fabric segmentation you are experiencing. A fabric might segment when you add a new switch to the fabric or upon fabric reconfiguration or bring up.The segmented fabric error message will occur on any switch to which the new switch is trying to connect.The new switch that is trying to join the fabric will show the E_Ports as unknown output from the switchShow command. If the fabric segments during a reconfiguration or bring up, you will have to search for a switch with unknown E_Ports, which can be determined by examining the switchShow output.You can also compare your current SAN profile to your baseline SAN profile to identify the missing switch.
Zoning Conflict A zone conflict and fabric segmentation can occur when introducing a single- or multiple-switch fabric into an existing fabric. As these conflicts may affect the connected online devices, the switches segment and await human intervention to determine the proper resolution.There is no way to identify the correct configuration without first investigating the nature of the conflict. If there are conflicts, it may be easier to clear the configuration on the conflicted switch and then have that switch absorb the zone information when it becomes part of the fabric. Typically, there are three conditions that will create a zone conflict: ■
Multiple zoning configurations enabled Enabling zoning on both fabrics when they are connected will create a zone conflict. Only one
323
140_SANs_08
324
8/14/01
5:26 PM
Page 324
Chapter 8 • SAN Troubleshooting
zone configuration can be enabled in a single fabric at a time. An example of this is if the Day configuration is enabled on one switch and the Night configuration is enabled on the other.The administrator will have to decide which one is appropriate and disable the other. ■
Zone definition type conflict This occurs when introducing a single- or multiple-switch fabric into an existing fabric that has zoning definitions already defined, but the definition type (in other words, alias, zone) is in conflict. An example of this would be a definition of Red as a zone defining one fabric, and Red as an alias definition on another fabric.This is a definition conflict and will segment the fabric.
■
Zone definition content conflict This occurs when introducing a single- or multiple-switch fabric into an existing fabric that has zoning definitions (in other words, alias, zone) already defined, but the content is in conflict.This is where the definition name and type match, but the content is different. An example of this would be a Red zone defined on both fabrics. On the first fabric, the Red zone was defined with domain 5, port 4, and the second fabric has the Red zone defined with domain 7, port 3. Both have a zone definition of Red, but the content is in conflict and will cause the fabric to segment. Again it will require that the administrator determine which Red zone is correct and either update the incorrect one or delete it. Once the fabrics merge, the proper Red zone will be propagated to all the switches in the fabric.
The workaround for this situation involves correcting the conflicts or clearing the zoning information on either the fabric or new switch, depending on which zoning configuration you consider to be correct and want to keep.You can clear a zoning configuration by issuing a cfgClear command followed by a cfgDisable command.You should first save a copy of the zone configuration by issuing cfgShow and saving the output to a file (in case you mistakenly delete the wrong configurations).The configUpload command is also useful for this operation. Figure 8.23 shows a zone conflict error message. Figure 8.23 Zone Conflict Error Message 0x10addf10 (tZone): May 15 09:37:01 (12) Error FABRIC-SEGMENTED, 3, port 4, zone conflict
140_SANs_08
8/14/01
5:26 PM
Page 325
SAN Troubleshooting • Chapter 8
Incompatible Fabric Parameters Certain system configuration settings are changed by issuing the command configure.The fabric parameter system configuration settings must be the same for every switch in the fabric.The fabric will segment if there is a difference between the parameters that exist in the fabric and the parameters on a switch that is trying to join the fabric.The following parameters must be consistent with the switch that is joining the fabric and the fabric: BB credit: (1..27) [16] R_A_TOV: (4000..120000) [10000] E_D_TOV: (1000..5000) [2000] Data field size: (256..2112) [2112] Sequence Level Switching: (0..1) [0] Disable Device Probing: (0..1) [0] Suppress Class F Traffic: (0..1) [0] SYNC IO mode: (0..1) [0] VC Encoded Address Mode: (0..1) [0] Core Switch PID Format: (0..1) [0] Per-frame Route Priority: (0..1) [0] Long Distance Fabric: (0..1) [0]
NOTE The range of values and defaults for Fabric OS 2.4.1a are shown in the list of parameters in this section. Fabric parameters are subject to change, and you should consult the documentation of the Fabric OS version you intend to use.
Figure 8.24 shows an example of an incompatible fabric parameters error message occurring on switch edge1, and the resulting switchShow data from that switch. It is necessary to track down the switch connected on ports 0 and 2 of switch edge1 and compare the fabric parameters from that switch to those of edge1. Once you identify the discrepancy, use the configure command to change the discrepant fabric parameters of the joining switch to those of switch edge1.
325
140_SANs_08
326
8/14/01
5:26 PM
Page 326
Chapter 8 • SAN Troubleshooting
Figure 8.24 Incompatible Fabric Parameters edge1:admin> switchShow switchName: edge1 switchType: 2.4 switchState: Online switchRole: Subordinate switchDomain: 2 switchId: fffc02 switchWwn: 10:00:00:60:69:11:f9:f7 switchBeacon: OFF port 0: sw Online E-Port (unknown) port 1: sw Online E-Port 10:00:00:60:69:10:9b:5b "core2" (upstream) port 2: sw Online E-Port (unknown) port 3: sw Online E-Port 10:00:00:60:69:10:9b:5b "core2" port 4: -- No_Module port 5: -- No_Module port 6: cu Online L-Port 7 public port 7: -- No_Module port 8: -- No_Module port 9: -- No_Module port 10: -- No_Module port 11: -- No_Module port 12: -- No_Module port 13: -- No_Module port 14: -- No_Module port 15: -- No_Module
Error 01 -------0x10f6f4d0 (tTransmit): May 17 18:10:38 (8) Error FABRIC-SEGMENTED, 3, port 0, incompatible flow control parameters
Edge1 0 1 2 3 4 5 6 7 8 9 10 1 12 12 14 15
?
Switch with incompatible fabric parameters
Domain ID Conflict A domain ID conflict can occur if you join a switch that is in the online state into a fabric, and the joining switch domain ID conflicts with the domain ID of
140_SANs_08
8/14/01
5:26 PM
Page 327
SAN Troubleshooting • Chapter 8
a switch in the fabric. Normally, domain IDs are automatically assigned; however, once a switch is online, the domain ID cannot change, as it would change the port addressing and potentially disrupt critical I/O.The resolution for this problem involves performing a switchDisable followed by a switchEnable on the joining switch.This will enable the joining switch to obtain a new domain ID as part of the process of coming online.The fabric principal switch will allocate the next available domain ID to the new switch during this process.
NOTE Changing domain IDs can have an impact on port zoning entries. Be sure to check to see if any port zoning entries exist for devices on a switch before changing its domain ID, and update any affected zones to reflect the change.
Message Queue Errors An MQ error is a message queue error.You can identify an MQ error message by looking for the two letters M and Q in the error message. MQ errors can result in edge devices dropping from the Name Server or preventing a switch from joining the fabric. MQ errors are rare and difficult to troubleshoot, and it is suggested that you resolve them by working with your switch supplier.When you encounter MQ errors, execute the supportShow command to capture debug information about the switch. A switch reboot will likely clear any associated problems.Then forward the supportShow data to your switch supplier for further investigation.
Troubleshooting Devices that Cannot Be Seen A host that is unable to access a SAN device is a more common SAN issue that can arise. Again, consider the virtual SAN cable analogy to start the troubleshooting process.We want to determine whether the SAN is the cause of the problem or whether it is an edge device issue.To do this you need to work your way along the virtual SAN cable to the edge device(s) that cannot be seen. Figure 8.25 depicts a flowchart that outlines the process for troubleshooting a missing device.
327
Follow fabric troubleshooting procedure
328 Problem Identified
Yes
Is it a zoning issue ?
Yes
Is storage device visible in name server ?
Yes
No
No
No
Timeout or Name Server conflict Escalate
Problem Identified
Yes
Node configuration issue ?
Problem Identified
Yes
Is there a port configuration conflict?
No
No
Name Server conflict - Escalate
Follow marginal link troubleshooting procedure
5:26 PM
Yes
No
Is the storage device present in switchShow?
8/14/01
Is it a fabric issue ?
Storage Device not visible to host
Figure 8.25 Troubleshooting Devices that Cannot Be Seen
140_SANs_08 Page 328
140_SANs_08
8/14/01
5:26 PM
Page 329
SAN Troubleshooting • Chapter 8
What to Look for in the Fabric The first step is to determine whether the missing device problem is a fabric issue. A quick way to determine this is to establish if the problem is localized to just a single missing device or multiple missing devices.You also want to ensure all switches are online in the fabric.You can quickly check your fabric status by issuing the command topologyShow to verify that the correct number of domains exist in your fabric.You can verify that the missing device is a localized issue by entering the command nsAllShow to establish the total number of devices in the fabric. If you suspect a fabric issue, since multiple devices are missing, follow the fabric troubleshooting process. If you suspect a missing device issue, since only one or two devices are unaccessible, move on to the next section,“Are the Host and Storage Visible via switchShow on Their Respective Switches?”
Are the Host and Storage Visible via switchShow on Their Respective Switches? Use the command switchShow on the switch to which the subject host is connected.Verify that the host port and the storage port are online. If both the storage and the host port are online, move on to the next section, as the virtual SAN cable is logically connected to both the storage and the host. If the port is not online, your host or storage might be malfunctioning, you might have a link initialization issue, or you might have a marginal link. If the edge port is not online or is a G_Port, this is analogous to having a disconnected cable. A host malfunction is a very broad term and can include problems such as incorrect or improperly installed HBA drivers, HBA parameters, or a faulty HBA. A storage malfunction can include an incorrect or improperly configured storage interface or a faulty storage interface.
NOTE A quick method of identifying the cause of a missing device is to visibly inspect your switch LEDs. Any steady or flashing yellow lights indicate that a port is not online and manual intervention is required.
Brocade SilkWorm switches by default automatically configure the appropriate port topology based on the connecting port topology, which is either N_Port or
329
140_SANs_08
330
8/14/01
5:26 PM
Page 330
Chapter 8 • SAN Troubleshooting
NL_Port, or in the case of a switch, an E_Port.This functionality is invaluable for SAN management, because it alleviates the SAN administrator from managing and maintaining the configuration for potentially thousands of ports. In some situations, it is necessary to configure a port for a particular topology by using one or more of the commands portCfgEport, portcfgFAport, or portcfgLport to lock the port into a certain state.This may help with an issue where the edge device supports multiple port topologies and does not initialize in the mode that is desired. A switch or port might also be configured for QuickLoop. First, check to see that the switch or port in question is configured correctly for the intended purpose. For example, if the attaching edge device is configured as an NL_Port and the switch port is configured as an F_Port, there is a conflict and that edge device might initialize as a G_Port. Initializing as a G_Port is just as bad as not initializing at all, as the associated device is essentially inaccessible.The G_Port, or generic port, is a transitional state defined in the standards as a device transitions to an F_Port or an E_Port. If the port connecting to your edge device is not intended to be a QuickLoop port, you will need to reconfigure that port, or the edge device might not initialize properly. If there is any conflict, resolve the conflict with the switch, by reconfiguring the port, or with the edge device and move onto the next section, “Do the Devices Show Up in the Name Server?” If the devices support both loop and fabric modes, utilize the fabric setting to get the best performance and fault isolation. See Figure 8.26 for the usage and examples of various port configuration commands. Switch core1 is configured for QuickLoop, as evidenced by the enabled entries in the QuickLoop mode column. Switch core1 port 8 is configured as a loop port, and no ports are configured as Fabric Assist (FA) ports.You can also use the command qlShow to determine if the switch is configured for QuickLoop. If the switch is in QuickLoop mode and no QuickLoop is required, you can issue a qlDisable command to disable QuickLoop for the entire switch. If QuickLoop is required, but is not needed for the port in question, use the qlPortDisable command for the port that needs to be changed. Figure 8.26 Port Configuration Examples core1:admin> qlportshowall
PortNum QuickLoop Mode
Port State
0
Enabled
fabric
E PORT
1
Enabled
fabric
E PORT Continued
140_SANs_08
8/14/01
5:26 PM
Page 331
SAN Troubleshooting • Chapter 8
Figure 8.26 Continued 2
Enabled
fabric
E PORT
3
Enabled
fabric
E PORT
4
Enabled
fabric
E PORT
5
Enabled
fabric
E PORT
6
Enabled
offline
7
Enabled
offline
8
Enabled
fabric
9
Enabled
offline
10
Enabled
fabric
11
Enabled
offline
12
Enabled
offline
13
Enabled
offline
14
Enabled
offline
15
Enabled
offline
core1:admin> portcfgLport Ports:
0
1
2
3
4
5
6
7 8
9
10
11 12
13
14
15
-------------------------------------------------------Lock
-
-
-
-
-
-
-
- YES
-
-
-
-
-
-
-
Private -
-
-
-
-
-
-
- -
-
-
-
-
-
-
-
6
7 8
9
10
11 12
13
14
15
core1:admin> portcfgFAport Ports:
0
1
2
3
4
5
---------------------------------------------------------
-
-
-
-
-
-
- -
-
-
-
-
-
-
-
If the port is not online or initializes as a G_Port, attempt to reinitialize the port by executing the commands portDisable and portEnable, supplying the port number in question as an argument to these commands. If this process works, monitor the situation carefully. If the host port consistently does not come online or comes up as a G_Port repeatedly, you might have a marginal link issue, a faulty HBA, HBA driver, or some type of configuration conflict between the host and the switch. At this point, you need to follow the process of troubleshooting a
331
140_SANs_08
332
8/14/01
5:26 PM
Page 332
Chapter 8 • SAN Troubleshooting
marginal link. If the link is not marginal, contact your switch supplier and HBA supplier to assist with further troubleshooting. Follow a similar process for the storage port. If the storage port is not online or is a G_Port, this is analogous to a disconnected cable at the storage end. Attempt to reinitialize the port by issuing a portDisable/portEnable. Next, rule out a marginal link, faulty storage equipment, and configuration conflict between the storage and the switch. If you are still unable to establish the root cause, work with your switch supplier and your storage supplier to assist with further troubleshooting.
Do the Devices Show Up in the Name Server? At this point, you have verified that the host and storage are logically connected to the virtual SAN cable, and it is now necessary to confirm that the two edge ports are able to communicate. Use nsShow on the switch to which the storage is connected and the switch to which the host is connected to verify that these edge devices are registered with the Name Server. If you intend to verify that an Emulex HBA located on switch core1 port 8 is registered with the Name Server, the data in Figure 8.27 would confirm this. Figure 8.27 nsShow Example—Verifying that an Emulex HBA Is Registered with the Name Server core1:admin> nsShow The Local Name Server has 2 entries { Type Pid N
COS
PortName
NodeName
TTL(sec)
011800; 2,3;10:00:00:00:c9:21:5f:a7;20:00:00:00:c9:21:5f:a7; na NodeSymb: [35] "Emulex LP8000 FV3.02
DV5-4.52A7 "
Fabric Port Name: 20:08:00:60:69:10:8d:fd N
011a00; 2,3;20:00:00:e0:69:f0:07:c6;10:00:00:e0:69:f0:07:c6; na Fabric Port Name: 20:0a:00:60:69:10:8d:fd
}
If the devices in question are registered with the Name Server, it is possible that you are experiencing a zoning mismatch or a host/storage issue. If one or both devices are not registered with the Name Server, it is possible that there is a
140_SANs_08
8/14/01
5:26 PM
Page 333
SAN Troubleshooting • Chapter 8
timeout or communication issue between the edge device(s) and the Name Server. Check with the edge device documentation to determine if there is a timeout setting or parameter that may help. If this does not work, contact the support organization for the product that appears to be timing out.
Rule Out Zoning Issues It is easy to rule out a zoning mismatch if zoning is not enabled. Check to see if zoning is enabled by issuing the cfgShow command. If the output states that no configuration is in effect, zoning is not enabled. If zoning is enabled, it is possible that the two edge devices are unable to communicate with each other due to zoning conflicts.To confirm whether this is the case, review the active zoning configuration.You can do this by again issuing the command cfgShow, as shown in Figure 8.28. In this example, host1 can access disk1, and host2 can access disk2, but host1 cannot access host2 or disk2, and host2 cannot access host1 or disk1. Confirm that the specific edge devices that need to communicate with each other are in the same zone. If they are not, and zoning is active, you need to update your zoning configuration before the edge devices in question are able to communicate with each other. For example, if host1 needs to get access to disk2, it is necessary to update the zoning configuration to enable this access. Once the zone changes are made via the command line or WEB TOOLS-based GUI, the devices should be able to access one another; however, some operating systems might require that you run a disk utility such as format or disk administrator. It is also possible that some operating systems might require a reboot to allow discovery of the new devices. Figure 8.28 Zoning Example core1:admin> cfgshow Defined configuration: cfg:
colors
red; yellow
zone:
red
host1; disk1
zone:
yellow
host2; disk2
alias: disk1
0,0
alias: disk2
0,1
alias: host1
1,14
alias: host2
1,15
Effective configuration: Continued
333
140_SANs_08
334
8/14/01
5:26 PM
Page 334
Chapter 8 • SAN Troubleshooting
Figure 8.28 Continued cfg:
colors
zone:
red
1,14 0,0
zone:
yellow
1,15 0,1
NOTE If zoning is active, any devices that are not explicitly defined in a zone together are not able to communicate with each other.
At this point, if you establish that there is no switch zoning mismatch, then you have established that the SAN virtual cable is working and that it is likely a host or storage issue. One possible host or storage issue that could be causing the “missing” devices is a mismatch with the HBA or storage-based zoning; be sure to check this first when troubleshooting the edge devices.
NOTE Incorrect or incomplete zoning is one of the most common causes of SAN communication problems. Checking for this is analogous to checking to see if a “malfunctioning” computer monitor is plugged in.
Edge Device Not in the Name Server Reaching this point implies that you have verified that the edge devices in question are connected to the switch, and that one or more of the edge devices are not registered in the Name Server. Attempt to reinitialize the edge device(s) with the Name Server by executing the commands portDisable and portEnable, supplying the port number(s) in question as an argument to these commands. If, after you do this, the devices successfully register with the Name Server, you have resolved the problem. However, pay attention to this issue because if the problem recurs, it indicates a complex problem that is best resolved by working with your
140_SANs_08
8/14/01
5:26 PM
Page 335
SAN Troubleshooting • Chapter 8
switch and edge device suppliers.You should also seek this type of assistance if after issuing a portDisable/portEnable, the devices do not register with the Name Server.This fact indicates a complex issue such as a communication conflict or timeout condition. Although edge devices should reconnect to the fabric and register when the port is disabled, some older devices might time out and no longer retry logging in. If this happens, you might need to reboot the device to get that device to reset and log into the fabric and Name Server.
Troubleshooting Marginal Links A marginal switch port is defined as a switch port that is either receiving a marginal incoming signal, or the switch receiver is not functioning properly. A marginal Nx_Port transmit can be caused by an Nx_Port failing optical component (GBIC or GLM) or a cable issue. A failing Fx_Port receiver can be caused by a failing switch optical component or a failing switch port, as depicted in Figure 8.29. Figure 8.29 Marginal Port Elements
Potential Faults A marginal Fx_Port (switch port) is termed a marginal Fx_Port receive
A marginal cable or Nx_Port GBIC is termed a marginal Nx_Port transmit A point-to-point N_Port or loop (NL_Port connection) Corresponding switch is either F_Port or FL_Port.
Marginal Point-to-Point/Fabric Device Links The impact of a marginal port can be significant. For example, a large storage device such as an HP XP512, an IBM Enterprise Storage Server, or EMC Symmetrix port might be accessed by potentially dozens of hosts.The marginal
335
140_SANs_08
336
8/14/01
5:26 PM
Page 336
Chapter 8 • SAN Troubleshooting
behavior of this storage device has the potential to impact all devices that access this storage port. Imagine that you are a part of a geographically distributed team of six workers.The primary communication for this team is via telephone. Assume that your telephone is functioning marginally (similar to a poor cellular connection). Anyone who wants to call you will not be able to communicate effectively with you. Conversely, anyone who you call will also be unable to communicate effectively with you. If you are a team leader for this group, the impact of your marginal telephone capabilities is significant, since many people utilize you as a resource. Note that the others in the group are free to communicate with each other without experiencing any impact from your telephone problems.The story can have a happy ending if you gain access to two telephones, and realizing the marginal nature of one telephone line, switch to the working telephone. Note that many SANs are constructed in a similar fashion to Figure 8.30, with dual paths between hosts and storage, and a single failure does not result in an I/O failure. In applications where availability is key, dual- or even triple-redundant fabrics are always recommended. Figure 8.30 Dual-Fabric SAN Design Server
Server
Server
Server
Fabric A
Fabric B
Tape drive
Tape drive
Data
Data
Data
Data
140_SANs_08
8/14/01
5:26 PM
Page 337
SAN Troubleshooting • Chapter 8
Marginal Loop Connections While a marginal point-to-point link affects only devices that access the pointto-point device, the ramifications of a malfunctioning loop-connected device can impact all devices in that loop. Extending the geographically distributed team analogy further, imagine that the only way the team communicates is via teleconference.Whenever the team needs to communicate, everyone dials in to a conference call. Unfortunately, the teleconference is disrupted by your marginal telephone link.What makes things even worse is that communication between any other team members is impossible or very difficult. For example, it is very difficult for one member to speak with another on the teleconference because your marginal telephone continually creates static on the teleconference. Brocade QuickLoop and Fabric Assist are unique Fibre Channel topologies that combine aspects of arbitrated loop and fabric topologies.They are composed of multiple private arbitrated loops (looplets) interconnected by a fabric. It can be best described as a Private Loop Fabric Attach, as compared to Private Loop Direct Attached (PLDA) or Fabric Loop Attachment (FLA).The FL_Port of each looplet is hidden from the NL_Ports. QuickLoop is a logical PLDA that complies with the FC-AL standard. Although NL_Port devices are attached to different arbitrated loops interconnected by a fabric, the fabric and the physical device locations are transparent. QuickLoop enables switches to be used in place of hubs in environments where all attached devices are private devices. Fabric Assist mode allows the configuration of a virtual private loop in which a private host can see and access public or private targets anywhere on the fabric. Such a private loop is called QuickLoop Fabric Assist mode zone. Fabric Assist mode enables private hosts to access public or private targets anywhere on the fabric, provided they are configured in the same Fabric Assist zone. A public target accessed by a private host remains public, with full fabric functionality. The nature of loops is such that the behavior of an unhealthy device on the loop can adversely impact the behavior of the remaining devices on the loop. For example, a marginal GBIC could degrade the signal to the point where the connected NL_Port (host or storage) device is no longer able to effectively communicate.This in turn causes the loop to reset.When a loop resets, so do the individual hosts or storage devices connected to that loop. Under normal circumstances, a loop reset does not cause any harm. However, if a device is constantly resetting, I/O flow can become severely restricted or halted. Loop Initialization Primitives (LIPs) are part of a healthy loop and are used for a variety of purposes—most commonly to signal other devices on the loop
337
140_SANs_08
338
8/14/01
5:26 PM
Page 338
Chapter 8 • SAN Troubleshooting
that a new device has been added, or that an existing device has left the loop. When a loop or NL_Port resets, LIPs are generated. However, an excessive number of LIPs will make a loop unstable. The Fibre Channel standards community is making great strides in further enhancing the functionality of loops. However, loops are starting to become a legacy issue. It is important to note that Fibre Channel and SilkWorm switches also support point-to-point topologies, which are not subject to the same disruptive behaviors that loops are.When a public device accesses a private device (known as translative mode), the LIP is not propagated to that public device, nor is that public device subject to disruption.
Nx_Port (Host/Storage) Behavior with a Marginal Port in the Loop When a marginal device disrupts the loop, a variety of symptoms can be present. Performance for devices connected to the QuickLoop or devices accessing a common device can be described as slow. Host logs (that is, /var/adm/messages, eventlog, or syslog) might indicate that I/O is timing out or that the interface is being reset.The switch LEDs should be green or a blinking green light. Green lights mixed with yellow lights or flashing yellow lights indicate that the ports are resetting themselves. Devices on the affected loop might FLOGI and/or PLOGI repeatedly onto the fabric as part of a reset process initiated by the HBA.This would show up on the console or telnet management session for the switch to which the affected device was attached. N_Port devices are less susceptible to disruption for reasons stated earlier.
Marginal GBIC/Cable You can use the er_enc_out statistic to identify a marginal GBIC. Active devices (such as disks) normally clean up an encoding error as these errors are encountered, and mark the frame as having bad CRC. Any er_enc_out errors are encoding errors outside a frame, and do not generate a CRC error. If a high count (for example, several thousand) or incrementing counts of er_enc_out errors are experienced on a particular port, this indicates that the signal is marginal between the connected device’s transmit port and the switch’s receive port. Because this situation is being recorded as encoding errors, the implication is that there is no active device cleaning up the errors between the switch receive and the connected device transmit.The diagnosis: marginal GBIC or cable on the connected device.
140_SANs_08
8/14/01
5:26 PM
Page 339
SAN Troubleshooting • Chapter 8
Connected Device Note that LIPs are normal in a healthy loop. An imbalance where the Lip_in count is larger than the Lip_out count indicates that the associated connected device is the originator of LIPs in the loop. A device that generates a large number of LIPs might be malfunctioning.The switch will propagate LIPs in accordance with the Fibre Channel specification. Propagated LIPs are recorded as Lip_out.
Fault Isolation Once a marginal port is identified, it is necessary to identify where the fault resides. Figure 8.31 depicts a suggested fault isolation process. Fault isolation on a loop is very difficult, which is one of the reasons why loops had limited success.
How the Switch Can Help: Fabric Watch and QuickLoop Zoning By virtue of being positioned between storage and host, the switch is a natural resource for gathering statistics and troubleshooting. As shown earlier, the switch can help mitigate the issues that arise when a marginal device disrupts a loop or other N_Port devices. Brocade Fabric Watch allows each switch to continuously monitor fabric elements for irregular conditions. Fabric Watch can assist in rapidly identifying and escalating potential problems.This proactive management improves the overall availability of the SAN. Specific to troubleshooting marginal links, Fabric Watch can detect such failing port symptoms as excessive CRC errors and proactively send an SNMP alert. It is also possible to telnet into the switch and quickly analyze statistics to identify the marginal port. To minimize the impact of a marginal device in a loop, you can utilize QuickLoop zoning or Fabric Assist to compartmentalize various host/storage pairs. QuickLoop zoning or Fabric Assist prevents LIPs from propagating between QuickLoop zones. In some respects, QuickLoop zoning turns one loop into multiple virtual loops. In Figure 8.32, a LIP generated by Host A in zone qlZone1 due to a marginal port does not propagate to qlZone2 or qlZone3.Without QuickLoop zoning, a marginal port has the potential to limit or halt I/O for all devices connected to the switch!
339
140_SANs_08
340
8/14/01
5:26 PM
Page 340
Chapter 8 • SAN Troubleshooting
Figure 8.31 Marginal Link Fault Isolation START
Move suspected marginal port cable to another port on the switch
Do the errors stop or symptoms go away ?
No
Cable or Nx_port issue
Yes Try a new cable
Switch port or switch GBIC is marginal
Do the errors stop or symptoms go away ?
Replace GBIC on marginal port
Yes
BAD Cable
Run portLoopBack test on marginal port
Does portLoopback test fail
Yes BAD port Replace mother board
No
Replaced GBIC is BAD
No
Follow Nx_port (i.e. HBA, storage interface) troubleshooting procedures
140_SANs_08
8/14/01
5:26 PM
Page 341
SAN Troubleshooting • Chapter 8
Figure 8.32 QuickLoop Zoning Example qlZone2
qlZone1
c0
c1
Host A
b8
Host C
ba
e0
0,0
0,1
0,2
0,3
0,4
0,5
e1
e2
e3
Host B
b9 qlZone3
Overview of SilkWorm Port Error Statistics Additional SilkWorm port statistics can be obtained by executing the following telnet commands: ■
portShow
■
portStatsShow
Use portStatsShow for error statistics (such as CRC, encoding, bad End of Frame [EOF], etc.), and use portShow for link-level and LIP statistics (such as link failure, loss of sync, loss of signal, etc.).The portShow command offers similar statistics to portStatsShow. However, the statistics gathered by portShow
341
140_SANs_08
342
8/14/01
5:26 PM
Page 342
Chapter 8 • SAN Troubleshooting
are updated in software whenever a port interrupt is received, while the statistics for portStatsShow are updated in hardware registers as they occur.The significance in this difference is that many errors, such as CRC errors, could occur between interrupts.The hardware counters (portStatsShow) will capture these between interrupt errors, while the software counters (portShow) might not. Another difference between the two commands is that portShow provides LIP statistics and link statistics (link failure, loss of signal, loss of sync), while portStatsShow does not. A partial listing of relevant portShow statistics follows: ■
Lip_in Number of LIPs transmitted from the connected device to the switch port. Does not apply to F_Port.
■
Lip_out Number of LIPs transmitted from the switch port to the connected device. Does not apply to F_Port.
■
Lip_rx Type of LIP (F7, F8) last received by the switch from the connected device. Does not apply to F_Port.
Troubleshooting I/O Pauses I/O pauses happen, and both the SAN and edge device can and should tolerate such events.The term I/O pause is somewhat generic. An I/O pause can be as harsh as the powering down of a host or storage device while I/O is in transit, which will cause I/O to cease. Alternatively, it can be as lightweight as a portlevel RSCN, which might be a problem for only the most latency-sensitive of applications. Most HBAs currently pause I/O during RSCN processing; however, updated drivers are expected to minimize this effect. Relative to the SAN, fabric events can also cause a pause in I/O. A fabric event can be broken down into a change, such as a switch reboot, and the resultant activity to respond to that change. In the case of a switch reboot, not only are the devices connected to that switch affected, but also devices connected to the fabric—even if the fabric is resilient.This is because the fabric needs to reroute, which takes less than a second, and because all devices connected to the SAN that have registered for state change notification must process a global RSCN. Edge devices such as HBAs and storage devices should be tolerant of such pauses in I/O. It is possible to adjust the settings for these devices to accommodate longer or shorter delays in I/O when a SAN event occurs. RSCNs are normal and key to SAN operation. Several applications are very sensitive to latency and/or RSCNs, such as video-on-demand and applications that are evolving into the SAN model, such as
140_SANs_08
8/14/01
5:26 PM
Page 343
SAN Troubleshooting • Chapter 8
tape backup. High latencies and large numbers of RSCNs can adversely affect these applications. Storage vendors, switch vendors, application vendors, and HBA vendors are working with the standards bodies (T11) as well as modifying their product implementations to handle these types of exceptions.Table 8.6 lists common events that cause fabric rerouting and/or fabric RSCNs. Table 8.6 Fabric Events and Their Impact Event SwitchDisable Disabling a switch in the fabric will require the fabric to reconfigure and a new set of data path routes to be established for the resulting downsized fabric. SwitchEnable The corresponding mode to the disable. A new switch added to the fabric will result in new route calculations to allow for the added ports. E_Port connection/disconnection Adding or removing an ISL will cause a fabric RSCN. A zone update, which occurs when you execute a cfgEnable or cfgDisable command. Adding/removing a switch to/from the fabric.
Generate Global RSCN?
Will Result in Reroute?
Yes
Yes
Yes
Sometimes
Yes
Sometimes
Yes
No
Yes
Sometimes
Troubleshooting fabric events and their adverse impact on applications and the SAN is a complex process. If you suspect that a fabric event is adversely affecting your SAN, work with your switch supplier for resolution.
343
140_SANs_08
344
8/14/01
5:26 PM
Page 344
Chapter 8 • SAN Troubleshooting
Summary It can be helpful to think of the SAN as a virtual cable when it comes to troubleshooting, approaching the problem by breaking components down to a host, the SAN virtual cable, and the storage.To the operating system, the SAN provides a link to a disk, just as a traditional SCSI connection would.Troubleshooting a SAN is more challenging, but still has many things in common with the traditional storage troubleshooting process. Switches are logically positioned in the middle of the network between hosts and storage, and have visibility to both storage and hosts.This visibility into both sides of the storage network enables you to use switches to determine the cause of any malfunction in the SAN. SAN troubleshooting should begin in the center of the SAN and proceed outward. Once you know where to start troubleshooting, the next question is how to proceed. Start the troubleshooting process by gathering a preliminary set of data, and then analyze this data to identify where the problem resides: the host, the fabric, or the storage. Next, gather additional data from the appropriate area and focus in on the cause of the problem. A plethora of data is available from the switches, hosts, and storage. Many tools are available to the SAN troubleshooter. Several of these tools are switch commands. Other tools involve viewing the switch LEDs, host information, Fibre Channel analyzers, and diagnostics available on many storage arrays. It is rarely possible to use a single tool to successfully troubleshoot a problem. It is more common is to use several tools in concert. A fabric problem is a pervasive issue that can often affect more than one device.When a fabric issue is experienced in a resilient SAN, it might have no impact on SAN functionality, because the SAN redundancy compensates for the marginal situation. However these “soft” errors can cause degradation in the performance of the enterprise application and thus require immediate attention. Fabric issues are normally associated with large fabrics, which are defined as fabrics consisting of 10 or more switches and 100 or more edge devices. A host that is unable to access a SAN device is a more common issue.This type of issue is classified as a missing device. Use of the commands switchShow and nsShow can quickly reveal the cause of the missing device. Missing device issues are normally limited to a few devices. If more devices are involved, it is likely a fabric issue. The impact of a marginal port can be significant. For example, a large storage device might be accessed by potentially dozens of hosts.The marginal behavior of this storage device then has the potential to impact all devices that access this
140_SANs_08
8/14/01
5:26 PM
Page 345
SAN Troubleshooting • Chapter 8
storage port. A marginal link involves the connection between the switch and the edge device. Isolating the exact cause of a marginal link involves analyzing and testing many of the components that make up the link: switch port, switch GBIC, cable, edge device GBIC, and the edge device. I/O pauses do happen, and both the SAN and edge device can and should tolerate such events.The term I/O pause is somewhat generic. An I/O pause can be as severe as the powering down of a host or storage device while I/O is in transit, which will cause I/O to cease. Alternatively, it can be as lightweight as a port-level RSCN, which might be a problem for only the most latency-sensitive applications. Relative to the SAN, fabric events can also cause a pause in I/O. Calibrating your edge devices to handle I/O pauses and troubleshooting I/O pauses is a complex process.
Solutions Fast Track The Troubleshooting Approach: The SAN Is a Virtual Cable ; Use the SAN’s visibility to both storage and hosts to start your trouble-
shooting process. ; The switchShow, nsShow, nsAllShow, errShow, and topologyShow
commands are extremely informational and invaluable to the troubleshooting process. ; The UNIX format command or HBA vendor-supplied utilities are also
helpful in troubleshooting. ; When you start the troubleshooting process, determine if the issue is
fabric related or device related. A fabric-related issue impacts many devices, and a device issue impacts only a few devices.
Troubleshooting the Fabric ; A fabric issue impacts many devices. A logical switch outage, such as
segmentation or physical switch outage, can cause many devices to drop out of the fabric. Problems with ISL initialization are also considered fabric issues.
345
140_SANs_08
346
8/14/01
5:26 PM
Page 346
Chapter 8 • SAN Troubleshooting
; The quickest way to narrow your search of a fabric problem is to com-
pare your baseline SAN profile to your current SAN profile and investigate discrepancies. ; A SAN profile includes the number of devices per switch (nsShow),
number of devices in the fabric (nsAllShow), and number of switches in the fabric (topologyShow).The errShow and switchShow commands are also helpful in tracking down fabric issues. ; Some fabric issues are caused by a mismatch in fabric service timeout
variables and the edge device timeout settings. Careful analysis of both the fabric and the edge devices is necessary to resolve this complex issue.
Troubleshooting Devices that Cannot Be Seen ; The first thing to check is that the missing device is logically connected
to the SAN as indicated by switchShow output. ; Next, check to see that the device is present in the Name Server, using
the command nsShow. If the device is not in the Name Server, it is invisible to the other devices in the fabric. ; Other common causes of missing devices are zone conflicts or
marginal links.
Troubleshooting Marginal Links ; Use portErrShow to establish if there are a relatively high number of
errors, such as CRC errors. Look for a steadily increasing number of errors to confirm a marginal link. ; A marginal link can impact multiple devices. For example, a shared
storage device with a marginal link can cause communication problems with all devices that access that shared storage. ; A marginal link can be caused by any of the components that make up
the link: switch port, switch GBIC, cable, edge device GBIC, and the edge device.
140_SANs_08
8/14/01
5:26 PM
Page 347
SAN Troubleshooting • Chapter 8
Troubleshooting I/O Pauses ; I/O pauses happen, and both the SAN and edge device can and should
tolerate such events. ; An I/O pause can be as harsh as the powering down of a host or storage
device while I/O is in transit, which will cause I/O to cease. Alternatively, it might be as lightweight as a port-level RSCN, which might be a problem for only the most latency-sensitive applications. Relative to the SAN, fabric events can also cause a pause in I/O. ; Several applications, such as video-on-demand and applications that are
evolving into the SAN model, such as tape backup, are very sensitive to latency and/or RSCNs. High latencies and large numbers of RSCNs can adversely affect these applications. ; Storage vendors, switch vendors, application vendors, and HBA vendors
are working with the standards bodies (T11) as well as modifying their product implementations to handle these types of exceptions.
Frequently Asked Questions The following Frequently Asked Questions, answered by the authors of this book, are designed to both measure your understanding of the concepts presented in this chapter and to assist you with real-life implementation of these concepts. To have your questions about this chapter answered by the author, browse to www.syngress.com/solutions and click on the “Ask the Author” form.
Q: When I activate a zone change (cfgEnable), I notice a pause in I/O and several of my hosts log warnings.What causes this?
A: When you issue a zone change, an RSCN is delivered to any host in the fabric that registers to receive an RSCN.The pause you notice is the initiator responding to the RSCN, which involves the initiator querying the Name Server and resolving any changes to the fabric.
Q: If I exhaust my troubleshooting options and cannot resolve an issue after reading this chapter, what should my next step be?
347
140_SANs_08
348
8/14/01
5:26 PM
Page 348
Chapter 8 • SAN Troubleshooting
A: Contact your switch supplier and request support. Provide the information outlined earlier in this chapter. Of special importance is the supportShow, which is ideally captured while the problem is happening.
Q: How can I tell if my fabric is segmented? A: Normally, a segmented fabric will generate an error message on the switch that segments.You can view errors by issuing the command errShow.
Q: How come my device inconsistently connects to the switch as either an N_Port or an NL_Port ?
A: It is likely that there is a bug in the port initialization of either the edge device or the switch. A short-term solution is to configure a port for a specific topology. For example, configure a port as an FL_Port by using the command portcfgLport. Longer term, you should resolve this behavior by escalating the problem to your switch supplier and your edge device supplier.
Q: What is a quick way to reinitialize to clear a fault or re-enable a link? A: The commands portDisable and portEnable will cause a port to reinitialize and potentially clear a fault. Doing so will cause the edge device to register with the Name Server.
140_SANs_09
8/14/01
3:40 PM
Page 349
Chapter 9
SAN Implementation, Maintenance, and Management
Solutions in this chapter: ■
Installation Considerations
■
Automating Switch Administration Activities
■
Brocade Zoning Considerations
■
Validating Your Fabric
■
SAN Maintenance
Summary Solutions Fast Track Frequently Asked Questions
349
140_SANs_09
350
8/14/01
3:40 PM
Page 350
Chapter 9 • SAN Implementation, Maintenance, and Management
Introduction Once you have completed your SAN design, you can then focus on implementation, management, and maintenance.To arrive at a design requires a significant data-gathering effort during which you establish the requirements that shape your SAN: application drivers, availability, scalability, manageability, and performance.With these decisions made, you can then create a SAN architecture that meets your needs.The process of deploying a SAN is iterative as you build, test, and refine your original design. Once you have a SAN design, the next step is to implement, maintain, and manage the SAN. SAN implementation is the process of taking your design from paper to physical setup. Implementation is an ongoing activity that is very visible during the middle stages of your SAN’s lifecycle. In transitioning to a management and maintenance mode, you will periodically implement changes such as SAN expansion, fabric upgrades, and node movement. SAN management and maintenance activities are also reactive, such as replacing a failed switch or Gigabit Interface Converter (GBIC) optical module. This chapter is organized similarly to how you would set up and run a SAN. First, we discuss topics that require thought prior to implementation, such as zoning, cabling, and installation decisions.Then we present topics such as how to validate your fabric prior to transitioning to production. Finally, once you have your fabric up and running, we discuss topics like managing your SAN with automation and maintenance topics such as adding devices to your fabric and fabric upgrades. This chapter provides unique tips and ideas as you deploy and manage your SAN, with a focus on practical techniques and tools that require minimal dependencies.You can directly apply the processes in this chapter to your SAN today. There are a wide variety of SAN management packages that offer varying degrees of functionality for SAN management, maintenance, and implementation, such as VERITAS (SANPoint Control), Computer Associates (Unicenter TNG), BMC (PATROL), Sun (HighGround), Hewlett-Packard (OpenView Storage Area Manager), SANavigator (SANavigator), Prisa (Visual SAN), Micromuse (Netcool), and IBM Tivoli Storage Network Manager (TSNM). There are many choices for developing your own SAN management and maintenance infrastructure using a mix of commercial packages and your own scripts and processes.Throughout this chapter, we provide examples of how to automate certain SAN management activities.The scripts discussed are freely downloadable and available for your use from the book’s Web site, found at www.syngress.com/ solutions.The level at which you use these scripts and the process outlined in this
140_SANs_09
8/14/01
3:40 PM
Page 351
SAN Implementation, Maintenance, and Management • Chapter 9
chapter depends on your technical skill level, what SAN management software you deploy, and your company’s information technology practices.
Installation Considerations Several decisions and considerations regarding your SAN solution are necessary prior to installation. Upfront planning and review result in significant time savings.This section identifies the areas of SAN installation that require planning, and the upfront decisions that you need to make. For example, it might be difficult to install or maintain an Ethernet connection for a remote SAN. In-band management via Internet Protocol over Fibre Channel (IPFC) is an option that addresses this issue.When you install your SAN, you want to be sure that the switches are running the appropriate level of Brocade Fabric OS, which is not always the latest release! You need to know which version of Fabric OS to use prior to installation. Other installation considerations include setting switch parameters and verifying that you have the necessary licenses to operate your SAN in accordance with the design requirements.
How to Cable Your SAN for Ease of Operation Installation time is when you should plan your cabling and implement a cable layout scheme that is manageable, flexible, and maintainable. An effective cable management scheme should not only enable ease of maintenance, but also be aesthetically pleasing.While aesthetics might not seem like a necessary design principal, it turns out that cable plans that “look nice” also usually turn out to be the ones that are easier to manage. The Inter-Switch Link (ISL) cabling plans in this chapter are optimized to facilitate clean cable management, and this should be easily achievable as long as normal cable management practices are followed. In particular, the cables used for ISLs should be carefully labeled and bundled so that they cannot be mistaken for host or storage cables. A well-managed ISL cable layout is shown in Figure 9.1. While the layout depicted in Figure 9.2 might look fairly clean, it does have some potential problems. If a switch in the middle of this set were to fail, it would be difficult to replace it without shutting down the other three.This is because the ISLs from the top switches run in front of the lower switches.The configuration in Figure 9.1 clearly does not have this problem. Another problem with the configuration in Figure 9.2 is the fact that the switches are stacked on a shelf rather than being rack-mounted. Even if the ISLs were cleaned up, it would still be difficult to remove a switch from the bottom of the stack.
351
140_SANs_09
352
8/14/01
3:40 PM
Page 352
Chapter 9 • SAN Implementation, Maintenance, and Management
Figure 9.1 An ISL Cable Layout That Is Easy to Maintain
Figure 9.2 An ISL Cable Layout That Is Difficult to Maintain
Figure 9.3 shows much the same configuration of switches shown in Figure 9.2, with a recommended cable layout scheme that is easy to maintain.The Figure 9.3 switches are also racked.
NOTE Ensure that ISLs run in front of only the switches to which they are connected. This will allow the switches to be removed without downtime for the fabric.
140_SANs_09
8/14/01
3:40 PM
Page 353
SAN Implementation, Maintenance, and Management • Chapter 9
Figure 9.3 The Switches From Figure 9.2 Are Recabled to Enable Ease of Maintenance
It is not always possible to use cables that are “cut to length” for the ISLs. If you are using cables that are excessively long, it is desirable to take up the slack in some manner, as shown in Figure 9.4. Ideally, do this away from the switch to avoid clutter at the switch itself. However, be sure that you do not exceed the bend radius specification of the optical cable. Figure 9.4 Take Up Slack to Avoid Clutter
Figure 9.5 depicts a high-performance, 32-port configuration that uses six switches.The switches are mounted with cable management above, below, and on both sides. Management can also be used in between switches, if needed. Only four of those switches have any ports available for user wiring.The other two are used exclusively for ISLs. It is desirable to have all available user ports in one contiguous block to ease cabling of edge devices and simplify troubleshooting and monitoring. Figure 9.5 shows the ports available to the user (edge ports).
353
140_SANs_09
354
8/14/01
3:40 PM
Page 354
Chapter 9 • SAN Implementation, Maintenance, and Management
Figure 9.5 Six Switches Racked for Edge Wiring and ISL Wiring
User wiring goes this way
ISL wiring goes this way
6' x 19" Rack
Clearly, the ISL wiring can be bundled to the top, bottom, and right side of the rack and kept completely separate from the user wiring that would run to the left. The ISLs within this group should all be formed using 1 meter cables, if you are using 1.5 U switches (such as the SilkWorm 2250), or 2 meter cables, if you are using 2 U switches (such as the SilkWorm 2800).The length of the ISLs to the other groups of switches will vary greatly depending on rack configuration and should therefore be measured beforehand. Note that these cable lengths apply only to the group depicted. For different SANs, different cable lengths might be required. The ISLs used to interconnect the switches in these configurations are assumed to be semipermanent. It is useful to have these semipermanent ISLs colored differently from the host/storage/other ISL cables. Most multimode Fibre Channel cables are orange. Using gray, black, blue, or some other different color for the ISLs should help to differentiate between ISL cables and edge device cables.
Racking Considerations If you are employing a dual-fabric SAN architecture, it is important that the duality be employed throughout the SAN implementation as shown in the lefthand configuration of Figure 9.6. Deploying two fabrics that are part of a SAN solution within the same rack makes that rack a single point of failure.The odds
140_SANs_09
8/14/01
3:40 PM
Page 355
SAN Implementation, Maintenance, and Management • Chapter 9
of a rack falling over of its own accord are low. However, it is possible to picture a contract cable management worker on a ladder falling off and hitting a rack, or a leak spraying water into a rack.The concept of dual fabrics is to avoid a single point of failure. For high-availability fabrics, ensure that you have separate power circuits available, as shown in the right-hand configuration of Figure 9.6. For dual power supply switches, use separate circuits for the left and right power supplies. If you are using single power supply switches, it is still important to use separate circuits if your SAN is configured for high availability.This means connecting half of the switches in your fabric to one circuit in such a way that if these switches are powered off, the other half, which are connected to a different circuit, can still comprise a working fabric. Figure 9.6 Racking and Powering for High Availability Separate Circuits
Rack A, Circuit A
Rack B, Circuit B
Hosts
SAN A
SAN B
Storage
355
140_SANs_09
356
8/14/01
3:40 PM
Page 356
Chapter 9 • SAN Implementation, Maintenance, and Management
In-Band or Out-of-Band Management? For some situations, it is not possible or practical to dedicate an Ethernet connection for each switch. It is possible to manage Brocade switches via direct Ethernet connections or via IPFC.When using Ethernet connections, it is only necessary to configure the switch IP information and attach an Ethernet cable to each switch. When using IPFC, you can use a single Ethernet connection to bridge to the other switches via IPFC.To configure IPFC, it is necessary to configure the switches and in some cases also to configure the Host Bus Adapters (HBAs) to run IPFC.There are advantages and disadvantages to an Ethernet bridge to IPFC approach, as presented in Table 9.1.The flexibility does exist to do in-band management with a single Ethernet connection. If you can do out-of-band management with Ethernet connections to each switch, you will need to allocate an IP address, Ethernet port, and cable for each switch.There is also the option of doing full IPFC-based management, with an IPFC-capable HBA connected to a switch that can talk to all other switches in the fabric via IPFC. Table 9.1 Advantages and Disadvantages of Using IPFC to Manage Your SAN Advantages
Disadvantages
Fewer or no Ethernet connections
Single point of management failure—only one Ethernet path or IPFC path to fabric. If a switch goes down anywhere in the fabric or gets “switch disabled,” all management capabilities stop at that point—no Brocade WEB TOOLS and no telnet support. This is no different than management via the Ethernet ports. In that case, you would have the management station and all the switches connected to an Ethernet switch or hub. If the Ethernet cable goes bad, you cannot manage any switch: single point of failure. Static IP addresses—no Dynamic Host Configuration Protocol (DHCP) support. No easy gateways exist for routing IPFC like you have on Ethernet unless you piece one together using routed devices on a UNIX box or use some kind of routing software for Windows NT.
Fewer Ethernet hubs/power Remote management
140_SANs_09
8/14/01
3:40 PM
Page 357
SAN Implementation, Maintenance, and Management • Chapter 9
IPFC In-Band Guidelines Figure 9.7 depicts an IPFC in-band configuration.The management station, where the browser runs, does not need to have a Fibre Channel interface, or be IPFC-capable: it only needs an Ethernet connection. Only one of the switches in the fabric needs an Ethernet connection, which must be in the same subnet as the management station. However, this is not strictly necessary.You can configure the default gateway on the management switch and also add an appropriate static route on the management station and all routers between it and the management switch. This is a bit complex, and probably not worthwhile in many cases. In addition, with this configuration correctly implemented, it is also possible to telnet into every switch in the fabric from the management station. Figure 9.7 A Five-Switch IPFC In-Band Setup Management Station IP: 192.168.164.109 Subnet: 255.255.255.0 GW:192.168.164.1
Same Ethernet IP can be used on all in-band switches. Note: Ethernet IP cannot be [0.0.0.0] or None.
SW1
SW2
SW3
SW4
SW5
Management Switch IP: 192.168.164.28 FC_IP: 172.17.50.1 GW: 192.168.164.1 (Gateway Switch)
IP: 192.0.0.1 FC_IP: 172.17.50.2 GW: 172.17.50.1
IP: 192.0.0.1 FC_IP: 172.17.50.3 GW: 172.17.50.1
IP: 192.0.0.1 FC_IP: 172.17.50.4 GW: 172.17.50.1
IP: 192.0.0.1 FC_IP: 172.17.50.5 GW: 172.17.50.1
Gateway on in-band switches must point to the first switch's FC_IP address. All switches' FC_IP address must be in the same subnet. A Static Route must be entered from the management station pointing to the FC_IP Subnet.
357
140_SANs_09
358
8/14/01
3:40 PM
Page 358
Chapter 9 • SAN Implementation, Maintenance, and Management
A guide for setting up a five-switch SAN for in-band management follows. You can adapt this guide to fit your SAN environment. A summary of the configuration is listed in Table 9.2.This summary highlights the relationship between the IPFC and the Ethernet IP addresses: ■
All switches must have their Fibre Channel IP addresses on the same subnet.
■
The management station and switch Ethernet port must have their IP addresses on the same subnet.
■
The management station must have a static route to the IPFC subnet, or the default gateway pointing to the Fibre Channel IP address of the switch connected to Ethernet. Either solution will work (for example, on a Solaris machine, route add IPFC mask IPFC MASK IPADDR metric 1).
■
The in-band managed switches not connected to Ethernet must have their default gateway set to the IPFC address of the switch that is connected to Ethernet. In Figure 9.7, switches 2, 3, 4, and 5 have their default gateway set to [172.17.50.1].These switches must have their Ethernet IP addresses set to an address that is different from the Ethernet IP subnet specified on Switch 1 (SW1).The Ethernet IP address cannot be [0.0.0.0] or None. The Ethernet IP address can be the same as illustrated in Figure 9.7, as long as the switches are not connected to the IP network.
■
The gateway address on Switch 1 (the gateway switch) should be set to the default gateway on the network. However, this is not required if Switch 1 and the management station are on the same subnet.
Setting Switch Parameters Before the switches are cabled together, certain parameters should be set.These include the IP information and the switch name, which should be the same as the host name that maps to the switch’s IP address. Set the IP address and switch name of each switch to an appropriate and unique ID.The gateway and subnet mask might also need to be set. See your network administrator for appropriate values. If possible, have a contiguous block of addresses reserved for all Brocade switches. It might also be beneficial to keep the last octet of these addresses below 239. One popular way to administer Fibre Channel domain IDs is to have them match the last octet of the IP address. For example, switch 192.168.62.100 would get domain ID 100. Since the highest valid domain ID is 239, this scheme works only if the last octet of the IP address is 239 or lower.
172.17.50.1 172.17.50.2 172.17.50.3 172.17.50.4 172.17.50.5
192.168.164.28 192.0.0.1 192.0.0.1 192.0.0.1 192.0.0.1
Management Switch 1 Switch 2 Switch 3 Switch 4 Switch 5
255.255.255.0
192.168.164.109
IPFC Address
Management Station
Subnet Mask
Ethernet IP Address
172.17.50.1 172.17.50.1 172.17.50.1 172.17.50.1
Static Route: route add 172.17.50.0 mask 255.255.255.0 192.168.164.28 metric 1 Gateway Switch
3:40 PM
192.168.164.1
192.168.164.1
Default Gateway Notes
8/14/01
Node
Table 9.2 Five-Switch IPFC Configuration Detail
140_SANs_09 Page 359
359
140_SANs_09
360
8/14/01
3:40 PM
Page 360
Chapter 9 • SAN Implementation, Maintenance, and Management
Switch Naming Tips Having a well thought-out switch-naming convention enables easy identification of physical switches if a problem arises. Use a switch-naming convention that scales across the organization, keeping in mind that the SAN might start small but can be extended enterprise-wide over time. If you have to change a switch name, it is very easy to do—just execute the command switchName. Changing a switch-naming convention is more difficult, as you will most likely have to change all the switch names in the SAN affected by the naming convention change. For example, if you evolved your SAN from a four-switch mesh to an eight-switch core/edge topology, you might want to rename your switches with either the term core or edge embedded in the name to reflect the role of the switch. Consider using the following items when making up the switch name field: ■
Incorporate an ID for the site or building where the switch is located.
■
Add a component to identify the floor or room where the switch is located.
■
Use the switch topology function (such as core or edge).
■
Add a component that shows to which organization or project the switch belongs.
■
Include the rack ID in the name to further detail switch location.
■
Embed the switch type into the switch name (such as the SilkWorm 2800, 2250, or 2400).
■
If redundant fabrics are being used, select an ID for complementary fabrics.
Example: CORE1_A_B6_230_R5 = core Switch 1, fabric A, building 6, room 230, rack 5 Note that switch names can be up to 19 characters long, must begin with a letter or digit, and must consist of letters, digits, and underscore characters. Spaces are not allowed.
140_SANs_09
8/14/01
3:40 PM
Page 361
SAN Implementation, Maintenance, and Management • Chapter 9
To set these parameters, execute the following steps: 1. If the switch has a serial port, connect to it with a serial cable and log in as the administrator. If the switch has a control panel instead of a serial port, use the buttons on the panel according to the Brocade Fabric OS documentation to set the IP address, netmask, and gateway, and then telnet in and perform the rest of the configuration as documented here. 2. To set the switchName parameter, use the switchName command: switch:admin> switchName "switch1" Updating flash ... switch1:admin>
3. Type ipAddrSet. A menu will appear. Answer the questions appropriately. Note that step 3 is not necessary if you enter the IP address via the front panel: switch1:admin> ipAddrSet Ethernet IP Address []: 192.168.163.110 Ethernet Subnetmask [255.255.255.0]: 255.255.255.0 Fibre Channel IP Address [none]: Fibre Channel Subnetmask [none]: Gateway Address []: 192.168.163.1 switch1:admin>
4. Connect the switch to the Ethernet and ping the address to verify that it has been set correctly.
What Fabric OS Version Should I Use? Deciding which version of Fabric OS to use can be a challenging process, especially if your SAN consists of multiple vendor edge devices or switches.The most recent version of Fabric OS might not always be the best version to use. In some cases, you might experience conflicting Fabric OS requirements, with multiple vendors each specifying a different version of Fabric OS. Many switch suppliers extensively test their SAN products with Brocade switches in varying configurations.To support their products and Brocade switches, they require that you run a specific version of Fabric OS. One suggestion is to work with your switch supplier and your SAN vendors to identify if there is an intersection of supported Fabric
361
140_SANs_09
362
8/14/01
3:40 PM
Page 362
Chapter 9 • SAN Implementation, Maintenance, and Management
OS versions. For example, your switch supplier might support Fabric OS versions v2.2.2 and v2.4.1. If your storage vendor and HBA vendor support v2.2.2, your choice would be to install Fabric OS v2.2.2. In some cases, there might not exist an intersection of support requirements, at which point you might want to use the version of Fabric OS recommended by your switch supplier or negotiate a support agreement with your SAN vendors. Another determining factor for running a version of Fabric OS is availability of features or support. There currently exist two Fabric OS trees, the v1.x tree for the SilkWorm 1000 series of switches and the v2.x tree for the SilkWorm 2000 series of switches.The naming convention for v2.x Fabric OS is formatted as dM.m.fp_t...t, with each variable replaced by the information specified in Table 9.3.The Fabric OS version used for examples in this book is v2.4.1.c.The software major version is 2, the minor version is 4, the maintenance version is 1, and the patch version is c. Many features and enhancements have been added since Fabric OS v2.0.Table 9.4 lists summaries of these feature and enhancement additions.The information in Table 9.4 can help you determine which Fabric OS is right for you, should you have the option to choose.The key is to establish which version of Fabric OS to run as part of the installation process. Table 9.3 How to Decode a Fabric OS Version Variable
Meaning
d
Deployment indicator
M
m
f
Format
Definition
Lowercase Indicates the deployment target for the letter release. Does not indicate any functional changes. Normally the letter “v”. Software Number Indicates a release that incorporates major significant functional changes to the version software, as compared to releases with a lower software major version. Generally follows architectural changes in the core operating system or hardware. Software Number Indicates a release that incorporates minor added functionality within a major version software version. Software Number Indicates a maintenance release for a maintenance minor software version. Usually indiversion cates a release of bug fixes only. (Brocade attempts to prevent functional changes from occurring in software Continued
140_SANs_09
8/14/01
3:40 PM
Page 363
SAN Implementation, Maintenance, and Management • Chapter 9
Table 9.3 Continued Variable
Meaning
p
Software patch version
t…t
Special type (nonproduction release)
Format
Definition
maintenance versions.) Any functional changes that do occur are clearly documented. Letter Indicates a release that incorporates a patch within a minor software maintenance version; otherwise, functionally identical to the maintenance version. Each patch incorporates all preceding patches for the same maintenance version; for example, v2.0.2c would incorporate the patches implemented in both v2.0.2a and v2.0.2b. Letter(s), Indicates a special nonproduction build possibly (“N” in the following definitions is the followed iteration of the build): by number ■ An Alpha release is “..._alphaN” (abbreviated to “aN” in bug lists). For example, “_alpha3 or “a3”. ■ A Beta release is “..._betaN” (abbreviated to “bN” in bug lists). For example, “_beta1” or “b1”. ■ A release candidate is “..._rcN” (abbreviated to “rcN” in bug lists). For example, “_rc2” or “rc2”.
Table 9.4 v2.x Fabric OS History with Feature and Enhancement Additions Fabric OS Version
New Feature
v2.2.0
■ ■ ■ ■
Fabric Watch Extended Fabrics FA Management Information Base (MIB) FC-GS3 Management Server
Enhancements ■
■
■ ■
Fabric Shortest Path First (FSPF) routing failover enhancements WEB TOOLS enhancements — Switch Status — Dramatically better Fabric View Switch Beaconing Serial ID Gigabit Interface Converters (GBICs) Continued
363
140_SANs_09
364
8/14/01
3:40 PM
Page 364
Chapter 9 • SAN Implementation, Maintenance, and Management
Table 9.4 Continued Fabric OS Version
New Feature
v2.2.1
Enhancements E_Port Enable / Disable Simple Network Management Protocol (SNMP) Access Control Lists (ACLs) ■ Extended Fabrics — Configurable on any switch — License on portal switches ■ ■
v2.2.2
■
■
■
■
v2.3.0 ■
SilkWorm 6400 Support —Group Definition Phase I —ISL Topology Check / Monitor —SNMP Group Support —New GETS for groups Fabric Watch trap enhancements —Traps now include thresholds SCSI Enclosure Services (SES) enhancements —Config File Upload/ Download via SES —Fabric OS Image Upload/ Download via SES —SupportShow via SES Fabric Access API v1.0 Switch Side QuickLoop Fabric Assist Mode —Reengineering of QuickLoop from Hub Emulation to Virtual Loops —Industry-leading LIP isolation —Loop Hosts talk to Fabric Targets
■
■
■ ■ ■
Fabric Watch —Alarm Enable/Disable —Threshold Reset to Defaults WEB TOOLS —Faster to load and run —Support for QLFA zoning —Support for ED5000 IOP Mode —Many enhancements Management Server —FC-GS-3 Platform Support FA MIB v2.2 FC-GS3 Name Server —More GET calls —More Register calls Continued
140_SANs_09
8/14/01
3:40 PM
Page 365
SAN Implementation, Maintenance, and Management • Chapter 9
Table 9.4 Continued Fabric OS Version
New Feature ■
v2.4.0
■
McData ED5000 switch interoperability with these constraints —Limited to 31 switches in fabric —WWN-based zoning, no hardware-enforced zoning —SilkWorm 1000 not supported in ED5000 mode —Zoning management limitations —All switches in fabric must run 2.3+ED5000 mode —No QuickLoop Fabric Assist Mode in ED5000 mode gets —No Alias Server or Management Server New features for SilkWorm 6400 —Fabric Manager 1.0 —Group Definition Phase 2 —Based on Management Server technology —Permits group management operations to be done on one switch rather than all —Group SupportShow
Enhancements
365
140_SANs_09
366
8/14/01
3:40 PM
Page 366
Chapter 9 • SAN Implementation, Maintenance, and Management
Licenses All switches must have fabric capability if you want to interconnect these switches. Brocade WEB TOOLS, Fabric Watch, and Zoning licenses are also desirable, but not required to build a fabric.You will need a QuickLoop license if you intend to integrate private hosts into your fabric.The SilkWorm 20x0 and the 22x0 switches offer varied fabric capabilities: ■
2010/2210 = No fabric license; loop switches
■
2040/2240 = Entry-fabric license; minimal switch-to-switch connectivity (single ISL support)
■
2050/2250 = Full-fabric license; unlimited switch-to-switch connectivity (multiple ISL support)
You should check each switch to verify that you have the licenses necessary to build your SAN solution.The command licenseShow is used to determine what licenses are installed on your switch, as shown in Figure 9.8. Note that a single key can enable multiple features. If this is the case, you will not have a one-to-one mapping of features and a license key. If you do not have the appropriate licenses, you will need to contact your switch supplier to acquire the necessary licenses. When acquiring a new license, it is necessary to supply the switch World-Wide Name (WWN), which is available from the output of the switchShow command, and the switch serial number, which is available from the switch chassis. Figure 9.8 Use licenseShow to Determine What Licenses Are Installed on Your Switch core1:admin> licenseShow SRzy9Sz9zeTS0zAG: Web license bbSz9eQb9zccT0AQ: Zoning license RdzdSRcSyzSe0eTn: QuickLoop license cSczRScd9RdTd0SY: Fabric license
140_SANs_09
8/14/01
3:40 PM
Page 367
SAN Implementation, Maintenance, and Management • Chapter 9
Automating Switch Administration Activities If you have to perform SAN administration activity more than once, consider writing a script.You can use the Tcl/Tk-based Expect scripting language to interface with the switch. In the future, you will also have the option to use the Fabric OS Application Programming Interfaces (APIs) for automating switch management functions. At the writing of this book, the Fabric OS APIs are available only to Brocade partners, with plans to make these APIs available to all switch users in the future.We discuss the following topics in this section: ■
Fabric OS APIs
■
Expect scripting
Using Expect to interface with the switch is not as powerful or effective a solution as using the APIs. However, if you need to implement a solution now, Expect is a good choice. Because of the power that the APIs deliver, further discussion is warranted to assist you with your planning. As we discuss the subjects in this chapter, we provide examples of how to automate related functions by using Expect.You can freely download and use the scripts mentioned in this book by accessing the book’s Web site (www.syngress.com/solutions). Several examples of the types of switch management functions you might want to automate follow later in this chapter: ■
Download new firmware to all of the switches in your fabric.
■
Reboot all of your switches at once or in a sequence.
■
Automate zone changes.
■
Facilitate troubleshooting.
Fabric OS APIs The Fabric OS API is a programming interface to allow applications to access fabric information and to perform control operations. Access to the switch functions is based on IP access either through out-of-band Ethernet or via IPFC from a suitable HBA. Host-resident libraries and header files are required. Support for Solaris,Windows 2000, and Hewlett-Packard HP-UX currently exists. Application programs are compiled and linked to the library interfaces.The library uses Remote Procedure Call (RPC) over a TCP/IP connection to a
367
140_SANs_09
368
8/14/01
3:40 PM
Page 368
Chapter 9 • SAN Implementation, Maintenance, and Management
switch to get information and perform control operations. A Perl interface is planned. One of the benefits of using the APIs over scripts is that it simplifies complex tasks into simple commands; commands that would require many lines of scripting can potentially be a single command via the API.The API will be rolled out for end-user support; however, the initial support is to third-party management applications. The application provider typically distributes host libraries and headers.Target users are SAN management application providers with availability to all switch users planned.The following companies provide SAN management applications: ■
VERITAS (SANPoint Control)
■
Computer Associates (Unicenter TNG)
■
BMC Software (PATROL)
■
Sun Microsystems (HighGround)
■
Hewlett-Packard (OpenView Storage Area Manager)
■
SANavigator (SANavigator)
■
Prisa (Visual SAN)
■
Micromuse (Netcool)
■
IBM Tivoli (TSNM)
The Fabric OS API is intended to provide the following operations: ■
Discovery applications can quickly discover the fabric topology (switches, ports, and routes) and devices within the SAN.
■
Zoning provides full access to Brocade Zoning management facilities. A transaction model with rollback manages multi-application access to safeguard against concurrent access.
■
Switch and port management provides application control of individual switches and ports. Applications have access to Switch, Ports, Port Statistics (PortStats), and Port Errors (PortErrors) objects for in-depth information of critical SAN information. Obtain firmware versions from all switches in your company.
■
Device management provides access to node and device objects that provide information about the end points within the SAN.
■
Route management provides access to route control information to assist users in discovering and managing routes within the fabric.
140_SANs_09
8/14/01
3:40 PM
Page 369
SAN Implementation, Maintenance, and Management • Chapter 9
Expect Scripting Expect is a powerful tool for managing the switch in an automated fashion using telnet commands; it not only automates applications such as telnet, ftp, passwd, fsck, rlogin, and tip, but it is also used for testing them.The Expect home page (http://expect.nist.gov) is an excellent source of information on Expect, the foundation software required by Expect (Tcl,Tk) and Expect applications. Another Web resource for Expect is the Tcl Developer Xchange, found at www.scriptics.com. Expect is available for a variety of UNIX, Microsoft, and Macintosh environments.
A Switch Management Wrapper Using Expect As mentioned earlier, the scripts discussed in this book are available on the book’s Web site (www.syngress.com/solutions). Although these scripts are not coding works of art, they are a great foundation to build utilities for your switch management. A wrapper that allows you to execute a single command on a switch is provided as an example (Figure 9.9).The name of the script is run_sw_cmd. The script takes two arguments: the command you wish to execute and the name of the switch you want to execute the command on.This wrapper enables you to run a switch command in an automated fashion.The hard part of the script and the majority of lines for this program are focused on establishing a “connection” (lines 1 through 62). Once the connection is made, it is very easy to just issue a command to the switch, and it takes only two lines (lines 63 through 64) to do this.The script is somewhat primitive since you need to set the user and password information in the script. However, there is nothing to prevent modification of the script to enable user and password arguments to be passed into the script. Because only one telnet session with the switch is permitted at a time, you cannot run an Expect script on a switch with an active telnet session. If you do, the Expect script will not be able to gain a connection. Figure 9.9 An Expect Script Wrapper for the SilkWorm Switch Usage: 1
run_sw_cmd
#!/usr/local/bin/expect
2 3
# Author:
Chris Beauchamp, Brocade Communications
4
# Date:
06/01/01 Continued
369
140_SANs_09
370
8/14/01
3:40 PM
Page 370
Chapter 9 • SAN Implementation, Maintenance, and Management
Figure 9.9 Continued 5 6
proc telnetLogin {user passwd prompt} {
7
expect {
8
timeout
9
{puts "FAIL\nTelnet attempt for $user timed out\n"
10
return 1
11
}
12
eof
13
{puts "FAIL\nTelnet login prompt for $user never happened\n"
14
return 1
15
}
16
# this is the case where we connect with the switch
17
"login:"
18
}
19
send "$user\r"
20
expect "Password:"
21
send "$passwd\r"
22
expect $prompt
23
return "0"
24
}
25 26
#
27
# main
28
#
29 30
# bail out if not enough args supplied
31
if {$argc != 2} {
32
puts "\nincorrect number of arguments supplied"
33
puts "
34
puts "\nexiting ..."
35
exit
36
}
\nusage: $argv0 "
Continued
140_SANs_09
8/14/01
3:40 PM
Page 371
SAN Implementation, Maintenance, and Management • Chapter 9
Figure 9.9 Continued 37 38
set cmd [lindex $argv 0]
39
set switch [lindex $argv 1]
40 41
# change these values if you have different password or user # requirements
42
set spasswd password
43
set suser admin
44
set sprompt admin>
45 46
set timeout 60
47 48
puts "telneting to switch"
49
spawn telnet $switch
50
set sw_spid $spawn_id
51 52
# exit since it was not possible to connect to the switch
53
catch {telnetLogin $suser $spasswd $sprompt} code
54
if {$code != 0} {
55
puts "unable to access switch"
56
exit
57
}
58 59
puts "switching context to switch telnet"
60
set spawn_id $sw_spid
61 62
# send the command
63
send "$cmd\r"
64
expect $sprompt
65
puts "\n"
66
return 0
371
140_SANs_09
372
8/14/01
3:40 PM
Page 372
Chapter 9 • SAN Implementation, Maintenance, and Management
You can also modify this script to read from a file of switches so that you can execute the script on multiple switches. Alternatively, as shown in Figure 9.10, you can call the script from a UNIX shell script or Perl to obtain the Fabric OS version from all switches in the fabric. Figure 9.10 Integrating run_sw_cmd Expect Script with UNIX Shell Scripts sun1# foreach switch ( core1 core2 edge1 edge2 edge3 ) ? echo $switch ? run_sw_cmd version $switch | grep "Fabric OS:" ? end core1 Fabric OS:
a2.4.1c
core2 Fabric OS:
a2.4.1a
edge1 Fabric OS:
a2.4.1a
edge2 Fabric OS:
a2.4.1a
edge3 Fabric OS:
a2.4.1a
Brocade Zoning Considerations If you use switch-based zoning, you need to determine if you want to use hard or soft Brocade Zoning, and how to manage your zones. A related zoning topic that you also need to explore is where to zone.This section addresses these particular issues. Brocade Zoning, which is an optionally licensed product, enables you to logically group devices into virtual SANs. Zoning is used to set up barriers between different operating environments, to deploy logical fabric subsets by creating defined user groups, or to create test and/or maintain areas that are separate within the fabric. Zoning is an all-or-nothing operation: once a zone is enabled, all devices must be defined in a zone, or each device will exist in a zone consisting of just that device, and that device will be inaccessible to other devices in the fabric. In effect, this sets up an access by inclusion policy such that, by rule, a host or storage device is not permitted to participate in the fabric until it is positively included in at least one zone.With Brocade Zoning, you can define multiple zoning configurations. However, only one zoning configuration is active
140_SANs_09
8/14/01
3:40 PM
Page 373
SAN Implementation, Maintenance, and Management • Chapter 9
at one time. It is possible to rapidly change zone configurations by just issuing the cfgEnable command. One use of this capability is to facilitate policy-based management.This capability can be used in many ways. For example, a policy can be defined to provide access to the tape library to Windows NT hosts during the day for continuous backup, but migrate access to the UNIX hosts at the end of the day. Alternatively, you might want to zone systems based on organizational structure.
Where to Zone? It is possible to zone at various points in the SAN, such as the HBA or at the host level, and you might even decide not to use switch-based zoning at all.You might also want to use switch zoning in combination with other zoning methods, such as using the HBA or storage controller to accomplish zoning, as each might have a different level of granularity. As zoning is a component of security, a combination of zoning at different locations in the SAN can be viewed as an additional level of security. Many customers feel that you can never have enough security.To provide context, see Figure 9.11 for the various zoning methods and where these methods can be employed. Much discussion surrounds the subject of where to zone. Major characteristics of zoning solutions include the need or lack of need for host resident software, zone configuration control, the ability of zoning to ease SAN management, the ability to zone at a Logical Unit Number (LUN) level, and security. If you use HBA zoning or a host resident zoning package, you need to install and maintain this software on all hosts that are part of the fabric. If one host is not running the host resident software, your fabric is subject to illicit access or data corruption, as the fabric is unprotected without the resident software installed. Host resident or HBA zoning software is also subject to configuration changes at multiple points, making management a challenge. Storage-based zoning, host resident zoning, and HBA zoning normally are capable of LUN-level zoning, which is a lower level of granularity than SAN switches can currently achieve. Due to the inherent risk mentioned in zoning at this upper application layer, it is advisable to supplement this solution with switch-based zoning as well to prevent a newly attached device from accessing storage until it is properly configured. In this mode, the administrator configures zoning at both the host and the switch. Doing so prevents any potential inappropriate data access if the host is not configured properly. As mentioned earlier, the SilkWorm zoning today cannot zone to the LUN level.To do LUN-level zoning, you will need to choose an additional zoning method. If you have multiple storage and HBA providers, it might be necessary
373
140_SANs_09
374
8/14/01
3:40 PM
Page 374
Chapter 9 • SAN Implementation, Maintenance, and Management
Figure 9.11 Where Zoning Can Happen in the SAN
Host Zoning
HBA Zoning
Hosts
SAN A
SAN B
Switch Zoning
Storage Zoning Storage
to learn, manage, and implement multiple zoning applications.The remainder of this section focuses on switch-based zoning. At a minimum, you will likely want to use switch zoning for the following reasons: ■
SilkWorm switches offer hard zoning, which is the most secure zoning available in your SAN.
■
Switch zoning provides a single point of control—you need to manage only one zoning interface as opposed to multiple HBA, storage, and host zoning interfaces.
■
Switch zoning minimizes the impact devices have on each other by limiting fabric activity such as Registered State Change Notification (RSCN) to only those zone members affected by the RSCN or limiting broadcast frames.
■
Some SAN devices can support only a limited number of device connections.With zoning, you can enforce the number of devices that exist in a zone to align with the edge device connection limits.
140_SANs_09
8/14/01
3:40 PM
Page 375
SAN Implementation, Maintenance, and Management • Chapter 9
Hard Zoning or Soft Zoning? Current Brocade SilkWorm switches support both hardware- and software-based zoning. As there is not a setting to turn on one or the other, it is often a point of confusion for administrators in terms of which one is being used.The type of zoning you use depends on how the zones are defined. If you use a device WWN or an Arbitrated Loop Physical Address (AL_PA) to define a zone object, you are using soft zoning. If you use a device physical port number, in the form (domain, port), you are using hard zoning (Figure 9.12). Figure 9.12 Hard and Soft Zone Examples core1:admin> cfgshow Defined configuration: cfg:
hard
green; yellow
cfg:
soft
red; blue
zone:
blue
jbod1; jbod2; softhost2
zone:
green
hardhost1; hardarray1
zone:
red
softjbod1; softjbod2; softhost1
zone:
yellow
hardhost2; hardarray2
alias: hardarray1 2,0 alias: hardarray2 3,9
Soft Zone
alias: hardhost1 0,8 alias: hardhost2 1,1 alias: softhost1 10:00:00:20:42:d9:78:31 alias: softhost2 20:00:00:50:37:d2:75:50 alias: softjbod1 21:00:00:20:37:d9:77:46 alias: softjbod2 21:00:00:20:37:d9:77:47
Effective configuration: cfg:
hard
zone:
green
0,8
zone:
yellow
1,1
2,0
3,9
Hard Zone
375
140_SANs_09
376
8/14/01
3:40 PM
Page 376
Chapter 9 • SAN Implementation, Maintenance, and Management
The difference between hard and soft zoning is that hard zoning is enforced at the Name Server and the Application-Specific Integrated Circuit (ASIC). Soft zoning is enforced only at the Name Server.With hard zoning, each ASIC maintains a list of source port IDs that have permission to access any of the ports on that ASIC, and the ASIC hardware itself will actually block inappropriate frames from passing through it, dropping them if they attempt to talk outside their zones.Your choice of zoning also influences how you maintain and operate your SAN. When a device requests a list of nodes from the Name Server, this is analogous to calling a telephone directory service.When the Name Server responds to a request, it returns nodes that the requesting device is allowed to access based on zoning definitions.When you contact a telephone directory service, unlisted telephone numbers are not returned. However, if you know the unlisted party’s telephone number or randomly guess an unlisted telephone number, there is nothing to prevent you from calling the unlisted party’s telephone number.With hard zoning, even if the device is aware of and attempts to use an “unlisted” port ID, the hardware will prevent communications from happening. Some edge devices either cache port IDs or bypass the Name Server under certain circumstances and will attempt to communicate with another device even though that device is not in the Name Server. Normally, this type of behavior is the nature of the device. For example, an initiator might not respond to an RSCN by design. An RSCN is normally sent to tell the initiator that the zones have changed and some devices that were previously being accessed are no longer available. If a device does not respond to this RSCN, it will continue to access even addresses that have been removed from its view of the Name Server. A malicious initiator might start scanning addresses to discover “live” ports. Hard zoning prevents initiators from accessing devices under such circumstances. If you are using soft zoning, these types of accesses are not prevented. Figure 9.13 shows the difference in security between hard zoning and soft zoning. Note that with hard zoning, you have protections at the Name Server and at the port, as depicted by the padlock icons.
140_SANs_09
8/14/01
3:40 PM
Page 377
SAN Implementation, Maintenance, and Management • Chapter 9
Figure 9.13 Hard Zoning Is More Secure than Soft Zoning With Hard Zoning you have zoning enforcement at the hardware level and with the Name Server
Simple Name Server
With Soft Zoning you have zoning enforcement at the Name Server only
Simple Name Server
377
140_SANs_09
378
8/14/01
3:40 PM
Page 378
Chapter 9 • SAN Implementation, Maintenance, and Management
Hard Zoning and Soft Zoning Differences When you zone by WWN (soft zoning), you have the flexibility of physically moving that device anywhere within the fabric without redefining your zones. This is because the device WWN has no dependencies on physical connection. Currently with hard zoning, the zone definition is based on the physical location of the edge device. If you move that edge device, you need to modify your zone definitions, since the zone definition is no longer valid.When you replace a failed device with a new device, you will need to modify your zone data with the new device WWN if you are using soft zoning. It is not necessary to modify a zone definition when replacing a hard-zoned device, since the device’s physical location is not changing. Hard zones are easier to implement since you just need to know the switch domain and port number of the device you want to zone.When you use soft zoning, however, you need to obtain the device WWN and it is harder to visualize the relationship between zone definition and a physical device. When you use hard zones, it is easier to replicate the zoning environment, since the domain and port identifiers do not need to be changed.You might want to replicate a zone environment when you implement the second fabric of your dual-fabric solution. Replicating SAN environments using soft zones is not as easy since re-entry of the unique WWNs associated with each SAN node is required. Because domain IDs are subject to change, hard zoning definitions might need to be redefined when a domain ID changes.
Zone Management Zoning is a fabric-wide resource administered from any switch in the fabric, which automatically distributes itself to every switch in the fabric. Zoning administration can be managed via telnet commands,WEB TOOLS, or the Fabric OS API to any switch in the fabric.You can use each of these zone management interfaces standalone or in combination with each other. The fabric provides maximum redundancy and reliability, since each switch stores the zoning information locally and can distribute it to any switch added to the fabric. For large zoning configurations or frequent zone changes, it might be desirable to automate these operations. Downloading the zoning configuration into a text file for manipulation and maintenance might also be desired.While the zoning information is redundantly distributed throughout the fabric, you are encouraged to make at least one backup copy of your zoning configuration by using the command configUpload.The configUpload command saves not
140_SANs_09
8/14/01
3:40 PM
Page 379
SAN Implementation, Maintenance, and Management • Chapter 9
only the switch configuration, but also the zoning configuration information to a file located on a specified host. Note that to enable a zone configuration with configDownload, you need to first disable the switch (use the command switchDisable).
Scripting Zoning Operations You have the option to use scripting to automate certain zoning operations. For example, you can create a script to automatically change a zoning configuration by enabling a predefined zone configuration.You might want to do frequent zone changes to virtually move a tape drive to different zones in a fabric as you perform your backup. Scripting is also effective for changing zoning configurations based on policy. For example, in a disaster recovery scenario, your policy might dictate to disable noncritical access to the SAN so that production systems can take over the resources used by noncritical systems. By automating the zone change process, you speed up the zone changes and minimize the potential for human error. With multiple zoning configurations defined in your fabric, it is quite easy to switch between configurations by issuing the cfgEnable command. If you need to change configurations frequently or based on policy, you might consider writing a script to cfgEnable the appropriate configuration.The script would be very similar to the script shown in Figure 9.9 (run_sw_cmd). Another option to leverage scripting for your zoning operations is to automate your zone creation with a script. Such a script would also serve as a backup to the zone configuration running in your SAN.You can modify this script to add or delete zone objects.When you need to restore a zone or implement zone changes, just execute the script.The script flow is as follows: 1. Log in to the switch. 2. Clear the existing zone objects. 3. Create the zone objects. 4. Enable the desired configuration. Figure 9.14 is a code fragment of an Expect script that can be used to create or modify a zone configuration.This script is called make_zone.You need to modify the zone entries within the script.The script is based on the script run_sw_cmd (Figure 9.9) and is also available on the book’s Web site (www.syngress.com/solutions).The syntax might appear a bit awkward or confusing, since you need to “escape” the double quotes (“) with a backslash (\) so that the double quotes are passed to the switch and not interpreted by the Expect script.
379
140_SANs_09
380
8/14/01
3:40 PM
Page 380
Chapter 9 • SAN Implementation, Maintenance, and Management
Figure 9.14 A Zone Creation Expect Script Usage: make_zone
Switch login code . . .
# clear out the existing configuration send "cfgclear\r" expect $sprompt
# create your zoning objects send "alicreate \"jbod1\",\"21:00:00:20:37:d9:77:46\"\r" expect $sprompt send "alicreate \"jbod2\",\"21:00:00:20:37:d9:77:47\"\r" expect $sprompt send "zonecreate \"red\",\"jbod1;jbod2;0,0;0,1\"\r" expect $sprompt send "zonecreate \"blue\",\"jbod1;jbod2;1,0;1,1\"\r" expect $sprompt send "zonecreate \"green\",\"2,0;2,1;3,0;3,1\"\r" expect $sprompt send "cfgcreate \"colors1\",\"red;green\"\r" expect $sprompt send "cfgcreate \"colors2\",\"blue;green\"\r" expect $sprompt
# enable the desired configurations send "cfgenable \"colors1\"\r" expect $sprompt puts "\n" return 0
140_SANs_09
8/14/01
3:40 PM
Page 381
SAN Implementation, Maintenance, and Management • Chapter 9
Zoning Tips The Brocade Zoning manual extensively documents the use of zoning.The following list of tips will guide you through your zoning implementation as a supplement to the Brocade manuals: ■
Minimal unwanted interactions To minimize unwanted interactions between devices and to facilitate fault isolation, limit the number of HBAs/initiators in a zone to one.The exception is clustering applications where HBAs need to communicate with each other.
■
Heterogeneous environments To reduce challenges related to operating system interoperability, zones can be created that fence off different operating systems. If a shared target such as a tape drive is needed, an overlapping zone can be used while still protecting the different operating systems from each other.
■
Aliases Use aliases to define your zone members. If a zone member changes, you need only to update the alias versus potentially changing multiple zone definitions. Aliases also give meaningful names to a device, much in the same way an IP name gives a meaningful name to an IP address. Aliases can be used for either single devices or for a group of multiple devices. From a service perspective it is the best method for getting a textual list of what is attached to what port.
■
Addition of a new switch To avoid zone conflicts and fabric segmentation when a new switch joins a fabric, clear and save the zone on the new switch prior to that switch joining the fabric. Do this with the commands cfgClear, cfgDisable, and cfgSave. A new switch added to the fabric automatically inherits the active zoning configuration information in the fabric and immediately begins enforcement.
■
Node and Port WWN When a zone member is specified by Node Name, then all ports on that device are in the zone.When a zone member is specified by Port Name, only that single device port is in the zone. A device has one Node WWN and one or more Port WWN(s). For flexibility, consider using the Node WWN for your zoning entries if you must use soft zoning.
■
Zone changes When issuing the configDownload command to enable a given zoning configuration, you should insert the keyword
381
140_SANs_09
382
8/14/01
3:40 PM
Page 382
Chapter 9 • SAN Implementation, Maintenance, and Management
“clear:” into the file immediately before the zoning lines.This will ensure that the new zones take effect and that there is no segmentation. ■
Backup When you have finished your zoning implementation, make a backup of your zoning data by using the command configUpload.
Validating Your Fabric Prior to transitioning your fabric to production, it is important to validate that the SAN you have implemented is ready.The time to identify and correct any problems is during the validation of your fabric and prior to transitioning to production. Doing so involves establishing your SAN profile, which we discuss in Chapter 8,“SAN Troubleshooting.” After baselining your SAN profile, you need to inject faults into the fabric to verify that the fabric and the edge devices are capable of recovering.The next step involves generating an I/O load in the SAN that approximates various application I/O profiles. Finally, you will want to run an I/O load on your SAN while also doing fault injection to approximate a worstcase scenario—a failure in your SAN while your SAN is in production. After completing the validation phase, you can then hand off the SAN to production. In the next section, we cover baselining your SAN profile, fault injection, running in I/O load, and using I/O generators.
Baseline Your SAN Profile We discuss the SAN profile process in Chapter 8.You need a baseline of your SAN so that you can quickly determine if the testing you execute results in any discrepancies.To baseline a SAN you need to bring up the SAN and all edge devices and verify that the fabric is stable and all devices are present and accounted for. Using the commands nsShow, nsAllShow, topologyShow, and switchShow, verify that the number of switches and devices you expect to be present are indeed visible to the fabric. If there is a discrepancy, refer to Chapter 8 for troubleshooting guidance. Once you verify that the correct number of switches and devices are present in the fabric, update your SAN profile form. In addition to the SAN profile information identified for collection in Chapter 8, you also need to identify the ISL ports in your fabric.You can do this by reviewing the output of switchShow on each switch in the fabric. Figure 9.15 provides a graphical depiction of a fabric with a SAN profile example.
140_SANs_09
8/14/01
3:40 PM
Page 383
SAN Implementation, Maintenance, and Management • Chapter 9
Figure 9.15 SAN Profile Example (Profile) Total Name Server Entries -------------9
Number Switches -------5
core1 name server entries: isl ports: number of ISL ports:
2 0: 1: 2: 3: 4: 5: 6
core2 name server entries: isl ports: number of ISL ports:
0 0: 1: 2: 3: 4: 5: 6
edge1 name server entries: isl ports: number of ISL ports:
7 0: 1: 2: 3: 4
edge2 name server entries: isl ports: number of ISL ports:
0 0: 1: 2: 3: 4
edge3 name server entries: isl ports: number of ISL ports:
0 0: 1: 2: 3: 4
core1
edge1
core2
edge2
edge3
383
140_SANs_09
384
8/14/01
3:40 PM
Page 384
Chapter 9 • SAN Implementation, Maintenance, and Management
The data extracted from the SAN shown in Figure 9.15 was extracted via a shell script.This shell script in turn used the Expect script run_sw_cmd. For large fabrics where you need to repeatedly capture a SAN profile, a script like get_san_profile is a real time saver.This shell script is available on the book’s Web site (www.syngress.com/solutions).
Fault Injection Fault injection is the process of creating scenarios in the SAN that mimic potential faults. It is effective for uncovering marginal connections and malfunctioning devices. Fabric and edge devices should gracefully recover after a fault is injected. The process of fault injection and SAN verification is straightforward: 1. Capture a SAN profile baseline. 2. Inject a fault. 3. Compare the SAN profile baseline to a current SAN profile. 4. Check edge devices to verify that no devices have dropped off (are no longer visible to the hosts or switches). 5. If there are any unexpected discrepancies, go to Chapter 8 for troubleshooting guidance. Fault injection should involve both the fabric and the edge devices. Power cycling and resetting are typical fault injection activities for edge devices.You can simulate an edge device going offline and online by doing a portDisable/portEnable for a particular edge port. For the fabric, you have several fault injection activities from which to choose: ■
Reboot a switch or power cycle a switch.
■
Disable and enable a switch (switchDisable/switchEnable).
■
Disable and enable ISL ports (portDisable/portEnable).
An exhaustive testing of the fabric and all edge devices is not usually warranted. However, spot-checking is useful prior to transitioning to production. For edge devices, select one or two representative hosts and storage devices for the edge device fault injection. Power cycle and/or reset these devices. After the device recovers from being power cycled or reset, check the other edge devices to verify that no devices, except the device being reset, are dropped. A dropped device is a device previously visible to the edge device that is no longer accessible: for example, a disk device that was visible via the UNIX format command
140_SANs_09
8/14/01
3:40 PM
Page 385
SAN Implementation, Maintenance, and Management • Chapter 9
but is no longer visible via the format command after a fault injection is considered dropped. A dropped device is considered an error that requires further troubleshooting. For the fabric, select two or three switches for reboot, power cycle, and disable/enable fault injection. If you are using a core/edge architecture, one of the switches used for fault injection should be a core switch. For the ISL testing, choose three to five ISLs spread across multiple switches to disable and then enable. After each fault injection, capture a SAN profile. Compare this SAN profile to the baseline. Also check the edge devices to see that no devices drop out. If there are no discrepancies, the SAN passes the test.
Running an I/O Load It is very important to establish a stable SAN prior to moving to testing a SAN with an I/O load. If the SAN is not stable prior to load testing, it becomes difficult to establish a root cause if problems arise, since these problems can be stability-related and/or load-related. Some problems only arise under load, such as marginal links.When you do load testing in your SAN, you should run a variety of load types, focusing on a load that is most similar to the type of I/O you expect in your SAN. Once you are able to test the SAN with a variety of loads, try doing so with fault injection.The level of fault injection during load testing should be less intensive than the fault injection phase of testing. A suggested level of fault injection testing to perform while the SAN is under load is as follows: ■
Reboot a switch.
■
Reboot/reset one storage device and one host (you can simulate this situation by using the portDisable/portEnable command).
■
Disable and enable two or three ISLs that are each located on a different switch.
This is the final test. If you can run I/O in your SAN while doing fault injection and you are able to recover after the fault, it is time to move to production. Some faults might cause some I/O not to recover.This can happen because the host driver is unable to recover I/O under certain circumstances (for example, tape I/O), or because timeout values on the edge devices or in the SAN require tuning. Adjusting timeout settings in the SAN is a complex process that involves edge device and switch settings.Timeout settings are normally configured in the HBA or storage device. Refer to the HBA or storage device
385
140_SANs_09
386
8/14/01
3:40 PM
Page 386
Chapter 9 • SAN Implementation, Maintenance, and Management
configuration documentation for the specifics of how to make these changes and what the impact is of doing so.
Types of Load I/O can be classified in three ways: either a read or a write, random or sequential, and I/O size. A second-order description of I/O is whether the I/O is bandwidth-intensive. If the I/O is bandwidth-intensive, it is more meaningful to measure this I/O by throughput (in MB/sec). If the I/O is not bandwidth-intensive, you normally measure this I/O in terms of I/O Per Second (IOPS). Usually, I/O is a mix of reads and writes. However, some applications are very biased. For example, a video server I/O activity will normally be almost 100 percent reads. I/O can further be classified as random or sequential. Some examples of random I/O are an e-mail server or an Online Transaction Processing (OLTP) server. Sequential I/O is characteristic of decision support (such as data warehousing) or scientific modeling applications. The third characteristic of I/O is the size of the I/O. I/O sizes typically range from 2 KB to over 1 MB. Table 9.5 lists the application I/O profiles that establish the typical magnitude of application bandwidth consumption. For SAN design performance purposes, I/O is classified by bandwidth utilization: light, medium, and heavy. It is important to ultimately support test assumptions by gathering actual data when possible.You can gauge the type of I/O activity in your existing environment by using I/O measurement tools such as iostat (UNIX) or diskperf (Microsoft). Table 9.5 Application I/O Profiles Application
Bandwidth Utilization
Read / Write Mix
Typical Access
Typical I/O Size
OLTP, e-commerce, e-mail, UNIX File System (UFS), Common Internet File Services (CIFS) OLTP (raw)
Light
80% read / 20% write
Random
8 KB
Light
Random
2 KB–4 KB
Customer Response Management (CRM)
Light
80% 20% 85% 15%
Random
2 KB–4 KB
read / write read / write
Continued
140_SANs_09
8/14/01
3:40 PM
Page 387
SAN Implementation, Maintenance, and Management • Chapter 9
Table 9.5 Continued Application Decision support, high-performance computing, seismic, imaging, technical computing Video server SAN applications: serverless backup, snapshots, thirdparty copy
Bandwidth Utilization
Read / Write Mix
Typical Access
Medium to Heavy
90% read / 10% write
Sequential 16 KB– 128 KB
Heavy
98% read / 2% write Variable
Sequential > 64 KB
Heavy
Typical I/O Size
Sequential > 64 KB
I/O Generators You need an I/O generator to place a load in the SAN. Use the application I/O profiles outlined in Table 9.5 as a guide for providing input to your I/O generator.You should run a light-bandwidth and a heavy-bandwidth I/O load in your SAN for testing, tweaking one of these profiles to match your anticipated load profile. An even better approach is to use the target applications for load testing. However, doing so is often difficult or not possible.When deciding on which tool to use for your testing, focus on the tool’s ability to do the following: ■
Generate variable I/O sizes
■
Generate sequential and random I/O
■
Generate a mix of reads/writes
■
Generate one or more process(es)/thread(s) per disk or LUN
For Microsoft environments, Iometer is a popular and robust tool that meets these requirements. Iometer is available from Intel (http://developer.intel.com/ design/servers/devtools/iometer) and the tool is free. For UNIX environments, finding a tool for testing I/O is more challenging.There are also public domain tools available, such as IOzone (www.iozone.org). More flexible and powerful tools are not in the public domain and you need to obtain these tools directly from storage and host suppliers. vxbench is a very powerful and flexible tool available from VERITAS that can generate the loads outlined in Table 9.5.You will need to contact a VERITAS representative to obtain a copy of vxbench.
387
140_SANs_09
388
8/14/01
3:40 PM
Page 388
Chapter 9 • SAN Implementation, Maintenance, and Management
Tips for Generating an I/O Load ■
Many I/O utilities do both reads and writes. Writes can be destructive and can cause loss of data. Make sure that the storage you are writing to does not contain data that you need.
■
For UNIX operating systems, use the raw devices to achieve maximum bandwidth. If you use a “cooked” device (a device with a file system), you will incur CPU overhead related to the file system and anomalous results due to buffering.
■
To achieve maximum bandwidth, use sequential I/O with one or two threads/processes per device if you are using a JBOD. When using a RAID device, you need to use multiple threads, starting with two and doubling your thread/process count until you observe a reduction in bandwidth.
■
To achieve maximum IOPS, start with two threads/processes per device and double your thread/process count until you see a reduction in IOPS. The switch does not measure IOPS, so you will need to use an external tool such as iostat (UNIX) or diskperf (Microsoft) to establish your IOPS.
■
To observe the performance for a particular switch, issue the command portPerfShow from a telnet session on that switch.
Many storage and host suppliers have internally developed I/O testing tools for UNIX and Microsoft environments that are available if you ask for them. A reliable standby in the UNIX environment is the tool dd, which is available from many UNIX operating systems. With dd, it is possible to perform variable size I/O and to generate one or more processes. However, generating random I/O or a mix of reads and writes is difficult. An example of running a heavy-bandwidth load using the UNIX utility dd and the Microsoft environment tool Iometer follows (Figures 9.16 and 9.17). For dd, it is necessary to only use one process per disk to achieve maximal bandwidth.
140_SANs_09
8/14/01
3:40 PM
Page 389
SAN Implementation, Maintenance, and Management • Chapter 9
Figure 9.16 Generating a Heavy-Bandwidth Load Using a Shell Script and the UNIX Utility dd #! /bin/csh -f
# 100% reads to 3 disks dd if=/dev/rdsk/c1t0d0s2 of=/dev/null bs=64k count=2000 & dd if=/dev/rdsk/c1t1d0s2 of=/dev/null bs=64k count=2000 & dd if=/dev/rdsk/c1t2d0s2 of=/dev/null bs=64k count=2000 &
# 100% writes to 4 disks dd if=/dev/zero of=/dev/rdsk/c1t3d0s0 bs=64k count=2000 & dd if=/dev/zero of=/dev/rdsk/c1t4d0s0 bs=64k count=2000 & dd if=/dev/zero of=/dev/rdsk/c1t5d0s0 bs=64k count=2000 & dd if=/dev/zero of=/dev/rdsk/c1t6d0s0 bs=64k count=2000 &
Figure 9.17 Generating a Heavy-Bandwidth Load Using the Tool Iometer—All Seven Disks Are Selected
389
140_SANs_09
390
8/14/01
3:40 PM
Page 390
Chapter 9 • SAN Implementation, Maintenance, and Management
To generate maximum Fibre Channel bandwidth of 100 MB/sec you will need multiple disks, if using a JBOD, and possibly multiple storage arrays, if the array is not capable of sustaining 100 MB/sec (Figures 9.18 and 9.19). For example, if you run a single disk in a JBOD, you are never going to hit 100 MB/sec—you will need to run a number of drives to saturate a link. Figure 9.18 The Access Pattern Is 100 Percent Sequential Read with an I/O Size of 64 KB
Figure 9.19 The Bandwidth Is 95+ MB/sec—Approaching Fibre Channel Maximum
140_SANs_09
8/14/01
3:40 PM
Page 391
SAN Implementation, Maintenance, and Management • Chapter 9
SAN Maintenance There are multiple maintenance functions that you will need to perform throughout the life of your SAN. Some of these maintenance activities will be planned, and others will happen unexpectedly. If your SAN is designed to be resilient or redundant, unexpected maintenance should have minimal or no impact on your SAN operations. Failed or malfunctioning devices are normally the cause of unexpected maintenance.We provide a suggested process in this section for each maintenance activity discussed, including verification procedures. Other detailed SAN maintenance procedures are available from Brocade reference manuals as well as whitepapers.The focus of this section is to present an overview of the tasks and actions necessary to do SAN maintenance that is integrated with the methodologies and process defined in this book: ■
Maintaining a configuration log
■
Backing up and restoring a switch configuration
■
Bringing up a fabric
■
Expanding a fabric: merging fabrics, adding a switch, or replacing a switch
■
Upgrading your fabric
■
Replacing or adding an edge device in the fabric
The Configuration Log: Key Information to Gather and Maintain about Your SAN A configuration log is an up-to-date compilation of information and configuration details about your SAN. Enough information about your SAN should exist in your configuration log so you can recreate your SAN based on it.Whenever you make a change to your SAN, you should also update your configuration log. The configuration log can exist in hardcopy, softcopy, or both.There are some aspects of your configuration log that are not printable, such as firmware. If you maintain your configuration log in softcopy and this softcopy is stored in your SAN, you should also maintain a hardcopy or disaster backup, in case a disaster makes your SAN inaccessible. Having a softcopy of your configuration log enables rapid searches and easy updates of your SAN configuration data.You will need to access your log for a variety of reasons:
391
140_SANs_09
392
8/14/01
3:40 PM
Page 392
Chapter 9 • SAN Implementation, Maintenance, and Management ■
Disaster recovery
■
Troubleshooting
■
Recreating a switch whose configuration is destroyed
■
Planning SAN additions (for example, replacing your core switches with large core switches)
■
Modifying or expanding a SAN design
■
Recovering accidentally deleted licenses
■
Recovering or reconfiguring a zoning configuration
The key to a successful configuration log is diligent updates.Without the updates, your configuration log is not very useful. A suggested structure for your configuration log resembles the following (Figure 9.20 shows a Microsoft Windows Explorer view of an online configuration log): 1. Detailed diagrams of your SAN: A. Switch topology B. Host and storage connections 2. Firmware log of all SAN devices: A. A table for all devices, listing the device and related firmware B. A directory structure containing an entry for each device’s firmware 3. A log book where any additions, deletions, or modifications to your SAN are logged 4. A directory structure for the switches: A. A copy of each switch’s configuration is maintained. Use the command configUpload to save a switch’s configuration to a host. B. supportShow information for each switch, captured after the SAN is tested and verified. 5. Your SAN profile 6. A script directory for any scripts you create 7. A zoning directory for zoning configurations
140_SANs_09
8/14/01
3:40 PM
Page 393
SAN Implementation, Maintenance, and Management • Chapter 9
Figure 9.20 Explorer Screenshot of a Configuration Log
Backing Up and Restoring a Switch Configuration When you implement a new SAN, change your switch configuration, add a new switch to your SAN, or replace a switch in your SAN, you should create a backup of each switch’s configuration on a host.You do so with the command configUpload, which generates an editable text file. You can then restore a switch’s configuration with the command configDownload. The direction of upload and download is relative to the switch. Sometimes is it confusing whether you are backing up a configuration or restoring a configuration.To back up a configuration, you upload to a host.To restore a configuration, you download from a host. You can also create a standard configuration profile, suitable for configuring all switches in your SAN, by stripping out the switch-specific data from the switch configuration file. A switch configuration profile enables you to perform rapid configuration of your switch or switches for initial implementation, additions, or replacements.The alternative is a manual and time-consuming configuration of fabric parameters, SNMP information, and Fabric Watch information.The switch configuration file can also be used as a backup for your zoning configuration or as a zoning reconfiguration tool. If you ever lose a switch’s license information, you can recover this information from the configuration backup data.When replacing a switch, you can reference the switch configuration backup for IP address information.
393
140_SANs_09
394
8/14/01
3:40 PM
Page 394
Chapter 9 • SAN Implementation, Maintenance, and Management
The configuration file is written as three sections.The first section contains the switch boot parameters (otherwise known as the switch’s identity) and is preceded by the heading [Boot Parameters]. It has variables such as the switch’s name and IP address.This section corresponds to the first few lines of output of the configShow command.The second section contains general switch configuration variables, such as diagnostic settings, fabric configuration settings, Fabric Watch setting, license key information, and SNMP settings.This section corresponds to the output of the configShow command (after the first few lines), although there are more lines uploaded than shown by the command.The second section is preceded by the heading [Configuration].The third section contains the zoning configuration. It corresponds to the output of the cfgShow command and is preceded by the heading [Zoning]. To create a standard switch configuration profile, strip out the [Boot Parameters] and [Zoning] headings and section data, leaving the [Configuration] heading and data.You might also want to strip out QuickLoop data from the configuration section if you are not running QuickLoop on every switch.To restore or load a configuration, it is necessary to disable the switch (switchDisable) before downloading the configuration information.To load a configuration, use the command configDownload, specifying the standard profile or backup configuration file as the configuration file. It is also possible to use the configuration file as a zoning backup and as a zoning reconfiguration tool. By stripping out the boot parameters and configuration information from the configuration file and then downloading the resulting zoning information, you can restore or change a zone configuration. If you are using the configuration file for zoning configuration changes, you need to insert the keyword clear: in the configuration file to clear out the existing SAN zone configuration and prevent zone conflicts.
Bringing Up a Fabric There are several instances, such as power failure, initial bring up, or fabric-wide firmware upgrade, when you will need to bring up an entire fabric.The ideal order of bring up is as follows: 1. Bring up the fabric. 2. Bring up the storage devices. 3. Bring up the hosts.
140_SANs_09
8/14/01
3:40 PM
Page 395
SAN Implementation, Maintenance, and Management • Chapter 9
This order stems from the fact that the host must have visibility to the storage, especially during boot when devices are configured: for the host to have visibility to the storage, the storage and SAN must be online.You can bring up the storage first and then bring up the SAN. However, it is recommended that you power up the SAN first. Unfortunately, this order is difficult to implement. Powering off or disconnecting edge devices is frequently very time consuming or very difficult to schedule. A more likely scenario involves bringing the SAN down via a power cycle or a reboot of all switches in the fabric, and then bringing the fabric back up while edge devices are powered on and connected. An example of unplanned bring up is a power outage.When a power outage occurs, the order in which hosts, storage, and switches come online is variable.To bring up a fabric, use the following steps as a guideline: 1. Bring up the switches. Either power on the switches or issue the command reboot to all of the switches in the fabric. 2. Verify the fabric. Once the fabric is up, you need to verify that all edge devices and switches are present. Use the SAN profile to compare the previous baseline of your fabric switch count and device count to a current profile. If you see any discrepancies, follow the troubleshooting procedures detailed in Chapter 8. Use topologyShow to verify that all switches are online and use nsAllShow to verify that the correct number of devices are present in the fabric. Even if you are able to execute the ideal order of bring up (fabric, storage, host), it is still necessary to compare the baseline SAN profile to the current SAN profile, since it is possible that all edge devices did not come back online. Note that this is becoming less of an issue, especially with newer edge devices.
Expanding a Fabric: Merging Fabrics, Adding a Switch, or Replacing a Switch Merging two fabrics, replacing a switch, and adding a switch to a fabric are similar processes. It is important that the zoning configurations and fabric configuration parameters are consistent between the new switch or fabric and the existing fabric. Execute the following steps when adding a switch or switches to the fabric: 1. If necessary, update your SAN profile with the current state of the SAN. 2. Resolve any zone conflicts.
395
140_SANs_09
396
8/14/01
3:40 PM
Page 396
Chapter 9 • SAN Implementation, Maintenance, and Management
3. Resolve any switch configuration parameter conflicts and make any necessary switch-specific configuration changes such as port configuration changes, enabling QuickLoop, SNMP, Fabric Watch settings, and other configuration changes. If you have a standard switch configuration, you can download this configuration with the command configDownload. 4. Resolve any domain ID conflicts or connect a disabled/powered-down switch. 5. Verify that the new switch or switches are licensed consistently with the existing fabric licensing scheme. 6. Check the new switch’s or switches’ Fabric OS version and, if possible, make the Fabric OS version consistent for the whole SAN. 7. Verify that your SAN devices are minimally impacted by an RSCN. If your SAN devices have difficulty handling RSCNs or your applications are adversely impacted, consider stopping I/O on those devices. 8. Connect the new switch or switches to the existing SAN. 9. Enable or power up the new switches. 10. Connect your edge devices. 11. Capture a new SAN profile to verify that the correct number of edge devices and switches are present in the fabric. If there are any disparities, reference Chapter 8 for guidance on troubleshooting. Once the correct number of switches and edge devices are accounted for, create a baseline SAN profile for future reference. 12. Back up the configuration for the added switch or switches with the command configUpload. To avoid zone conflicts, it is simplest to clear out the zone configurations for the new switch or switches by executing the commands cfgClear, cfgDisable, and cfgSave on the switch(es) being added. If you are merging multiple fabrics, select one of the fabrics as the active fabric; add the zone entries from the nonactive fabrics to the active fabric zoning configuration; and then clear out the nonactive fabric switches’ zone information. Once you add the “blank” switches into the fabric, these blank switches will absorb the zoning configuration of the active fabric. Certain configuration parameters in the fabric must be the same.To review your switch configuration parameters, issue the command configShow. You
140_SANs_09
8/14/01
3:40 PM
Page 397
SAN Implementation, Maintenance, and Management • Chapter 9
must resolve any conflicts in fabric configuration parameters before adding a new switch to the fabric. For example, if there is a difference with the variable Error Detect Timeout Value (E_D_TOV), it is necessary to either change this setting on the new switches or the existing switches so that the value is consistent on all switches that are going to be part of the same fabric. You can compare fabric configurations from your new switch and the existing fabric by examining the output from the command configShow. You can upload your standard switch settings from a configuration file to ensure consistency of your switch configuration throughout the fabric.You can create and restore switch configurations by using the commands configUpload and configDownload. If a backup of your switch configuration is not current, execute a configUpload to capture a current backup of your switch configuration and license information.When merging fabrics or adding a new switch to a fabric, you need to check for domain ID conflicts and resolve these conflicts by changing one of the conflicting domain IDs. If you bring a disabled switch or powered-down switch into a fabric, you do not need to resolve domain ID conflicts, since the new switch will negotiate an acceptable domain ID. When a disabled or powered-down switch joins the fabric, if there is a domain ID conflict, the added switch will negotiate a new domain ID. If domain IDs change, verify your zone definitions to identify and correct any hard zones affected by the domain ID change. Recall that a domain and a port number define a hard zone. Also, some edge devices might have dependencies on a device port ID, which is a function of the domain ID. If the domain ID changes, it might be necessary to reboot your host or have your host rescan for devices.
NOTE What is a domain ID? The Fibre Channel specification Fabric Generic Requirements (FC-FG), available from the Technical Committee T11 of the National Committee for Information Technology Standards (NCITS) at www.t11.org, defines the concept of a domain as “the highest or most significant hierarchical level in the three-level addressing hierarchy.” A SilkWorm switch is considered a domain. The domain number uniquely identifies the switch in a fabric. Within a fabric, a domain is identified by an address ranging from 1 to 239 (domain ID). The range of allowed values varies depending on the switch model and other system settings. SilkWorm switches automatically assign domain IDs as part of the switch initialization process.
397
140_SANs_09
398
8/14/01
3:40 PM
Page 398
Chapter 9 • SAN Implementation, Maintenance, and Management
To maintain a full feature set across all switches in a single fabric, you should run the same version of Fabric OS on all switches in that fabric. Before adding a new switch, check the license information with the command licenseShow to verify that a consistent license set exists with the new switch and the existing fabric.When you add a new switch to a fabric, you should try to do so when I/O is quiescent.You can verify if any I/O is occurring in your fabric by issuing the command portPerfShow on each switch in your fabric. When you add a new switch or switches, there will be a pause in any active I/O as the fabric reconfigures and edge devices respond to RSCNs. If you successfully tested your fabric with fault injections while generating an I/O load, you will have an accurate idea about how your fabric and edge devices will respond to the new switch addition(s).
Upgrading Your Fabric We describe both processes in the following two sections. A hot upgrade (also called a rolling upgrade) requires that your edge devices be configured with redundant paths and software capable of managing path failover.With a hot upgrade, you reboot one switch at a time for the new firmware to take effect.With a cold upgrade, you reboot all of your switches at the same time.You would perform a hot upgrade if you were unable to take down an entire fabric. A cold upgrade should take your fabric down only for a few minutes as you reboot for the new firmware to take effect.
Issues Applicable to Both Hot and Cold Upgrades The actual process of downloading firmware (firmwareDownload) does not require you to take the switch down. For the new firmware to take effect, you do need to reboot your switch.Wait until all switches are running new firmware before configuring any new software features or zoning parameters. It is recommended that all switches be upgraded to the same firmware level, to support all features in the current fabric; however, rolling upgrades are possible and supported.When performing a rolling upgrade, note that the new functionality might not be available on the switches until all switches are running the new version of Fabric OS.
140_SANs_09
8/14/01
3:40 PM
Page 399
SAN Implementation, Maintenance, and Management • Chapter 9
NOTE To minimize the reboot process, use the fastboot command after the firmware download. This skips the Power-On Self-Test (POST) and goes right to loading code and bringing up the switch. The fastboot time is approximately 30 seconds compared to approximately two minutes if POST is run during a normal reboot.
Performing a Hot Fabric Upgrade 1. If necessary, update your SAN profile with the current state of the SAN. 2. Make sure the switch has redundant paths for devices attached to it. If possible, force the I/O path on the devices to fail over to a neighboring switch using software provided on those devices. 3. Verify that there is no traffic on the switch, using the perfShow telnet command if manual failover was possible in step 2. 4. Download the new firmware (firmwareDownload) onto the switch. 5. Reboot the switch for the firmware to take effect.When you take a switch down, it will cause the fabric to reconfigure and you will see a pause in any outstanding I/O as the fabric reconfigures.When the switch re-enters the fabric, you will also see a pause in I/O as the fabric reconfigures. 6. Capture a new SAN profile to verify that the correct number of edge devices and switches are present in the fabric. If there are any disparities, reference Chapter 8 for guidance on troubleshooting. 7. Re-enable the redundant paths for the attached devices. 8. Repeat steps 2 through 7 until all switches in the fabric have been upgraded. 9. Configure any new software features or zoning parameters. 10. Capture a new SAN profile to verify that the correct number of edge devices and switches are present in the fabric. If there are any disparities, reference Chapter 8 for guidance on troubleshooting and resolving the discrepancies. 11. Create a baseline SAN profile for future reference.
399
140_SANs_09
400
8/14/01
3:40 PM
Page 400
Chapter 9 • SAN Implementation, Maintenance, and Management
Performing a Cold Fabric Upgrade 1. If necessary, update your SAN profile with the current state of the SAN. 2.
Verify that there is no traffic on the switch, using the perfShow telnet command.
3. Download the new firmware (firmwareDownload) onto all switches in the fabric. 4. Reboot all switches in the fabric. 5. Capture a new SAN profile to verify that the correct number of edge devices and switches are present in the fabric. If there are any disparities, reference Chapter 8 for guidance on troubleshooting and resolving these discrepancies. 6. Configure any new software features or zoning parameters. 7. Capture a new SAN profile to verify that the correct number of edge devices and switches are present in the fabric. If there are any disparities, reference Chapter 8 for guidance on troubleshooting. 8. Create a baseline SAN profile for future reference.
How to Automate firmwareDownload Using the run_sw_cmd script as a base, you can automate the download of firmware to a switch.You can then call this script from a “for” loop to download to multiple switches.The following excerpt in Figure 9.21 is from a script called fw_download, which is available on the book’s Web site (www.syngress.com/ solutions). Figure 9.21 Excerpt from a Firmware Download Automation Expect Script # using ftp version of fw download -
note requires cleartext password
set cmd "firmwareDownload \"192.168.162.102\",\"root\",\"/book/v2.4.1c\",\"fooba r\"" # send the command send "$cmd\r" expect $sprompt
140_SANs_09
8/14/01
3:40 PM
Page 401
SAN Implementation, Maintenance, and Management • Chapter 9
Replacing or Adding an Edge Device in the Fabric When you add a new device to the fabric, you need to do some work ahead of time. Ideally, you will connect the new device to a switch with the highest concentration of devices that the new device is expected to access. Carefully consider placing new devices on the core of a core/edge fabric, since taking up core switch ports with edge devices limits your expansion capabilities. See Chapter 7 for more detail on device placement in the SAN. If you are using zoning, you need to update your zoning configuration with information from the new device. If you are doing AL_PA zoning or hard zoning and you replace a device on the same port, you do not need to make any zoning changes. Remember that an AL_PA is an arbitrated loop physical address. An AL_PA is an 8-bit address and is used to identify a private loop device (for example, ef ). An AL_PA might also be part of a public loop 24-bit address (for example, 0102ef ). AL_PAs are “soft,” meaning the address is dynamically assigned, or AL_PAs are “hard,” meaning the AL_PA is manually set.There might be some port configuration requirements such as QuickLoop, or a port might require “locking” to a G_Port (F_Port or E_Port) or L_Port. By default, Brocade SilkWorm ports are auto-configuring, the switch ports will match the topology of the edge port, and “locking” a port should not be necessary. Configuring a switch port for G_Port locks the switch port topology to a point-to-point connection, and configuring for an L_Port locks the switch port for a loop connection. If you are adding a new device, establish whether any port configuration dependencies exist and make any necessary changes. Adding a new device should have minimal impact on any active I/O. Devices within the new device’s zone might experience a pause in I/O as they respond to the RSCN notifying the SAN that there has been a change when the device is added.The following steps detail how to add or replace a device in the fabric: 1. Choose a location for the device in the SAN when making an addition. 2. If necessary, update your SAN profile with the current state of the SAN. 3. Update your zoning configuration to reflect the new edge devices prior to connection.This is extremely important as most devices perform discovery during login, so the zones need to be in place.
401
140_SANs_09
402
8/14/01
3:40 PM
Page 402
Chapter 9 • SAN Implementation, Maintenance, and Management
4. If necessary, make any port changes such as locking a port to specific topology or enabling QuickLoop. 5. Connect your new device or replace the existing device. If replacing a server, be sure to remove it from the zone, and if using software-based zoning, either reboot it or disconnect it from the SAN, as it might potentially still access mounted and known storage subsystems. 6. Capture a new SAN profile to verify that the correct number of edge devices and switches are present in the fabric. If there are any disparities, reference Chapter 8 for guidance on troubleshooting. For example, if there are 42 devices in the SAN before you connect your F_Port device, expect 43 devices to exist after you connect the device. 7. Create a baseline SAN profile for future reference.
140_SANs_09
8/14/01
3:40 PM
Page 403
SAN Implementation, Maintenance, and Management • Chapter 9
Summary Several decisions and considerations regarding your SAN solution are necessary prior to installation. Installation time is when to plan your cabling and to implement a cable layout scheme that is manageable, flexible, and maintainable. An effective cable management scheme not only enables ease of maintenance, but it is also aesthetically pleasing. Key areas to pay attention to when cabling are ISL cabling and taking up slack.When racking your switches, make sure to avoid single points of failure.This means separating redundant fabrics into different racks and powering resilient fabrics in such a way that a power failure does not cause the fabric to fail. For some situations, it is not possible or practical to dedicate an Ethernet connection for each switch. It is possible to manage Brocade switches via direct Ethernet connections or via IPFC. Using IPFC currently does have some disadvantages, such as a single point of management failure; however, this is no different from management via the Ethernet ports. In that case, you would have the management station and all the switches connected to an Ethernet switch or hub. If the Ethernet cable goes bad, you cannot manage any switch. Other areas that require thought prior to installation are a switch-naming convention, the version of Fabric OS to install, and ensuring that the correct licenses have been purchased.You should have a well thought-out switch-naming convention to enable easy identification of a physical switch in case a problem arises. Choose your Fabric OS version before you implement your SAN.The Fabric OS selection process might involve several of your SAN device providers, since Fabric OS support levels vary between SAN device vendors. If you have to do a SAN administration activity more than once, consider writing a script.You can use the Tcl/Tk-based Expect scripting language to interface with the switch. In the future, you will also have the option to use the Fabric OS APIs for automating switch management functions.With scripting you can automate many activities, such as downloading firmware, rebooting switches, automating zone changes, and facilitating troubleshooting. A wrapper script that enables the automation of SilkWorm switch commands is provided as an example and as a tool to use in your SAN.You can use this script as is, or modify it to suit your needs. If you use switch-based zoning, you need to determine if you want to use hard or soft Brocade Zoning and how to manage your zones. A related zoning topic that you also need to explore is where to zone. Zoning enables you to logically group devices into virtual SANs. Zoning is used to set up barriers between different operating environments, to deploy logical fabric subsets by creating
403
140_SANs_09
404
8/14/01
3:40 PM
Page 404
Chapter 9 • SAN Implementation, Maintenance, and Management
defined user groups, or to create test areas or maintenance areas that are separate within the fabric. Zoning is an all-or-nothing operation: once a zone is defined, all devices must be defined in a zone, or each device will exist in a zone consisting of just that device, and that device will be inaccessible to other devices in the fabric. It is possible to zone at various points in the SAN, such as the HBA or at the host level, and you might even decide not to use switch-based zoning at all.You might also want to use switch zoning in combination with other zoning methods such as using the HBA or storage controller to accomplish zoning. Zoning offers many advantages, such as a single point of management, hard zoning, and the ability to create virtual SANs. Hard zoning is the most secure zoning available because it is enforced at the Name Server and at the ASIC.There are differences between hard zoning and soft zoning, such as flexibility in moving zoned devices in the fabric and ease of implementation. A domain and a port number define a hard zone, while a device WWN defines soft zoning.You can script repetitive zoning operations. Scripting large zoning configurations might also be easier than using other zoning management interfaces, such as WEB TOOLS. Prior to transitioning your fabric to production, it is important to validate that the SAN you have implemented is ready.The time to identify and correct any problems is during the validation of your fabric and prior to transitioning to production. Doing so involves establishing your SAN profile and injecting faults into the fabric to verify that the fabric and the edge devices are capable of recovering. Generating an I/O load in the SAN that approximates various application I/O profiles is also an important part of the SAN validation process.To approximate a worst-case scenario—a failure in your SAN while your SAN is in production—you will want to run an I/O load on your SAN while also doing fault injection. Suggested fault injections involve rebooting or power cycling your switches and edge devices. I/O can be classified in three ways: a read or a write, random or sequential, and I/O size. A second-order description of I/O is whether the I/O is bandwidth-intensive.When you do load testing in your SAN, you should run a variety of load types, focusing on a load that is most similar to the type of I/O you expect in your SAN. Once you are able to test the SAN with a variety of loads, try doing so with fault injection.The level of fault injection during load testing should be less intensive than the fault injection phase of testing.There are several publicly available load generators, such as Iometer. You might also want to ask your storage or host provider for tools they use in their load testing.
140_SANs_09
8/14/01
3:40 PM
Page 405
SAN Implementation, Maintenance, and Management • Chapter 9
There are multiple maintenance functions that you will need to perform throughout the life of your SAN. A suggested process is provided for several maintenance activities: creating a configuration log, backing up and restoring a switch configuration, bringing up a fabric, expanding a fabric (merging fabrics, adding a switch, or replacing a switch), upgrading your fabric, and replacing or adding an edge device in the fabric. Other detailed SAN maintenance procedures are available from Brocade reference manuals as well as whitepapers. A diligently maintained configuration log can help you with disaster recovery, troubleshooting, recreating a switch whose configuration is destroyed, SAN design modifications or expansion, recovering accidentally deleted licenses, and recovering a zoning configuration.When you implement a new SAN, change your switch configuration, add a new switch to your SAN, or replace a switch in your SAN, you should create a backup of each switch’s configuration on a host using the command configUpload.
Solutions Fast Track Installation Considerations Ensure that ISLs run in front of only the switches to which they are
connected.This will allow the switches to be removed without downtime for the fabric. When racking your switches, be sure to avoid single points of failure.
This means separating redundant fabrics into different racks and powering resilient fabrics in such a way that a power failure does not cause the fabric to fail. Carefully consider solely using in-band management of your switch.
Consider using both in-band and out-of-band management. Have a well thought-out switch-naming convention to enable easy iden-
tification of a physical switch in case a problem arises. If you intend to implement a large fabric, work closely with your switch
supplier to identify a version of Fabric OS that supports the size of SAN you intend to build.
405
140_SANs_09
406
8/14/01
3:40 PM
Page 406
Chapter 9 • SAN Implementation, Maintenance, and Management
Automating Switch Administration Activities If you have to do a SAN administration activity more than once,
consider automating the activity with a script. Use Expect for automating your SAN administration activities today and
consider using Fabric OS APIs when they become available. Take the Expect script example (run_sw_cmd) and modify it for your
SAN administration activities. If you use Expect scripting, you need the supporting software. See the
following URL for Expect installation guidance: http://expect.nist.gov.
Brocade Zoning Considerations Determine whether you want to use hard or soft zoning prior to
implementing your zone scheme. Hard zoning is more secure than soft zoning, as hard zoning is enforced
at the Name Server and at the hardware level and will actually block inappropriate access. There are differences between hard zoning and soft zoning from a
maintenance perspective. For example, you need to update your zone information if you replace a device that is part of a soft zone. Consider using a script to build and maintain large zoning configura-
tions. Scripts can also be helpful to implement disaster recovery policies that are implemented in zoning.
Validating Your Fabric Baseline your fabric first so that you can quickly identify failures when
you validate your SAN. Use the SAN profile as your baseline. You can automate many validation activities, such as taking your SAN
profile and fault injection. The key to fault injection is to establish how your entire system behaves
when a fault is encountered.
140_SANs_09
8/14/01
3:40 PM
Page 407
SAN Implementation, Maintenance, and Management • Chapter 9
Key fabric fault injections include: switch power cycle, ISL
disable/enable, and switchDisable/switchEnable. Key edge device fault injections include: reset or power cycle the edge device.You can simulate this event by doing a portDisable/portEnable. Running a load in your SAN can shake out issues like marginal links.
The final test is to run a load on your SAN while you do fault injections. If your SAN is able to handle this test, you are ready for production. Pick a minimum of two load types for your SAN I/O testing: one that
approximates your SAN application load, and a load that is bandwidthintensive. Ask your host supplier, HBA supplier, or storage supplier for the tools
they use for I/O testing in a UNIX environment. If you want to use a Microsoft/Intel tool, Iometer is a good choice.
SAN Maintenance If possible, use the “cold” upgrade process for fabric upgrades. It takes
only a few minutes of downtime. When adding a switch to the fabric, clear out the zone information. It is simplest to power down or disable your switch prior to connecting
that switch to an existing SAN. Doing so will avoid domain ID conflicts. A diligently maintained SAN configuration log can help you with dis-
aster recovery, troubleshooting, recreating a switch whose configuration is destroyed, SAN design modification or expansion, recovering accidentally deleted licenses, and recovering a zoning configuration. Back up your switch configuration with the command configUpload
whenever you add or replace a switch. Maintaining a baseline SAN profile is essential to many SAN mainte-
nance activities. Make sure you know how to create and maintain a SAN maintenance profile. The act of loading firmware does not impact SAN operations.The
process of activating does impact SAN operations because a reboot is required.
407
140_SANs_09
408
8/14/01
3:40 PM
Page 408
Chapter 9 • SAN Implementation, Maintenance, and Management
Frequently Asked Questions The following Frequently Asked Questions, answered by the authors of this book, are designed to both measure your understanding of the concepts presented in this chapter and to assist you with real-life implementation of these concepts. To have your questions about this chapter answered by the author, browse to www.syngress.com/solutions and click on the “Ask the Author” form.
Q: Do I have to take my SAN down to perform a fabric upgrade? A: No. If you have your SAN configured such that all edge devices have dual paths, it is possible to perform a “hot” upgrade.
Q: When I merged my fabrics, several disks were no longer accessible from the hosts.What happened?
A: If you were using hard zoning, it is possible that the domain IDs of one of the switches you merged changed.This change in domain ID might have invalidated some of your zones.
Q: Are rogue hosts a real threat to soft zoning? A: Hosts that intentionally bypass the Name Server are as likely a threat to security as a device that caches a Name Server entry or does not respond to RSCNs.
Q: Why do I have to use a complex scripting language to manage my SAN? A: You do not have to write any scripts to manage your SAN.WEB TOOLS and other commercially available SAN management software is available to perform a variety of SAN management tasks. Scripting is for users who prefer the flexibility and power that scripting enbles for their SAN management tasks.
Q: When will the Brocade APIs be available for end-user use? A: Check with your switch supplier.The current target date is planned for the end of 2001.
140_SANs_AppFT
8/14/01
3:42 PM
Page 409
Appendix
Building SANs with Brocade Fabric Switches Fast Track
This Appendix will provide you with a quick, yet comprehensive, review of the most important concepts covered in this book.
409
140_SANs_AppFT
410
8/14/01
3:42 PM
Page 410
Appendix • Building SANs with Brocade Fabric Switches Fast Track
❖ Chapter 1: Introduction to SANs
Overview of SANs
SAN technology evolved from direct-attach interconnects like Small Computer Systems Interface (SCSI).
Fibre Channel supports SCSI, Internet Protocol (IP), and the Fibre Channel Virtual Interface (FC-VI) Protocol.
The distance between Fibre Channel nodes can be as much as 10 km.
Fibre Channel supports copper, multimode optical, and single-mode optical media.
SAN technology has moved from Fibre Channel Arbitrated Loop to full Fibre Channel switch fabric.
Taming the Storage Monster
Data storage needs are increasing rapidly.
Requirements due to databases, e-mail, multimedia, and the Internet have dramatically increased the required amount of storage for data.
Disk farms, storage arrays, and storage consolidation are the keys to solving the storage problem.
Benefits of Building a SAN
Fibre Channel is ideal for supporting high-availability configurations and business-critical back-end operations, due to the ability to set up redundant networks and clusters.
SAN technology allows for storage consolidation and data pooling for more efficient use of storage resources.
Backup windows are shrinking, and backup traffic on the LAN can be easily reduced by using a SAN to reduce network congestion due to backup.
Block-level, high-speed access through SCSI-Fibre Channel Protocol (FCP) can accelerate data access between storage and hosts, and can free up host resources that would be occupied serving files and data through IP.
140_SANs_AppFT
8/14/01
3:42 PM
Page 411
Building SANs with Brocade Fabric Switches Fast Track • Appendix
Chapter 1 Continued
Cluster protocol access through FC-VI frees up CPU cycles in hosts and enables clustered database operations.
One of the major advantages of SAN technology is its long-distance capability for disaster tolerance.
When to Deploy a SAN
The most important part of determining whether to deploy a SAN is to focus on the actual business application that will be served with the SAN deployment.
Speed and bandwidth requirements determine if the technology is right for the application. Compared with other technologies, such as IP-based file sharing and Network Attached Storage (NAS), the Fibre Channel protocol provides for more usable bandwidth and faster data transfer.
A SAN is ideal for block-level access to shared storage.
Fibre Channel works well for centralized access to storage arrays, redundant connections, clustered configurations, and disaster tolerance.
Steps to a Successful SAN Deployment
Data collection Evaluate the goals of the deployment to determine options in achieving high availability, redundancy, fault tolerance, data consolidation, cost reduction, and so forth.
Data analysis Investigate the hardware and software options that support those goals.
Architecture development Design and install a SAN testbed to set up configuration and components. Select the software and hardware carefully to avoid any interoperability problems.
Testing the prototype Test the configuration for interoperability, functionality, error handling, and fault tolerance.
Transition existing hardware in a controlled release to production Stage the deployment by rolling out the setup gradually, making changes on a limited basis to minimize risk.
411
140_SANs_AppFT
412
8/14/01
3:42 PM
Page 412
Appendix • Building SANs with Brocade Fabric Switches Fast Track
❖ Chapter 2: Fibre Channel Basics
The Architecture of SANs
A Fibre Channel SAN provides the advantages of increased speed, reliability, and scalability.
Fibre Channel presently transmits at 1062.5 Gbit/sec over single- and multimode optical and copper cabling.
A SAN implemented using the Fibre Channel protocol incorporates the benefits of a channeled connection and a network.
A SAN is constructed from three primary types of elements: initiating devices, switches, and target devices.
A target device is a storage device on a SAN. Device enclosures like tapes, JBODs, or RAIDs are the most common type of target device.
An initiating device is a device that actively seeks out and interacts with target devices on the SAN.
Switches create the foundation of the Fibre Channel SAN. A group of interconnected switches is called a fabric.
Fibre Channel Protocol Basics
Fibre Channel is primarily used to transport the SCSI and IP protocols.
Devices are identified by an 8-bit Arbitrated Loop Physical Address (AL_PA) in an arbitrated loop topology, and a 24-bit address for switched fabric connections.
Frames start with a primitive Start Of Frame (SOF) and end with an End Of Frame (EOF) primitive.
There are five Fibre Channel layers, designated FC-0 through FC-4.
The FC-0 layer is the physical media layer and includes the media selection and connectors.
The FC-1 layer is the signal encoding and decoding layer.The FC-1 layer uses 8b/10b encoding.
The FC-2 layer is the Fibre Channel protocol layer.
140_SANs_AppFT
8/14/01
3:42 PM
Page 413
Building SANs with Brocade Fabric Switches Fast Track • Appendix
Chapter 2 Continued
The FC-3 layer is the Fibre Channel common services layer.The services are servers in a Fibre Channel fabric that manage connections between devices connected remotely through the switched fabric.
The FC-4 layer is the Fibre Channel ULP mappings layer.
Classes of Service
Classes of service specify what mechanisms are required for transmission of different types of data.
Class 1—Acknowledged connection-oriented service.
Class 2—Acknowledged connectionless service.
Class 3—Unacknowledged connectionless service.
Class 4—Fractional bandwidth connection-oriented service.
Class F—Used for inter-switch communication.
Storage Network Topologies
There are three primary types of topologies in Fibre Channel: point-topoint, arbitrated loop, and switched fabric (also called point-to-point).
The primary use of the point-to-point topology is to connect devices directly to switches or other bridge devices.
The arbitrated loop topology allows up to 127 devices in a ring formation to share the bandwidth of a single line without a switch.
Fabrics allow thousands of devices to be interconnected.
Switches have three types of ports. FL_Ports are fabric loop ports that attach arbitrated loops to the fabric. F_Ports are fabric ports that connect single devices to the fabric in a point-to-point topology. E_Ports connect a switch to another switch.
Fabric-attached devices have a three-part address.The first segment indicates the physical switch, the second part indicates the physical port, and the last part is the arbitrated loop address in a loop device or 0x00 for a fabric device.
413
140_SANs_AppFT
414
8/14/01
3:42 PM
Page 414
Appendix • Building SANs with Brocade Fabric Switches Fast Track
Chapter 2 Continued
Fabric Services
Switches exchange information in their servers so that all individual switch servers contain the same information.This creates distributed servers.
The fabric port is used to log a device into the fabric.The response frame from login assigns the device its 24-bit address.
The Name Server is used as a database to register and store information about all devices on the fabric.
The Fabric Controller at well-known address 0xFFFFFD provides state change notification service to registered nodes. State change notification is a service that notifies devices when a change in fabric topology occurs.
The Management Server provides information about the fabric without stipulation as to zone.
❖ Chapter 3: SAN Components and Equipment
Overview of Fibre Channel Equipment
Understanding the features of your Fibre Channel equipment is key when building a robust infrastructure.
A Fibre Channel network is comprised of cabling, GBICs, hubs, switches, HBAs, and routers.
Fibre Channel shares much of the same terminology as Ethernet networking, but the functionality of similarly named equipment is not necessarily identical.
Cabling and GBICs
Copper cabling is almost always terminated with either an HSSDC or DB-9 male connector.
Multimode optical fiber is terminated using a variety of optical connectors, including SC, LC, and MT-RJ.
Single-mode fiber is the most expensive media type, but preferable for long distances.
140_SANs_AppFT
8/14/01
3:42 PM
Page 415
Building SANs with Brocade Fabric Switches Fast Track • Appendix
Chapter 3 Continued
Single-mode fiber, because of its small diameter (9 µm), has the highest transmission speed potential.
Copper cabling is available in two types: active and passive. Active copper lines provide twice the distance of passive copper lines.
The HSSDC connector was specifically designed as a Gigabit copper connector, improving density and performance over the DB-9 style connector.
GBICs are removable transceivers used in all types of Fibre Channel devices, including switches, hubs, and HBAs.
GBICs offer the option of interfacing with almost all types of connectors.
A Media Interface Adapter (MIA) converts DB-9 copper connectors to optical SC connectors.
Using Hubs
Hubs serve as a very basic level for connecting different ports in a network together.
Hubs can connect up to 127 devices together in an FC-AL loop.
Simple hubs contain no intelligence, just electrical connections.
Managed hubs provide a level of error tolerance and management features.
Managed hubs provide LIP isolation, automatic port bypass, signal retiming, and management interfaces.
Fibre Channel LIPs can be a major source of problems in arbitrated loop configurations.
To avoid an earlier generation of problems due to loop architecture, most people are moving to switched fabric devices instead.
Using Switches and Fibre Channel Fabrics
Switches are classified into three categories: entry-level, scalable fabric, and core fabric switches.
Entry-level switches are focused on small workgroups of 8 to 16 ports, usually are geared toward low cost, and deliver limited scalability and management. Fabric switches provide the capability to cascade switches together to create larger fabrics.
415
140_SANs_AppFT
416
8/14/01
3:42 PM
Page 416
Appendix • Building SANs with Brocade Fabric Switches Fast Track
Chapter 3 Continued
A core fabric switch is designed for interconnecting multiple edge switches to form multihundred-port SANs.
HBAs are used to connect servers to the network.They map SCSI commands in the operating system to Fibre Channel frames on the network. HBAs range from low-end, loop-only devices to high-end, fabric multipathing adapters.
Major protocols supported by HBAs are SCSI-FCP for storage, IPFC for networking, and VIFC for clustering.
HBAs either support 1 Gbit/sec or 2 Gbit/sec speeds, with current generation cards supporting 1 Gbit/sec, and emerging cards supporting both.
HBAs can be found in single one-port configurations or dual-port adapters for higher density.
LUN masking enables control of access to devices in the network from the HBA.
Persistent binding is the mapping of a Fibre Channel device into an operating system at a specific device location.
Dynamic discovery is the capability to dynamically add and remove drives from your system without reboot.
HBA API support is an important feature that allows management of your HBA by SAN management software.
Remote booting is the use of an HBA to boot an operating system image across the SAN and is used to dynamically change hosts and enable ease of disaster recovery.
Connecting Legacy Devices into Your SAN
Α Fibre Channel router, which is also known as a bridge, allows legacy parallel SCSI devices to attach to your Fibre Channel SAN.
A Fibre Channel router plugs into Fibre Channel on one side and a SCSI bus on the other.
Frames are translated from SCSI-FCP to parallel SCSI bus signals through routers.
140_SANs_AppFT
8/14/01
3:42 PM
Page 417
Building SANs with Brocade Fabric Switches Fast Track • Appendix
Chapter 3 Continued
Routers provide many different features, including different numbers of SCSI buses and different support for parallel SCSI protocols and termination.
Advanced features include selective LUN presentation, extended copy support, and various management interfaces.
Selective LUN presentation is the capability of a router to mask the presence of devices to different hosts in the network and allow for better security and control over resources.
Extended copy support (third-party copy) allows software to directly back up data on the SAN, saving CPU and network traffic.
Available management interfaces include telnet, SNMP, Ethernet, and serial ports.
Bridging and Routing to IP Networks and Beyond
Fibre Channel-to-DWDM technology multiplexes Fibre Channel signals onto higher bandwidth fiber for transmission over MAN distances (up to 100 km).
Use of DWDM is transparent to Fibre Channel switches, except for buffer settings.
It is necessary to increase buffer credit settings to handle the long distances/ delays involved in MANs.
Fibre Channel can also be transported across IP networks like ATM and Gigabit Ethernet.
FC_IP (not to be confused with IPFC) encapsulates Fibre Channel frames in the IP protocol and can be used for remote backup and extending SAN distances.
Fibre Channel Storage
Fibre Channel storage is important as the core of the data storage on your network.
Fibre Channel storage ranges from simple JBOD devices to multiterabyte storage arrays.
417
140_SANs_AppFT
418
8/14/01
3:42 PM
Page 418
Appendix • Building SANs with Brocade Fabric Switches Fast Track
Chapter 3 Continued
A JBOD is a cabinet of independent disks, all connected into the Fibre Channel network in a loop.
Hosts individually address disks in a JBOD.
RAID arrays provide additional protection and performance to your storage.
Different RAID levels are appropriate for different applications.
High-end storage arrays add support for multiple terabytes of data. Other types of connections include parallel SCSI, ESCON, and FICON.
High-end arrays also generally include a large amount of cache, which is used to speed up data access.
Selective LUN presentation is the ability of high-end storage to control access by hosts to data and to ensure data integrity.
LUN export across multiple ports is used for redundancy and high availability, but requires dynamic multipathing software or drivers to work.
Snapshot backup volumes are used to enable backup on live databases and data images.
❖ Chapter 4: Overview of Brocade
SilkWorm Switches and Features Selecting the Right Switch
Identify your requirements for availability, port density, functionality, and cost.
Decide whether you need an arbitrated loop or full-fabric environment.
Learn which switch functions best satisfy your requirements.
Consider what strategic direction you want to take, and whether your current switches will scale easily to meet your needs.
Understanding the Brocade Fabric OS
Fabric OS is the operating system for all Brocade SilkWorm switches.
Key functions include auto-discovery, in-order frame delivery, zoning, and others.
140_SANs_AppFT
8/14/01
3:42 PM
Page 419
Building SANs with Brocade Fabric Switches Fast Track • Appendix
Chapter 4 Continued
Fabric OS provides the capability to work with other storage management applications.
Using Optional Brocade Features
You can use Brocade Zoning to isolate devices into separate, virtual SANs.
Zoning is ideal for multiple customer environments where data security is critical.
Extended Fabrics enables the benefits of Fibre Channel technology at distances up to 100 km.
Fabric Watch tracks switch and fabric events to help you optimize fabricwide performance and proactively identify problems before they happen.
QuickLoop integrates private loop-based devices into switched fabric environments.
QuickLoop helps support legacy devices to protect existing investments while also providing performance and reliability advantages.
WEB TOOLS is an advanced monitoring tool that sends alerts about fabric events to help prevent downtime.
You can use a Web browser interface and Java plug-in to monitor switchedfabric SANs from any workstation in your enterprise.
Future Capabilities in the Brocade Intelligent Fabric Services Architecture
The Brocade Intelligent Fabric Services Architecture includes the SilkWorm family of fabric switches, advanced fabric services, open fabric management tools, and enterprise-class security products.
ISL Trunking is an optional software product ideal for optimizing performance of Brocade 2 Gbit/sec Fibre Channel fabric switches.
Frame Filtering enables a variety of new capabilities for monitoring and managing SAN fabrics and enhancing both security and reliability.
Secure Fabric OS is the most comprehensive SAN security architecture available, addressing vulnerabilities in the SAN fabric and supporting multiple authentication methods.
419
140_SANs_AppFT
420
8/14/01
3:42 PM
Page 420
Appendix • Building SANs with Brocade Fabric Switches Fast Track
❖ Chapter 5: The SAN Design Process
Looking at the Overall Lifecycle of a SAN
The SAN design process is a cycle.
This process consists of seven phases: 1. Data Collection 2. Data Analysis 3. Architecture Development 4. Prototype and Test 5. Transition 6. Release to Production 7. Maintenance
Whenever there is a fundamental change to the SAN, the cycle should repeat.
Conducting Data Collection
Data collection is the foundation on which a SAN is built.
You should interview everybody who has an interest in the project.
During the interview process, create a technical requirements document.
Analyzing the Collected Data
There are several things that you need to get out of data analysis: —The number of different fabrics that will make up the SAN solution —The port count and performance characteristics of each fabric —An estimate of the hardware required to meet these requirements
You might be able to localize traffic for better performance if you can create well-defined groups.
Prepare an ROI proposition to justify your SAN project.
140_SANs_AppFT
8/14/01
3:42 PM
Page 421
Building SANs with Brocade Fabric Switches Fast Track • Appendix
❖ Chapter 6: SAN Applications
and Configurations Configuring a High-Availability Cluster
HA clusters are used for redundant, fail-safe installations of mission-critical business applications.
Clustering provides availability, manageability, and scalability.
Availability is the capability of a cluster to tolerate hardware, network, or software errors.
The most common use of clustering is two servers configured to share storage through Fibre Channel.
Redundant HBAs and switches should be used to provide fault tolerance.
The use of dynamic multipathing software, drivers, or HBAs can provide higher levels of availability to your cluster.
Using a SAN for Storage Consolidation
Storage consolidation enables administrators to centralize storage resources.
Consolidation provides more efficient use of storage, enhances manageability, and improves accessibility.
Almost any layout of a storage network can be used for storage consolidation.
Consolidation requires attention paid to how operating systems treat shared volumes.
In order to properly partition data in a consolidation environment, you need to use fabric zoning, LUN masking on storage or the host, or software to control permissions.
It is generally best to use fabric zoning even when also using another access control product to achieve a more effective security model, and to provide a “broadcast container,” which can increase the scalability and reliability of a SAN.
An example of a typical storage consolidation setup is a shared SAN used to provide data storage for a Web farm, where many servers read the same disks to present data.
421
140_SANs_AppFT
422
8/14/01
3:42 PM
Page 422
Appendix • Building SANs with Brocade Fabric Switches Fast Track
Chapter 6 Continued
Storage LUN masking is used to ensure that only specific hosts are allowed access to specific logical units of a storage array.The advantage of storage LUN masking is that the storage guarantees which host is allowed access to any volume.
HBA LUN masking is also used to limit what storage a host can see, and requires that every host in the network participate in the same masking scheme.
Software partitioning provides another type of control over LUN presentation, but it generally requires upper-level software and demands that every host in the network be loaded with that software.
Switch zoning, available in Brocade switches, provides a convenient way to allocate storage to hosts, and to consolidate different departments into a single company network.
Switch zoning does not currently support control at the LUN level, only at the port and WWN levels. Upcoming products will add this capability. For now, other access control techniques might need to be used in addition to switch zoning to provide access control at the LUN level.
Storage LUN masking provides another way to control access to volumes in a shared SAN.
High-end storage arrays provide the capability to specify the port or node WWN of a host HBA, and specify which volumes in the array will respond to requests.
By using storage LUN masking, you can ensure that only hosts with permission can read or write from a specified volume.
Storage LUN masking requires the participation of the storage only to enforce permissions.
HBAs provide access control to volumes through LUN masking.
LUN masking controls which volumes an operating system can see through a particular HBA.
HBA LUN masking requires the participation of all of the hosts in the network to avoid contention for storage resources.
140_SANs_AppFT
8/14/01
3:42 PM
Page 423
Building SANs with Brocade Fabric Switches Fast Track • Appendix
Chapter 6 Continued
LAN-Free Backup Configuration
Traditional backup systems used SCSI direct-attached tape storage.The LAN-based client-server backup model, although an improvement, cannot account for ever-increasing amounts of data through the LAN connection. LAN-free backups using storage networks solve LAN-based problems by offloading traffic from the LAN and increasing bandwidth.
SAN Server-Free Backup
Server-free backup is the use of a SAN to remove backup traffic from a LAN.
Backup is done directly on the SAN for each device, rather than each host being involved in data transfer.
Third-party copy provides an even more efficient way to transfer data to tape, freeing a backup server from needing to directly access disks and copy data to tape.
Making Your Enterprise Disaster Tolerant
Fibre Channel SANs are ideal for mirroring and accessing data across large distances.
It is now possible to separate critical systems many miles apart.
Brocade switches provide extended credits on ISLs to enable high performance and reliable long-distance operation.
❖ Chapter 7: Developing a SAN Architecture
Identifying Fabric Topologies and SAN Architectures
A fabric consists of one or more interconnected Fibre Channel switches.
A SAN includes one or more related fabrics and everything attached to them.
In a resilient core/edge fabric topology, two or more switches act as a core to interconnect multiple edge switches.This is the best “general-use” topology available, especially when combined with the dual-fabric approach to SAN architecture.
423
140_SANs_AppFT
424
8/14/01
3:42 PM
Page 424
Appendix • Building SANs with Brocade Fabric Switches Fast Track
Chapter 7 Continued
In order to select the right topology, you must first decide the requirements for your SAN architecture.This includes redundancy and scalability in addition to port count.
In general, the cascade, ring, full mesh, and partial mesh are best used in architectures where the individual fabrics that comprise the SAN will not change much.This could be true in a static, low-growth environment, or in a “SAN islands” design.
The resilient core/edge topology is the best choice for general use and for situations where SAN requirements are either unknown or might change frequently.
The resilient core/edge topology can be combined with dual fabrics to achieve maximum performance, reliability, and scalability.
Working with the Core/Edge Topology
The core/edge topology offers a number of key advantages over other topologies. Core/edge fabrics are: —Easy to scale without downtime. —Capable of scaling to a large number of ports. —Flexible in terms of their cost-to-performance ratios. (You can add switches to the core with a clear knowledge of how doing so will affect both cost and performance.) —Easy to understand, manage, and performance-tune. —Well-tested and reliable.
Several core/edge fabrics can be used as “cookie-cutter fabrics” when design information is incomplete or might change frequently.
Determining Levels of Availability
There are four levels of availability that a SAN architecture might employ. The dual-fabric, resilient approach is the most reliable and the most frequently recommended.
140_SANs_AppFT
8/14/01
3:42 PM
Page 425
Building SANs with Brocade Fabric Switches Fast Track • Appendix
Chapter 7 Continued
In most cases, this approach is not more expensive to implement than the other three approaches, and it might be less expensive in some cases.
This approach allows for the failure of anything up to and including an entire fabric without application downtime.
Configuring Traffic Patterns
Tiered fabrics allow simplified management and storage resource planning, but are the worst-case scenario from the standpoint of locality.
Locality is the most effective approach to performance tuning in a SAN, but it is frequently unattainable.
You should view locality as a “moving target,” since network complexity increases over time. However, it is worth getting as much locality as is practical into a SAN, since all SANs benefit in several ways from this technique.
Evaluating Performance Considerations
Over-subscription is never a bad thing in and of itself. It is only when oversubscription becomes congestion that problems might arise.
Latency is almost never a driving consideration in real-world SAN performance, since fabric latency is at least one order of magnitude lower than typical disk subsystem latency. Exceptions to this rule include clustering software and some highly performance-sensitive applications.
In almost all cases, considerations outside the fabric will limit performance, such as CPU speed of hosts or the I/O profile of an application.
❖ Chapter 8: SAN Troubleshooting
The Troubleshooting Approach: The SAN Is a Virtual Cable
Use the SAN’s visibility to both storage and hosts to start your troubleshooting process.
425
140_SANs_AppFT
426
8/14/01
3:42 PM
Page 426
Appendix • Building SANs with Brocade Fabric Switches Fast Track
Chapter 8 Continued
The switchShow, nsShow, nsAllShow, errShow, and topologyShow commands are extremely informational and invaluable to the troubleshooting process.
The UNIX format command or HBA vendor-supplied utilities are also helpful in troubleshooting.
When you start the troubleshooting process, determine if the issue is fabric related or device related. A fabric-related issue impacts many devices, and a device issue impacts only a few devices.
Troubleshooting the Fabric
A fabric issue impacts many devices. A logical switch outage, such as segmentation or physical switch outage, can cause many devices to drop out of the fabric. Problems with ISL initialization are also considered fabric issues.
The quickest way to narrow your search of a fabric problem is to compare your baseline SAN profile to your current SAN profile and investigate discrepancies.
A SAN profile includes the number of devices per switch (nsShow), number of devices in the fabric (nsAllShow), and number of switches in the fabric (topologyShow).The errShow and switchShow commands are also helpful in tracking down fabric issues.
Some fabric issues are caused by a mismatch in fabric service timeout variables and the edge device timeout settings. Careful analysis of both the fabric and the edge devices is necessary to resolve this complex issue.
Troubleshooting Devices that Cannot Be Seen
The first thing to check is that the missing device is logically connected to the SAN as indicated by switchShow output.
Next, check to see that the device is present in the Name Server, using the command nsShow. If the device is not in the Name Server, it is invisible to the other devices in the fabric.
Other common causes of missing devices are zone conflicts or marginal links.
140_SANs_AppFT
8/14/01
3:42 PM
Page 427
Building SANs with Brocade Fabric Switches Fast Track • Appendix
Chapter 8 Continued
Troubleshooting Marginal Links
Use portErrShow to establish if there are a relatively high number of errors, such as CRC errors. Look for a steadily increasing number of errors to confirm a marginal link.
A marginal link can impact multiple devices. For example, a shared storage device with a marginal link can cause communication problems with all devices that access that shared storage.
A marginal link can be caused by any of the components that make up the link: switch port, switch GBIC, cable, edge device GBIC, and the edge device.
Troubleshooting I/O Pauses
I/O pauses happen, and both the SAN and edge device can and should tolerate such events.
An I/O pause can be as harsh as the powering down of a host or storage device while I/O is in transit, which will cause I/O to cease. Alternatively, it might be as lightweight as a port-level RSCN, which might be a problem for only the most latency-sensitive applications. Relative to the SAN, fabric events can also cause a pause in I/O.
Several applications, such as video-on-demand and applications that are evolving into the SAN model, such as tape backup, are very sensitive to latency and/or RSCNs. High latencies and large numbers of RSCNs can adversely affect these applications.
Storage vendors, switch vendors, application vendors, and HBA vendors are working with the standards bodies (T11) as well as modifying their product implementations to handle these types of exceptions.
❖ Chapter 9: SAN Implementation,
Maintenance, and Management Installation Considerations
Ensure that ISLs run in front of only the switches to which they are connected.This will allow the switches to be removed without downtime for the fabric.
427
140_SANs_AppFT
428
8/14/01
3:42 PM
Page 428
Appendix • Building SANs with Brocade Fabric Switches Fast Track
Chapter 9 Continued
When racking your switches, be sure to avoid single points of failure.This means separating redundant fabrics into different racks and powering resilient fabrics in such a way that a power failure does not cause the fabric to fail.
Carefully consider solely using in-band management of your switch. Consider using both in-band and out-of-band management.
Have a well thought-out switch-naming convention to enable easy identification of a physical switch in case a problem arises.
If you intend to implement a large fabric, work closely with your switch supplier to identify a version of Fabric OS that supports the size of SAN you intend to build.
Automating Switch Administration Activities
If you have to do a SAN administration activity more than once, consider automating the activity with a script.
Use Expect for automating your SAN administration activities today and consider using Fabric OS APIs when they become available.
Take the Expect script example (run_sw_cmd) and modify it for your SAN administration activities.
If you use Expect scripting, you need the supporting software. See the following URL for Expect installation guidance: http://expect.nist.gov.
Brocade Zoning Considerations
Determine whether you want to use hard or soft zoning prior to implementing your zone scheme.
Hard zoning is more secure than soft zoning, as hard zoning is enforced at the Name Server and at the hardware level and will actually block inappropriate access.
There are differences between hard zoning and soft zoning from a maintenance perspective. For example, you need to update your zone information if you replace a device that is part of a soft zone.
140_SANs_AppFT
8/14/01
3:42 PM
Page 429
Building SANs with Brocade Fabric Switches Fast Track • Appendix
Chapter 9 Continued
Consider using a script to build and maintain large zoning configurations. Scripts can also be helpful to implement disaster recovery policies that are implemented in zoning.
Validating Your Fabric
Baseline your fabric first so that you can quickly identify failures when you validate your SAN. Use the SAN profile as your baseline.
You can automate many validation activities, such as taking your SAN profile and fault injection.
The key to fault injection is to establish how your entire system behaves when a fault is encountered.
Key fabric fault injections include: switch power cycle, ISL disable/enable, and switchDisable/switchEnable. Key edge device fault injections include: reset or power cycle the edge device.You can simulate this event by doing a portDisable/portEnable.
Running a load in your SAN can shake out issues like marginal links.The final test is to run a load on your SAN while you do fault injections. If your SAN is able to handle this test, you are ready for production.
Pick a minimum of two load types for your SAN I/O testing: one that approximates your SAN application load, and a load that is bandwidthintensive.
Ask your host supplier, HBA supplier, or storage supplier for the tools they use for I/O testing in a UNIX environment. If you want to use a Microsoft/Intel tool, Iometer is a good choice.
SAN Maintenance
If possible, use the “cold” upgrade process for fabric upgrades. It takes only a few minutes of downtime.
When adding a switch to the fabric, clear out the zone information.
It is simplest to power down or disable your switch prior to connecting that switch to an existing SAN. Doing so will avoid domain ID conflicts.
429
140_SANs_AppFT
430
8/14/01
3:42 PM
Page 430
Appendix • Building SANs with Brocade Fabric Switches Fast Track
Chapter 9 Continued
A diligently maintained SAN configuration log can help you with disaster recovery, troubleshooting, recreating a switch whose configuration is destroyed, SAN design modification or expansion, recovering accidentally deleted licenses, and recovering a zoning configuration.
Back up your switch configuration with the command configUpload whenever you add or replace a switch.
Maintaining a baseline SAN profile is essential to many SAN maintenance activities. Make sure you know how to create and maintain a SAN maintenance profile.
The act of loading firmware does not impact SAN operations.The process of activating does impact SAN operations because a reboot is required.
140_SANs_index
8/14/01
3:43 PM
Page 431
Index A
B
ACK frame, 43 active/active storage controllers, 198, 200 active/passive storage controllers, 198, 200 Adapters, Media Interface (MIAs). See Media Interface Adapters (MIAs) addressing, switched fabric, 48–49, 358, 361 administrative activities, automating, 367–372 Expect scripting and, 369–372 Fabric OS API and, 367–368 Alias Server, 50, 52, 134 analyzer, Fibre Channel, 287, 314–316 any-to-any connectivity, 268–269 APIs Fabric OS API, 367–368, 378 HBA API, 101–103 application service providers (ASPs), data sharing and, 12 Application-Specific Integrated Circuit (ASIC), 80 applications assessing need for SAN in interviews, 16–17 selecting during design phase, 165 switch management via, 94 arbitrated loop topology, 4, 5, 33, 39, 47–48 architecture, SAN. See SAN architecture asynchronous transfer mode (ATM), 31 automatic device discovery, 133 Fabric OS and, 133 Host Bus Adapters (HBAs) and, 101 automatic path failover, 134 automatic port bypass, managed hubs and, 78 availability levels, SAN architecture and, 256–260 availability, high-availability (HA) cluster, 197
backups, configuration switch configuration file, 393–394 zoning configuration, 382 backups, network accelerating cycling of, 14 collecting information on in interviews, 167 LAN-free, 212–213 reducing network congestion from, 13 remote, 218–219 SAN-based server-free, 213–216 bad_eof error statistic, 297 bandwidth assessing need for SAN, 17 Fibre Channel hub, 34 baseline SAN profiles, 308–311, 317–318, 382–384 block-level protocols, data access speed and, 14 bonded SC connectors, 72 bridges, Fibre Channel, 35, 60, 64–65, 106–109 extended copy support by, 108–109 management interfaces for, 109 number of SCSI buses on, 107 SCSI termination type and, 108 selective LUN presentation by, 108 types of SCSI ports available on, 108 broadcasting, IP over Fibre Channel (IPFC), 88, 96–97 Brocade Fabric Access Layer API, 137 Brocade Fabric Assist, 79, 138–139, 147, 337 Brocade Fabric Manager, 131 Brocade Fabric OS, 132–135, 228 adding switches with, 248 automatic device discovery, 133 command-line interface, 135 continuous port monitoring, 133 dynamic routing services, 134 431
140_SANs_index
432
8/14/01
3:43 PM
Page 432
Index
Fabric Access Layer API, 137 Fabric OS API and, 367–368, 378 Fibre Channel services provided by, 133–134 history of features and enhancements, 363–365 in-band interface, 135 Management Information Bases (MIB) provided by, 135 switch beaconing and, 135 syslog daemon interface, 135 universal port support by, 133 version, selecting which to use, 361–365 Brocade Fabric Watch, 75, 138, 147, 339, 366 Brocade Intelligent Fabric Services Architecture, 140–143 frame filtering and, 142 hardware-enforced zoning, 142 ISL Trunking, 140–142 performance analysis and, 143 Secure Fabric OS and, 143 Brocade QuickLoop, 79, 80, 138–139 isolating marginal port faults, 339 private loop devices and, 165–166, 337 Brocade Remote Switch, 218 Brocade Secure Fabric OS, 143 Brocade SilkWorm switches, 124–132, 146, 147 entry-level series, 126–128 licensing, 136, 322 Metropolitan Area Networking (MAN) and, 219 port error statistics, 339, 341–342 scalable, 128–131 selecting most appropriate, 124–126 SilkWorm 1000, 300, 301 SilkWorm 12000 Core Fabric Switch, 126, 131–132 SilkWorm 2000, 126, 300, 301 SilkWorm 2010, 126, 127 SilkWorm 2040, 126, 127
SilkWorm 2050, 126, 128 SilkWorm 2200, 126 SilkWorm 2210, 127 SilkWorm 2240, 127 SilkWorm 2250, 128 SilkWorm 2400, 126, 128, 129 SilkWorm 2800, 126, 128, 129–130 SilkWorm 6400 Integrated Fabric, 126, 130–131, 256 SwitchType values, 302 zoning of, 373–375 See also switches, Fibre Channel Brocade SOLUTIONware guides, 158, 196 Brocade Web site, 158 Brocade WEB TOOLS, 139–140, 147, 366, 378, 408 Brocade Zoning, 136, 146–147, 372–373 licensing for, 366, 372 private loop devices and, 165–166 buffer credits, switch port, 86–87 buffer-to-buffer flow control, 43 business goals, identifying, 153, 158 business requirements, identifying, 158–159
C cable testers, 315 cabling, 61, 65–68, 164 copper, 61, 65–66 layout of, 351–354 multimode optical, 36, 61, 66–67 single-mode optical, 36, 61, 68–69 camTest command, 289 cascade topology, 236–237 compared with other topologies, 247 resiliency of, 257, 258 central memory diagnostics, 289 centralMemoryTest command, 289 cfgClear command, 324 cfgDisable command, 324, 343 cfgEnable command, 343, 347 cfgShow command, 286, 324, 333
140_SANs_index
8/14/01
3:43 PM
Page 433
Index
channel protocols, 31 channels, 31, 38 Chaparral products Fibre Channel/SCSI bridges, 215 Network Storage, 14 character encoding, Fibre Channel, 36, 41 Class F service, 45, 50 classes of service, Fibre Channel, 37, 39, 43–45 Class 1, 43–44 Class 2, 44, 84 Class 3, 44, 84 Class 4, 44 Class F, 44, 50, 84 Cluster Server, Microsoft’s, 11, 200–202 clustering techniques, FC-VI standard and, 14–15 clusters, high-availability. See HighAvailability (HA) clusters cmemRetentionTest command, 290 CMI bus connection diagnostics, 289 cmiTest command, 289 cold fabric upgrades, 398, 400 combination adapters, 98 command-line interface, Fabric OS, 135 complex core resilient core/edge topology, 244–245, 247 components, 61–65 attention to those in production during SAN planning, 166 evaluating pre-existing, 165–166 redundant HA cluster, 199 validation of, 166 See also specific components composite resilient core/edge topology, 245–246 Computer Associates Unicenter TNG, 350, 368 configDownload command, 393, 394 configShow command, 286, 394 configUpload command, 324, 382, 393 configuration logs, 391–393
433
configuration management software, HBA, 101 congestion, network, 13, 233, 270 connectors, Fibre Channel, 61–62, 69–73 D-B9 connectors, 69–70 high-density optical connectors, 72–73 HSSDC connectors, 70–71 SC connectors, 71–72 copper cabling, 61, 65–66 copper connectors, 62, 69–71 core/edge fabric, 228, 229–230, 242–246 adding edge switches to, 248–250 compared to other topologies, 247 complex core/edge, 244–245 resiliency of, 256, 257, 258 scaling without downtime, 248 simple core/edge, 244 target designs for, 253–256 upgrading core switches, 250–253 core switches, 81–82, 242–243, 250–253 adding new to core/edge fabric, 251 configuring new core/edge fabric, 251–252 core team, SAN identifying people to include, 156–157, 193–194 interview process for, 157–176 costs Brocade SilkWorm switches, 125, 126 cabling media, 65, 67, 68 cascade topology, 236, 237 complex core resilient core/edge topology, 245 full-mesh topology, 238 partial-mesh topology, 242 ring topology, 238 CPU speed, pre-SAN performance data and, 170 CRC errors, 295 crc_err error statistic, 296–297 crossPortTest command, 290 Crossroads Systems, 14, 215
140_SANs_index
434
8/14/01
3:43 PM
Page 434
Index
D D characters, 41 data access, increasing speed of, 14 data analysis, SAN design process and, 153, 194 port requirements, establishing, 182–187 Return On Investment (ROI) proposition, 153, 159, 187–189 SAN grouping process, 178–182 data characters, 41 data collection, SAN design process and, 153, 156–177, 194 backup information, 167 business problem identification, 158 business requirement identification, 158–159 component testing needs, 166–167 components, identifying those in place, 165–166 components, identifying those in production, 166 current performance data, 168–172 design interview form, 175–176 host information, 160–162 initiator-to-target communications matrix, 167–168 maintenance downtime, 174–175 node information, 160–161 performance, determining future needs, 172–174 physical assessment of hardware, 176–177 processing collected data, 177–182 SAN-enabled applications needed, 165 SAN implementation downtime, 174 selecting people to interview, 156–157 storage device information, 162–163 storage facility information, 164 technical requirement identification, 159–160 timeline creation from, 175–176 data movers, 14, 214 data replication techniques, 218
data sharing, 12, 18–19, 203–212 file-level sharing, 19 LUN masking and, 210–211 resource sharing, 19 software management of, 211–212 switch zoning and, 208–210 volume-level sharing, 19 with Web farms, 206–207 See also switch zoning data storage consolidating with SANs, 9–10, 11–13, 203–212 increased need for, 8–9 sharing among multiple hosts, 12, 203–212 database servers, HA cluster configuration, 198–200, 226 DataCare SANsymphony, 12 DB-9 connectors, 62, 69–70 DB-9 serial cabling, 69, 70 dd tool, UNIX, 388–389 debuggers Fibre Channel analyzer, 314–315 portLog, 314 protocol analyzers, 315–316 dedicated connection types, 43 Dense Wave Division Multiplexing (DWDM), 109–110, 217, 219 design interview form, 176 Destination ID (D_ID), switch, 302 devices adding to fabrics, 401–402 automatic discovery of, 133 collecting information on in interviews, 160–164 determining if existing require additional hardware, 165–166 timeout at bring up, 321–322 troubleshooting missing, 279–283, 327–335 DiagErr# message, 293 diagClearError command, 290
140_SANs_index
8/14/01
3:43 PM
Page 435
Index
diagDisablePost command, 290 diagEnablePost command, 290 diagHelp command, 289 diagnostic switch commands, 289–308 errShow, 278, 292–295, 318 help, 291–292 nsAllShow, 286, 320, 329, 382, 395 nsShow command. See nsShow command portDisable, 249, 302, 318, 330, 334–335 portEnable, 302, 318, 330, 334–335 portErrShow, 286, 295–297 portLoopbackTest, 289, 290 show vs. dump, 291 supportShow. See supportShow command switchEnable, 251, 318 switchShow. See switchShow command topologyShow, 307–308, 309, 320, 329 diagnostics, storage array, 287 diagShow switch command, 286, 290 disabled switches, 300 disaster tolerance, SANs and, 15–16, 31, 216–221 data replication and remote backup, 218–219 Metropolitan Area Networks (MANs) and, 219–221 Disk Administrator,Windows, 312, 317 disk farms, 9 disk I/O performance, 169–172, 385–390 disk monitoring tools, 169–170 disk seek time, 171 diskmon feature,Windows NT, 169 diskperf -yd command, 172 diskperf -yv command, 172 diskperf utility, 172, 386 disks storage on, 32, 33, 64, 111 troubleshooting missing, 279–283 distance requirements, SANs and, 17–18 distributed fabric services, 41 domain IDs, switch, 48, 300–301, 319 conflicting, 326–327
435
merging fabrics and, 397 setting, 358 downstream information, 302, 303, 304 downtime determining acceptable for maintenance and changes, 174–175 determining acceptable for SAN implementation, 174 scaling core/edge networks without, 248–253 drivers, determining if full fabric, 165–166 dual-fabric SANs, 248 HA clusters and, 199, 202 nonresilient, 257 resilient, 257 dual-port adapters, 98 DWDM. See Dense Wave Division Multiplexing (DWDM) dynamic discovery Fabric OS and, 133 Host Bus Adapters (HBAs) and, 101 dynamic routing services, Fabric OS, 134
E e-mail, data storage and, 8 E_Ports (ISLs), 48–49, 87 cable layout and, 351–354 incomplete initialization of, 318–319 ISL over-subscription ratio, 231–232 load sharing through, 134 port configuration conflicts and, 322–323 trunking, 140–142 edge devices. See nodes edge ports, 243 edge switches, 243, 248–250 18-switch core/edge SAN design, 255 Electrically Erasable Programmable ReadOnly Memory (EEPROM), 74–75 Emulex HBA configuration utility, 101, 102 enc_in error statistic, 296 enc_out error statistic, 297 End Of Frame (EOF) primitive, 40
140_SANs_index
436
8/14/01
3:43 PM
Page 436
Index
end-to-end flow control, 43 er_disc_c3 error statistic, 297 er_enc_out statistic, 338 errDump command, 286 error logs, 292–295 error messages, 284, 285, 293, 324, 338 error statistics, 296–297, 341–342 errors displaying logged, 292–295 MQ errors, 318, 327 errShow command, 278, 292–295, 318 Ethernet, 31 in-band/out-of-band switch management, 356 router management by, 109 Exchange Mail Server, 226 Expect scripting firmwareDownload automation by, 400 switch management by, 369–372 zoning management by, 379–380 Extended Copy command, 108–109, 214, 215 Extended Fabrics, 136–137 Extreme SCSI, 169, 170 EZ Fibre, JNI’s, 312
F F_Ports, 48–49, 87 Fabric Access Layer API, 137 Fabric Assist, 79, 138–139, 147, 337 Fabric Configuration Server, 51 fabric licenses, 322, 366 Fabric Login (FLOGI) frame, 50 Fabric Loop Attachment (FLA), 337 Fabric Manager, 131 Fabric OS API, 367–368, 378 fabric port count, 230 fabric segmentation, 318, 323–327, 348 fabric services, 36, 38, 41, 49–52, 85–86 Alias Server, 50, 52, 134 Fabric/Switch Controller, 51
Login Server, 38, 50 Management Server, 51, 85, 86, 133 Name Server. See Name Server Registered State Change Notification (RSCN), 51, 85–86, 284, 343–344, 347 switches and, 85–86 Time Server, 52, 86 Fabric Shortest Path First (FSPF), 90–91, 134, 228, 233, 256 Fabric/Switch Controller, 38, 50 fabric topologies, 229, 235 cascade topology, 236–237 comparison of properties of, 247 complex, 246 congestion and, 270 core/edge fabric, 242–246, 248–256 distance and, 234 factors affecting performance, 270–271 full-mesh topology, 239–240 over-subscription and, 270 partial-mesh topology, 240–242 resiliency of, 256 ring topology, 237–238 SAN architecture availability and, 257–260 scalability of, 236 vs. SANs, 275 fabric troubleshooting, 316–327 comparing SAN profiles to identify problem, 317–318 domain ID conflicts, 326–327 fabric information available from hosts, 317 fabric licenses and, 322 I/O pauses and, 343–344 incompatible fabric parameters, 325–326 MQ errors, 327 Name Server discrepancies, 320–321 port configuration conflicts, 322–323, 347 switch LEDs and, 288 symptoms indicative of fabric problem, 316
140_SANs_index
8/14/01
3:43 PM
Page 437
Index
timeout of devices at bring up, 321–322 topologyShow command and, 309, 320, 321, 329, 382, 395 zoning conflicts, 323–324 See also SAN troubleshooting Fabric Watch, 75, 138, 147, 339, 366 Fabric Zone Server, 51 fabrics, 5, 34, 36, 49–52 adding edge devices to, 401–402 adding switches to, 395–398 bringing up, 321–322, 394–395 defined, 229 licenses for, 322, 366 merging, 395, 408 segmented, 318, 323–327, 348 timeout of devices at bring up, 321–322 troubleshooting. See fabric troubleshooting upgrading, 398–400, 408 verification of, 382–383, 395 zone management and, 378–379 fabricShow command, 250, 251, 286 fan-in, 234 fan-out, 234 faShow command, 286 fastboot command, 399 fault injection techniques, 384–385 fault tolerance of High-Availability (HA) clusters, 199, 200 of SANs, 10–11 faultShow command, 286 FC-0 Fibre Channel layer, 36, 40 FC-1 Fibre Channel layer, 41 FC-2 Fibre Channel layer, 36, 41 FC-3 Fibre Channel layer, 36, 41 FC-4 Fibre Channel layer, 36, 41–42 FC-AL. See Fibre Channel Arbitrated Loop (FC-AL) FC-GS-3 standard, 41, 50 FC-VI standard, 15 FC_IP (Fibre Channel across IP), 110
437
FCIA. See Fibre Channel Industry Association (FCIA) FCP/SCSI protocol, 96 Fiber Distributed Data Interface (FDDI), 31, 43 fiber-optic cables, 36, 61, 65 fiber-optic connectors, 62, 71–73 high-density optical connectors, 72–73 SC optical connectors, 71–72 Fibre Channel, 30–58 broadcasting via, 88 character encoding, 36, 41 classes of service, 37, 39, 43–45 cost of, 28 CPU requirements, 170 disk seek time requirements, 171 distances supported, 2, 39 HBA speed requirements, 171 history of, 2 hot-plug systems and, 104–105 interoperability of, 28, 58 layers, 36–37 PCI bus speed requirements, 170–171 protocols supported, 2, 31, 39, 42 RAID speed requirements, 171 RAM requirements, 171 resources on, 8 routing IP over to Gigabit Ethernet, 65, 110–111 routing over IP networks, 110 SCSI performance vs., 57 speed of, 14, 31, 38 standards, 35 switched fabric installation, 5, 6 topologies, 37, 39 transfer rates, 35 Fibre Channel analyzers, 287, 314–316 Fibre Channel Arbitrated Loop (FC-AL), 4, 5, 33, 39, 47–48, 60 Fibre Channel common services layer. See FC-3 Fibre Channel layer Fibre Channel disks, 33
140_SANs_index
438
8/14/01
3:43 PM
Page 438
Index
Fibre Channel Industry Association (FCIA), 8, 22, 35, 58 Fibre Channel layers, 36–37, 40–42 FC-0, 36, 40 FC-1, 36, 41 FC-2, 36, 41 FC-3, 36, 41 FC-4, 36, 41–42 Fibre Channel Management MIB, 93 Fibre Channel Protocol (FCP), 96 Fibre Channel protocol layer. See FC-2 Fibre Channel layer Fibre Channel standards projects, 35 Fibre Channel Storage Area Network (SAN). See SANs (Storage Area Networks) Fibre Channel-to-Dense Wavelength Division Multiplexing (DWDM), 16, 109–110 Fibre Channel-to-Gigabit Ethernet bridges, 65, 110–111 FibreAlliance Management Information Base (MIB), 92–93 FibreAlliance MIB, 109 fields, frame, 39–40 file-level sharing, 19 File Transfer Protocol (FTP), router management by, 109 firmware upgrades, 89, 398–400 cold fabric upgrade, 400 hot fabric upgrade, 399 scripting of, 400 FL_Ports, 48–49, 87 flow control, 43 format UNIX command, 312, 317 14-switch core/edge SAN design, 255–256 frame filtering, 142 frames, 37, 39–40, 41 classes of service and, 43–45 headers, 39–40 free ports, 243 full-mesh topology, 239–240, 247 fw_downloand script, 400
G GBICs. See Gigabit Interface Connectors (GBICs) Get All Next (GA_NXT) request, 51 get_san_profile script, 384 Get_Time frame, 52 Get_Time_Response frame, 52 Gigabit Ethernet, 14, 30, 57 cost of, 28 routing Fibre Channel to, 110–111 Gigabit Interface Connectors (GBICs), 60, 61–62, 73–75, 121 Brocade SilkWorm switches and, 126 disadvantages of, 74, 75 GBIC ports on equipment, 74 serialized, 74–75
H HA clusters. See High-Availability (HA) clusters hard zoning, 83–84, 303, 374, 375–378 hardware forwarding, 38 hardware, SANs, 21, 61–65 identifying pieces in place during design phase, 165–166 initiating devices, 33 interconnecting devices, 33–34 physical assessment of in design process, 176–177 selecting, 21–22 target devices, 32–33 See also specific pieces HBA. See Host Bus Adapters (HBAs) HBA API, 101–103 headers, frame, 37, 39–40 heartbeat, network, 198, 202 help command, 291–292 Hewlett-Packard LUN Manager product, 212 OpenView, 12, 350, 368 high-availability applications, 198–200
140_SANs_index
8/14/01
3:43 PM
Page 439
Index
high availability, ensuring with SANs, 10–11, 31 High-Availability (HA) clusters, 196–202, 257 active/active model, 198 active/passive model, 198 advantages of, 197 database servers and, 198–200 fault tolerance of, 199, 200 high-availability applications and, 198–200 Microsoft Cluster Server and, 200–202 redundancy and, 199 storage devices for, 200 zero-downtime failovers, 199 high-availability storage devices, 200 high-density optical connectors, 62, 72–73 high-end storage arrays, 113–114 LUN export across multiple ports by, 113–114 selective LUN presentation by, 113 snapshot backup volumes by, 114 High-Performance Parallel Interface (HiPPI), 31, 38, 42–43 HighGround, Sun’s, 350 hop counts, 231, 276 Host Bus Adapters (HBAs), 33, 60, 64, 95 combination adapters, 98 configuration management software for, 101 connecting hosts to fabric with, 95 default LUN access permissions and, 100 drivers for, 172 dual in HA clusters, 199 dynamic discovery by, 101 Fabric Assist and, 139 fabric-capable, 98–99 HBA API and, 101–103 hot-plug systems and, 104–106 LUN mapping (persistent binding) and, 99–100 LUN masking and, 99 ports available on, 98 private (loop-based), 98
439
protocol access permissions, 100 protocols supported, 95, 96–97 QuickLoop and, 139 remote booting and, 103–104 speed requirements, 97, 171, 172 static discovery by, 101 storage partitioning by, 210–211 types of, 95–96 zoning, 373 host tier switches, 263 hosts, 33 checking for with switchShow command, 329 collecting information on in interviews, 160–162 troubleshooting information available from, 287, 312–313, 317 hot fabric upgrades, 398, 399 hot-plug systems, 104–106, 126 hot-swappable components, 86, 125 HSSDC-2 connectors, 71 HSSDC connectors, 62, 70–71 hubs, Fibre Channel, 4, 34–35, 60, 63, 76–80 LIP process and, 77, 78–79 managed, 35, 60, 76–78 simple electrical, 34, 35, 60, 76 vs. switches, 57
I I/O generators, 387–390 I/O load, 385–390 generation of, 387–390 types of, 386–687 I/O pauses, 342–343, 347 IBM DB2, 15, 226 in-band switch management, 356–358 incomplete ISL initialization, 318–319 InfiniBand technologies, 30, 57, 58, 71 initiating devices, 33 initiator-to-target communications, mapping, 167–168 installation, SAN. See SAN installation
140_SANs_index
440
8/14/01
3:43 PM
Page 440
Index
Integrated Drive Electronics (IDE), 2 Intelligent Fabric Services Architecture, 140–143 frame filtering and, 142 hardware-enforced zoning, 142 ISL Trunking, 140–142 performance analysis and, 143 Secure Fabric OS and, 143 Intelligent Peripheral Interface (IPI), 42 Inter-Switch Links (ISL; E_ports), 48–49, 87 cable layout and, 351–354 incomplete initialization of, 318–319 ISL over-subscription ratio, 231–232 load sharing through, 134 port configuration conflicts and, 322–323 trunking, 140–142 interconnecting devices, 33–34 Internal Rate of Return (IRR) calculations, 189 Internet, data storage and, 9 Internet Protocol (IP), 2, 31, 39, 42, 96–97 Internet Service Providers (ISPs), 12–13 interoperability labs, 22–23 interviews, SAN design process and, 150, 153 backup information, identifying required, 167 business problem identification, 158 business requirements identification, 158–159 component testing needs, 166–167 current performance data, 168–172 design interview form, 175–176 host information, collecting, 160–162 identifying people to interview, 156–157, 193–194 implementation, determining acceptable downtime for, 174 initiator-to-target communications matrix, 167–168 maintenance downtime, determining acceptable, 174–175
performance, determining future needs, 172–174 SAN-enabled applications desired, 165 storage device information, 162–163 storage facility information, 164 technical requirement identification, 159–160 timeline creation from, 175–176 Iometer, Intel’s, 169, 170, 387, 389 iostat utility, Sun Solaris, 169, 386 IOzone, 387 IP addresses, setting switch, 358, 361 IP networks routing Fibre Channel across, 110 routing over Fibre Channel to Gigabit Ethernet, 110–111 IP protocol, 2, 31, 39, 42, 96–97 IP targets, 32, 33 IPFC protocol, 88, 96, 356–358 iSCSI, 57–58 ISLs (E_Ports). See Inter-Switch Links (ISLs)
J Java-based Web pages, switch management by, 93–94 JNI EZ Fibre, 312 Just a Bunch Of Disks (JBOD), 32, 33, 64, 111
K K characters, 41 Key Distribution server, 50
L LAN-based backup configurations, 212–213, 214 LAN-free backup configurations, 212–216 latency, 231 LC connectors, 72 LEDs, switch, 287–289, 318, 329
140_SANs_index
8/14/01
3:43 PM
Page 441
Index
legacy devices, connecting to SANs, 106 Legato NetWorker, 13 license, fabric, 322, 366 licenseShow command, 286, 322, 366 LIP (Loop Initialization Primitive), 77–78, 337–338, 339 LIP process, 77, 78–79 LIP storm, 79 Lip_in count, 339, 342 Lip_out count, 339, 342 Lip_rx count, 342 load testing, I/O, 385–390 locality, performance optimization through, 266–268 logical unit number. See LUN (Logical Unit Number) Login Server, 38, 50 logs, configuration, 391–393 logs, error, 292–295 loop environments, 4, 5, 33, 90 isolating marginal port faults, 339 LIP imbalances and, 339 LIP process and, 78–79 marginal GBICs and, 338 marginal loop connections, 337–338 marginal port behavior on disrupted, 338 migrating to switched fabrics, 79–80 Loop Initialization Primitive (LIP), 77, 337–338, 339 loop ports. See FL_Ports loop zoning, 77 loopPortTest command, 290 LUN (Logical Unit Number) access permissions, 100 high-availability (HA) clusters and, 200 LUN-level zoning, 373, 374 Microsoft Cluster Server configurations, 200–202 selective presentation of, 108 LUN Manager, Hewlett-Packard’s, 212 LUN mapping (persistent binding), 99–100 LUN masking, 122, 205, 226
441
HBA-based, 99 hot-plug systems and, 105 storage partitioning by, 210–211
M make_zone script, 379–380 MAN technologies. See Metropolitan Area Networking (MAN) manageability, high-availability cluster, 197 managed Fibre Channel hub, 34–35, 60, 76–78 Management Information Base (MIB), 92–93, 135 Management Server, 38, 85, 86, 133 mapping, LUN, 99–100 marginal GBICs, 338 marginal loop connections, 337–338 marginal point-to-point/fabric device links, 335–336 marginal ports, 335 disrupted loops and, 338 fault isolation and, 339, 340 marginal GBICs, 338 marginal loop connections, 337–338 marginal point-to-point/fabric device links, 335–336 portErrShow command, 295–297 masking, LUN, 122, 205, 226 HBA-based, 99 hot-plug systems and, 105 storage partitioning by, 210–211 media, cabling, 61, 65–68, 164 Media Interface Adapters (MIAs), 75 mesh topologies, 238 compared to other topologies, 247 full-mesh topology, 239–240 partial-mesh topology, 240–242 resiliency of, 256, 257, 258 metadata servers, 212 Metropolitan Area Networking (MAN), 15–16, 217, 219–221, 238 MIAs. See Media Interface Adapters (MIAs)
140_SANs_index
442
8/14/01
3:43 PM
Page 442
Index
Micromuse’s Netcool, 350, 368 Microsoft Cluster Server (MSCS), 11, 200–202 Microsoft Exchange databases, 11 Microsoft SQL Server, 226 Microsoft Windows Hardware Quality Lab (WHQL), 202 migration process, 154 missing devices, troubleshooting, 279–283, 327–335 locating on Name Server with nsShow, 332–333 port configuration conflicts, 329–332 switchShow command and, 329–332 zoning mismatches and, 333–335 MQ errors, 318, 327 mqShow command, 286 MSCS. See Microsoft Cluster Server (MSCS) MT-RJ connectors, 72–73 multi-LUN devices, 208 multicast groups, 52 multimode optical cables, 36, 61, 66–67 multipathing software, 199
N Name Server, 38, 41, 50–51, 85, 133–134 hard zoning and, 376 missing devices and, 280–283, 303–307, 332–335 nsAllShow command, 286, 320, 329, 382, 395 nsShow command, 281–282, 303–307, 320–321 names, switch, 360 Net Present Value (NPV) calculations, 189 NetBackup,VERITAS’, 13 Netcool, Micromuse’s, 350, 368 Network Attached Storage (NAS), 17, 18 network backups
accelerating cycling of, 14 collecting information on in interviews, 167 LAN-free configurations for, 212–213 reducing network congestion from, 13 remote, 218–219 SAN-based server-free, 213–216 Network Data Management Protocol (NDMP), 214 network heartbeats, 198, 202 network protocols, 31 NetWorker, Legato’s, 13 node count, 230 node WWNs, 306–307 nodes, 230 adding to fabrics, 401–402 collecting information on in interviews, 160–164 missing from Name Server, 334–335 over-subscription of, 232 timeout of at bring up, 321–322 WWNs, 307 nonresilient dual-fabric SANs, 257 nsAllShow command, 286, 320, 329, 395 nsShow command, 286 fabric validation using, 382 troubleshooting fabrics with, 320–321 troubleshooting missing devices with, 281–282, 332–335 troubleshooting SANs with, 303–307
O Open Fibre Control (OFC), 67 OpenView, Hewlett-Packard’s, 12, 350, 368 optical cabling, 36, 61, 65 optical connectors, 62, 71–73 Oracle Parallel Server, 15, 226 out-of-band switch management, 356 over-subscription, 231
140_SANs_index
8/14/01
3:43 PM
Page 443
Index
P parallel bus SCSI, 2–4 parallel SCSI termination, 108 parityCheck command, 290 partial-mesh topology, 240–242, 247 PATROL, BMC’s, 350, 368 payback period, 189 payloads, 36, 39 PCI bus speed, pre-SAN, 170–171 PCI hot-plug systems, 104–106 perfmeter, Sun Solaris, 169 perfmon,Windows, 169 performance Brocade Intelligent Fabric Services Architecture and, 143 of cascade topology, 237 of complex core resilient core/edge topology, 245 determining future performance needs, 172–174 evaluating pre-SANs, 168–172 factors affecting, 270–271 of full-mesh topology, 240 of partial-mesh topology, 242 of ring topology, 237–238 Peripheral Component Interconnect (PCI) hot-plug systems, 104–106 persistent binding (LUN mapping), 99–100 persistent logging, 293 phantom devices, 302, 303 PLOGI/FLOGI timeout failures, 321–322 point-to-point connections, 87 point-to-point topology, 37, 39, 45–47 port addressing, 307–308 port-based zoning, 83 port counts, 230 port indicators, LED, 287–289 port internal loopback diagnostic, 290 Port Login (PLOGI) frame, 50 port register diagnostics, 289 port WWNs, 306–307
443
portCfgEport command, 322–323, 330 portcfgFport command, 330 portCfgLport command, 286, 330 portDisable command, 249, 302, 318, 330, 334–335 portEnable command, 302, 318, 330, 334–335 portErrShow command, 286, 295–297 portFlagsShow command, 286 portLog debugging tool, 314 portLogDump command, 286, 314 portLogShow command, 314 portLoopbackTest command, 289, 290 portRegShow command, 286 portRegTest command, 289 portRouteShow command, 286 ports, 48–49, 50 automatic bypass of by managed hubs, 78 buffer credits available per switch port, 86–87 collecting information on in interviews, 160–161 configuration conflicts, 322–323, 348 determining number needed, 182–187 display of port state information, 297–303, 318 E_Ports. See E_Ports (ISLs) edge, 243 F_Ports, 48–49, 87 FL_Ports, 48–49 free, 243 manual configuration of, 330 marginal, 335–342 monitoring of by Fabric OS, 133 number on Brocade SilkWorm switches, 125 number on HBAs, 98 number on routers, 107, 108 self-configuring, 87–88 types of SCSI on routers, 108 universal port support by Fabric OS, 133 portSemShow command, 286
140_SANs_index
444
8/14/01
3:43 PM
Page 444
Index
portShow command, 341, 342 portStatsShow command, 278, 341, 342 power supplies, redundancy and, 86, 125 primitives, 36, 39–41 principal switches, 300 Prisa’s Visual SAN, 350, 368 private devices, 302, 303 private Host Bus Adapters (HBAs), 98 Private Loop Direct Attached (PLDA), 337 private loop drivers, 165–166 Private Loop Fabric Attach (PLFA), 337 problem descriptions, troubleshooting and, 284–285 protocol analyzers, 315 prototypes, 21, 153–154 psShow switch command, 286
Q qlDisable command, 330 qlPortDisable command, 330 qlShow command, 286, 330 QuickLoop, 79, 80, 138–139, 330, 337, 339, 366 QuickLoop CAM diagnostics, 289
R R_RDY primitive, 43 racking, 354–355 RAID storage arrays, 32, 33, 64, 111–112 controller speed requirements, 171 switch zoning and, 208 RAM (Random Access Memory), 171 ramTest command, 289 redundancy, 86, 125, 126, 256–257 redundant fabric architecture, 175, 199, 256–257, 258–260 Registered State Change Notification (RSCN), 51, 85–86 I/O pauses and, 343–344, 347 new device registration and, 284 reliability cascade topology, 236, 237
complex core resilient core/edge topology, 245 full-mesh topology, 240 partial-mesh topology, 241, 242 ring topology, 238 remote backups, 218–219 remote booting, 103–104 Remote Switch, 218 resiliency, SANs and, 126, 256 resilient core/edge fabric topology, 242–246 adding edge switches to, 248–250 defined, 229–230 upgrading core switches, 250–253 resilient dual-fabric SANs, 257 resource sharing, 19 Return Merchandise Authorization (RMA) system, 173–174 Return On Investment (ROI) proposition, 153, 159, 187–189, 190 RFT_ID frame, 50–51 ring topology, 237–238 compared to other topologies, 247 resiliency of, 256, 257, 258 ROI proposition. See Return On Investment (ROI) proposition rolling upgrades, HA clusters and, 197, 198–199, 202 routers, Fibre Channel, 35, 60, 64–65, 106–109 extended copy support by, 108–109 management interfaces for, 109 number of SCSI buses on, 107 SCSI termination type and, 108 selective LUN presentation by, 108 types of SCSI ports available on, 108 run_sw_cmd script, 369, 372, 379, 384, 400
S SAN architecture, 153, 230–231, 253–256 any-to-any connectivity and, 268–269 availability models, 256–260 core/edge fabric designs, 253–256
140_SANs_index
8/14/01
3:43 PM
Page 445
Index
development of, 153 Intelligent Fabric Services Architecture, 140–143 leveraging tiers, 261–265 localizing traffic, 266–268 SAN-based server-free backups, 213–216 SAN-based third-party copy data movers, 214, 215–216 SAN configurations, 196–221, 226 disaster tolerance solutions, 216–221 High-Availability (HA) clusters, 196–202 LAN-free backup configurations, 212–213 SAN-based server-free backups, 217 storage consolidation configurations, 203–212 SAN grouping, 178–182 SAN installation, 21, 350 creating baseline profile at, 382–383 ISL cable layout for, 351–354 racking considerations, 354–355 setting switch parameters, 358–359, 361 zoning considerations, 372–382 SAN lifecycle, 190 architecture development phase, 153 data analysis phase, 153, 177–189 data collection phase, 153, 156–177 maintenance, 155 migration process, 154 overview of, 151–155 prototypes, 153–154 release to production, 154–155 testing, 154 SAN maintenance, 155 adding edge devices to fabrics, 401–402 adding switches to fabrics, 395–398 bringing up fabrics, 321–322, 394–395 configuration log for, 391–393 downtime requirements for, 174–175 fabric upgrades, 398–400 merging fabrics, 395, 408 replacing switches on fabrics, 395–398 switch configuration backups, 393–394
445
SAN management, 350–351 automation of administrative activities, 367–372 in-band IPFC, 356–358 out-of-band, 356 SAN port count, 230 SAN profiles, 308–311, 317–318, 382–384 SAN troubleshooting, 278–316 creating problem descriptions, 284–285 creating SAN profiles, 308–311 Fibre Channel analyzers, 314–316 gathering supportShow information, 285–287 I/O pauses, 342–343 marginal links, 335–342 missing devices, 279–283, 327–335 portLog debugging tool, 314 SAN profiles, 308–311, 317–318, 382–384 scenario describing, 279–283 switch diagnostics and, 289–308 switch LEDs and, 287–289 using information from hosts for, 312–313 “virtual cable” approach to, 278–279 See also fabric troubleshooting SAN verification fault injection techniques, 384–385 I/O load and, 385–390 SAN profiles and, 382–383 SANavigator, 350, 368 SANergy,Tivoli’s, 12, 212 SANmark Conformance Documents (SCDs), 35 SANmark suite, 8, 22, 35, 58 SANs (Storage Area Networks), 229 acceleration of backup cycles by, 14 applications best served by, 16–17 arbitrated loop topology, 4, 5, 33, 37, 39, 47–48 architecture overview, 30–36 availability levels, 257–260 cluster protocol access and, 14–15 configuration log for, 391–393
140_SANs_index
446
8/14/01
3:43 PM
Page 446
Index
configuration, typical, 7 creation of, 2–7 data access speeds and, 14, 31 deployment of, 20–24, 154–155, 174 design process, 153, 156–177, 187–189, 194 disaster tolerance with, 15–16, 31, 216–221 distances supported, 4, 17 fabrics vs., 275 high availability features of, 10–11, 31 High Availability (HA) clusters and, 196–202 initiating devices, 33 installing. See SAN installation lifecycle of, 151–155 maintenance requirements, 155 managing data storage with, 9–10 Network Attached Storage (NAS) vs., 18 network congestion and, 13 point-to-point topology, 37, 39, 45–47 port requirements, 182–187 prototypes, 21, 153–154 release to production, 154–155 remote booting and, 103–104 resources on, 8 Return On Investment (ROI) proposition, 153, 159, 187–189 scalability of, 31 speed of, 17, 31 storage consolidation and data pooling through, 11–13, 203–212 storage servers vs., 31, 32 switched fabric topology, 37, 39, 48–49 target devices and, 32–33 testing, 21, 23, 154, 382–390 tiered, 261–265 when to deploy, 16–20 See also fabrics; Fibre Channel SANsymphony, DataCare’s, 12 SC connectors, 62, 71–72
SCDs. See SANmark Conformance Documents (SCDs) scripting, automated of firmwareDownload, 400 SAN profiles and, 384 switch operations, 369–372 zoning operations, 379–380 SCSI buses, 107 SCSI disks, 33 SCSI Enclosure Services (SES), 94 SCSI-FCP standard, 41 SCSI protocol, 2, 28, 31, 39, 41, 42, 57 SCSI-to-Fibre Channel routers, 65 Secure Fabric OS, 143 segmented fabrics, 323–327, 348 selective LUN presentation, 108, 113 serial ports router/bridge management by, 109 switch management by, 91 Serial Storage Architecture (SSA), 3 serialized ID GBICs, 74–75 server cycles, increasing, 14–15 server-free backups, SAN-based, 213–216. See also LAN-free backup configurations services, fabric. See fabric services SES. See SCSI Enclosure Services (SES) setGbicMode command, 290 setSplbMode command, 290 shared file system technique, 206 signal retiming, managed hubs and, 78 signaling interface, Fibre Channel, 40 SilkWorm series switches. See Brocade SilkWorm switches simple electrical hubs, 34, 35, 60, 76 Simple Name Server, 133. See Name Server simple resilient core/edge topology, 244, 247 single fabric, nonresilient SAN architecture, 257 single fabric, resilient SAN architecture, 257 single-mode fiber-optic cable, 36, 61, 68–69
140_SANs_index
8/14/01
3:43 PM
Page 447
Index
Single Point Of Failure (SPOF), 233–234 single-port adapters, 98 snapshot backup volumes, 114 SNIA. See Storage Networking Industry Association (SNIA) SNMP Management Information Base (MIB), 92, 121, 135 SNMP polling, 295–296 SNMP (Simple Network Management Protocol) router/bridge management by, 109 switch management by, 91–93 soft zoning, 83, 375–378, 408 software identifying existing in design phase, 165–166 selecting, 21–22, 165–166 storage partitioning, 205, 211–212, 226 Solaris, 169–170, 287, 312, 317 special character (K), 41 speed assessing need for, 17, 122 auto-negotiation of by switches, 88 HBA capabilities, 97 spinFab command, 290 spinSilk command, 290 sramRetentionTest command, 290 ST connectors, 71 star topology network, 228, 242–244 Start Of Frame (SOF) primitive, 39 State Change Registration (SCR) frame, 51 static discovery, HBAs and, 101 Storage Area Networks (SANs), Fibre Channel. See SANs (storage area networks) storage arrays, 9, 11, 21, 287. See also RAID storage arrays storage consolidation, SANs and, 11–13, 203–212 partitioning software, 211–212 shared storage with Web farms, 206–207 switch zoning and, 208–210
447
storage devices, 64 collecting information on in interviews, 162–163 connecting legacy to SANs, 106 drivers for, 204 High-Availability (HA) clusters and, 200 high-end storage arrays, 113–114 individual disk drives, 33, 64, 111 Just a Bunch Of Disks (JBOD), 32, 33, 64, 111 RAID storage arrays, 64, 111–112, 208 storage consolidation SAN configurations, 203, 204 storage LUN masking, 105 storage network topologies, 37, 45–49 arbitrated loop topology, 37, 39, 47–48 point-to-point topology, 37, 39, 45–47 switched fabric topology, 37, 39, 48–49 Storage Networking Industry Association (SNIA), 8, 103 Storage Networking World Conference, 8 storage partitioning with LUN masking, 210–211 software for, 205, 211–212, 226 with switch zoning, 51 See also switch zoning storage tier switches, 263 subordinate switches, 300 Sun Microsystems products HighGround, 350, 368 Jira management standard, 109 Solaris, 169–170, 287, 312, 317 supportShow command, 285–286, 289, 290, 318 switch beaconing, Fabric OS and, 135 switch commands, 289–290 case sensitivity of, 291 diagnostic. See diagnostic switch commands show vs. dump, 291 See also specific commands switch configuration file, 393–394
140_SANs_index
448
8/14/01
3:43 PM
Page 448
Index
switch domain IDs, 300–301, 319, 326–327, 358, 361, 397 switch LEDs device troubleshooting and, 329 fabric troubleshooting and, 318 SAN troubleshooting and, 287–289 switch ports, buffer credits per, 86–87 switch zoning, 83–84, 226, 372–382 adding new switches and, 381, 396–397 aliases and, 381 automatic scripting of activities, 379–380 backing up of configurations, 382 clearing zones, 381 conflicts with, 323–324 determining where to zone, 373–375 hard, 83–84, 374, 375–378 hardware enforced, 142 hot-plug systems and, 105–106 management of, 378–379 minimizing unwanted interactions, 381 mismatch of and missing devices, 333–334 Node and Port WWN and, 381 port-based, 83 soft, 83, 375–378, 408 storage partitioning and, 84, 205, 208–210 switchBeacon command, 302 switchDisable command, 251, 300, 343 switched fabric ports. See F_Ports switched fabric topology, 5, 37, 39, 48–49 switchEnable command, 251, 318, 343 switches, Fibre Channel, 4, 33–34, 36, 60, 63, 80–83 adding new to fabrics, 395–398 adding new to zoned SAN, 381 auto-negotiation of speed by, 88 automated management of, 369–372 automatic configuration of by Fabric OS, 133–134 Brocade SilkWorm series of. See Brocade SilkWorm switches buffer credits available per switch port, 86–87
cable layout and, 351–354 classes of service and, 84 configuration backups for, 393–394 core. See core switches Destination ID (D_ID), 302 disabled, 300 edge, 243, 248–250 equipment redundancy and, 86 exchange of information by, 49–50 fabric licenses and, 322, 366 fabric switches, 81 FSPF compliance, 90–91 host tier, 263 hubs vs., 57 in-band switch management, 356–358 IPFC broadcasting and, 88 loop operation capabilities, 90 management interfaces, 91–94 naming, 360 out-of-band switch management, 356 principal, 300 restoring configuration of, 393–394 selecting most appropriate, 124–126 self-configuring ports and, 87–88 setting parameters for, 358–359, 361 storage tier, 263 subordinate, 300 switched fabric topology and, 48–49 upgrading, 89 zoning and. See switch zoning switchShow command, 278, 279, 286 fabric troubleshooting with, 307–308, 318–319 fabric validation using, 382 SAN troubleshooting with, 297–303, 307–308 troubleshooting missing devices with, 279–280, 329 SwitchType values, 302 syslog daemon interface, Fabric OS, 135 syslogIpaAdd command, 293 syslogIpRemove command, 293
140_SANs_index
8/14/01
3:43 PM
Page 449
Index
syslogIpSho command, 293 system DRAM diagnostics, 289
T T11, 35 tape drives, 32, 33, 171 target devices, 32–33 Task Name, 294, 295 technical goals, identifying, 153 technical requirements document, 160, 190 technical requirements, identifying SAN, 159–160, 190 telnet, switch management by, 91, 92, 378 tempShow switch command, 285 10-switch core/edge SAN design, 253–254 termination, type of SCSI, 108 third-party copy backup systems, 214, 215–216 third-party copy devices, 14, 214, 215–216, 217 tiered SANs, 261–265 tiers, leveraging, 261–265 Time Server, 38, 41, 50, 52, 86 timeline, creation of development, 175–176, 190 timeout failures, edge device, 321–322 Tivoli products SANergy, 12, 212 Storage Network Manager, 350, 368 Token Ring protocol, 31 too_long error statistic, 297 topologyShow command, 307–308, 309, 320, 321, 329, 382, 395 traceShow command, 286 traffic patterns any-to-any connectivity and, 268–269 evaluating future traffic, 172–174 Fabric OS dynamic routing, 134 initiator-to-target mapping, 167–168 leveraging tiers, 261–265 localizing traffic, 266–268 pre-SANs performance data on, 169–172
449
SAN grouping process and, 178–182 transitioning, 154 transmission words, 36 Troika products HBA driver, 172 SAN Command utility, 101, 102, 312 troubleshooting fabrics, 316–327 missing devices, 279–283, 327–335 SANs (Storage Area Networks), 278–316 tools for, 287–316 20-switch core/edge SAN design, 254
U ULP information, 36 ULP mapping, 42 unbonded SC connectors, 72 unconfirmed domains, 300–301, 319 Unicenter TNG, 350, 368 UNIX dd tool for I/O generation, 388–389 format command, 312, 317 performance monitoring, 169–170 unmanaged Fibre Channel hub, 34, 35, 60 Unzoned Name Server, 51 upgrades acceptable downtime for, 174–175 fabric, 398–400, 408 Fibre Channel switch, 89 firmware, 89, 398–400 rolling with HA clusters, 197, 198–199, 202 upstream devices, 302, 303, 304 uptime switch command, 285
V /var/adm/messages file, Solaris’s, 287, 312, 317 VERITAS products Cluster Server, 11 Dynamic Multipathing, 172, 199
140_SANs_index
450
8/14/01
3:43 PM
Page 450
Index
NetBackup, 13 SANavigator, 103 SANPoint control, 103, 350, 368 Volume Manager, 12, 212 vxbench, 387 version switch command, 285 Virtual Interface (VI) protocol, 2, 15, 31, 42, 97 Visual SAN, Prisa’s, 350, 368 volume-level sharing, 19 Volume Manager,VERITAS’, 12 Vxbench,VERITAS’, 387
W Web-based router management, 109 Web-based switch management, 93–94 Web farms, storage sharing with, 206–207 Web sites Brocade, 158 Expect, 369 Intel (Iometer), 387 IOzone, 387
router management through, 109 switch management through, 93–94 WEB TOOLS, 139–140, 147, 366, 378, 408 well-known addresses, 50, 51, 52 WHQL. See Microsoft Windows Hardware Quality Lab (WHQL) Windows 2000 Advanced Server, 200 Windows 2000 Data Center, 200, 202 Windows 2000 OS, 99 Windows NT, 99, 169, 200 wiring, 61, 65–68, 164 World-Wide Names (WWNs), 307, 366 World Wide Web, data storage needs and, 9 WWN ports, 306–307 WWN spoofing, 143
X X3T11, 35
Z zero-downtime failover, 199 zoning, switch. See switch zoning
140_SANs_BM
8/14/01
3:38 PM
Page 451
140_SANs_BM
8/14/01
3:38 PM
Page 452
Additional SAN Information Resources from Brocade To learn more about SANs and SAN technologies, you can visit the Brocade Web site at www.brocade.com.This site contains a wide range of useful resources designed for many types of audiences—from high-level managers and other key decision makers to system administrators and network architects. In particular, the Brocade SAN solution center and educational Web pages offer a wealth of timely information.
Brocade SAN Solution Center Accessible at www.brocade.com/san, the Brocade SAN solution center includes a variety of information about Brocade products and SAN technologies in the form of: ■
White papers
■
Product data sheets
■
Brocade SOLUTIONware configuration summaries
■
Standards updates
■
Industry news
■
Tools for understanding Return On Investments
Brocade Education Services Accessible at www.brocade.com/education_services, Brocade education provides information about the full range of Brocade educational offerings, including: ■
Brocade SAN certification requirements
■
Course descriptions
■
Registration forms
■
Special promotions
140_SANs_BM
8/14/01
3:38 PM
Page 453
Brocade Networking Storage Conference 2002 If you want to interact with the SAN experts, just plan to attend the second annual Brocade Storage Networking Conference.Visit www.brocade.com/san to get the latest information about registration, early-bird discounts, and previews of the agenda and speakers.
140_SANs_BM
8/14/01
3:38 PM
Page 454
SYNGRESS SOLUTIONS… AVAILABLE NOW ORDER at www.syngress.com
ASP Configuration Handbook: A Guide to ISPs Internet Service Providers (ISPs) are confronted with shrinking profit margins, fierce competition, and undifferentiated products. To successfully overcome these new economic realities, ISPs have begun to expand beyond just offering bandwidth and connectivity, by providing customers with “outsource services” such as software applications and electronic commerce solutions This book is the blueprint for a successful ISP-to-ASP transition. ISBN: 1-928994-26-1 Price: $49.95 US, $77.95 CAN
AVAILABLE NOW ORDER at www.syngress.com
Administering Cisco QoS in IP Networks Administering Cisco QoS for IP Networks discusses IP Quality of Service (QoS) and how it applies to Enterprise and Internet Service Provider (ISP) environments. It reviews routing protocols and quality of service mechanisms available today on Cisco network devices and it will provide you with examples and exercises for a hands-on experience designed to give you the background to implement these capabilities in your network. ISBN: 1-928994-21-0 Price: $59.95 US, $92.95 CAN
AVAILABLE DECEMBER 2001 ORDER at www.syngress.com
Juniper Networks Internet Routing Guide Upstart Juniper Networks has created a new generation of faster routers that are extremely price competitive with similar Cisco models. Juniper Networks Internet Routing Guide is a comprehensive, hands-on guide on how to install, configure, and troubleshoot Junipers core router family. This book will immediately meet the demands of network engineers responsible for the installation and configuration of Juniper Routers. It also covers the latest “M-Class” routers, as well as configuration strategies, and the JUNOS software environment. ISBN: 1-928994-76-8 Price: $69.95 US, $108.95 CAN
[email protected]
