OS/embedded concepts

Storage System Architecture

Shashi Shekhar's photo
Shashi Shekhar
·

4 min read

1. Basics of Storage Systems

Definition

A storage system is a collection of hardware and software resources designed to store, manage, and retrieve data efficiently and reliably.

Key Components

  1. Storage Media:

    • Hard Disk Drives (HDDs): Traditional magnetic storage.

    • Solid-State Drives (SSDs): Faster, non-volatile memory-based storage.

    • Optical Discs: CDs, DVDs for archival purposes.

    • Tape Drives: Used for backups and archiving.

  2. Storage Controllers:

    • Manage data flow between the storage media and the host system.

    • Examples: RAID controllers, NVMe controllers.

  3. Interfaces:

    • Connect storage devices to the host system.

    • Examples:

      • SATA, SAS (Serial Attached SCSI) for HDDs and SSDs.

      • NVMe (Non-Volatile Memory Express) for high-speed SSDs.

      • Fibre Channel, iSCSI for networked storage.

  4. Protocols:

    • Define communication rules between storage and host.

    • Examples: SCSI, ATA, NVMe, SMB (Server Message Block).

Categories of Storage

  1. Primary Storage:

    • Volatile memory, like RAM.

    • Directly accessible by the CPU.

    • Used for active processes.

  2. Secondary Storage:

    • Non-volatile memory, like HDDs and SSDs.

    • Stores data persistently.

  3. Tertiary Storage:

    • Archival and backup storage.

    • Includes tape drives and optical media.

  4. Quaternary Storage:

    • Remote storage, such as cloud storage.

2. Storage System Architectures

1. Direct Attached Storage (DAS)

  • Description:

    • Storage devices are directly connected to a host (e.g., via SATA or SAS).

    • No network is involved.

  • Advantages:

    • High performance for individual systems.

    • Simplicity and cost-effectiveness.

  • Disadvantages:

    • Limited scalability and sharing.

2. Network Attached Storage (NAS)

  • Description:

    • A dedicated device or appliance that provides file-based storage over a network.

    • Uses protocols like NFS (Linux), SMB/CIFS (Windows).

  • Advantages:

    • Centralized file storage and sharing.

    • Easy management and backup.

  • Disadvantages:

    • May face performance bottlenecks on busy networks.

3. Storage Area Network (SAN)

  • Description:

    • A high-speed network that provides block-level storage to multiple servers.

    • Uses Fibre Channel, iSCSI, or FCoE (Fibre Channel over Ethernet).

  • Advantages:

    • High performance and scalability.

    • Suitable for large-scale, enterprise environments.

  • Disadvantages:

    • Complex setup and high cost.

4. Object Storage

  • Description:

    • Stores data as objects, each with metadata and a unique identifier.

    • Examples: Amazon S3, OpenStack Swift.

  • Advantages:

    • Scalability for unstructured data.

    • Best for cloud storage and big data.

  • Disadvantages:

    • Not optimized for low-latency random access.

3. File Systems

Definition

A file system organizes and manages data on storage devices, making it accessible to users and applications.

Common File Systems

  1. FAT32:

    • Legacy system for compatibility across devices.

  2. NTFS:

    • Advanced file system for Windows, supporting permissions, encryption.

  3. EXT4:

    • Linux file system known for performance and reliability.

  4. ZFS:

    • Advanced file system with features like snapshots, RAID, and data integrity checks.

  5. Btrfs:

    • Modern Linux file system focusing on scalability and fault tolerance.

4. Advanced Concepts

1. RAID (Redundant Array of Independent Disks)

  • Combines multiple disks for redundancy and/or performance.

  • Levels:

    • RAID 0: Striping (performance).

    • RAID 1: Mirroring (redundancy).

    • RAID 5: Striping with parity (balance of redundancy and performance).

2. Virtualization

  • Abstracts physical storage into logical storage pools.

  • Examples: VMware vSAN, Microsoft Storage Spaces.

3. Tiered Storage

  • Automatically moves data between different types of storage based on usage.

  • Examples: Hot storage (SSD) for frequently accessed data, cold storage (HDD) for archival.

4. Software-Defined Storage (SDS)

  • Storage functionality is implemented in software rather than hardware.

  • Examples: Ceph, GlusterFS.

5. Cloud Storage

  • Off-premise storage provided over the internet.

  • Examples: Amazon S3, Google Cloud Storage, Azure Blob Storage.

6. Data Deduplication and Compression

  • Reduces storage usage by eliminating duplicate copies of data and compressing files.

7. NVMe and NVMe-over-Fabrics

  • NVMe is a protocol designed for SSDs, offering high-speed data access.

  • NVMe-oF extends NVMe over network fabrics like Ethernet or Fibre Channel.

5. Security in Storage Systems

  • Encryption: Encrypting data at rest and in transit.

  • Access Control: Restricting access to authorized users.

  • Snapshots and Backups: Regular snapshots for quick recovery.

  • Data Integrity: Checksums to prevent corruption.