HDFS Architecture

The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems.However, the differences from other distributed file systems are significant. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware. HDFS provides high throughput access to application data and is suitable for applications that have large datasets. HDFS relaxes a few POSIX requirements to enable streaming access to file system data. HDFS was originally built as infrastructure for the Apache Nutch web search engine project. HDFS is now an Apache Hadoop subproject.


HDFS has a master/slave architecture. An HDFS cluster consists of a single Name Node, a master server that manages the file system namespace and regulates access to files by clients. In addition, there are a number of DataNodes, usually one per node in the cluster, which manage storage attached to the nodes that they run on. HDFS exposes a file system namespace and allows user data to be stored in files. Internally, a file is split into one or more blocks and these blocks are stored in a set of Data Nodes. The NameNode executes file system namespace operations like opening, closing, and renaming files and directories.It also determines the mapping of blocks to Data Nodes. The Data Nodes are responsible for serving read and write requests from the file system’s clients. The Data Nodes also perform block creation, deletion, and replication upon instruction from the Name Node.


The NameNode and DataNode are pieces of software designed to run on commodity machines. These machines typically run a GNU/Linux operating system (OS). HDFS is built using the Java language; any machine that supports Java can run the Name Node or the Data Node software. Usage of the highly portable Java language means that HDFS can be deployed on a wide range of machines. A typical deployment has a dedicated machine that runs only the Name Node software. Each of the other machines in the cluster runs one instance of the Data Node software. The architecture does not preclude running multiple Data Nodes on the same machine but in a real deployment that is rarely the case.
The existence of a single Name Node in a cluster greatly simplifies the architecture of the system. The Name Node is the arbitrator and repository for all HDFS metadata. The system is designed in such a way that user data never flows through the NameNode

Comments

Post a Comment

Popular posts from this blog

WES Verification Process for GTU Students

Image
WES Credentials are required for Canada Immigration. The following are the hassle-free steps to get WES Credential for the GTU Student. Step 1: Create WES Account  and register for the credentials verification. Where will you use your evaluation? Canada Select an Application ECA Application for IRCC Provide necessary personal details and create an account. Fill up the Personal Information Add credentials under Your Education.  Take care of the below points while adding credentials. If you had lateral entry into the second year (or third semester) of a Bachelor of Technology or Bachelor of Engineering, you must also provide documentation for prior study, e.g. Diploma in Engineering from a State Board of Technical Education. For applicants submitting a Master’s Degree or Postgraduate Diploma for evaluation, you must also provide your bachelor’s degree documents. WES cannot complete evaluation without these documents. The only exceptions where your bache...
Read more

Introduction to Apache Hadoop

Hadoop is an open source framework for writing and running distributed applications that process large amounts of data. Distributed computing is a wide and varied field, but the key distinctions of Hadoop are that it is : Accessible : Hadoop runs on large clusters of commodity machines or on cloud computing services such as Amazon’s Elastic Compute Cloud (EC2). Robust : because it is intended to run on commodity hardware, Hadoop is architected with the assumption of frequent hardware malfunctions. It can gracefully handle most such failures. Scalable : Hadoop scales linearly to handle larger data by adding more nodes to the cluster. Simple : Hadoop allows users to quickly write efficient parallel code. Hadoop’s accessibility and simplicity give it an edge over writing and running large distributed programs. Even college students can quickly and cheaply create their own Hadoop cluster. On the other hand, its robustness and scalability make it suitable for even the most dem...
Read more