webhdfspy¶

A Python wrapper library to access Hadoop WebHDFS REST API

Installation¶

To install webhdfspy from PyPI:

$ pip install webhdfspy

Python versions¶

webhdfspy requires Python 3.9+

Usage¶

>>> import webhdfspy
>>> client = webhdfspy.WebHDFSClient("localhost", 50070, "username")
>>> print(client.listdir('/'))
[]
>>> client.mkdir('/foo')
True
>>> print(client.listdir('/'))
[{'group': 'supergroup', 'permission': '755', ...}]
>>> client.create('/foo/foo.txt', "just put some text here", overwrite=True)
True
>>> print(client.open('/foo/foo.txt'))
just put some text here
>>> client.remove('/foo')
True

Using a context manager:

>>> with webhdfspy.WebHDFSClient("localhost", 50070, "username") as client:
...     client.listdir('/')
[]

HTTPS support:

>>> client = webhdfspy.WebHDFSClient("host", 9871, "user", scheme="https")

Binary and large files¶

open() returns text. Use read() for binary data, and stream() or copytolocal() for files too large to hold in memory:

>>> client.read('/data/image.png')
b'\x89PNG\r\n...'
>>> with client.stream('/data/huge.parquet') as chunks:
...     for chunk in chunks:
...         process(chunk)
>>> client.copytolocal('/data/huge.parquet', 'local.parquet')
True

High availability¶

Pass every namenode and the client fails over when the active one is standby or unreachable:

>>> client = webhdfspy.WebHDFSClient(["nn1", "nn2"], 9870)

Secured clusters¶

auth accepts any requests auth handler, and verify/cert control TLS:

>>> from requests_kerberos import HTTPKerberosAuth
>>> client = webhdfspy.WebHDFSClient(
...     "host", 9871, scheme="https",
...     auth=HTTPKerberosAuth(),
...     verify="/etc/pki/ca-trust/ca-bundle.crt",
... )

Delegation tokens authenticate subsequent requests:

>>> token = client.get_delegation_token("renewer")
>>> client.set_delegation_token(token["urlString"])

API Documentation¶

Client for Hadoop WebHDFS REST API.

Supports context manager protocol for automatic resource cleanup:

with WebHDFSClient("host", 50070, username="user") as client:
    client.listdir("/")

For an HA cluster, pass every namenode; the client transparently fails over to the next one when the active namenode is standby or unreachable:

WebHDFSClient(["nn1.example.com", "nn2.example.com"], 9870)

append(path: str, file_data: Any, buffersize: int | None = None, encoding: str = 'utf-8') → bool¶

Append data to a file.

Parameters:

path – path of the file
file_data – data to append, as text, bytes or a file object
buffersize – size of the buffer used to transfer the data
encoding – codec used to encode file_data when it is text

cancel_delegation_token(token: str) → bool¶

Cancel a delegation token.

Parameters:: token – the delegation token

chmod(path: str, permission: str) → bool¶

Set the permissions of a file or directory.

Parameters:

path – path of the file/dir
permission – permissions in octal (e.g. "755")

close() → None¶: Close the underlying HTTP session, unless it was supplied by the caller.

copyfromlocal(local_path: str, hdfs_path: str, overwrite: bool | None = None) → bool¶

Copy a file from the local filesystem to HDFS.

Parameters:

local_path – path of the local file
hdfs_path – HDFS destination path
overwrite – whether to overwrite an existing file

copytolocal(hdfs_path: str, local_path: str, chunk_size: int = 65536) → bool¶

Download a file from HDFS to the local filesystem.

The file is streamed, so its size is not limited by available memory.

Parameters:

hdfs_path – path of the HDFS file
local_path – local destination path
chunk_size – number of bytes transferred per chunk

create(path: str, file_data: Any, overwrite: bool | None = None, encoding: str = 'utf-8') → bool¶

Create a new file in HDFS.

Uses the two-step WebHDFS create protocol (NameNode redirect then DataNode upload).

Parameters:

path – the file path to create
file_data – the data to write, as text, bytes or a file object
overwrite – whether to overwrite an existing file
encoding – codec used to encode file_data when it is text

environ_home() → str¶: Return the home directory of the user.

get_checksum(path: str) → dict[str, Any]¶

Return the checksum of a file.

Parameters:: path – path of the file
Returns:: FileChecksum dict

get_content_summary(path: str) → dict[str, Any]¶

Return the content summary of a directory.

Parameters:: path – path of the directory
Returns:: ContentSummary dict

get_delegation_token(renewer: str) → dict[str, Any]¶

Get a delegation token.

Pass the token’s urlString to set_delegation_token() (or the token constructor argument) to authenticate with it.

Parameters:: renewer – the user who can renew the token
Returns:: Token dict

listdir(path: str = '/') → list[dict[str, Any]]¶

List all the contents of a directory.

Parameters:: path – path of the directory
Returns:: a list of FileStatus dicts

mkdir(path: str, permission: str | None = None) → bool¶

Create a directory hierarchy, like mkdir -p.

Parameters:

path – the path of the directory
permission – dir permissions in octal (e.g. "755")

property namenode_url: str¶: Base URL of the namenode currently believed to be active.

open(path: str, offset: int | None = None, length: int | None = None, buffersize: int | None = None, encoding: str = 'utf-8') → str¶

Open a text file and return its contents as a string.

Use read() for binary files and stream() or copytolocal() for files too large to hold in memory.

Parameters:

path – path of the file
offset – starting byte position
length – number of bytes to read
buffersize – size of the buffer used to transfer the data
encoding – codec used to decode the data

Returns:

the file data as text

read(path: str, offset: int | None = None, length: int | None = None, buffersize: int | None = None) → bytes¶

Read a file and return its contents as bytes.

Parameters:

path – path of the file
offset – starting byte position
length – number of bytes to read
buffersize – size of the buffer used to transfer the data

Returns:

the file data as bytes

remove(path: str, recursive: bool = False) → bool¶

Delete a file or directory.

Parameters:

path – path of the file or dir to delete
recursive – delete content in subdirectories

rename(src: str, dst: str) → bool¶

Rename a file or directory.

Parameters:

src – path of the file or dir to rename
dst – destination path

renew_delegation_token(token: str) → int¶

Renew a delegation token.

Parameters:: token – the delegation token
Returns:: new expiration time in ms since epoch

set_delegation_token(token: str | None) → None¶

Authenticate subsequent requests with a delegation token.

Pass None to stop sending a token.

Parameters:: token – the urlString of a token from get_delegation_token()

set_owner(path: str, owner: str | None = None, group: str | None = None) → bool¶

Set the owner and/or group of a file or directory.

Parameters:

path – path of the file/dir
owner – new owner name
group – new group name

set_replication(path: str, replication_factor: int) → bool¶

Set the replication factor of a file.

Parameters:

path – path of the file
replication_factor – number of replications (>0)

set_times(path: str, modificationtime: int | None = None, accesstime: int | None = None) → bool¶

Set modification and/or access time of a file.

Parameters:

path – path of the file
modificationtime – modification time in ms since epoch
accesstime – access time in ms since epoch

status(path: str) → dict[str, Any]¶

Return the FileStatus of a file or directory.

Parameters:: path – path of the file/dir
Returns:: a FileStatus dictionary

stream(path: str, offset: int | None = None, length: int | None = None, buffersize: int | None = None, chunk_size: int = 65536) → Iterator[Iterator[bytes]]¶

Stream a file’s contents without holding it all in memory.

Yields an iterator of byte chunks:

with client.stream("/big.bin") as chunks:
    for chunk in chunks:
        process(chunk)

Parameters:

path – path of the file
offset – starting byte position
length – number of bytes to read
buffersize – size of the buffer used to transfer the data
chunk_size – number of bytes yielded per chunk

Exceptions¶

class webhdfspy.WebHDFSException(msg: str)¶: Base exception for WebHDFS errors.

class webhdfspy.WebHDFSRemoteException(message: str, status_code: int, exception: str = '', java_class_name: str = '')¶: Exception raised when WebHDFS returns a RemoteException.

class webhdfspy.WebHDFSConnectionError(msg: str, cause: Exception | None = None)¶: Exception raised when a connection to WebHDFS fails.

WebHDFS documentation¶

https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/WebHDFS.html

webhdfspy¶

Installation¶

Python versions¶

Usage¶

Binary and large files¶

High availability¶

Secured clusters¶

API Documentation¶

Exceptions¶

WebHDFS documentation¶

Indices and tables¶

webhdfspy

Navigation

Related Topics