s3tail package

Submodules

s3tail.background_writer module

class s3tail.background_writer.BackgroundWriter(writer, done_callback=None)

Bases: threading.Thread

exception WriteAfterDone

Bases: exceptions.Exception

Indicates when an action is taken after requested to stop.

BackgroundWriter.join(timeout=None)

BackgroundWriter.mark_done()

BackgroundWriter.run()

BackgroundWriter.write(data)

s3tail.cache module

class s3tail.cache.Cache(path, hours)

Bases: future.types.newobject.newobject

cleanup()

lookup(name)

open(name, reader)

readers = []

s3tail.cli module

s3tail.old_file_cleaner module

class s3tail.old_file_cleaner.OldFileCleaner(path, hours)

Bases: threading.Thread

run()

s3tail.s3tail module

Utility to help “tail” AWS logs stored in S3 generated by S3 bucket logging or ELB logging.

class s3tail.s3tail.S3Tail(config, bucket_name, prefix, line_handler, key_handler=None, bookmark=None, region=None, cache_path=None, hours=24)

Bases: future.types.newobject.newobject

An object that wraps the process of downloading and extracting lines from S3 files.

Upon creation of the tail, the caller can next invoke S3Tail.watch() to begin the process of downloading files from S3 (or, opening them from the local file system cache) and invoking the provided line_handler to allow the caller to process each line in the file.

Parameters:

• config – the configuration wrapper for saving bookmarks
• bucket_name – the name of the S3 bucket from which files will be downloaded
• prefix – what objects in the S3 bucket should be matched
• line_handler – a function that will expect to be called for each line found in the downloaded files
• key_handler – a function that will expect to be called for each file
• bookmark – a location or name for where to pick up from a previous run
• region – a region to use when connection to the S3 bucket
• cache_path – the path for where the cache should live (None will disable caching)
• hours – the number of hours to keep files in the cache (0 will disable caching)

BUFFER_SIZE = 1048576

Describes the number of bytes to read into memory when parsing lines.

MAX_BUFFER_SIZE = 5242880

Describes the maximum amount of buffer to read into memory when parsing lines.

exception MismatchedPrefix

Bases: exceptions.Exception

Indicates when a prefix is provided that does not overlap with the requested bookmark.

S3Tail.cleanup()

Wait on any threads remaining and cleanup any unflushed state or configuration.

S3Tail.get_bookmark()

Get a bookmark to represent the current location.

S3Tail.stop(*args)

Request that a running watch should terminate processing at the next earliest convenience.

This can be most useful if the tail is running in a separate thread and/or if the caller is trying to process an interrupt condition (i.e. from a signal or keyboard request). The arguments are ignored and allow this to be directly passed in as a signal handler:

signal.signal(signal.SIGPIPE, tail.stop)

S3Tail.watch()

Begin watching and reporting lines read from S3.

This call will not return until all the files are read and processed or until a callback indicates the need to terminate processing early.

Before reading each file, the optional key_handler provided when created will be invoked with the name of the S3 key. If the key_handler returns a “falsey” value the key will be skipped and the tail will move on to the next key.

For every line parsed from the files found in S3, the line_handler provided when created will be invoked passing along the line number and line to the callback. If the line_handler returns a result (i.e. if it is not None), processessing is terminated and the result will be returned from the call to watch.

Module contents