org.duracloud.services.hadoop.base

Class SimpleFileRecordReader

java.lang.Object

org.duracloud.services.hadoop.base.SimpleFileRecordReader

All Implemented Interfaces:

org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Direct Known Subclasses:

ICRecordReader

public class SimpleFileRecordReader
extends Object
implements org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Record reader, used to provide a set of key/value pairs for each file in a file split. This reader assumes the file split is a single file and creates one key/value pair per file where: key = the file path value = the output path (for results)

Field Summary

Fields

Modifier and Type	Field and Description
protected org.apache.hadoop.mapred.JobConf	jobConf
protected org.apache.hadoop.mapred.Reporter	reporter

Constructor Summary

Constructors

Constructor and Description
SimpleFileRecordReader(org.apache.hadoop.mapred.FileSplit inputSplit, org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.mapred.Reporter reporter)

Method Summary

Methods

Modifier and Type	Method and Description
void	close()
org.apache.hadoop.io.Text	createKey() Create an empty Text object in which the key can be stored
org.apache.hadoop.io.Text	createValue() Create an empty Text object in which the value can be stored
protected String	getOutputPath()
long	getPos()
float	getProgress()
boolean	next(org.apache.hadoop.io.Text key, org.apache.hadoop.io.Text value) Store the next record to be processed in the provided key/value objects.
protected boolean	verifyProcessFile(String filePath) Verifies that the file at the provided file path should, indeed, be processed.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

jobConf

protected org.apache.hadoop.mapred.JobConf jobConf

reporter

protected org.apache.hadoop.mapred.Reporter reporter

Constructor Detail

SimpleFileRecordReader

public SimpleFileRecordReader(org.apache.hadoop.mapred.FileSplit inputSplit,
                      org.apache.hadoop.mapred.JobConf jobConf,
                      org.apache.hadoop.mapred.Reporter reporter)

Method Detail

next

public boolean next(org.apache.hadoop.io.Text key,
           org.apache.hadoop.io.Text value)
             throws IOException

Store the next record to be processed in the provided key/value objects. Returns true the first time the method is called for a given file and false thereafter.

Specified by:

next in interface org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Throws:

IOException

getOutputPath

protected String getOutputPath()

verifyProcessFile

protected boolean verifyProcessFile(String filePath)

Verifies that the file at the provided file path should, indeed, be processed. This method is provided for purposes of being overridden to specify when a file should or should not be passed along to the mapper for processing. The default implementation always returns true.

Parameters:

filePath - the path of the file to be verified

Returns:

true if the file should be processed by the mapper, false otherwise

createKey

public org.apache.hadoop.io.Text createKey()

Create an empty Text object in which the key can be stored

Specified by:

createKey in interface org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

createValue

public org.apache.hadoop.io.Text createValue()

Create an empty Text object in which the value can be stored

Specified by:

createValue in interface org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

getPos

public long getPos()
            throws IOException

Specified by:

getPos in interface org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Throws:

IOException

close

public void close()
           throws IOException

Specified by:

close in interface org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Throws:

IOException

getProgress

public float getProgress()
                  throws IOException

Specified by:

getProgress in interface org.apache.hadoop.mapred.RecordReader<org.apache.hadoop.io.Text,org.apache.hadoop.io.Text>

Throws:

IOException