Package org.apache.accumulo.core.iterators.user

Class IntersectingIterator

  • All Implemented Interfaces:
    SortedKeyValueIterator<Key,​Value>, YieldingKeyValueIterator<Key,​Value>
    Direct Known Subclasses:
    IndexedDocIterator
    public class IntersectingIterator
extends Object
implements SortedKeyValueIterator<Key,​Value>
    This iterator facilitates document-partitioned indexing. It involves grouping a set of documents together and indexing those documents into a single row of an Accumulo table. This allows a tablet server to perform boolean AND operations on terms in the index. The table structure should have the following form: row: shardID, colfam: term, colqual: docID When you configure this iterator with a set of terms (column families), it will return only the docIDs that appear with all of the specified terms. The result will have an empty column family, as follows: row: shardID, colfam: (empty), colqual: docID This iterator is commonly used with BatchScanner or AccumuloInputFormat, to parallelize the search over all shardIDs. This iterator will *ignore* any columnFamilies passed to seek(Range, Collection, boolean) as it performs intersections over terms. Extending classes should override the IntersectingIterator.TermSource.seekColfams in their implementation's init(SortedKeyValueIterator, Map, IteratorEnvironment) method. An example of using the IntersectingIterator is available at https://github.com/apache/accumulo-examples/blob/master/docs/shard.md

    • Field Detail

      • nullText

        protected org.apache.hadoop.io.Text nullText

      • currentPartition

        protected org.apache.hadoop.io.Text currentPartition

      • currentDocID

        protected org.apache.hadoop.io.Text currentDocID

      • topKey

        protected Key topKey

      • value

        protected Value value

    • Constructor Detail

      • IntersectingIterator

        public IntersectingIterator()

    • Method Detail

      • getPartition

        protected org.apache.hadoop.io.Text getPartition​(Key key)

      • getTerm

        protected org.apache.hadoop.io.Text getTerm​(Key key)

      • getDocID

        protected org.apache.hadoop.io.Text getDocID​(Key key)

      • buildKey

        protected Key buildKey​(org.apache.hadoop.io.Text partition,
                       org.apache.hadoop.io.Text term)

      • buildKey

        protected Key buildKey​(org.apache.hadoop.io.Text partition,
                       org.apache.hadoop.io.Text term,
                       org.apache.hadoop.io.Text docID)

      • buildFollowingPartitionKey

        protected Key buildFollowingPartitionKey​(Key key)

      • deepCopy

        public SortedKeyValueIterator<Key,​Value> deepCopy​(IteratorEnvironment env)
        Description copied from interface: SortedKeyValueIterator
        Creates a deep copy of this iterator as though seek had not yet been called. init should be called on an iterator before deepCopy is called. init should not need to be called on the copy that is returned by deepCopy; that is, when necessary init should be called in the deepCopy method on the iterator it returns. The behavior is unspecified if init is called after deepCopy either on the original or the copy. A proper implementation would call deepCopy on the source.
        Specified by:
        deepCopy in interface SortedKeyValueIterator<Key,​Value>
        Parameters:
        env - IteratorEnvironment environment in which iterator is being run.
        Returns:
        SortedKeyValueIterator a copy of this iterator (with the same source and settings).

      • getTopKey

        public Key getTopKey()
        Description copied from interface: SortedKeyValueIterator
        Returns top key. Can be called 0 or more times without affecting behavior of next() or hasTop(). Note that in minor compaction scope and in non-full major compaction scopes the iterator may see deletion entries. These entries should be preserved by all iterators except ones that are strictly scan-time iterators that will never be configured for the minc or majc scopes. Deletion entries are only removed during full major compactions.

        For performance reasons, iterators reserve the right to reuse objects returned by getTopKey when SortedKeyValueIterator.next() is called, changing the data that the object references. Iterators that need to save an object returned by getTopKey ought to copy the object's data into a new object in order to avoid aliasing bugs.

        Specified by:
        getTopKey in interface SortedKeyValueIterator<Key,​Value>
        Returns:
        K

      • getTopValue

        public Value getTopValue()
        Description copied from interface: SortedKeyValueIterator
        Returns top value. Can be called 0 or more times without affecting behavior of next() or hasTop().

        For performance reasons, iterators reserve the right to reuse objects returned by getTopValue when SortedKeyValueIterator.next() is called, changing the underlying data that the object references. Iterators that need to save an object returned by getTopValue ought to copy the object's data into a new object in order to avoid aliasing bugs.

        Specified by:
        getTopValue in interface SortedKeyValueIterator<Key,​Value>
        Returns:
        V

      • hasTop

        public boolean hasTop()
        Description copied from interface: SortedKeyValueIterator
        Returns true if the iterator has more elements. Note that if this iterator has yielded (@see YieldingKeyValueIterator.enableYielding(YieldCallback)), this this method must return false.
        Specified by:
        hasTop in interface SortedKeyValueIterator<Key,​Value>
        Returns:
        true if the iterator has more elements.

      • next

        public void next()
          throws IOException
        Description copied from interface: SortedKeyValueIterator
        Advances to the next K,V pair. Note that in minor compaction scope and in non-full major compaction scopes the iterator may see deletion entries. These entries should be preserved by all iterators except ones that are strictly scan-time iterators that will never be configured for the minc or majc scopes. Deletion entries are only removed during full major compactions.
        Specified by:
        next in interface SortedKeyValueIterator<Key,​Value>
        Throws:
        IOException - if an I/O error occurs.

      • encodeColumns

        protected static String encodeColumns​(org.apache.hadoop.io.Text[] columns)
        Returns:
        encoded columns

      • encodeBooleans

        protected static String encodeBooleans​(boolean[] flags)
        Returns:
        encoded flags

      • decodeColumns

        protected static org.apache.hadoop.io.Text[] decodeColumns​(String columns)

      • decodeBooleans

        protected static boolean[] decodeBooleans​(String flags)
        Returns:
        decoded flags

      • seek

        public void seek​(Range range,
                 Collection<ByteSequence> seekColumnFamilies,
                 boolean inclusive)
          throws IOException
        Description copied from interface: SortedKeyValueIterator
        Seeks to the first key in the Range, restricting the resulting K,V pairs to those with the specified columns. An iterator does not have to stop at the end of the range. The whole range is provided so that iterators can make optimizations. Seek may be called multiple times with different parameters after SortedKeyValueIterator.init(org.apache.accumulo.core.iterators.SortedKeyValueIterator<K, V>, java.util.Map<java.lang.String, java.lang.String>, org.apache.accumulo.core.iterators.IteratorEnvironment) is called. Iterators that examine groups of adjacent key/value pairs (e.g. rows) to determine their top key and value should be sure that they properly handle a seek to a key in the middle of such a group (e.g. the middle of a row). Even if the client always seeks to a range containing an entire group (a,c), the tablet server could send back a batch of entries corresponding to (a,b], then reseek the iterator to range (b,c) when the scan is continued. columnFamilies is used, at the lowest level, to determine which data blocks inside of an RFile need to be opened for this iterator. This set of data blocks is also the set of locality groups defined for the given table. If no columnFamilies are provided, the data blocks for all locality groups inside of the correct RFile will be opened and seeked in an attempt to find the correct start key, regardless of the startKey in the range. In an Accumulo instance in which multiple locality groups exist for a table, it is important to ensure that columnFamilies is properly set to the minimum required column families to ensure that data from separate locality groups is not inadvertently read.
        Specified by:
        seek in interface SortedKeyValueIterator<Key,​Value>
        Parameters:
        range - Range of keys to iterate over.
        seekColumnFamilies - Collection of column families to include or exclude.
        inclusive - boolean that indicates whether to include (true) or exclude (false) column families.
        Throws:
        IOException - if an I/O error occurs.

      • setColumnFamilies

        public static void setColumnFamilies​(IteratorSetting cfg,
                                     org.apache.hadoop.io.Text[] columns)
        Encode the columns to be used when iterating.

      • setColumnFamilies

        public static void setColumnFamilies​(IteratorSetting cfg,
                                     org.apache.hadoop.io.Text[] columns,
                                     boolean[] notFlags)
        Encode columns and NOT flags indicating which columns should be negated (docIDs will be excluded if matching negated columns, instead of included).