BufferedIterator

The idea is to process an item or a set of items after another and behind the scenes make only the necessary/minimum calls to load those items without worrying about it. If I want to process 1000 videos, I surely don't want to have to make 1000 web service calls but I also do not want to load 1000 videos at once.



It would look like this:

for (Iterator<MyItem> i = .....; i.hasNext(); ) {
 MyItem oItem = i.next();
 ....process oItem....
}

I can batch them by say 10 or 20. It would look like this:

int oFrom = 1;
int oTo = 10;
int oInc = 10;
FindParameters oParams = new FindParameters(oFrom, oTo).with(...some filters...).sortBy(....some sort...);
Findings<Video> oFindings = mVideoBO.find(oParams);
while (!oFindings.getResults().isEmpty()) {
 for (Video v : oFindings.getResults()) {
  ...process item...
 }
 oFrom += oInc;
 oTo += oInc;
 oParams.from(oFrom).to(oTo);
 oFindings = mVideoBO.find(oParams);
}

That's a lot of code: I need over 10 lines of error-prone code and logic before processing any item. It's difficult to maintain and impossible to reuse (without a copy-paste-tweak).

The simple idea is to refactor all this and end up with something like the following:

FindParameters oParams = new FindParameters().with(...some filters...).sortBy(....some sort...);
mVideoBO.findAndProcess(oParams, new IProcessor<Video>() {
 private void process(final Video pItem) {
  ...process item...
 }
});

There are different ways to accomplish this and I chose to work with the iterator and lazy loading.
I first define my own extension to Iterator<T>:

public interface IIterator<T> extends Iterator<T> {}

I then create my PaginationIterator<T> which basically gets more items at every call to next():

public abstract class PaginationIterator<T> implements IIterator<T> {
 private long mTo;
 private int mIncrement;
 private long mCursor;
 
 public PaginationIterator(long pFrom, long pTo, int pIncrement) {
  mTo = pTo;
  mIncrement = pIncrement;
  mCursor = pFrom;
 }

 public T next() {
  if (!hasNext()) return null;
  long oFrom = mCursor;
  long oTo = Math.min(mTo, mCursor + mIncrement - 1);  
  T oItems = getItems(oFrom, oTo);
  mCursor += mIncrement;
  return oItems;
 }
 
 public boolean hasNext() {
  return (mCursor < mTo);  
 }
 
 public void remove() {
  throw new RuntimeException("Not supported");
 }
 
 protected abstract T getItems(long pFrom, long pTo);
 
 protected void setTo(int pValue) {
  mTo = pValue;
 }
 
 public long getCursor() {
  return mCursor;
 }
 public int getIncrement() {
  return mIncrement;
 }
 public long getTo() {
  return mTo;
 }
}

I can now create a concrete extension of that iterator, specifying how the items are loaded given a pFrom and pTo.
I have now something like the following:

for (Iterator<List<Video>> i = new MyVideoIterator(1, 100, 10); i.hasNext(); ) {
 List<Video> o10Videos = i.next();
 for (Video v : o10Videos) {
  ....process v....
 }
}

I'm here processing a set of items instead of each item individually. The next step is therefore to create another iterator reusing the logic of that PaginationIterator:

public class BufferedIterator<T, U extends Iterable<T>> implements IIterator<T> {
 
 private Iterator<T> mCurrentBufferedIterator;
 private IIterator<U> mIterator;
 
 public BufferedIterator(IIterator<U> pIterator) {
  mIterator = pIterator;
 }
 
 public boolean hasNext() {
  return mIterator.hasNext() || (mCurrentBufferedIterator != null && mCurrentBufferedIterator.hasNext());
 }
 
 public T next() {
  if ((mCurrentBufferedIterator == null || !mCurrentBufferedIterator.hasNext()) && hasNext()) getNextBatch(mIterator.next());
  return mCurrentBufferedIterator == null ? null : mCurrentBufferedIterator.next();
 }
 
 private void getNextBatch(U pNext) {
  if (pNext == null) return;
  mCurrentBufferedIterator = pNext.iterator();
 }
 
 public void remove() {
  throw new RuntimeException("Not supported.");
 }
}

With this we now have:

for (Iterator<Video> i = new BufferedIterator<Video>(new MyPaginationIterator(1, 100, 10)); i.hasNext(); ) {
 Video v = i.next();
 ....process v....
}

I can now embed the whole thing in a method of my BOs:

public void findAndProcess(final FindParameters pParams, final IProcessor<T> pProcessor) throws CoreException {
 MyPaginationIterator<T> oIterator = new MyPaginationIterator<T>(this, pParams, 1, 100, 10);
 BufferedIterator<T, List<T>> oBufferedIterator = new BufferedIterator<T, List<T>>(oIterator);
 for (Iterator<T> i = oBufferedIterator; i.hasNext;) {
  pProcessor.process(i.next());
 }
}

And I finally end up with what I wanted earlier:

FindParameters oParams = new FindParameters().with(...).sortBy(...);
mVideoBO.findAndProcess(oParams, new IProcessor<Video>() {
 private void process(final Video pItem) {
  ...process item...
 }
});