Ignite with Mongodb

Thanks to the previous posts, we now know how to query data in Ignite. So far, we were either persisting data in memory (in which case, it would go off as soon as the node is down) or on file-system. However, we may also use the database of our choice as the persistence layer. One such experiment that I tried out was using Mongodb.

Mongodb, as we know, is a very popular NoSQL database. In this example, I have used flapdoodle to start an embedded mongo instance. Also, I have used morphia, that is a wrapper library on monogodb for data operations. It makes our lives much easier.

Note, the way we query data on ignite doesn’t change. In fact, we are going to reuse the same methods that I have covered in my previous post. But, now onwards, it is going to interact to the database through the CacheStore class that we write.

We extend the class CacheStoreAdapter and everytime a data operation takes place, ignite is going to consult this implementation class on how to query/read, write or delete data. Please note that this class will not be called everytime, but only when we need to interact with the underlying datastore (in this case mongodb).

For our usecase, we have overridden the method load(). This method will be called whenever the IgniteCache.get method is called.

@Override
public Data load(Integer key) throws CacheLoaderException {
  Data data = morphia.find(Data.class).field("id").equal(key).get();

  System.out.println("Loaded data: " + data);

  return data;
}

Similarly, we override the write method of the class, that will be called whenever IgniteCache.put method is called.

@Override
public void write(Cache.Entry<? extends Integer, ? extends Data> e) throws CacheWriterException {
  morphia.save(e.getValue());

  System.out.println("Stored data: " + e.getValue());
}

Also, we need to override the method loadCache that will be called whenever IgniteCache.loadCache or IgniteCache.localLoadCache method is called. This is particularly important as Ignite does not load the data in memory from the underlying database (but it does for file based persistence). So we have to take care of the situation by explicitly calling IgniteCache.loadCache method.

@Override
public void loadCache(IgniteBiInClosure<String, Data> clo, Object... args){
  datastore.find(Data.class).asList()
           .forEach( x -> {

                       System.out.println("Loaded " + x.toString());

                       clo.apply(x.getId(), x);
                  });
}

The strategy of loading the data of course depends on the use case. In this case, we are loading the entire data from mongo to memory just for the purpose of demo. In a clustered environment, we should only load a subset of the data based on partition.

Finally, we set the CacheConfiguration and ask it to use the MongoCacheStore class as store.

CacheConfiguration<Integer, Data> cacheConfig = new CacheConfiguration<>(CACHE_NAME);

cacheConfig.setCacheMode(CacheMode.PARTITIONED);

cacheConfig.setIndexedTypes(Integer.class, Data.class);//Important to set indexed type.. else will throw SQL exception: table not found!

cacheConfig.setCacheStoreFactory(FactoryBuilder.factoryOf(CacheMongoStore.class));

cacheConfig.setReadThrough(true);

cacheConfig.setWriteThrough(true);

 
// Set write-behind flag for synchronous/asynchronous persistent store update

cacheConfig.setWriteBehindEnabled(WRITE_BEHIND_FLAG);

 
IgniteCache<String, Data> cache = ignite.getOrCreateCache(cacheConfig);

  if(cache.size() <= 0)

    cache.localLoadCache(null);

And we are all set to go!

Note

From here onwards, we are going to use the same scanqueries and the textqueries to query data from ignite. It is ignite who decides when to call the respective methods of the CacheStore. The entire code is available in github (link in reference) for fork and to play around.

PEACE!!

Reference

1. https://github.com/talk2sjb/ignite-example

2. https://github.com/gridgain/gridgain-advanced-examples/