No npy files after extracting lmdb using vilbert configuration

After unpacking with lmdb feature, there are two files shown under the following directory
datasets/hateful_memes/defaults/features/detectron.lmdb
14664257536 May 6 12:29 data.mdb
8192 May 6 12:29 lock.mdb

However, the feature reader expects individual files to be read in its
def get(self, item) function
in antmmf/datasets/database/features_database.py
e.g., its item input is
item = {‘id’: 14238, ‘img’: ‘img/14238.png’, ‘label’: 0, ‘text’: ‘most religions are religions of peace’, ‘feature_path’: ‘14238.npy’}
but there is no npy files

This file helps to generate npy files: https://github.com/Muennighoff/vilio/blob/master/fts_lmdb/lmdb_conversion.py

1 Like