Zi@SSKrSSKrSSKrSSKrSSKJrJrJrJ r J r J r SSK J r "SS5r"SS\5r\R "5rS S S \S 34H)q\R&R)[$5(dM) O S qSr"SS\5rSr"SS\5r"SS\5r"SS\5r"SS\5r"SS\5r"SS\5rSSjrg) N) fvecs_read ivecs_read bvecs_mmap fvecs_mmap bvecs_iterbvecs_iter_chunked)knnc^\rSrSrSrSrSrSSjrSrSSjr SS jr SS jr S r S r S rg)Datasetz*Generic abstract class for a test dataset cJSUlSUlSUlSUlSUlg)z1the constructor should set the following fields: L2Ndmetricnqnbntselfs p/var/www/eduai.edurigo.com/question_generate/ques_gen_env/lib/python3.13/site-packages/faiss/contrib/datasets.py__init__Dataset.__init__s% c[5e)z&return the queries as a (nq, d) array NotImplementedErrorrs r get_queriesDataset.get_queries !##rNc[5e)z&return the queries as a (nt, d) array rrmaxtrains r get_trainDataset.get_trainr"rc[5e)z&return the queries as a (nb, d) array rrs r get_databaseDataset.get_database#r"rc## UR5nUupEURU-U-URUS--U-pv[XgU5HnX8[X-U5v M g7f)areturns an iterator on database vectors. bs is the number of vectors per batch split = (nsplit, rank) means the dataset is split in nsplit shards and we want shard number rank The default implementation just iterates over the full matrix returned by get_dataset. rN)r)rrangemin rbssplitxbnsplitranki0i1j0s rdatabase_iteratorDataset.database_iterator'sg    46)477dQh+?6+IB#BRWb)* *$sA!A#c[5e)z6return the ground truth for k-nearest neighbor search rrks rget_groundtruthDataset.get_groundtruth5r"rc[5e)z)return the ground truth for range search r)rthreshs rget_groundtruth_rangeDataset.get_groundtruth_range9r"rc SURSURSURSURSUR3 $)Nzdataset in dimension z, with metric z , size: Q z B z T rrs r__str__Dataset.__str__=sD'x~dkk]K77)3twwis477)= >rcUR5RURUR4:XdeURS:a@UR SS9nURSUR4:XdSUR<35eUR 5RURUR4:XdeURSS9RURS4:Xdeg)z7runs the previous and checks the sizes of the matrices r{)r%zshape= )r;N) r shaperrrr&r)rr<)rxts r check_sizesDataset.check_sizesAs!''DGGTVV+<<<< 77Q;-B88TVV}, GBHH.G G,  "((TWWdff,====##b#)//DGGR=@@@r)rrrrrN)rr)__name__ __module__ __qualname____firstlineno____doc__rr r&r)r7r<r@rCrJ__static_attributes__rrr r s35$$$ +$$>Arr cB\rSrSrSrS SjrSrS SjrSrS Sjr S r g) SyntheticDatasetKzGA dataset that is not completely random but still challenging to index c[RU5 XX44uUlUlUlUlSnX2-U-n[ RRU5n U RX4S9n [ R"XRXq55n XRU5S-S--n [ R"U 5n U RS5n XPlU SUUlXX#-UlXU-SUlg)N )sizeg?float32)r rrrrrnprandom RandomStatenormaldotrandsinastyperrIr1xq) rrrrrrseedd1nrsxs rrSyntheticDataset.__init__Ps,-2M)$'  GbL YY " "4 ( IIA7I # FF1ggbn % a#% & FF1I HHY  CR&rw-GH+rcUR$rL)rfrs rr SyntheticDataset.get_queriesb wwrNcBUbUO URnURSU$rL)rrIr$s rr&SyntheticDataset.get_traines#'38wwy!!rcUR$rL)r1rs rr)SyntheticDataset.get_databaseirorc[URURUURS:Xa[R 5S$[R 5S$)Nrr)r rfr1rfaiss METRIC_L2METRIC_INNER_PRODUCTr:s rr< SyntheticDataset.get_groundtruthlsW GGTWWa#{{d2EOO    8=8R8R    r)rrrrrr1rfrI)ri:rL)d rOrPrQrRrSrr r&r)r<rTrUrrrWrWKs $" rrWz/datasets01/simsearch/041218/z7/mnt/vol/gfsai-flash3-east/ai-group/datasets/simsearch/z/home/z/simsearch/data/zdata/cUqgrL)dataset_basedir)paths rset_dataset_basedirr~sOrc>\rSrSrSrSrSrS SjrSrS Sjr S r g) DatasetSIFT1MzS The original dataset is available at: http://corpus-texmex.irisa.fr/ (ANN_SIFT1M) c[RU5 SuUlUlUlUl[ S-Ulg)N)rN順@B'zsift1M/r rrrrrr|basedirrs rrDatasetSIFT1M.__init__2,G)$'&2 rc2[URS-5$)Nzsift_query.fvecsrrrs rr DatasetSIFT1M.get_queries$,,);;<ww%B!BB 4<<,#>?IJJrc ## [URS-5nUupEURU-U-URUS--U-pv[XgU5Hn[ X8[ X-U55v M g7f)Nrr)rrrr,rr-r.s rr7DatasetDeep1B.database_iteratorso  |3 4 46)477dQh+?6+IB#B2#bgr"234 4$r)rrrrrr)rrLrMrrUrrrrs( /KMK5rrc8\rSrSrSrS SjrSrSrS SjrSr g) DatasetGloveiz< Data from http://ann-benchmarks.com/glove-100-angular.hdf5 Nc&SSKnU(aS5eU(d [S-nURUS5UlSUlSuUlUlURSRSUlURSRSUl g) Nrznot implementedzglove/glove-100-angular.hdf5rIP)ryrtraintest) h5pyr|File glove_h5pyrrrrHrr)rlocdownloadrs rrDatasetGlove.__init__ s}...|!$BBC))C-  //'*003//&)//2rcx[R"URS5n[R"U5 U$)Nrr^arrayrru normalize_L2rrfs rr DatasetGlove.get_queriess, XXdoof- . 2 rcx[R"URS5n[R"U5 U$)Nrrrr1s rr)DatasetGlove.get_databases, XXdoog. / 2 rcPURSnUbUS::deUSS2SU24nU$)N neighborsry)rrs rr<DatasetGlove.get_groundtruths6 __[ ) =8O8ArrEB r)rrrrrr)NFrL rOrPrQrRrSrr r)r<rTrUrrrrs 3  rrc4\rSrSrSrSrSrSrS SjrSr g) DatasetMusic100i'zC get dataset from https://github.com/stanis-morozov/ip-nsw#dataset c[RU5 SuUlUlUlUlSUl[S-Ulg)N)ryrrrrz music-100/) r rrrrrrr|rrs rrDatasetMusic100.__init__-s9,@)$' &5 rcn[R"URS-SS9nURSS5nU$)Nzquery_music100.binr]rrryr^fromfilerreshapers rr DatasetMusic100.get_queries3s1 [[(<\rSrSrSrSrSrS SjrSrS Sjr S r g) DatasetGIST1MiEzS The original dataset is available at: http://corpus-texmex.irisa.fr/ (ANN_GIST1M) c[RU5 SuUlUlUlUl[ S-Ulg)N)irrrzgist1M/rrs rrDatasetGIST1M.__init__Krrc2[URS-5$)Nzgist_query.fvecsrrs rr DatasetGIST1M.get_queriesPrrNcZUbUO URn[URS-5SU$)Nzgist_learn.fvecsrr$s rr&DatasetGIST1M.get_trainSrrc2[URS-5$)Nzgist_base.fvecsrrs rr)DatasetGIST1M.get_databaseWrrcb[URS-5nUbUS::deUSS2SU24nU$)Nzgist_groundtruth.ivecsryrrs rr<DatasetGIST1M.get_groundtruthZrrrrLrzrUrrrrErrrc\\rSrSrSrS SjrSrSSjrSrSSjr SS jr SS jr S r S r g)DatasetDINO10Biaa Data from https://dl.fbaipublicfiles.com/large_objects/dino_vitl_10B/ The dataset contains 10 billion 1024-d vectors extracted from image patches from the YFCC100M dataset, using a Dino-ViT-L 16 model (facebook/dinov3-vitl16-pretrain-lvd1689m). The dataset is sharded in multiple chunked .bvecs files. Downloading instructions can be obtained with "wget https://dl.fbaipublicfiles.com/large_objects/dino_vitl_10B/README.md". Supported sizes : 100k 200k 500k 1M ... 5B 10B listed in supported_nbs (see __init__). c[RU5 /SQnX;aU(d[SUSU35e[RR [ 5(d[S[ 35e[ S-UlURS-Ul[RR UR5(dSUR35eURS-Ul [RR UR5(dS US UR35eURS -S -[U5-S -S-Ul URS-Ul Xl SUlSUlSUlSUlg)N)ri@ i rii@KLri-1iri ieri5wlr Td( zUnsupported dataset size: z, supported values are: z0Provided dataset base directory does not exist: zdino_vitl_10B/chunked_base_10Bz2Index path should exist, check your dataset path: zqueries_clean.bvecsz*Queries path should exist as dataset size z is supported: zgts/gts_dino_patch__zk10.npyztrain_queries_99M.bvecsirir)r r ValueErrorosr}existsr|rindexdir queriesdirstrgtsdirtrain_queriesdirrrrrr)rrignore_supported supported_nbss rrDatasetDINO10B.__init__hsk] "+;9"=UVcUdef fww~~o..OP_O`ab b&)99  '99 ww~~dmm,,r0bcgcpcpbq.rr,,,)>>ww~~doo.. B2\]_\``optpppA1B B.llV+.??#b'ICOR[[ $ /H H rcB[UR5n[U5$)z!Get all vectors as a single array)rrr)rqueriess rr DatasetDINO10B.get_queries}sT__-  rNclUbUS:a [S5e[[UR5SU5$)z,Get training query vectors as a single arrayNrzThe training set is potentially too large to fit in RAM (400 GB of data). Please use train_iterator or use maxtrain parameter below 10_000_000 to get the first maxtrain training vectors.)rrrrr$s rr&DatasetDINO10B.get_trainsA  x*4%'cd d 4#8#89)8DEErcURS:a [S5e[[URURS9R 55$)z*Get all database vectors as a single arrayrzThe dataset is potentially too large to fit in RAM. Please use database_iterator or use a dataset size equal to or below 10_000_000. batch_size)rrrr r__next__rs rr)DatasetDINO10B.get_databasesG 77Z %'mn n.t}}QZZ\] ]rc## Sn[URUS9HcnX#RS-UR:aUSURU- n[ U5v X#RS- nX R:dMc g g7f)z_Iterator over the database of size nb, corresponding to the first nb vectors in the .bvecs filerr$N)r rrHrr)rr/ total_readbatchs rr7 DatasetDINO10B.database_iteratorst ' "EEKKN*TWW43tww345/ ! ++a. (JWW$ Fs A6B<Bc#\# [URUS9Hn[U5v M g7f)z;Iterator over all training query vectors in the .bvecs filer$N)rrr)rr/r*s rtrain_iteratorDatasetDINO10B.train_iterators' 5 5"EE5/ !Fs*,c~US:a [S5e[R"UR5nUSS2SU24nU$)zGet ground truth from .npy filerZz+Ground truth files only available for k<=10N)rr^rr)rr;gtss rr<DatasetDINO10B.get_groundtruths= r6%&ST Tggdkk"!RaR%j rcg)N euclideanrUrs rdistanceDatasetDINO10B.distancesr) rrrrrrrrrr)FrL)r)rZ)rOrPrQrRrSrr r&r)r7r-r<r4rTrUrrr r as1 *! F ^ " rr cJUS:Xa [5$US:Xa [5$URS5(aUS:XaSO [USS5n[ US9$URS 5(aWUS S nUSS :XaS [US S5-nO.US:XaSnO%USS:XaS[US S5-nO SU-5e[ US9$US:Xa [ 5$US:Xa [US9$URS5(aUS:XaSO [US S 5n[US9$[SU-5e)zconverts a string describing a dataset to a Dataset object Supports sift1M, bigann1M..bigann1B, deep1M..deep1B, music-100 and glove sift1Mgist1Mbigannbigann1Brr)rrr\NMrrrr;zdid not recognize suffix )rz music-100glove)rdinodino10Brzunknown dataset ) rr startswithintrrrrr  RuntimeError)datasetrdbsizeszsufs rdataset_from_namerFsH ( H    H % % J.C" 4F&))   F # #  9 s5":.F d]F 2Y# Ccr O+F =5= =5'' K   G X..   F # ##*i#7S=M((-788r)deep1MF)rnumpyr^rugetpassvecs_iorrrrrr exhaustive_searchr r rWgetuserusernamer|r}rr~rrrrrrrr rFrUrrrNs   dc"8A8Av% w% \ ??  (A  *+-O ww~~o&& -O G:4%5G%5P-5G-5` 7 Fg<G8GWGR'9r