20 node hadoop cluster with Cloudera Hadoop 5.12.0
| Project | Collection name | Total # of tweet | Started at | Collection tool | Analysis service |
|---|---|---|---|---|---|
| IDEAL | Archive DB | 2,069,322,393 | 2012 | yTK1) | |
| IDEAL | 1% sampling | 74,807,578 | 2015 | DMI-TCAT2) | Analysis |
| IDEAL | User following | 11,212,862 | 2015 | DMI-TCAT2) | Analysis |
| IDEAL | Keyword tracking | 21,646,879 | 2015 | DMI-TCAT2) | Analysis |
| GETAR | Collection | 341,852,272 | 2015 | yTK1) | |
| GETAR | Collection | 2,325,125,326 | 2016.9 | SFM3) | |
| NIH | Keyword tracking | 660,660 | 2015 | DMI-TCAT2) | Analysis |
| Total | Google Table | 4,844,627,970 |
Open source tools for collecting tweets
| Project | Collection name | Hosted by | Service | Location |
|---|---|---|---|---|
| IDEAL | IA webpage collection | Internet Archive | Archive-it | IA link |
| IDEAL | IA webpage collection (downloaded) |
Virginia Tech (DLRL) |
Hadoop | /data/IACollections in head node |