User Tools

Site Tools


wiki:ponyland:storage

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Last revision Both sides next revision
wiki:ponyland:storage [2019/05/15 17:00]
wessel created
wiki:ponyland:storage [2019/05/16 14:02]
wessel
Line 3: Line 3:
 === The problem === === The problem ===
  
-If you have a lot (=thousands) of small items to store (experimental sentences, tweets, audio fragments, individual calculations,​ etc), it might be tempting to store them as individual files so the file system more or less reflects your data structure. This has the important downside, however, that the file system will become incredibly slow: a simple command like ''​ls''​ is no longer instant, loading your data with a script might take ages, and most importantly:​ it will be practically impossible to move around your files, for example if they need to be moved to another disk for maintenance reasons.+If you have a lot (=**thousands**) of small (**less than 100 kB**) items to store (experimental sentences, tweets, audio fragments, individual calculations,​ etc), it might be tempting to store them as individual files so the file system more or less reflects your data structure. This has the important downside, however, that the file system will become incredibly slow: a simple command like ''​ls''​ is no longer instant, loading your data with a script might take ages, and most importantly:​ it will be practically impossible to move around your files, for example if they need to be moved to another disk for maintenance reasons.
  
 === The solution === === The solution ===
Line 11: Line 11:
   - Store everything in larger files, where each line is one item   - Store everything in larger files, where each line is one item
   - Use an sqlite or mysql database. Mysql databases can be requested via the [[admin]].   - Use an sqlite or mysql database. Mysql databases can be requested via the [[admin]].
 +
 +=== Further reading ===
 +
wiki/ponyland/storage.txt ยท Last modified: 2019/05/16 14:02 by wessel