Grid Storage System
From Hamsterworks Wiki!
I'm working on an grid storage subsystem optimized for utilizing the large network bandwidth and spare storage capacity in desktop PCs to provide a short-term reliable, high-capacity, high throughput for sequential storage - perfect for staging backups.
Storage is divided into three storage classes, two of which have subclasses.
- Reliable. These nodes are usually available 24x7 short outages are tolerated, but if a node is absent too long it will be ejected into a lower class. These are divided into two subclasses, allowing easy placement decisions to achieve RAID10
- (B) Blue
- (Y) Yellow
- On Demand. These nodes can usually be woken up as required using Wake On Lan. These are divided into three storage classes, and the system tries to have all data stored in the On Demand class triple mirrored.
- (R) Rock
- (P) Paper
- (S) Scissors
These nodes are still tracked, but data stored on these nodes needs to replicated into the other classes as soon as possible.
All blocks a node has ownership of is scored based on where they currently reside, and re-scored based on if an a copy is removed or created on a node in a different storage class. The highest scoring action is taken until no more improvements are possible. All storage decisions are made locally, avoiding the need for centralized control (although in the initial design a global block registry is maintained on the four most highly available nodes.
Blocks on offline "on demand" nodes are also scored (not sure who does this at the moment) and if required they are awoken).
Local decision logic
Storage pools are scored to see if they are under pressure for capacity. One consideration for this scoring is to allow sufficient capacity for node failures.
Once the overall state of the system is known, the blocks are stored based on the following table
|B||Y||R||P||S||Plenty of storage||short of reliable storage||short of on demand storage||short in both classes|
... another 28 scores need to be added.
An extra point is added when an extra copy is located in online unreliable storage, an extra 2 points when copies exist in the same On Demand storage class, and an extra five points if the copy is located in a reliable storage class.