Mercurial > hg > silvermirror

comparison silvermirror-whitepaper.txt @ 14:5f95af14b51c

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
notes
author Jeff Hammel <jhammel@mozilla.com>
date Sun, 24 Mar 2013 12:46:35 -0700
parents 584a847d2491
children
comparison
equal deleted inserted replaced
13:584a847d2491 14:5f95af14b51c
1 SilverMirror Whitepaper 1 = SilverMirror Whitepaper =
2 2
3 It is necessary to maintain parallel directory structures of various 3 It is necessary to maintain parallel directory structures of various
4 resources across an arbitrary number of computers. The traditional 4 resources across an arbitrary number of computers. The traditional
5 approach is the central server model, where files live in one 5 approach is the central server model, where files live in one
6 canonical location and the network is used to give access to the data. 6 canonical location and the network is used to give access to the data.
10 10
11 Distributed version control, often of nominal use (in the case where a 11 Distributed version control, often of nominal use (in the case where a
12 canonical trunk exists) is ideally suited to provide mirroring of 12 canonical trunk exists) is ideally suited to provide mirroring of
13 desired resources across computers. 13 desired resources across computers.
14 14
15 Implementation 15 == Implementation ==
16 16
17 A front end to a DVCS - most likely mercurial but potentially bzr - 17 A front end to a DVCS or other - unison is completed,
18 mercurial is up next -
18 will be written to keep resources in sync across an arbitrary number 19 will be written to keep resources in sync across an arbitrary number
19 of computers. The front end, called SilverMirror, may be used to push 20 of computers. The front end, called SilverMirror, may be used to push
20 or pull changes to resources. Optionally, a daemon will monitor 21 or pull changes to resources. Optionally, a daemon will monitor
21 changes to resources and push or pull changes at desired intervals. 22 changes to resources and push or pull changes at desired intervals.
22 23
26 versioned, any change to the resource's directory structure should be 27 versioned, any change to the resource's directory structure should be
27 mirrored across machines without user intervention. Files matching a 28 mirrored across machines without user intervention. Files matching a
28 pattern may be ignored, either globally or on a per resource basis, 29 pattern may be ignored, either globally or on a per resource basis,
29 for the purpose of versioning. 30 for the purpose of versioning.
30 31
31 Configuration 32 == Configuration ==
32 33
33 SilverMirror is configured via an INI file containing a section for 34 SilverMirror is configured via an INI file containing a section for
34 each resource and a section for application configuration. 35 each resource and a section for application configuration.
35 36
36 The main section, denoted [::SilverMirror::], has the following options: 37 The main section, denoted [::SilverMirror::], has the following options:
59 * hosts: hosts to push/pull from 60 * hosts: hosts to push/pull from
60 61
61 In order to ensure coherency among resources, all relevant 62 In order to ensure coherency among resources, all relevant
62 configuration options must be synced prior to push/pull transactions. 63 configuration options must be synced prior to push/pull transactions.
63 64
64 Default Configuration: 65 Default Configuration::
65 66
66 [::SilverMirror::] 67 [::SilverMirror::]
67 conflict = ClobberRemote 68 conflict = ClobberRemote
68 69
69 Example of a more complex configuration: 70 Example of a more complex configuration::
70 71
71 [::SilverMirror::] 72 [::SilverMirror::]
72 conflict.push = ClobberRemote 73 conflict.push = ClobberRemote
73 conflict.pull = ClobberLocal 74 conflict.pull = ClobberLocal
74 75
75 Push 76 == Push ==
76 77
77 Push changes to remote resources. When resources are pushed, first 78 Push changes to remote resources. When resources are pushed, first
78 changes are pulled from each remote host in turn, conflicts between 79 changes are pulled from each remote host in turn, conflicts between
79 local and remote changes are resolved (see Behavior on Conflicts), 80 local and remote changes are resolved (see Behavior on Conflicts),
80 then local modifications are pushed. This is done to keep the 81 then local modifications are pushed. This is done to keep the
83 When new files are added to the resource they are automatically added 84 When new files are added to the resource they are automatically added
84 to the hg repository. When resource files are edited the changes are 85 to the hg repository. When resource files are edited the changes are
85 pushed to the repository. When a conflict occurs between local 86 pushed to the repository. When a conflict occurs between local
86 resources and remote resources, the conflict handler is used. 87 resources and remote resources, the conflict handler is used.
87 88
88 Pull 89 == Pull ==
89 90
90 Get changes to the cloud filesystem resources. If no host is 91 Get changes to the cloud filesystem resources. If no host is
91 specified, pull changes from all known + accessible hosts. 92 specified, pull changes from all known + accessible hosts.
92 93
93 Namespaced Resources 94 == Namespaced Resources ==
94 95
95 It is possible to maintain versioning of a subdirectory within a 96 It is possible to maintain versioning of a subdirectory within a
96 resource. 97 resource.
97 98
98 Example: 99 Example::
99 100
100 [docs] 101 [docs]
101 directory = /path/to/docs 102 directory = /path/to/docs
102 103
103 [docs:private] 104 [docs:private]
104 105
105 This configuration snippet describes a resource, [docs:private], 106 This configuration snippet describes a resource, [docs:private],
106 namespaced within the [docs] resource. [docs:private] inherits 107 namespaced within the [docs] resource. [docs:private] inherits
107 configuration and behavior from [docs] but may be dealt with 108 configuration and behavior from [docs] but may be dealt with
108 separately. For example, some computers in the cloud may not have 109 separately. For example, some computers in the cloud may not have
118 from its namespace (private) and the directory of its parent resource. 119 from its namespace (private) and the directory of its parent resource.
119 So its base directory is /path/to/docs/private . If a relative path 120 So its base directory is /path/to/docs/private . If a relative path
120 was specified in the directory option of the [docs:private] section, 121 was specified in the directory option of the [docs:private] section,
121 it would be joined with the base directory of [docs]. 122 it would be joined with the base directory of [docs].
122 123
123 Behavior on Conflicts 124 == Behavior on Conflicts ==
124 125
125 Conflict handlers are set via setuptools entry points. Several 126 Conflict handlers are set via setuptools entry points. Several
126 conflict handlers are provided with SilverMirror: 127 conflict handlers are provided with SilverMirror:
127 128
128 * ClobberLocal: replace local changes with changes from remote files 129 * ClobberLocal: replace local changes with changes from remote files
135 136
136 The conflict handler may also be specified from the command line: 137 The conflict handler may also be specified from the command line:
137 138
138 silvermirror push -d ClobberRemote 139 silvermirror push -d ClobberRemote
139 140
140 Command Line Usage 141 == Command Line Usage ==
141 142
142 silvermirror [push|pull] [resource] [options] 143 silvermirror [push|pull] [resource] [options]
143 144
144 In the simplest invocation, SilverMirror is used with no command line 145 In the simplest invocation, SilverMirror is used with no command line
145 arguments: 146 arguments::
146 147
147 silvermirror 148 silvermirror
148 149
149 This pushes changes of the resource as determined by the current 150 This pushes changes of the resource as determined by the current
150 working directory after pulling outstanding changes from all 151 working directory after pulling outstanding changes from all
151 applicable remote computers and invoking the conflict handler for 152 applicable remote computers and invoking the conflict handler for
152 push. If the current working directory is not within a resource, all 153 push. If the current working directory is not within a resource, all
189 SilverMirror does not desire to duplicate versioning on directories 190 SilverMirror does not desire to duplicate versioning on directories
190 already under version control (svn, bzr, hg). These resources are 191 already under version control (svn, bzr, hg). These resources are
191 automatically ignore. In a future implementation, these resources 192 automatically ignore. In a future implementation, these resources
192 would optionally be checked out or updated upon a pull. 193 would optionally be checked out or updated upon a pull.
193 194
194 Automatic Syncronization 195 == Automatic Syncronization ==
195 196
196 SilverMirror includes a script that will automatically invoke 197 SilverMirror includes a script that will automatically invoke
197 syncronizing the resources in a specified period of time. This daemon, 198 syncronizing the resources in a specified period of time. This daemon,
198 called silvermirrord, is invoked from the command line with options 199 called silvermirrord, is invoked from the command line with options
199 parallel to the silvermirror program. One additional option, -s, tells 200 parallel to the silvermirror program. One additional option, -s, tells
203 is noninteractive, otherwise the daemon will hang forever. 204 is noninteractive, otherwise the daemon will hang forever.
204 205
205 As an alternative, the silvermirror program may be invoked from a cron 206 As an alternative, the silvermirror program may be invoked from a cron
206 job. 207 job.
207 208
208 Future Work 209 == Events ==
210
211 SilverMirror [TODO] may be hooked up to an event listener for
212 filesystem changes and sync on them.
213
214 == Future Work ==
209 215
210 SilverMirror implements a cloud filesystem which may be accessed 216 SilverMirror implements a cloud filesystem which may be accessed
211 nearly transparently by an arbitrary number of computers. Several 217 nearly transparently by an arbitrary number of computers. Several
212 improvements could extend SilverMirror to solve several deficiencies 218 improvements could extend SilverMirror to solve several deficiencies
213 of modern filesystem. 219 of modern filesystem.
233 SilverMirror backup scheme on portable media. This not only 239 SilverMirror backup scheme on portable media. This not only
234 includes a backup of the files, but also the SilverMirror program and 240 includes a backup of the files, but also the SilverMirror program and
235 all of the files necessary to create a new SilverMirror node. 241 all of the files necessary to create a new SilverMirror node.
236 242
237 * SilverMirror + Firefox Sync 243 * SilverMirror + Firefox Sync
244
245 == Vision ==
246
247 Essentially, SilverMirror is intended as a portable filesystem on a
248 filesystem type interface.